BAA1030 - Data Analytics and Story Telling

.title[
# BAA1030 - Data Analytics and Story Telling
]
.subtitle[
## Lecture 9: Visualisation in python with plotnine
]
.author[
### Damien Dupré - Dublin City University
]

---

# Python so far

We have already seen how to:

- Subset values with `pl.filter()`
- Subset columns with `pl.select()`
- Compute new columns with `pl.with_columns()`
- Aggregate be per using `pl.group_by()` and `agg()`

More importantly we have seen how to chain multiple transformations.

Let's start to to make some fancy visualisations!

---

# Google Colab

In your webrowser (Chrome, Firefox, ...):

1. Open these same slides on a tab to copy-paste the examples

- From Loop: Lectures > Lecture 9
  - Or from the URL: https://damien-dupre.github.io/BAA1030/lectures/lecture_9

2. In another tab, go to: https://colab.research.google.com

- Sign in or Sign up (if not already done)
  - In your workspace, Click "New Notebook" (if not already done)

---

# The gapminder Dataset

The dataset used today is called `gapminder`.

Each row in this table corresponds to a country at a specific year. For each row, we have 6 columns:

- **country**: Name of country.
- **year**: Year of the observation (between 1952 and 2007).
- **pop**: Number of people living in the country.
- **continent**: Which of the five continents the country is part of. 
- **lifeExp**: Life expectancy in years.
- **gdpPercap**: Gross domestic product (in US dollars).

to use the `gapminder` dataset, use the function `pl.read_csv` on the url:

> https://raw.githubusercontent.com/damien-dupre/damien-dupre.github.io/refs/heads/main/gapminder.csv

``` python
import polars as pl
gapminder=pl.read_csv("https://raw.githubusercontent.com/damien-dupre/damien-dupre.github.io/refs/heads/main/gapminder.csv")
```

---

# 1. Basic Visualistions with plotnine

---

# Using plotnine

The plotnine package provides an **easy way to create and to customise your plots**.

Most used packages would include Matplotlib or Seaborn. However plotnine is based on the Grammar of Graphics, which makes is much easier to use and powerful.

``` python
from plotnine import *
```

**“the grammar of graphics”** defines a set of rules for constructing statistical graphics by combining different types of layers.

From the first (top) to the last (bottom), the mandatory layers are :

---

# Data

In plotnine, the **data** layer is the name of the object that contains the variables to plot.

The visualisation is initiated with the function `ggplot()`

``` python
(ggplot(data=my_dataframe) # the argument name can be omitted too
)
```

Let's try to use the gapminder data:

``` python
(ggplot(data=gapminder) # or ggplot(gapminder)
)
```

Only a grey frame is displayed! We need to define the axes (aesthetic mapping) and their corresponding layers.

**Let's add additional layers with the symbol `+`.**

---

# Aesthetic Mapping

The **aesthetic mapping** refers to the frame of the plot, `x` for the variable in the x-axis and `y` for the variable in the y-axis. There are many more aesthetics such as `color`, `fill` and `group` but let's focus on the x and y axes for the moment. The mapping is performed by a function called `aes()` for aesthetics.

This is an essential concept, the mapping of a plot (frame axes) is built from aesthetics.

``` r
(ggplot(data=my_dataframe) 
  + aes(x="my_xaxis_variable", y="my_yaxis_variable")
)
```

The arguments names of `aes()` can be omitted if filled in the right order:

``` r
(ggplot(my_dataframe) 
  + aes("my_xaxis_variable", "my_yaxis_variable")
)
```

Obviously, some type of layers are not suitable for all aesthetics, it depends on how many variables are included and their type (categorical or continuous).

---

# Aesthetic Mapping

.pull-left[
**Aesthetic mapping represents** not only the variables defined as **x-axis** and **y-axis** but also, **colours** of borders (colors), colours of shapes (fill), **shapes**, **size**, ...
]

``` python
(ggplot(data=my_dataframe)
  + aes(
      x="my_xaxis_variable", 
      y="my_yaxis_variable",
      color="my_color_variable",
      fill="my_fill_variable",
      shape="my_shape_variable",
      size="my_size_variable"
  )  
)
```
]

The following code is adding not only a x- and y-axis to a plot, but also colours to the points:

``` python
(ggplot(data=gapminder)
  + aes(
     x="gdpPercap", 
     y="lifeExp", 
     color="continent"
  )
)
```
]

.pull-right[
<img src="lecture_9_files/figure-html/unnamed-chunk-13-1.png" style="display: block; margin: auto;" />
]

---

# Geometries

**Geometries**, which are shapes we use to represent our data. There is a dedicated function for every type of shape to plot but all start with `geom_`

|function          |shape    |
|------------------|---------|
|`geom_point()`    |point    |
|`geom_line()`     |line     |
|`geom_bar()`      |bar      |
|`geom_histogram()`|histogram|
|`geom_boxplot()`  |boxplot  |
|...               |...      |

See here the exhaustive list of all available geometry functions: <https://plotnine.org/reference/#geoms>

<span><i class="fas  fa-arrow-circle-right faa-horizontal animated faa-slow " style=" color:blue;"></i></span> Note: Not all data is suitable for all types of geometries. You have to find the geometry that corresponds to your data.

---

# Geometries

Example for **scatter plots**:

``` python
(ggplot(data=my_dataframe)
  + aes(x="my_xaxis_variable", y="my_yaxis_variable") 
  + geom_point() 
)
```

Example for **line graph**:

``` python
(ggplot(data=my_dataframe)
  + aes(x="my_xaxis_variable", y="my_yaxis_variable") 
  + geom_line()
)
```

Example for **bar graph**:

``` python
(ggplot(data=my_dataframe)
  + aes(x="my_xaxis_variable", y="my_yaxis_variable") 
  + geom_col()
)
```

---

# Geometries Applied to gapminder

Example for **scatter plots**:

``` python
(ggplot(data=gapminder)
  + aes(x="gdpPercap", y="lifeExp", color="continent")
  + geom_point()
)
```

---

# Geometries Applied to gapminder

Example for **line graph**:

``` python
(ggplot(data=gapminder)
  + aes(x="year", y="lifeExp", color="country")
  + geom_line()
  + guides(color="none") # removes legend because too many countries
)
```

---

# Geometries Applied to gapminder

Example for **bar graph**:

``` python
gapminder_avg_continent_2007 = (gapminder
  .filter(pl.col("year") == 2007)
  .group_by(pl.col("continent"))
  .agg(pl.col("lifeExp").mean().alias("m_lifeExp"))
)

(ggplot(data=gapminder_avg_continent_2007)
  + aes(x="continent", y="m_lifeExp", fill="continent")
  + geom_col() 
)
```

---
class: title-slide, middle

## Live Demo

---
class: title-slide, middle

## Exercise

Create:

- A `ggplot()` layer with the `gapminder` data,
- An `aes()` layer containing `continent` as x, `lifeExp` as y, and `continent` as color,
- And a `geom_boxplot()` layer:

``` python
(_ _ _(_ _ _)
  + aes(x=_ _ _, y=_ _ _, color=_ _ _)
  + _ _ _()
)
```

- A `ggplot()` layer with the `gapminder` data,
- An `aes()` layer containing `year` as x, `pop` as y, and `continent` as fill,
- And a `geom_col()` layer:

``` python
(_ _ _(_ _ _)
  + aes(x=_ _ _, y=_ _ _, fill=_ _ _)
  + _ _ _()
)
```

---
class: inverse, mline, center, middle

# 2. Advanced Visualistions with plotnine

---

# Inherited Propriety of Geometries

You can add as many geometry layers as you want, however repeating the mapping for each geometry layer is very redundant.

Thankfully, if all your geometry layers are using the same aesthetics mapping, __it is possible to include this mapping inside the `ggplot()`__, then all the geometry layers will have the same mapping:

``` python
(ggplot(gapminder, aes(x="year", y="lifeExp", color="country")) 
  + geom_point() 
  + geom_line()
)
```

If aesthetics are different for several geometry layers, it is also __possible to declare the aesthetics in the geometry__:

``` python
(ggplot(gapminder) 
  + geom_point(aes(x="year", y="lifeExp", shape="continent")) 
  + geom_line(aes(x="year", y="lifeExp", color="country"))
)
```

---

# Themes

Now, to make the plot more professional, let's remove that standard grey background using a different theme.

Many themes come built into the plotnine package. My preference is `theme_bw()` but once you start typing `theme_` a list of options will pop up.

``` python
(ggplot(gapminder)
  + aes(x="gdpPercap", y="lifeExp", color="continent")
  + geom_point()
  + theme_bw()
)
```

---

# Themes

- `theme_538`
- `theme_bw`
- `theme_classic`
- `theme_dark`
- `theme_gray`
- `theme_light`
- `theme_linedraw`
- `theme_matplotlib`
- `theme_minimal`
- `theme_seaborn`
- `theme_tufte`
- `theme_void`
- `theme_xkcd`
]

.pull-right[
<img src="https://sakaluk.wordpress.com/wp-content/uploads/2016/02/4282183.jpg" style="display: block; margin: auto;" />
]

---

# Facets

Faceting is used to __split a particular visualisation by the values of another variable__.

This will create multiple copies of the same type of plot with matching x and y axes, but whose content will differ.

For example, suppose we were interested in looking at the evolution of life expectancy by continent from 1952. We could “split” this figure for each continent. In other words, we would plot a scatter plot for each continent separately.

We do this by adding `facet_wrap("continent")` layer.

---

# Facets

``` python
(ggplot(gapminder)
  + aes(x="year", y="lifeExp", color="continent")
  + geom_point()
  + facet_wrap("continent")
  + theme_classic()
)
```

---

# Facets

We can also specify the number of rows and columns in the grid by using the `nrow` and `ncol` arguments inside of `facet_wrap()`.

For example, suppose we would like our faceted figure to have 1 rows instead of 2. We simply add an `nrow=1` argument to `facet_wrap("continent")`:

``` python
(ggplot(gapminder)
  + aes(x="year", y="lifeExp", color="continent")
  + geom_point()
  + facet_wrap("continent", nrow=1)
  + theme_classic()
)
```

---

# Labels

Ggplot has a layer called `labs()` in order to change the name of the axis labels very quickly.

``` python
(ggplot(gapminder)
  + aes(x="year", y="lifeExp", color="continent")
  + geom_point()
  + facet_wrap("continent", nrow=1)
  + labs(
      x="Year (from 1952 to 2007)",
      y="Life Expectancy",
      title="Evolution of life expectancy from 1952 to 2007 per continent."
  )
  + theme_classic()
)
```

`labs()` can rename x, y and title as well as many more texts like colour, subtitle, or caption

---

# Statistics and Special Effects

Instead of creating summaries inside the data frame object, ggplot has some function to calculate and display them automatically.

The first special effect is the `geom_smooth()` layer. `geom_smooth()` is a classic geometry layer but which displays linear and non-linear trends.

``` python
(ggplot(gapminder)
  + aes(x="year", y="lifeExp", color="continent")
  + geom_point()
  + geom_smooth()
  + facet_wrap("continent", nrow=1)
  + theme_classic()
)
```

---

# Statistics and Special Effects

`geom_smooth()` has one important extra argument called `method`. If method has the value __"lm"__, a linear regression will be shown. If method has the value __"loess"__ or __"gam"__, a non-linear regression will be shown.

``` python
(ggplot(gapminder)
  + aes(x="year", y="lifeExp", color="continent")
  + geom_point()
  + geom_smooth(method="lm")
  + facet_wrap("continent", nrow=1)
  + theme_classic()
)
```

---

# Additional Options

``` python
(ggplot(gapminder)
  + aes(x="year", y="lifeExp", color="continent")
  + geom_point(alpha=0.2, size=1)
  + geom_smooth(method="lm")
  + facet_wrap("continent", nrow=1)
  + scale_x_continuous(breaks=[1960, 1980, 2000])
  + labs(
      x="Year (from 1952 to 2007)",
      y="Life Expectancy",
      title="Evolution of life expectancy from 1952 to 2007 per continent."
  )
  + theme_classic()
  + theme(text=element_text(size=20))
  + guides(color="none")
)
```

---

# Additional Options

---

# Map Visualisations

You will need a new package to read the world shape file:

``` python
import geopandas as gp
```

To do a Map visualisation with plotnine, we __need the shape__ of each country.

``` python
shape_world = gp.read_file("https://public.opendatasoft.com/api/explore/v2.1/catalog/datasets/world-administrative-boundaries/exports/shp")
```

``` python
(ggplot()
  + geom_map(shape_world)
)
```
]

.pull-right[
<img src="lecture_9_files/figure-html/unnamed-chunk-46-13.png" style="display: block; margin: auto;" />
]

---

# Map Visualisations

To obtain colours that are matching values of a specific variable, these variables need to be joined to the original data that we want to display on the map. `geom_map()` will use these variables to fill the countries' with a gradient of colour.

For example, let's display the population for each country in 2007:

``` python
pop_2007 = (gapminder
  .filter(pl.col("year") == 2007)
)

map_pop_2007 = (shape_world
  .merge(gapminder.to_pandas(), left_on="name", right_on="country", how="outer")
)

(ggplot(data=map_pop_2007)
  + aes(fill="pop")
  + geom_map()
  + coord_fixed()
)
```

<span><i class="fas  fa-exclamation-triangle faa-flash animated faa-slow " style=" color:red;"></i></span> Do NOT merge shape_world with an objects non-filtered and do NOT use the resulting object for any other visualisation.

---

# Map Visualisations

Use the `coord_fixed()` layer to keep relationship between latitude and longitude scales

---

# Map Visualisations

`gapminder` and the `shape_world` objects have different spelling for the countries. For example "United States of America" can be "USA" or just "United States".

To solve this, we can merge them with a new object providing ISO codes for countries regardless of their spelling, and that you can use to match them across datasets.

``` python
# 1. Read the csv file containing the iso3 convention for all spellings
country_code = pl.read_csv("https://raw.githubusercontent.com/adrivsh/country_names/refs/heads/master/names_to_iso.csv")
# 2. Join country_code and gapminder to identify the iso3 code for each country
gapminder_code = (gapminder.join(country_code, on="country", how="left"))
# 3. Filter gapminder_code to have only values for 2007
gapminder_code_2007 = (gapminder_code.filter(pl.col("year") == 2007))
# 4. Merge the values with the shape file for each country using the iso3
shape_gapminder_code_2007 = (shape_world
  .merge(gapminder_code_2007.to_pandas(), on="iso3", how="outer")
)
# 5. Plot the map
(ggplot(data=shape_gapminder_code_2007)
  + aes(fill="pop")
  + geom_map()
  + coord_fixed()
)
```

---

# Map Visualisations

---

# Map Visualisations

``` python
(ggplot(data=shape_gapminder_code_2007)
  + aes(fill="pop")
  + geom_map()
  + scale_fill_gradient(low="yellow", high="red")
  + labs(
      title="Differences between countries regarding their population in 2007",
      subtitle="Countries in grey have no data due to a mismatch with their names",
      caption="Source: gapminder",
      x="Longitude",
      y="Latitude",
      fill="Country Population"
  )
  + coord_fixed()
  + theme_bw()
)
```

---

# Map Visualisations

---
class: title-slide, middle

## Live Demo

---
class: title-slide, middle

## Exercise

Build a ggplot with the `gapminder` dataset and one `aes()` layer which contains the x as `continent`, y as `lifeExp` and colour as `continent`. Use `geom_boxplot()` as geometry and `year` as a facet variable, use a theme of your choice as well and changes axis labels:

``` python
(ggplot(...
```

Build a ggplot with the `gapminder` dataset and:

- One `geom_line()` layer which contains the x as `year`, y as `lifeExp` and group as `country` in its `aes()`,
- One `geom_smooth()` layer which contains the x as `year`, y as `lifeExp` and colour as `continent` in its `aes()` ,
- One `facet_wrap()` for each continent.

``` python
(ggplot(_ _ _)
  + geom_line(aes(_ _ _))
  + geom_smooth(aes(_ _ _))
  + facet_wrap(_ _ _) 
)
```

---
class: inverse, mline, center, middle

# 3. Combine ggplot() and Transformations

---

# Combine ggplot() and Transformations

A very powerful way to create figures is to use a __ggplot at the end of data transformations__.

Indeed, having a data frame object as first argument of the `ggplot()` function is similar to chaining it to the `ggplot()` function using `>>`:

``` python
# this classic representation:
(ggplot(data=gapminder)
  + aes(x="gdpPercap", y="lifeExp", color="continent")
  + geom_point()
)
# is the same as:
(gapminder
  >>
  ggplot()
  + aes(x="gdpPercap", y="lifeExp", color="continent")
  + geom_point()
)
# is the same as:
(gapminder
  >>
  ggplot(aes(x="gdpPercap", y="lifeExp", color="continent"))
  + geom_point()
)
```

The layers are still added with the `+` symbol.

---

# filter() to ggplot()

You can easily display only the data for a specific section of you interest. For example, let's filter the data only for Ireland:

``` python
(gapminder
  .filter(pl.col("country") == "Ireland")
  >>
  ggplot(aes("year", "lifeExp", color="country"))
  + geom_line()
)
```

---

# with_columns() to ggplot()

If you need to display a variable that has to be created beforehand, you can always include a mutate statement in the chain.

For example, let's create the variable `gdpPercountry` which is the result of the multiplication between countries' population and countries' gdpPercap. Then let's display this information for Ireland and France:

``` python
(gapminder
  .with_columns(gdpPercountry=pl.col("gdpPercap")*pl.col("pop"))
  .filter(pl.col("country") == "Ireland") 
  >>
  ggplot(aes("gdpPercountry", "lifeExp", color="country"))
  + geom_line()
)
```

---

# group_by() and agg() to ggplot()

Finally, one of the most useful possibilities will be to aggregate variables per groups and to display this information in figures.

For example, let's create the average population per continent and display how this average evolves with time:

``` python
(gapminder
  .group_by(pl.col("year"), pl.col("continent"))
  .agg(pl.col("pop").mean().alias("m_pop"))
  >>
  ggplot(aes("year", "m_pop", color="continent"))
  + geom_line()
)
```

---

# group_by() and agg() to ggplot()

We can also compare the sum of the population by continent for the year 2007:

``` python
(gapminder
  .filter(pl.col("year") == 2007)
  .group_by(pl.col("continent"))
  .agg(pl.col("pop").sum().alias("s_pop"))
  >>
  ggplot(aes("continent", "s_pop", fill="continent")) 
  + geom_col()
)
```

---

# Thanks for your attention and don't hesitate to ask if you have any question!

[<svg aria-hidden="true" role="img" viewBox="0 0 496 512" style="height:1em;width:0.97em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:currentColor;overflow:visible;position:relative;"><path d="M165.9 397.4c0 2-2.3 3.6-5.2 3.6-3.3.3-5.6-1.3-5.6-3.6 0-2 2.3-3.6 5.2-3.6 3-.3 5.6 1.3 5.6 3.6zm-31.1-4.5c-.7 2 1.3 4.3 4.3 4.9 2.6 1 5.6 0 6.2-2s-1.3-4.3-4.3-5.2c-2.6-.7-5.5.3-6.2 2.3zm44.2-1.7c-2.9.7-4.9 2.6-4.6 4.9.3 2 2.9 3.3 5.9 2.6 2.9-.7 4.9-2.6 4.6-4.6-.3-1.9-3-3.2-5.9-2.9zM244.8 8C106.1 8 0 113.3 0 252c0 110.9 69.8 205.8 169.5 239.2 12.8 2.3 17.3-5.6 17.3-12.1 0-6.2-.3-40.4-.3-61.4 0 0-70 15-84.7-29.8 0 0-11.4-29.1-27.8-36.6 0 0-22.9-15.7 1.6-15.4 0 0 24.9 2 38.6 25.8 21.9 38.6 58.6 27.5 72.9 20.9 2.3-16 8.8-27.1 16-33.7-55.9-6.2-112.3-14.3-112.3-110.5 0-27.5 7.6-41.3 23.6-58.9-2.6-6.5-11.1-33.3 2.6-67.9 20.9-6.5 69 27 69 27 20-5.6 41.5-8.5 62.8-8.5s42.8 2.9 62.8 8.5c0 0 48.1-33.6 69-27 13.7 34.7 5.2 61.4 2.6 67.9 16 17.7 25.8 31.5 25.8 58.9 0 96.5-58.9 104.2-114.8 110.5 9.2 7.9 17 22.9 17 46.4 0 33.7-.3 75.4-.3 83.6 0 6.5 4.6 14.4 17.3 12.1C428.2 457.8 496 362.9 496 252 496 113.3 383.5 8 244.8 8zM97.2 352.9c-1.3 1-1 3.3.7 5.2 1.6 1.6 3.9 2.3 5.2 1 1.3-1 1-3.3-.7-5.2-1.6-1.6-3.9-2.3-5.2-1zm-10.8-8.1c-.7 1.3.3 2.9 2.3 3.9 1.6 1 3.6.7 4.3-.7.7-1.3-.3-2.9-2.3-3.9-2-.6-3.6-.3-4.3.7zm32.4 35.6c-1.6 1.3-1 4.3 1.3 6.2 2.3 2.3 5.2 2.6 6.5 1 1.3-1.3.7-4.3-1.3-6.2-2.2-2.3-5.2-2.6-6.5-1zm-11.4-14.7c-1.6 1-1.6 3.6 0 5.9 1.6 2.3 4.3 3.3 5.6 2.3 1.6-1.3 1.6-3.9 0-6.2-1.4-2.3-4-3.3-5.6-2z"/></svg> @damien-dupre](http://github.com/damien-dupre)  
[<svg aria-hidden="true" role="img" viewBox="0 0 640 512" style="height:1em;width:1.25em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:currentColor;overflow:visible;position:relative;"><path d="M579.8 267.7c56.5-56.5 56.5-148 0-204.5c-50-50-128.8-56.5-186.3-15.4l-1.6 1.1c-14.4 10.3-17.7 30.3-7.4 44.6s30.3 17.7 44.6 7.4l1.6-1.1c32.1-22.9 76-19.3 103.8 8.6c31.5 31.5 31.5 82.5 0 114L422.3 334.8c-31.5 31.5-82.5 31.5-114 0c-27.9-27.9-31.5-71.8-8.6-103.8l1.1-1.6c10.3-14.4 6.9-34.4-7.4-44.6s-34.4-6.9-44.6 7.4l-1.1 1.6C206.5 251.2 213 330 263 380c56.5 56.5 148 56.5 204.5 0L579.8 267.7zM60.2 244.3c-56.5 56.5-56.5 148 0 204.5c50 50 128.8 56.5 186.3 15.4l1.6-1.1c14.4-10.3 17.7-30.3 7.4-44.6s-30.3-17.7-44.6-7.4l-1.6 1.1c-32.1 22.9-76 19.3-103.8-8.6C74 372 74 321 105.5 289.5L217.7 177.2c31.5-31.5 82.5-31.5 114 0c27.9 27.9 31.5 71.8 8.6 103.9l-1.1 1.6c-10.3 14.4-6.9 34.4 7.4 44.6s34.4 6.9 44.6-7.4l1.1-1.6C433.5 260.8 427 182 377 132c-56.5-56.5-148-56.5-204.5 0L60.2 244.3z"/></svg> https://damien-dupre.github.io](https://damien-dupre.github.io)  
[<svg aria-hidden="true" role="img" viewBox="0 0 512 512" style="height:1em;width:1em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:currentColor;overflow:visible;position:relative;"><path d="M16.1 260.2c-22.6 12.9-20.5 47.3 3.6 57.3L160 376V479.3c0 18.1 14.6 32.7 32.7 32.7c9.7 0 18.9-4.3 25.1-11.8l62-74.3 123.9 51.6c18.9 7.9 40.8-4.5 43.9-24.7l64-416c1.9-12.1-3.4-24.3-13.5-31.2s-23.3-7.5-34-1.4l-448 256zm52.1 25.5L409.7 90.6 190.1 336l1.2 1L68.2 285.7zM403.3 425.4L236.7 355.9 450.8 116.6 403.3 425.4z"/></svg> damien.dupre@dcu.ie](mailto:damien.dupre@dcu.ie)