Who am I?

Laurent Bergé

• Assistant Prof., BxSE, Univ. of Bordeaux
• @lrberge
  
• laurent.berge@u-bordeaux.fr
  

What do I do?

My fields

• Applied economics ( $=$ Data + methods)
• Economics of Innovation ( $=$ Large data)
• (a bit of) Statistics ( $=$ Computational methods)

R and me

This course

Data analysis II

1. Data visualization
2. Webscraping

Data visualization

This crash course is not...

• really a tutorial on how to make graphs in R

it's rather about...

• opening your eyes on what makes a good statistical graph

• learning the tradoffs in graph making

Some resources on graph making in R

ggplot2

• Andrew Heiss' data viz class

• Emi Tanaka's data viz workshop

• Garrick Aden-Buie's guide

Base R

• Notes de cours de l'université de Laval (French)

Numbers suck

Sort these countries by GDP per capita:

Numbers suck

Sort these countries by GDP per capita:

Numbers suck II

Numbers

Graph

Anscombe

What can you say about these numbers?

x	y	mean_X	sd_X	mean_Y	sd_Y	cor_XY
55.4	97.2	54.3	16.8	47.8	26.9	-0.0645
51.5	96
46.2	94.5
42.8	91.4
40.8	88.3
38.7	84.9
35.6	79.9
33.1	77.6
...	...

Same measures; but different data ⇒ different stories.

Numbers suck III

Numbers

Graph

Do you see a pattern?

Do you see a pattern?

Because love is an axis story

No, it's still because numbers suck

• remember that we're very limited human beings!

• we only can grasp and understand the world with our 5 senses

• abstract concepts (e.g. numbers) are tied to these physical senses

• to compare numbers we need to visualize them in our head

• when we graphically represent numbers, we cut the middle man

Visualization: why?

• powerful way to understand the data

• powerful way to send a message (not the same as the previous point!)

• most graphs suck (Excel do you hear me?): it's an easy way to stand out

Why R?

• Powerful: imagination is the limit, you can graph anything you have in your mind (really)

• Versatile: there are so many packages...

  • Cartography? cartography, leaflet, etc
  • 3D + kickass light effects? rayshader
  • animated? gganimate
  • dynamic? highcharter

• Communication: smooth integration in HTML documents, create a website in minutes (Rmarkdown)

Shiny web apps

Maps

Graph

Code

Source: Lorena Abad Crespo

# David Kidd, Kingston University London (2019)
# https://storymaps.arcgis.com/stories/79eeffa9f54d429687c17fa8267d3ba2
# Thinkquest historical boundaries 1815
#https://web.archive.org/web/20080328104539/http://library.thinkquest.org:80/C006628/download.html
library(tidyverse)
library(sf)
library(ggpomological)
library(ggimage)
library(MapColoring)
library(ggspatial)
extrafont::loadfonts(device = "win")
humboldt = read_sf("data/kingstonUniLondon/humboldt_route.geojson") |> 
  filter(Journey == "America") |> 
  mutate(
    angle = case_when(
      Date == "June-July 1799" ~ 0,
      Date == "Dec. 1800" ~ 340,
      Date == "Mar 1804" ~ 25,
      Date == "May 1804" ~ 50,
      Date == "July 1804" ~ 10,
      is.na(Date) ~ NA_real_
    ),
    hjust = case_when(
      Date == "June-July 1799" ~ 1.1,
      Date == "Dec. 1800" ~ -0.3,
      Date == "Mar 1804" ~ 0.5,
      Date == "May 1804" ~ 0.5,
      Date == "July 1804" ~0.5,
      is.na(Date) ~ NA_real_
    ),
    vjust = case_when(
      Date == "June-July 1799" ~ 3.5,
      Date == "Dec. 1800" ~ 0.5,
      Date == "Mar 1804" ~ 0.6,
      Date == "May 1804" ~ 0.5,
      Date == "July 1804" ~0.5,
      is.na(Date) ~ NA_real_
    )
  )
countries = read_sf("data/thinkquest/1815/cntry1815.shp") |> 
  st_set_crs(4326) |> 
  mutate(fillcol = as.factor(getColoring(as_Spatial(countries)))) |> 
  st_transform(st_crs(humboldt))
bbox = humboldt |> st_bbox()
g = ggplot() +
  geom_sf(data = countries,
          aes(fill = fillcol),
          col = NA, alpha = 0.6,
          show.legend = FALSE
        ) +
  geom_sf(
    data = humboldt,
    linetype = "longdash",
    color = "#2b323f",
    size = 0.5
  ) +
  geom_sf_text(
    data = humboldt,
    aes(label = Date, angle = angle, hjust = hjust, vjust = vjust),
    family = "Homemade Apple", 
    nudge_y = 250000
  ) +
  annotation_north_arrow(
    style = north_arrow_nautical(
      line_col = "#a89985",
      text_family = "Homemade Apple",
      text_col = "#a89985",
      fill = c("#a89985", "white"),
    )
  ) +
  annotate(
    "text",
    x = -1500000, y = -900000, color = "#6b452b",
    family = "Homemade Apple", size = 5,
    label = "Alexander von Humboldt\ntravels to the Americas\n1799-1804"
  ) +
  scale_fill_pomological() +
  labs(
    caption = glue::glue(
      "#30DayMapChallenge | Day 24: Historical | ",
      "Data: David Kidd, Kingston University London (2019), ThinkQuest | Created by @loreabad6",
    )
  ) +
  coord_sf(
    xlim = c(bbox["xmin"], bbox["xmax"]),
    ylim = c(bbox["ymin"], bbox["ymax"])
  ) +
  labs(x = NULL, y = NULL) +
  theme_pomological("Homemade Apple", 16) +
  theme(
    plot.caption = element_text(
      family = "mono", size = 9, hjust = 0.5,
      color = "#6b452b", face = "bold"
    )
  )
ggbackground(g, ggpomological:::pomological_images("background"))
ggsave(filename = "maps/day24.png",
       height = 20, width = 28, units = "cm")

Animations

Graph

Code

Source: Jamie Hudson

# ultra_running.R
# Jamie Hudson
# Created: 27 October 2021
# Edited: 27 October 2021
# Data: Benjamin Nowak by way of International Trail Running Association (ITRA)
# load libraries ------------------------------------------------------------
library(tidytuesdayR)
library(tidyverse)
library(janitor)
library(ggmap)
library(XML)
library(gganimate)
library(showtext)
library(colorspace)
library(ggtext)
library(shadowtext)
font_add_google("Cabin")
font_add_google("Shrikhand")
showtext_opts(dpi = 320)
showtext_auto(enable = TRUE)
# load dataset ------------------------------------------------------------
ultra_rankings <- readr::read_csv('https://t.co/JNJTpFTKqI?amp=1') %>% 
  clean_names()
race <- readr::read_csv('https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2021/2021-10-26/race.csv')
# read in GTX data for UTMB course 
# code from Sascha Wolfer https://www.r-bloggers.com/2014/09/stay-on-track-plotting-gps-tracks-with-r/
# data from https://www.plotaroute.com/search?keyword=utmb
options(digits=10)
# Parse the GPX file
pfile <- htmlTreeParse(file = "data/UTMB.gpx", error = function(...) {
}, useInternalNodes = T)
elevations <- as.numeric(xpathSApply(pfile, path = "//trkpt/ele", xmlValue))
times <- xpathSApply(pfile, path = "//trkpt/time", xmlValue)
coords <- xpathSApply(pfile, path = "//trkpt", xmlAttrs)
lats <- as.numeric(coords["lat",])
lons <- as.numeric(coords["lon",])
# wrangle data ------------------------------------------------------------
race_df <- full_join(ultra_rankings, race)
utmb_21 <- race_df %>% 
  filter(race_year_id == 72496)
geodf <- data.frame(lat = lats, lon = lons, ele = elevations, time = times)
lat <- c(min(geodf$lat) -0.06, max(geodf$lat) + 0.1)
lon <- c(min(geodf$lon) - 0.1, max(geodf$lon) + 0.1)
bbox <- make_bbox(lon,lat)
geodf <- geodf %>% 
  slice(which(row_number() %% 5 == 1))
utmb_times <- utmb_21 %>% 
  dplyr::select(time_in_seconds) %>% 
  mutate(pos = row_number()) %>% 
  pivot_wider(names_from = pos, values_from = time_in_seconds) %>% 
  slice(rep(1, each = nrow(geodf)))
# function to repeat times n times
rep.x <- function(x, na.rm=FALSE) (x / nrow(geodf) * row_number())
geodf_times <- cbind(geodf, utmb_times) %>% 
  dplyr::select(-c("time")) %>% 
  mutate_at(.vars = vars("1":"1526"), 
            rep.x) %>% 
  pivot_longer(-c(lon, lat, ele), names_to = "id", 
               values_to = "time") %>% 
  mutate(time = as.numeric(time))
# D'Haene and Dauwalter
two_runners <- geodf_times %>%
  filter(id %in% c(1,7))
# dataframe of route
route <- geodf %>% 
  dplyr::select(lon, lat, ele)
# download stamenmap for background
map_background <- get_stamenmap(bbox,
                    zoom = 12,
                    source="stamen",
                    maptype = "terrain-background",
                    color="bw")
# plot ------------------------------------------------------------
plot <- ggmap(map_background) +
  geom_path(data = route, mapping = aes(x = lon, y = lat, color = ele, group = 1),
            size = 4, lineend = "round") +
  scale_color_viridis_c(option = "magma") +
  geom_rect(xmin = 6.5, xmax = 7.3, ymin = 46.09, ymax = 46.2, 
            fill = "grey", alpha = 0.4) +
  geom_jitter(geodf_times, mapping = aes(x = lon, y = lat, group = id),
              colour = "white", fill = "white", pch = 25,
              size = 0.4, width = 0.002, height = 0.002) +
  geom_jitter(two_runners, mapping = aes(x = lon, y = lat, group = id),
              colour = "white", fill = "black",
              pch = rep(c(21, 22), 2858), size = 2.5, width = 0.002, height = 0.002) +
  geom_segment(aes(x = route$lon[1] - 0.01, 
                   y = route$lat[1] + 0.01,
                   xend = route$lon[1] + 0.01, 
                   yend = route$lat[1] - 0.01),
               colour = "yellow") +
  geom_shadowtext(label = "Start/End",
                  x = route$lon[1] + 0.014, y = route$lat[1] - 0.014, 
                  size = 3, hjust = 0, family = "Cabin", check_overlap = TRUE,
                  colour = "black", bg.colour = "white", bg.r = 0.2) +
  geom_shadowtext(label = "Elevation (m)",
                  x = 6.917, y = 45.623, 
                  size = 3, hjust = 0.5, family = "Cabin", check_overlap = TRUE,
                  colour = "black", bg.colour = "white", bg.r = 0.2) +
  geom_shadowtext(label = "1000",
                  x = 6.785, y = 45.635, 
                  size = 2.5, hjust = 0.5, family = "Cabin", check_overlap = TRUE,
                  colour = "black", bg.colour = "white", bg.r = 0.2) +
  geom_shadowtext(label = "2500",
                  x = 7.07, y = 45.635, 
                  size = 2.5, hjust = 0.5, family = "Cabin", check_overlap = TRUE,
                  colour = "black", bg.colour = "white", bg.r = 0.2) +
  guides(colour = guide_colorbar(title.position = 'bottom', title.hjust = 0.5,
                                 barwidth = unit(15, 'lines'), barheight = unit(0.8, 'lines'))) +
  annotate(geom = "text", label = "Ultra-Trail du Mont-Blanc 2021",
           x = 6.917, y = 46.162, size = 8, hjust = 0.5, family = "Shrikhand") +
  annotate(geom = "richtext", label = "A total of 1526 runners completed the 2021 edition of the UTMB® race in Chamonix, France. Starting at 17:00 the course  \nundulates over 170km, and was eventually won by Francois D'Haene in a time of 20 hours 45 minutes and 59 seconds.  \nCourtney Dauwalter was the first female runner to finish in 22 hours 30 minutes and 54 seconds.",
           x = 6.917, y = 46.125, size = 2.8, hjust = 0.5, family = "Cabin",
           fill = NA, label.color = NA) +
  annotate(geom = "richtext", label = "*The map below shows the average speed of each finisher. D'Haene is the black circle, and Dauwalter the black square*",
           x = 6.917, y = 46.1, size = 2.8, hjust = 0.5, family = "Cabin",
           fill = NA, label.color = NA) + 
  labs(colour = "Elevation (m)",
       caption = "@jamie_bio | source: International Trail Running Association (ITRA)") +
  theme_minimal() +
  theme(
    axis.line = element_blank(),
    axis.text.x = element_blank(),
    axis.text.y = element_blank(),
    axis.ticks = element_blank(),
    axis.title.x = element_blank(),
    axis.title.y = element_blank(),
    panel.grid.major = element_blank(),
    panel.grid.minor = element_blank(),
    panel.border = element_blank(),
    plot.caption = element_text(family = "Cabin", size = 8),
    legend.position=c(0.5, 0.025),
    legend.justification = "bottom",
    legend.direction = "horizontal",
    legend.text = element_blank(),
    legend.title = element_blank())
anim <- plot +
  transition_reveal(time)
animate(anim, nframes = 200, 
        height = 8, width = 6.5, units = "in", res = 150)
anim_save(paste0("ultra_running_", format(Sys.time(), "%d%m%Y"), ".gif"))

Nice spider stuff

Purpose of a graph

Good graphs have a main purpose.

Main purposes:

• looking nice

• send a message

• exploratory

Purpose of a publication graph

Good graphs have a main purpose.

Main purposes:

• looking nice

• send a message ( $=$ we're here!)

• exploratory

Publication graph

What's the best graph?

1. informative ( $=$ clear and not misleading)
2. pleasant

But...

• the medium should not take precedence on the content!
• graphs which are too beautiful (too arty) distract the reader from the message

The classic misleading graph

Problem?

Reply

Source: the bird

Source: the bird

Maybe too nice?

Basic

Too nice?

The 3 components of a graph

Content

• the information you want to share/observe

• typically: relationships, time series, consequence of events, conditional distributions

• you can stack several content in the same graph

Clarity

• how easy it is to understand the content

• defined by its measure: the time it takes to extract a piece of information $\Rightarrow$ the lower the time, the higher the clarity

Attractiveness

• is the graph pleasant to look at?

• do you stare at it for its own sake?

Graphs as a maths problem

Graph making is just about optimization

Precision approach (us)

$$\begin{array}{ccc}grap{h}^{\ast }& =& \arg \underset{g}{max}clarity\left(g\right)\\ & s.t.& content\left(g\right)\ge \tau \\ & & attractiveness\left(g\right)\ge \eta \end{array}$$

Infography approach

$$\begin{array}{ccc}grap{h}^{\ast }& =& \arg \underset{g}{max}attractiveness\left(g\right)\\ & s.t.& content\left(g\right)\ge \tau \\ & & clarity\left(g\right)\ge \gamma \end{array}$$

Tradeoffs

• adding content necessarily reduces clarity (there are solutions to limit that)

• the relationship between attractiveness and content/clarity is not clear. Graphs which are visually too attractive usually reduce clarity or weaken the message.

Example (iris again)

Graph 1

Criticism 1

Graph 2

Criticism 2

• the contrast is so so due to the grey background: ↘ clarity

• the colors are OK to identify the species

• the grid helps to make comparisons between distant points

⇒ clarity is OK overall but the attractiveness is so so. We get that these variables are good to discriminate the species: content is OK.

• font hard to read ↘ clarity

• no grid ↘ clarity

• it is visually attractive (at least to me!):

  • the font is nice (although hard to read)
  • the background texture is nice
  • the pencil look of the numbers/box is nice
  • = it's really nice!

⇒ attractiveness +++ (to me!). However, because there are so many nice things to look at, we lose track of the content. The design choices, although great, reduce clarity greatly.

This is not a graph that you should put in your report!

Lessons of the rules

• Clarity is cardinal

  • what is this graph about?

  • what does it represent?

  • what's the value of that point?

  • a graph should be self explanatory

• Attractiveness is important, but second tier

  • hooks the reader

  • makes the reader stay longer and maybe decide to put in some efforts to understand your graph

The reader as a shopper

The money is the effort. The reader has little money.

• Clarity $=$ value for money

  • you can extract more content with lower effort

  • if you don't get much for your money, you just switch to another product

• Attractiveness: increases the amount of money you wanna spend. It's the same effect as advertising:

  Buy that ticket to become a millionaire!

  vs

  Buy that ticket and you may become a millionaire if you win a lottery with probability one over one billion.

Typology of graphs

For publication (i.e. for the world)

• low content: to send a single or two messages
• clarity should be maximal: the reader must understand fast
• high attractiveness: to hook the reader

For exploration (i.e. for yourself)

• high content: you want the most information in a single graph
• tolerance for lower clarity: you know what you're doing, that's the price to pay to dispose of more content in a single graph. Although you still need a minimum of clarity
• attractiveness: isn't the priority

You still have the same optimization problem to solve but the thresholds are different!

Attractiveness

• what is considered nice largely depends on a consensus which can evolve over time

• it depends on preferences that vary between persons and even within persons (just have a look at your haircuts of 10 years ago!)

• we won't cover attractiveness since we can't please everyone

But...

• there are guiding principles of proportions and colors

• good news: clear graphs usually look ok

Content

• tailor your content, ask yourself:

  • is that information important?
  • what would the graph look like without it?
  • cut anything not necessary

• add elements of context if needed (do they strengthen your point?)

• is the graph faithful?

• hierarchy of information! (the main message should be emphasized vàv other messages/the elements of context)

• do I get the takeaway just from looking at the graph? Or do I need to read the text / get an oral explanation to get the central message?

The good news

There are many tips to make graphs clear!

Colors

• colors are extremely powerful: used properly, they add an extra layer of information without requiring any extra graph-space and have almost no processing cost for the reader

Moral of the story: Abuse colors!

Colors: what for?

• two main uses of color:

  • to distinguish categories
  • to represent intensity

• two different usages $=$ two very different color picks!

Colors: OK, but which colors?

Colors: a tricky topic!

Colors: a mini primer

Colors: general rule

Don't use pure colors!

They're a bit of an eyesore. They're very bright, making them hard to read. That's why I had to use a heavy font.^⋆ Note that the brightness depends on the hue: hues are not equal light-wise!

These are the same hues. I've just reduced saturation. The text is easier to read. I can even remove the heavy font!

Colors: distinguishing categories

• use colors that are "different" but have some harmony

• how to find harmonious color sets?

Don't do it yourself!

• Adobe color website: to create palettes or to find existing ones

• tons of color palettes in R

Distinct colors: 3 rules

General rules (and like French grammar, there are always exceptions!)

1. use colors that are "clear-cut" (we keep colors in mind using their names, ex: using blue + mid-blue-mid-green + green makes it hard to remember)

2. use different hues, not only different shades (shade variations are harder to remember and discern than hue variations; having both is even better)

3. don't use colors to distinguish too many categories:

   • 2-4: best
   • 2-5: to avoid, but can be OK if palette is good and depends on the graph
   • 2>5: forget about it (too big a hit on clarity)

Distinct colors: Rule 3

OK, but what if I really have to display 6+ categories?

Distinct colors: Example

Graph 1

Graph 2

Colors: intensity

Two main types of things to represent:

• min-max: dichotomic representation (you can use >2 colors though)
• negative-zero-positive: at least three colors (dichromatic works well)

Colors: intensity

Type of colors

min-max

neg-0-pos

Color: Accessibility

• about 5% of the population is colorblind

Color accessibility in R

Two dedicated packages (among others):

• scico (stands for scientific colors)
• viridis

Color accessibility

scico

viridis

Shapes

• it's fairly easy for humans to discern shapes
• however it's more difficult and less automatic than for colours
• add shapes in scatterplots in conjunction with colours to facilitate reading
• very useful in B&W

Shapes: example

Shape

Sh. + Col.

BW

BW + Sh.

Harnessing visual perception

• humans are pretty good at comparing heights and widths
• in contrast, they're pretty bad at comparing surfaces

$\Rightarrow$ use bars to compare numbers, use a grid to help make comparisons

Comparisons: Example

Difficult

Easy

Can you see a difference?

Exactly the same data... incomparably easier to read.

Helping the reader

• the data in your graph is central
• the text describing the graph is not less central!

Without description, a graph is worthless!

Describing a graph: First commandment

Describing a graph: Other main textual components

• the legend, if there are 2+ types of data
• the title¹

Helping the reader: Tips

• the text must be read easily: readable font + min. font size

• hierarchy: the explanations should not take more space than the data. More important information should be more emphasized.

• repeat information

• add a grid when relevant

• minimize the distance between the legend and the data: especially with 4+ categories

Making the text easy to read

1. you must choose a readable font
2. ensure the size of the text is easily readable (but not too big)

Font family

Font families

showtext

Graph

There are three main font families:

• serif
• sans-serif
• monospace

• sans-serif are recognized as the font family with the highest readability
• in general, don't use serif fonts in graphs.
• mono fonts can be nice when you have words with the same size that stack (like country codes)

In R you can change the font in graphs with showtext:

pacman::p_load(showtext)
font_add_google("Fira Code", "fira")
font_add_google("Merriweather", "merri")
showtext_auto()
plot(iris$Petal.Length, iris$Sepal.Length, col = iris$Species, 
     pch = 16, bty = "L", ann = FALSE)
title(xlab = "Petal length (default sans serif)", cex.lab = 1.2)
title(ylab = "Sepal length (serif: Merriweather)", 
      family = "merri", cex.lab = 1.2)
mtext(text = "Three varieties of iris flowers (mono: Fira Code)", 
      side = 3, line = 1, font = 2, adj = 0, cex = 1.7, family = "fira")

Font size

• do you think that once you find a graph displaying nicely on your screen, you can just ggsave it and it will be fine?

Think twice!

• the final location of your graph is your document, not your screen!
• the graph will be rescaled to fit its width in the document $\Rightarrow$ the font may become too small!

Font size: A helper function

Some functions may help you to deal with that: pdf_fit/png_fit from fplot:

• use setFplot_page to define the size of the final document if needed (by default it's an A4 page with some usual margins)

# starts recording.
# - pt = 11 will save the graph with 11pt font size
# - w2h = 1.75 means that the width to height ratio is 1.75 (wide graph)
pdf_fit("path.pdf", pt = 11, w2h = 1.75)
# your graph
# ends recording.
# The final look of your graph is displayed in the viewer pane
fit_off()

Repeat + grid: An example

Minimize the legend-to-data distance

Difficult

Easy

Can you spot the problems?

Context

The graph

Criticism

I will show you a graph coming from a top (top) publication.

The research was careful and the results are of great relevance: there is no doubt on the quality and the importance of the research done.

Despite the stellar work, the graphs could be improved (i.e. ↗ clarity), at no cost.

Among others...

Major

The legend takes as much space as the graph!!!!!!!! (it burned my eyes!)

Minor

Adding a light grid would facilitate the reading, it's almost impossible to compare points.