library(ggplot2)
library(dplyr)
library(hexbin)

This is a continuation of the lab from Monday. Please add these questions to the end of Monday’s lab instead of creating a new document.

Today we move from displaying distributions to displaying relationships between variables. In fast, we were starting to veer into this territory towards the end of Monday’s portion of the lab. We will continue using the Marriage data and our sample of flights from Monday.

One of the main considerations when choosing a visualization is what variables we want to examine and what types of variables they are. Robert Kabacoff, in Modern Data Visualization with R, organizes many graphical displays according to the combination of categorical and quantitative variables being considered, as well as the number of variables being displayed. A graph designed to examine one quantitative variable, for example, serves a different purpose from one designed to examine the relationship between two quantitative variables or several variables of different types.

Our main goal in these labs is to introduce you to a variety of visualization options and what they can reveal about data. The types and number of variables help narrow down which graphs might be appropriate, but they usually do not determine a single “correct” graph. Different visualizations can emphasize different features of the same data. As the course continues, we will occasionally return to these choices and think more carefully about when a particular graph is useful for answering a question in context.

As before, you are not expected to memorize every new function. Focus on what each graph reveals and how a basic graph can be modified when we want to emphasize something different.

# run this code to update your flights_sample data
## Warning, next parts will not work without updating

flight_pool = subset(
  nycflights13::flights,
  carrier %in% c("AA", "B6", "DL", "UA", "WN") &
    !is.na(arr_delay) & !is.na(dep_delay)
)

set.seed(230)
flights_sample = flight_pool[
  sample(nrow(flight_pool), size = 6000),
]

airline_names = c(
  AA = "American", B6 = "JetBlue", DL = "Delta",
  UA = "United", WN = "Southwest"
)

flights_sample$airline = factor(
  airline_names[flights_sample$carrier], levels = airline_names
)

flights_sample$origin = factor(
  flights_sample$origin, levels = c("EWR", "JFK", "LGA")
)

flights_sample = flights_sample[
  , c("airline", "origin", "dest", "month", "day",
      "dep_delay", "arr_delay", "distance", "air_time")
]





Two Categorical Variables

You have already worked with bar charts involving two categorical variables in 209. Here is a quick refresher using airline and departure airport.

ggplot(flights_sample, aes(x = airline, fill = origin)) +
  geom_bar(position = "dodge") +
  labs(x = "Airline", y = "Number of flights", fill = "Origin") +
  scale_fill_viridis_d() + 
  theme_minimal()

# (optional) add this on and watch the magic
geom_text(
    aes(label = after_stat(count)),
    stat = "count",
    position = position_dodge(width = 0.9),
    vjust = -0.3
  )

The grouped bars compare counts. We can instead make every airline’s bar the same height and compare proportions.

ggplot(flights_sample, aes(x = airline, fill = origin)) +
  geom_bar(position = "fill") +
  labs(x = "Airline", y = "Proportion", fill = "Origin") +
  scale_fill_viridis_d() +
  theme_minimal()

Question 2.1

  1. Suppose you are interested in understanding how the five airlines use the three NYC airports. What does the proportional graph make easier to learn about the airlines? What kinds of questions about the airlines and airports would this graph help you answer?

  2. Suppose instead that you are interested in the number of flights operated by each airline from the three airports. Which graph is more useful? Briefly explain. Also give one specific question about these data that you could investigate with the count graph but not the proportional graph alone.





Building Graphs with Layers

Recall that a ggplot can contain multiple layers. We can use those layers to add useful information rather than treating every graph as a fixed chart type.

Here are the arrival delays for our five airlines.

ggplot(flights_sample, aes(x = airline, y = arr_delay)) +
  geom_boxplot() +
  coord_cartesian(ylim = c(-50, 100)) +
  labs(x = "Airline", y = "Arrival delay (minutes)") +
  theme_minimal()

Suppose we also want to show the mean arrival delay for each airline. We can add it as another layer.

ggplot(flights_sample, aes(x = airline, y = arr_delay)) +
  geom_boxplot() +
  stat_summary(fun = mean, geom = "point", size = 3) +
  coord_cartesian(ylim = c(-50, 100)) +
  labs(x = "Airline", y = "Arrival delay (minutes)",
       title = "Arrival delays with airline means") +
  theme_minimal()

The boxplot still summarizes the distribution, while the added point shows another quantity we may care about.

We can also add a meaningful reference value.

ggplot(flights_sample, aes(x = airline, y = arr_delay)) +
  geom_boxplot() +
  stat_summary(fun = mean, geom = "point", size = 3) +
  geom_hline(yintercept = 0, linetype = "dashed") +
  coord_cartesian(ylim = c(-50, 100)) +
  labs(x = "Airline", y = "Arrival delay (minutes)",
       title = "Arrival delays by airline") +
  theme_minimal()

Here, zero has a natural interpretation: arriving exactly on schedule.

Question 2.2

  1. What do the added points represent in the boxplot? How is it different from the horizontal line inside each box?

  2. Why might the line at zero be useful when interpreting this particular variable? What does it allow us to easily compare?





Comparing Many Groups

The previous graphs compared five airlines. Sometimes we want to compare a numerical quantity across many more categories.

Suppose we are interested in whether flights to some destinations tend to experience larger arrival delays than others. The following code finds the 15 most common destinations in our sample and calculates the mean arrival delay for each.

Don’t worry about understanding all of the data manipulation code yet. We will begin learning how to perform these operations with dplyr next week.

destination_delays = flights_sample |>
  group_by(dest) |>
  summarize(n = n(), mean_delay = mean(arr_delay), .groups = "drop") |>
  arrange(desc(n)) |>
  slice_head(n = 15)

One way to display these summaries is a Cleveland dot plot. Rather than representing each value with the length of a bar, the numerical value is represented by the position of a point.

ggplot(destination_delays,
       aes(x = mean_delay, y = reorder(dest, mean_delay))) +
  geom_point(size = 3) +
  geom_vline(xintercept = 0, linetype = "dashed") +
  labs(x = "Mean arrival delay (minutes)", y = "Destination",
       title = "Mean arrival delay for common destinations") +
  theme_minimal()

Question 2.3

  1. Suppose you were interested in identifying destinations associated with relatively large or small average delays. What features of this graph make that comparison easy?

  2. Why might a dot plot be preferable to a bar chart when comparing a numerical quantity across many categories?

  3. What does each point in this graph represent? In particular, why should you not interpret a point as representing an individual flight?

Showing Estimates with Intervals

Sometimes we want a graph to communicate not only an estimate, but also the uncertainty associated with that estimate.

The following code calculates the mean arrival delay for each airline along with a 95% confidence interval for that mean.

airline_delays = flights_sample |>
  group_by(airline) |>
  summarize(
    n = n(), mean_delay = mean(arr_delay), sd_delay = sd(arr_delay),
    se_delay = sd_delay / sqrt(n),
    ci_delay = qt(.975, df = n - 1) * se_delay,
    .groups = "drop"
  )

We can represent the means with points and the confidence intervals with error bars.

ggplot(airline_delays, aes(x = airline, y = mean_delay)) +
  geom_point(size = 3) +
  geom_errorbar(
    aes(ymin = mean_delay - ci_delay, ymax = mean_delay + ci_delay),
    width = .15
  ) +
  geom_hline(yintercept = 0, linetype = "dashed") +
  labs(x = "Airline", y = "Mean arrival delay (minutes)",
       title = "Mean arrival delay with 95% confidence intervals") +
  theme_minimal()

Question 2.4

  1. Compare this graph with the layered boxplot from Question 2. Both contain points representing airline means. What information does the boxplot provide that this graph does not? What information does this graph provide that the boxplot does not?

  2. Suppose your main interest were estimating and comparing mean arrival delays among airlines. Which of these two graphs would be more directly suited to that question? Explain your reasoning.

A note about these intervals: The interpretation of these confidence intervals is more complicated than it may first appear. The original flight data are administrative records rather than a random sample from a larger population; for the period and flights represented, they are closer to a census. Although we can calculate standard errors and confidence intervals from these data, interpreting what population or source of uncertainty those intervals refer to requires assumptions beyond what we will consider in this course. The important point for now is that being able to calculate a confidence interval does not automatically mean that it has a meaningful inferential interpretation.





Much ado about a lot of points

Now suppose our interest shifts to a relationship between two quantitative variables, asking a question like: How are departure delays related to arrival delays?

A scatterplot is a natural graph for this question, but our sample contains 6,000 flights.

ggplot(flights_sample, aes(x = dep_delay, y = arr_delay)) +
  geom_point() +
  coord_cartesian(xlim = c(-25, 200), ylim = c(-75, 250)) +
  labs(x = "Departure delay (minutes)", y = "Arrival delay (minutes)",
       title = "Departure and arrival delays") +
  theme_minimal()

Many observations overlap. As a result, an area containing hundreds of flights may not look very different from an area containing dozens.

A hexagonal bin plot, or hexplot, takes a different approach. Instead of representing every flight with its own point, it divides the plotting region into hexagons. The color of each hexagon represents the number of observations falling in that region.

ggplot(flights_sample, aes(x = dep_delay, y = arr_delay)) +
  geom_hex(bins = 35) +
  coord_cartesian(xlim = c(-25, 200), ylim = c(-75, 250)) +
  labs(x = "Departure delay (minutes)", y = "Arrival delay (minutes)",
       fill = "Flights",
       title = "Concentration of departure and arrival delays") +
  theme_minimal()

A related approach is a two-dimensional density plot. Like the hexplot, this graph emphasizes where observations are concentrated rather than individual flights. Instead of dividing the graph into bins, however, it uses a smooth estimate of the density (probability distribution) of observations.

ggplot(flights_sample, aes(x = dep_delay, y = arr_delay)) +
  geom_density_2d_filled(bins = 5) +
  labs(x = "Departure delay (minutes)", y = "Arrival delay (minutes)",
       fill = "Density", title = "Density of departure and arrival delays") +
  theme_minimal()

This graph is not particularly effective for these data! Most flights are highly concentrated within a relatively small region, so most of the graph is assigned to areas with very low density. A useful graph type does not necessarily produce a useful graph for every dataset.

We can modify the idea by displaying the density as contour lines and layering the individual observations back onto the graph.

ggplot(flights_sample, aes(x = dep_delay, y = arr_delay)) +
  geom_point(alpha = .08) +
  geom_density_2d() +
  labs(x = "Departure delay (minutes)", y = "Arrival delay (minutes)",
       title = "Departure and arrival delays with density contours") +
  theme_minimal()

The contour lines work somewhat like those on a topographic map, except that they represent estimated density rather than elevation. The points allow us to see the individual observations while the contours help identify regions where observations are more densely concentrated. This is another example of how layers can be combined when no single graphical element shows everything we want to see.

Question 2.5

  1. Compare the scatterplot and hexplot. What does the hexplot make easier to see about the relationship between departure and arrival delays? What information does the scatterplot preserve that the hexplot does not?

  2. Why is the filled two-dimensional density plot not particularly effective for these data? What feature of the data causes the problem?

  3. How does adding the individual points to the density contours change what you can learn from the graph?

  4. None of these representations is universally better than the others. Suppose you were interested in (1) an individual unusual flight, (2) how many flights occur in different regions of the graph, or (3) the general regions of greatest concentration. Which representation would be most useful for each purpose?





Relationships Over Time

Time introduces another kind of relationship. Suppose we want to know whether average arrival delays changed over the course of 2013.

The following code uses the full set of flights from our five airlines rather than the 6,000-flight sample. It calculates one mean arrival delay for each month.

monthly_delays = flight_pool |>
  group_by(month) |>
  summarize(mean_delay = mean(arr_delay), .groups = "drop")

Because months have a meaningful order, we can display these values using a line graph.

ggplot(monthly_delays, aes(x = month, y = mean_delay)) +
  geom_line() +
  geom_point(size = 3) +
  geom_hline(yintercept = 0, linetype = "dashed") +
  scale_x_continuous(breaks = 1:12, labels = month.abb) +
  labs(x = "Month", y = "Mean arrival delay (minutes)",
       title = "Mean arrival delay during 2013") +
  theme_minimal()

The points show the twelve monthly summaries. Connecting them emphasizes how the quantity changes as we move through time.

Question 2.6

  1. Suppose you were interested in whether delays were better or worse during different parts of the year. What features or patterns do you notice in this graph that you might want to investigate further?

  2. Why does it make sense to connect January to February to March and so on? Why would it generally not make sense to connect American to JetBlue to Delta in one of our earlier graphs?

Adding Airline Distinction

The previous graph combines all five airlines. But perhaps the pattern over time is different for different airlines.

We can instead calculate a mean for each airline during each month.

monthly_airline_delays = flight_pool |>
  mutate(airline = factor(airline_names[carrier], levels = airline_names)) |>
  group_by(airline, month) |>
  summarize(mean_delay = mean(arr_delay), .groups = "drop")

One approach is to distinguish the airlines using color.

ggplot(monthly_airline_delays,
       aes(x = month, y = mean_delay, color = airline, group = airline)) +
  geom_line() +
  geom_point() +
  geom_hline(yintercept = 0, linetype = "dashed") +
  scale_x_continuous(breaks = 1:12, labels = month.abb) +
  scale_color_viridis_d() +
  labs(x = "Month", y = "Mean arrival delay (minutes)",
       color = "Airline", title = "Monthly arrival delays by airline") + 
  theme_minimal()

Another possibility is to separate the airlines into panels.

ggplot(monthly_airline_delays, aes(x = month, y = mean_delay)) +
  geom_line() +
  geom_point() +
  geom_hline(yintercept = 0, linetype = "dashed") +
  facet_wrap(~ airline) +
  scale_x_continuous(
    breaks = c(1, 4, 7, 10),
    labels = month.abb[c(1, 4, 7, 10)]
  ) +
  labs(x = "Month", y = "Mean arrival delay (minutes)",
       title = "Monthly arrival delays by airline") +
  theme_minimal()

The underlying information is the same. What changes is how the airline variable is represented.

Question 2.7

Suppose you are interested in whether the airlines experienced similar or different patterns of delays over the course of the year.

  1. Which graph makes it easier to compare different airlines during the same month? Explain.

  2. Which graph makes it easier to concentrate on how one particular airline changed throughout the year? Explain.

  3. If you were preparing one figure for someone interested in seasonal patterns in airline performance, which representation would you choose? What about their question makes your choice useful?

Exploring other Graphs

There are far more ways to visualize relationships than we can cover in two class periods.

Browse the graph examples in the Categorical vs. Quantitative section of Modern Data Visualization with R:

https://rkabacoff.github.io/datavis/Bivariate.html#categorical-vs.-quantitative

Choose one type of graph that we did not use in either Monday’s or Wednesday’s lab. You do not need to reproduce the graph or understand all of the R code used to create it.

Question 2.8

  1. What type of graph did you choose? In your own words, describe what information the graph is designed to show.

  2. Imagine you had a dataset you were interested in exploring related to a hobby you enjoy. Describe a question or situation for which your chosen graph would be useful in the context of this data. Explain why you might choose it instead of one of the graphs we used in this lab.