In our first dplyr lab, most of the operations we
performed changed a data frame while continuing to retain information
about individual observations. We filtered observations, selected
variables, created new variables, and changed the order of the
observations.
Today we are going to do something slightly different. Sometimes we don’t need another variable describing every observation. Instead, we want to use a collection of observations to calculate information about that collection.
Before introducing any new dplyr functions, let’s return
to an idea from our functions lab.
Earlier in the semester, we divided vector functions into two useful categories:
Consider:
x = c(4, 7, 8, 10, 11)Before running the following code, think about how many values each line should return.
mean(x)
median(x)
sd(x)
x - mean(x)
log(x)x - mean(x) and log(x) return one value
corresponding to each value in x. The mean, median, and
standard deviation each use the entire vector to calculate one
value.
This is the distinction behind the names mutate() and
summarize() in dplyr.
Consider the following small dataset:
scores = data.frame(
student = c("A", "B", "C", "D", "E"),
section = c("1", "1", "1", "2", "2"),
score = c(72, 81, 84, 88, 95)
)
scores## student section score
## 1 A 1 72
## 2 B 1 81
## 3 C 1 84
## 4 D 2 88
## 5 E 2 95What does one observation represent in these data?
Suppose we want to know the mean score. We could calculate it directly from the variable:
mean(scores$score)## [1] 84However, when working with data frames we will often want to calculate several summaries and keep the results organized together.
The summarize() function creates a new data frame
containing the summaries we ask it to calculate:
scores |>
summarize(
mean_score = mean(score),
median_score = median(score)
)## mean_score median_score
## 1 84 84Look carefully at what happened to the rows. We started with five
observations, where each row represented one student. After
summarize(), we have one row containing information
about those five students as a collection.
Inside summarize(), we give each new summary a name and
then specify how it should be calculated:
new_name = summarize_function(variable)We can calculate several different summaries at the same time:
scores |>
summarize(
mean_score = mean(score),
median_score = median(score),
sd_score = sd(score),
minimum = min(score),
maximum = max(score)
)## mean_score median_score sd_score minimum maximum
## 1 84 84 8.514693 72 95These functions answer different questions about the distribution of scores. The mean and median describe the center in different ways, the standard deviation describes variation, and the minimum and maximum describe the observed extremes.
Another useful way to summarize a distribution is with quantiles, which we’ve covered in 209.
A quantile gives us a value below which a particular percentage of
observations fall. We can calculate quantiles using the aptly named
quantile() function:
# Q1
quantile(scores$score, probs = 0.25)## 25%
## 81# Q2 = median
quantile(scores$score, probs = 0.50)## 50%
## 84# Q3
quantile(scores$score, probs = 0.75)## 75%
## 88The probs argument tells R which quantile we want,
written as a proportion. For example, 0.25 corresponds to
the 25th percentile and 0.90 corresponds to the 90th
percentile.
We can also use quantile() inside
summarize():
scores |>
summarize(
q25 = quantile(score, probs = 0.25),
med = quantile(score, probs = 0.50),
q75 = quantile(score, probs = 0.75)
)## q25 med q75
## 1 81 84 88It should be noted that there is a base R function
called summary() that gives the 5-number summary + average
of a variable. This is useful so we don’t need to code up a
dplyr summarize if we are just analyzing one
variable and want these specific values. summarize is much
more flexible, however.
summary(scores$score)## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 72 81 84 84 88 95Suppose we want to calculate the mean body mass of the penguins in our data. Without grouping, we could use:
penguins |>
summarize(
mean_mass = mean(body_mass_g, na.rm = TRUE)
)## # A tibble: 1 × 1
## mean_mass
## <dbl>
## 1 4202.This treats all of the observations in penguins as one
collection. The result is a single row containing the mean body mass
across all penguins.
Now compare this to:
penguins |>
group_by(species) |>
summarize(
mean_mass = mean(body_mass_g, na.rm = TRUE)
)## # A tibble: 3 × 2
## species mean_mass
## <fct> <dbl>
## 1 Adelie 3701.
## 2 Chinstrap 3733.
## 3 Gentoo 5076.The only new operation is group_by(species). The
group_by() function tells dplyr to treat
observations with different values of species as separate
groups. Importantly, group_by() itself does not calculate
anything or remove any observations. Instead, it changes how the
operations that follow are performed.
As a result, summarize() now calculates a separate mean
for each species, giving us one row for each group rather than one row
for all penguins.
We will need these packages and datasets.
# make sure to install if necessary
library(dplyr)
library(palmerpenguins)
library(dslabs)
library(gapminder)
colleges = read.csv("https://remiller1450.github.io/data/Colleges2019.csv")Recall from our work with functions that a summarize function takes a vector containing many values and returns a single summary value.
Some familiar examples include: mean,
median, sd, min,
max, sum, and quantile.
The summarize() function in dplyr creates a
new data frame containing summary information calculated from the
observations in the original data.
For example:
penguins |>
summarize(
mean_mass = mean(body_mass_g, na.rm = TRUE),
median_mass = median(body_mass_g, na.rm = TRUE)
)## # A tibble: 1 × 2
## mean_mass median_mass
## <dbl> <dbl>
## 1 4202. 4050Inside summarize(), we give each new summary variable a
name and specify how it should be calculated. Here,
mean_mass is calculated by passing body_mass_g
to mean(), while median_mass is calculated by
passing the same variable to median().
The original penguins data contain hundreds of
observations. After using summarize(), the resulting data
frame contains one row. The individual penguins are no longer
represented by separate rows; instead, the row contains the summaries we
asked R to calculate.
Because we have not grouped the data, all of the penguins are currently treated as one collection.
Use the penguins data for this question.
A. Use summarize() to create a data
frame containing the mean, median, standard deviation, minimum, and
maximum of flipper_length_mm.
B. What does the resulting row tell us about penguin flipper lengths?
C. The original penguins data contain
hundreds of observations. How many “observations” does the
summarize result contain? Explain why.
D. Suppose we replaced summarize() with
mutate() in Part A. Without running the code, how would you
expect the resulting data frame to differ? Then verify if your
prediction is correct.
quantile(penguins$body_mass_g, probs = 0.25, na.rm = TRUE)## 25%
## 3550The probs argument specifies the percentile as a
proportion. For example:
quantile(penguins$body_mass_g, probs = 0.10, na.rm = TRUE)## 10%
## 3300quantile(penguins$body_mass_g, probs = 0.90, na.rm = TRUE)## 90%
## 5400calculate the 10th and 90th percentiles.
Use the colleges data for this question.
A. Find the 25th, 50th, and 75th percentiles of
Cost.
B. Interpret the 75th percentile in context.
C. Using summarize, find the IQR of the
median student debt variable (median is part of the name).
D. Suppose a college has median student debt above the 90th percentile. What does this tell us about that college relative to the others in these data?
So far, each summary has described the entire dataset.
Often we want to calculate the same summary separately for different collections of observations. For example, rather than finding the mean body mass of all penguins, we might want to find the mean body mass for each species.
The group_by() function tells dplyr that
observations should be treated as belonging to separate groups:
penguins |>
group_by(species) |>
summarize(
mean_mass = mean(body_mass_g, na.rm = TRUE)
)## # A tibble: 3 × 2
## species mean_mass
## <fct> <dbl>
## 1 Adelie 3701.
## 2 Chinstrap 3733.
## 3 Gentoo 5076.group_by(species) does not calculate a
summary or remove any observations. Instead, it tells dplyr
to treat observations with different values of species as
separate groups. The operation that follows then works separately within
those groups.
Here, summarize() reduces the observations in each
species to one row, and mean() is applied separately to the
body masses within each species. The result therefore contains one row
for each species.
Compare this with:
Consider the following two pipelines:
penguins |>
summarize(
median_flipper = median(flipper_length_mm, na.rm = TRUE)
)penguins |>
group_by(island) |>
summarize(
median_flipper = median(flipper_length_mm, na.rm = TRUE)
)A. Without running the code, how many rows do you expect each result to contain? Explain.
Now run both pipelines.
B. What does one row represent in each resulting data frame?
C. Modify the second pipeline to calculate the 25th percentile, median, and 75th percentile of flipper length for each species.
D. In the 2nd pipeline, change the
group_by(island) argument to
group_by(island, species). Before running the code, how
many groups do you expect this to create? Explain your reasoning. Then
run the code and verify whether you are correct. If you are not correct,
explain what happened.
n()The number of observations being summarized is itself often useful information. (Report your sample sizes, researchers!)
Inside summarize(), the helper function n()
returns the number of observations currently being summarized.
For example:
penguins |>
group_by(species) |>
summarize(
n_penguins = n(),
mean_mass = mean(body_mass_g, na.rm = TRUE)
)## # A tibble: 3 × 3
## species n_penguins mean_mass
## <fct> <int> <dbl>
## 1 Adelie 152 3701.
## 2 Chinstrap 68 3733.
## 3 Gentoo 124 5076.Unlike mean() or median(), n()
does not operate on a particular variable. It just counts the rows of
the object. Because the data are currently grouped by species,
n() counts the number of observations within each
species.
If the data were not grouped, n() would instead count
the number of observations in the entire data frame.
The gapminder data contain information about countries
around the world at several points in time.
head(gapminder)## # A tibble: 6 × 6
## country continent year lifeExp pop gdpPercap
## <fct> <fct> <int> <dbl> <int> <dbl>
## 1 Afghanistan Asia 1952 28.8 8425333 779.
## 2 Afghanistan Asia 1957 30.3 9240934 821.
## 3 Afghanistan Asia 1962 32.0 10267083 853.
## 4 Afghanistan Asia 1967 34.0 11537966 836.
## 5 Afghanistan Asia 1972 36.1 13079460 740.
## 6 Afghanistan Asia 1977 38.4 14880372 786.Recall that one observation represents one country during one particular year.
Suppose we want to compare life expectancy among continents in 2007.
Construct a data frame containing one row for each continent and the following information:
Arrange the continents from largest to smallest median life expectancy.
A. What does one observation in your resulting data frame represent?
B. Which continent has the greatest variation in life expectancy?
C. Why is the number of countries useful information to include with the other summaries?
Grouping is useful for more than summarize().
Recall that mutate() creates or modifies variables while
retaining the individual observations. Without grouping, a calculation
such as
# run in your console
penguins |>
mutate(
mean_mass = mean(body_mass_g, na.rm = TRUE)
)uses the complete body_mass_g variable.
mean() therefore calculates the mean body mass across all
of the penguins, and that same value is placed in mean_mass
for every observation.
Now consider:
# run in your console
penguins |>
group_by(species) |>
mutate(
mean_mass = mean(body_mass_g, na.rm = TRUE)
)group_by(species) changes which values are supplied to
mean() during the calculation. The mean is calculated
separately using the body masses within each species.
Unlike summarize(), however, mutate() does
not collapse those observations. Every penguin remains
in the data. Each penguin simply receives the mean calculated from the
observations belonging to its species.
This gives us a useful way to calculate information about an individual observation relative to the group it belongs to.
Use the penguins data for this question.
A. Create a variable called
species_mean_flipper containing the mean flipper length for
each penguin’s species.
B. Create another variable called
difference_from_species_mean containing each penguin’s
flipper length minus the mean for its species.
C. What does a positive value of
difference_from_species_mean mean? What does a negative
value mean?
D. Arrange the data so that the penguins furthest above their species mean appear first. What does the first observation tell you?
E. Why would using summarize() instead
of mutate() prevent us from answering this question about
individual penguins?
ungroup()When we use group_by(), the grouping remains attached to
the resulting data frame. This means that later operations may continue
to be performed separately within the groups even after the calculation
for which we originally created the groups is finished.
For example:
# run this in your console, check columns
penguins |>
group_by(species) |>
mutate(
species_mean_mass = mean(body_mass_g, na.rm = TRUE)
) |>
mutate(
another_mean = mean(body_mass_g, na.rm = TRUE)
)At the second mutate(), the data are still grouped by
species. Therefore, another_mean is also calculated
separately within each species.
If we are finished performing calculations within the groups, we can
remove the grouping using ungroup():
# run this in your console, check columns
penguins |>
group_by(species) |>
mutate(
species_mean_mass = mean(body_mass_g, na.rm = TRUE)
) |>
ungroup() |>
mutate(
overall_mean_mass = mean(body_mass_g, na.rm = TRUE)
)ungroup() does not remove observations, remove
variables, or change any values that have already been calculated. It
changes how later operations treat the
observations.
Before ungroup(), calculations are performed separately
within species. After ungroup(), the observations are once
again treated as a single collection unless we group them again.
Consider the following pipeline:
penguins |>
group_by(species) |>
mutate(
species_median = median(body_mass_g, na.rm = TRUE)
) |>
ungroup() |>
mutate(
overall_median = median(body_mass_g, na.rm = TRUE)
)A. What value does species_median
contain for each penguin?
B. What value does overall_median
contain?
C. Why is ungroup() necessary if we
want overall_median to represent all penguins?
D. Did ungroup() remove or modify any
observations or variables? What did it change?
We have now used grouping in two different ways.
With summarize():
penguins |>
group_by(species) |>
summarize(
mean_mass = mean(body_mass_g, na.rm = TRUE)
)## # A tibble: 3 × 2
## species mean_mass
## <fct> <dbl>
## 1 Adelie 3701.
## 2 Chinstrap 3733.
## 3 Gentoo 5076.the observations belonging to each species are reduced to one summary row.
With mutate():
# verify operation in your console
penguins |>
group_by(species) |>
mutate(
mean_mass = mean(body_mass_g, na.rm = TRUE)
)the individual observations remain, but the calculation is performed separately within each species.
For the final question of today, return to the colleges
data. Suppose we are interested only in colleges located in the
Plains region.
Which colleges have unusually high costs relative to what is typical for colleges in the same state?
A. Before investigating this question, create a data
frame containing only colleges in the Plains region and
select only the variables that you think will be useful for the
analysis.
B. Use your data from Part A to investigate the question above and identify the college that stands out the most. You will need to decide how to define typical and what calculations and data manipulation operations are useful. Which college did you identify? Explain how you determined that it stood out relative to other colleges in its state.
C. Which college has the highest cost among all
colleges in the Plains region? Is this the same college as
Part B?
D. Explain why these two approaches to identifying a college with unusually high cost can produce different answers.
Today we used summarize() to reduce observations to
useful numerical summaries and group_by() to control which
observations are used together in a calculation.
We also saw that grouping affects more than summarize().
When a grouped data frame is passed to mutate(), the
individual observations remain, but calculations are performed
separately within each group.
Finally, ungroup() allows us to remove the current
grouping when a later calculation should use the entire data frame.
As you work with grouped data, periodically ask yourself: Which observations should be used together for this calculation?
Material added on Wednesday:
Question 8:
At the beginning of our first data manipulation lab, we looked at the
career of Manny Ramirez using the Batting data from the
Lahman package. Recall that one observation in these data
represents one player’s statistics for one team during one season. As a
result, a player who plays for multiple teams during the same season
will have multiple observations for that year.
Let’s return to these data and investigate Manny Ramirez’s performance over his career.
library(Lahman)
data(Batting)A. Filter the Batting data to contain
only observations for Manny Ramirez (relevant id is
"ramirma02". Then use group_by() and
summarize() to create a data frame containing one
observation for each year of his career. For each year, calculate his
total number of home runs (HR), total number of games
played (G), and the number of observations that were
combined to produce each yearly summary.
B. What does the number of observations calculated in Part A tell us about Manny’s career? Identify any seasons for which this number is greater than one and explain why this occurred.
C. Add a variable to your summarized data that gives Manny’s number of home runs per game for each season. Among seasons in which Manny played at least 100 games, in which season did he have the highest number of home runs per game?
D. In which season did Manny hit the most total home runs? Is this the same season that you identified in Part C?
E. Suppose instead that we had simply filtered the
original Batting data to Manny Ramirez and arranged the
observations from largest to smallest according to HR.
Explain why this would not necessarily tell us which season Manny hit
the most home runs.
Question 9:
For this question, consider all players in the Batting
data from 2010 through 2015. We will investigate home runs at several
different levels by changing how the data are grouped and
summarized.
Begin by filtering the Batting data to contain only
observations from 2010 through 2015.
A. Create a data frame containing one observation for each player. For each player, calculate the total number of home runs they hit from 2010 through 2015. Which player hit the most home runs during this period?
B. Using the data you created in Part A, determine how many players hit at least 100 total home runs from 2010 through 2015.
C. Starting again with the original filtered data, create a data frame containing one observation for each year. Calculate the total number of home runs hit by all players during each year. Which year had the most home runs? Which year had the fewest?
D. Starting again with the original filtered data, create a data frame containing one observation for each team during each year. Calculate the total number of home runs hit by each team during each season. Which team-season had the most home runs from 2010 through 2015?
E. Using the team-by-year summaries from Part D, calculate the mean and median number of home runs hit by teams during each year. Your final data frame should contain one observation for each year.
F. Think about how the meaning of an observation
changed throughout this question. What does one observation represent in
the data you created in Parts A, C, D, and E? Explain how the variables
supplied to group_by() determine what one observation
represents after using summarize().
Question 10:
The following dataset details expenditures on K-12 education in the U.S. for different years. For this exercise, you are going to play the role of a public administrator in Iowa who is interested in comparing money spento on education in Iowa to other states.
k12expenditures = read.csv("https://nfriedrichsen.github.io/data/k12expenditures.csv")Data Source: NCES public education finance data. These data were reconstructed from the original federal data to match the expenditures dataset used in Bradley Boehmke’s Data Wrangling with R.
A. Begin by inspecting the
k12expenditures data. How many observations and variables
are there? What does one observation represent? Describe what
information is contained in the variables of this data set.
B. Use an appropriate dplyr function to
investigate the Division variable. Briefly describe the
part of the country represented by each division.
C. Using the 2011 expenditures, investigate how Iowa compares with the other states in its division. Use an appropriate numerical summary to make this comparison and explain what you find.
D. Now compare Iowa’s 2011 expenditure with the other 49 states. Create a new data frame containing only the other 49 states and use an appropriate numerical summary of their 2011 expenditures to make this comparison. Describe what you find.
E. In Part D, you reduced the expenditures of 49 different states to a single summary value. What information about the other states is lost by doing this?
F. Let’s investigate the other 49 states in more detail. Divide these states into an upper and lower half based on their 2011 expenditures. Create separate data frames containing the states in each half and use appropriate numerical summaries to compare these groups with Iowa. Tell me what you find.