Preamble

Real data rarely arrives in the form we want it in.

In the previous labs, we focused primarily on accessing data and understanding various ways R can store that data. In Lab 1, we also reviewed how data frames are organized into rows and columns and how individual observations and variables can be accessed. In Lab 2, we used logical conditions to identify and subset observations that meet particular criteria.

In this lab, we will begin modifying data frames or making new ones. When we manipulate data in R, we can either overwrite an existing object with the result or give the result a new name so that both versions remain available.

For example, consider the College Scorecard data we have used previously:

# load in colleges data
colleges = read.csv("https://nfriedrichsen.github.io/data/Colleges2019.csv")

Suppose we only wanted to work with colleges in Iowa. We could overwrite the colleges data with this smaller data set:

# remove all non-iowa colleges from the data set
colleges = subset(colleges, State == "IA")

After running this code, colleges contains only the Iowa colleges and the original version with different states is no longer stored in our R Environment. If I want to go back to analyze a different subset, I have to reload the original colleges data set above. Having to constantly reload the same data is not efficient; it slows down our knitting process by making the computer do more work and, with larger data sets, can make things take considerably longer.

Instead, we could give the result a new name:

# reload the original data
colleges = read.csv("https://nfriedrichsen.github.io/data/Colleges2019.csv")

# make the new subset its own data frame
iowa_colleges = subset(colleges, State == "IA")

Now colleges still contains the original data, while iowa_colleges contains our modified version.

There is a trade-off here; every new object we create takes up some memory, so we wouldn’t necessarily want to make endless copies of a huge data set. Neither approach is always correct. The important consideration we need to make is whether the original data may still be useful later.

Remember that changing an object in R does not change the raw data file from which it was imported. If we make a mistake, we can rerun our code from the beginning and import the original data again, which is one reason that making reproducible code (thus documenting our operations) is important.

Lab

For this lab, we will continue working with the Happy Planet data used in the previous labs:

HappyPlanet = read.csv("https://nfriedrichsen.github.io/data/HappyPlanet.csv")

Coercion – Review

Recall that we introduced three important types of vectors in the first lab:

  1. numeric vectors - for example: x = c(1,2,3)
  2. character vectors - for example: x = c("A","B","C")
  3. logical vectors - for example: x = c(TRUE, FALSE, TRUE)

Many functions require their inputs to be of a certain type, but that is not always how data is stored, or sometimes it is converted to another type when we import it into R. Fortunately, data can be coerced into another type using the as. family of functions:

## A character vector where the text strings are numbers
x <- c("1","12","123")
typeof(x)
## [1] "character"
## Coerce 'x' to a numeric vector
x <- as.numeric(x)
x
## [1]   1  12 123
typeof(x)
## [1] "double"

Notice that we overwrote x in this example. After the assignment is performed, the original character version of x is no longer stored in the Environment.

Question 1: (Sort of review) Check the type of the variable Region in HappyPlanet. Then coerce Region into a character variable and overwrite the original Region variable with the result. Use typeof() to verify the change.

Factor Variables

Categorical variables are often stored as either character variables or factors in R. A factor represents a categorical variable using a set of possible categorical labels known as levels.

Factors are particularly useful when the categories have a meaningful ordering that we want R to recognize. Consider the following vector containing different months:

## A vector containing different months
mons <- c("March","April","January","November","January", "September",
          "October","September","November","August","January","November",
          "November","February","May","August","July","December",
          "August","August","September","November","February","April")

## Convert it to a factor
mons_unordered = factor(mons)

By default, the factor levels are arranged alphabetically:

levels(mons_unordered)
##  [1] "April"     "August"    "December"  "February"  "January"   "July"     
##  [7] "March"     "May"       "November"  "October"   "September"
barplot(table(mons_unordered))

However, alphabetical ordering does not make much sense for months. (This graph is bad, don’t make graphs like this). We can specify the order of the levels manually:

mons_ordered = factor(mons,
                      levels = c("January","February","March","April",
                                 "May","June","July","August","September",
                                 "October","November","December"),
                      ordered = TRUE)

levels(mons_ordered)
##  [1] "January"   "February"  "March"     "April"     "May"       "June"     
##  [7] "July"      "August"    "September" "October"   "November"  "December"
barplot(table(mons_ordered))

Question 2: The Ames Housing data contain information from the Ames Assessor’s Office used in computing assessed values for individual residential properties sold in Ames, IA from 2006 to 2010.

A more detailed description of variables can be found at this link.

ames_housing = read.csv("https://nfriedrichsen.github.io/data/AmesHousing.csv")

  • A. Check the type of the variable MS.SubClass and compare it with the description of this variable given in the information link above. Based upon your assessment, explain whether this variable should be coerced to a different type.

  • B. The variable Exter.Cond describes the exterior condition of a house using the categories Po, Fa, TA, Gd, and Ex, corresponding to Poor, Fair, Typical/Average, Good, and Excellent.

Create an ordered factor from Exter.Cond whose levels run from Poor to Excellent. Store the result as an object named cond_ordered. Use levels() to verify the ordering and use barplot() and table() to construct a bar chart of the resulting factor.

Creating and Combining Data Frames

Recall from Lab 1 that data frames are composed of vectors of the same length. We can use the data.frame() function to combine vectors into a data frame.

Suppose we recorded the temperature and reaction time for three trials of a chemistry experiment:

temperature = c(20, 40, 60)
reaction_time = c(84.2, 43.7, 29.1)

experiment = data.frame(temperature, reaction_time)
experiment
##   temperature reaction_time
## 1          20          84.2
## 2          40          43.7
## 3          60          29.1
typeof(experiment)
## [1] "list"

Each vector becomes a variable (column) in the resulting data frame, and corresponding positions in the vectors become observations (rows). Notice that a data frame is stored as a list of column vectors (variables) in R.

Column Binding

Another way to combine objects is the cbind() function. The c in cbind() stands for column. For example:

temperature = c(20, 40, 60)
reaction_time = c(84.2, 43.7, 29.1)

experiment = cbind(temperature, reaction_time)
experiment
##      temperature reaction_time
## [1,]          20          84.2
## [2,]          40          43.7
## [3,]          60          29.1
typeof(experiment)
## [1] "double"

Both examples place temperature and reaction_time beside one another as columns. However, without directly invoking the data.frame function, R treats this resulting object as a mathematical matrix. Although the matrix has named columns, it is not stored as a list of variable vectors like a data frame, and so the following operation which otherwise should work, does not.

experiment$reaction_time
## Error in `experiment$reaction_time`:
## ! $ operator is invalid for atomic vectors

The cbind() operation is generally useful when we already have an existing data frame, and we want to add information for each observation corresponding to a new variable. For example, suppose the concentration used in each experimental trial was stored separately in a different vector:

concentration = c(.10, .15, .20)

experiment = data.frame(temperature, reaction_time) # create data frame
experiment = cbind(experiment, concentration)
experiment # add concentration column
##   temperature reaction_time concentration
## 1          20          84.2          0.10
## 2          40          43.7          0.15
## 3          60          29.1          0.20

When using cbind(), R assumes that the observations are already in the correct order. Therefore, you should only use cbind() when you know that the rows correspond to the same observations.

To summarize, data.frame() is primarily used to construct a new data frame, while cbind() describes the operation of adding columns together. If you are using cbind() to add columns to a data frame, then the use of data.frame(df, newvar) vs cbind(df, newvar) are generally interchangeable.

Row Binding

We can similarly use rbind() to combine objects by rows.

Sometimes it can be useful to aggregate several data frames with the same structure into one larger data frame. For example, we might want to combine results from multiple experiments or aggregate several years of data from the same source. The rbind() function will append the rows of one or more data frames to an initial data frame, provided their variables are compatible:

df1 <- data.frame(Year = 2024, val = rnorm(3))
df2 <- data.frame(Year = 2025, val = rnorm(3, mean = 10))

rbind(df1, df2)
##   Year         val
## 1 2024 -0.06919795
## 2 2024 -0.16191750
## 3 2024 -1.11008560
## 4 2025  8.26355472
## 5 2025 10.14050837
## 6 2025 10.61355374

We can also bind an arbitrary number of data frames:

df3 <- data.frame(Year = 2026, val = rnorm(3, mean = -5))

rbind(df1, df2, df3)
##   Year         val
## 1 2024 -0.06919795
## 2 2024 -0.16191750
## 3 2024 -1.11008560
## 4 2025  8.26355472
## 5 2025 10.14050837
## 6 2025 10.61355374
## 7 2026 -5.61068190
## 8 2026 -4.09419745
## 9 2026 -5.28988243

The distinction between these last two operations is:

  • rbind() adds observations (rows).
  • cbind() adds variables (columns).

For rbind(), the data frames should use the same variables. For cbind(), the columns must all have the observations (rows) in the same order; otherwise your observations will have their variable values jumbled around.

Question 3:

The colleges data set can be loaded using:

colleges <- read.csv("https://nfriedrichsen.github.io/data/Colleges2019.csv")

  • A. Find the rows corresponding to Iowa State University and Grinnell College.

  • B. Make two data frames consisting of each row of data respectively.

  • C. Use rbind() to combine these into a new data frame and print the results.

Creating New Variables

Once we have a data frame, we will frequently want to construct new variables using information already contained in the data.

We have previously used $ to access an existing variable:

HappyPlanet$Happiness
##   [1] 5.5 5.6 4.3 7.1 5.0 7.9 7.8 5.3 5.3 5.8 7.6 6.6 3.0 6.1 6.5 5.9 4.7 7.6
##  [19] 5.5 3.6 5.9 2.9 4.9 3.9 8.0 4.0 5.4 6.3 6.7 7.3 3.6 3.9 8.5 6.4 6.7 7.2
##  [37] 6.9 8.1 5.7 7.6 6.4 6.7 6.7 5.6 4.0 8.0 7.1 4.3 7.2 4.7 6.8 7.4 4.0 6.5
##  [55] 5.2 7.0 7.2 5.7 7.8 5.5 5.7 5.6 5.4 8.1 7.1 6.9 6.7 6.8 6.0 6.1 3.7 6.3
##  [73] 6.7 5.0 6.2 5.4 4.7 5.8 7.7 5.5 3.7 4.4 6.6 3.8 7.1 5.0 7.7 5.7 5.7 5.6
##  [91] 3.8 4.5 5.3 7.7 7.8 7.1 3.8 4.8 8.1 5.6 5.0 7.8 6.9 5.9 5.5 6.5 5.9 5.9
## [109] 5.9 4.2 7.7 4.5 6.0 3.6 7.1 6.1 7.0 5.0 7.6 5.4 4.5 7.9 7.7 5.9 5.1 2.4
## [127] 6.3 2.6 6.7 5.9 5.5 4.5 5.3 7.2 7.4 7.9 6.8 6.0 6.9 6.5 5.2 4.3 2.8

We can also use $ and assignment to create a new variable. For example, the following logical condition produces one value for every country in the Happy Planet data:

HappyPlanet$Happiness > 6
##   [1] FALSE FALSE FALSE  TRUE FALSE  TRUE  TRUE FALSE FALSE FALSE  TRUE  TRUE
##  [13] FALSE  TRUE  TRUE FALSE FALSE  TRUE FALSE FALSE FALSE FALSE FALSE FALSE
##  [25]  TRUE FALSE FALSE  TRUE  TRUE  TRUE FALSE FALSE  TRUE  TRUE  TRUE  TRUE
##  [37]  TRUE  TRUE FALSE  TRUE  TRUE  TRUE  TRUE FALSE FALSE  TRUE  TRUE FALSE
##  [49]  TRUE FALSE  TRUE  TRUE FALSE  TRUE FALSE  TRUE  TRUE FALSE  TRUE FALSE
##  [61] FALSE FALSE FALSE  TRUE  TRUE  TRUE  TRUE  TRUE FALSE  TRUE FALSE  TRUE
##  [73]  TRUE FALSE  TRUE FALSE FALSE FALSE  TRUE FALSE FALSE FALSE  TRUE FALSE
##  [85]  TRUE FALSE  TRUE FALSE FALSE FALSE FALSE FALSE FALSE  TRUE  TRUE  TRUE
##  [97] FALSE FALSE  TRUE FALSE FALSE  TRUE  TRUE FALSE FALSE  TRUE FALSE FALSE
## [109] FALSE FALSE  TRUE FALSE FALSE FALSE  TRUE  TRUE  TRUE FALSE  TRUE FALSE
## [121] FALSE  TRUE  TRUE FALSE FALSE FALSE  TRUE FALSE  TRUE FALSE FALSE FALSE
## [133] FALSE  TRUE  TRUE  TRUE  TRUE FALSE  TRUE  TRUE FALSE FALSE FALSE

Since the resulting logical vector has one value for every row, we can store it as a new variable in the data frame:

HappyPlanet$HighHappiness = HappyPlanet$Happiness > 6

The data frame now contains an additional column:

colnames(HappyPlanet)
##  [1] "Country"        "Region"         "Happiness"      "LifeExpectancy"
##  [5] "Footprint"      "HLY"            "HPI"            "HPIRank"       
##  [9] "GDPperCapita"   "HDI"            "Population"     "HighHappiness"
table(HappyPlanet$HighHappiness)
## 
## FALSE  TRUE 
##    77    66

Notice that this uses the same logical conditions that we studied in Lab 2. Instead of using the logical vector only to count or subset observations, we have stored it as a new variable.

We can create numerical variables in the same way. For example, suppose we wanted life expectancy measured in approximate months instead of years:

HappyPlanet$LifeExpectancyMonths = 12 * HappyPlanet$LifeExpectancy

In both cases, the expression on the right side of = produces a vector containing one value for each observation. That vector is then stored as a new column of the data frame.

Question 4:

  • A. Create a new logical variable in HappyPlanet named HighLifeExpectancy that identifies countries whose life expectancy is at least 75 years.

  • B. Use table() to summarize the new variable.

  • C. How many countries have both HighLifeExpectancy == TRUE and a Happiness score greater than 6?

Modifying Existing Variables

Creating a new variable and modifying an existing variable use very similar syntax. The difference is whether the name on the left side of the assignment already exists in the data frame.

## Create a new variable
my_data$new_variable = ...

## Replace an existing variable
my_data$existing_variable = ...

We already modified a variable in Question 1 when we replaced Region with a character version of itself. In that situation, we decided that we no longer needed the original numerical representation of Region.

We can also modify only particular elements of a variable. Recall that square brackets allow us to access particular positions in a vector:

x = c(10, 20, 30, 40)
x[2]
## [1] 20

Assignment can be used with the same indexing syntax:

x[2] = 100
x
## [1]  10 100  30  40

Logical vectors can also be used to identify the positions that should be changed:

x[x > 50] = 50 # replace values 'larger than 50' with 50
x
## [1] 10 50 30 40

Thus, the logical conditions from Lab 2 can be used not only to find observations, but also to specify which values should be modified.

When deciding whether to overwrite an existing variable or create a new one, consider whether you may need the original information later. Changing how a variable is stored, such as coercing a variable from character to numeric, may provide little reason to preserve the old version. A transformation that changes the values themselves may be more useful to store as a new variable.

Question 5:

Using the ames_housing data:

Part A: Find the number of homes in this data set with missing values for the variable Garage.Type. Hint: is.na() and sum functions

Part B: Create a subset of ames_housing named ames_missing that contains homes with a missing value for the variable Garage.Type. What is the average Garage.Area of these homes?

Part C: Create a subset of ames_housing named ames_complete that contains no missing values for the variable Garage.Type. What is the average Garage.Area of these homes? (Hint: You should not get NA for the value)

Part D: A simple plot of the Garage.Area variable shows it is right-skewed. Many methods we’ve seen in 209 required variables to have a Normal distribution. A common fix for this, is that if we have a right-skewed distribution, some times applying the log-transform (taking the log of a variable) results in a transformed variable that looks Normal.

Using ames_complete, create a new variable named LogGarageArea containing the natural logarithm of Garage.Area. Use the log() function to perform this. Verify that the original Garage.Area still exists within this data set. Then run the following line of code exactly as written and describe the shape of the distribution of log-Garage-Area.

hist(ames_complete$LogGarageArea)