# Purrr and a Bit of Dplyr

Tuesday September 21, 2021

# Last week: Data frames and apply

• Data frames are a representation of the “classic” data table in R: rows are observations/cases, columns are variables/features
• Each column can be a different data type (but must be the same length)
• subset(): function for extracting rows of a data frame meeting a condition
• split(): function for splitting up rows of a data frame, according to a factor variable
• apply(): function for applying a given routine to rows or columns of a matrix or data frame
• lapply(): similar, but used for applying a routine to elements of a vector or list
• sapply(): similar, but will try to simplify the return type, in comparison to lapply()
• tapply(): function for applying a given routine to groups of elements in a vector or list, according to a factor variable

# Part I

Motivation: tidyverse

Here’s a basic breakdown for common iteration tasks that we encounter in R: we iterate over …

• elements of a list
• dimensions of an array (e.g., rows/columns of a matrix)
• sub data frames induced by one or more factors

All of this is possible in base R, using the apply family of functions: lapply(), sapply(), apply(), tapply(), etc. So why look anywhere else?

Answer: because some alternatives offer better consistency

• With the apply family of functions, there are some inconsistencies in both the interfaces to the functions, as well as their outputs
• This can both slow down learning and also lead to inefficiencies in practice (frequent checking and post-processing of results)

However, the world isn’t black-and-white: base R still has its advantages, and the best thing you can do is to be informed and well-versed in using all the major options!

# Why not plyr?

The plyr package used to be one of the most popular (most downloaded) R packages of all-time. It was more popular in the late 2000s and early 2010s

• All plyr functions are of the form **ply()
• Replace ** with characters denoting types:
• First character: input type, one of a, d, l
• Second character: output type, one of a, d, l, or _ (drop)

It is no longer under active development and that development is now happening elsewhere (mainly in the tidyverse). However, some people still like it. If you want to learn about it, you can check out the notes from a previous offering of this course

# What is purrr?

purrr is a package that is part of the tidyverse. It offers a family of functions for iterating (mainly over lists) that can be seen as alternatives to base R’s family of apply functions

• Compared to base R, they are more consistent
• Compared to plyr, they can often be faster

Credit: Jenny Bryan’s tutorial on purrr provided the inspiration for this lecture

# What is the tidyverse?

The tidyverse is a coherent collection of packages in R for data science (and tidyverse is itself a actually package that loads all its constituent packages). Packages include:

• Data wrangling: dplyr, tidyr, readr
• Iteration: purrr
• Visualization: ggplot2

This week we’ll cover purrr and a bit of dplyr. Next week we’ll do more dplyr, and some tidyr. (Many of you will learn ggplot2 in Statistical Graphics 36-315)

Loading the tidyverse so that we can get all this functionality (plus more):

library(tidyverse)

# Part II

map() and friends

# The map family

purrr offers a family of map functions, which allow you to apply a function across different chunks of data (primarily used with lists). Offers an alternative base R’s apply functions. Summary of functions:

• map(): apply a function across elements of a list or vector
• map_dbl(), map_lgl(), map_chr(): same, but return a vector of a particular data type
• map_dfr(), map_dfc(): same, but return a data frame

# map(): list in, list out

The map() function is an alternative to lapply(). It has the following simple form: map(x, f), where x is a list or vector, and f is a function. It always returns a list

my.list = list(nums=seq(0.1,0.6,by=0.1), chars=letters[1:12],
bools=sample(c(TRUE,FALSE), 6, replace=TRUE))
map(my.list, length)
## $nums ## [1] 6 ## ##$chars
## [1] 12
##
## $bools ## [1] 6 # Base R is just as easy lapply(my.list, length) ##$nums
## [1] 6
##
## $chars ## [1] 12 ## ##$bools
## [1] 6

# map_dbl(): list in, numeric out

The map_dbl() function is an alternative to sapply(). It has the form: map_dbl(x, f), where x is a list or vector, and f is a function that returns a numeric value (when applied to each element of x)

Similarly:

• map_int() returns an integer vector
• map_lgl() returns a logical vector
• map_chr() returns a character vector
map_dbl(my.list, length)
##  nums chars bools
##     6    12     6
map_chr(my.list, length)
##  nums chars bools
##   "6"  "12"   "6"
# Base R is a bit more complicated
as.numeric(sapply(my.list, length)) 
## [1]  6 12  6
as.numeric(unlist(lapply(my.list, length)))
## [1]  6 12  6
vapply(my.list, FUN=length, FUN.VALUE=numeric(1))
##  nums chars bools
##     6    12     6

# Applying a custom function

As before (with the apply family), we can of course apply a custom function, and define it “on-the-fly”

library(repurrrsive) # Load Game of Thrones data set
class(got_chars)
## [1] "list"
class(got_chars[[1]])
## [1] "list"
names(got_chars[[1]])
##  [1] "url"         "id"          "name"        "gender"      "culture"     "born"        "died"        "alive"       "titles"      "aliases"     "father"
## [12] "mother"      "spouse"      "allegiances" "books"       "povBooks"    "tvSeries"    "playedBy"
map_chr(got_chars, function(x) { return(x$name) }) ## [1] "Theon Greyjoy" "Tyrion Lannister" "Victarion Greyjoy" "Will" "Areo Hotah" "Chett" "Cressen" ## [8] "Arianne Martell" "Daenerys Targaryen" "Davos Seaworth" "Arya Stark" "Arys Oakheart" "Asha Greyjoy" "Barristan Selmy" ## [15] "Varamyr" "Brandon Stark" "Brienne of Tarth" "Catelyn Stark" "Cersei Lannister" "Eddard Stark" "Jaime Lannister" ## [22] "Jon Connington" "Jon Snow" "Aeron Greyjoy" "Kevan Lannister" "Melisandre" "Merrett Frey" "Quentyn Martell" ## [29] "Samwell Tarly" "Sansa Stark" # Extracting elements Handily, the map functions all allow the second argument to be an integer or string, and treat this internally as an appropriate extractor function map_chr(got_chars, "name") ## [1] "Theon Greyjoy" "Tyrion Lannister" "Victarion Greyjoy" "Will" "Areo Hotah" "Chett" "Cressen" ## [8] "Arianne Martell" "Daenerys Targaryen" "Davos Seaworth" "Arya Stark" "Arys Oakheart" "Asha Greyjoy" "Barristan Selmy" ## [15] "Varamyr" "Brandon Stark" "Brienne of Tarth" "Catelyn Stark" "Cersei Lannister" "Eddard Stark" "Jaime Lannister" ## [22] "Jon Connington" "Jon Snow" "Aeron Greyjoy" "Kevan Lannister" "Melisandre" "Merrett Frey" "Quentyn Martell" ## [29] "Samwell Tarly" "Sansa Stark" map_lgl(got_chars, "alive") ## [1] TRUE TRUE TRUE FALSE TRUE FALSE FALSE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE FALSE TRUE TRUE TRUE TRUE FALSE TRUE FALSE ## [28] FALSE TRUE TRUE Interestingly, we can actually do the following in base R: [() and [[() are functions that act in the following way for an integer x and index i • [(x, i) is equivalent to x[i] • [[(x, i) is equivalent to x[[i]] (This works whether i is an integer or a string) sapply(got_chars, [[, "name") ## [1] "Theon Greyjoy" "Tyrion Lannister" "Victarion Greyjoy" "Will" "Areo Hotah" "Chett" "Cressen" ## [8] "Arianne Martell" "Daenerys Targaryen" "Davos Seaworth" "Arya Stark" "Arys Oakheart" "Asha Greyjoy" "Barristan Selmy" ## [15] "Varamyr" "Brandon Stark" "Brienne of Tarth" "Catelyn Stark" "Cersei Lannister" "Eddard Stark" "Jaime Lannister" ## [22] "Jon Connington" "Jon Snow" "Aeron Greyjoy" "Kevan Lannister" "Melisandre" "Merrett Frey" "Quentyn Martell" ## [29] "Samwell Tarly" "Sansa Stark" sapply(got_chars, [[, "alive") ## [1] TRUE TRUE TRUE FALSE TRUE FALSE FALSE TRUE TRUE TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE FALSE TRUE TRUE TRUE TRUE FALSE TRUE FALSE ## [28] FALSE TRUE TRUE # Part III map_dfr(), map_dfc(), dplyr # map_dfr() and map_dfc(): list in, data frame out The map_dfr() and map_dfc() functions iterate a function call over a list or vector, but automatically combine the results into a data frame. They differ in whether that data frame is formed by row-binding or column-binding map_dfr(got_chars, [, c("name", "alive")) ## # A tibble: 30 × 2 ## name alive ## <chr> <lgl> ## 1 Theon Greyjoy TRUE ## 2 Tyrion Lannister TRUE ## 3 Victarion Greyjoy TRUE ## 4 Will FALSE ## 5 Areo Hotah TRUE ## 6 Chett FALSE ## 7 Cressen FALSE ## 8 Arianne Martell TRUE ## 9 Daenerys Targaryen TRUE ## 10 Davos Seaworth TRUE ## # … with 20 more rows # Base R is much less convenient data.frame(name = sapply(got_chars, [[, "name"), alive = sapply(got_chars, [[, "alive")) ## name alive ## 1 Theon Greyjoy TRUE ## 2 Tyrion Lannister TRUE ## 3 Victarion Greyjoy TRUE ## 4 Will FALSE ## 5 Areo Hotah TRUE ## 6 Chett FALSE ## 7 Cressen FALSE ## 8 Arianne Martell TRUE ## 9 Daenerys Targaryen TRUE ## 10 Davos Seaworth TRUE ## 11 Arya Stark TRUE ## 12 Arys Oakheart FALSE ## 13 Asha Greyjoy TRUE ## 14 Barristan Selmy TRUE ## 15 Varamyr FALSE ## 16 Brandon Stark TRUE ## 17 Brienne of Tarth TRUE ## 18 Catelyn Stark FALSE ## 19 Cersei Lannister TRUE ## 20 Eddard Stark FALSE ## 21 Jaime Lannister TRUE ## 22 Jon Connington TRUE ## 23 Jon Snow TRUE ## 24 Aeron Greyjoy TRUE ## 25 Kevan Lannister FALSE ## 26 Melisandre TRUE ## 27 Merrett Frey FALSE ## 28 Quentyn Martell FALSE ## 29 Samwell Tarly TRUE ## 30 Sansa Stark TRUE Note: the first example uses extra arguments; the map functions work just like the apply functions in this regard # dplyr The map_dfr() and map_dfc() functions actually depend on another package called the dplyr, hence require the latter to be installed What is dplyr? It is another tidyverse package that is very useful for data frame computations. You’ll learn more soon, but for now, you can think of it as providing the tidyverse alternative to the base R functions subset(), split(), tapply() # filter(): subset rows based on a condition head(mtcars) # Built in data frame of cars data, 32 cars x 11 variables ## mpg cyl disp hp drat wt qsec vs am gear carb ## Mazda RX4 21.0 6 160 110 3.90 2.620 16.46 0 1 4 4 ## Mazda RX4 Wag 21.0 6 160 110 3.90 2.875 17.02 0 1 4 4 ## Datsun 710 22.8 4 108 93 3.85 2.320 18.61 1 1 4 1 ## Hornet 4 Drive 21.4 6 258 110 3.08 3.215 19.44 1 0 3 1 ## Hornet Sportabout 18.7 8 360 175 3.15 3.440 17.02 0 0 3 2 ## Valiant 18.1 6 225 105 2.76 3.460 20.22 1 0 3 1 filter(mtcars, (mpg >= 20 & disp >= 200) | (drat <= 3)) ## mpg cyl disp hp drat wt qsec vs am gear carb ## Hornet 4 Drive 21.4 6 258 110 3.08 3.215 19.44 1 0 3 1 ## Valiant 18.1 6 225 105 2.76 3.460 20.22 1 0 3 1 ## Cadillac Fleetwood 10.4 8 472 205 2.93 5.250 17.98 0 0 3 4 ## Lincoln Continental 10.4 8 460 215 3.00 5.424 17.82 0 0 3 4 ## Dodge Challenger 15.5 8 318 150 2.76 3.520 16.87 0 0 3 2 # Base R is just as easy with subset(), more complicated with direct indexing subset(mtcars, (mpg >= 20 & disp >= 200) | (drat <= 3)) ## mpg cyl disp hp drat wt qsec vs am gear carb ## Hornet 4 Drive 21.4 6 258 110 3.08 3.215 19.44 1 0 3 1 ## Valiant 18.1 6 225 105 2.76 3.460 20.22 1 0 3 1 ## Cadillac Fleetwood 10.4 8 472 205 2.93 5.250 17.98 0 0 3 4 ## Lincoln Continental 10.4 8 460 215 3.00 5.424 17.82 0 0 3 4 ## Dodge Challenger 15.5 8 318 150 2.76 3.520 16.87 0 0 3 2 mtcars[(mtcars$mpg >= 20 & mtcars$disp >= 200) | (mtcars$drat <= 3), ]
##                      mpg cyl disp  hp drat    wt  qsec vs am gear carb
## Hornet 4 Drive      21.4   6  258 110 3.08 3.215 19.44  1  0    3    1
## Valiant             18.1   6  225 105 2.76 3.460 20.22  1  0    3    1
## Cadillac Fleetwood  10.4   8  472 205 2.93 5.250 17.98  0  0    3    4
## Lincoln Continental 10.4   8  460 215 3.00 5.424 17.82  0  0    3    4
## Dodge Challenger    15.5   8  318 150 2.76 3.520 16.87  0  0    3    2

# group_by(): define groups of rows based on columns or conditions

head(group_by(mtcars, cyl), 2)
## # A tibble: 2 × 11
## # Groups:   cyl [1]
##     mpg   cyl  disp    hp  drat    wt  qsec    vs    am  gear  carb
##   <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1    21     6   160   110   3.9  2.62  16.5     0     1     4     4
## 2    21     6   160   110   3.9  2.88  17.0     0     1     4     4
• This doesn’t actually change anything about the way the data frame looks
• Only difference is that when it prints, we’re told about the groups
• But it will play a big role in how dplyr functions act on the data frame

# summarize(): apply computations to (groups of) rows of a data frame

# Ungrouped
summarize(mtcars, mpg = mean(mpg), hp = mean(hp))
##        mpg       hp
## 1 20.09062 146.6875
# Grouped by number of cylinders
summarize(group_by(mtcars, cyl), mpg = mean(mpg), hp = mean(hp))
##        mpg       hp
## 1 20.09062 146.6875

Note: the use of group_by() makes the difference here

# Base R, ungrouped calculation is not so bad
c("mpg" = mean(mtcars$mpg), "hp" = mean(mtcars$hp))
##       mpg        hp
##  20.09062 146.68750
# Base R, grouped calculation is getting a bit ugly
cbind(tapply(mtcars$mpg, INDEX=mtcars$cyl, FUN=mean),
tapply(mtcars$hp, INDEX=mtcars$cyl, FUN=mean))
##       [,1]      [,2]
## 4 26.66364  82.63636
## 6 19.74286 122.28571
## 8 15.10000 209.21429
sapply(split(mtcars, mtcars$cyl), FUN=function(df) { return(c("mpg" = mean(df$mpg), "hp" = mean(df$hp))) }) ## 4 6 8 ## mpg 26.66364 19.74286 15.1000 ## hp 82.63636 122.28571 209.2143 aggregate(mtcars[, c("mpg", "hp")], by=list(mtcars$cyl), mean)
##   Group.1      mpg        hp
## 1       4 26.66364  82.63636
## 2       6 19.74286 122.28571
## 3       8 15.10000 209.21429

# Summary

• The tidyverse is a collection of packages for common data science tasks
• purrr is one such package that provides a consistent family of iteration functions
• map(): list in, list out
• map_dbl(), map_lgl(), map_chr(): list in, vector out (of a particular data type)
• map_dfr(), map_dfc(): list in, data frame out (row-binded or column-binded)
• dplyr is another such package that provides functions for data frame computations
• filter(): subset rows based on a condition
• group_by(): define groups of rows according to a condition
• summarize(): apply computations across groups of rows