---
title: "Plyr: `d*ply()`"
author: "Statistical Computing, 36-350"
date: "Friday November 11, 2016"
---

Reminder: split-apply-combine and `plyr`
===

- The split-apply-combine framework is very helpful for breaking up large data analysis tasks, on ragged data, into manageable chunks 
- Apply functions are very important in the context of the split-apply-combine paradigm
- The `plyr` package makes life easier by making the input and outputs types (array, data frame, or list?) as explicit as possible

`d*ply()` : the input is a data frame
===

The signature for all `d*ply()` functions is:

```
d*ply(.data, .variables, .fun, ...)
```

- `.data` : a data frame
- `.variables` : variable (or variables) to split the data frame by
- `.fun` : the function to be applied to each piece
- `...` : additional arguments to be passed to the function

Note that this looks like:

```
tapply(X, INDEX, FUN, ...)
```

Strikes data set, revisited
===

Recall, data set on political economy of strikes: 

```{r}
strikes.df = read.csv("http://www.stat.cmu.edu/~ryantibs/statcomp-F16/data/strikes.csv")
head(strikes.df)

# Function to compute coefficients from regressing number of strikes (per 
# 1000 workers) on leftwing share of the government
my.strike.lm = function(country.df) {
  coef(lm(strike.volume ~ left.parliament, data=country.df))
}

# Getting regression coefficients separately for each country, old way:
strikes.list = split(strikes.df, f=strikes.df$country)
strikes.coefs = sapply(strikes.list, my.strike.lm)
head(strikes.coefs)
```

(Continued)
===

```{r}
# Getting regression coefficient separately for each country, new way, in 
# three formats:
library(plyr)
strikes.coefs.a = daply(strikes.df, .(country), my.strike.lm)
head(strikes.coefs.a) # Get back an array, note the difference to sapply()

strikes.coefs.d = ddply(strikes.df, .(country), my.strike.lm)
head(strikes.coefs.d) # Get back a data frame

strikes.coefs.l = dlply(strikes.df, .(country), my.strike.lm)
head(strikes.coefs.l) # Get back a list
```

Splitting on two (or more) variables
===

The function `d*ply()` makes it very easy to split on two (or more) variables: we just specify them, separated by a "," in the `.variables` argument

```{r}
# First create a variable that indicates whether the year is pre 1975, and add
# it to the data frame
strikes.df$yearPre1975 = strikes.df$year <= 1975
# Then use (say) ddply() to compute regression coefficients for each country 
# pre and post 1975
strikes.coefs.1975 = ddply(strikes.df, .(country, yearPre1975), my.strike.lm)
dim(strikes.coefs.1975) # Note that there are 18 x 2 = 36 rows
head(strikes.coefs.1975)
```

(Continued)
===

```{r}
# We can also create factor variables on-the-fly with I(), as we've seen before
strikes.coefs.1975 = ddply(strikes.df, .(country, I(year<=1975)), my.strike.lm)
dim(strikes.coefs.1975) # Again, there are 18 x 2 = 36 rows
head(strikes.coefs.1975)
```

Parallelization
===

What happens if we have a really large data set and we want to use split-apply-combine?

If the individual tasks are unrelated, then we should be speed up the computation by performing them **in parallel**

The `plyr` functions make this quite easy: let's take a look at the full signature for `daply()`:

```
daply(.data, .variables, .fun = NULL, ..., .progress = "none",
  .inform = FALSE, .drop_i = TRUE, .drop_o = TRUE, .parallel = FALSE,
  .paropts = NULL)
```

The second to last argument `.parallel` (default FALSE) is for parallelization. If set to TRUE, then it performs the individual tasks in parallel, using the `foreach` package

The last argument `.paropts` is for more advanced parallelization, these are additional arguments to be passed to `foreach`

(Continued)
===

For more, read the `foreach` package first. May take some time to set up the parallel backend (this is often system specific)

But once set up, parallelization is simple and beautiful with `**ply()`! The difference is just, e.g., `daply(strikes.df, .(country), my.strike.lm)` versus `daply(strikes.df, .(country), my.strike.lm, .parallel=TRUE)`