---
title: 'Lecture 15: Better Apply Functions with `plyr`'
author: "Statistical Computing, 36-350"
date: "Wednesday November 11, 2015"
---

Outline
===

- Quick review: split-apply-combine
- Quick review: apply functions in base R
- New apply functions from `plyr`

General strategy
===

These are tools that, when put together, give us three general steps:

- **Split** whatever data object we have into meaningful chunks
- **Apply** the function of interest to this division
- **Combine** the results into a new object

As in:

```
x = split(strikes, strikes$country) # Split
y = sapply(x, my.strike.lm)         # Apply and combine
```

`*apply()` in base R
===

- `apply()` : arrays
- `lapply()` : list or vector, output list
- `sapply()` : list or vector, simplify to vector
- `vapply()` : list or vector, safer simplify to vector
- `tapply()`, `by()`, `aggregate()` : data frames(tables)
- `mapply()` : multiple vectors(special case of 2d array)

Useful set, but some issues: inconsistent output

Back to the drawing board: `plyr`
===

We have three common data structures beyond vectors: 
arrays(matrices), data frames, lists

```{r}
library(plyr)
```

Most popular R package of all time(most downloads)!

Functions within: `**ply()`, replace `**` with characters denoting types:

- First character: input type, one of `a`, `d`, `l` 
- Second character: output type, one of `a`, `d`, `l`, or `_` (drop)


`a*ply()` : input arrays
===

```
processed = a*ply(.data, .margins, .fun, ...)
```

- `.data` : an array
- `.margins` : subscripts which the function gets applied over
- `.fun` : the function to be applied
- `...` : additional arguments to function

Looks like

```
processed = apply(data, margin, function)
```

---

Let's test it out!

```{r}
my.boring.array = array(1:27, c(3,3,3))
rownames(my.boring.array) = c("Curly", "Larry", "Moe")
colnames(my.boring.array) = c("Groucho", "Harpo", "Zeppo")
dimnames(my.boring.array)[[3]] = c("Bart", "Lisa", "Maggie")
adply(my.boring.array, 1, sum)
```

---

```{r}
alply(my.boring.array, 1, sum)
```

---

```{r}
aaply(my.boring.array, 1, sum)
```

---

```{r}
adply(my.boring.array, 2:3, sum)
```

---

```{r}
alply(my.boring.array, 2:3, sum)
```

---

```{r}
aaply(my.boring.array, 2:3, sum)
```

One more:
===

Return no values, but use outcomes in secondary functions

```{r}
pdf("lotsaplots.pdf")
a_ply(my.boring.array, 2:3, plot)
graphics.off()
```

`d*ply()` : operate on data frames
===

```
processed = d*ply(.data, .(splitvariable), .fun, ...)
```

- `.data` : an array
- `.(...)` : arguments to split by
- `.fun` : the function to be applied
- `...` : additional arguments to function

Looks like

```
processed = tapply(data, data$splitvariable, function)
```
---

Strikes example again:

```{r}
strikes = read.csv("http://www.stat.cmu.edu/~ryantibs/statcomp-F15/lectures/strikes.csv")
my.strike.lm.better = function(country.df) {
  return(lm(strike.volume ~ left.parliament + unemployment + inflation,
            data=country.df)$coefficients)
}
```

---

Syntax: for data frame inputs, include the variables for the splitting in a 
stand-in function `.()` : syntax like `by()`

```{r}
results = daply(strikes, .(country), my.strike.lm.better)
results[1:5,]
```

---

```{r}
results = ddply(strikes, .(country), my.strike.lm.better)
results[1:5,]
```

---
```{r}
results = dlply(strikes, .(country), my.strike.lm.better)
results[1:3]
```


`l*ply()` : operate on lists
===

```
processed = l*ply(.data, , .fun, ...)
```

- `.data` : an array
- `.fun` : the function to be applied
- `...` : additional arguments to function

Looks like

```
processed = lapply(data, function, ...)
```

---

```{r}
strikes.split = split(strikes, strikes$country)
results = laply(strikes.split, my.strike.lm.better)
results[1:5,]
```

---

```{r}
results = ldply(strikes.split, my.strike.lm.better)
results[1:5,]
```

---

```{r}
results = llply(strikes.split, my.strike.lm.better)
results[1:3]
```

More dimensions
===

NHL data from the 2013-2014 season

```{r}
print(load(url("http://www.stat.cmu.edu/~ryantibs/statcomp-F15/lectures/nhl-2013.RData")))
head(teamtable)
colnames(teamtable)
teamtable = subset(teamtable, scorediffcat < 6)
```

---

Split by team name, add up performance measures:

```{r}
teamperf = ddply(teamtable, .(team), 
                 function(dd) colSums(dd[,c("GF","GA","HIT","HIT_TAKEN")]))
head(teamperf)
```

---

Split by team name and home/away, add up performance measures

```{r}
teamperf = ddply(teamtable, .(team, home), 
                 function(dd) colSums(dd[,c("GF","GA","HIT","HIT_TAKEN")]))
head(teamperf)
```

---

Split by team name and game ID: the lethal case from last time

```{r}
teamperf = daply(teamtable, .(team, gcode), 
                 function(dd) colSums(dd[,c("GF","GA","HIT","HIT_TAKEN")]))
dim(teamperf)  # Worst case: 30 x 1323 rows.
head(teamperf)
```

Summary
===

- Apply functions are very important in the context of the split-apply-combine
  paradigm
- `plyr` makes life easier by making the input and outputs types (array, data frame,
  or list?) as explicit as possible