---
title: "The Apply Family"
author: "Statistical Computing, 36-350"
date: "Wednesday October 12, 2016"
---

The apply family
===

R offers a family of **apply functions**, which allow you to apply a function across different chunks of data. Offers an alternative to explicit iteration using `for()` loop. Can be simpler and faster than a `for()` loop, though not always

Below is a summary. We'll cover `apply()` today, and the rest next time

- `apply()`: apply a function to rows or columns of a matrix or data frame
- `lapply()`: apply a function to elements of a list or vector
- `sapply()`: same as the above, but simplify the output (if possible)
- `tapply()`: apply a function to levels of a factor vector

`apply()`, rows or columns of a matrix or data frame
===

The `apply()` function takes inputs of the following form:

- `apply(x, MARGIN=1, FUN=my.fun)`, to apply `my.fun()` across rows of a matrix or data frame `x`
- `apply(x, MARGIN=2, FUN=my.fun)`, to apply `my.fun()` across columns of a matrix or data frame `x`

```{r}
x = matrix(rnorm(9), 3, 3) # Create a 9 x 9 matrix of random normals
x
apply(x, MARGIN=1, FUN=min) # Smallest entry in each row
apply(x, MARGIN=1, FUN=sum) # Sum of entries in each row
```

(Continued)
===

```{r}
head(state.x77) # Matrix of states data, 50 states x 8 variables
apply(state.x77, MARGIN=2, FUN=max) # Maximum entry in each column
apply(state.x77, MARGIN=2, FUN=which.max) # Index of the max in each column
apply(state.x77, MARGIN=2, FUN=summary) # Summary of each col, get back matrix!
```

Applying a custom function 
===

For a custom function, we can just define it before hand, and the use `apply()` as usual

```{r}
# Our custom function
my.fun = function(v) {  v.mean = mean(v) 
  q1 = quantile(v, prob=0.1)
  q2 = quantile(v, prob=0.9)
  cat(paste("The 0.1 quantile is", q1, "! "))
  cat(paste("The 0.9 quantile is", q2, "!\n"))
  v.mean = mean(v) # Regular mean
  v.trimmed.mean = mean(v[q1 <= v & v <= q2]) # Trimmed mean!
  c(v.mean, v.trimmed.mean)
}

mat = apply(state.x77, MARGIN=2, FUN=my.fun) # We get back a matrix
mat # First row is the mean, second row is the trimmed mean
```

Applying a custom function "on-the-fly"
===

Instead of defining a custom function before hand, we can just define it "on-the-fly". Sometimes this is more convenient

```{r}
# Compute trimmed means, defining custom function on-the-fly
apply(state.x77, MARGIN=2, FUN=function(v) { 
  q1 = quantile(v, prob=0.1)
  q2 = quantile(v, prob=0.9)
  mean(v[q1 <= v & v <= q2]) # Trimmed mean!
})
```

Applying a function that takes extra arguments
===

Sometimes we want to use a function over rows or columns of a matrix, that takes **extra arguments** (besides the row or column itself). We can pass these as inputs to `apply()`, as in: `apply(x, MARGIN=1, FUN=my.fun, extra.arg.1, extra.arg.2)`, for two extra arguments `extra.arg.1`, `extra.arg.2` to be passed to `my.fun()`

```{r}
# Function that gets indices of the biggest 3 entries of v, then returns the
# corresponding 3 elements of names.v
top.3.names = function(v, names.v) { names.v[order(v, decreasing=TRUE)[1:3]] }
# Now we'll run this function on each column of state.x77. Note: here v will
# be a column, and for names.v, we'll pass in rownames(state.x77), i.e., the
# state names
apply(state.x77, MARGIN=2, FUN=top.3.names, names.v=rownames(state.x77))
```

What's the return argument?
===

What kind of data type will `apply()` give us? Depends on what function we pass. Summary, say, with `FUN=my.fun()`:

- If `my.fun()` returns a single value, then `apply()` will return a vector
- If `my.fun()` returns k values, then `apply()` will return a matrix with k rows (note: this is true regardless of whether `MARGIN=1` or `MARGIN=2`)
- If `my.fun()` returns different length output for different inputs, then `apply()` will return a list
- If `my.fun()` returns a list, then `apply()` will return a list

Optimized functions for special tasks
===

**Don't overuse** the apply paradigm! There's lots of special functions that **optimized** are will be both simpler and faster than using `apply()`. E.g.,

- `rowSums()`, `colSums()`: for computing row, column sums of a matrix
- `rowMeans()`, `colMeans()`: for computing row, column means of a matrix
- `max.col()`: for finding the maximum position in each row of a matrix

Combining these functions with logical indexing and vectorized operations will enable you to do quite a lot. E.g., how to count the number of positives in each row of a matrix?

```{r}
x
# DON'T do this (much slower for big matrices)
apply(x, MARGIN=1, function(v) { sum(v>0) })
# DO do this (much faster, simpler for big matrices)
rowSums(x > 0)
```