---
title: 'Lecture 2: More Data Structures'
author: "Statistical Computing, 36-350"
date: "Wednesday September 2, 2015"
---

Outline
===

- Arrays
- Matrices
- Lists
- Data frames
- Structures of structures

Vector structures, starting with arrays
===

Many data structures in R are made by adding bells and whistles to vectors, 
i.e., they are "vector structures"

Most useful: **arrays**

```{r}
x = c(7, 8, 10, 45)
x.arr = array(x, dim=c(2,2))
x.arr
```

`dim` says how many rows and columns; filled by columns

Can have 3, 4, ...  arrays; `dim` is vector of arbitrary length

---

Some properties of our array:

```{r}
dim(x.arr)
is.vector(x.arr)
is.array(x.arr)
```

---

```{r}
typeof(x.arr)
str(x.arr)
attributes(x.arr)
```

`typeof()` returns the type of the array elements

`str()` gives the structure: here, a numeric array, with two 
dimensions, both indexed 1--2, and then the actual numbers

Exercise: try all these with `x`

Accessing and indexing arrays
===

Can access a 2d array either by pairs of indices or by the 
underlying vector:

```{r}
x.arr[1,2]
x.arr[3]
```

---

Omitting an index means "all of it":

```{r}
x.arr[c(1:2),2]
x.arr[,2]
```

Functions on arrays
===

Many functions applied to a vector-structure like an array 
will just boil things down to the underlying vector:

```{r}
which(x.arr > 9)
```

This happens unless the function is set up to handle 
arrays specifically

---

Many functions _do_ preserve array structure:

```{r}
y = -x
y.arr = array(y,dim=c(2,2))
y.arr + x.arr
```

Others specifically act on each row or column of the 
array separately:

```{r}
rowSums(x.arr)
```

<small>(We will see a lot more of this idea soon)</small>

Example: houses prices in Pennsylvania
===

Census data for California and Pennsylvania on housing prices,
by Census "tract"

```{r}
calif_penn = read.csv("http://www.stat.cmu.edu/~cshalizi/uADA/13/hw/01/calif_penn_2011.csv")
penn = calif_penn[calif_penn[,"STATEFP"]==42,]
coefficients(lm(Median_house_value ~ Median_household_income, data=penn))
```

Fit a simple linear model, predicting median house price from median 
household income

---

It turns out census tracts 24--425 are Allegheny county

Tract 24 has a median income of \$14,719; actual median house 
value is \$34,100; is that above or below what's predicted?

```{r}
34100 < -26206.564 + 3.651*14719
```

Tract 25 has income \$48,102 and house price \$155,900

```{r}
155900 < -26206.564 + 3.651*48102
```

What about tract 26?

---

We could just keep plugging in numbers like this, but that's

- boring and repetitive
- error-prone (what if I forget to change the median income, 
or drop a minus sign from the intercept?)
- obscure if we come back to our work later (what are these 
numbers, again?)

Use variables and names
===

```{r}
penn.coefs = coefficients(lm(Median_house_value ~ Median_household_income, data=penn))
penn.coefs
allegheny.rows = 24:425
allegheny.medinc = penn[allegheny.rows,"Median_household_income"]
allegheny.values = penn[allegheny.rows,"Median_house_value"]
allegheny.fitted = penn.coefs["(Intercept)"] +  
  penn.coefs["Median_household_income"]*allegheny.medinc
```

---

```{r}
plot(x=allegheny.fitted, y=allegheny.values,
     xlab="Model-predicted median house values",
     ylab="True median house values",
     xlim=c(0,5e5), ylim=c(0,5e5))
abline(a=0, b=1, col="red")
```

Running example: resource allocation 
====

Factory makes cars and trucks, using labor and steel

- a car takes 40 hours of labor and 1 ton of steel
- a truck takes 60 hours and 3 tons of steel
- resources: 1600 hours of labor and 70 tons of steel 
each week

Matrices
===

In R, a matrix is a specialization of a 2d array

```{r}
factory = matrix(c(40,1,60,3), nrow=2)
factory
is.array(factory)
is.matrix(factory)
```

could also specify `ncol`; to fill by rows, use `byrow=TRUE` 

Elementwise operations with the usual arithmetic and comparison 
operators (e.g., `factory/3`)

Compare whole matrices with `identical()` or `all.equal()`

Matrix multiplication
===

Has its own special operator, written `%*%`:

```{r}
six.sevens = matrix(rep(7,6), ncol=3)
six.sevens
factory %*% six.sevens # [2x2] * [2x3]
```

<small>(What happens if you try `six.sevens %*% factory`?)</small>

Multiplying matrices and vectors
===

Numeric vectors can act like proper vectors:

```{r}
output = c(10,20)
factory %*% output
output %*% factory
```

<small>(R silently casts the vector as either a 1-column or 
1-row matrix, as appropriate)</small>

Matrix operators
===

Transpose:
```{r}
t(factory)
```

Determinant:
```{r}
det(factory)
```

The matrix diagonal
===

The `diag()` function can be used to extract the diagonal 
entries of a matrix:

```{r}
diag(factory)
```

It can also be used to change the diagonal:

```{r}
diag(factory) = c(35,4)
factory
```

Re-set it for later:

```{r}
diag(factory) = c(40,3)
```

Creating a diagonal or identity matrix
===

```{r}
diag(c(3,4))
diag(2)
```

<small>(How do you get a 1 x 1 matrix containing a single entry 2?)</small>

Inverting a matrix
===

```{r}
solve(factory)
factory %*% solve(factory)
```

Why is it called "solve" anyway?
===

Solving the linear system $Ax = b$ for $x$:

```{r}
available = c(1600,70)
solve(factory,available)
factory %*% solve(factory,available)
```

Names in matrices
===

We can name either rows or columns or both, with 
`rownames()` and `colnames()`

These are just character vectors, and we use the same function to 
get and to set their values

Names help us understand what we're working with

Names can be used to coordinate different objects

---

```{r}
rownames(factory) = c("labor","steel")
colnames(factory) = c("cars","trucks")
factory
available = c(1600,70)
names(available) = c("labor","steel")
```

---

```{r}
output = c(20,10)
names(output) = c("trucks","cars")
factory %*% output # But we've got cars and trucks mixed up!
factory %*% output[colnames(factory)]
all(factory %*% output[colnames(factory)] <= available[rownames(factory)])
```

Note that last lines don't have to change if we add motorcycles 
as output or rubber and glass as inputs (abstraction again)

Doing the same thing to each row or column
===

Take the mean: `rowMeans()`, `colMeans()`, input is matrix,
output is vector.  Also `rowSums()`, `colSums`

`summary()`: vector-style summary of column

```{r}
colMeans(factory)
summary(factory)
```

---

`apply()`, takes 3 arguments: 

- the array or matrix, 
- then 1 for rows and 2 for columns, 
- then a name of the function to apply to each

```{r}
rowMeans(factory)
apply(factory, 1, mean)
```

<small>(What would `apply(factory, 1, sd)` do?)</small>

Lists
====

Sequence of values, not necessarily all of the same type

```{r}
my.distribution = list("exponential", 7, FALSE)
my.distribution
```

Most of what you can do with vectors you can 
also do with lists

Accessing pieces of lists
===

Can use `[ ]` as with vectors  
Or use `[[ ]]`, but only with a single index  
`[[ ]]` drops names and structures, `[ ]` does not

```{r}
my.distribution[2]
my.distribution[[2]]
my.distribution[[2]]^2
```
<small>(What happens if you try `my.distribution[2]^2`?)</small>
<small>(What happens if you try `[[ ]]` on a vector?)</small>

Expanding and contracting lists
===

Add to lists with `c()` (also works with vectors):

```{r}
my.distribution = c(my.distribution,7)
my.distribution
```

---

Chop off the end of a list by setting the length to something 
smaller (also works with vectors):

```{r}
length(my.distribution)
length(my.distribution) = 3
my.distribution
```

---

Pluck out all but one piece of a list (also works with vectors):

```{r}
my.distribution[-2]
```

<small>(What happens if you try `my.distribution[[-2]]`?)</small>

Naming list elements
===
We can name some or all of the elements of a list:

```{r}
names(my.distribution) = c("family","mean","is.symmetric")
my.distribution
my.distribution[["family"]]
my.distribution["family"]
```

---

Lists have a special shortcut way of using names, 
with `$`:

```{r}
my.distribution[["family"]]
my.distribution$family
```

Names in lists (continued)
===

Creating a list with names:

```{r}
another.distribution = list(family="gaussian",
                            mean=7, sd=1, is.symmetric=TRUE)
```

Adding named elements:

```{r}
my.distribution$was.estimated = FALSE
my.distribution[["last.updated"]] = "2015-09-01"
```

Removing a named list element, by assigning it the 
value `NULL`:

```{r}
my.distribution$was.estimated = NULL
```

Key-value pairs
===

Lists give us a natural way to store and look up data by _name_, 
rather than by _position_

A really useful programming concept with many names: 
**key-value pairs**, **dictionaries**, **associative arrays**

If all our distributions have components named `family`, we 
can look that up by name, without caring where it is (in what position
it lies) in the list

Data frames
===

The classic data table, $n$ rows for cases, $p$ columns for variables

Lots of the really-statistical parts of R presume data frames  

Not just a matrix because _columns can have different types_

Many matrix functions also work for data frames (e.g.,`rowSums()`, 
`summary()`, `apply()`)

<small>(But no matrix multiplication with data frames, 
even if all columns are numeric!)</small>

---

```{r}
a.matrix = matrix(c(35,8,10,4), nrow=2)
colnames(a.matrix) = c("v1","v2")
a.matrix
a.matrix[,"v1"]  # Try a.matrix$v1 and see what happens
```

---

```{r}
a.data.frame = data.frame(a.matrix,logicals=c(TRUE,FALSE))
a.data.frame
a.data.frame$v1
a.data.frame[,"v1"]
a.data.frame[1,]
colMeans(a.data.frame)
```

Adding rows and columns
===

We can add rows or columns to an array or data frame with 
`rbind()` and `cbind()`, but be careful about forced type conversions

```{r}
rbind(a.data.frame,list(v1=-3,v2=-5,logicals=TRUE))
rbind(a.data.frame,c(3,4,6))
```

Structures of structures
===

So far, every list element has been a single data value

List elements can be other data structures, e.g., vectors and matrices:
```{r}
plan = list(factory=factory, available=available, output=output)
plan$output
```

Internally, a data frame is basically a list of vectors
(all of the same length)

---

List elements can even be other lists  
which may contain other data structures  
including other lists  
which may contain other data structures ...

This **recursion** lets us build arbitrarily complicated 
data structures from the basic ones

Most complicated objects are (usually) lists of data structures

Example: eigen-decomposition
===

`eigen()` finds eigenvalues and eigenvectors of a matrix

Returns a list of a vector (the eigenvalues) and a matrix 
(the eigenvectors)

```{r}
eigen(factory)
class(eigen(factory))
```

---
With complicated objects, you can access parts of parts 
(of parts ...)

```{r}
factory %*% eigen(factory)$vectors[,2]
eigen(factory)$values[2] * eigen(factory)$vectors[,2]
eigen(factory)$values[2]
eigen(factory)[[1]][[2]] # NOT [[1,2]]
```

Summary
===

- Arrays add multi-dimensional structure to vectors
- Matrices act like you'd hope they would
- Lists let us combine different types of data
- Data frames are hybrids of matrices and lists, allowing each column to have a different basic type
- Recursion lets us build complicated data structures out of simpler ones