---
title: 'Recitation: Midterm Review'
output: html_document
---

We've already covered a lot of material this semester.  Let's consider what we'll do with R in a typical application:

* Read in some data -- maybe something nicely formatted, maybe something we have to parse.

* Identify interesting subsets of the data (sometimes)

* Fit a model to the data

* And/Or do something over and over again

* Make our results presentable ot others (usually with a plot)

You're asked to do all of these things on the midterm. We'll take a short look at each of them now.

***

Reading in Data: The Easy Version
---

Sometimes we have nicely formatted data, in which case we can use `read.table()` or `read.csv()`. These give us a data frame.

```{r}
pros = read.table('http://statweb.stanford.edu/~tibs/ElemStatLearn/datasets/prostate.data')
class(pros)
```

Let's compare the top of the file and the resulting data frame:

From the file:
```
	lcavol	lweight	age	lbph	svi	lcp	gleason	pgg45	lpsa	train
1	-0.579818495	2.769459	50	-1.38629436	0	-1.38629436	6	  0	-0.4307829	T
2	-0.994252273	3.319626	58	-1.38629436	0	-1.38629436	6	  0	-0.1625189	T
3	-0.510825624	2.691243	74	-1.38629436	0	-1.38629436	7	 20	-0.1625189	T
4	-1.203972804	3.282789	58	-1.38629436	0	-1.38629436	6	  0	-0.1625189	T
5	 0.751416089	3.432373	62	-1.38629436	0	-1.38629436	6	  0	 0.3715636	T
```

Our data frame:
```{r}
head(pros,5)
```

***

Reading in Data: Parsing with Regular Expressions
---

We looked at a trickier example in Homework 4. One entry from the file looked like this:

```
<tr><td colspan="1" rowspan="1" class="date"><div style="display:block;" class="skedStartDateSite">Wed Oct 7, 2015</div><div style="display:none;" class="skedStartDateLocal">Wed Oct 7, 2015</div></td><td colspan="1" rowspan="1" class="team"><!-- Away --><a onclick="loadTeamSpotlight(jQuery(this));" rel="NYR" shape="rect" href="javascript:void(0);"><img title="New York Rangers" alt="New York Rangers" class="team-logo  " src="http://cdn.nhle.com/nhl/images/logos/teams/rangers_logo.svgz?v=8.47"/></a><div class="teamName"><a onclick="loadTeamSpotlight(jQuery(this));" rel="NYR" shape="rect" href="javascript:void(0);">NY Rangers</a></div></td><td colspan="1" rowspan="1" class="team"><!-- Home --><a onclick="loadTeamSpotlight(jQuery(this));" rel="CHI" shape="rect" href="javascript:void(0);"><img title="Chicago Blackhawks" alt="Chicago Blackhawks" class="team-logo  " src="http://cdn.nhle.com/nhl/images/logos/teams/blackhawks_logo.svgz?v=8.47"/></a><div class="teamName"><a onclick="loadTeamSpotlight(jQuery(this));" rel="CHI" shape="rect" href="javascript:void(0);">Chicago</a></div></td><td colspan="1" rowspan="1" class="time"><!-- Time --><div style="display:block;" class="skedStartTimeEST">8:00 PM ET</div><div style="display:none;" class="skedStartTimeLocal">7:00 PM CT</div></td><td colspan="1" rowspan="1" class="tvInfo"><!-- Network -->
NBCSN</td><td colspan="1" rowspan="1" class="skedLinks"><!-- Button Links --><a class="btn" shape="rect" href="http://www.ticketmaster.com/Chicago-Blackhawks-tickets/artist/805913?intcmp=tm204060&amp;wt.mc_id=NHL_LEAGUE_TIX_LIST_PR_CHI"><span>TICKETS›</span></a><a class="btn" shape="rect" href="http://www.ticketsnow.com/NHL/chicago-blackhawks-tickets/?intcmp=tm204030&amp;wt.mc_id=NHL_LEAGUE_TIX_LIST_EX_CHI"><span>BUY/SELL›</span></a></td></tr>
```

We read it in as a raw text file:

```{r}
nhl1516 = readLines('http://www.stat.cmu.edu/~ryantibs/statcomp-F15/homework/NHL1516.html')
length(nhl1516)
```

As a brief example, let's remember how we extracted the date. The relevant part of each entry looked like this:

```
<div style="display:block;" class="skedStartDateSite">Wed Oct 7, 2015</div><div style="display:none;" class="skedStartDateLocal">Wed Oct 7, 2015</div>
```

We used regular expressions to isolate the parts we were interested in.  What does `.*` do? Why do we match all the way out to `skedStartDateLocal`?

```{r}
datematch = regexpr("DateSite\">.*</div>.*skedStartDateLocal",nhl1516)
dates = regmatches(x=nhl1516,m=datematch)
head(dates)
```

We can then pull out just the part of the string we want:

```{r}
date = substr(dates,start=11,stop=25)
head(date)
```

***

Interesting Subsets of the Data
---

It's straightforward to generate vectors of `TRUE` and `FALSE` in R. We can then use those vectors to extract interesting portions of the data.

For example, going back to the prostate data set, suppose we wanted to compare young vs. old patients:

```{r}
young_patients = (pros$age <= 55)
old_patients = (pros$age > 55)
head(young_patients)
```

We could then compare the younger and older patients:

```{r}
mean( pros$lweight[young_patients] )
mean( pros$lweight[old_patients] )
```

We can also look directly at the data frame, restricted to those values:

```{r}
head(pros[young_patients,])
```

Why do we need the comma?

***

Fit a Model
---

Let's revisit an example from the lecture. Using the prostate cancer data, we had a proposed model: $$\mathrm{lpsa} = a \mathrm{lcavol} + b \mathrm{lweight}$$

We used `lm` to fit this model to the data:

```{r}
pros.lm.1 = lm(lpsa ~ lcavol + lweight, data=pros)
summary(pros.lm.1)
```

We specified the model with `lpsa ~ lcavol + lweight`.

We could extract the coefficients:

```{r}
coef(pros.lm.1)
```

We could also extract the fitted values for each data point:

```{r}
head(fitted(pros.lm.1))
```

***

Doing Things Over and Over Again
---

This is one of the reasons we use computers -- they're good at doing the same thing over and over again without making mistakes or getting bored. In R, we have three main tools:

* `for` and `while` loops

* Functions from the `apply` family

* Defining our own functions

***

Loops
---

`for` loops execute a set of commands a fixed number of times, `while` loops execute a set of commands until a condition is satisfied. Let's look at a simple for loop:

```{r}
for(i in 1:ncol(pros)){
  print( colnames(pros)[i] )
  print( mean(pros[,i]) )
}
```

In this case, we could have accomplished a similar result using `lapply`:

```{r}
lapply(pros,mean)
```

***

Functions
---

Functions are a tool to break big problems into a series of small problems. As an added bonus, you may find that the solutions to the small problems are something you can use over and over again.

Basic syntax:

```{r}
first_col_summary = function(df){
  this.name = colnames(df)[1]
  this.mean = mean(df[,1])
  this.sd   = sd(df[,1])
  return( list(name=this.name,mean = this.mean,sd=this.sd) )
}
first_col_summary(pros)
```

***

A pesky detail
---

If you create a variable within the function, it won't "survive" outside the function.

```{r}
numberchanger = function(){
  the_best_number = 4
  return(the_best_number)
}
numberchanger()
#the_best_number
```

***

A pesky detail (cont.)
---

If a variable outside the function and inside the function have the same name, the function will ignore the "outside" copy of the variable.  It won't modify it, and will work with its own copy.

```{r}
the_best_number = 2
numberchanger = function(){
  the_best_number = 4
  return(the_best_number)
}
numberchanger()
the_best_number
```

***

A pesky detail (cont.)
---

If a variable isn't defined within the function, but is defined outside, the function will refer to the outside value:

```{r}
the_best_number = 2
numberchanger = function(){
  return(the_best_number)
}
numberchanger()
the_best_number
```

But remember, if I try to modify `the_best_number` from inside my function, R will instead create a local copy of `the_best_number`. Generally speaking, getting variable values this way instead of as arguments is a poor programming practice.

***

Presenting Your Results
---

Usually, we ploy `x,y`:

```{r}
plot(pros$lpsa,pros$lcavol)
```

When plotting columns from the same data frame, we can instead plot `y~x`:

```{r}
plot(lpsa~lcavol,data=pros)
```

***

Nicer Plots
---

Label everything:

```{r}
plot(lpsa~lcavol,data=pros,xlab='log cancer volume',ylab='log PSA score',main='Nice Plot!')
```

Use color to make your plots more informative:

```{r}
plot(lpsa~lcavol,data=pros[young_patients,],xlab='log cancer volume',ylab='log PSA score',main='Nice Plot!',col='red')
points(pros$lcavol[old_patients],pros$lpsa[old_patients],col='blue')
```

And don't forget how to plot things side-by-side by setting `mfrow`:

```{r}
par(mfrow=c(1,2))
plot(lpsa~lcavol,data=pros,xlab='log cancer volume',ylab='log PSA score',main='Super')
plot(lpsa~lweight,data=pros,xlab='log cancer weight',ylab='log PSA score',main='Sweet!')
```