---
title: 'Recitation 2: Strings and RegExp'
author: "Statistical Computing, 36-350"
date: "Monday September 28, 2015"
output: pdf_document
---

Strings: A Brief Review
---

Intituively, a string is a seris of letters, numbers, and other symbols. R stores these using the R character type.

```{r}
my.strings = character(3)
my.strings[1] = "Strings"
my.strings[2] = "Are"
my.strings[3] = "Fantastic!"
my.strings
class(my.strings)
typeof(my.strings)
```

***

More About Strings
---

R treats strings and vectors differently, which means we have new functions for manipulating strings. 

`nchar` instead of `nrow` or `length`

```{r}
nchar(my.strings)
nrow(my.strings)
length(my.strings)
length(my.strings[[1]])
```

***

Substring
---

We used **slices** of vectors, matrices, etc. For strings, we use **substrings**.

```{r}
substr(my.strings,start=1,stop=4)
substring(my.strings,first=5)
```

`substr` and `substring` have some subtle differences. 

***

Regular Expressions: A Brief Review
---

We'll often want to extract interesting parts of a string. Regular expressions provide a way to describe those interesting parts. You've had a comprehensive introduction to these in lecture, I'll just highlight a few important points.

We can use plain strings:
```{r}
look.for = "ing"
grepl(look.for,my.strings)
```

We can specify ranges of characters, e.g. `[A-Z]1,`[1-9]`, `[A-z]`. Can also use POSIX expressions like `[[:space:]]` and `[[:punct:]]`. Can also look for anything besides a newline, using `.`.

We can specify how many times we want to see something, e.g. 
* '[A-Z]*' (Any capital letters zero or more times)
* '[1-9]+' (Any numbers one or more times)

***

Excluding Things
---

We can *exclude* things. This is important because regex matching is greedy.

```{r}
my.string = "<HTML>words<TAG>"
strsplit(my.string,"<.*>")
strsplit(my.string,"<[^>]*>")
```

***

Our text for the day: The second Republican primary debate (courtesy CNN):
---

```{r}
linc = readLines("http://www.stat.cmu.edu/~ryantibs/statcomp-F15/recitations/RepublicanDebate.clean.txt") 
length(linc)
hist(nchar(linc))
```

Each line is a paragraph. We can have some fun parsing the text!

***

The transcript has a separate line for APPLAUSE. How many times did the audience applaud?

```{r}
grep('APPLAUSE',linc)
length(grep('APPLAUSE',linc))
```

Here's another way to get a number:

```{r}
head(grepl('APPLAUSE',linc),10)
sum(grepl('APPLAUSE',linc))
```

Note that `grep` and `grepl` return the same information, but in different forms.

***

How many times did each candidate speak?
---

```{r}
candidates = c('BUSH','TRUMP','FIORINA','RUBIO','CARSON','CRUZ','WALKER','KASICH','CHRISTIE','PAUL','HUCKABEE')
spoke = rep(0,length(candidates))
names(spoke)=candidates
for (name in candidates){
  spoke[name] = length(grep(name,linc))
}
spoke
```

This code searches the full text for lines that contain the candidate names (in ALL CAPS). 

***

How many times did each candidate mention jobs?
---

```{r}
jobscount = rep(0,length(candidates))
names(jobscount) = candidates
for (name in candidates){
  name.index = grepl(name,linc)
  jobscount[name] = sum(grepl('[Jj]obs',linc[name.index]))
}
jobscount
```

Does this actually count the number of times each candidate used the word?

***

Another pass at counting jobs references:
---

```{r}
realjobscount = rep(0,length(candidates))
names(realjobscount) = candidates
for (name in candidates){
  name.index = grepl(name,linc)
  allwords = strsplit(linc[name.index],"([[:space:]]|[[:punct:]])+")
  allwords = unlist(allwords)
  word.table = table(allwords)
  jobsct = 0
  if (!is.na(word.table['jobs'])){
    jobsct = jobsct + word.table['jobs']
  }
  if (!is.na(word.table['Jobs'])){
    jobsct = jobsct + word.table['Jobs']
  }
  realjobscount[name] = jobsct
}
realjobscount
```

Note that here, `strsplit` returns a list of lists. `unlist` converts that into one flat list, which is more convenient for table. What would happen if we ran this code without the `if` statements?

***

Other things we could do:
---

* Frequently used words for each candidate
* Unusual words for each candidate
* Time series for words of interest
* ???

***

Another source: MLB Standings (via ESPN)
---

```{r}
linc = readLines("http://espn.go.com/mlb/standings") 
length(linc)
line = linc[249]
nchar(line)
```

In the source for this page all of the data are in one line.

***

Let's split up the line with all the data:
---

Here's an excerpt from that line:

```
<span class="teams sprite-mlb-teams-25 sprite-25-team-14"></span></a><a  name="&lpos=mlb:standings:team" href="/mlb/team/_/name/tor"><span><span class="team-names">Toronto Blue Jays</span><abbr title="Toronto Blue Jays">TOR</abbr></span></a></td><td style="white-space:no-wrap;" class="">90</td><td style="white-space:no-wrap;" class="">65</td><td style="white-space:no-wrap;" class="">.581</td><td style="white-space:no-wrap;" class="">-</td><td style="white-space:no-wrap;" class="center">53-28</td><td style="white-space:no-wrap;" class="center">37-37</td><td style="white-space:no-wrap;" class="">853</td><td style="white-space:no-wrap;" class="">631</td><td style="white-space:no-wrap;" class=" positive">+222</td><td style="white-space:no-wrap;" class="">W4</td><td style="white-space:no-wrap;" class="">7-3</td><td style="white-space:no-wrap;" class="">100.0%</td></tr><tr class=" standings-row"><td style="text-align:left;white-space:nowrap;" class="team "><a  name="&lpos=mlb:standings:team" href="/mlb/team/_/name/nyy"><span class="teams sprite-mlb-teams-25 sprite-25-team-10"></span></a><a  name="&lpos=mlb:standings:team" href="/mlb/team/_/name/nyy"><span><span class="team-names">New York Yankees</span><abbr title="New York Yankees">NYY</abbr></span></a></td><td style="white-space:no-wrap;" class="">86</td><td style="white-space:no-wrap;" class="">69</td><td style="white-space:no-wrap;" class="">.555</td>
```

What can we use to split?

***

Splitting the string
---

```{r}
teams = strsplit(line,"class=\"team-names\">")[[1]]
teams[[2]]
teams = teams[2:length(teams)]
num.teams = length(teams)
num.teams
```

Note that there are 30 MLB teams.  Phew!

***

The data are surrounded by HTML tags -- let's split on the tags.
---

Tags start with "`<`" and end with "`>`".

```{r}
teaminfo = strsplit(teams,"<[^>]*>")
teaminfo[[1]]
teaminfo[[2]]
```

***

Let's put some of the data into a data frame:
---

```{r}
teamnames  = character(num.teams)
wins   = rep(0, num.teams)
losses = rep(0, num.teams)
for (i in 1:30){
  teamnames[i]  = teaminfo[[i]][[1]]
  wins[i]   = as.numeric( teaminfo[[i]][[8]] )
  losses[i] = as.numeric( teaminfo[[i]][[10]] )
}
wl = data.frame(wins=wins,losses=losses)
rownames(wl) = teamnames
wl
```

***

Which teams have 90 or more wins?
---

```{r}
subset(wl,wins>=90)
```

Let's go Bucs!