---
title: "Splitting and Searching with Regexes"
author: "Statistical Computing, 36-350"
date: "Wednesday September 16, 2016"
---

Splitting on a regex
===

Regexes, we've seen, are powerful tools for matching patterns of text. Turns out, we can also split strings based on them: use `strsplit()` with `split` equal to a regex

```{r}
strsplit("Just .. gotta  .... keep ......  going.......", 
  split=" *\\.+ *")[[1]]
strsplit("A semicolon; is used to; join two; or more ideas; in a sentence",
         split=";? +")
strsplit("Mercedes S550 iPhone 5s Titleist 915D Stat Computing v2", 
         split=" *[[:alpha:]]*[0-9]+[[:alpha:]]* *")
```

Example: Trump's RNC speech
===

```{r}
trump.lines = readLines("http://www.stat.cmu.edu/~ryantibs/statcomp-F16/data/trump.txt")
trump.text = paste(trump.lines, collapse=" ")
trump.words.naive = strsplit(trump.text, split=" ")[[1]]
head(trump.words.naive, 10)
trump.words = strsplit(trump.text, split="[[:space:]]|[[:punct:]]")[[1]]
head(trump.words, 10)
```

The second looks better! And a sanity check:

```{r}
grep("[[:punct:]]", trump.words, value=TRUE)
```

So there's really no punctuation marks in the second set of words

(Continued)
===

Let's get rid of the empty string and explore the word counts a bit

```{r}
sum(trump.words == "")
trump.words = trump.words[trump.words != ""] # Get rid of empty strings
trump.wordtab = table(trump.words)
head(sort(trump.wordtab, decreasing=TRUE), 5)
plot(trump.wordtab)
```

Searching with a regex
===

Scraping data from the web is not easy. Often times there is a lot of HTML code and/or other "junk" mixed in with text or data that we care about

Unfortunately there is no general recipe for extracting data from a generic webpage, this is what makes web scraping hard; but regexes can help a lot (recall earthquake data from Wednesday's lab)

(Continued)
===

General steps:

- Read the webpage into R with `readLines()`, print out some of the results
- Find where the data starts, by (say) using `grep()` and a regex literal
- Find out where the data ends, in the same way
- If the data lines are contiguous, then just pull these lines
- If not, then get lines using `grep()` and a regex pattern
- One you have the data lines, study the pattern of data you want to extract
- Use `regexpr()` and `regmatches()` with a regex pattern to get the data 
- Do some sanity checks to make sure that we have extracted the "right" data

Example: fastest 100m sprint times
===

Getting data lines is pretty easy, because the data is contiguous

```{r}
sprint.lines = readLines("http://www.alltime-athletics.com/m_100ok.htm")
length(sprint.lines)
head(sprint.lines, 5)
(i.first = min(grep("Usain Bolt", sprint.lines)))
(i.last = max(grep("Julian Forte", sprint.lines)))
i.last - i.first + 1 # Sanity check
sprint.lines[i.first:(i.first+4)]
sprint.lines = sprint.lines[i.first:i.last] # Throw away everything else
```

(Continued)
===

Now let's extract the times themselves

```{r}
time.pattern = "[[:space:]]+(9|(10))\\.[0-9]{2}[[:space:]]"
test.vec = c("  9.58 ", " 10.05  ", "13.81")
as.numeric(grep(time.pattern, test.vec, value=TRUE))
sprint.times = as.numeric(regmatches(sprint.lines, regexpr(time.pattern, sprint.lines)))
sprint.times[1:10]
length(sprint.times) # Sanity check ... uh oh
head(grep("Obadele Thompson", sprint.lines, value=TRUE), 1)
```

(Continued)
===

Second attempt at extracting times, catch trailing "A" (stands for wind assisted)

```{r}
time.pattern = "[[:space:]]+(9|(10))\\.[0-9]{2}([[:space:]]|A)"
sprint.times = regmatches(sprint.lines, regexpr(time.pattern, sprint.lines))
length(sprint.times) # Sanity check 
sprint.times = as.numeric(regmatches(sprint.times, regexpr("(9|(10))\\.[0-9]{2}", sprint.times)))
length(sprint.times) # One more sanity check
sprint.times[1:10]
plot(sprint.times, type="l", xlab="Rank", ylab="Time") 
```

(Continued)
===

Some motivation for learning cool plot tools, which we'll cover next week!

```{r}
sprint.times.unique = unique(sprint.times)
sprint.times.table = table(sprint.times)
n.unique = length(sprint.times.unique)
plot(sprint.times, type="n", xlab="Rank", ylab="Time", yaxt="n",
     main=paste("The", length(sprint.times), "fastest 100m dash times"))
axis(side=2, at=sprint.times.unique, labels=sprint.times.unique)
(mar = par("usr"))
xleft = mar[1]; xright = mar[2]
rect(xleft, sprint.times.unique[-n.unique], xright, sprint.times.unique[-1],
     col=c("pink","orange","lightblue"), border=NA, density=20)
abline(h=sprint.times.unique)
lines(sprint.times, lwd=2)
text(2700+50*rnorm(n.unique-1), 
     (sprint.times.unique[-n.unique]+sprint.times.unique[-1])/2,
     labels=sprint.times.table[-n.unique], cex=0.75)
```