---
title: "Summarizing Text"
author: "Statistical Computing, 36-350"
date: "Wednesday September 7, 2016"
---

Counting words
===

Our most basic tool for summarizing text: **word counts**, retrieved using `table()`

```{r}
trump.lines = readLines("http://www.stat.cmu.edu/~ryantibs/statcomp-F16/data/trump.txt")
trump.text = paste(trump.lines, collapse=" ")
trump.words = strsplit(trump.text, split=" ")[[1]]
trump.wordtab = table(trump.words)
```

The output of `table()` 
===

What did we get? Alphabetically sorted unique words, and their counts = number of appearances

```{r}
class(trump.wordtab)
length(trump.wordtab)
trump.wordtab[1:10]
plot(trump.wordtab, xlab="Words", ylab="Frequency") # Table objects can be specially plotted
```

The names are words, the entries are counts
===

Note: this is actually a vector of numbers, and the words are the names of the vector

```{r}
trump.wordtab[1:5]
trump.wordtab[2] == 34
names(trump.wordtab)[2] == "–"
```

Fetching specific word counts
===

Reminder: you can index a vector by names (not just by positions)

```{r}
trump.wordtab[100]
trump.wordtab["already"] # Same thing
```

We can now use this to look up whatever words we want

```{r}
trump.wordtab["America"] 
trump.wordtab["great"]
trump.wordtab["wall"]
trump.wordtab["Canada"] # NA means Trump never mentioned Canada
```

Most frequent words
===

Let's sort in decreasing order, to get the most frequent words

```{r}
trump.wordtab.sorted = sort(trump.wordtab, decreasing=TRUE)
length(trump.wordtab.sorted)
head(trump.wordtab.sorted, 20) # First 20
tail(trump.wordtab.sorted, 20) # Last 20
```

Notice that punctuation matters, e.g., "Yet" and "Yet," are treated as separate words (not ideal)

Visualizing frequencies
===

Let's use a histogram to visualize frequencies

```{r}
nw = length(trump.wordtab.sorted)
plot(1:nw, trump.wordtab.sorted, type="l", xlab="Rank", ylab="Frequency")
```

A pretty drastic looking trend! It looks as if $\mathrm{Frequency} \propto (1/\mathrm{Rank})^a$ for some $a>0$

Zipf's law
===

This phenomenon, that frequency tends to be inversely proportional to a power of rank, is called **Zipf's law** 

For our data, Zipf's law approximately holds, with $\mathrm{Frequency} \approx C(1/\mathrm{Rank})^a$ for $C=215$ and $a=0.57$

```{r}
C = 215; a = 0.57
trump.wordtab.zipf = C*(1/1:nw)^a
cbind(trump.wordtab.sorted[1:8], trump.wordtab.zipf[1:8])
```

Not perfect, but not bad

Visualizing Zipf's law
===

We can plot the original sorted word counts, and those estimated by our formula law on top

```{r}
plot(1:nw, trump.wordtab.sorted, type="l", xlab="Rank", ylab="Frequency")
curve(C*(1/x)^a, from=1, to=nw, col="red", add=TRUE)
```

We'll learn about plotting tools in detail in a couple of weeks