---
title: "Bag-of-Words"
author: "Statistical Computing, 36-350"
date: "Friday September 9, 2016"
---

Two documents of interest
===

Let's get two documents of interest into R, before we can start comparing them, and caculate word counts for each. (Note: we convert all words to lower case)

```{r}
# First one
trump.lines = readLines("http://www.stat.cmu.edu/~ryantibs/statcomp-F16/data/trump.txt")
trump.words = strsplit(paste(trump.lines, collapse=" "), split=" ")[[1]]
trump.wordtab = table(tolower(trump.words))

# Second one
clinton.lines = readLines("http://www.stat.cmu.edu/~ryantibs/statcomp-F16/data/clinton.txt")
clinton.words = strsplit(paste(clinton.lines, collapse=" "), split=" ")[[1]]
clinton.wordtab = table(tolower(clinton.words))
```

Simple pre-processing
===

Turns out that both tables actually has an entry for the empty string, ""; let's get rid of these entries

```{r}
trump.wordtab[names(trump.wordtab) == ""] # We have 1 empty word with Trump
clinton.wordtab[names(clinton.wordtab) == ""] # We have 312 empty words with Clinton
trump.wordtab = trump.wordtab[names(trump.wordtab) != ""] 
clinton.wordtab = clinton.wordtab[names(clinton.wordtab) != ""] # Let's get rid of them
```

Basic comparisons
===

Let's make some basic comparisons. (Note: we know these are imperfect, because our words may contain punctuation marks, etc. Ignore for now, but next week we'll see a nice way to clean this up)

```{r}
# Who had the longer speech (spoke more words)?
sum(trump.wordtab)
sum(clinton.wordtab)

# Who used more unique words?
length(trump.wordtab)
length(clinton.wordtab)

# Who repeated themselves less (higher percentage of unique words)?
length(trump.wordtab) / sum(trump.wordtab) * 100
length(clinton.wordtab) / sum(clinton.wordtab) * 100

# Who used "great" more? 
trump.wordtab["great"]
clinton.wordtab["great"]
```

How to go beyond the basics?
===

Suppose we want to make more advanced comparisons. E.g., given a new (third) document, is it more like the first or the second?

We must think of a way of representing our two documents. We want our representation to: 

1. Be easy to generate from the raw documents, and be easy to work with
2. Highlight important aspects of the documents, and suppress unimportant aspects

Think we need some fancy representation? Think again! We pretty much already have what we need

Bag-of-words model
===

The **bag-of-words** model is a central to information retrieval and natural language processing. The idea: a document is nothing more than a bag of words, i.e., order doesn't matter, only the word counts

This is like what we're already doing! Key difference is that we must calculate word counts for words that appear across **all documents**

```{r}
# Let's set up our third document, the query document
query.text = "Make America great again."
query.words = strsplit(query.text, split=" ")[[1]]
query.wordtab = table(tolower(query.words))

# Let's get all the words, then just consider the unique ones, sorted alphabetically
all.words = c(names(trump.wordtab), names(clinton.wordtab), names(query.wordtab))
all.words.unique = sort(unique(all.words))
length(all.words.unique)
```

Document-term matrix
===

This is a matrix that is $(\text{# of documents}) \times (\text{# of unique words})$, or for us, 3 x 2686. Each row contains the word counts for one document

```{r}
dt.mat = matrix(0, 3, length(all.words.unique))
rownames(dt.mat) = c("Trump", "Clinton", "Query")
colnames(dt.mat) = all.words.unique
dt.mat[1, names(trump.wordtab)] = trump.wordtab
dt.mat[2, names(clinton.wordtab)] = clinton.wordtab
dt.mat[3, names(query.wordtab)] = query.wordtab
dt.mat[, 1:10]
```

Computing word count differences
===

Which words had the biggest (absolute) differences in counts between our two documents?

```{r}
dt.diff = dt.mat[1,] - dt.mat[2,]
dt.diff[sample(length(dt.diff), 5)]
dt.diff.abs.sorted = sort(abs(dt.diff), decreasing=TRUE) 
dt.diff.abs.sorted[1:20] # Top 20 biggest absolute differences
```

Now we don't know the direction though, e.g., who said "believe" more?

Reminder: `order()`
===

The `sort()` function gives us the values in sorted order; its close friend, the `order()` function gives us the indices that correspond to sorted order

```{r}
x = c(5, 2, 7)
sort(x, decreasing=TRUE)
order(x, decreasing=TRUE)
x[order(x, decreasing=TRUE)]
```

Now back to our running data set

```{r}
inds.sorted = order(abs(dt.diff), decreasing=TRUE)
dt.diff[inds.sorted[1:20]] # Top 20 biggest differences, with signs
```

Computing correlations
===

Which of our two documents is "closest" to the query? Depends on how we define "closest"; but one reasonable way is to use **correlation**, easily computed with the `cor()` function

```{r}
x = 1:10; y = 1:10; z = 5*(1:10)
cor(x, y)
cor(x, z)
(y = 1:10 + rnorm(10))
cor(x, y)
```

Now back to our running data set

```{r}
cor(dt.mat[1,], dt.mat[3,])
cor(dt.mat[2,], dt.mat[3,])
```

IDF weighting
===

How well does our bag-of-words model meet our two goals: 1. easy to set up and use, 2. helpful at teasing out important aspects, and muting unimportant ones?

Not so strong on the second goal (recall the word count differences). But an extension called **inverse-document-frequency** or IDF weighting can help a lot with this

Helps when we have many documents in our collection (also called our corpus). Suppose we have $D$ documents total. IDF weighting works as follows: e.g., the word count
$$
(\text{# of times Trump said great})
$$
becomes the weighted word count
$$
(\text{# of times Trump said great}) \times 
\log\bigg(\frac{D}{\text{# of documents with word great}}\bigg)
$$
(Note: you'll see this as TF-IDF, where the TF stands for **term-frequency**)

So when all documents contain the word in question, the weight is zero. Better to have a large corpus, say $D$ in the hundreds or thousands, not $D=2$ like us