---
title: "Reading in Text"
author: "Statistical Computing, 36-350"
date: "Wednesday September 7, 2016"
---

Text from the outside
===

How to get text, from an external source, into R? Use the `readLines()` function

```{r}
trump.lines = readLines("http://www.stat.cmu.edu/~ryantibs/statcomp-F16/data/trump.txt")
class(trump.lines) # We have a character vector
length(trump.lines) # Many lines (elements)!
trump.lines[1:3] # First 3 lines
```

Another example
===

That seemed to work really well! Let's try again

```{r}
wiki.lines = readLines("https://en.wikipedia.org/wiki/Donald_Trump")
length(wiki.lines) # Many, many lines
wiki.lines[1:6] # First 6 lines
```

Scraping web text is not easy!
===

The difference between these two examples: the first was a text file put there by your professor, the second was a bonafide website (in HTML)

To get "usual" text from the website, we have to go to line 216, and even then it's not that readable

```{r}
wiki.lines[216]
```

We'll learn more next week; for now, we'll just look at a nice text files

Reading text from a local file
===

We don't need to use the web; `readLines()` can be used on a local file. The following code would read in a text file from your professor's computer:

```{r, error=TRUE}
trump.lines.2 = readLines("~/Dropbox/teaching/f16-350/lectures/text/trump.txt")
```

This will cause an error for you, unless your folder is set up exactly like the professor's laptop! So using web links is more robust

Reconstitution
===

Fancy word, but all it means: make one long string, then split the words

```{r}
trump.text = paste(trump.lines, collapse=" ")
trump.list = strsplit(trump.text, split=" ")
```

The big long string
===

Let's investigate, starting with the big long string

```{r}
class(trump.text)
length(trump.text)
substr(trump.text, 1, 300)
nchar(trump.text)
```

The split up words
===

Let's investigate the result of splitting up the long string

```{r}
class(trump.list) # Remember why it's a list?
length(trump.list) 
trump.words = trump.list[[1]]
length(trump.words)
trump.words[1:10] 
```

Sorting the words
===

We can sort strings (just like numbers), with `sort()`

```{r}
trump.words.sorted = sort(trump.words) # Default is increasing order
length(trump.words.sorted)
head(trump.words.sorted) # Peak at the start
tail(trump.words.sorted) # Peak at the end
```

Notice that punctuation marks are treated as words (not ideal)

Getting the unique words
===

We can get just the unique (sorted) words, using `unique()`

```{r}
trump.words.sorted.unique = unique(trump.words.sorted)
length(trump.words.sorted.unique) 
length(trump.words.sorted.unique) / length(trump.words.sorted) # About 36% unique words
head(trump.words.sorted.unique)
tail(trump.words.sorted.unique)
```