---
title: 'Lecture 4: Text Basics'
author: "Statistical Computing, 36-350"
date: "Monday September 21, 2015"
---

Outline
===

- Introduction to strings and string operations
- Extracting and manipulating string objects
- Introduction to general search

Why characters?
===

A lot of data out there is in character form!

- Web pages can be scraped
- Email messages can be analyzed 
- Survey responses can be processed

Even if you only care about numbers eventually, you may have to know
how to extract them from text 

The simplest distinction
===

**Character**: a symbol in a written language, specifically 
what you can enter at a keyboard: letters, numerals, punctuation, 
space, newlines, etc.  
```
'L', 'i', 'n', 'c', 'o', 'l', 'n'
```

**String**: a sequence of characters bound together  
```
"Lincoln"
```

<small>(Note: R does not have a separate type for characters and 
strings)</small> 

```{r}
class("L")
class("Lincoln")
```

Making strings
===

Use single or double quotes to construct a string;
whatever you use as the starting choice has to match the end.
Why do we prefer double quotes?

```{r}
"Lincoln"
"Abraham Lincoln"
"Abraham Lincoln's Hat"
"As Lincoln never said, \"Four score and seven beers ago\""
```

---

Use `nchar()` to get the length of a  string:

```{r}
nchar("Lincoln")
nchar("Abraham Lincoln's Hat")
nchar("As Lincoln never said, \"Four score and seven beers ago\"")
```

Whitespace
===

The space `" "` is a character; so is the empty string `""`

Some characters are special, so we have "escape characters" to specify 
them in strings

- Quotes within strings: `\"`
- Tab: `\t`
- New line: `\n` 

The character data type
===

One of the atomic data types, like `numeric` or `logical`

Can go into scalars, vectors, arrays, lists, or be the type 
of a column in a data frame

```{r}
length("Abraham Lincoln's beard")
length(c("Abraham", "Lincoln's", "beard"))
nchar("Abraham Lincoln's beard")
nchar(c("Abraham", "Lincoln's", "beard"))
```

Character-valued variables
===

They work just like others, e.g., with vectors:

```{r}
president = "Lincoln"
nchar(president) 
presidents = c("Fillmore","Pierce","Buchanan","Davis","Johnson")
presidents[3]
presidents[-(1:3)]
```

Substring operations 
===

**Substring**: a smaller string from a bigger string, but still
a string in its own right

A string itself is not a vector or a list, so we cannot use subscripts 
like `[[ ]]` or `[ ]` to extract substrings; we must use `substr()` instead

```{r}
phrase = "Christmas Bonus"
substr(phrase, start=8, stop=12)
```

We can also use `substr()` to replace elements:

```{r}
substr(phrase, 13, 13) = "g"
phrase
```

Substrings with vectors
===

`substr()` vectorizes over all its arguments:

```{r}
presidents
substr(presidents,1,2) # First two characters
substr(presidents,nchar(presidents)-1,nchar(presidents)) # Last two
substr(presidents,20,21) # No such substrings so return the empty string
substr(presidents,7,7) # Can you explain?
```

Dividing strings into vectors
===

`strsplit()` divides a string according to key characters,
by splitting each element of the character vector `x` at appearances 
of the pattern `split`, its second argument

```{r}
scarborough.fair = "parsley, sage, rosemary, thyme"
strsplit(scarborough.fair, ",")
strsplit(scarborough.fair, ", ")
```

Pattern is recycled over elements of the input vector:

```{r}
strsplit(c(scarborough.fair, "Garfunkel, Oates", "Clement, McKenzie"), ", ") 
```

Note that it outputs a `list` of character vectors---why is this a 
good format here?

Casting data types into strings
===

Converting one variable type to another is called **casting**: 

```{r}
as.character(7.2)            # Obvious
as.character(7.2e12)         # Obvious
as.character(c(7.2,7.2e12))  # Obvious
as.character(7.2e5)          # Not quite so obvious
```

Building strings from multiple parts
===

The `paste()` function is very flexible! With one vector argument, 
works like `as.character()`:

```{r}
paste(41:45)
```

With 2 or more vector arguments, combines them with recycling:

```{r}
paste(presidents,41:45)
paste(presidents,c("R","D"))# Not historically accurate!
paste(presidents,"(",c("R","D"),41:45,")")
```

---

We can change the separator between pasted-together items
by setting the `sep` argument:

```{r}
paste(presidents, "(", 41:45, ")", sep="_")
paste(presidents, " (", 41:45, ")", sep="")
```

<small>(Exercise: what happens if you give `sep` a vector?</small>) 

<!-- 
A more complicated example of `paste()` recycling
===

```{r}
paste(c("HW","Lab"),rep(1:15,each=2))
```

<small>(Exercise: Convince yourself of why this works as it does)</small>
-->

Condensing multiple strings
===

We can produce one big string by setting the `collapse` argument:

```{r}
paste(presidents, " (", 41:45, ")", sep="", collapse="; ")
```

<small>(Note: the default value of `collapse` is `NULL`, i.e., it won't 
collapse anything)</small>

A function for writing regression formulas
===

R has a standard syntax for models: outcome and predictors

```{r}
my.formula = function(dep,indeps,df) {
  rhs = paste(colnames(df)[indeps], collapse=" + ")
  return(paste(colnames(df)[dep], "~", rhs, collapse=""))
}
my.formula(2,c(3,5,7),df=state.x77)
```


Text of some importance
===
  ...
  *If we shall suppose that American slavery is one of those
  offenses which, in the providence of God, must needs come, but which, having
  continued through His appointed time, He now wills to remove, and that He
  gives to both North and South this terrible war as the woe due to those by
  whom the offense came, shall we discern therein any departure from those
  divine attributes which the believers in a living God always ascribe to Him?
  Fondly do we hope, fervently do we pray, that this mighty scourge of war may
  speedily pass away.  Yet, if God wills that it continue until all the wealth
  piled by the bondsman's two hundred and fifty years of unrequited toil shall
  be sunk, and until every drop of blood drawn with the lash shall be paid by
  another drawn with the sword, as was said three thousand years ago, so still
  it must be said "the judgments of the Lord are true and righteous
  altogether."*
  ...

Reading text from a file
===

```{r}
linc = readLines("http://www.stat.cmu.edu/~ryantibs/statcomp-F15/lectures/lincoln.txt") 
length(linc)
head(linc)
```

`linc` is a vector of strings, one element per line of text

A hint of next time: search
===

Narrowing down entries: use `grep()` to find which strings have a 
matching search term

```{r}
grep("God", linc)
grepl("God", linc)
linc[grep("God", linc)]
```


Reconstituting
===

Make one long string, then split the words

```{r}
linc = paste(linc, collapse=" ")
linc.words = strsplit(linc, split=" ")[[1]]
length(linc.words)
head(linc.words)
```

Counting words 
===

Tabulate how often each word appears, put in order:

```{r}
wc = table(linc.words)
wc = sort(wc,decreasing=TRUE)
head(wc,20)
```

`names(wc)` gives all the distinct words in `linc.words` 
(types); `wc` contains how often they appear (tokens)

A few undesirable results
===

The empty string is the third-most-common word:
```{r}
names(wc)[3]
```

Commas matter:

```{r}
wc["years"]
wc["years,"]
```

---

Capitalization matters:
```{r}
wc["that"]
wc["That"]
```

All of this can be fixed if we learn how to work with text patterns 
and not just pure strings

Summary
=======

- Text is data, just like everything else
- `substr()` extracts and substitutes
- `strsplit()` splits a big string into smaller ones
- `paste()` bunches together smaller strings into a big one
- `table()` for counting how many tokens belong to each type

Next time: searching for text patterns using regular expressions


