---
title: Introduction to the Course
author: Cosma Shalizi
date: 28 August 2018, 36-467/667
output: slidy_presentation
bibliography: locusts.bib
---

## Data over Space and Time

- Statistical study for processes that unfold over time, or space, or both
- History, science, and lots of practical policy & technology
- Not everything:
    + Good experiments
	+ Good surveys
	+ Wishful thinking

## Special Statistical Issues

- The goal: learn how _here_ and _now_ depends on _there_ and _then_
- The two problems:
    + Everything depends on everything else $\Rightarrow$ basic theory doesn't apply
    + We don't get multiple samples of time or of space
    + $\Rightarrow$ $n=1$, always

## Course Mechanics

- Syllabus: http://www.stat.cmu.edu/~cshalizi/dst/18
    + All assignments will be posted there
	+ Information about readings and other course resources ditto
- Canvas as a gradebook and to distribute some electronic readings
- Piazza for question-answering

Office hours:

- Me, Wednesdays 1:00--3:00 in Baker Hall 229C
- TAs, TBD

## Textbooks

Gidon Eshel, _Spatiotemporal Data Analysis_

![](https://press.princeton.edu/sites/default/files/styles/large/public/covers/9780691128917_0.png?itok=B8g255-h)

**Required**, but full text PDF through JSTOR

Expect to read some of this nearly every week

## Textbooks

Peter Guttorp, _Stochastic Modeling of Scientific Data_

```{r, fig.retina=NULL, out.height=300, echo=FALSE}
knitr::include_graphics("https://images.tandf.co.uk/common/jackets/amazon/978041299/9780412992810.jpg")
```

- **Recommended**; required excerpts will be distributed through Canvas

## Textbooks

Paul Teetor, _The R Cookbook_

![](https://covers.oreillystatic.com/images/9780596809164/cat.gif)

**Recommended**; consult as needed


## Lectures

- Clarification, amplification, examples, alternatives
- _Complements_ to the readings
    + $\therefore$ Do the readings ahead of time
- No electronics, except for documented need
- Graded in-class exercises most lectures


## Assignments {.smaller}

1. In-class exercises in small groups (15%)
   + In class, on paper, 15% of your grade
   + Will get you started on the homework, or get you over hurdles
   + Expect one of these most class meetings
2. Weekly homework $\times 12$ (40%)
   + Data analysis & computing & a little theory
   + Always turned in electronically via Canvas
   + Usually due at 6 pm on Wednesdays most weeks
   + Lowest 2 dropped, no questions asked
   + If you turn in all 12, lowest _3_ dropped
   + NO LATE HOMEWORK FOR ANY REASON
3. Take-home mid-term (20%) and final exams (25%)
   + Midterm due 11 October; final due 14 December
   + One week (at least) to work on each
   + Electronic submission via Canvas

## R Markdown

- R Markdown is a straightforward way to combine text, math, code, and
  code output in one document
- Enforces _reproducibility_: your numbers, figures, etc., are generated
  by the code, so results always match
- Use is _required_ for this class
    + Your homework and exams will lose points if you don't use it
	+ You will lose more points as the semester goes on
	+ Resources for learning R Markdown on the syllabus page

## Collaboration, Cheating & Plagiarism

- Except for the in-class group assignments, you need to work on your own
- Full policy for this class is part of the syllabus
- Read that policy and the [CMU policy on academic integrity](http://www.cmu.edu/policies/student-and-student-life/academic-integrity.html)
- Turn in Homework 0 on these policies by Thursday
    + Not graded, but you won't get credit for any work until you complete it successfully

## Any questions?


## What are the big issues?

- We see $X$ at time $t$ and point $r$, $X(r,t)$, and what to know how it
relates to $Y$ at time $s$ and point $q$, $Y(q,s)$

Problems:

0. Basic statistical theory is about independent, identically distributed (IID) data.
1. But we (usually) only see _one_ realization of a whole process.
2. Every observation is dependent on every other observation.
3. Basic statistical theory then says that $n=1$ and can't draw any inferences.

## How are we going to deal with these issues?

- Methods for describing relationships and finding patterns in the data
- Especially methods for _predicting_ $X(r,t)$ from $Y(q,s)$
- _Incorporate_ dependence into statistical theory, so we can say when methods will work
- Quantifying uncertain is best done through modeling and simulation

## Why is this worth knowing?

- Italo Calvino, "All at One Point" (1963) imagines the world un-extended in time or space[^calvino]

[^calvino]: Calvino's _Cosmicomics_ is a precious part of our common cultural heritage.

- Otherwise, every branch of science deals with data spread over space and time
- Our examples will come from
    + Geology
	+ Climatology
	+ Meteorology
	+ Ecology
	+ Epidemiology
	+ Demography
	+ Economics
	+ Neuroscience
	+ Physics
- Could (and might) add examples from technology, policy, business, etc.

## So what are we going to cover? {.smaller}

- Exploratory methods for spatio-temporal data
- Linear prediction
- Inference with dependent data
- Generative models
- Simulations and simulation-based inference
- Organized by _method_, skip around on _subject matter_

#### What we will not be covering

- ARIMA (etc.) models --- take 36-618
- Finance --- have a care for your soul

## Exploratory data analysis {.smaller}

- You know that we always start with EDA
- Impose few or no assumptions, try to find patterns
    + _Or_: check whether some pattern holds
- All the usual stuff can still be useful
    + Scatter-plots, histograms, cross-tabulations...
- Special tools for spatio-temporal data:
    + Local averaging to remove noise and reveal trends
	+ Removing trends to reveal random fluctuations
	+ Measuring dependence through covariance
	+ Finding patterns of covariance
	+ Finding periodic oscillations

Start with something vivid, which is also HW 1


## Cherry blossoms in Kyoto

![](https://farm2.staticflickr.com/1726/27534132847_78448546b1.jpg)

> Cherries at the Hirano shrine in Kyoto [(David Montasco on flickr)](https://www.flickr.com/photos/david_montasco/27534132847/)

Flowering of cherry trees has been a central part of Japanese art & culture for well over a millennium

## _Hanami_


```{r, fig.retina=NULL, out.width=400, echo=FALSE}
knitr::include_graphics("https://www.loc.gov/exhibits/cherry-blossoms/cultural-history/Assets/cb0019_enlarge.jpg")
```

> Kitao Shigemasa, _Sangatsu, Asukayam Hanami_ = _Third Lunar Month, Blossom Viewing at Asuka Hill_, c. 1776, via [Library of Congress](https://www.loc.gov/exhibits/cherry-blossoms/cherry-blossoms-in-japanese-cultural-history.html)

Notice the date in the title!

## This is data!

- Ancient diaries[^shonagon], poetry, etc., and modern newspapers, record _when_ cherry trees in Kyoto came into bloom
     + Kyoto because it's the ancient capital and has been a continuous seat of art & culture

[^shonagon]: The _Pillow Book of Sei Shonagon_ is a precious part of our common cultural heritage.


## Cherry blossoms track climate

- Japan gets cold

![](https://farm3.staticflickr.com/2682/32049377133_92a73aab96.jpg)

> Snow at the Hirano shrine [(yopparainokobito on flickr)](https://www.flickr.com/photos/kobikobi/32049377133/)

- Cherries only blossom when it gets warm enough

- $\therefore$ The date when cherry trees are in full flower tells us about how warm the year was
    + Date of first flowering is also informative, but less often recorded


## A data set

- Assembled by [Prof. Yasuyuki Aono](http://atmenv.envi.osakafu-u.ac.jp/aono/kyophenotemp4/)
    + Data points going back to the early 800s
    + Almost continuous for modern times
    + Re-formatted version at [http://www.stat.cmu.edu/~cshalizi/dst/18/data/kyoto.csv]
- For each year, the day of the year (1--366) of full bloom
    + April 1 = `r 31+28+31+1` or `r 31+29+31+1`
    + Aono had to search out the ancient records, poems, histories, etc.
    + and convert dates before 1873 to our (Gregorian) calendar and (common) era (Meiji revolution)

	

## A data set

```{r}
kyoto <- read.csv("http://www.stat.cmu.edu/~cshalizi/dst/18/data/kyoto.csv")
plot(Flowering.DOY ~ Year.AD, data=kyoto, type="l",
     ylab="Day in year of full flowering", xlab="Year (AD)",
     main="Cherry blossoms in Kyoto")
```



## Problems

- How to fill in the data _between_ the observations? (**interpolation**)
    + When did the cherries bloom in 1015?
- How to extend _beyond_ the observations? (**extrapolation**)
    + Make a guess for 2020, _or_ for 800
- How to remove measure noise _in_ the observations? (**filtering**)
    + Poets can be sloppy about dates
- How to separate year-to-year **fluctuations** from longer-term **trends**?
    + How do we model fluctuations?
	+ How do we model trends?
- How seriously should we treat the warming since about 1800?  (**inference**)
    + Could it be an artifact of denser observations?
	+ Does the climate just do stuff like this occasionally?
	+ How long do we need to wait to be confident?

## Smoothed cherry blossoms


```{r, echo=FALSE}
plot(Flowering.DOY ~ Year.AD, data=kyoto, type="l",
     ylab="Day in year of full flowering", xlab="Year (AD)",
     main="Cherry blossoms in Kyoto")
lines(with(na.omit(kyoto), smooth.spline(x=Year.AD, y=Flowering.DOY)), col="blue", lwd=4)
abline(lm(Flowering.DOY ~ Year.AD, data=kyoto), col="green", lwd=4)
```

Both curves come from averaging nearby values

## More concrete problems


- How do we fit either the blue or the green line to the data?
- How can we tell which one is a better fit?
- There ought to be margins of error --- are the two curves even incompatible?
- What do we mean by a "trend" anyway?

## Break for small-group exercise

Which of the two curves do you think is better?  Or are they both bad?
Or, if you can't tell, what more information would you need?

## We're going to need to build some concepts

- Stochastic process = collection of random variables over time or space or
both, typically dependent, say $X(t)$

- Trend = central tendency of the process, say $\mu(t) = \mathbb{E}[X(t)]$

- Fluctuations = difference from the trend

- How can we figure out the trend if we just see $X$ once?


## By Thursday:

- Make sure you're on Canvas & Piazza for the course
    + Let me know if not
- Homework 0
    + Read the course policy on collaboration, cheating and plagiarism
	+ Read the university policy on academic integrity
	+ Read the excerpt from Taurabian's _Manual for Writers_ (Canvas)
	+ Complete Homework 0
- Readings: Eshel, ch. 7; Guttorp, introduction and ch. 1

## Take-aways

- Everything is statistically dependent on everything else
- Dependence means what happens _here_ and _now_ gives us information about _there_ and _then_, so we can predict (interpolate, extrapolate, filter, forecast)
- Dependence means the statistical theory we've learned needs to be fixed
- For now, we will focus on _describing_ dependence
- Next time: trends and smoothing