--- title: Introduction to the Course author: Cosma Shalizi date: 1 September 2020, 36-467/667 output: slidy_presentation bibliography: locusts.bib --- ```{r, include=FALSE} # General set-up options library(knitr) opts_chunk$set(size="small",background="white", highlight=FALSE, cache=TRUE, autodep=TRUE, tidy=TRUE, warning=FALSE, message=FALSE, echo=FALSE) ``` ## Data over Space and Time - Statistical study of processes that unfold over space, or time, or both ## Data over Space and Time - Statistical study of processes that unfold over space, or time, or both + Data over _space_: ```{r, fig.retina=NULL, out.width=700, echo=FALSE} knitr::include_graphics("nyt-2020-08-31-space.jpg") ``` (From [https://www.nytimes.com/interactive/2020/us/coronavirus-us-cases.html] on 2020-08-31) ## Data over Space and Time - Statistical study of processes that unfold over space, or time, or both + Data over space + Data over _time_: ![](nyt-2020-08-31-time.jpg) (From [https://www.nytimes.com/interactive/2020/us/coronavirus-us-cases.html] on 2020-08-31) ## Data over Space and Time - Statistical study of processes that unfold over space, or time, or both + Data over space + Data over time + Data over space _and_ time: ```{r, fig.retina=NULL, out.width=800, echo=FALSE} knitr::include_graphics("nyt-2020-08-31-st.jpg") ``` (From [https://www.nytimes.com/interactive/2020/us/coronavirus-us-cases.html] on 2020-08-31) ## Data over Space and Time - Statistical study of processes that unfold over space, or time, or both > - Applications: History, science, and lots of practical policy & technology > - Not everything: > + Good experiments > + Good surveys > + Wishful thinking ## Special Statistical Issues > - The goal: learn how _here_ and _now_ relates to _there_ and _then_ > - The two problems: > 1. Everything depends on everything else $\Rightarrow$ basic theory doesn't apply > 2. We don't get multiple samples of time or of space > + $\Rightarrow$ $n=1$, always > - Nonetheless, we will see how to do statistical inference with dependent, spatio-temporal data ## Course Mechanics {.smaller} - Syllabus: [http://www.stat.cmu.edu/~cshalizi/dst/20] + All assignments will be posted there + Information about readings and other course resources ditto - Canvas: turn in assignments, gradebook, some readings that can't go on the public web - Piazza for question-answering #### Homework Every week, Thursdays at 6 pm (Pittsburgh) #### After-Lecture Questions Online through Canvas, the day after each class, _short_ answers #### Office hours - Me, W 2:00--3:00 by Zoom, Th 2:00--3:00 by Piazza - TAs, TBD ## Textbooks Gidon Eshel, _Spatiotemporal Data Analysis_ ```{r, fig.retina=NULL, out.height=300, echo=FALSE} knitr::include_graphics("https://pup-assets.imgix.net/onix/images/9780691128917.jpg?w=640") ``` **Required**, but [full text PDF through JSTOR](https://www.jstor.org/stable/j.ctt7zthhv) ## Textbooks Peter Guttorp, _Stochastic Modeling of Scientific Data_ ```{r, fig.retina=NULL, out.height=300, echo=FALSE} knitr::include_graphics("https://images.tandf.co.uk/common/jackets/amazon/978041299/9780412992810.jpg") ``` - Required excerpts will be distributed (as PDFs) through Canvas + It's a good book and I recommend reading the whole thing, but $$$ ## Textbooks Paul Teetor, _The R Cookbook_ ![](https://covers.oreillystatic.com/images/9780596809164/cat.gif) **Recommended**; consult as needed ## Lectures - Clarification, amplification, examples, alternatives - _Complements_ to the readings + $\therefore$ Do the readings ahead of time - Please ask questions - I will _often_ ask you to solve some problem during lecture + not graded, _but_ + students find the discussion of the solutions very helpful + and more helpful if you've tried it yourself - _Not_ recorded ## Assignments/Grading 1. After-class review questions and exercises (10%) 2. Weekly homework (90%) NO exams ## After-class review questions and exercises - 10% of your total grade - Specific questions about stuff we went over in class, and/or in the readings - Turn in through Canvas, the day after each class + Due at 6 pm (Pittsburgh time) the next day so $> 24$ hr - Should take $\approx 10$ minutes + But it's untimed - Usually short-answer questions, may sometimes be multiple choice - Hand-written math OK as needed + Scan or just take a picture with your phone and upload + Dark ink on white paper works best - All equally weighted - Lowest 4 dropped, no questions asked ## Homework - 90% of your total grade - Data analysis & computing & a little theory - Must use R Markdown - Always turned in electronically via Canvas/Gradescope - Due at 6 pm on Thursday (Pittsburgh time) every week + Except this first week (no homework) + and Thanksgiving week (also no homework) - Lowest 3 dropped, no questions asked + If you turn in all 13, and get at least 60% on each, lowest _4_ dropped - NO LATE HOMEWORK FOR ANY REASON + Turn in as many incomplete or rough versions as you like well ahead of the deadline ## R Markdown - R Markdown is a straightforward way to combine text, math, code, and code output in one document - Enforces _reproducibility_: your numbers, figures, etc., are generated by the code, so results always match - Use is _required_ for this class + Your homework will lose points if you don't use it * You will lose more points as the semester goes on * Resources for learning R Markdown on the syllabus page ## Collaboration, Cheating & Plagiarism - Everything you turn in for a grade needs to be your own work, _or_ an acknowledged borrowing from an approved source - Full policy for this class is part of the syllabus - Read that policy and the [CMU policy on academic integrity](http://www.cmu.edu/policies/student-and-student-life/academic-integrity.html) - Turn in Homework 0 on these policies by Thursday + Not graded, but you won't get credit for any work until you complete it successfully ## Any questions? ## What are the big issues? - We see $X$ at time $t$ and point $r$, $X(r,t)$, and what to know how it relates to $Y$ at time $s$ and point $q$, $Y(q,s)$ Problems: 0. Basic statistical theory is about independent, identically distributed (IID) data. 1. But we (usually) only see _one_ realization of a whole process. 2. Every observation is dependent on every other observation. 3. Basic statistical theory says that $n=1$ and refuses to draw any inferences. ## How are we going to deal with these issues? - Methods for describing relationships and finding patterns in the data - Especially methods for _predicting_ $X(r,t)$ from $Y(q,s)$ - _Incorporate_ dependence into statistical theory, so we can say when methods will work - Quantifying uncertain through modeling and simulation ## Why is this worth knowing? - Italo Calvino, "All at One Point" (1963) imagines the world un-extended in time or space[^calvino] [^calvino]: Calvino's _Cosmicomics_ is a precious part of our common cultural heritage. - Otherwise, every branch of science deals with data spread over space and time - Our examples will come from + Geology + Climatology + Meteorology + Ecology + Epidemiology + Demography + Economics - Could (and might) add examples from neuroscience, physics, technology, policy, business, etc. ## So what are we going to cover? {.smaller} - Exploratory methods for spatio-temporal data - Linear prediction - Inference with dependent data - Generative models - Simulations and simulation-based inference - Organized by _method_, skip around on _subject matter_ #### What we will not cover - ARIMA (etc.) models --- take 36-618 - Finance ### To get started... - Something vivid, but more cheerful than Covid ## Cherry blossoms in Kyoto ![](https://farm2.staticflickr.com/1726/27534132847_78448546b1.jpg) > Cherries at the Hirano shrine in Kyoto [(David Montasco on flickr)](https://www.flickr.com/photos/david_montasco/27534132847/) Flowering of cherry trees has been a central part of Japanese high art & culture for well over a millennium ## _Hanami_ ```{r, fig.retina=NULL, out.width=400, echo=FALSE} knitr::include_graphics("https://www.loc.gov/exhibits/cherry-blossoms/cultural-history/Assets/cb0019_enlarge.jpg") ``` > Kitao Shigemasa, _Sangatsu, Asukayam Hanami_ = _Third Lunar Month, Blossom Viewing at Asuka Hill_, c. 1776, via [Library of Congress](https://www.loc.gov/exhibits/cherry-blossoms/cherry-blossoms-in-japanese-cultural-history.html) Notice the date in the title! ## This is data! - Ancient diaries[^shonagon], poetry, etc., and modern newspapers, record _when_ cherry trees in Kyoto came into bloom + Kyoto because it's the ancient capital and has been a continuous seat of art & culture [^shonagon]: The _Pillow Book of Sei Shonagon_ is a precious part of our common cultural heritage. ## Cherry blossoms track climate - Japan gets cold ![](https://farm3.staticflickr.com/2682/32049377133_92a73aab96.jpg) > Snow at the Hirano shrine [(yopparainokobito on flickr)](https://www.flickr.com/photos/kobikobi/32049377133/) - Cherries only blossom when it gets warm enough - $\therefore$ The date when cherry trees are in full flower tells us about how warm the year was + Date of first flowering is also informative, but less often recorded ## A data set - Assembled by [Prof. Yasuyuki Aono](http://atmenv.envi.osakafu-u.ac.jp/aono/kyophenotemp4/) + Data points going back to the early 800s + Almost continuous for modern times + Re-formatted version at [http://www.stat.cmu.edu/~cshalizi/dst/20/data/kyoto-2020.csv] - For each year, the day of the year (1--366) of full bloom + April 1 = `r 31+28+31+1` or `r 31+29+31+1` + Aono had to search out the ancient records, poems, histories, etc. + and convert dates before 1873 to our calendar[^calendar] [^calendar]: The traditional Japanese calendar system (from 645) didn't have an accumulating count of years the way we do, but rather reckoned years by "name eras", so a given year would be called something like "year $k$ of the reign of Emperor So-and-so." (Some emperors had more than one name era, and the name of the era was not the emperor's name but one he chose, but that was the basic idea.) This is as though we called this year 3 of Trump, called 2016 year 8 of Obama, etc. Part of the work of compiling data like this is to keep track of when, in our terms, each name era began. Japan in fact still has name eras for some official purposes (this is year 3 of the Reiwa era), but in 1873, as part of the Meiji Revolution, the government adopted the Gregorian calendar and the common era, the year-numbering scheme formerly known as AD/BC. ("Meiji" is itself an era name.) More-or-less similar schemes, where the count of years resets when the ruler changes, have been very common across the world; unending sequential year numbers seem to have been invented twice, by the Seleucid dynasty in what we now call the Middle East around 300 BCE, with [remarkable consequences](https://aeon.co/essays/when-time-became-regular-and-universal-it-changed-history), and in Central America, in the form of the "long count" calendar used by the Mayans and other civilizations and reckoning days since 11 August 3114 BCE. (The calendar was certainly invented much more recently and we don't know why that was their zero-day.) ## A data set ```{r} kyoto <- read.csv("http://www.stat.cmu.edu/~cshalizi/dst/20/data/kyoto-2020.csv") plot(Flowering.DOY ~ Year.AD, data=kyoto, type="l", ylab="Day in year of full flowering", xlab="Year (AD)", main="Cherry blossoms in Kyoto") ``` ## Problems {.smaller} > - How to fill in the data _between_ the observations? (**interpolation**) > + When did the cherries bloom in 1015? > - How to extend _beyond_ the observations? (**extrapolation**) > + Make a guess for 2021, _or_ for 800 > - How to remove measurement noise _in_ the observations? (**filtering**) > + Poets can be sloppy about dates > - How to separate year-to-year **fluctuations** from longer-term **trends**? > + How do we model fluctuations? > + How do we model trends? > - How seriously should we treat the warming since about 1800? (**inference**) > + Could it be an artifact of denser observations? > + Does the climate just do stuff like this occasionally? > + How long do we need to wait to be confident? ## Smoothed cherry blossoms ```{r, echo=FALSE} plot(Flowering.DOY ~ Year.AD, data=kyoto, type="l", ylab="Day in year of full flowering", xlab="Year (AD)", main="Cherry blossoms in Kyoto") lines(with(na.omit(kyoto), smooth.spline(x=Year.AD, y=Flowering.DOY)), col="blue", lwd=4, lty="dotted") abline(lm(Flowering.DOY ~ Year.AD, data=kyoto), col="green", lwd=4, lty="dashed") ``` Both curves come from averaging observed values ## More concrete problems - How do we fit either the dotted blue or the dashed green curve to the data? - How can we tell which curve is a better fit? - There ought to be margins of error --- are the two curves even incompatible? - What do we mean by a "trend" anyway? ## We're going to need to build some concepts - Stochastic process = collection of random variables over time or space or both, typically dependent, say $X(t)$ - Trend = central tendency of the process, say $\mu(t) = \mathbb{E}[X(t)]$ - Fluctuations = difference from the trend - How can we figure out the trend if we just see $X$ once? ## By Thursday: - Make sure you're on Canvas & Piazza for the course + Let me know if not - Homework 0 + Read the course policy on collaboration, cheating and plagiarism + Read the university policy on academic integrity + Read the excerpt from Taurabian's _Manual for Writers_ (Canvas) + Homework 0 isn't graded but you need to do it to _get_ a grade for anything else ## Take-aways - Everything is statistically dependent on everything else - Dependence means what happens _here_ and _now_ gives us information about _there_ and _then_, so we can predict - Dependence means the statistical theory we've learned needs to be fixed - For now, we will focus on _describing_ dependence - Next time: EDA and graphics