---
title: 36-462/36-662, Data Mining
author: Cosma Shalizi
date: Lecture 1, 14 January 2020 --- Welcome to the course
output: slidy_presentation
---

## Welcome!

![](https://live.staticflickr.com/311/32174260996_3d3f68747b_z_d.jpg)

## Agenda for today

> - Course mechanics
>    + All of the details are in the syllabus
> - General orientation to the course

## What is data mining?

> - Finding useful patterns in large collections of data
>     + a.k.a. "knowledge discovery in data bases"
> - Two sides:
>     + Methods for **coming up with** candidate patterns
>     + Methods for **checking** candidate patterns
> - The most useful and most checkable patterns are ones which let us _predict_

## What are we going to learn about

So many things!


## Nearest neighbors

> - "This new case will do what similar cases did" is surprisingly powerful
> - Need good ways to define "similar"
> - Need good ways to find similar cases in big data sets

## Prediction and decision trees

![](classification-tree-we-can-believe-in.jpg)

- Simple, binary-choice models for prediction
- Plus ways of combining many trees to get "forests"

## Nonlinear features and kernels

- Recycle everything we know about linear models by using new, nonlinear features of the data
- Avoid having to actually calculate the transformations by using tailored similarity measures ("kernels")

## Information measures

- How much information does one feature give us about another?
- How much information does the model give us about predictions?
- How much information does the data give us about the models?


## Dimension reduction

![](https://live.staticflickr.com/3560/3487720211_1df38f25e8.jpg)

> - More than 3D is very hard for us to grasp
>    + _Maybe_ 5 if you use color and animation well
> - Somehow reduce the huge number of features to something more manageable _but_ still intelligible


## Clustering

> - Divide the data into groups, without knowing what the right groups are
> - Using probability models so this isn't _totally_ arbitrary

## Checking our guesses

> - Cross-validation for seeing how well our models _actually_ predict
>    + Divide the data, fit the model to one part, evaluate predictions on the other
> - Bootstrap for seeing how much uncertain we should have about the model
>    + Sub-divide the data, fit the model to only part, see how much things change over many random sub-samples
> - How do we account for the fact that we've chosen our model to fit the data?

## Applications

> - Could have looked at astronomy, biology, marketing, medicine, supply-chain management, social welfare services...
> - Will instead focus on three:
>     + Information retrieval
>     + Recommendation engines
>     + Criminal risk/recidivism prediction
>     + You live with the first two and hopefully will never be on the receiving end of the third


## Information retrieval

![](https://live.staticflickr.com/3281/5826310009_1834b058ab_z.jpg)

> - Everything important about data mining in a small package
>     + Finding similar items by content
>	  + Using features to define "similar content"
>	  + Making predictions about patterns
>	  + Evaluating predictions about patterns
	


## Recommendation engines

> - "You may also like"
> - Uses: nearest neighbors, clustering, dimension reduction, classification, ...
> - Abuses/unintended consequences...

## Fairness in prediction

> - How do we keep our models from just reproducing the injustices of the society around us?
>    + _Can_ we keep our models from doing this?
>    + What counts as an injustice?
> - Case study: how risky is it to release this just-arrested person, before their trial?



## Waste, fraud and abuse

![](https://web.archive.org/web/20080901072600if_/http://failblog.files.wordpress.com/2008/01/camerafail.jpg)

> - Sometimes data mining just won't work
>     + Bad data
>	  + Overwhelming data and the curse of dimensionality

## Waste, fraud and abuse

![](https://web.archive.org/web/20080901072600if_/http://failblog.files.wordpress.com/2008/01/camerafail.jpg)

> - Sometimes data mining is just wrong

## Where did this come from?

> - Statistics
>     + Exploratory data analysis
>	  + Regression, especially regression by matching
>	  + Principal components and factor analysis
>	  + Classifiers
>	  + Clustering
> - Computer science
>    + Databases
>	 + Pattern recognition (= classsifiers)
>    + Dimension reduction (= PCA and factor analysis)
>	 + Clustering (= clustering)
>    + Nearest neighbors (= matching)
>    + Machine learning (= predictive statistical modeling)

## Where did this _really_ come from?

```{r, fig.retina=NULL, out.width=400, echo=FALSE}
knitr::include_graphics("mound-city-1870.jpg")
```

- "The state, the coldest of all cold monsters"



## Where did this _really_ come from?

![](http://farm3.static.flickr.com/2411/2404562785_5b887699de.jpg?v=0)

- We invented data-processing machines, _before_ computers, because we were already keep tabs on so much about so many people

## Where did this _really_ come from?

![](https://live.staticflickr.com/369/19194898203_6f9b3bba5f_c.jpg)

> - Computers made it _really easy_ to **create**, **store** and **analyze** data
>     + Create: sensors (phones, cameras, cash registers...), digitization (doing everything via computer)
>     + Store: hard disks, disk arrays, data-bases...
>     + Analyze: that's where we come in
>     + Organizations create and store the data even when they don't have particular analyses in mind
> - Tension: flexibility to find many different patterns vs. vulnerability to noise and coincidence
>    + All the data is great: brilliant ideas from 50--100 years ago become practical!
>    + Why we look a lot of different ways of finding patterns
>    + All the data is a problem: If you run a gazillion analyses, (gazillion/20) will be significant at the 5% level
>    + Why we look at how to avoid fooling ourselves

## What will you need to know?

> - 36-401, modern regression
> - $=$ Linear statistical models **in R**
> - $=$ Actual experience with predictive modeling of data
>    + $+$ Mathematical statistics (for notions of inference and error)
>    + $+$ Probability (for notions of distributions and risk)
>    + $+$ Linear algebra through eigenvalues and eigenvectors (**essential** for multivariate data)
>    + $+$ Calculus (**essential** for optimization)

## Course mechanics

- Class meetings
- Readings
- Homework
- Class homepage: [http://www.stat.cmu.edu/~cshalizi/dm/20]
    + Full syllabus with all the details
	+ Links to course assignments, due dates, etc.
	+ What to read
- Canvas: submitting work, gradebook, some readings
- Piazza: question-answering
	


## Class meetings

> - Lecture: me explaining and demonstrating stuff, you asking questions
> - In-class exercises: you checking your understanding
> - No electronics

## In-class exercises

- Short ($< 20$) minute problem-solving exercises
- Pencil-and-paper, not electronics
- Groups of up to 4
- Most if not all class meetings


## Reading

> - Most class meetings will have **required** reading: do it!
> - Many will have _recommended_ reading: try to do it
> - Most will have background reading: if you get interested

## Reading: Textbook

![_Principles of Data Mining_](https://mitpress.mit.edu/sites/default/files/styles/large_book_cover/http/mitp-content-server.mit.edu%3A18180/books/covers/cover/%3Fcollid%3Dbooks_covers_0%26isbn%3D9780262082907%26type%3D.jpg?itok=GrWC_te-)

## Reading: Textbook

```{r, fig.retina=NULL, out.width=400, echo=FALSE}
knitr::include_graphics("https://mathbabe.files.wordpress.com/2016/02/weaponsmath-r4-6-06.jpg")
```

## Reading: Textbook

![_The Ethical Algorithm: The Science of Socially Aware Algorithm Design_](https://global.oup.com/academic/covers/pop-up/9780190948207)

## Reading: Recommended

![_Statistical Learning from a Regression Perspective_](https://media.springernature.com/w306/springer-static/cover-hires/book/978-3-319-44048-4)

## Reading: Recommended

![_Elements of Statistical Learning_](https://web.stanford.edu/~hastie/ElemStatLearn/CoverII_small.jpg)

## Homework

- Implementing methods on actual data
- Working out some of the mathematical details
- Practicing interpreting and communicating the results
- One assignment per week, 14 in all
    + Released by Friday each week (sometimes earlier)
    + Usually due Thursdays at 10 pm via Canvas

## Homework

- 10% of each homework will be graded on the quality & clarity of your communication
    + There will be a rubric for this on each assignment
- Most (if not all) homeworks will also have an online assignment
    + Easy if you've done the reading
	+ Usually due Sundays at 10 pm via Canvas
    + Online assignments will be about 10% of your grade

## Grading

- Homework: 90% of your grade
    + **Lowest 2** homework+online grades dropped automatically
	+ No late homework
	+ If you do all 14 homeworks with a minimum grade of $\geq 60$%, **lowest 3** grades get dropped
- In-class exercises: 10% of your grade
    + **Lowest 3** dropped automatically
	+ If you do all exercises with minimum of $\geq 60$%, lowest **5** dropped
- No exams
- Grade boundaries: 90 for an A, 80 for a B, etc.
    + 662: 97 for an A+, etc.

## Time expectations

> - This is a 9 **credit-hour** class
> - You spend 3 hours in lecture each week
> - $\Rightarrow$ **6 hours** working on the class outside of lecture each week
>    + averaged over the semester
>    + Talk to me if it's taking much longer than that





## Cheating, collaboration & plagiarism

> - Don't
> - You can talk to each other, you can read whatever you like, but everything you turn in **must** be your own work
>    + Exception to "read whatever you like": Don't read old solutions, or share this year's
>    + Exception to "all work must be your own: Working together is OK for in-class exercises
> - Full policy in the syllabus
> - You will need to do a HW 0 about the class cheating policy before getting graded

## Homework format

- We will use [R Markdown](http://rmarkdown.rstudio.com/) to integrate your code directly in to your writing
    + Write a source file that's mostly ordinary text, plus the R code you want
	+ "Knit" to an HTML or PDF with the text plus the output of the code (figures, tables, numbers)
- Ensures **computational reproducibility**: your results really came from the code that you say/think they came from
- You will turn in both your raw R Markdown file _and_ the knitted PDF or HTML

## Switch to R Studio

Specifically `welcome.Rmd`

## Some lessons from the demo

- Using R Markdown is almost no extra work over just using R and copying-and-pasting your results in
    + Eventually it's _less_ work than copying-and-pasting _and_ it's more reliable
- Trees are a characteristic data-mining model
	+ Very flexible (can fit _anything_, with a big enough tree...)
	+ Good at things linear models are bad at (thresholds, interactions)
    + Almost never realistic (... but when is the linear model right?)

## Next time: The truth about linear regression

- A review, without the mythology, to set us up for more powerful prediction methods
- Do the reading!
