Agenda for today

• Course mechanics
  • All of the details are in the syllabus
• General orientation to the course

What is statistical learning?

• Statistical learning:
  • how to fit predictive models
  • to training data
  • usually by solving an optimization problem
  • so the model will probably predict well
  • on average
  • on new data

Course mechanics

• Class meetings
• Readings
• Homework
• Class homepage: [http://www.stat.cmu.edu/~cshalizi/dm/22]
  • Full syllabus with all the details
  • Links to course assignments, due dates, etc.
  • What to read when
• Gradescope: submitting almost all your work
• Canvas: submitting reading questions, gradebook, solutions
• Piazza: question-answering

Class meetings

• Lecture: me explaining and demonstrating stuff, you asking questions
• In-class exercises: you checking your understanding
• No electronics when we’re in person
• No recordings

In-class exercises

• Short (\(< 20\)) minute problem-solving exercises related to lecture and homework
• Pencil-and-paper, not electronics
• Groups of up to 4 when we’re in person
• Most if not all class meetings, due via Gradescope the next day

Reading

• Most class meetings will have key reading: do it!
• Many will have suggested reading: try to do some of it
• Most will have background reading: if you get interested

Reading: Textbook

Principles of Data Mining

Reading: Textbook

Reading: Textbook

The Ethical Algorithm: The Science of Socially Aware Algorithm Design

Reading: Textbook

Statistical Learning from a Regression Perspective

Reading: Recommended

Elements of Statistical Learning

Homework

• Implementing methods on actual data
• Working out some of the mathematical details
• Practicing interpreting and communicating the results
• One assignment per week, 12 in all
  • Released by Friday each week (sometimes earlier)
  • Usually due Thursdays at 6 pm via Gradescope

Homework

• 10% of each homework will be graded on the quality & clarity of your communication
  • There will be a rubric for this on each assignment
• Most (if not all) homeworks will also have an online assignment
  • Easy if you’ve done the reading
  • Usually due Mondays at 6 pm via Canvas
  • Online assignments will be about 10% of each homework

Grading

• Homework: 90% of your grade
  • Lowest 3 homework grades dropped automatically
  • No late homework
  • If you do all homeworks with a minimum grade of \(\geq 60\)%, lowest 4 grades get dropped
• In-class exercises: 10% of your grade
  • Lowest 5 dropped automatically
  • If you do all exercises with minimum of \(\geq 60\)%, lowest 6 dropped
• No exams
• Grade boundaries: 90 for an A, 80 for a B, etc.
  • 662: 97 for an A+, etc.

Time expectations

• This is a 9 credit-hour class
• You spend 3 hours in lecture each week
• \(\Rightarrow\) 6 hours working on the class outside of lecture each week
  • averaged over the semester
  • Talk to me if it’s taking much longer than that

Cheating, collaboration & plagiarism

• Don’t
• You can talk to each other, you can read whatever you like, but everything you turn in must be your own work
  • Exception to “read whatever you like”: Don’t read old solutions, or share this year’s
  • Exception to “all work must be your own: Working together is OK for in-class exercises
• Full policy in the syllabus
• You will need to do a HW 0 about the class cheating policy before anything else will be graded

Homework format

• We will use R Markdown to integrate your code directly in to your writing
  • Write a source file that’s mostly ordinary text, plus the R code you want
  • “Knit” to an HTML or PDF with the text plus the output of the code (figures, tables, numbers)
• Ensures computational reproducibility: your results really came from the code that you say/think they came from
• Keep your raw R Markdown; expect to be randomly picked to turn it in about once this semester

What are we going to learn about

So many things!

Nearest neighbors

• “This new case will do what similar cases did” is surprisingly powerful
• Need good ways to define “similar”
• Need good ways to find similar cases in big data sets

Prediction and decision trees

• Simple, binary-choice models for prediction
• Plus ways of combining many trees to get “forests”

Nonlinear features and kernels

• Recycle everything we know about linear models by using new, nonlinear features of the data
• Avoid having to actually calculate the transformations by using tailored similarity measures (“kernels”)
• Random nonlinear functions are surprisingly powerful

Dimension reduction

• More than 3D is very hard for us to grasp
  • Maybe 5 if you use color and animation well
• Somehow reduce the huge number of features to something more manageable but still intelligible

Clustering

• Divide the data into groups, without knowing what the right groups are
• Using probability models so this isn’t totally arbitrary

Mathematical tools to make all this work

• Decision theory to think carefully about what’s a good prediction
• Optimization to actually fit good models
• Regularization to keep the models from getting too weird

Checking our guesses

• Estimates of prediction error on new data
  • Models are always optimistic
• Approximate formulas based on optimization theory
• Cross-validation for seeing how well our models actually predict
  • Divide the data, fit the model to one part, evaluate predictions on the other

Applications

• Could have looked at astronomy, biology, marketing, medicine, supply-chain management, social welfare services…
• Will instead focus on four:
  • Recommendation engines
  • Credit scoring and loan decisions
  • Criminal risk/recidivism prediction
  • Predictive policing
  • You live with the first two and hopefully will never be on the receiving end of the others

Recommendation engines

• “You may also like”
• Uses: nearest neighbors, clustering, dimension reduction, classification, …
• Abuses/unintended consequences…

Fairness in prediction

• How do we keep our models from just reproducing the injustices of the society around us?
  • Can we keep our models from doing this? Should we?
  • What counts as an injustice?
• Case study: credit scores, predicting who will repay loans, and loan-making
• Case study: how risky is it to release this just-arrested person, before their trial?
• Case study: what happens when the police try to predict where there will be crime? when they try to predict who will do the crimes?

Waste, fraud and abuse

• Sometimes statistical learning just won’t work
  • Bad data
  • No useful predictions to be made
  • Overwhelming data and the curse of dimensionality

Waste, fraud and abuse

• Sometimes statistical learning is just the wrong thing to do

Where did this come from?

• Statistics worked out lots of ideas about how to make predictions and how to evaluate predictions
  • Regression, especially regression by matching and by nonlinear functions
  • Principal components and factor analysis
  • Classifiers and discriminant analysis
  • Clustering and mixture models
  • Cross-validation
  • Many of these ideas weren’t very practical in the 1960s, or 1920s, or even 1800s…
• Computer scientists had been interested in getting machines to learn almost from the first computers in the 1940s
• Between 1980 and 1995, some computer scientists started using those statistical tools, and statisticians started using models and algorithms from CS
  • Or from theoretical biology (“neural networks”), physics, etc.
• “Data mining”, “Knowledge discovery in data bases”, “Statistical learning”, “Machine learning”…

What will you need to know?

• 36-401, modern regression
• \(=\) Linear statistical models in R
• \(=\) Actual experience with predictive modeling of data
  • \(+\) Mathematical statistics (for notions of inference and error)
  • \(+\) Probability (for notions of distributions and risk)
  • \(+\) Linear algebra through eigenvalues and eigenvectors (essential for multivariate data)
  • \(+\) Calculus (essential for optimization)

Next time: The truth about linear regression

• A review, without the mythology, to set us up for more powerful prediction methods
• Do the reading!

Backup: Where did this really come from?

• “The state, the coldest of all cold monsters”

Backup: Where did this really come from?

• We invented data-processing machines, before computers, because we were already keep tabs on so much about so many people
• We have traditions of data-driven prediction and decision-making going back hundreds of years

Backup: Where did this really come from?

• Computers made it really easy to create, store and analyze data
  • Create: sensors (phones, cameras, cash registers…), digitization (doing everything via computer)
  • Store: hard disks, disk arrays, data-bases…
  • Analyze: that’s where we come in
  • Organizations create and store the data even when they don’t have particular analyses in mind

Backup: Where did this really come from?

• Tension: flexibility to find many different patterns vs. vulnerability to noise and coincidence
  • All the data is great: brilliant ideas from 50–100 years ago become practical!
  • Why we look a lot of different ways of finding patterns
  • All the data is a problem: If you run a gazillion analyses, (gazillion/20) will be significant at the 5% level
  • Why we look at how to avoid fooling ourselves

Backup: Where did this really come from?

• If we hadn’t spent centuries re-shaping our societies to collect and act on data, dropping in people skilled in statistical learning wouldn’t accomplish very much
• Even dropping in statistical learners and computers wouldn’t accomplish very much