---
title: "Final Course Evaluation report for 36-617, Fall 2022"
author: "Brian Junker"
date: "12/06/2022"
output: 
  pdf_document:
    keep_tex: true
header-includes:
  - \usepackage{hyperref}
    
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)

##    includes:
##      in_header: "fix-tables.tex"

```

On page 3 below is a summary of your ratings for 36-617, and on the succeeding pages are your text comments and questions for me.

Once again, you were really thoughtful and helpful in your comments about the course!  I will say a bit more about your comments below, but here I'll just say that I think you've hit on several impportant points about the course.

Here are a few comments and reactions to things that caught my eye in your ratings, comments and questions:

\textbf{Ratings}

\begin{itemize}

\item I'm pretty happy with your ratings of how well you understand things, and especially your interest in the course.  

\item Although many think the difficulty of the course was about right, roughly half the class think it was too hard (``a bit too hard'', or ``way too hard'').  I think that's related in part to how many topics we cover, and in part to the fact that the course is fast-paced.

\item With respect to pacing, a little more than half think the pace was about right, and almost half think it was a bit too fast.  I see many of your written comments are about the course being a bit fast-paced too, and I think this is something to work on in the syllabus for future semesters.

\end{itemize}

One thing I didn't ask you to rate is whether you were especially stressed, either by this course or by the totality of things you had to do for the MSP program. Some of your written comments speak to that stress, and it is something we should work on for the program.  Some stress is OK, and even healthy because it shows that you care, but too much stress makes it hard to think well and be happy.  


\textbf{Your Comments}

\begin{itemize}
\item Once again, it's nice to see that the zoom recordings are useful, especially as the material has strayed farther away from things that you already sort of knew coming into the class. It's also good to see that office hours and Piazza are still really helpful.
\item Generally people seemed to be happy to learn about nonparametric regression, multilevel models and Bayes.  Several people would have liked more time on nonparametrics and Bayes.  
\item There was a mix of comments suggesting either that the pacing of the course was about right, or it went a bit fast.  It is definitely a fast-paced course, and the impression of fast pace is exacerbated by the many topics we cover.  There really isn't enough time to get in-depth about anything, just enough time to expose you to ideas and things that you will be able to look up in textbooks, on the web, etc., when you need them later in your professional career.
\item There are a noticeable number of comments about being lost or somewhat lost in the Bayesian stuff. (On the other hand, most people seem to ``get'' the multi-level stuff pretty well, which is great!)  In part that's related to the general goals of the course, to expose you to things and ideas that you can get more detail on later in your career when you need them.  But it may also be that the idea of putting Bayes at the end of 617 just doesn't work very well, and that it would be better, in the future,  to skip Bayes in 617 and offer Bayes as a separate semester (e.g. a spring ``topics'' course).  That would give more time to topics more directly related to regression in 617, and more time in the Bayes ``topics'' course to go into more detail about the reasoning behind Bayesian modeling, selection of prior distributions, how and why MCMC works etc.
\item I tried to make  most of the homeworks a more reasonable length in the second half of the semester, but hw09 was killer and I didn't really give you enough time to do it.  Several of your written comments speak to this.  It was also challenging for you to have an assignment on Bayes due in the same week as your final IDMRAD paper.
\item On a related note, while many people appreciated what they were learning from building an IDMRAD paper from the ground up, absolutely no one liked the rather compressed time schedule for the project.  That's really my fault: I should have started the whole process at least a week earlier, to give you more time to do and absorb the work in hw09, before trying to construct an IDMRAD paper around it.  I just dropped the ball on how much time was left in the semester, after accounting for Thanksgiving break.  This is definitely something to do better on, in the future! 

\end{itemize}

\textbf{Your Questions \& Things You Learned}

\begin{itemize}
\item People really seemed to appreciate learning about mixed-effects and multi-level models, which is great.  I think people appreciated the exposure to Bayes as well but many people were hoping to learn more and/or in greater depth about Bayes.  As I suggested above, we may need to think about splitting Bayes off into a separate course.

\item Someone asked how much of what we learned transfers to other programming systems, like Python.  Fitting lm, lmer and gmlmer models seems pretty far along in the Python world (see for example \tt\url{https://www.slideshare.net/PyData/diamond-mixed-effects-models-in-python})\rm but I am less sure that a full set of diagnostic tools for regression and multilevel models exists in Python (you can always roll your own, but that's not really a good solution for most people).  See for example \tt\url{https://towardsdatascience.com/going-from-r-to-python-linear-regression-diagnostic-plots-144d1c4aa5a}\rm and \tt\url{https://stackoverflow.com/questions/46607831/python-linear-regression-diagnostic-plots-similar-to-r}\rm.  Stan can certainly be connected to python, but again I'm less sure about a Python analogue to bayesplot, for example.

\item A couple people mentioned that they have a better sense of how to do a complete data analysis from raw data to final report, and/or that interpretability is the most important part of any analysis.  I'm really glad to see that. 

Remember, ordinary regression, generalized linear models and multilevel models (and even GAMs) have the great advantage of interpretability, so if you are working with someone who wants to understand and be able to communicate the mechanisms underlying the data you have, these models can be quite useful.  If you are working on a problem where the main concern is good prediction, then methods with more of an ML flavor will generally do better, at the cost of interpretability.  Sometimes your boss or client will want both, of course, and then you have to manage a tradeoff between predictive accuracy and interpretability.  In any case, it is almost always worth trying a linear model or glm first, in part to see if a model with lots of  interpretability will suffice to answer your boss's or client's question, and in part to serve as a ``baseline'' to try to beat with sexier methods that you may be asked to use.

\item A couple people were confused about Bayes' rule.  I'm not quite sure what the confusion is, yet.  Maybe it's really confusion about how to specify a good likelihood and prior for any particular problem; people also asked about that, as well as more information about MCMC.  For a general understanding of Bayes, I think Lynch Ch 3 is quite good. For a general understanding of MCMC I like Lynch Ch 4.  (Both chapters are in the Week 13 and week 14 folders for you to read, and you can get the whole book for free from springer link online.)  Specifying good likelihoods and priors for complex data like our multilevel data is an art that takes practice.

\end{itemize}
I really enjoy teaching 617 and getting to work with the MSP students each year, and this is no exception.  Thank you for a great semester!

--BJ

\clearpage


```{r echo=F, fig.height=8}
library(stringr)
library(kableExtra)
library(ggplot2); theme_set(theme_bw())
library(gridExtra)

data <- read.csv("eval-data.csv",header=T)

attach(data)

## par(mfrow=c(3,2))

## myplot <- function(x,title=deparse(substitute(x))) {
##     tab <- rep(0,5)
##     names(tab) <- paste(1:5)
##     raw <- table(x)
##     for (i in names(raw)) {
##         tab[i] <- tab[i] + raw[i]
##     }
##     barplot(tab,xlab=title)
## }

myplot <- function(x,title=deparse(substitute(x))) {
  
  x <- sapply(strsplit(x,","),function(x) x[[1]])
  x <- substr(x,1,20)
  
  w <- names(table(x))
  w <- w[order(as.numeric(sapply(strsplit(w,":"),function(x) x[[1]])))]
  
  x <- factor(x,levels=w)
  
  g <- ggplot(data=data,aes(x=x)) 
  
  g <- g + geom_bar() + xlab("") + ggtitle(title) + 
    theme(axis.text.x = element_text(angle = 45,vjust=0.35))
  
  return(g)
}

g1 <- myplot(Understanding,title="My Understanding is ( 1, 2, 3, 4, 5)...")

g2 <- myplot(Interest,"My Interest is ( 1, 2, 3, 4, 5)...")

g3 <- myplot(hard.easy.number,title="The Difficulty is (-2,-1, 0, 1, 2)...")

g4 <- myplot(slow.fast.number,title="The Pace is (-2,-1, 0, 1, 2)...")

## myplot(useful,title="Useful")

grid.arrange(g1,g2,g3,g4,ncol=2)

detach()

```


```{r echo=F}
attach(data)

com_table <- function (x) {

com <- x

for (i in 1:length(com)) {
    loc <- str_locate_all(com[i],"Â")[[1]][,1]
##  loc <- str_locate_all(com[i],"PP ")[[1]][,1]
    if (length(loc)>0) {
        for (j in 1:length(loc)) {
            substr(com[i],loc[j],loc[j]) <- " "
        }
    }
    com[i] <- paste(com[i],"\n\n")
##    com[i] <- paste(com[i],"\n\n---------------------------------------\n\n")
##    if (str_length(com[i])>0) {
##        cat(com[i],"\n\n---------------------------------------\n\n")
##    }
}

set.seed(143256)

com <- com[str_length(com)!=0]
com <- com[!is.na(com)]
com <- com[com!="NA"]
com <- com[com!="N/A"]
com <- com[com!=""]
com <- data.frame(text=sample(com))
n <- dim(com)[1]

return(com)

}


```

\clearpage

```{r echo=F}
x <- com_table(Comments) 

x %>% kbl(col.names="Comments",longtable=TRUE) %>% column_spec(1,width="6in") %>% kable_styling(latex_options = c("hold_position", "repeat_header")) 

```

\clearpage

```{r echo=F}
x <- com_table(Questions) 

x %>% kbl(col.names="Questions",longtable=TRUE) %>% column_spec(1,width="6in") %>% kable_styling(latex_options = c("hold_position", "repeat_header")) 


detach(data)

```

