Unexpected finding from a big study on what ChatGPT did to colleges: "once the COVID-19 disruption is modeled separately, the introduction of ChatGPT had no detectable effect on grades.. course evaluations for subject understanding, interest, and relative workload show no change"
**How does GenAI affect grades in college? It can help learn, but it can also substitute for learning ("substitution hypothesis"). And how do students *experience* courses that GenAI can easily ace?**
We spent year+ trying to answer these questions, using some of the best data:
- Intuition: If GenAI helps learn, grades should increase in courses regardless of what assessment method they use. If GenAI substitutes learning, grades should increase disproportionately in courses with "susceptible" assessments (i.e. take-home essays).
- Data: admin data from one of largest U.S. universities
- Method: Compare grades and student evals in susceptible vs. non-susceptible courses before and after ChatGPT
Findings:
- No disproportionate increase in grades in GenAI-susceptible courses
- No disproportionate change in student evals
Interpretation:
- Not clear that GenAI results in learning substitution or reduced student satisfaction
- This could be for lots of reasons: students use it to learn, faculty adjust courses, etc.
Caveat:
- Analyses like these use lots of assumptions and researcher-degrees-of-freedom. So results should always be taken as suggestive rather than conclusive
- COVID is a big confounder, and results can be sensitive to how transient or permanent you assume its effects are