For my UC Berkeley MIDS DATASCI 201 final project, our team studied how large language models can be integrated into 8th-grade science classrooms in New Jersey to improve comprehension rather than encourage misuse. We focused on a central policy and learning question: how does guided use of an education-focused model (KhanMigo) compare with unguided use of general-purpose free LLMs for student learning outcomes, engagement quality, and equity across income and baseline performance groups.
The study design used stratified sampling across 50 New Jersey public schools and split students into a control group (free public LLM access) and an intervention group (KhanMigo + teacher guidance). We proposed measuring outcomes with pre/post NJSLA science performance, prompt usage patterns (frequency, prompt type, and work-deferral behavior), and qualitative interviews/surveys from students, teachers, parents, and administrators. Planned analyses included two-sample t-tests, ANOVA subgroup comparisons, linear regression on score improvement, NLP-based response quality evaluation (relevance/correctness/depth), and Wilcoxon signed-rank testing for paired survey shifts.
Our project emphasized actionable policy output for the NJ Board of Education, including recommendations for AI literacy, prompt-engineering support, and implementation guardrails that reduce inequity and overdependence risks. We also documented key limitations, including teacher-effect confounding, subject-specific generalizability, and the challenge of separating model effects from guidance effects. Project presentation: Beyond the Ban - DATASCI 201 Final Project.