California Bar Exam Controversy: AI-Generated Questions Raise Questions on Fairness

The revelation by the State Bar of California that some questions in the problematic February 2025 bar exam were developed with the assistance of artificial intelligence has sparked outrage and concern among legal educators and applicants. The announcement, made public by the legal licensing body, also stated that it plans to request the California Supreme Court to adjust scores for those who took the exam—a test already marred by technical failures such as platform crashes, screen lags, and submission errors that left many test-takers unable to complete their exams. This confirmation of AI involvement has left experts dubious about the integrity of the exam process. Leah Wilson, the State Bar’s executive director, maintains confidence in the validity of the test despite these issues, asserting that the multiple-choice questions were designed to fairly assess the competence of test-takers. However, the fallout from the exam has led to significant criticism, such as from Mary Basick, assistant dean at the University of California, Irvine, who described the situation as 'unbelievable.' Katie Moran, a bar preparation professor, highlighted the predicament of employing non-lawyers to draft exam questions, raising ethical questions about the qualifications necessary for such a high-stakes assessment. The initial exam protocol, which included 171 scored multiple-choice questions, relied heavily on questions crafted by Kaplan and a number of others derived from first-year law exams. Of particular note is the 23 questions created by ACS Ventures, the State Bar’s psychometrician, with AI assistance. The intensity of the criticism from various educational authorities, including a collective of 14 law school deans who have called for a return to traditional testing methods, demonstrates the significant credibility crisis facing California’s bar exam process. Andrew Perlman of Suffolk University Law School expressed that while AI may have potential applications in education and testing, its use must come with stringent oversight and vetting to ensure that quality standards are upheld. As skepticism toward AI's efficacy in such critical fields continues to mount, the legal community is left to grapple with the question of whether technology can ever truly fill the role of human experts, particularly in domains as sensitive as legal licensing. The future may see increased reliance on AI tools, posing a challenge to educational institutions to adapt and ensure that future lawyers are adequately equipped for the legal profession, both in terms of knowledge and ethical practice.