CaseCrunch, a recent AI startup founded by Cambridge law students, has won a challenge against 112 lawyers in predicting outcomes of claims with the UK Financial Ombudsman for mis-selling Payment Protection Insurance (PPI). The competition provided participants with factual scenarios for PPI mis-selling claims and asked for a yes or no prediction on whether the claim would be upheld. The scenarios were real, decided cases. Lawyers were permitted to complete their predictions in an unsupervised environment and were invited to use all available resources that they would normally use in performing their work. A Technology Judge and a Legal Judge supervised the fairness of the competition.
Competition results
There were 775 predictions submitted by the participants. Lawyers successfully predicted 62.3% of outcomes, whereas the application CaseCruncher Alpha predicted 86.6% of outcomes.
What are the implications of this early competition between human lawyers and AI?
It seems clear that in narrow areas, AI systems are already at or even surpassing the performance of humans to predict legal outcomes. As Ludwig Bull, Scientific Director of CaseCrunch, notes: “Evaluating these results is tricky. These results do not mean that machines are generally better at predicting outcomes than human lawyers. These results show that if the question is defined precisely, machines are able to compete with and sometimes outperform human lawyers. The use case for these systems is clear. Legal decision prediction systems like ours can solve legal bottlenecks within organisations permanently and reliably.”
While it may remain unclear what the ultimate application of AI in law will be, it is clear that in coming years AI will disrupt the traditional practice of law. By driving down cost and removing routine work, AI systems should allow human lawyers to focus on strategic elements of a case and client interaction, and ultimately make legal services more accessible to the public.
Additional resources
- To learn more about the competition, visit CaseCrunch.
- View a video by the legal judge Felix Steffk (LLM, PHD) Univeristy of Cambridge Lecturer in Law describing the competition and results on YouTube.
