Do LLMs Truly Understand When a Precedent Is Overruled? Large language models (LLMs) with extended context windows show promise for complex legal reasoning tasks, yet their ability to understand long legal documents remains insufficiently evaluated. Developing long-context benchmarks that capture realistic, high-stakes tasks remains a significant challenge in the field, as most existing evaluations rely on simplified synthetic tasks that fail to represent the complexity of real-world document understanding. Overruling relationships are foundational to common-law doctrine and commonly found in judicial opinions. They provide a focused and important testbed for long-document legal understanding that closely resembles what legal professionals actually do. We present an assessment of state-of-the-art LLMs on identifying overruling relationships from U.S. Supreme Court cases using a dataset of 236 case pairs. Our evaluation reveals three critical limitations: (1) era sensitivity -- the models show degraded performance on historical cases compared to modern ones, revealing fundamental temporal bias in their training; (2) shallow reasoning -- models rely on shallow logical heuristics rather than deep legal comprehension; and (3) context-dependent reasoning failures -- models produce temporally impossible relationships in complex open-ended tasks despite maintaining basic temporal awareness in simple contexts. Our work contributes a benchmark that addresses the critical gap in realistic long-context evaluation, providing an environment that mirrors the complexity and stakes of actual legal reasoning tasks.

Looking forward to read this paper that won the Best Student Paper award at #Jurix2025: "Do LLMs Truly Understand When a Precedent Is Overruled?" by Li Zhang, Jaromir Savelka, and Kevin Ashley. https://doi.org/10.48550/arXiv.2510.20941

#Precedent #LegalNLP #LNLP #NLLP #LLM #LegalAI #LawAndAI

🍽️ Lunch is on us — huge thanks to our sponsor HumanAds (ERC project)
🙌🏆 Best Presentation Award sponsored by @bloomberglp.bsky.social
🧠✨🎉 Get ready for an amazing day of #NLLP #LegalTech discussions — both in-person & online!

👏 EVERYBODY READY? 🫳🎤

🎉 We have a (appropriately colored) poster! 🇨🇳📜

📢 Check the CfP:
🔗 nllpw.org/workshop/call/

📄 Direct submissions (🗓️ deadline: 26/Aug): 🔗 openreview.net/group?id=EMN...

📝 ARR commits (🗓️ deadline: 2/Sep):
🔗 openreview.net/group?id=EMN...

✍️ Did you already start writing?

#NLLP #legaltech #NLProc

NLLP Workshop 2025 The seventh workshop on Natural Legal Language Processing (NLLP 2025) explores methods and applications of Natural Language Processing for the Legal Domain by focusing on legal text and text with lega...

☀️ We’re here too - getting ready for the 7th NLLP workshop (8/Nov @emnlpmeeting.bsky.social ) 🚀✨
The updated Call for Papers is coming soon — stay tuned! 📣📄

💙 Follow us & spread the word about NLLP🌍💬

🔗 Keep an eye on the website for updates! 🖥️🔍 nllpw.org/workshop/

#NLLP #legaltech #nlproc #nlp