Joint work with Tunde Oluwaseyi Ajayi, Seth Aycock, Zain Muhammad Mujahid, Vladana Perlić, Ekaterina Borisova, and Markarit Vartampetian — a research collaboration that started as a mini-project at the Advanced Language Processing School (ALPS 2025, alps.imag.fr)!

LLM-as-a-qualitative-judge: automating error analysis in natural language generation Prompting large language models (LLMs) to evaluate generated text, known as LLM-as-a-judge, has become a standard evaluation approach in natural language generation (NLG), but is primarily used as a q...

Paper: arxiv.org/abs/2506.09147

Code to try it on your task: github.com/tunde-ajayi/...

Poster: drive.google.com/file/d/1_xEc...

I will be presenting LLM-as-a-qualitative-judge at the MME workshop @ #EACL2026!

Will be happy to chat about the importance of analysing frequent error cases in NLP applications and how we attempt to automate it in LLM-as-a-qualitative-judge!

Mar 28, 16:50, poster hall
#NLProc

Joint work with Tunde Oluwaseyi Ajayi, Seth Aycock, Zain Muhammad Mujahid, Vladana Perlić, Ekaterina Borisova, and Markarit Vartampetian — a research collaboration that started as a mini-project at the Advanced Language Processing School (ALPS 2025, alps.imag.fr)!

📣 LLM-as-a-qualitative-judge: automating error analysis in natural language generation

TLDR: our approach outputs a summary of error types and their counts in an NLP system

📜 Paper: arxiv.org/abs/2506.09147
💻 Code to try it on your task: github.com/tunde-ajayi/...

#NLProc #LLM

Applications for #ALPS2026 are now open!

lig-alps.imag.fr/index.php/ap...

👉What? Advanced Language Processing School
👉When? 29.03.26-03.04.26
👉Where? French Alps
👉For whom? Phd & advanced MS students, practitioners in academia and industry

Photos from the last run below!

#NLProc

If you are leaving #acl2025 thinking "I wish for a week of time to think what's next", we have you covered:

alps.imag.fr

Announcing ALPS 2025, our 5th NLP winter school in 🇫🇷 We guarantee great scientific discussions, a fantastic scenery and great speakers:

Arrived in Singapore for #ICLR2025 and will be presenting PROVENCE on Friday, Poster session 3 at 10am, poster #255!

Blogpost: huggingface.co/blog/nadiinc...

Will be happy to meet & chat about #LLMs, #RAG, #InformationRetrieval and #MultilingualNLP :)

#NLProc @naverlabseurope

A small photo-report from this year's run of our Advanced Language Processing School #ALPS2025 in the French Alps near Grenoble: 50+ participants, 9 invited speakers, 5 cool days!

Excited for the next year's run of ALPS! Stay tuned!
lig-alps.imag.fr

#ALPS2025 #NLP #NLProc #WinterSchool #FrenchAlps