🏆ADVSCORE won an Outstanding Paper Award at #NAACL2025
🚨 Don't miss out on our poster presentation *today at 2 pm* by Yoo Yeon (first author).
📍Poster Session 5 - HC: Human-centered NLP
💼 Highly recommend talking to her if you are hiring and/or interested in Human-focused Al dev and evals!
A screenshot of a paper showing the title - "Pairscale: Analyzing Attitude Change in Online Communities" by Rupak Sarkar, Patrick Wu, Kristina Miler, Alexander Hoyle and Philip Resnik.
Are you tired of using traditional stance detection to measure the polarity of text? Our #NAACL25 paper proposes an approach that uses pairwise comparisons to order texts on a continuous scale, capturing both implicit and explicit evidence in language.
📍Today in Hall 3 from 4-5:30pm
Come say hi!
Read more at aclanthology.org/2025.naacl-l...!
A huge shoutout to my advisor
@rachelrudinger
and everyone else in the CLIP lab at UMD for their support and feedback :-) (6/6)
This helps us build groups of examples that evaluate the same pieces of knowledge, allowing us to measure under what *contexts* an LLM can correctly draw a particular inference ("inferential consistency"). We find that LLMs still exhibit room for improvement on this front. (5/n)
We propose a method to pinpoint the particular pieces of knowledge a defeasible reasoning example aims to evaluate by identifying the atom(s) that are most critical in determining the overall label of a defeasible NLI example. (4/n)
We also explore how atomic hypothesis decomposition can help us better understand the complexities of defeasible reasoning, a softer inference task that requires models to weigh the effects of multiple, sometimes competing, pieces of evidence on a hypothesis. (3/n)
For example, after decomposing hypothesis from an NLI premise-hypothesis pair into atoms, we can measure whether its judgment on the overall pair is consistent with its set of judgments on each premise-atom sub-problem in a logical way. (2/n)
I'll be presenting this work with @rachelrudinger at #NAACL2025 tomorrow (Wednesday 4/30) in Albuquerque during Session C (Oral/Poster 2) at 2pm! 🔬
Decomposing hypotheses in traditional NLI and defeasible NLI helps us measure various forms of consistency of LLMs. Come join us!
Ah, thanks Joe!! :) And a huge thank you to you for all your early feedback -- it definitely helped the way we framed the concept of atomic inference.