Thanks a lot for raising this! In our case, we specifically selected the downstream tasks based on empirical findings from humans linking the questionnaires and their constructs to the corresponding behaviors.
📄 Paper: arxiv.org/abs/2510.11254
A very big thank you to my amazing collaborators @marlutz.bsky.social, @indiiigo.bsky.social, and @mstrohm.bsky.social!
3/3
For all 3 constructs we looked at –sexism, racism, and morality– the correlations between tests scores and behavior in a related downstream task are only weak positive, or even negative.
📢 Our results call for LLM-specific evaluations instead of applying tests originally developed for humans.
2/3
Are you using survey-style questionnaires designed for humans to measure characteristics of LLMs?
In our #EACL2026 paper, we evaluate both the reliability and validity of such tests and found that their scores do not reflect real-world model behavior. In fact, they can be deceptive!
🧵1/3
🚨New paper alert🚨
🤔 Ever wondered how the way you write a persona prompt affects how well an LLM simulates people?
In our #EMNLP2025 paper, we find that using interview-style persona prompts makes LLM social simulations less biased and more aligned with human opinions.
🧵1/7
Thrilled to talk about how seemingly small decisions in silicon sampling can have a large impact on simulated survey responses 👀 Join us on Oct 29th! 👈
👋 #ACL2025NLP 🇦🇹 @marlutz.bsky.social and I are presenting our poster on demographic representativeness of LLMs today!
🕦 10:30-12:00
📍 Hall X5 (board 1 or 14 according to different sources 🧐)
Here’s the paper on ACL anthology: aclanthology.org/2025.finding...
Drop by!
Chair for Data Science in the Economic and Social Sciences at University of Mannheim having lots of fun at #ic2s2 @janajung.bsky.social @wanlo.bsky.social @indiiigo.bsky.social @jrupprec.bsky.social @maximiliankreutner.bsky.social and Stefano Balietti
Poster Session 1 is live in the Atrium! Explore the work and cast your daily vote—just scan the QR code to submit your favorite poster ID. Everyone has one vote per day. #ic2s2
LLMs can generate synthetic survey responses, e.g. for imputation, but how reliable are they? 📋
At #IC2S2, I'll be sharing our research on the robustness of AI-generated responses to perturbations and if they mirror human survey biases. 🤖
Come by my poster on Tuesday between 1:30 and 3:30 p.m.
Really excited to also present this work at #IC2S2 next week in Norrköping! 🎉 I'd love to discuss how to produce LLM survey responses at my poster on Wed at 13:30 (Poster Session 2, Poster ID 68) 📊
Very excited to head to #IC2S2 next week! 🎉
In our project, we tested whether a psychological assessment can measure sexism in LLMs, and found that applying such tools to LLMs is not as straightforward as it seems.
Find me and my poster at Poster Session 1 (Tue 12:30-14:30) — hope to see you there