I gave talks at MIT and Harvard this week about "Science with synthetic data". How can generative models help us learn about the actual world (e.g., social systems) in a principled way? Lots of interesting conversations -- more convinced than ever that there's nuanced issues to navigate here.
14/ This work will be presented as a spotlight talk today at #COLM2025 SocialSim workshop and at NeurIPS 2025.
Paper: arxiv.org/abs/2508.06635
Code: github.com/lasilab/valid-synth-inference
13/ I really enjoyed working on this project with the brilliant and kindest @shantanug.bsky.social and great mentors @zacharylipton.bsky.social @donskerclass.bsky.social @brwilder.bsky.social
12/ This framework provides a foundation for easily extensible estimation methods that can safely incorporate the growing variety and quality of synthetic data sources.
11/ At a fundamental level, this work takes a step towards understanding how synthetic data from foundation models can be used to support valid inference. As the usage and promise of FMs continue to grow, so too will the complexity of pipelines that incorporate their outputs.
10/ Empirically, we observe large gains in estimation performance (lower MSE + tighter confidence intervals with valid coverage) across diverse computational social science tasks, with benefits most pronounced in low label regimes.
9/ In other words, in the worst case where synthetic data is *completely* uninformative (bad quality), including it does not hurt, at least asymptotically.
8/ When they are independent from each other, the variance reduces to the optimal variance based only on the real data.
7/ Precisely: The GMM measures the cross-correlations between the synthetic and real data, producing a combination of these moments that reduces the variance of the real data moments if there is information from the synthetic data moments.
6/ Why and when does synthetic data help? We found that the incorporation of synthetic data leads to more precise estimation and tighter confidence intervals when its moments are predictive of the real data moments
5/ Prospectively, it was not intuitive whether the incorporation of additional moments based solely on synthetic data (defined in terms of a separate parameter from the target) would yield any benefits (or even affect) the estimation of the target parameter of the real data.
4/ We propose a solution via a new estimator based on generalized method of moments (GMM) that allows us to incorporate these multiple sources of information by adding moments.
3/ Problem: Naively aggregating these different sources of information leads to highly biased estimates, due to differences in the underlying distribution
2/ In limited labeled regimes, LLMs provide practitioners a cheap alternative to attain imperfect labels and even generate entirely new synthetic samples
💡Can we trust synthetic data for statistical inference?
We show that synthetic data (e.g., LLM simulations) can significantly improve the performance of inference tasks. The key intuition lies in the interactions between the moment residuals of synthetic data and those of real data
Should LLMs be used to review papers? AAAI is piloting LLM-generated reviews this year. I wrote a blog post arguing that using LLMs as reviewers can have bad downstream consequences for science by centralizing judgments about what constitutes good research.
bryanwilder.github.io/files/llmrev...
Photo of Johns Hopkins Campus
I'm recruiting PhD students for Fall 2025! CS PhD Deadline: Dec. 15th.
I work on safe/reliable ML and causal inference, motivated by healthcare applications.
Beyond myself, Johns Hopkins has a rich community of folks doing similar work. Come join us!
would love to join!