Key findings:
🎭 Facial expressions predict engagement better than text/speech sentiment
📚 Informational content outperforms emotional support content!
🕺 Effects of cross-modal interactions are threshold-dependent (e.g. humor in text and meme visuals boost engagement when both are salient in a post.)

We applied the methodology on 160K+ mental-health-related TikToks to understand how text, visuals, and audio shape engagement.

Interpreting Multimodal Communication at Scale in Short-Form Video: Visual, Audio, and Textual Mental Health Discourse on TikTok Short-form video platforms integrate text, visuals, and audio into complex communicative acts, yet existing research analyzes these modalities in isolation, lacking scalable frameworks to interpret th...

New preprint out!

Social media is trending towards short-form video and multimodal content, but most empirical work still analyzes text, visuals, and audio separately. In this paper, we introduce a scalable + interpretable pipeline to analyze them jointly.

Paper: arxiv.org/abs/2601.15278