Modeling Community Attitude through Reaction Tone: A Human-AI Collaborative Framework for Evaluating LLM Alignment with Linguistic Behaviors in Online Communities

The research cuts to the heart of a growing trend and its profound limitations. We are increasingly leaning on large language models as social simulators—to test narratives, predict public sentiment, or understand cultural friction. This paper, through its CARE framework, delivers a sobering reality check: these models are often just generating a kind of statistical puppet show. They can mimic the vocabulary of a group but not its voice. The distinction is everything. A human community's response to a news event isn't just a collection of keywords and sentiment scores; it's a layered performance of identity, history, and shared understanding, what anthropologist Clifford Geertz called "thick description." An online group's reaction to, say, an economic policy shift is shaped by its collective memory of past betrayals, its in-jokes, its internal power dynamics, and the specific platform vernacular it uses. LLMs, even when explicitly prompted with "you are a group of conservative millennials from the Midwest," tend to output a caricature. They can regurgitate stereotypical positions but miss the how—the sarcasm, the defensive humor, the weary resignation, the specific metaphorical language—that constitutes authentic social discourse.

The finding that explicit community prompts fail to improve simulation fidelity is particularly damning. It suggests the problem isn't merely one of missing data or a poorly crafted system prompt. It's architectural. These models are trained on the open internet's sprawling, decontextualized soup of text. They learn what things are said in association with certain labels, but not the embodied, contingent why. Real community discourse is dynamic and event-contingent; it's a live, negotiated performance. An LLM, by contrast, is a static snapshot, reflecting a blended average of discourse patterns. When confronted with a new event, it doesn't "think" as that community; it generates a plausible-sounding reaction based on probabilistic correlations from its training data. This is why the "behavioral signatures" diverge across frontier models—each has been shaped by different data blends and alignment choices, leading to different styles of caricature.

This work forces a difficult question for the entire field of computational social science: Are we just building better, more persuasive mirrors that reflect our own expectations back at us? If an LLM simulation of a Twitter subculture feels "right," is it because it's accurate, or because it confirms the stereotypes the researcher already held? The realism gap isn't just a technical metric; it's an epistemological trap. We risk mistaking fluency for understanding, and coherence for truth. The call for validation through "human-AI collaboration" is a nod to this, acknowledging that the ground truth of social dynamics can only be partially captured in code and fully interpreted by human ethnographers who understand context.

The implications extend beyond academic benchmarks. If companies use LLMs to model consumer communities for product testing, or policymakers use them to simulate public reaction to a crisis, the consequences of this gap become tangible. Decisions could be made based on a hollow echo of public sentiment rather than its substance. The path forward likely involves more than scaling models or better alignment with human feedback. It may require fundamentally new approaches to training that incorporate temporal dynamics and relational context, treating language not as isolated text but as acts situated in a social world. Until then, we should treat LLM social simulations for what this research shows them to be: impressive tools for generating text, but poor substitutes for the rich, messy, and deeply human practice of community itself.