Generative AI has made polished social-media copy nearly free to produce, and a plausible worry is that cash-strapped local governments adopting it will pay for the savings with reach: residents may read machine-drafted posts as less authentic and engage with them less. We test that worry directly, within account, across 767,131 Facebook posts from roughly 4,900 U.S. city and county governments in all 50 states and DC. After we control for length, theme, writing register, and formality, posts that read as more machine-like draw 2–3% fewer likes and comments per standard deviation of machine-likeness, and no reliably different share count.
Status: v0.1.0, in active development. Feedback welcome.
craft operationalizes the C-R-A-F-T framework of Ko, Tai, and Webb Williams into 12 short R functions. It does not call LLM APIs itself: it takes the outputs you already have — labels, confidences, rationales — and walks them through five steps.
Step
Functions
What it does
Construct
role()
Documents how the LLM is being used (annotator, ML system, silicon participant) and which metric family applies.
If you use craft, please cite both the framework paper and the package:
Ko, Hyein, and Yuehong Cassandra Tai. “Can We Trust LLM-Generated Data? The CRAFT Framework for Measurement and Inference in Political Science.” Under review.
Tai, Yuehong Cassandra, and Hyein Ko. 2026. craft: A CRAFT Pipeline for Evaluating LLM-Generated Data. R package version 0.1.0. https://github.com/casstai/craft-r
Large language models (LLMs) are increasingly used for text annotation in social science, but standard performance metrics do not explain why errors occur. Low F1 may reflect limitations of the annotation system, defects in the written codebook, or inconsistency in reference labels. We propose a pre-deployment codebook audit for LLM-assisted annotation. The audit uses class-specific metrics and confusion matrices to identify problematic classes and boundaries, structured document-level review to attribute error sources, and targeted codebook revision followed by held-out evaluation.
Large language models (LLMs) are increasingly used to annotate social-science texts, but high accuracy does not establish that they apply researcher-supplied category definitions: predictions may instead reflect prior associations with category labels or labeled demonstrations. We introduce a counterfactual audit that reassigns definitions among output labels, distinguishing adherence to supplied definitions from preservation of a dataset’s original mapping. Across three political-text datasets, models rarely follow reassigned definitions, and standard fine-tuning strengthens the original mapping without reliably increasing definition adherence.
Large language models are increasingly used for data generation in political and social science, yet the discipline lacks a shared standard for validating their output. Existing frameworks address pieces of the workflow, mostly covering a single stage. We propose C-R-A-F-T, a five-step framework that connects construct definition through inferential adjustment within a single, model-agnostic specification: C-onstruct roles and tasks; R-eport dual-track metrics; A-ssess stability across prompts, and models; F-ield human audit and adjudication; T-ranslate to inference incorporating uncertainty.
Despite recent advances in understanding the capabilities and limits of generative artificial intelligence (GenAI) models, we are just beginning to understand their capacity to assess and reason about the veracity of content. We evaluate multiple GenAI models across tasks that involve the rating of, and reasoning about, the credibility of information. The information in our experiments comes from content that subnational U.S. politicians post to Facebook. We find that GPT-4o outperforms other models, but all models exhibit only moderate agreement with human coders.
The rise in hate speech targeting minority communities underscores the urgent need for effective tools to detect and address harmful content in digital communication. We examine over 3 million tweets posted by state legislators between 2020 and 2021, focusing on messages directed at Asian communities. To address the nuanced nature of hate speech, we develop three comprehensive definitions for identifying hate speech. With a human-in-the-loop approach, our fine-tuned BERT-NLI model achieved improved classification performance.
Elected officials increasingly communicate across multiple social media platforms. We argue that platforms are not neutral channels: the same political pressures can be expressed differently across them. Using 6.19 million Facebook and X posts from 6,356 U.S. state legislators, we examine climate attention and stance. Ideology structures climate communication differently across platforms: in 2020–2021, liberal legislators devote substantially more climate attention on X, whereas ideological differences in climate stance are stronger on Facebook.