Can We Trust LLM-Generated Data? The CRAFT Framework for Measurement and Inference in Political Science

Ko, Hyein, and Yuehong Cassandra Tai (co-lead author)
(2026)

code  software

Large language models are increasingly used for data generation in political and social science, yet the discipline lacks a shared standard for validating their output. Existing frameworks address pieces of the workflow, mostly covering a single stage. We propose C-R-A-F-T, a five-step framework that connects construct definition through inferential adjustment within a single, model-agnostic specification: C-onstruct roles and tasks; R-eport dual-track metrics; A-ssess stability across prompts, and models; F-ield human audit and adjudication; T-ranslate to inference incorporating uncertainty. We illustrate the framework with state legislators’ online climate stances and clarify its scope.

The framework is implemented in the craft R package.