Topics covered
Subject ▸ LLMs
craft: A CRAFT Pipeline for Evaluating LLM-Generated Data
Status: v0.1.0, in active development. Feedback welcome.
craft operationalizes the C-R-A-F-T framework of Ko, Tai, and Webb Williams into 12 short R functions. It does not call LLM APIs itself: it takes the outputs you already have — labels, confidences, rationales — and walks them through five steps.
| Step | Functions | What it does |
|---|---|---|
| Construct | role() |
Documents how the LLM is being used (annotator, ML system, silicon participant) and which metric family applies. |
| Report | reliab(), reliab_pairs(), valid(), dual() |
Reliability (Cohen’s κ, weighted κ, Fleiss’, Krippendorff’s α, ICC, percent agreement) paired with validity (precision, recall, F1, accuracy, balanced accuracy, MCC). |
| Assess | stab() |
Cross-prompt and cross-model stability. |
| Field audit | audit(), disagree(), tau_sens() |
Surfaces disagreements and low-confidence agreements; confidence-threshold sensitivity. |
| Translate | dsl_fit(), dsl_cmp() |
Design-based supervised learning, so misclassification uncertainty propagates into inference. |
| Report | report() |
Emits a reproducibility supplement capturing the model, version, prompt, metrics, thresholds, audit decisions, and DSL results. |
Installation
# install.packages("remotes")
remotes::install_github("casstai/craft-r")
The dsl dependency (Egami et al.):
remotes::install_github("naoki-egami/dsl")
Quick start
library(craft)
# C: document the role-task
rt <- role("annotator", "classify climate stance",
gold = TRUE, prompt_type = "few-shot")
# R: reliability and validity in one call
dual(ratings, gold = ratings$human, pred = ratings$gpt5,
reliability_method = "kripp")
Documentation
- Getting Started (5-minute tutorial)
- Full walkthrough: climate stance
- Transparency report for journal submission
Citation
If you use craft, please cite both the framework paper and the package:
Ko, Hyein, and Yuehong Cassandra Tai. “Can We Trust LLM-Generated Data? The CRAFT Framework for Measurement and Inference in Political Science.” Under review.
Tai, Yuehong Cassandra, and Hyein Ko. 2026. craft: A CRAFT Pipeline for Evaluating LLM-Generated Data. R package version 0.1.0. https://github.com/casstai/craft-r
When F1 Is Not Enough: Auditing Codebooks for LLM-Assisted Annotation
Large language models (LLMs) are increasingly used for text annotation in social science, but standard performance metrics do not explain why errors occur. Low F1 may reflect limitations of the annotation system, defects in the written codebook, or inconsistency in reference labels. We propose a pre-deployment codebook audit for LLM-assisted annotation. The audit uses class-specific metrics and confusion matrices to identify problematic classes and boundaries, structured document-level review to attribute error sources, and targeted codebook revision followed by held-out evaluation.In Name Only: What LLM Annotators Learn from Codebooks, Examples, and Fine-Tuning?
This project asks what large language models actually learn when they are given a codebook, a set of labeled examples, or fine-tuning data for an annotation task — and whether the resulting labels reflect the construct a researcher intended.