Topics covered
Subject ▸ R package
craft: A CRAFT Pipeline for Evaluating LLM-Generated Data
Status: v0.1.0, in active development. Feedback welcome.
craft operationalizes the C-R-A-F-T framework of Ko, Tai, and Webb Williams into 12 short R functions. It does not call LLM APIs itself: it takes the outputs you already have — labels, confidences, rationales — and walks them through five steps.
| Step | Functions | What it does |
|---|---|---|
| Construct | role() |
Documents how the LLM is being used (annotator, ML system, silicon participant) and which metric family applies. |
| Report | reliab(), reliab_pairs(), valid(), dual() |
Reliability (Cohen’s κ, weighted κ, Fleiss’, Krippendorff’s α, ICC, percent agreement) paired with validity (precision, recall, F1, accuracy, balanced accuracy, MCC). |
| Assess | stab() |
Cross-prompt and cross-model stability. |
| Field audit | audit(), disagree(), tau_sens() |
Surfaces disagreements and low-confidence agreements; confidence-threshold sensitivity. |
| Translate | dsl_fit(), dsl_cmp() |
Design-based supervised learning, so misclassification uncertainty propagates into inference. |
| Report | report() |
Emits a reproducibility supplement capturing the model, version, prompt, metrics, thresholds, audit decisions, and DSL results. |
Installation
# install.packages("remotes")
remotes::install_github("casstai/craft-r")
The dsl dependency (Egami et al.):
remotes::install_github("naoki-egami/dsl")
Quick start
library(craft)
# C: document the role-task
rt <- role("annotator", "classify climate stance",
gold = TRUE, prompt_type = "few-shot")
# R: reliability and validity in one call
dual(ratings, gold = ratings$human, pred = ratings$gpt5,
reliability_method = "kripp")
Documentation
- Getting Started (5-minute tutorial)
- Full walkthrough: climate stance
- Transparency report for journal submission
Citation
If you use craft, please cite both the framework paper and the package:
Ko, Hyein, and Yuehong Cassandra Tai. “Can We Trust LLM-Generated Data? The CRAFT Framework for Measurement and Inference in Political Science.” Under review.
Tai, Yuehong Cassandra, and Hyein Ko. 2026. craft: A CRAFT Pipeline for Evaluating LLM-Generated Data. R package version 0.1.0. https://github.com/casstai/craft-r