Skip to the content
Living Work Library · Project 03

AI Build-vs-Buy Framework

Before I compared a single model, I found out my measuring stick was bent.

A 125M-parameter specialist beat six foundation models on a locked human-labeled benchmark. The recommendation was neither build nor buy. It was a hybrid operating model.

Research project · no production deployment Seven-class classification

A five-person MBA research team. I led the scope, owned the build and codebase, and ran the model evaluation. Human labeling and adjudication were shared.

← Selected Work
LOCKED HOLDOUT · N = 1,000CONSENSUS-ADJUDICATED
Agreed label Disagreement Adjudicated

Fictional complaint fragments · aggregate agreement metrics measured. Source described only as a public sample from the CFPB Consumer Complaint Database.

Chapter 01 · the question

Call an API, or build a specialist?

When a seven-class classification capability runs continuously at volume, should the organization call a general model API or build and fine-tune a small specialist?

Chapter 02 · the measuring stick was bent

Three annotators, one label each.

KRIPPENDORFF’S α0.3974 THREE-WAY AGREEMENT50.0%
Fragments arriving Holdout locked · n = 1,000
Beat 1 of 7 · Fragments arriving

Fictional complaint fragments · aggregate agreement metrics measured. Public sample from the CFPB Consumer Complaint Database.

Chapter 03 · repairing the evaluation

The codebook, not the model, was the first thing to fix.

Before revision α 0.3974 Full three-way agreement was 50.0%. The disagreement showed that the definitions needed revision.
After revision α 0.5479 Definitions were rewritten and boundaries between adjacent classes were made explicit. Three-way agreement rose to 63.0%.
Residual Disagreement did not disappear. Genuinely ambiguous complaints remained. Those went to consensus adjudication, and the adjudicated set was locked as the holdout — n = 1,000, seven classes.
What changed Class definitions were rewritten with decision rules for the two classes annotators most often confused, and edge cases were given worked examples.
What stayed hard Complaints describing two problems at once stayed contested. They were adjudicated by consensus rather than forced into one reading by a rule.
Why it had to come first A benchmark built on unreliable labels would have ranked the models by the noise in the labels, not by their behaviour.
Chapter 04 · the head-to-head

One locked benchmark, read three ways.

Axis begins at 0.70
Macro-F1 Led the decision
Foundation 0.7900
Specialist 0.8492
MCC Confirmed it
Foundation 0.8364
Specialist 0.8634
Accuracy Reported, not decision-leading
Foundation 0.9060
Specialist 0.9275
Metric 1 of 3 · Macro-F1

Best foundation model and the RoBERTa specialist on the locked consensus-adjudicated holdout, n = 1,000, seven classes. Architectures other than the RoBERTa specialist are reported as “best foundation model” without per-architecture attribution.

Chapter 05 · what the metrics meant

Accuracy stayed visible. It did not lead the decision.

Two classes represented 83.3% of the locked holdout; macro-F1 and MCC carry more information about minority-class performance.

Macro-F1Every class counts equally, so a rare class cannot be ignored.
MCCReads the whole confusion matrix, including the classes a model quietly avoids.
AccuracyDominated by the two majority classes; reported, not decision-leading.
Chapter 06 · the operating decision Splitting the work into three roles
  1. General model Bootstrapping and taxonomy refresh Fast to start, useful for drafting and revising the label set when categories change.
  2. Specialist Recurring classification The steady, high-volume work: stronger macro-F1 and MCC on the locked holdout, and an evaluation pass of roughly 15 minutes against roughly four hours for the GenAI path.
  3. People Ambiguous and high-risk cases The residual disagreement never went away. Cases that stayed contested belong with people, by design.
One decision axis · bootstrap → run → adjudicate

Neither build nor buy. A hybrid operating model.

Repair the evaluation system before trusting the model comparison.

Step 1 of 4 · General model

A five-person MBA research project. Teja led scope, owned the build and codebase, and ran the head-to-head evaluation. Human labeling and adjudication were shared. Research project · no production deployment. Source described only as a public sample from the CFPB Consumer Complaint Database; complaint fragments shown anywhere in this artifact are fictional.

← Return to Selected Work