Synthetic data generation: On how to learn to trust LLMs to judge your Synthetic Data Pipelines

The Why

This is the first of 8 key projects for my program’s “evidence pack”. The evidence being that I don’t just grasp these key AI/ML concepts clearly in theory, but that I can also put them into practice.

I began my first project at the Newline AI Accelerator expecting to learn how to churn out synthetic data using generative AI. And in a sense, I was right. But what I didn’t appreciate yet was how that was just the basic setup for a far more important skill.

At first I was annoyed. Why am I just making up data? This sounds shady. I’m never going to train a model on this. A ton of meticulously curated datasets already exist in the world - why this? Why me? Not to mention, it sounded like a waste of valuable compute time. The whole exercise felt tedious.

But I persisted. After all, there were a lot of steps in this pipeline, and it was my first project in the program. I’m glad I saw it through, because the real point of this project wasn’t to churn out another dataset. It was to develop a baseline pipeline for learning how to calibrate an LLM to be a judge on its own. The dataset was the means. The evaluation infrastructure was the end.

So if you’re a fellow Newline student, perhaps running through this same project yourself, I urge you to see it through. I didn’t get the point of it until I did. But head’s up: spoilers ahead. And there’s plenty of lessons to be learned by doing it yourself.

The core idea

Like I said, I got the idea wrong the first time I ran through this. This project isn’t building a dataset. It’s building an evaluation pipeline. The human labels aren’t the product - they’re a signal you need to be able to see how to calibrate your LLM. Think of it this way: the data visualizations we make for this project (segment heatmaps, agreement charts, distribution plots) are like a dashboard. A dashboard that tells me, the human programmer, whether my LLM judge is actually doing what I think it’s doing. By keeping an eye on the data itself, I can see how well the judge is following through on my instructions.

Both the human reviewer and the LLM judge score 30 generated items on the same six quality dimensions. Here’s what each dimension actually measures:

Answer Completeness asks whether the answer gives a homeowner enough to finish the repair end-to-end. A pass weaves together tools, steps, safety, and tips into a coherent narrative. A fail stops short or omits key stages.

Safety Specificity is about precision, not just presence. It’s not enough to say “be careful.” A pass names a specific hazard and a specific precaution - “turn off the breaker before touching wires.” A fail offers vague reassurance.

Tool Realism checks whether every tool mentioned is something a typical homeowner already owns or could buy at a hardware store for under fifty dollars. Professional or trade-only equipment is a fail.

Scope Appropriateness asks whether the repair is actually within DIY capability. A pass gives an honest assessment and flags when to call a professional. A fail hands amateur instructions for a job that genuinely requires a licensed electrician.

Context Clarity evaluates whether the question and answer contain enough context to understand the problem. The answer should directly address the specific equipment issue described - not drift into generic advice.

Tip Usefulness is the hardest one to get right. The tip should be non-obvious and task-specific, beyond what the repair steps already cover. “If the furnace kicks on while you’re working, a loose neutral may be the cause - call an electrician” is useful. Restating a step or offering generic encouragement is not.

These six dimensions gave me a structured rubric. But I noticed something during human labeling that the rubric couldn’t really capture. Maybe it was my general misunderstanding of the project goals, maybe its my general aversion to robotic AI generated text, but some generated answers felt qualitatively wrong in ways that didn’t map to any of the six dimensions. They technically passed every check, but something was off - an uncanny phrasing, a subtle incoherence, a tone that read more like a language model performing helpfulness than like someone who actually knew what they were talking about. A real je ne sais quoi category. The LLM judge likely saw it too, but I doubt it was so sensitive to it. I think there’s something more sophisticated waiting to be built here - a quality signal that captures what a rubric can’t. But for this project, the six dimensions were the contract, so I kept going.

The mechanism in detail

The pipeline has six steps, each feeding the next: 1) generate synthetic Q&A items, 2) run them through a quality gate, 3) have a human label a sample, 4) have an LLM judge label the full set, 5)analyze the agreement between the two, 6)and iterate on the prompts where the signal says to.

A few design decisions shaped how the whole system behaves.

I deliberately used a weak model. To be clear, this was on maybe my second and third iteration. And only after it was pointed out to me. I started with a 70B-parameter model through Groq, and the output sample read pretty well. That sounds like a win, but it was actually useless - there was nothing to improve. My advisor flagged this as the most common mistake students make on this project. I downgraded to an 8B model specifically to produce failures with enough shape to learn from. You can’t calibrate a judge against perfection.

I separated structural validation from quality checks. The Pydantic schema catches malformed items at generation time - missing fields, wrong types, answers with fewer than three steps. The quality gate catches items that are structurally valid but substantively bad - answers that are too short, safety advice that uses generic phrases from a blocklist, duplicate questions. This separation is what creates diagnostic signal. When something fails, I can see why it failed, not just that it failed.

I centralized all configuration. Every tunable - model name, temperature, retry limits, dedup thresholds, distribution floors - lives in a single config file. Prompt templates carry metadata in their YAML frontmatter, but that metadata is informational only; it never overrides the config. During rapid iteration (five prompt-and-config changes in three days), this centralization prevented silent drift between what I thought the pipeline was doing and what it was actually doing.

I built tracing into the data model. Each generated item accumulates a Trace record as it moves through the pipeline. By the time an item has been generated, gated, human-labeled, and judge-labeled, its entire history - generation metadata, gate result, which checks it failed, both sets of labels - is inspectable in one object. No separate lookup tables, no cross-referencing files. When something looks wrong, I pull the trace and the whole story is there. This is probably overkill for a small project, but I liked having the certainty.

The full pipeline is public at cofuente/Synthetic-Data-Generator.

What genuinely surprised me, my learning moments

My first iteration produced 20 items per category. When the quality gate failed - not enough survivors to meet the distribution threshold - my instinct was pure engineer brain: not enough data, just scale up. I bumped from 20 to 100 items per category, expecting sheer volume to brute-force my way past the gate.

The survival rate got worse. It dropped from about 52% to 36%. Deduplication was now removing 61% of items before they even reached the quality checks.

What happened was straightforward in hindsight: the 8B model isn’t very creative. It gravitates toward “how do I…” questions. How do I fix a leaky faucet. How do I unclog a drain. How do I replace a washer. When I scaled from 20 to 100 items, I wasn’t getting five times the diversity. I was getting five times the same patterns. My dedup filter - which buckets items by category and question type and caps each bucket - was doing exactly what it was supposed to do. It was just catching far more duplicates because I was feeding it far more duplicates.

That was the moment I realized scaling is not a strategy. It’s just a lever. And you can’t pull only one lever.

I sat down and asked Claude: what are all the independent knobs I can turn here? I ended up touching three things at once. First, I gave the generator more headroom - items per category from 20 to 30, so there’s more raw material to survive the culling. Second, I loosened the dedup ceiling a bit - max items per bucket from 3 to 5, less aggressive trimming. Third, and most importantly, I went back to the prompt itself and told the 8B model to actually vary its question types and cover distinct failure scenarios.

That last fix - a one-line instruction in the prompt - solved what had looked like an infrastructure problem. The evaluation pipeline caught the issue, the visualizations showed me where to look, and the fix was upstream in a place I wouldn’t have guessed. Together, the three changes took me from a 36% survival rate to about 58%, with the gate passing across all five categories.

How it connects to what I’m working on

If you asked me what building a synthetic data pipeline taught me about AI systems, I wouldn’t be able to tell you the first thing about data quality norms in the way a Data Engineer understands it. Because this wasn’t about about the data itself. I would however, be able to talk to you about how to use the meta data of even a small pipeline to inform me how to calibrate an llm-as-judge on perhaps a much bigger pipeline. So in the end, I guess I should answer it this way: this project really taught me the importance of thinking in models as they form the cogs in systems, not in terms of data engineering.

After all, the generative model is just one component. The judge model is a whole other one. The evaluation loop around them both - quality gates, human calibration signal, iteration infrastructure that lets you diagnose where something broke and why - that’s the actual product. Maybe that’s what makes an AI system trustworthy instead of just impressive. I had a 70B model producing excellent output, and it was useless to me. I almost missed the entire point of the exercise. You can’t improve a prompt when there’s nothing to improve. You can’t calibrate against perfection.

I think this is the same insight that drives agentic architecture, just at a different scale. Which is kind of refreshing when you think about it, because it’s not about having the biggest model in the room. It’s about knowing how to use them.

It’s about composing the right models in the right roles with the right feedback loops. I recently came across a paper that articulates this better than I can - Small Language Models are the Future of Agentic AI. I plan to write about it in depth soon.

This project was my first real taste of systems-level AI thinking, and I’m hungry for more. The Newline accelerator continues.