sia.hackernoon.com

Moving from theory to a tangible, working system that turns AI mistakes into high-quality training data.

In the first part of this series, we talked about the messy middle of AI development, which is the frustrating gap between a promising 85% prototype and a production-ready 99% system. We established that the key isn't just a better model, but a system that learns from every mistake.

Today, we're going to get our hands dirty and construct a simple, working web application that demonstrates the core loop of a data flywheel. By the end of this article, you will have corrected an AI's mistake and generated a perfect, fine-tuning-ready dataset from your work.

We'll be using the correction_deck_quickstart example from our open-source framework, Foundry. This example is self-contained, requires no external services like Docker or Redis, and proves just how powerful the core pattern can be.

The Scenario: A Flawed Invoice AI

Imagine we've built an AI to extract structured data from invoices. We feed it an image of an invoice, and we want it to return a clean JSON object. On its first pass, the AI does a decent job, but it's not perfect. It produces this flawed output:

{
  "supplier_name": "Lone Star Provisins Inc.", // <-- TYPO!
  "invoice_number": "785670",
  "invoice_date": "2025-08-20",
  "inventory_items": [
    {
      "item_name": "TAVERN HAM WH", 
      "total_quantity": 15.82, 
      "total_unit": "LB", 
      "total_cost": 87.80
    },
    {
      "item_name": "ONIONS YELLOW JBO", 
      "total_quantity": 5, // <-- WRONG QUANTITY! Should be 50.
      "total_unit": "LB", 
      "total_cost": 35.50
    }
  ]
}

Our goal is to build a system that allows a human to easily fix these two errors and, crucially, captures those fixes for retraining.

The Three Core Components of Our Flywheel

To build this, our Foundry framework relies on three simple but powerful Python abstractions:

Job: Think of this as a ticket in a tracking system. It's a database model that represents a single unit of work for the AI. It holds the input_data (the invoice image), the initial_ai_output (the flawed JSON above), and a place to store the corrected_output once a human has fixed it.
CorrectionRecord: This is the golden ticket. When a human saves their correction, we don't just update the Job. We create a separate, self-contained CorrectionRecord. This record is purpose-built for fine-tuning. It stores a clean copy of the original input, the AI's bad attempt, and the human's "ground truth" correction. It’s a perfect, portable training example.
CorrectionHandler: This is the business logic. It's a simple class that orchestrates the process: it takes the submitted form data from the web UI, validates it, updates the Job, creates the CorrectionRecord, and handles exporting all the records into a training file.

These three pieces work together to form the backbone of our flywheel. Now, let's see them in action.

Let's Build It: The Quickstart in Action

If you're following along, clone the Foundry repository, navigate to the examples/correction_deck_quickstart directory, and install the dependencies.

Step 1: Run the Quickstart Script

From your terminal, simply run:

python quickstart.py

You'll see a message that a local web server has started on http://localhost:8000.

--- Foundry Quickstart Server running at http://localhost:8000 ---
--- Open the URL in your browser to use the Correction Deck. ---
--- Press Ctrl+C to stop the server and complete the flywheel. ---

Step 2: Use the Correction Deck UI

Open that URL in your browser. You'll see a simple Correction Deck UI. On the left is the source invoice image. On the right is a web form pre-filled with the AI's flawed data.

Your task is to be the human in the loop. Make these two corrections:

Fix the Typo: Change Lone Star Provisins Inc. to Lone Star Provisions Inc.
Fix the Quantity: Change the quantity for ONIONS YELLOW JBO from 5 to 50.

Click Save Correction.

Step 3: Complete the Flywheel

Now, go back to your terminal, and stop the server by pressing Ctrl+C. The script automatically triggers the final step of the flywheel: exporting your work. You'll see this output:

--- Server stopped. ---

--- Exporting approved corrections to fine-tuning format... ---
--- Data successfully exported to 'corrected_data.jsonl' ---

--- QUICKSTART COMPLETE ---

You did it. You just completed one full turn of the data flywheel.

The Payoff: The Perfect Training File

Open the examples/correction_deck_quickstart directory. You'll find a new file: corrected_data.jsonl. This is the prize. This is the tangible result of your work, captured and formatted perfectly for fine-tuning a modern AI model.

Let's look inside. It contains a single line of structured JSON:

{"contents": [{"role": "user", "parts": [{"fileData": {"mimeType": "image/jpeg", "fileUri": "/static/example_invoice.jpeg"}}, {"text": "Extract the key business data from the provided input."}]}, {"role": "model", "parts": [{"text": "{\"supplier_name\": \"Lone Star Provisions Inc.\", \"invoice_number\": \"785670\", \"invoice_date\": \"2025-08-20\", \"inventory_items\": [{\"item_name\": \"TAVERN HAM WH\", \"total_quantity\": 15.82, \"total_unit\": \"LB\", \"total_cost\": 87.8}, {\"item_name\": \"ONIONS YELLOW JBO\", \"total_quantity\": 50.0, \"total_unit\": \"LB\", \"total_cost\": 35.5}]}"}]}]}```

This might look complex, but it's the exact conversational format that models like Google's Gemino and OpenAI's GPT series expect for fine-tuning.

"role": "user": This is the prompt. It contains the input image (fileUri) and the instruction we gave the AI.
"role": "model": This is the perfect response. It contains the JSON string with your corrections applied.

We have successfully turned a few seconds of human effort into a high-quality, machine-readable training example. Now, imagine doing this for 100 corrections. Or 1,000. You are no longer just fixing errors; you are actively and efficiently building a dataset that will eliminate this entire class of errors in the next version of your model.

What's Next?

We've proven the core loop of the flywheel: Correct -> Capture -> Format for Training.

This is a powerful start, but it's an offline process. We waited for the AI to finish its batch, and then we corrected its work. But what if we could be more interactive? What if a pipeline could be running, encounter something it doesn't understand, and intelligently pause itself to ask a human for help in real time?

In the next article in this series, we'll build exactly that. We will construct a resilient, Human-in-the-Loop pipeline that knows when it's in trouble and isn't afraid to ask for clarification.

Your First AI Data Flywheel in Under 100 Lines of Python