URL to Training Data

Python data preparation

A tool that turns web-page text into data files for AI training while preserving source and heading information.

The need to solve

Converting long web content into training data requires preserving source, heading and table context while making processing repeatable.

The pipeline prepares source-aware JSONL dialogues or CPT raw data, with batch processing, retries and processed-URL tracking.

I worked on scraping, cleaning, token/heading-aware chunking and JSONL generation in a Python pipeline, adding a raw-data mode without LLM calls.

Explore development details

For readers interested in the technical approach: workflows, technologies and implementation decisions.

Turning web pages into training data involves source preparation, context-preserving chunking and separate paths for the chosen generation mode.

Load URLs and processing history
Prepare batch input; processed-URL tracking helps control duplicate work.
Collect the page
The scraping stage retrieves source-page content.
Clean the content
Remove page noise and prepare text suitable for training data.
Chunk by headings and token limits
Split text while preserving section headings and the context of long tables.
Choose the generation mode
Structured examples use the Gemini path; CPT raw mode takes the output path without an LLM.
Generate examples with Gemini
Generate training examples from chunks using batch processing and retries.
Write JSONL with source metadata
Include the source URL and heading metadata alongside generated training examples.

Skip the LLM stage
Use cleaned source content without calling Gemini.
Write raw training data
Prepare raw text for CPT instead of generating structured examples.

Retry the failed work
Batch / retry mechanisms revisit a failed generation attempt.
Return the result to output
Pass successful results back to the output process with source metadata.

Source and heading context travel with the data.A schematic illustration of how the product works.

Worked on a web-scraping, cleaning and token/heading-aware chunking pipeline with Hugging Face tokenizers, preserving context when splitting long tables and lists.
Developed Gemini API generation of JSONL dialogue data with source/heading metadata, retry handling for failed generation and batch-processing workflows.
Added a CPT raw-data mode that bypasses LLM calls; configured processed-URL tracking and repeatable data preparation with Docker/Compose.