Scaling Synthetic Data ;

Most small models are bottlenecked not by architecture, but by the quality of the data they see at pretraining. By using LLaMA-3 to rephrase raw web text into clean, structured batches before any training begins, the resulting synthetic corpus is far denser in signal ; setting up a compact model to punch well above its weight class.

The pipeline has two discrete stages before any model work begins. Raw web text is streamed in manageable chunks and passed to LLaMA-3, which rewrites each batch into a clean, structured Q/A format. The resulting synthetic corpus is saved as numbered batch files ; ready for downstream use with no further transformation required.

Background ; The BeyondWeb Result

The DatalogyAI team's BeyondWeb paper demonstrated that language models trained on synthetic datasets can be up to 6× more efficient than those trained solely on raw web text. Their approach focused on building synthetic corpora as a drop-in replacement for standard pretraining data ; same model architecture, markedly better data quality.

This project takes that insight one step earlier in the stack. Instead of swapping the corpus after the fact, the raw text is rephrased by a large model at the preprocessing stage ; before any vocabulary or tokenizer decisions are made. The synthetic signal is injected at the very root of the pipeline, shaping everything downstream.

The Synthesis Pipeline

The pipeline is implemented as a streaming batch processor. OpenWebText is read in compressed .jsonl.gz format and divided into chunks of 50–500 lines. Each chunk is passed as a prompt to a locally running LLaMA-3 instance, which rewrites the raw passages into a structured Q/A format. The refined output is written out as sequentially numbered .jsonl batch files.

Each refined entry follows the schema {"q": "…", "a": "…"}. The question field captures the topic or intent extracted from the raw passage; the answer field contains the rephrased, structured explanation produced by LLaMA-3. This format is deliberate ; it imposes consistent structure that a smaller model can absorb far more efficiently than the inconsistently structured prose characteristic of raw web text.

Output Format & Structure

The repository organises output into two directories. The raw/ folder holds the original OpenWebText files unchanged. The refined/ folder contains the LLaMA-rephrased output in sequentially numbered batch files ; each one self-contained and independently resumable. Processing can be paused and restarted at any batch boundary without re-processing earlier output.

What Rephrasing Actually Does

Noise removal

Raw OpenWebText contains HTML residue, broken sentence fragments, promotional boilerplate, and encoding artifacts inherited from Common Crawl scraping. LLaMA-3 rephrasing strips these implicitly ; the model produces grammatically coherent, well-formed prose regardless of how malformed the input passage was.

Structure imposition

The Q/A output format imposes a consistent structure across the entire corpus. Every entry has a clear topic (the question) and a focused explanation (the answer). This is fundamentally different from the stream-of-consciousness structure of most web prose ; and it is a structure that models learn to reproduce and reason within far more reliably.

Information density

LLaMA-3 compresses and distils each raw passage when rephrasing. Redundant or low-information sentences are dropped. The rephrased answer conveys the core content of the original passage in fewer tokens ; meaning the corpus contains more distinct concepts per training token than the equivalent raw web text.

---

Technical Notes

Batch size selection. Batches of 50–500 lines balance context length against LLaMA-3's effective rephrasing coherence. Very small batches lose the inter-sentence context needed for well-formed Q/A output; very large batches exceed the model's effective attention window. 200-line batches are the practical sweet spot for most OpenWebText passages.

GPU requirements. LLaMA-3 rephrasing requires a minimum of 15 GB VRAM. Google Colab T4 (16 GB) is sufficient for smaller batch sizes; A100 access is recommended for production-scale corpus generation. The preprocessing script (dataset.py) runs locally with any compatible GPU and does not require Colab specifically ; Colab is used here for convenience and storage mounting.

Resumability. Each batch file is written atomically before the next batch begins. If the pipeline is interrupted, it can be restarted from the last successfully written batch with no data loss and no re-processing of earlier output. The sequential naming convention (batch_data00001.jsonl, batch_data00002.jsonl, …) makes gap detection straightforward.

Output validation. Each line in a refined batch file should parse as valid JSON with non-empty q and a fields. The repository includes sample output across qa_batch_00001.txt through qa_batch_00019.txt ; useful for inspecting rephrasing quality across a range of source passages before committing to full-corpus processing.

---