MetaSynth - Diverse Synthetic Data Generation via Meta Prompting

https://arxiv.org/abs/2504.12563

Based on the sources, METASYNTH is a novel method for generating diverse synthetic data by employing meta-prompting and agentic scaffolding.

Here's a breakdown of its key aspects:

- Diversity Enhancement: The primary goal of MetaSynth is to address the issue of low diversity in synthetic data generated by Large Language Models (LLMs), which can negatively impact their downstream applicability. MetaSynth aims to achieve diversity through a process where a meta language model orchestrates multiple "expert" LLM agents to collaboratively generate data.
- Meta-Prompting Driven: MetaSynth leverages the concept of meta-prompting, where an LLM (the meta-LM) itself writes the prompts to guide other LLM agents in the data generation process. This approach has been shown to elicit more diverse and creative outputs compared to template-based prompting.
- Agentic Scaffolding: The meta-LM acts as an orchestrator in a centralised multi-agent system (MAS), overseeing communication between various specialised "expert" agents. These agents, such as "Seed Keyword Extraction Expert," "Domain Expert," "Summarizer Expert," and "Content Analyst Expert," have specific roles in the data generation and diversification process. The meta-LM dynamically composes the choice of agent and their instructions based on the sub-task.
- Conditional Instance Generation: MetaSynth incorporates Conditional Instance Generation, where the meta-LM maintains a memory of previously generated instances and categorises them to ensure that each newly synthesised instance (document or instruction) is distinct from all prior ones. This process involves expanding seed keywords with related terms and comparing new instances to existing ones using summaries generated by a "Summarizer Expert".
- MetaSynth-Instruct: This is a specific application of MetaSynth focused on generating and iteratively evolving complex instructions for instruction pre-training. Notably, MetaSynth-Instruct evolves instructions purely from synthetic documents generated by MetaSynth, without relying on human-written text. It also synthesises training data for fine-tuning encoder models, outperforming template-based prompting in this context.
- Workflow Example (Document Synthesis): The meta-LM consults experts to extract keywords from seed documents, instructs a domain expert to generate a document based on these keywords, summarises the document, and then uses a content analyst to assess its diversity compared to previous documents. If deemed insufficiently diverse, the meta-LM instructs a new domain expert with an enriched keyword set to rewrite the document from a fresh perspective.
- Diversity Measurement: MetaSynth's effectiveness in generating diverse data is evaluated using seven automated metrics, including the Task2Vec diversity coefficient, compression ratio, N-gram diversity, Remote Clique, Chamfer Distance, and Mean Inverse Frequency (MIF). Evaluations show that MetaSynth significantly improves data diversity compared to template-based prompting.
- Domain Adaptation: Experiments demonstrate that using only 25 million tokens of synthetic data generated with MetaSynth is sufficient to effectively adapt a well-trained LLM (Mistral-7B-v0.3) to specialised domains like Finance and Biomedicine without compromising general capabilities. Continual pre-training with MetaSynth data outperforms the base LLM and models trained on template-prompted data. Mixing real data with MetaSynth-generated data is often unnecessary when the synthetic data is sufficiently diverse.
- Comparison to Template Prompting: MetaSynth is contrasted with a strong baseline of template-based prompting, which uses static prompt templates with in-context examples of real data. MetaSynth consistently demonstrates higher diversity and better performance in domain adaptation compared to this baseline.
- Limitations: Despite its benefits, MetaSynth has limitations, including a significant inference cost due to the iterative refinement process and the need for substantial multi-threading capabilities. The agentic workflow can also be prone to breakdowns, and there are limitations on the length of synthesised documents. Furthermore, automated diversity metrics may not always perfectly align with human judgments, and biases in the underlying LLM can influence the generated data.

In summary, MetaSynth represents an advancement in synthetic data generation by using a meta-prompting approach with multiple expert agents to create more diverse data, which proves effective for domain adaptation and fine-tuning, although it comes with certain computational costs and potential limitations.

Видео MetaSynth - Diverse Synthetic Data Generation via Meta Prompting канала Denis Kropp