The Thing AI Does Not Know It Is Doing

Image on Unsplash

There is a problem with training AI on AI-generated data. The problem is that it works too well for too long, and then it stops working altogether.

A researcher named Ilia Shumailov at Google DeepMind published a paper showing what happens when you feed an AI model its own output repeatedly. In the beginning, the output looks fine, but after a few iterations, it starts to degrade. After five or six generations, trained purely on its own data, the model converges to a mean and stops producing anything useful (Shumailov et al., 2024). Researchers called it "model collapse." It is also sometimes called "autophagy," after the process where a cell consumes itself.

An AI model trained on human-generated data learns the full distribution of that data, including the rare cases and the outliers. When you train a new model on that first model's output, you lose the rarest cases first. The new model never saw them, so it cannot reproduce them (IBM, 2024). Train a third model on the second model's output, and you lose more of the tails. After enough iterations, the distribution converges to the middle, and the model produces only bland averages (Shumailov et al., 2024). Any errors in generation get amplified in the next iteration, so mistakes compound.

As AI-generated content floods the internet, future training datasets will inevitably include more synthetic data. Unless it gets actively filtered out, we will be training the next generation of models on the previous generation's mistakes. 

To get more clarity, consider what happens in a practical scenario. A company trains an AI to write customer support responses. That AI's outputs become part of the training data for the next version. The second version learns not just from the first version's good responses but also from its mistakes and biases. The third version compounds those errors further. After a few cycles, the model is producing nonsense because it has drifted so far from human language that it is essentially hallucinating (Seaton Tongue, 2025).

This problem is especially significant in a domain where AI is being deployed recursively. For example, medical AI trained on outputs from previous medical AI, legal AI trained on previous models' legal summaries, and educational AI trained on past AI's explanations. Each cycle, the connection to ground truth gets fainter.

Companies are well aware of this situation. The mitigation strategies are all about maintaining "data provenance," meaning tracking where data came from and filtering out synthetic content before it contaminates the training pipeline (Ada Lovelace Institute, 2025). Some companies are hiring people specifically to curate AI-generated data, ranking which synthetic outputs are worth keeping and which should be discarded. Others are investing heavily in fresh, human-generated data pipelines to continuously introduce real-world signals back into training.

But here is the hard part. The scale of AI deployment now means that human curation cannot keep pace. You cannot hire enough people to review and filter the volume of synthetic data being generated. The problem grows faster than the solution (Ada Lovelace Institute, 2025).

The even harder part is that this creates an incentive for companies to cut corners. If you need fresh, human-generated data to prevent model collapse, you have two choices. Either you invest in massive data curation infrastructure and hiring, or you just... do not worry about it and hope the collapse takes long enough that no one notices (Mixflow, 2025). Given that most companies are focused on shipping fast and iterating quickly, the financial pressure makes the second option look much more tempting (New York Times, 2024).

What researchers are still unsure about is how fast this problem will actually manifest in production. In controlled experiments, models collapse visibly, but in the real world, where training data is a blend of human and synthetic sources, the degradation might be gradual enough to hide for a while. You might deploy a model that is 5% worse than it should be. Six months later, it would be 15% worse. A year later, it might just be unusable. But by then, the company has already shipped it to thousands of customers (Winn Solutions, 2025).

To tackle this problem, fresh human-generated data must flow into training pipelines continuously, data curation must become part of standard practice, and models need to be monitored continuously for signs of degradation, not just at deployment but over time. The alternative is a world where AI systems keep getting worse at their jobs because they trained on other AI systems that were quietly getting worse.

What makes this situation unique is that unlike other AI problems, this one does not announce itself. The scary part is that a model collapse does not throw an error message, but just gradually stops working. By the time it’s noticed, the model has already been in production for months.