Meta's AI Writes 22,000 Tests — Then Deletes Them All - JiTTesting

Meta built an AI that writes 22,000 tests, finds bugs humans miss, and then deletes every single test. On purpose.

It's called JiTTests (Just-in-Time Tests) — and it catches 4x more bugs than traditional testing while reducing human review by 70%.

In this video, I break down exactly how it works — the 6-step pipeline, the mutation testing strategy, and the peer-reviewed results from FSE 2025.

⏱ TIMESTAMPS
0:00 — The problem with traditional testing
1:02 — Why your tests are lying to you
2:07 — Test debt at scale
2:33 — Meta's different approach: JiTTests
2:55 — Step 1: New code lands
3:04 — Step 2: LLM infers intent
3:23 — Step 3: Create mutants
3:40 — Step 4: Generate catching tests
4:00 — Step 5: Assessors filter results
4:24 — Step 6: Engineer gets notified (then test is deleted)
4:52 — The evidence (peer-reviewed numbers)
5:48 — 15% vs 2.4% — Targeted vs Coverage-only
6:11 — The counterintuitive finding
6:38 — Code walkthrough: Old Way vs New Way
7:25 — Five problems killed in one stroke
7:44 — Should you stop writing tests?
8:09 — Paper and resources

📄 RESOURCES
Research Paper: https://arxiv.org/abs/2601.22832
Meta Engineering Blog: https://engineering.fb.com/2025/02/05/security/revolutionizing-software-testing-llm-powered-bug-catchers-meta-ach/

📊 KEY STATS FROM THE PAPER
- 4x more bugs caught vs traditional hardening tests
- 20x more effective vs coincidental test failures
- 70% reduction in human review load
- 22,126 tests analyzed across 7 Meta platforms
- 15% mutant kill rate (ACH) vs 2.4% (TestGen-LLM)
- 51% of generated tests also raised code coverage

🔗 CONNECT
YouTube: https://www.youtube.com/@AyyazTech
Subscribe for weekly deep dives on AI, automation, and software engineering.

This video covers Meta's JiTTests system for automated test generation using large language models (LLMs). The system uses mutation testing, intent inference, and an ensemble of rule-based and LLM-based assessors to find real bugs in production code — then discards every test after use. Zero maintenance. Zero flaky test debt.

Presented at FSE 2025 (Foundations of Software Engineering), Trondheim, Norway.

#Meta #AITesting #SoftwareEngineering #JiTTest #MutationTesting #LLM #AI #CodingAutomation #DevTools #MetaEngineering #SoftwareTesting #UnitTesting #MachineLearning #AIForDevelopers #TestAutomation #CodeQuality #Programming

Видео Meta's AI Writes 22,000 Tests — Then Deletes Them All - JiTTesting канала AyyazTech