Close Menu
NCIJ Network NCIJ Network
    What's Hot

    EU chief diplomat Kallas blasts Russian peace demands as ‘absurd’ – POLITICO

    November 16, 2026

    UK has lost more than it gained from Brexit, Burnham expected to say

    November 16, 2026

    Democrats Will Wield Enormous Power in Statehouses. What Will They Do?

    November 16, 2026
    Facebook X (Twitter) Instagram
    Trending
    • EU chief diplomat Kallas blasts Russian peace demands as ‘absurd’ – POLITICO
    • UK has lost more than it gained from Brexit, Burnham expected to say
    • Democrats Will Wield Enormous Power in Statehouses. What Will They Do?
    • Meet Meta’s JiTTests: LLM-Generated Tests Designed to Fail Before Risky Code Lands
    • Wildfire Lawsuits Against Utilities Fuel a Debate Over Who Should Pay
    • Ed Miliband defends doing 2015 election interview with Russell Brand, saying he was ‘taking him on’ not endorsing him – UK politics live | Politics
    • Physicists use laser light to unlock a crystal’s hidden electrical powers
    • Rare flat-headed cat prefers elevated palms in Malaysian forest, says study
    • About
      • Our Team
      • Editorial Policy
      • Editorial Independence
      • International Support
    • Trust & Standards
      • AI Usage Policy
      • Conflict of Interest Policy
      • Corrections Policy
      • Ethics Policy
      • Fact-Checking Policy
      • Source Protection
    • Get Involved
      • Guide for Sources
      • Support Independent Journalism
    • Legal
      • Cookie Policy
      • Privacy Policy
      • Terms of Use
    Facebook X (Twitter) Instagram
    NCIJ Network NCIJ Network
    Monday, November 16
    • Home
    • World
    • Ai
    • Business
    • Politics
    • Health
    • Crypto
    • Science
    • Technology
    • Cybersecurity
    • Defense & Security
    • Economy
    • Energy
    • Europe
    • More
      • Fact Check
      • Investigations
      • Opinion & Analysis
      • Environment
    NCIJ Network NCIJ Network
    Home»Artificial Intelligence

    Meet Meta’s JiTTests: LLM-Generated Tests Designed to Fail Before Risky Code Lands

    NCIJ NETWNCIJ NETWORKBy NCIJ NETWNCIJ NETWORKNovember 16, 2026 Artificial Intelligence No Comments4 Mins Read
    Share
    Facebook Twitter LinkedIn Pinterest Email





    Meta has moved Catching Just-in-Time Tests (JiTTests) from research prototype into its production infrastructure. Meta JiTTests is an LLM-based system that writes executable tests designed to fail on a risky code change. A failing test is concrete evidence that a diff changed behavior. Meta team reports that JiTTests caught 74 privacy SEVs at the diff stage in the first half of 2026. The method is described in the FSE 2026 research paper ‘Just-in-Time Catching Test Generation at Meta‘.

    TL;DR

    • Scale: Runs on Meta backend code with hundreds of millions of lines. The research paper analyzes 22,126 generated tests.
    • Runs on: Meta’s internal infrastructure, overnight on spare capacity. Judge models in the paper: Llama3.3-70B, Gemini 3 Pro, Claude Sonnet 4. Not publicly available.
    • Performance: Engineers report a 60% true-positive rate at full scale (self-reported).
    • Best: 74 privacy SEVs caught before landing in H1 2026; privacy SEV occurrence down 20%.
    • Worst: The initial deployment confirmed only 8 of 41 reported catches (19.5%).
    • Bottom line: Best: executable proof of a bug before merge. Worst: internal only, with self-reported numbers.

    What are Catching JiTTests?

    Catching JiTTests are LLM-generated tests that should fail when a diff introduces a bug. Traditional hardening tests pass when written and guard against future regressions. Catching tests target the change under review, before it lands. Meta formalized the details in its 2025 paper ‘Harden and Catch’.

    Most AI bug tools act like faster human reviewers. They read the diff and leave comments. JiTTests instead execute code and show a concrete behavioral difference. Meta team concedes this takes more computational steps. It argues the approach catches bug classes that LLM or agent review can miss.

    How does the JiTTests pipeline work?

    The production pipeline has 5 stages:

    1. Targeting: Runs overnight on spare capacity. It selects the previous day’s risky diffs, using a mechanism similar to Meta’s Diff Risk Score.
    2. Test generation: 2 diff-aware workflows produce candidate tests (explained below).
    3. LLM-as-Judge ensemble: Several LLMs score each failing test using the test code, failure trace, diff and inferred intent.
    4. RubFake: The Rule-Based False-Positive Killing Environment adds deterministic checks. Its rules flag patterns like reflection, broken mocks and not-implemented exceptions.
    5. Human-on-the-loop: The system decides whether to contact an engineer, based on the combined ranking.

    Dodgy diff vs intent-aware: what is the difference?

    The dodgy diff workflow assumes every diff is buggy. It treats the diff as a mutant of its parent and generates tests that tell them apart.

    The intent-aware workflow first infers what the diff should do from its code, title and summary. It lists risks, builds a mutant for each, and writes tests to kill them. Tests that then fail on the real diff become ‘weak catches.’

    What do the benchmarks show?

    The research team analyzes 22,126 generated tests. Diff-aware methods produced 4x more candidate catches than hardening tests. They produced 20x more than tests that failed by coincidence.

    The intent-aware workflow produced weak catches in 6.4% of its tests and on 7.9% of diffs. Dodgy diff scored 2.5% and 4.0%. Meta’s earlier ACH mutation-guided hardening scored 0.8% and 1.6%. The rule-based and LLM assessors cut human review load by 70%.

    Code designed fail JiTTests lands LLMGenerated Meet Metas risky Tests
    NCIJ NETWNCIJ NETWORK
    • Website

    Keep Reading

    Cancer ‘sponge’ tests for bad indigestion could be offered to more people

    Bland Launches Tofu-1, a Full-Duplex Speech-to-Speech Model for Calls

    Cmpnd Releases dspy-wsp: Weight-Space Programming Comes to DSPy

    Stanford Research Open-Sources Memorilla, a Latent Memory Module for Frozen LLMs

    Vercel Releases UI SDK: 50+ UI Behaviors for AI Coding Agents

    Trump to Meet With Putin and Xi in China, With Nuclear Proliferation on the Agenda

    Add A Comment
    Leave A Reply Cancel Reply

    Editors Picks

    EU chief diplomat Kallas blasts Russian peace demands as ‘absurd’ – POLITICO

    November 16, 2026

    UK has lost more than it gained from Brexit, Burnham expected to say

    November 16, 2026

    Democrats Will Wield Enormous Power in Statehouses. What Will They Do?

    November 16, 2026

    Meet Meta’s JiTTests: LLM-Generated Tests Designed to Fail Before Risky Code Lands

    November 16, 2026
    Latest Posts

    Are lemons ‘manmade’ fruits that don’t occur naturally? Here’s the bitter truth

    August 20, 2026

    There is nothing scientific about The Great ADHD Myth? ‘documentary’ | Gina Rippon

    August 20, 2026

    Despite SafeSport Ban, Girls Volleyball Coach Ryan Richardson Continues to Work With Kids — ProPublica

    August 20, 2026

    Subscribe to News

    Get the latest sports news from NewsSite about world, sports and politics.

    NCIJ Network is an independent digital news platform delivering trusted investigative journalism, European and global news, in-depth analysis, and fact-based reporting with accuracy, transparency, and integrity.

    Facebook X (Twitter) Instagram Pinterest YouTube

    EU chief diplomat Kallas blasts Russian peace demands as ‘absurd’ – POLITICO

    November 16, 2026

    UK has lost more than it gained from Brexit, Burnham expected to say

    November 16, 2026

    Democrats Will Wield Enormous Power in Statehouses. What Will They Do?

    November 16, 2026

    Subscribe to Updates

    Get the latest creative news from FooBar about art, design and business.

    Type above and press Enter to search. Press Esc to cancel.