Close Menu
NCIJ Network NCIJ Network
    What's Hot

    For decades, we were told to avoid full-fat dairy. New research says otherwise

    November 7, 2026

    A US Senator’s Push to Fast-Track Pipelines Would Benefit His Former Company

    November 7, 2026

    U.S. Midterms Results See Slew of Younger Members Enter the Senate

    November 7, 2026
    Facebook X (Twitter) Instagram
    Trending
    • For decades, we were told to avoid full-fat dairy. New research says otherwise
    • A US Senator’s Push to Fast-Track Pipelines Would Benefit His Former Company
    • U.S. Midterms Results See Slew of Younger Members Enter the Senate
    • New wave of bomb threats hit European airlines
    • Voters in several states deliver wins for ballot measures fighting money in politics • OpenSecrets
    • Russell Brand has finally faced justice. But other abusers in comedy are still hiding in plain sight | Sarah Solemani
    • Wright’s Misleading Claims About Electricity Prices in Red vs. Blue States
    • Two dead after bus crashes in France, officials say
    • About
      • Our Team
      • Editorial Policy
      • Editorial Independence
      • International Support
    • Trust & Standards
      • AI Usage Policy
      • Conflict of Interest Policy
      • Corrections Policy
      • Ethics Policy
      • Fact-Checking Policy
      • Source Protection
    • Get Involved
      • Guide for Sources
      • Support Independent Journalism
    • Legal
      • Cookie Policy
      • Privacy Policy
      • Terms of Use
    Facebook X (Twitter) Instagram
    NCIJ Network NCIJ Network
    Saturday, November 7
    • Home
    • World
    • Ai
    • Business
    • Politics
    • Health
    • Crypto
    • Science
    • Technology
    • Cybersecurity
    • Defense & Security
    • Economy
    • Energy
    • Europe
    • More
      • Fact Check
      • Investigations
      • Opinion & Analysis
      • Environment
    NCIJ Network NCIJ Network
    Home»Artificial Intelligence

    Zyphra AI Introduces Zyphra Hierarchical Attention (ZHA): A Path to True Subquadratic Attention

    NCIJ NETWNCIJ NETWORKBy NCIJ NETWNCIJ NETWORKNovember 3, 2026 Artificial Intelligence No Comments5 Mins Read
    Share
    Facebook Twitter LinkedIn Pinterest Email

    Zyphra has released Zyphra Hierarchical Attention (ZHA), a sparse attention method built around a tree-structured indexer. ZHA targets a known gap in sparse attention: the indexer that picks tokens still scans the whole context. Zyphra’s technical paper shows ZHA can be distilled into frozen pretrained models with about 4B tokens. On Qwen3.6-35B-A3B it stays close to dense attention on reasoning, agentic coding and 256K retrieval benchmarks.

    TL;DR

    • Size: It is a method that adds 81,920 trainable parameters to Qwen3.6-35B-A3B (about 0.00024%). Tested up to 256K context.
    • Runs on: Retrofits frozen pretrained models after a 4B-token KL warmup at 32K context. Speedups come from a modeled H100 BF16 roofline. No optimized kernels released. An FP4 router variant was also tested.
    • Performance: Within a few points of dense attention on 7 reasoning, coding and retrieval benchmarks.
    • Best: LongBench v2 hits 63.4% (deep tree) vs 58.0% dense, using about 1.1% of keys.
    • Worst: On NoPE-based Nemotron-3.5-Lightning-30B-A3B, deep-tree RULER-13 drops 25.3 points below dense.

    What is Zyphra Hierarchical Attention (ZHA)?

    ZHA is a sparse attention method from Zyphra that finds relevant keys by searching a tree. It keeps the full KV cache. The tree only decides which cached keys each query reads.

    Fixed-state methods like SSMs and sliding windows saturate at long contexts. Fine-grained sparse methods like DeepSeek Sparse Attention score every past token with an indexer. The final attention is sparse, but the indexer still grows with the full context.

    Near 1M tokens, cited work shows indexing takes 60 to 90% of layer time. ZHA makes the search itself sparse.

    How does ZHA’s tree indexer work?

    Building the tree

    ZHA splits keys into contiguous chunks and compresses each chunk into a summary key. Summaries are grouped and compressed again, level by level. With chunk size 4, 6 levels cover 4,096 tokens per top node.

    Summaries come from a small set of learned reference queries. Each reference query gives a tangent-plane estimate of a chunk’s log attention mass. The max over these planes approximates how much attention any future query would give the chunk.

    How does top-p routing pick tokens?

    At decode time, a query scores the top-level nodes. It keeps the smallest set holding a fraction p of estimated attention mass. It then expands only those children, repeating until it reaches raw KV entries.

    Top-p makes the budget adaptive. A single-fact lookup touches few branches. A query that needs many scattered facts expands more. The first token is always kept as an attention sink.

    How does ZHA perform against dense attention and MSA?

    Zyphra team converted Qwen3.6-35B-A3B with 2 trees: deep (chunk 4, 6 levels) and wide (chunk 256, 1 level). Both use 16 reference queries and p = 0.95. It also trained a MiniMax Sparse Attention (MSA) indexer on 40B tokens, 10x ZHA’s budget.

    Benchmark (Qwen3.6-35B-A3B) Dense ZHA deep ZHA wide MSA
    AIME 2026 93.1 91.2 90.8 91.7
    SWE-bench Verified 67.4 67.4 68.4 63.6
    Terminal-Bench 2.1 49.9 46.7 53.3 46.1
    MMLU-Pro 84.9 85.3 85.2 85.3
    GPQA Diamond 84.8 85.0 85.0 83.1
    RULER-13 (256K) 96.0 93.3 96.2 77.1
    LongBench v2 (thinking) 58.0 63.4 62.4 49.3

    Source: arXiv 2610.42112, Figure 5. All numbers self-reported by Zyphra.

    The biggest gap is on retrieval. MSA drops to 77.1% on RULER-13, while both ZHA trees stay above 93%. On LongBench v2, the deep tree and MSA read similar key shares (1.1% vs 1%). The deep tree scores 14.1 points higher.

    Across 28 tree geometries, RULER accuracy held steady when attending to 10 to 30% of keys. At fixed compression, deeper trees kept accuracy at higher sparsity. A 4-parameter model predicts RULER accuracy from geometry with 1.44-point RMSE.

    attention Hierarchical Introduces path Subquadratic true ZHA Zyphra
    NCIJ NETWNCIJ NETWORK
    • Website

    Keep Reading

    Zyphra ZYFA: FP8 FlashAttention for AMD MI355X, Up to 1.76x Faster

    Thinking Machines Releases Inkling 2 Small, a 594B MoE With 17B Active

    Franklin Templeton uses OpenAI to support investment research

    Baidu Releases ERNIE 5.5 Flash Base: A 280B MoE Base Model With 14B Active Parameters

    Merge builds product and business tools with OpenAI technology

    Google Releases Gemini Omni 1.2 Flash: 15-Second AI Video at $0.07 per Second

    Add A Comment
    Leave A Reply Cancel Reply

    Editors Picks

    For decades, we were told to avoid full-fat dairy. New research says otherwise

    November 7, 2026

    A US Senator’s Push to Fast-Track Pipelines Would Benefit His Former Company

    November 7, 2026

    U.S. Midterms Results See Slew of Younger Members Enter the Senate

    November 7, 2026

    New wave of bomb threats hit European airlines

    November 7, 2026
    Latest Posts

    Microsoft confirms outage affecting search in Microsoft 365 apps

    August 18, 2026

    Alvys launches AI agents for freight TMS workflows

    August 18, 2026

    Andy Burnham says he’s not embarrassed about hoax message exchange

    August 18, 2026

    Subscribe to News

    Get the latest sports news from NewsSite about world, sports and politics.

    NCIJ Network is an independent digital news platform delivering trusted investigative journalism, European and global news, in-depth analysis, and fact-based reporting with accuracy, transparency, and integrity.

    Facebook X (Twitter) Instagram Pinterest YouTube

    For decades, we were told to avoid full-fat dairy. New research says otherwise

    November 7, 2026

    A US Senator’s Push to Fast-Track Pipelines Would Benefit His Former Company

    November 7, 2026

    U.S. Midterms Results See Slew of Younger Members Enter the Senate

    November 7, 2026

    Subscribe to Updates

    Get the latest creative news from FooBar about art, design and business.

    Type above and press Enter to search. Press Esc to cancel.