Apr 7th ~ Apr 14th
#103 Latest AI Research Explained Simply

🗞️ Industry News in 1 Line

1. ♥ 5.5k Following its initial debut last month, MiniMax has now made the weights for MiniMax M2.7 openly available to the public under a restrictive license that limits commercial use and derivative works. The model shows SoTA performance in software engineering and command-line tasks, achieving a 56.22% on SWE-Pro and 57.0% on Terminal Bench 2. You can try it today on Hugging Face.

   

    

2. ♥ 10k Meta Superintelligence Lab (MSL) recently released their first ever model Muse Spark, a natively multimodal reasoning model that features a "contemplating mode" for complex, parallel agent orchestration. The model serves as the backbone for Meta AI's new deep reasoning and shopping capabilities, demonstrating performance competitive with other leading frontier models like GPT Pro and Gemini Deep Think. You can try it on Meta AI Platform for free.

   

3. ♥ 10k Z.ai has launched GLM-5.1, an open-source model that currently leads the open-weight rankings with a state-of-the-art 58.4 score on SWE-Bench Pro. The model is specifically optimized for long-horizon agentic tasks, capable of running autonomously for up to eight hours to solve complex engineering and database optimization problems. You can try it on Hugging Face or via the API.

   

Thunder Compute: The cheapest cloud GPU

H100 @ $1.38/GPU/hr!!!

Thunder Compute has one of the cheapest cloud GPUs for developers. They offer on-demand GPU cloud instances in enterprise-grade data centers for a fraction of the price of competitors.

With on-demand H100 sitting at $1.38/GPU/hr, you’d get best-in-class reliability and networking, compared to other competitors that offer at least $4/GPU/hr.

They have additional features like:

• VSCode extension and CLI which let you connect to instances without SSH config.

• Snapshots to save instance state and restore on any number of instances

• Templates for ComfyUI, Ollama, Unsloth Studio, and more

• $20 of free credit for students

Create a GPU instance now!

Advertise with The AI Timeline!

In-Place Test-Time Training

Feng et al. [ByteDance Seed, Peking University]

♥ 1k   Test time training  

Current LLMs follow a strict “train then deploy” rule, meaning once they are released, their underlying knowledge is completely frozen. They cannot adjust their internal wiring to absorb continuous streams of new information in real time.

While scientists have tried a workaround called Test-Time Training (allowing a tiny fraction of the model to update on the fly) it historically required changing the system's architecture and undertaking a massive, costly retraining process.

To overcome this, researchers designed a brilliant upgrade called In-Place Test-Time Training. Instead of bolting on brand-new components to the system, they realized they could simply repurpose existing ones. They targeted ubiquitous processing centers inside the model, known as MLP blocks, which normally store the static knowledge acquired during initial training.

Efficiency analysis of In-Place TTT.

The team unlocked the final layer of these blocks to act as a flexible, fast-updating memory. Because this elegant drop-in design leaves the original architecture perfectly intact, it preserves the system's foundational knowledge while seamlessly granting it the ability to adapt as it processes new data.

The team paired this structural cleverness with an efficient engine that updates the memory in scalable chunks, avoiding heavy computing bottlenecks. Additionally, they aligned this real-time learning with the system's natural goal of predicting the next word.

TriAttention: Efficient Long Reasoning with Trigonometric KV Compression

Mao et al. [MIT, NVIDIA, ZJU]

♥ 1.1K   Attention   bycloud’s pick  

Modern LLMs generate incredibly long chains of thought to solve logic puzzles, but this creates a massive memory bottleneck. Every thought the model holds onto is stored in a cache, and as reasoning grows, this memory gets completely overwhelmed. Until now, the best solution was deleting older memories based on recent observations. However, just like someone forgetting the beginning of a math problem midway through, this causes the system to lose critical context.

Q/K concentration and its implications for attention.

To solve this, researchers looked deeper into the architecture of the model, the raw space before the system applies rotational math to track word positions. Here, they noticed a beautifully consistent pattern. The specific components the model uses to match questions with answers naturally cluster around stable centers, regardless of the actual text.

Performance comparison on Qwen3-8B.

Because these centers never wander, researchers realized they could predict exactly which memories the model would need in the future based purely on distance. By using a mathematical curve known as a trigonometric series, they mapped out the natural distance preferences of the model to perfectly score memory importance.

Performance trade-offs on AIME25 (Qwen3-8B)

Building on this insight, the team designed a system named TriAttention. Instead of guessing which past thoughts matter based on recent windows, it calculates the true future importance of every piece of data.

Researchers demonstrated that their method perfectly matches the reasoning accuracy of an uncompressed model, yet it slashes memory usage by nearly eleven times and runs two and a half times faster. This shortcut frees up enough memory that advanced AI can now run smoothly on a single consumer graphics card.

Neural Computers

Zhuge et al. [Meta AI, KAUST]

♥ 1.2K   LLM Computer Use  

Think about how computers operate today: the hardware, the operating system, the applications, and the AI tools navigating them are all completely separate pieces. Researchers are trying to solve this fundamental fragmentation by asking a bold question: what if a single AI model could actually be the entire computer?

Currently, traditional computers execute explicit programs, AI agents click around those programs from the outside, and predictive models guess what a screen should look like next. To bridge this gap, scientists are building Neural Computers.

Instead of relying on a rigid, traditional stack of physical processors, memory banks, and standard code, this approach combines computation, working memory, and user inputs into one continuously learning neural network.

To test this ambitious idea, the team built early prototypes using advanced video generation technology, observing how the AI handled both text-heavy command-line terminals and traditional visual desktops. By feeding the system streams of user actions, text prompts, and starting screen visuals, the network's internal state essentially became the computer’s processor and RAM.

The researchers discovered that these neural systems can intuitively learn the physical rules of our digital worlds from observation alone. The models successfully rendered fast-scrolling text, aligned precise cursor movements, and perfectly simulated short-term desktop responses like hovering over menus or clicking buttons.

While they still require careful instruction to solve complex math or maintain focus over long periods, this fascinating discovery proves the foundational building blocks of a completely neural computer are already within our reach.

Neural Computer: A New Machine Form Is Emerging

A research essay on Neural Computer: how it differs from agents, world models, and conventional computers; what runtime and CNC would mean; what current prototypes already show; and how software and hardware might change.

METAUTO.ai • Mingchen Zhuge

Interleaved Head Attention

Duvvuri et al. [Meta, UT Austin, UC Berkeley, Harvard University, MIT]

♥ 487   LLM Attention  

When modern artificial intelligence reads a prompt, it relies on independent processors called "attention heads." Think of these heads as a team of isolated researchers, where each person analyzes a document in a sealed room without speaking to their colleagues. While this works for simple facts, researchers realized it creates a massive bottleneck for multi-step reasoning.

If you ask an AI where the author of a specific book was born, the system must first identify the author, then find their birthplace. Because these processors cannot communicate during their computation, standard models are forced to rely on an inefficient, ever-growing number of isolated heads to piece together these chains of logic.

Overview of Interleaved Head Attention (IHA).

To solve this, researchers developed a brilliant new approach called Interleaved Head Attention. Instead of forcing processors to work in isolation, the system constructs "pseudo-heads" that actively blend information from the entire team before analyzing the text. By mixing their perspectives together, these virtual processors can suddenly share context.

Rather than learning just one pattern per head, this collaborative mixing allows a single processor to recognize multiple complex patterns simultaneously. It literally multiplies the model's ability to connect the dots while capturing complex, overlapping relationships.

RULER long-context results after 64k fine-tuning.

The researchers proved that this technique requires vastly less underlying code to achieve the same reasoning power as older models. When put to the test, this cross-mixing improved the system's ability to retrieve multiple facts hidden in extremely long documents by up to twenty percent.

Furthermore, when fine-tuned for complex logic, it boosted accuracy on advanced math problem-solving benchmarks by nearly six percent.

Twitter tweet

Apr 1st ~ Apr 7th
#102 Latest AI Research Explained Simply

🗞️ Industry News in 1 Line

1. ♥ 2k Arcee.ai has introduced Trinity-Large-Thinking, a new open-weights language model designed specifically for complex agent workflows and long-horizon tool use. This model offers improved multi-turn coherence, stable instruction following, and the efficiency required for production-scale deployments. Try the model out for yourself via API on OpenRouter or explore it on Hugging Face.

   

2. ♥ 7.2k Google has introduced Gemma 4, a new family of open models released under the permissive Apache 2.0 license. The lineup features four distinct sizes tailored for various deployment needs, including a 31B dense model for raw performance, a 26B Mixture-of-Experts (MoE) variant for low latency, and efficient 2B and 4B options optimized for edge devices. You can download the model weights and start fine-tuning for specific tasks today by checking it out on Hugging Face or Ollama.

   

3. ♥ 5.8k Z.ai has launched GLM-5V-Turbo, a native vision coding model capable of translating multimodal inputs (such as design drafts, videos, and UI screenshots) directly into executable code. Backed by a new CogViT visual encoder and collaborative reinforcement learning across over 30 task types.

   

4. ♥ 4.6k Alibaba's Qwen team has introduced Qwen3.5-Omni, a new family of native multimodal models designed to seamlessly process and integrate text, image, audio, and video inputs. It is available in Plus, Flash, and Light variants, the models feature massive context capacities capable of handling up to 10 hours of audio, along with advanced real-time capabilities like emotion-controlled voice interaction and "Audio-Visual Vibe Coding." You can explore the real-time voice demo on Hugging Face or access the models via the Alibaba Cloud API.

   

Intuitive AI Academy - NEW MoE Chapter!

My latest project: Intuitive AI Academy has the perfect starting point for you! We focus on building your intuition to understand LLMs, from transformer components, to post-training logic. All in one place.

We just added a new chapter on MoE, that goes through the history, the key techniques, and the current state of MoE that frontier model uses. With over 10,000 words written!

We currently have an early bird offer, where you would get 40% off on the yearly plan for our early users.

Use code: TIMELINE

Check Out Intuitive AI Academy

Advertise with The AI Timeline!

HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention

Xu et al.

♥ 256   Attention  

When we ask an LLM to analyze a massive document, it uses a clever trick called sparse attention, where it only focuses on the most relevant words instead of processing everything equally. The internal tool that selects these important words still has to scan every single word in the document one by one.

Comparison of the DSA token-wise indexer (left) and HISA hierarchical block-level coarse filter followed by token-level refinement (right)

To solve this, researchers developed an elegant workaround called Hierarchical Indexed Sparse Attention. Instead of a flat, word-by-word scan, this new method uses a brilliant two-step strategy. First, it chunks the massive document into larger blocks and looks at a quick summary of each block to instantly filter out the irrelevant sections.

Latency comparison of the indexer kernel between the original DSA

Once the bulk of the text is safely discarded, the system zooms in on the surviving blocks, scanning only those specific words to find the exact information the AI needs. It is much like skimming the chapter titles of a textbook to find the right section before reading the actual sentences.

By rewriting this search path, researchers managed to speed up the scanning process by nearly four times for exceptionally long texts, all while perfectly preserving the model's accuracy.

Embarrassingly Simple Self-Distillation Improves Code Generation

Zhang et al. [Apple]

♥ 1.5k   Distillation   bycloud’s pick  

Teaching an AI to write better code requires expensive human examples, a smarter teacher model, or highly complex reward systems that verify every single line of code. This heavy reliance on outside help has become a massive bottleneck in AI development. Researchers recently asked a fascinating question to address this: could a model pull itself up by its bootstraps, improving its own capabilities using absolutely nothing but its own raw, unverified outputs?

Simple self-distillation (SSD) is embarrassingly simple, yet yields substantial LiveCodeBench v6 gains across five models spanning two families, three scales, with both instruct and thinking variants.

This can be done through a method called simple self-distillation. Researchers asked the AI to generate solutions to coding prompts (without running test cases or checking if the code actually worked) and then retrained the AI on those exact responses. Remarkably, this caused a massive leap in performance across several different models, with the absolute biggest gains seen on the hardest coding challenges. Rather than just memorizing a single dominant way to solve a problem, the AI actually preserved its ability to explore multiple viable solution paths.

SSD improves every evaluated model on LiveCodeBench, with the largest gains on medium and hard problems

To understand why this works, researchers uncovered a fascinating tug-of-war inside the model called the precision-exploration conflict. When writing code, an AI encounters "locks" (moments requiring rigid exactness with zero ambiguity) and "forks", i.e. moments requiring creative exploration to choose a problem-solving approach.

Training and evaluation temperatures compose through a broad effective-temperature band, while truncation raises the achievable pass@1 within that band.

Normally, adjusting an AI's generation settings forces a strict compromise: making it flexible enough to navigate creative forks causes it to make sloppy, distracting errors at the rigid locks.

Path-Constrained Mixture-of-Experts

Gu et al. [Apple, Google]

♥ 356   MoE  

In "Mixture-of-Experts" architectures, instead of activating the entire AI for every word, this design acts like a traffic system, routing each piece of information only to specialized mini-programs, or experts. But there is a catch.

Historically, these models make independent routing decisions at every single layer, creating an astronomical number of possible pathways. Because the vast majority of these paths are never explored during training, researchers realized this scattered approach represents a massive inefficiency. They wondered if they could guide information along more intentional, organized routes.

Spectrum of routing constraints in MoE architectures.

When peering inside these systems, researchers discovered something fascinating: language naturally organizes itself. Even without strict guidance, words cluster into a tiny fraction of specific pathways based on their linguistic purpose, with dedicated paths emerging for things like punctuation, names, or action verbs.

To amplify this natural structure, the team introduced a streamlined approach called PathMoE. Instead of letting every layer make its own isolated traffic decisions, PathMoE groups consecutive layers into blocks that share the exact same routing rules. Because neighboring layers process similar information, this gentle constraint guides the data along highly concentrated, specialized routes without restricting the model's overall potential.

Main results on Fineweb-100B with 0.9B total / 0.37B active MoE architecture. Throughput is reported per GPU and memory reports peak active GPU memory.

By simply encouraging these natural pathways, models equipped with PathMoE demonstrated consistent improvements in accuracy and language comprehension. This method naturally keeps the AI's workload perfectly balanced, eliminating the need for the clunky, manual tuning formulas engineers previously relied on.

Screening Is Enough

Nakanishi [RIKEN]

♥ 762   Attention  

AI models distribute a fixed budget of attention across every piece of information they read. Because this budget is strictly capped, the system evaluates data relatively, comparing words against one another rather than valuing them on their own merits. As a document grows, this fixed attention dilutes.

A new architecture called Multiscreen introduces a beautifully intuitive solution to this problem through a mechanism researchers call "screening." Instead of forcing words to compete for a slice of a fixed attention pie, screening judges every piece of information entirely independently against a strict, absolute threshold. If a piece of data is useful, it passes the screen; if it is irrelevant, it is completely discarded.

Long-context perplexity comparison between 353M Transformer and 286M Multiscreen models.

By eliminating global competition among data points, the model cleanly filters out the noise and confidently gathers only the information that actually matters, allowing it to adapt its focus without getting overwhelmed by sheer volume.

Researchers found that Multiscreen achieves comparable performance using roughly forty percent fewer parameters than standard models. Even more impressively, a vastly scaled-down version of Multiscreen consistently outperformed much larger standard models in retrieving specific information, all while cutting processing delays by over three times on massive texts.

Mar 24th ~ Mar 31th
#101 Latest AI Research Explained Simply

🗞️ Industry News in 1 Line

1. ♥ 5.5k GLM-5.1 by z.ai is now officially available to all GLM Coding Plan users. To start using the new model, simply update your configuration file, such as ~/.claude/settings.json, by manually changing the model name to "glm-5.1." If you look at the benchmarks, it doesn’t look that impressive, but what’s great is that it offers 3× usage of the Claude Pro plan.

   

2. ♥ 2.2k Meta has released SAM 3.1, a drop-in update that introduces "object multiplexing" to track up to 16 objects simultaneously in a single forward pass. This update doubles video processing throughput and removes memory bottlenecks, making high-performance AI applications feasible on smaller, more accessible hardware. View on GitHub.

   

3. ♥ 15k Meta has introduced TRIBE v2, a new foundation model trained on over 500 hours of fMRI data to predict how the human brain responds to visual and auditory stimuli. This "digital twin" of neural activity achieves a nearly 3x improvement in zero-shot predictions over previous methods. View on GitHub.

   

4. ♥ 38k Google has released TurboQuant, which is a compression algorithm designed to optimize the key-value (KV) cache of Large Language Models. This new method reduces memory requirements by at least sixfold, allowing developers to run massive models on significantly more modest hardware. Moreover, it also delivers up to an 8x speedup in inference. Most importantly, TurboQuant achieves these performance gains with zero loss in accuracy, ensuring that model quality remains untouched despite the extreme compression.

   

   TurboQuant demonstrates robust KV cache compression performance across the LongBench benchmark relative to various compression methods

   The hype for Google’s TurboQuant is facing pushback from the research community as the paper contains serious technical inaccuracies and misleading comparisons. Lead researchers from the RaBitQ project claim that TurboQuant misrepresents their methodology and fails to acknowledge fundamental similarities, specifically regarding the use of the Johnson-Lindenstrauss transform. According to public statements, these flaws were flagged to the authors prior to submission, yet the paper was allegedly published and promoted without the necessary corrections.

Intuitive AI Academy - NEW Distillation Chapter!

My latest project: Intuitive AI Academy has the perfect starting point for you! We focus on building your intuition to understand LLMs, from transformer components, to post-training logic. All in one place.

We currently have an early bird offer, where you would get 40% off on the yearly plan for our early users.

Use code: TIMELINE

Check Out Intuitive AI Academy

Advertise with The AI Timeline!

Composer 2 Technical Report

Cursor Research Team

♥ 5.3k   Coding  

When developers ask AI models to write code, the models perform brilliantly on laboratory tests but stumble in the messy, ambiguous world of actual software engineering. Researchers realized that existing AI coding benchmarks were simply too neat, providing detailed instructions for isolated bugs. Developers constantly deal with vague bug reports, massive codebases, and confusing production logs. To bridge this gap, scientists set out to build a specialized assistant that genuinely thinks like a seasoned engineer navigating the authentic friction of daily development.

Overview of a single grouped GEMM training flow in our Mixture-of-Experts layer.

This paper introduces Composer 2, which is a new model that dramatically improves how AI handles long-term programming tasks. The researchers achieved this through a clever two-step training process.

Example CursorBench task

First, they immersed the model in massive amounts of code to build profound baseline knowledge. Then, they put it through rigorous reinforcement learning, simulating actual user sessions where the AI practiced solving diverse problems. To help the model stay perfectly focused during lengthy programming tasks, the team introduced a brilliant self-summarization technique.

Instead of getting overwhelmed by a long history of commands, the model constantly writes little internal summaries for itself. This preserves crucial context while discarding clutter.

To prove this approach works, the team bypassed traditional public tests and created a rigorous new evaluation based entirely on authentic engineering scenarios. In these demanding simulations, which require extensive code changes from incredibly brief instructions, the model demonstrated remarkable accuracy and coherence.

Claudini: Autoresearch Discovers State-of-the-Art Adversarial Attack Algorithms for LLMs

Panfilov et al. [MATS, ELLIS Institute Tubingen & Max Planck Institute for Intelligent Systems, Tubingen AI Center, Imperial College London]

♥ 1.5k   Coding Vulnerabilities   bycloud’s pick  

Can AI automatically conduct research to make future technology safer? Researchers built an automated security researcher to see if an AI could invent entirely new methods for pressure-testing digital vulnerabilities.

To test this, the research team built a continuous loop where the AI reviewed dozens of existing security-testing algorithms, wrote new software to improve them, and then evaluated its own creations. Instead of simply generating tricky text prompts to bypass safety filters, the AI engineered the underlying mathematical algorithms that actively search for these vulnerabilities.

Claudini Strongly Outperforms a Classical AutoML Method.

By intelligently splicing together previous techniques and writing clever mechanisms to avoid getting stuck during its analysis, the AI made massive strides. It successfully generated novel algorithms that dramatically outperformed human-made baselines, jumping from a success rate of less than ten percent to forty percent on a targeted safety filter.

The most remarkable finding is just how adaptable these new AI-authored algorithms proved to be. When scientists tested these methods on a completely different, highly secured model that the AI had never even encountered, the new algorithms bypassed the defenses with a flawless hundred percent success rate.

LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels

Maes et al. [Mila & Université de Montréal, New York University, Samsung SAIL, Brown University]

♥ 3.7k   LLM World Models  

AI needs an internal "world model" to predict the consequences of its actions. Researchers have tried teaching AI to build these models directly from raw video pixels, compressing complex visual scenes into a streamlined imagination space. However, this process faces a frustrating hurdle known as representation collapse.

When asked to predict the future, the AI often takes the lazy route, mapping every image to the exact same uniform representation just to guarantee a perfect prediction score. To prevent this, previous systems relied on highly fragile workarounds.

LeWorldModel Training Pipeline.

This paper developed LeWorldModel, which is a highly stable world model from scratch using just two simple rules. First, the model predicts the next compressed state of its environment based on a given action. Second, a clever mathematical regulator forces these compressed representations to stay continuously diverse, spreading them out to match a natural bell-curve distribution. By enforcing this varied shape, the model is strictly prevented from collapsing into a single, lazy answer.

Pseudo-code for the training procedure of LeWorldModel.

By reducing the complex tuning down to a single parameter, this compact model trains on one standard graphics card in just a few hours. Impressively, it plans up to forty-eight times faster than bulkier alternatives across complex control tasks. Even more fascinating, without explicit physics lessons, the model organically learns to track object locations and registers mathematical surprise when shown impossible events like spontaneous teleportation.

LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels

End-to-end joint-embedding predictive architecture from pixels.

le-wm.github.io

Self-Distillation of Hidden Layers for Self-Supervised Representation Learning

Lowe et al. [Vector Institute, Carleton University, Dalhousie University, University of British Columbia, University of Guelph]

♥ 780   Distillation  

Teaching an AI to understand the visual world is complex. For a long time, scientists have had to choose between two extreme training methods, both with frustrating limitations. One approach forces the AI to perfectly reconstruct raw, low-level pixels. While this keeps the system safely grounded in reality, it leaves the AI struggling to grasp big-picture concepts without a lot of extra hand-holding.

Multi-layer self-distillation with Bootleg.

The other approach asks the AI to predict only highly abstract, final-stage ideas. However, because the system is essentially generating its own study material in a continuous loop, it often becomes unstable, loses touch with the actual image, and completely breaks down during training.

To solve this, this research has introduced Bootleg. Instead of making the AI guess only the most basic details or only the most complex final concepts, they tasked it with predicting multiple hidden layers of information all at once. This beautifully mirrors how the human brain processes sights, where early visual processing picks up simple edges and colors, while deeper brain regions recognize complete objects.

By forcing the AI to simultaneously predict early, middle, and late stages of understanding, the system has to compress a wealth of varied knowledge through a tight informational bottleneck. This multi-level approach brilliantly keeps the AI anchored to actual visual features while it masters complex, abstract ideas.

Results for masked self-supervised learning with Bootleg and baselines.

The researchers discovered that this technique creates a dramatically smarter system, outperforming previous methods by significant margins in both recognizing what is in an image and mapping out exact scenes.

Mar 17th ~ Mar 24th
#100 Latest AI Research Explained Simply

🗞️ Industry News in 1 Line

1. ♥ 1.3k Xiaomi has announced the release of MiMo-V2-Pro, Omni, and TTS models. These new models introduce advanced features with global top-tier agent performance, multimodal interaction for seeing and hearing, and expressive voice synthesis. You can test these models on the web or via the API portal.

   

2. ♥ 1.5k MiniMax has launched its M2.7 model, which uses a recursive self-evolution architecture that contributed to an 88% win-rate over its predecessor. This model achieves state-of-the-art performance in software engineering benchmarks and high-fidelity document editing, while demonstrating enhanced agentic capabilities with 97% skill adherence. Try it today on the web.

   

Intuitive AI Academy - NEW Distillation Chapter!

My latest project: Intuitive AI Academy has the perfect starting point for you! We focus on building your intuition to understand LLMs, from transformer components, to post-training logic. All in one place.

We just added a new chapter on Distillaion!

We currently have an early bird offer, where you would get 40% off on the yearly plan for our early users.

Use code: TIMELINE

Check Out Intuitive AI Academy

Advertise with The AI Timeline!

Attention Residuals

Kimi Team

♥ 15k   Attention  

LLMs rely on residual connections to pass information through deep stacks of layers. While these connections act as a vital "gradient highway", they work like a blunt instrument. In current architectures, every layer simply adds its output to a uniform sum of all previous layers. This approach causes the information representing the model’s "hidden state" to swell uncontrollably as it moves deeper. Over time, this leads to a dilution effect where early, important information becomes buried and difficult for the model to retrieve effectively, effectively limiting how well the model can leverage its own depth.

Overview of Attention Residuals.

Researchers have introduced "Attention Residuals" (AttnRes) to replace this clumsy, fixed accumulation with a smarter mechanism. AttnRes allows each layer to selectively aggregate information from previous layers using learned, input-dependent weights. Instead of blindly adding everything together, the model now chooses which previous layers are most relevant to its current task.

PyTorch-style pseudo code for Block Attention Residuals.

To ensure this doesn't create excessive memory demands in massive models, the team developed "Block AttnRes," which organizes layers into smaller groups. This allows the model to maintain the benefits of selective, smart aggregation while keeping memory and communication costs efficient.

Experiments on large-scale models, including a 48B-parameter architecture, showed that this method successfully prevents the hidden-state growth that plagued previous models. By creating a more uniform distribution of signals and gradients across all layers, AttnRes consistently improves performance on complex reasoning, math, and coding benchmarks.

Mamba-3: Improved Sequence Modeling using State Space Principles

Lahoti et al. [Carnegie Mellon University, Princeton University, Together AI, Cartesia AI]

♥ 1.5k   Mamba   bycloud’s pick  

Transformer based AI models are incredibly powerful, but they suffer from a "bottleneck" in efficiency. As models grow, the computational effort required to generate each new piece of information, and the memory needed to store that context increases very fast.

This paper has introduced Mamba-3, a new architecture designed with an "inference-first" mindset to solve these efficiency challenges. By revisiting the mathematical foundations of State Space Models (SSMs), the team implemented three core improvements.

First, they developed a more expressive discretization method called "exponential-trapezoidal," which allows the model to handle data dynamics with greater precision than previous versions.

Second, they incorporated a complex-valued state update rule. This acts like a "rotary" mechanism that enables the model to track states, such as solving arithmetic parity tasks, that were previously impossible for similar linear models to master.

Finally, they shifted to a multi-input, multi-output (MIMO) formulation. This clever adjustment allows the model to perform more computation during the memory-heavy decoding phase without increasing the actual size of its state or slowing down its response time.

These refinements allow Mamba-3 to achieve a remarkable balance. At the 1.5B scale, it outperforms top-tier competitors in downstream accuracy while simultaneously matching the language-modeling capabilities of its predecessor at half the state size.

Prefill and Prefill+Decode latency across sequence lengths.

GitHub - state-spaces/mamba: Mamba SSM architecture

Mamba SSM architecture. Contribute to state-spaces/mamba development by creating an account on GitHub.

github.com/state-spaces/mamba

V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning

Mur-Labadia et al. [FAIR at Meta, Universidad de Zaragoza]

♥ 1.3k   JEPA  

To be truly useful, an AI needs to be a "jack-of-all-trades": it must understand both the big picture (like identifying a person’s action) and fine-grained local details (like pinpointing the exact edge of a glass for a robot to grasp). Previous models, such as the V-JEPA family, excelled at global understanding but struggled to extract precise local information, resulting in noisy, fragmented visual representations.

V-JEPA 2.1 Architecture

This gap limited their effectiveness in tasks that demand spatial precision, such as depth estimation or delicate robotic manipulation. The team discovered that the "missing link" was in the training objective. Traditional V-JEPA models only practiced predicting what was missing from an image or video, the masked, hidden patches.

V-JEPA 2.1 ViT-G performance across dense and global prediction tasks.

Because the model wasn’t forced to analyze the visible parts of the scene, it essentially learned to treat those visible sections as global summaries rather than detailed spatial maps. To fix this, this paper has introduced a "dense prediction loss," which forces the model to learn from both the hidden and the visible parts of the input.

By supervising every part of the scene, the model is compelled to build a coherent, fine-grained understanding of where objects actually exist in space. They further refined this by applying the learning signal at multiple layers deep within the network, a method called "deep self-supervision", and by using specialized tokenizers that handle images and videos in their native formats.

These changes, combined with scaling the model and the diversity of training data, resulted in a system capable of state-of-the-art performance in both high-level action forecasting and precise, low-level spatial tasks like depth perception.

GitHub - facebookresearch/vjepa2: PyTorch code and models for VJEPA2 self-supervised learning from video.

PyTorch code and models for VJEPA2 self-supervised learning from video. - facebookresearch/vjepa2

github.com/facebookresearch/vjepa2

Temporal Straightening for Latent Planning

Wang et al. [New York University, Brown University, University of Toronto]

♥ 1.2k   JEPA  

When AI agents interact with complex environments, they typically translate high-dimensional sensory data, like raw video, into a "latent space" to make decisions. For a computer trying to navigate, this means the shortest path between two points in its digital map doesn't actually correspond to the shortest physical path. Because these internal maps are so twisted, gradient-based planners, which rely on finding the smoothest way to reach a goal, often get stuck or perform poorly, forcing researchers to rely on computationally expensive search-based methods.

UMaze: ground-truth geodesic distance.

UMaze: ResNet-global after straightening.

This paper introduces a way to "straighten" these maps, helping AI agents see the world more linearly so they can plan their actions more efficiently. The researchers developed a method called "temporal straightening" to fix the distorted geometry of latent spaces.

Latent Curvature and Open-Loop GD Success Rate for Different Encoders. Higher cosine similarity indicates lower curvature.

This is based on a simple idea: as the agent moves, the model tracks its latent trajectory and minimizes the "curvature" by keeping the velocity vectors of consecutive steps as aligned as possible. It forces the AI to learn representations where movement in the latent space feels like travel along a straight, predictable line rather than a chaotic curve.

By adding a straightening objective to the standard training process, the model learns to map observations in a way that Euclidean distance, the simplest way to measure "how far" a goal is, finally matches reality. This has a profound effect on planning: because the path is straighter, the math behind gradient-based optimization becomes much more stable.

In experiments, this approach drastically improved success rates across a variety of navigation and manipulation tasks, allowing agents to reach goals with far greater precision and efficiency without needing the heavy compute power typically required for complex decision-making.

Mar 10th ~ Mar 17th
#99 Latest AI Research Explained Simply

🗞️ Industry News in 1 Line

1. ♥ 24k Claude 3.5 models (Opus and Sonnet) now support a 1-million-token context window, and allow users to process large codebases, extensive document sets, and up to 600 images or PDF pages per request. This update is available across all plans and is integrated by default into Claude Code at standard pricing.

   

2. ♥ 44k Google Maps has integrated Gemini AI to help you explore and navigate more easily. You can now use a new "Ask Maps" feature to get conversational answers to specific, real-world questions, like finding a place to charge your phone or a well-lit tennis court. Additionally, a new "Immersive Navigation" tool is rolling out, which provides vivid 3D visuals and more detailed route guidance to help you navigate your surroundings with more confidence.

3. ♥ 5.3k Hermes Agent by Nous Research is an open-source, Python-based tool designed to grow with you by utilizing a multi-level memory system and persistent machine access, similar to OpenClaw. It works across your CLI and various messaging platforms, and offers developers an extensible framework for complex tasks like subagent management and programmatic tool calling.

   

4. ♥ 6.2k Covenant-72B is the largest decentralized LLM pre-training run in history. It is a 72B parameter model trained across commodity internet connections without centralized clusters or whitelisting. By using innovative techniques like SparseLoCo for bandwidth efficiency and a blockchain-based "Gauntlet" system for validation, the project achieved performance levels competitive with models trained in traditional data centers. Try it in browser.

   

5. ♥ 19k Yann LeCun’s new startup Advanced Machine Intelligence (AMI) has secured $1.03 billion in one of the largest seed rounds in history to develop AI systems capable of advanced reasoning, persistent memory, and world-model understanding. These models are designed to understand the physical world while featuring persistent memory and the ability to reason, plan, and operate safely.

Intuitive AI Academy - NEW Distillation Chapter!

My latest project: Intuitive AI Academy has the perfect starting point for you! We focus on building your intuition to understand LLMs, from transformer components, to post-training logic. All in one place.

We just added a new chapter on Distillation too!

We currently have an early bird offer, where you would get 40% off on the yearly plan for our early users.

Use code: TIMELINE

Check Out Intuitive AI Academy

Advertise with The AI Timeline!

OpenClaw-RL: Train Any Agent Simply by Talking

Wang et al. [Princeton University]

♥ 674   RL  

Every time we interact with an AI, it receives immediate feedback: a follow-up question, a software error, or a screen transition. Existing AI systems throw this valuable experience away. They treat our replies merely as context for their very next move, missing a massive opportunity to learn. Researchers wanted to solve this by capturing these everyday reactions, what they call next-state signals, and turning them into a live, continuous learning stream.

We can build a system where an AI naturally improves simply by being used, turning ordinary conversations and software tasks into a seamless training loop without needing to pause for offline updates. The researchers built a unified framework called OpenClaw-RL.

The AI receives two powerful forms of information. First, there are evaluative signals, which act like a simple score indicating whether an action succeeded or frustrated the user. Second, there are directive signals. When a user corrects an AI by explaining how it should have responded, or when a software tool outputs a detailed error, it provides a clear map for improvement.

The framework extracts these specific textual hints to give the AI rich, word-by-word guidance. Because the system is built asynchronously, the AI can chat with a user, a background judge can evaluate its performance, and a training engine can update the AI's core behavior all at the exact same time, without ever interrupting the workflow.

By combining basic scoring with these rich textual hints, researchers found that personal assistants rapidly adapt their tone, becoming much more natural after just a handful of conversations.

Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights

Gan and Isola [MIT CSAIL]

♥ 818   Pretraining   bycloud’s pick  

Teaching an AI a new skill feels like searching for a microscopic needle in a haystack. Researchers have long believed that adapting a massive, billion-parameter model required highly complex, meticulous, step-by-step mathematical adjustments just to find a version of the system that performed a specific task well. Blindly guessing the right settings was considered mathematically impossible and entirely out of the question.

However, this study discovered that as models grow larger and undergo extensive initial training, adapting them to specific tasks like mathematical reasoning or coding no longer requires such rigid, painstaking effort. The heavy lifting of learning has already been done, signaling an exciting era where customizing powerful technology is becoming surprisingly natural, fast, and accessible.

The researchers found that scaling up these models fundamentally transforms their underlying structure. Instead of a desolate landscape where a good solution is a lone needle, massive models are surrounded by a dense, flourishing "thicket" of specialized solutions.

By making random, tiny adjustments to the model's underlying numbers, the researchers uncovered an abundance of hidden specialists waiting nearby. One random tweak might produce an expert in creative writing, while another creates a brilliant chemist, each specializing in one area while forgetting others.

To harness this rich diversity, the team tested a beautifully simple approach: they generated thousands of random tweaks simultaneously, kept the top performers for a specific task, and had them vote on the final answer. This parallel guess-and-check strategy matched the accuracy of today’s most advanced training methods but operated in a fraction of the time because it avoided slow, sequential updates.

Neural Thickets · MIT

Diverse Task Experts Are Dense Around Pretrained Weights

thickets.mit.edu

IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse

Bai et al. [Tsinghua University, Z.ai]

♥ 540   Attention  

AI models use an attention mechanism to connect different pieces of information, but as the text gets longer, this demands a staggering amount of computing power. Developers recently introduced a clever shortcut called sparse attention, which uses a specialized "indexer" tool to scan the text and pick only the most relevant words for the AI to focus on at each step. It is a brilliant fix, but researchers ran into a new bottleneck. This indexer operates independently at every single layer of the AI network. As the text grows, just running this indexer consumes a massive chunk of the system's processing time, slowing everything down.

Side-by-side comparison of inference loops.

Looking closely at how these models process data, researchers noticed an incredible inefficiency. They discovered that consecutive layers of the AI were repeatedly selecting almost the exact same important words, often with a near-perfect overlap. To solve this, the researchers developed an elegant, hopeful solution called IndexCache.

Instead of forcing every layer to do the heavy lifting of scanning and selecting information, IndexCache designates a few specific layers as "Full" layers to run the indexer. The remaining layers become "Shared" layers, which simply borrow the selected words from the nearest Full layer. The team created two ways to apply this: a training-free method that calculates the absolute best pattern of Full and Shared layers for existing models, and a training-aware method that actively teaches the AI to share this data.

Training-free IndexCache at 1/2, 1/4, and 1/8 indexer retention. ‘Long’ and ‘G&R’ aggregate benchmark scores.

By simply reusing this cached information, researchers eliminated 75 percent of the heavy indexer computations with negligible drops in the model's reasoning quality. When tested on massive systems, this straightforward change nearly doubled the speed at which the AI reads information and significantly accelerated its ability to generate answers.

Training Language Models via Neural Cellular Automata

Lee et al. [MIT, Improbable AI Lab]

♥ 1.5k   LLM Sampling  

LLMs rely on massive amounts of human-written text to learn how to reason and communicate. However, this approach faces a looming wall: high-quality human data is finite, often riddled with biases, and mixes actual reasoning with messy, subjective language. They investigated whether models could learn the fundamental mechanics of reasoning by training on purely synthetic, non-linguistic data before ever seeing a human sentence.

To test this, researchers turned to neural cellular automata (NCA), algorithmic systems that generate complex, ever-changing grid patterns using simple, local rules. Unlike static text, these patterns can be generated cheaply and in infinite supply, allowing for precise control over the "complexity" of the training data.

The team pre-trained models on these synthetic grid trajectories, then followed up with standard training on natural language. The results show that models that practiced on just 164 million synthetic tokens learned faster and performed better than those trained solely on significantly larger amounts of traditional internet text.

While traditional text training can cause a model to rely on human biases or semantic shortcuts, the synthetic grids force the model to focus purely on tracking long-range patterns and inferring underlying rules. Furthermore, the researchers found they could tune the complexity of this synthetic data to match specific domains; for instance, code benefited from simpler rules, while math and web text thrived on higher complexity.

Training Language Models via Neural Cellular Automata

hanseungwook.github.io/blog/nca-pre-pre-training

GLM-OCR Technical Report

Duan et al. [Zhipu AI, Tsinghua University]

♥ 1.1k   LLM RL  

Modern information systems rely heavily on extracting knowledge from complex, visually dense documents like financial reports, invoices, and scientific papers. While recent multimodal AI models have improved how we read these documents, they often suffer from a major drawback: their massive size makes them slow, memory-intensive, and difficult to deploy in practical, real-world settings.

Architecture and workflow of the GLM-OCR framework

The researchers developed GLM-OCR, a lightweight, highly optimized framework that packs significant power into a compact 0.9-billion-parameter design. The system merges a specialized visual encoder with a streamlined language decoder. What makes it particularly clever is the shift away from standard “one-token-at-a-time” generation, which is notoriously slow for structured documents.

Instead, the researchers implemented a Multi-Token Prediction mechanism that allows the model to predict several tokens simultaneously. By using a shared-parameter scheme to keep memory usage low, the model effectively boosts throughput, the speed at which it processes data, by roughly 50% without sacrificing accuracy.

To handle real-world complexities, the system employs a two-stage pipeline. First, it uses an analysis module to detect the layout of a document, breaking a complex page into manageable regions. These regions are then processed in parallel, allowing for faster and more robust recognition of everything from handwritten text to complicated table structures.

zai-org/GLM-OCR · Hugging Face

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

huggingface.co/zai-org/GLM-OCR

Mar 3rd ~ Mar 10th
#98 Latest AI Research Explained Simply

🗞️ Industry News in 1 Line

1. ♥ 6.8k Sarvam AI has announced the open-source release of its 30B and 105B parameter models, which were developed entirely in-house to target both global benchmarks and Indian language tasks. The model weights are now available on Hugging Face and AIKosh, featuring day-zero support for SGLang with vLLM compatibility expected soon.

   

2. ♥ 23k OpenAI has launched GPT-5.4 Thinking and GPT-5.4 Pro models across ChatGPT, the API, and Codex. These new models are designed to enhance reasoning, coding, and agentic workflows. It also includes features such as advanced deep web research capabilities and the ability for users to interrupt the model mid-process to provide real-time instructions or course corrections.

   

3. ♥ 15k Andrej Karpathy has developed an "autoresearch" agentic workflow designed to autonomously optimize neural network training through iterative experimentation. When applied to his nanochat project, the tool identified 20 additive changes that reduced the "Time to GPT-2" training benchmark from 2.02 hours to 1.80 hours.

   

Intuitive AI Academy - NEW MoE Chapter!

My latest project: Intuitive AI Academy has the perfect starting point for you! We focus on building your intuition to understand LLMs, from transformer components, to post-training logic. All in one place.

We just added a new chapter on MoE, that goes through the history, the key techniques, and the current state of MoE that frontier model uses. With over 10,000 words written!

We currently have a early bird offer, where you would get 40% off yearly plan for our early users.

Use code: TIMELINE

Check Out Intuitive AI Academy

Advertise with The AI Timeline!

Speculative Speculative Decoding

Kumar et al. [Stanford University, Princeton University, Together AI]

♥ 22k   Decoding  

The primary hurdle in making LLMs feel instantaneous is a phenomenon known as the sequential bottleneck. Standard AI models generate text one word, or "token", at a time, which fails to fully use the massive parallel computing power of modern hardware. Researchers previously introduced "speculative decoding", a method where a small, fast model drafts a few guesses for a larger model to verify. The drafting model has to wait for the larger model to finish checking its work before it can start guessing the next set of words. This creates a persistent lag that limits how fast even the most advanced systems can communicate.

Ordinary speculative decoding (SD) requires the verifier to wait idly for the draft to speculate.

To eliminate this idle time, researchers have developed Speculative Speculative Decoding (SSD) via an optimized algorithm called Saguaro. The biggest change is decoupling the "guesser" from the "checker" entirely. While the large target model is busy verifying a current batch of text, Saguaro’s draft model looks ahead and predicts several possible outcomes of that verification. It prepares a menu of "potential futures." If the larger model confirms one of these predicted outcomes, the system can immediately provide the next set of words without any drafting delay. This parallel approach effectively hides the time spent guessing, transforming a sequential process into a streamlined, continuous flow.

By using a clever "geometric fan-out" strategy, Saguaro focuses its computational effort on the most likely verification results, ensuring the "speculation cache" is highly accurate even at higher creative temperatures. This method is entirely lossless, meaning it achieves the exact same high-quality output as the original model but at much higher speeds.

Advantage of geometric fan out strategy increases at higher temperatures, improving both speculation cache hit rate (right) and thus end-to-end speed (left).

Initial results show that Saguaro can deliver text up to five times faster than traditional generation methods and twice as fast as previous state-of-the-art speculative techniques.

Beyond Language Modeling: An Exploration of Multimodal Pretraining

Tong et al. [FAIR, New York University]

♥ 424   Multimodal LLMs   bycloud’s pick  

AI is getting good at manipulating language, but it lacks a fundamental grasp of the physical world. Researchers describe this limitation using the "allegory of the cave": current models have mastered the description of shadows on a wall, text, without ever seeing the actual objects casting those shadows. Because text is a human abstraction, it is a "lossy" version of reality that misses the raw physics, geometry, and causality of our environment.

Overview of this study.

By training models to "see" and "read" simultaneously from birth, we can build a more grounded intelligence that understands the world’s dynamics directly.

The researchers developed a unified model using a framework called Transfusion, which trains a single "brain" to perform two different tasks at once: predicting the next word in a sequence and reconstructing visual frames through a process called diffusion.

Examples of training data.

They discovered that the most effective way to do this is by using a single, high-quality visual representation known as a Representation Autoencoder. This contradicts the traditional belief that you need different "eyes" for understanding an image versus creating one; instead, a unified representation excels at both while keeping the model’s language skills sharp.

The study also showed that learning from diverse visual data, like video and image-text pairs, actually improves the model’s performance on downstream tasks like reasoning and planning. To manage this complexity, they utilized a "Mixture-of-Experts" architecture.

Multimodal co-training exceeds unimodal performance.

This design allows the model to naturally evolve specialized internal "experts" for different tasks. It learned to dedicate more capacity to language while efficiently processing the massive amounts of data required by vision. Most impressively, this unified training allowed "world modeling" capabilities to emerge.

The model could predict the physical outcome of actions, like navigating a robot through a room, using simple text commands, proving that a truly multimodal foundation can bridge the gap between human language and physical reality.

Scaling laws for unified dense models.

Beyond Language Modeling: An Exploration of Multimodal Pretraining

Empirical insights on representation, data, architecture, and scaling for native multimodal pretraining.

beyond-llms.github.io

FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling

Zadouri et al. [Princeton University, Meta, Colfax Research, NVIDIA, Georgia Tech, Together AI]

♥ 430   Attention  

Latest AI hardware is significantly faster at basic matrix multiplication, but other components, like the units responsible for specialized math or moving data around, haven't kept the same pace. This creates a digital traffic jam where the fastest parts of a processor are frequently left idling, waiting for slower sections to finish their work.

FlashAttention-4 was designed to bridge this gap, by offering a clever software design that can overcome the physical limitations of even the most advanced hardware.

FlashAttention-4 forward pipeline.

The researchers taught the software to find creative shortcuts around these hardware bottlenecks. Because the chip’s dedicated unit for calculating exponentials is often the slowest link in the chain, they developed a way to emulate its functions using more plentiful, general-purpose math units.

FlashAttention-4 backward computation graph (5 MMA operations + 2 elementwise operations), showing the 1-CTA MMA mode software pipeline order across the prologue, main loop, and tail.

By using a mathematical technique called polynomial approximation, the software effectively mimics the specialized unit, achieving nearly identical accuracy at much higher speeds. Additionally, they introduced a "conditional rescaling" method that intelligently skips redundant calculations unless they are truly necessary for precision.

Backward pass TFLOPS on B200 (FP16/BF16) with head dimension 128.

By keeping more data in high-speed "tensor memory" and coordinating tasks so that different parts of the chip work in perfect sync, FlashAttention-4 can reach incredible speeds, hitting up to 1613 trillion operations per second on the latest Blackwell GPUs.

SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration

Chen et al. [Sun Yat-sen University, Alibaba Group]

♥ 855   LLM Evaluation  

Software engineering is rarely about writing a perfect piece of code in one go; it is an ongoing marathon of maintenance, updates, and evolving requirements. While current AI models have become quite good at solving isolated, one-shot coding tasks, these models are often tested on their ability to provide a quick fix rather than their capacity to sustain a healthy codebase over time.

Unlike previous benchmarks, SWE-CI proposes an evolution-based evaluation.

To address this, a new benchmark called SWE-CI shifts the focus from simple functional correctness to long-term maintainability, providing a much-needed lens into how AI agents handle the messy, continuous reality of software development.

The researchers built this benchmark using actual evolutionary histories from real-world software repositories, with tasks spanning an average of seven months and dozens of consecutive updates. To mirror a professional environment, they employed a dual-agent system where one AI acts as an Architect to identify gaps and set requirements, while a second Programmer agent implements the changes.

Data curation process of SWE-CI.

Success is measured by a novel metric called EvoScore, which specifically rewards agents that make decisions facilitating future growth rather than just immediate fixes.

The results show that the newest AI models are improving at an accelerating pace, but most still struggle to prevent regressions, the frustrating phenomenon where adding a new feature accidentally breaks an old one. This discovery highlights that the next great frontier for AI developers isn't just writing code that works, but writing code that lasts.

SWE-CI uses an architect-programmer dual-agent workflow to model the continuous integration cycle of professional software teams in the real world.

Real Money, Fake Models: Deceptive Model Claims in Shadow APIs

Zhang et al. [CISPA Helmholtz Center for Information Security]

♥ 805   LLM RL  

Access to frontier models is restricted by high costs, complex payment barriers, or geographical limitations. To bridge this gap, many researchers and developers have turned to "shadow APIs", third-party services that promise the same power as official models like GPT-5 or Gemini through unofficial, often cheaper channels. While these services appear to democratize access, they operate in a digital gray market with almost no transparency.

Researchers recently set out to investigate whether these shadow services are truly delivering what they advertise or if they are quietly undermining the integrity of the scientific work built upon them.

Landscape of the shadow APIs.

The investigation revealed a concerning gap between marketing promises and technical reality. By auditing these services through "model fingerprinting", a technique that identifies an AI by analyzing the unique statistical patterns and "signatures" in its responses, researchers discovered that nearly half of the shadow services were not using the models they claimed.

In a classic "bait-and-switch", premium proprietary models were frequently swapped for cheaper, open-source alternatives behind the scenes. This deception led to significant performance collapses; in high-stakes fields like medicine and law, accuracy dropped by nearly half when compared to official versions.

Fingerprinting results via LLMmap matched model and mean cosine distance D with standard.

The researchers found that while these shadow services might handle simple tasks well, they often fail during complex reasoning and display unpredictable safety behaviors. In addition to identifying the mismatch, the study used statistical testing to prove that the outputs from these services were fundamentally different from the official sources.

Feb 24th ~ Mar 4th
#97 Latest AI Research Explained Simply

🗞️ Industry News in 1 Line

1. ♥ 10k Google has begun rolling out Nano Banana 2, its latest image generation model. The updated model uses real-time web search data to improve real-world accuracy and introduces the ability to render clear, multilingual text for designs like posters and logos. Additionally, Nano Banana 2 brings faster generation speeds alongside enhancements to lighting, textures, and overall image detail. Try it today via the Gemini app and web interface.

   

2. ♥ 3.7k OpenAI has signed a classified deployment contract with the Department of War, insisting that a "cloud-only" architecture, an internal safety stack, and cleared engineers will somehow strictly prevent the military from using their models for autonomous lethal weapons or mass NSA surveillance. Entrusting a tech corporation to independently self-police lethal and intelligence applications sounds straight out of a black mirror episode.

3. ♥ 20k Alibaba's Qwen team has launched the Qwen 3.5 Small Model Series, a family of native multimodal models ranging from a highly compact 0.8B to a remarkably capable 9B designed for edge devices and lightweight agents. Both the base and instruct models are now available on Hugging Face and ModelScope. Additionally, the entire suite is already optimized for local deployment via Ollama.

   

4. ♥ 8k After small series, Alibaba has also introduced the Qwen 3.5 Medium Model Series, which includes the 27B, 35B-A3B, and 122B-A10B models designed to bridge the gap between mid-sized and frontier AI capabilities. Highlighting a shift toward architectural efficiency over parameter size, the new 35B-A3B model notably outperforms the previous generation's massive 235B model thanks to improved data quality and reinforcement learning. Try it in browser.

5. ♥ 8.2k Google has announced Gemini 3.1 Flash-Lite, its fastest and most cost-effective Gemini 3 model to date. It has dynamic "thinking levels" and the model instantly processes high-volume queries while scaling its reasoning for complex edge cases, delivering a 2.5X faster time-to-first-token than its 2.5 Flash predecessor. Try it in browser.

   

Intuitive AI Academy - NEW MoE Chapter!

My latest project: Intuitive AI Academy has the perfect starting point for you! We focus on building your intuition to understand LLMs, from transformer components, to post-training logic. All in one place.

We just added a new chapter on MoE, that goes through the history, the key techniques, and the current state of MoE that frontier model uses. With over 10,000 words written.

We currently have a early bird offer, where you would get 40% off yearly plan for our early users.

Use code: TIMELINE

Check Out Intuitive AI Academy

Advertise with The AI Timeline!

Learning Without Training

Ryan O’Dowd [Claremont Graduate University]

♥ 720   LLM Training  

Engineers usually build machine learning models by guessing a structure and running exhaustive optimization processes to train them. Researchers wanted to know if there was a more elegant way to overcome the hurdles of high-dimensional, noisy data without relying on these brute-force training methods.

The current approach assumes a model will eventually learn the underlying patterns, but it lacks constructive mathematical guarantees. By rooting their approach in classical approximation theory, we can save immense computational power while tackling complex problems like tracking brain diseases or analyzing hyperspectral images.

Normalized histogram of the density of interest (left), paired with our density estimation by σ128 based on 3900 samples (right).

This paper discovered how to mathematically construct highly accurate models directly on unknown, complex data surfaces, known as manifolds. This method bypasses the need to map out the entire geometry of the dataset first; it only requires knowing the dimension of the data.

The team unlocked a breakthrough in transfer learning by figuring out how to successfully lift learned information from just a localized portion of one data space and apply it to a completely different domain. As a result, adapting a massive model to a new problem no longer requires processing the entire original dataset.

Finally, the researchers reimagined data classification by treating it like a signal separation problem. By mathematically estimating the underlying sources of these signals, their new algorithm quickly zeroes in on the most informative data points.

Doc-to-LoRA: Learning to Instantly Internalize Contexts

Charakorn et al. [Sakana AI, Minerva University]

♥ 1.4k   LoRA   bycloud’s pick  

Let’s assume you asked an AI to analyze a massive technical manual. Currently, every time you ask a follow-up question, the system has to re-read the entire document. This repetitive reading eats up massive amounts of computing power, memory, and time.

While researchers can technically train the AI to memorize the document permanently, that traditional training process is painfully slow, expensive, and completely impractical for quick updates.

To solve this, researchers developed a brilliant workaround called Doc-to-LoRA, or D2L. Instead of forcing the main AI to constantly re-read text or undergo grueling training, they built a specialized, lightweight helper system called a hypernetwork.

This helper reads the document exactly once and instantly generates a tiny, customized plug-in. Think of it like instantly downloading a new skill directly into the AI's brain. Once this plug-in is attached, the main AI can answer subsequent queries fluidly without ever needing the original text in its prompt. It performs this complex mental compression in just a single step, completely bypassing the steep costs of standard training.

QA performance on SQuAD compared to the used context length ratio (left), update latency (middle), and additional memory needed for model updates (right).

The results are incredibly promising. In testing, D2L successfully hunted down specific facts hidden inside massive walls of text, achieving near-perfect accuracy on documents over four times larger than the AI’s normal limits.

It works drastically faster and uses far less memory than previous memorization methods. It can even translate visual information from image-based models into these text plug-ins, allowing a text-only AI to classify images.

Long document QA performance. LLMLingua-2 compresses the input with [20%, 40%, 60%, 80%, 90%] compression rates from right to left (gray dots).

The Geometry of Noise: Why Diffusion Models Don't Need Noise Conditioning

Sahraee-Ardakan et al. [Google]

♥ 382   Diffusion Models  

Can you restore a severely damaged painting without knowing how much damage was originally done? Standard diffusion models avoid this problem by relying on a strict timer that tells them exactly how much "noise" or corruption they are dealing with at any given step.

Recently, researchers have been incredibly hopeful about "autonomous" models that strip away this timer, learning a single rule to handle everything from pure static to nearly perfect data. However, this creates a profound mathematical paradox.

As these blind models approach the clean data, the underlying mathematical landscape forms an infinitely deep pit. The directional signals diverge completely, creating a severe geometric singularity. By all conventional logic, these models should become hopelessly unstable and crash.

The Singular Geometry of the Marginal Energy Landscape.

Yet, researchers have beautifully resolved this mystery. They discovered that these autonomous systems are actually charting a course across a unified map called "Marginal Energy."

More importantly, the scientists proved that the models naturally develop a hidden geometric shock absorber. As the AI approaches that infinitely deep mathematical pit, this built-in feature perfectly counteracts the extreme steepness. It transforms a catastrophic plunge into a smooth, stable descent known as a Riemannian gradient flow.

Generative performance on Fashion MNIST.

Models attempting to directly predict the noise act like faulty amplifiers, magnifying tiny errors until the system catastrophically breaks down. Conversely, models based on predicting "velocity" inherently absorb that uncertainty into a smooth, stable drift.

dLLM: Simple Diffusion Language Modeling

Zhou et al. [UC Berkeley, UIUC]

♥ 120   Diffusion LLM  

Language models traditionally generate text strictly left to right. But recently, researchers have found a promising alternative: diffusion language models. These systems can generate words in any order and iteratively refine their answers, unlocking highly flexible AI.

GitHub - ZHZisZZ/dllm: dLLM: Simple Diffusion Language Modeling

dLLM: Simple Diffusion Language Modeling. Contribute to ZHZisZZ/dllm development by creating an account on GitHub.

However, the underlying code was scattered across complex, isolated research repositories, making it incredibly difficult for developers to reproduce results or build upon each other’s work. To solve this, researchers created dLLM, a unified open-source framework that elegantly standardizes the development pipeline so the community can innovate together.

Inference pipeline: sampler swap from vanilla to FastdLLM MDLM sampler.

The framework seamlessly connects the core pillars of AI development: training, generation, and testing. Using a highly modular design, dLLM allows developers to snap different components together effortlessly. A researcher can easily swap out a training method or plug in a high-speed generation algorithm without rewriting the model's core architecture.

By standardizing how these models are evaluated, researchers also uncovered a hidden quirk: diffusion models are intensely sensitive to tiny adjustments in generation settings. A single tweaked parameter can drastically alter a model's performance, highlighting exactly why a transparent, shared testing environment is vital for meaningful progress.

Terminal Visualizer showing transition from masked to decoded tokens.

Perhaps the most hopeful breakthrough is how this framework democratizes AI research. The team proved that building these dynamic models does not require massive supercomputers. Using their new recipes, they successfully transformed standard, off-the-shelf systems (including traditional discriminative architectures like BERT) into functional diffusion chatbots.

Sensitivity to decoding hyperparameters.

They achieved this with minimal computing power and simple fine-tuning, requiring no structural changes to the original models.

dllm-hub (dLLM)

Org profile for dLLM on Hugging Face, the AI community building the future.

Feb 19th ~ Feb 24th
#96 Latest AI Research Explained Simply

🗞️ Industry News in 1 Line

1. ♥ 8.2k Replit has introduced Replit Animation, a new tool that lets users create animated videos in minutes using conversational prompts. It is powered by Gemini 3.1 Pro and makes it easier to produce polished, shareable content without traditional editing software.

   

2. ♥ 22k Anthropic has released Claude Sonnet 4.6, which comes with stronger performance in complex spreadsheet tasks, multi-step web forms, and expanded integrations through Excel MCP connectors. The Claude API now also supports more accurate web search, dynamic filtering, and general availability of code execution, memory, and programmatic tool use.

   

3. ♥ 49k Anthropic has announced Claude Code Security to scan codebases for vulnerabilities and suggest targeted patches for human review. This release triggered a sharp selloff in cybersecurity stocks, with companies like JFrog, CrowdStrike, Okta, and Cloudflare all recording declines. The system uses reasoning to trace data flows and flag subtle errors, while enforcing a human-in-the-loop safeguard to ensure developer oversight. If you want to try it, then join the waitlist here.

Learn LLMs Intuitively - Intuitive AI Academy

Want to learn about LLMs, but never have a good place to start?

My latest project: Intuitive AI Academy has the perfect starting point for you! We focus on building your intuition to understand LLMs, from transformer components, to post-training logic. All in one place.

content overview

We currently have a early bird offer, where you would get 40% off yearly plan for our early users.

Use code: TIMELINE

Check Out Intuitive AI Academy

Advertise with The AI Timeline!

Unified Latents (UL): How to train your latents

Heek [Google DeepMind Amsterdam]

♥ 2.1k   ALDiffusion  

To create high-quality images and videos efficiently, models usually compress data into a "latent" space (a digital shorthand that represents the original image in a smaller, more manageable package). If you compress the data too much, the AI loses fine details like textures and sharp edges; if you don't compress it enough, the AI becomes incredibly slow and expensive to train.

Schematic overview of our model, include the Encoder, the prior latent diffusion model, and the diffusion decoder model.

Traditional methods often rely on manual tuning to strike this balance, which is often more of an art than a science. Researchers have recently introduced a framework called Unified Latents (UL) to turn this guesswork into a systematic, more efficient process. By co-training the compression and generation steps together, they have found a way to maintain stunningly high-quality details while actually lowering the computational cost required for training.

Instead of treating the compressed representation as a static container, Unified Latents uses a "diffusion prior" to monitor and regularize the information flow. By linking the noise level produced during the encoding process directly to the precision of the diffusion model, researchers could create a mathematically tight way to control the latent bitrate.

A selection of samples from a text-to-image trained with Unified Latents

They also paired this with a diffusion-based decoder, which is remarkably better at reconstructing high-frequency details than previous methods. This unified approach allows the system to navigate the trade-off between compression and quality with much greater precision.

Fast KV Compaction via Attention Matching

Zweiger et al. [Massachusetts Institute of Technology]

♥ 196   Attention   bycloud’s pick  

AI models can take on more complex tasks like long-form coding and multi-day conversations, but they face a significant memory hurdle known as the key-value (KV) cache bottleneck. Every word the model processes adds to a digital "short-term memory" that can quickly balloon into several gigabytes of data.

Until now, researchers have managed this by either summarizing the text (which often strips away vital nuances) or by using expensive optimization techniques that take hours of computing time to compress a single document.

This paper has introduced a technique called Attention Matching that changes how we think about memory compaction. Instead of relying on slow, iterative training to condense information, this approach treats memory like a mathematical puzzle that can be solved directly in "latent space".

By focusing on how a model "pays attention" to specific pieces of information, the researchers found they could create a compact version of the memory that mimics the original's behavior. They discovered that this problem can be broken down into smaller sub-problems with efficient, closed-form solutions, allowing them to bypass the slow trial-and-error process of traditional machine learning.

Accuracy vs. compaction ratio across methods.

This new framework can shrink a model's memory by up to 50 times in just a matter of seconds, rather than hours, with almost no impact on the quality of the model's output.

Experiential Reinforcement Learning

Shi et al. [KRAFTON, University of Wisconsin–Madison, UC Berkeley, Microsoft Research]

♥ 1k   LLM RL  

Teaching artificial intelligence to navigate complex tasks is often a game of high-stakes guessing. In standard reinforcement learning, a model typically receives a single "reward" signal. This makes it incredibly difficult for the AI to pinpoint exactly where it tripped up or how to adjust its behavior for the next try.

Researchers recognized that this "blind" trial-and-error is far less efficient than how humans naturally learn. When we fail at a task, we don't just try again at random; we stop, reflect on what went wrong, and form a mental plan to do better. To solve this, researchers introduced Experiential Reinforcement Learning (ERL), a new approach designed to turn silent failures into structured, durable lessons.

In Experiential Reinforcement Learning (ERL), instead of learning from feedback or outcome directly

The researchers developed a clever "experience-reflection-consolidation" loop that embeds human-like reasoning directly into the training process. Instead of moving on immediately after an attempt, the model receives feedback from its environment and is prompted to generate a verbal reflection on its own performance.

It looks at its errors and produces a self-critique that guides a refined second attempt at the same task. If this second try succeeds, the model "internalizes" the successful correction. This is the breakthrough moment: by training the model to reproduce the improved behavior from the original task alone, the AI eventually learns to skip the reflection step entirely.

Conceptual comparison of learning dynamics in RLVR and Experiential Reinforcement Learning (ERL)

This ensures that the final model is both smarter and faster, maintaining high performance at deployment without any extra computational cost.

In complex multi-step tasks like Sokoban, which require deep planning and spatial reasoning, ERL improved performance by a staggering 81% over standard methods. It also showed reliable gains in agentic reasoning tasks that involve using external tools to answer questions.

Overview of Experiential Reinforcement Learning (ERL).

By allowing the AI to accumulate "corrective knowledge" in a persistent memory, researchers have created a way for models to build on their past successes rather than repeating the same mistakes.

GLM-5: from Vibe Coding to Agentic Engineering

Zhipu AI & Tsinghua University

♥ 290   LLM Agents  

Researchers are working to shift our relationship with AI from "vibe coding" (where humans provide constant prompts) toward a more autonomous era of "agentic engineering." Traditional models often struggle with the sheer computational cost of keeping track of long conversations, or they lose their way during tasks that require hours of planning and execution.

GLM-5 was designed to overcome these bottlenecks by creating a more independent assistant that can plan, implement, and iterate on technical challenges with minimal human intervention.

To make this possible, the researchers introduced a dynamic mechanism called DeepSeek Sparse Attention. Rather than the model straining to analyze every single word in a massive document with equal intensity (which is incredibly expensive and slow), it now identifies which tokens are truly important for the task at hand.

This allows the model to manage massive amounts of data, such as entire codebases or long-term business simulations, while significantly reducing the hardware power required. The model also features an "interleaved thinking" process where it pauses to reason before every action it takes.

It can "preserve" these thoughts across a long conversation, ensuring it doesn't lose its train of thought when moving between different stages of a project. By using a new asynchronous training infrastructure that allows the model to learn from complex, it was able to reach an unprecedented ability to solve end-to-end software engineering challenges.

Feb 9th ~ Feb 16th
#95 Latest AI Research Explained Simply

🗞️ Industry News in 1 Line

1. ♥ 5.7k Google DeepMind has upgraded its Gemini 3 Deep Think reasoning mode to tackle complex scientific and engineering challenges. This new mode is setting new performance records on frontier benchmarks like ARC-AGI-2. You can access it via Google AI Ultra subscription and is available to researchers via a Vertex AI Early Access Program.

   

2. ♥ 3.1k Zhipu AI’s GLM-5 has demonstrated the ability to manage over 700 tool calls and 800 context handoffs during continuous 24-hour operations. This advancement allows AI agents to handle highly complex, multi-step workflows, such as autonomously reverse-engineering software code from visual video demonstrations. Currently #1 open source model, and beating Gemini-3-Pro-high.

   

3. ♥ 8.9k MiniMax has launched M2.5, an open-source model that delivers state-of-the-art performance in coding, search, and agentic tool-calling, including a top-tier 80.2% score on SWE-Bench. This model is optimized for productivity and it is 37% faster and can be used as a drop-in replacement for Claude Code via its coding plan.

   

4. ♥ 4.2k Alibaba has published Qwen3.5-397B-A17B, its first open-weight, native multimodal model featuring a sparse Mixture of Experts (MoE) architecture that is optimized for real-world agents. It is licensed under Apache 2.0 and it supports over 200 languages and delivers up to 19 times the decoding throughput of its predecessor through advanced reinforcement learning environment scaling.

   

Learn LLMs Intuitively - Intuitive AI Academy

Want to learn about LLMs, but never have a good place to start?

My latest project: Intuitive AI Academy has the perfect starting point for you! We focus on building your intuition to understand LLMs, from transformer components, to post-training logic. All in one place.

content overview (a total of 100k words explainer so far!)

Check Out Intuitive AI Academy

Advertise with The AI Timeline!

ViT-5: Vision Transformers for The Mid-2020s

Wang et al. [Johns Hopkins University, UC Santa Cruz]

♥ 1k   ViT  

LLMs are sprinting ahead with rapid architectural refinements, but Vision Transformers (ViTs) have remained largely stagnant since their debut in 2020. Vision models struggle with stability issues and a limited ability to handle complex spatial reasoning.

ViT architecture

The research team developed ViT-5 by systematically testing five years of AI advancements to see which ones actually improve a model's "eyesight." They discovered that simply copying language model tricks doesn't always work; for instance, a popular method for filtering information in text models actually caused "over-gating" in vision, making the internal representations too sparse to be useful.

Instead, they found success by combining a more efficient normalization method with a clever dual-positioning system. This allows the model to understand where every pixel is relative to its neighbors while still maintaining a "big picture" sense of the entire image.

To further refine performance, the researchers introduced "register tokens," which act like digital scratchpads to clean up visual artifacts and help the model focus on what is semantically important. They also implemented a technique called QK-normalization, which smoothed out the training process and eliminated the frustrating "error spikes" that often crash large-scale AI projects.

The final model can handles images of varying sizes with ease and consistently outperforms previous standards in identifying objects and generating new images.

SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning

Xia et al. [UNC-Chapel Hill, University of Chicago, University of California San Diego, NEC Labs America, University of California Berkeley, University of California Santa Cruz]

♥ 455   Skill   bycloud’s pick  

Even the most advanced models often treat every new task as a blank slate. Researchers have long tried to give these agents a memory, but simply feeding them long, messy logs of past actions often results in "noisy" confusion that slows the system down.

The team behind SKILLRL realized that for AI to truly evolve, it shouldn't just record what happened; it needs to distill those experiences into compact, actionable skills. This team developed a framework that transforms raw, verbose interaction data into a structured "SkillBank."

Instead of saving every redundant step of a task, the system uses a teacher model to extract the core logic behind a success and the critical lessons from a failure. These insights are organized into a hierarchy: general principles for broad strategy and specialized tactics for specific tasks.

To make this work, the researchers introduced a recursive evolution process. As the agent practices using reinforcement learning, it doesn't just improve its own performance; it simultaneously updates its library.

When the agent hits a new type of roadblock, the system analyzes the failure, writes a new "skill" to handle it, and adds it to the collection. This co-evolution creates a virtuous cycle where the agent becomes more efficient and avoids "context bloat," using ten to twenty times less data than raw logs.

The results are striking, showing that smaller, open-source models can actually outperform massive, closed-source giants like GPT-4o by using this structured expertise.

Performance on ALFWorld and WebShop.

Learning a Generative Meta-Model of LLM Activations

Luo et al. [UC Berkeley, Transluce]

♥ 353   LLM Activations  

Current tools for analyzing LLMs rely on rigid mathematical guesses that don't quite capture the messy, organic complexity of an AI’s inner workings. When we try to "steer" an AI toward a specific persona or sentiment, the intervention often corrupts the model’s internal logic, causing it to descend into repetitive or nonsensical babble.

To achieve this, the team developed a "meta-model" called a Generative Latent Prior, or GLP. Rather than forcing the data into simple boxes, they trained a massive diffusion model on over a billion internal states of a functioning language model. This meta-model effectively learns the "natural habitat" of the AI's internal activations.

When a human intervention pushes the AI into an unnatural or "off-manifold" state, the GLP acts as a corrective guide. It "denoises" the internal signal, pulling it back onto a path that makes sense to the language model while preserving the intended change in behavior.

GLP generates activation samples near-indistinguishable from real activations

This approach follows reliable scaling laws, meaning that as they invested more computing power, the meta-model became significantly better at predicting and refining the AI’s internal states. They also found that the meta-model’s own internal units, which they call "meta-neurons," are exceptionally good at isolating specific, human-understandable concepts like geography or mathematics.

Feb 3rd ~ Feb 9th
#94 Latest AI Research Explained Simply

🗞️ Industry News in 1 Line

1. ♥ 72k ByteDance has released Seedance 2.0 in China, a powerful new AI tool that empowers creators to generate stunningly realistic, high-resolution videos from simple prompts. It is so advanced and its output so lifelike that it is already being hailed as a game-changing force which can revolutionize the filmmaking and visual effects industries.

2. ♥ 10k OpenAI has launched GPT-5.3-Codex, which has the ability to handle long-running, complex tasks that involve research and tool use, effectively acting as an interactive collaborator. It is 25% faster and has set new industry standards in performance (real-world software engineering to building complex games and apps from scratch).

   

3. ♥ 39k Anthropic has launched Claude Opus 4.6, its latest model for agentic coding, reasoning, and complex knowledge work. It now supports brand-consistent slide creation in PowerPoint, sophisticated multi-step planning in Excel, and autonomous "agent teams" for parallelized coding. Claude Opus 4.6 provides creators with an incredibly smart and efficient partner for handling long-running, high-stakes projects.

   

4. ♥ 961 Intern-S1-Pro has launched as a 1T open-source model that delivers state-of-the-art scientific reasoning, rivaling the world's leading closed-source models in complex AI4Science tasks. It uses Fourier Position Encoding for superior time-series modeling and integrates with vLLM, this powerful tool provides researchers with an incredibly efficient and accessible way to master complex physical signals and advanced multimodal data. Get weights on Hugging Face or try via API.

   

Learn LLMs Intuitively - Intuitive AI Academy

Want to learn about LLMs, but never have a good place to start?

My latest project: Intuitive AI Academy has the perfect starting point for you! We focus on building your intuition to understand LLMs, from transformer components, to post-training logic. All in one place.

content overview (a total of 100k words explainer so far!)

We currently have a New Year New Me launch offer, where you would get 50% off yearly plan FOREVER for our early users. Only a few left & the deal will not be back!

Use code: 2026

Check Out Intuitive AI Academy

Advertise with The AI Timeline!

Generative Modeling via Drifting

Deng wt al. [MIT, Harvard University]

♥ 1.1k   LLMs  

Generative modeling is about teaching a computer to map simple random noise into complex, meaningful data, like a realistic image. Until now, leading approaches work by taking a noisy sample and refining it step-by-step until it becomes clear. This multi-step requirement makes the actual generation process computationally heavy and time-consuming. The researchers wanted to know whether it is possible to condense this complex evolution into a single, instant step without sacrificing quality.

Illustration of drifting a sample.

The researchers introduced a new approach called "Drifting Models," which reimagines how a neural network learns to create data. Instead of forcing the model to refine an image iteratively every time it runs, this approach allows the network to act as a single-pass generator.

The biggest change is the introduction of a "drifting field" that operates during the training process. This field governs how the distribution of generated samples moves, or "drifts," toward the target data distribution as the network optimizes. The system is designed to seek equilibrium: when the generated samples match the real data, the drift becomes zero, and the model stabilizes.

Evolution of samples.

This method effectively moves the iterative evolution out of the inference stage and into the learning stage. Once the model is trained, it can generate high-fidelity content in just one step. The results are highly encouraging, with the method achieving state-of-the-art performance on ImageNet benchmarks.

System-level comparison: ImageNet 256×256 generation in pixel space

Learning to Reason in 13 Parameters

Morris et al. [FAIR at Meta, Cornell University, Carnegie Mellon University]

♥ 2K   Reasoning   bycloud’s pick  

Researchers are teaching models to "reason" and thoughtfully work through math and logic problems rather than just predicting the next word. Fine-tuning a massive model to improve these cognitive skills has been a heavy computational lift, which requires engineers to adjust millions, if not billions, of internal connections.

It is a common belief that learning a complex new behavior requires a substantial overhaul of the model’s weights, which makes personalization expensive and difficult to scale. However, this paper suggests that when it comes to reasoning, a model might not need to learn nearly as much new information as previously thought; perhaps it just needs a very precise, microscopic nudge in the right direction.

Using Qwen2.5-7B-Instruct as a base model, our TinyLoRA achieves performance within 5% of full finetuning on GSM8K with only 13 parameters. Dashed lines indicate untrained and full-FT baselines.

This paper introduced a method called TinyLoRA, which showed that large language models can learn to reason effectively by updating as few as thirteen parameters (a change so small it occupies merely 26 bytes of computer memory).

By using reinforcement learning, they were able to train a massive model to achieve over 90% accuracy on complex math problems, effectively matching the performance of traditional methods that update thousands of times more data. The researchers found that this extreme efficiency works because reinforcement learning acts as a highly effective filter.

TinyLoRA performance during training on MATH.

Standard training methods train the model with "noisy" data that it tries to memorize, but reinforcement learning provides a clean, sparse signal (simply telling the model if its logic was right or wrong). This clarity allows the model to ignore irrelevant details and isolate the specific neural pathways needed for reasoning.

Maximum Likelihood Reinforcement Learning

Tajwar et al. [Carnegie Mellon University, Tsinghua University, Zhejiang University, UC Berkeley, Impossible Inc.]

♥ 714   LLM RL  

Researchers use RL to teach AI models how to solve tasks with clear right-or-wrong outcomes, such as writing code or solving complex math proofs. The authors of this paper suggest that this strategy is actually just a rough, first-order approximation of a far more powerful mathematical goal: maximum likelihood.

The problem has always been that targeting maximum likelihood directly in these complex, open-ended scenarios was considered computationally intractable. The research team set out to fix this misalignment, asking if there was a principled way to trade more computing power for a better, more mathematically rigorous learning objective without hitting a computational wall.

The researchers introduced a framework called MaxRL, which elegantly bridges the gap between standard reinforcement learning and the ideal maximum likelihood objective. They demonstrated that the learning signal can be viewed as a mathematical series where standard methods only optimize the very first term. MaxRL unlocks a "compute-indexed" family of objectives. By generating more sample attempts (or "rollouts") during training, the algorithm incorporates higher-order terms from this series, creating a progressively more accurate approximation of the ideal objective.

Results on Qwen3-4B. MaxRL Pareto-dominates GRPO across all benchmarks, achieving similar or better Pass@1 while significantly improving Pass@K. This translates to 7.9×–19.2× gains at test-time scaling efficiency.

This approach effectively transforms extra computing time into a smarter learning signal. The study reveals that MaxRL focuses the model’s attention on harder, lower-probability successes rather than just reinforcing easy wins.

Example maze: successful navigation (left) vs. failure case (right).

Additionally, the method consistently outperforms existing techniques in mathematical reasoning and navigation tasks. The researchers found that their approach prevents the model from "overfitting," or losing its creativity, as training progresses.

Kimi K2.5: Visual Agentic Intelligence

Kimi team at Moonshot AI

♥ 16K   Visual LLM  

There is a difference between how models process language and how they understand images, combined with their inability to handle massive, multi-step tasks efficiently. Until now, visual capabilities were often bolted onto text models like an afterthought, and "agents" worked linearly. The team behind Kimi K2.5 created a model where vision and language are treated as equal partners from day one, while simultaneously reinventing how AI manages complex workflows.

Vision RL training curves on vision benchmarks starting from minimal zero-vision SFT.

It has two clever architectural innovations. First, rather than teaching the model to read and then teaching it to see, researchers integrated visual data and text together at the very beginning of the training process. This "early fusion" created a surprising synergy: learning to interpret images actually made the model better at text-based reasoning, and text training helped it understand visuals without needing specific image examples. It turns out that a balanced diet of inputs helps the AI generalize better across the board, creating a shared understanding where vision refines text and text bootstraps vision.

An agent swarm has a trainable orchestrator that dynamically creates specialized frozen subagents and decomposes complex tasks into parallelizable subtasks for efficient distributed execution.

Secondly, the researchers introduced a framework called "Agent Swarm." Instead of a single AI struggling through a long to-do list, Kimi K2.5 acts as an orchestrator. It looks at a complex problem, breaks it down into smaller pieces, and assigns them to specialized sub-agents that work in parallel.

Overview of training stages: data composition, token volumes, sequence lengths, and trainable components.

By treating the AI as a manager of a digital swarm rather than a solo worker, the system can handle heavy workloads in coding and research that would typically overwhelm a standard model.

Performance and token efficiency of some reasoning models. Average output token counts (in thousands) are shown in parentheses.

We are seeing ground-breaking discoveries in the field of AI every couple of days, and in 2026, AI is entering a new period where static models are becoming a thing of the past. A lot of new research are starting to move towards systems that learn, adapt, and evolve in real time. For nearly a decade (2017-2025), we were following a simple approach: train → freeze → deploy. Models became fixed artifacts the moment training ended, unable to grow with new data or shifting environments.

But in 2026, we are seeing a new shift where the boundary between training and inference is disappearing. This is giving rise to a new age of Adaptive Intelligence. Continual Learning (CL) is no longer just about preventing catastrophic forgetting, but it’s about active adaptation through Test-Time Training (TTT) and the stability offered by Reinforcement Learning (RL) compared to traditional Supervised Fine-Tuning (SFT). 

In this post, we will see how this shift is redefining how AI systems are evolving, and turning them into dynamic partners rather than frozen tools.

Let’s get started!

What is Continual Learning in LLMs

For over a decade, we built models under the following assumptions: "Training" was the expensive, compute-heavy phase where knowledge was forged, and "Inference" was the cheap, static phase where that knowledge was applied. This separation created a fundamental brittleness. A model frozen at the end of training could effectively handle the "average" case it saw during supervision, but it lacked the plasticity to adapt to the specific nuances of a new, complex problem instance.

In 2026, we are seeing a lot of new views of a neural network not as a fixed artifact. Instead, we view the inference process itself as a continuous learning loop. The model does not just read the test instance; it trains on it.

This shift is currently driven by two distinct but complementary breakthroughs: TTT for Context (solving the memory bottleneck) and TTT for Discovery (solving the search bottleneck).

TTT for Context

Jan 26th ~ Feb 2nd
#93 Latest AI Research Explained Simply

🗞️ Industry News in 1 Line

1. ♥ 34k Google DeepMind has released Project Genie, which is an experimental prototype that uses the Genie 3 world model to generate interactive virtual environments from text and visual prompts. This new tool allows users to design, edit, and explore their own worlds in real-time. Google subscribers in US can try it today on Google Labs.

   

2. ♥ 15k Claude has rolled out a suite of interactive integrations for paid subscribers. Csers can connect directly with tools like Slack, Figma, Asana, and Box. These updates enable users to draft messages, visualize diagrams, manage timelines, and query data from apps like Hex and Clay without leaving the interface.

   

3. ♥ 2.9k Z.ai has released GLM-OCR, a 0.9B-parameter OCR model tuned for complex documents (tables, formulas, code-heavy pages). They got SOTA results on major doc-understanding benchmarks and runs fast, up to 1.86 pages/sec on PDFs. Now available through API, their website, and HuggingFace.

   

   GLM-OCR benchmark

Quantization-Aware Distillation for NVFP4 Inference Accuracy Recovery

Xin et al. [NVIDIA]

♥ 4.3k   Quantization  

The race to make Artificial Intelligence more efficient is facing a common problem. As engineers try to shrink massive language models into ultra-fast, energy-saving 4-bit formats (specifically NVFP4), they often encounter a steep penalty: the compressed models become less intelligent.

Comparison of quantization-aware training (QAT) and quantization-aware distillation (QAD).

A common fix has been to simply retrain the compressed model, but this is no longer practical. Modern AI training has become a labyrinth of complex steps (such as supervised fine-tuning, reinforcement learning, and model merging) that is incredibly difficult to replicate perfectly. Researchers faced a common challenge: finding a way to preserve a model’s complex reasoning capabilities during compression without needing to retrace every complicated step of its original education.

The team discovered that a technique called Quantization-Aware Distillation (QAD) can change how compressed models recover their lost accuracy. Instead of forcing the compressed model to relearn tasks from raw data, the researchers set up a digital mentorship. The original, full-precision model acts as a "teacher," and the compressed 4-bit model acts as a "student." Through a mathematical process involving KL divergence, the student stops trying to solve problems from scratch and instead focuses entirely on mimicking the teacher’s exact output patterns.

Impact of training data on QAD for AceReason Nemotron 1.1 7B.

This method successfully restores compressed models to nearly the same accuracy as their full-sized counterparts. Since the student is learning behavior directly from the teacher rather than facts from a textbook, it does not require the original, high-quality training datasets. The researchers found that the student could recover its capabilities even when looking at partial data or synthetic information, making high-performance, energy-efficient AI far more accessible than previously thought.

Shaping capabilities with token-level data filtering

Rathi and Radford [Anthropic, Stanford]

♥ 882   Data filtering   bycloud’s pick  

Building safe artificial intelligence often feels like a cat-and-mouse game. Currently, developers train massive models on the entire internet, which inevitably means the AI learns dangerous information (like how to synthesize bioweapons) alongside useful facts.

Safety teams then try to suppress this knowledge after the fact, essentially putting a muzzle on the model. The problem is that because the dangerous knowledge is still buried deep inside the system, adversaries can often find ways to break the muzzle and retrieve the information.

Operationalizing token filtering.

Researchers recently tackled this foundational issue by asking a simple but difficult question: Is it possible to surgically remove dangerous concepts from the training data itself, so the model never learns them in the first place? They tested this by trying to teach an AI general biology while completely preventing it from learning medical advice, using this as a proxy for blocking dangerous capabilities.

Traditionally, if a training document contained harmful information, engineers would discard the entire file. This is a blunt instrument that wastes valuable context and unrelated knowledge. Instead, these researchers developed a method called token-level filtering. Think of it like a government redacting a classified file: instead of shredding the whole page, they simply take a black marker to the specific dangerous words and phrases while leaving the surrounding sentences visible.

Token filtering scales better than document filtering.

By identifying and hiding these specific pieces of information during the training process, they found they could effectively lobotomize the model’s ability to perform dangerous tasks without hurting its general intelligence or ability to understand related topics.

What makes this approach so hopeful is how well it scales. The study revealed that this surgical filtering actually becomes more effective as the AI models get larger and more powerful, creating a massive efficiency gap between the effort required to train a safe model versus the effort required to make it dangerous again.

Data filtering decreases MCQ performance on the forget domain without substantial damage to the retain domain.

Even more surprisingly, the researchers found that "forgetting" the data didn’t blind the model completely. These filtered models could still be easily trained to recognize and refuse questions about the forbidden topics, proving that an AI doesn't need to know how to build a bomb to know it should refuse to help you build one.

Reinforcement Learning via Self-Distillation

Hübotter et al. [ETH Zurich, Max Planck Institute for Intelligent Systems, MIT, Stanford]

♥ 320   LLM Scaling Law  

Current methods for teaching artificial intelligence to handle complex reasoning are like a particularly harsh school exam: the model attempts a math problem or writes code, and the system simply tells it "Pass" or "Fail." This approach, known as Reinforcement Learning with Verifiable Rewards, creates a significant bottleneck.

It is incredibly difficult for a model to figure out exactly which step in a long chain of reasoning caused the failure when the only feedback is a single score. Researchers realized that the software environments these models operate in usually offer much more detail (like error messages) that were being ignored.

Training progression of Olmo3-7B-Instruct on Chemistry.

To solve this, researchers developed a method called Self-Distillation Policy Optimization (SDPO), which allows the model to act as its own mentor. When the model generates an answer that fails, it takes the detailed feedback from the environment and re-evaluates its original attempt. With this new context, the model can retrospectively see exactly where it went wrong and calculate what it should have done differently. It then "distills" this insight back into its own network, effectively updating its behavior to match this wiser version of itself.

Self-Distilled Policy Optimization (SDPO) Loop

This approach transforms error messages from simple penalties into dense, actionable lessons. The study found that this self-teaching method allowed models to reach high levels of accuracy much faster than traditional methods, often requiring four times fewer attempts to reach the same performance. The models became more efficient reasoners as they learning to avoid the circular logic and verbal filler that often plague AI "thinking" processes.

Test-time self-distillation on hard coding problems.

TensorLens: End-to-End Transformer Analysis via High-Order Attention Tensors

Atad et al. [Tel Aviv University]

♥ 902   Transformer Interpretability  

If you try to understand how a massive, complex machine works, then you will probably be able to inspect one gear at a time. Researchers have the same problem in analyzing Transformer models. While scientists can examine individual "attention heads" or specific layers to guess how the model processes information, they have struggled to see the full picture.

Previous attempts to map the model's global behavior used rough averages or incomplete combinations of these parts, which ignored components like normalization or feed-forward networks.

Transformers are re-formulated as data controlled linear operators, characterized by an inputdependent high-order attention tensor T.

To solve this, the team introduced a new mathematical framework called TensorLens. Instead of treating the Transformer as a collection of disjointed parts, they successfully reformulated the entire architecture into a single, unified concept known as a high-order interaction tensor. This approach captures every element of the computation (including the attention mechanisms, feed-forward networks, activations, and residual connections) and expresses them as one cohesive "linear operator" that adapts based on the input data.

A schematic visualization of our method, where each sub-component of the transformer architecture

By replacing simple matrices with these high-order tensor structures, the researchers created a theoretically grounded way to represent exactly how the model transforms information from start to finish. Their validation showed that this method yields much richer and more accurate representations of the model’s behavior than previous aggregation techniques.

Perturbation Tests in NLP

Jan 20th ~ Jan 27th
#92 Latest AI Research Explained Simply

🗞️ Industry News in 1 Line

1. ♥ 11k Moltbot is THE personal AI assistant everyone imagined AI to be. Previously known as Clawdbot, it is an open source AI assistant that can connect messaging platforms like WhatsApp, Telegram, and Slack to a private gateway running on your own hardware. It can run seamlessly on Linux and Windows (via WSL2), and it even integrates with iMessage (requires macOS hardware) and automate everything for you. Just be aware of security when setting it up!

2. ♥ 44k Claude has released a beta integration that adds AI reasoning directly into your spreadsheets to analyze complex formulas and multi-tab dependencies. The tool allows you to debug errors, update assumptions, and build models while preserving your original structure, all accessible via a simple keyboard shortcut. Unlike standard chat interfaces, it provides transparent explanations with cell-level citations, ensuring you can verify the logic behind every calculation.

   

3. ♥ 6.1k Qwen has open-sourced the Qwen3-TTS family, which is a suite of high-performance speech generation models (1.7B and 0.6B) that deliver state-of-the-art voice cloning and design across 10 languages. It is built on a novel non-DiT architecture with Dual-Track modeling, the tool enables ultra-low latency streaming (outputting audio after a single character input) and allows for precise control over emotion, tone, and prosody via natural language instructions. Try it on Hugging Face or via the Alibaba Cloud API.

   

4. ♥ 4.4k Excalidraw has added a new "smarter, faster, stronger" text-to-diagram feature that uses a streaming chat interface for real-time visual generation. This update is accessible now within the free version of the tool (with some usage limits) and remains open-source on GitHub, though users may need to refresh their browser cache or use incognito mode to ensure they aren't loading the older build.

   

5. ♥ 7.4k Moonshot AI has released Kimi K2.5, an open-source visual agentic model that achieves state-of-the-art performance on major benchmarks, including HLE (50.2%) and BrowseComp (74.9%). The model excels at converting multimodal inputs (like chats, images, and videos) into fully functional, aesthetic websites and features a beta "Agent Swarm" capability that allows up to 100 self-directed sub-agents to collaborate on complex tasks 4.5× faster than single-agent setups. Try it now or download from Hugging Face.

   

Learn LLMs Intuitively - Intuitive AI Academy

Want to learn about LLMs, but never have a good place to start?

Intuitive AI Academy has the perfect starting point for you! We focus on building your intuition to understand LLMs, from transformer components, to post-training logic. All in one place.

We have just added a brand new chapter, an in-depth walkthrough of the current LoRA research landscape. Coming up next is MoE.

content overview (a total of 110k words explainer so far!)

We currently have a New Year New Me launch offer, where you would get 50% off yearly plan FOREVER for our early users. Use code: 2026

Check Out Intuitive AI Academy

Advertise with The AI Timeline!

Memorization Dynamics in Knowledge Distillation for Language Models

Borkar et al. [Meta Superintelligence Labs, Meta Central Applied Science, FAIR at Meta, Northeastern University, Carnegie Mellon University]

♥ 502   Knowledge Distillation 

Knowledge distillation is a process where a highly capable "teacher" AI helps train a smaller, more efficient "student" model. It allows us to capture the teacher's intelligence without the massive computational cost. However, large models are notorious for memorizing specific snippets of their training data, from phone numbers to unique sentences.

Experimental framework

The researchers of this paper want to know does the student simply parrot the teacher's memories, or can it learn to think like the teacher while forgetting the specific, sensitive data that was used for training?

Overlap of memorized examples.

The study shows that when a student model learns from a teacher, it actually acts as a privacy filter. These distilled models ended up memorizing significantly less training data, cutting the rate of memorization by more than half compared to models trained the standard way. It turns out that the student primarily holds onto "easy" information that is simple to compress and understand, while successfully rejecting the complex, specific examples that the teacher had memorized.

Shannon Entropy vs. Log-Probability Analysis.

By allowing the student to mimic the teacher’s uncertainty rather than forcing it to memorize hard answers, the distillation process naturally prevents the model from overfitting to specific training examples. The student effectively learns the general skills without retaining the exact data used to teach them.

Learning to Discover at Test Time

Yuksekgonul et al. [Stanford University, NVIDIA, Astera Institute, UC San Diego, Together AI]

♥ 680   LLM   bycloud’s pick  

Until now, most AI models have functioned like the student who stops learning once the test begins; they are "frozen" and can only search for answers based on past training. To succeed, they need to learn and adapt in the moment, internalizing new concepts from their own failed attempts.

TTT-Discover continues to train an LLM on a single problem at test time.

The researchers for this paper have developed a method called "Test-Time Training to Discover". Instead of trying to be good at everything on average, this approach allows the AI to become a hyper-specialist on a single problem. By using reinforcement learning during the test phase, the model treats its own search attempts as new training data, ignoring safe, average answers to hunt for the single, exceptional "eureka" moment.

In mathematics, the system improved upon the bounds of a problem posed by Erdős in 1955, discovering a complex, asymmetric solution that had eluded both human mathematicians and previous AI systems.

In computer engineering, it wrote code for GPU chips that ran significantly faster than kernels hand-optimized by human experts. It achieved these state-of-the-art results using an open-source model, outperforming larger, proprietary models simply by having the freedom to learn while it worked.

Results in two AtCoder Heuristic Competitions.

By shifting the focus from massive, static training to dynamic, problem-specific adaptation, this research has opened the door to solving highly specific challenges in biology, engineering, and mathematics that currently seem out of reach.

Toward Efficient Agents: Memory, Tool learning, and Planning

Yang et al. [Shanghai Artificial Intelligence Laboratory, Fudan University, University of Science and Technology of China, Shanghai Jiaotong University, Chinese Academy of Sciences, The Chinese University of Hong Kong (Shenzhen), Hong Kong Polytechnic University, Wuhan University, Tsinghua University]

♥ 96   LLM Agents Survey  

AI models can generate text, but we need "agents" capable of actively interacting with the world to execute complex workflows. However, this autonomy comes with a heavy price. Unlike a standard chatbot that answers a question and moves on, an agent operates in a recursive loop (planning, acting, remembering, and observing).

This creates a compounding accumulation of information, where the output of one step becomes the costly input for the next. This paper aims to solve the problem of "context window saturation" to ensure that AI systems remain sustainable, responsive, and accessible to everyone rather than becoming too computationally expensive to run.

From LLMs to agents: standalone reasoning to trajectory-level reasoning with memory, planning, and tool learning, while introducing additional cost sources.

This survey suggests that creating an "efficient agent" does not simply mean building a smaller model; it requires optimizing the entire system to maximize success while minimizing resource consumption. The researchers found that efficiency gains come from rethinking three specific areas: memory, tool usage, and planning.

In terms of memory, the field is moving away from forcing an agent to re-read every past interaction. Instead, new techniques allow agents to compress history into summaries or "latent" states, essentially giving the AI a working memory that retains the gist of a conversation without the computational weight of the raw text.

Efficient memory overview.

When it comes to using external tools, we should focus on optimizing how the agent interacts with its environment. Rather than testing tools sequentially or randomly, efficient agents are now using sophisticated retrieval systems to identify the correct tool instantly. Advanced agents can also perform parallel execution, where agents can run multiple tasks simultaneously rather than waiting for one to finish before starting another, drastically reducing the latency between a thought and an action.

Efficient tool learning comprises three stages

The researchers propose reimagining how agents plan their next moves by treating "thinking" as a limited resource. Instead of allowing for unbounded reasoning, new methods introduce the concept of "budgeted deliberation."

Overview of Efficient Planning.

By using reinforcement learning, systems are being trained to value brevity, effectively rewarding the agent not just for getting the correct answer, but for solving the problem with the fewest possible steps and the least amount of computational effort.

Jan 13th ~ Jan 19th
#91 Latest AI Research Explained Simply

🗞️ Industry News in 1 Line

1. ♥ 5.9k Z.ai has released GLM-4.7-Flash, a lightweight ~30B-class local model aimed at coding and agentic workflows, with support for long-context writing, translation, and roleplay. Weights are available on Hugging Face, and the API includes a free GLM-4.7-Flash tier (1 concurrency) plus a faster GLM-4.7-FlashX option.

   

2. ♥ 9.5k OpenAI says it will begin testing ads in ChatGPT in the coming weeks for logged-in adults on the Free and Go tiers in the U.S., with ads clearly separated and labeled and responses not influenced by advertising. OpenAI also says chats won’t be shared or sold to advertisers, and Pro, Business, and Enterprise tiers will remain ad-free

3. ♥ 6.9k Google has introduced Personal Intelligence inside Gemini App, an opt-in beta that lets it securely connect your Gmail, Photos, Search, and YouTube history to deliver more personalized help.

4. ♥ 7.4k Google DeepMind has released TranslateGemma, a new family of open translation models built on Gemma 3 with 55-language support in 4B, 12B, and 27B sizes. It’s designed for efficient, low-latency translation (including text-in-image) and is available on Hugging Face and Kaggle.

   

Learn LLMs Intuitively - Intuitive AI Academy

Want to learn about LLMs, but never have a good place to start?

Intuitive AI Academy has the perfect starting point for you! We focus on building your intuition to understand LLMs, from transformer components, to post-training logic. All in one place.

This is my latest project, and we will include write-up accompanying my latest videos on the frontier of AI research, all to help you keep up without the hassle of digging through papers and understanding math.

content overview (a total of 100k words explainer so far!)

We currently have a New Year New Me launch offer, where you would get 50% off yearly plan FOREVER for our early users. Use code: 2026

Check Out Intuitive AI Academy

Advertise with The AI Timeline!

Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings

Gelberg et al. [Sakana AI, University of Oxford]

♥ 1.7k   Positional Embeddings  

Teaching LLMs to process long documents is often very costly. Models are trained on shorter snippets of text to save computing power. But when they are asked to handle longer sequences, like analyzing a whole book instead of a chapter, they struggle because they rely on specific "positional embeddings" (mathematical tags that mark the order of words). Until now, fixing this required an expensive second phase of training to retune the model for longer contexts.

DroPE matches RoPE’s in-context perplexity.

This paper made a counterintuitive discovery that the tools used to help models learn word order are actually holding them back. The team found that while positional embeddings act as a guide during the initial learning process, they become a hindrance later on, which prevents the model from generalizing to lengths it hasn't seen before.

RoPE transformers have higher positional bias gradients at initialization.

By simply removing these embeddings after the initial training (a method they call DroPE), the model acts more flexibly. Surprisingly, once these rigid guides were removed and the model underwent a very brief recalibration, it could immediately understand and retrieve information from sequences far longer than anything it had seen before. This method effectively unlocks long-context capabilities without compromising the model’s performance on standard tasks.

Length generalization results on larger models.

By proving that complex fine-tuning isn't always necessary to upgrade a model's memory, this approach opens the door to systems that can digest vast amounts of information (from entire codebases to legal archives) at a fraction of the current cost.

GitHub - SakanaAI/DroPE: Extending the Context of Pretrained LLMs by Dropping Their Positional Embedding

Extending the Context of Pretrained LLMs by Dropping Their Positional Embedding - SakanaAI/DroPE

github.com/SakanaAI/DroPE

STEM: Scaling Transformers with Embedding Modules

Sadhukhan et al. [Carnegie Mellon University, Meta AI]

♥ 257   Transformers 

When we need a smarter LLM, we often try to get a bigger model, which makes them slow and expensive to run. To solve this problem, researchers are trying to build sparse models by teaching the model to activate only a small fraction of its "brain" for any given task.

However, current methods for doing this are notoriously fickle. They often suffer from training instability and create complex communication bottlenecks between computer chips, all while making the model’s decision-making process harder for humans to interpret.

Schematics of (a) SwiGLU FFN, (b) MoE FFN, and (c) STEM with a single prefetched token embedding.

The team developed a new architecture called STEM (Scaling Transformers with Embedding Modules) that simplifies how models process information. Instead of forcing every piece of data through the same complex mathematical transformations, the system uses a static, specialized approach.

Knowledge injection/edit demonstration.

By replacing a computationally heavy slice of the network with a simple, dedicated lookup table for each specific word, they successfully decoupled the model's storage capacity from its processing cost. This approach proved remarkably stable, effectively removing about one-third of the usual parameters required for these calculations while actually improving accuracy on knowledge-heavy tasks.

STEM-based knowledge editing schemes for length-mismatched source (ns) and target (nt) entity tokenizations.

Because this information is organized by specific tokens rather than hidden inside a "black box" of varying experts, the model became much more interpretable. The researchers found they could even "edit" the model's knowledge in a transparent way without retraining the entire system.

Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models

Cheng et al. [Peking University, DeepSeek-AI]

♥ 2.8k   LLM Memory   bycloud’s pick  

LLMs are getting better at complex reasoning, but they don’t have a built-in mechanism to simply "remember" fixed information. To recall a common phrase, a specific name, or a formulaic pattern, a standard Transformer model must spend valuable processing power computing the answer from scratch, effectively "thinking" its way to a memory every single time.

What if we gave the model a dedicated, instantly accessible memory bank for static facts, allowing its neural circuitry to focus entirely on difficult problem-solving? This approach acknowledges that language consists of two distinct parts (creative composition and rote knowledge) and suggests that our AI architectures should reflect that duality.

The team introduced "Engram," a sophisticated memory module that modernizes classic text-analysis techniques to function as a high-speed lookup table for the model. By analyzing how to balance this new memory system with traditional computation, they discovered a distinct "U-shaped" scaling law, pinpointing the optimal ratio between processing power and static memory.

When they built a large-scale model using these principles, the results were counterintuitive. While the system predictably improved at factual retrieval, it saw even larger gains in complex reasoning, mathematics, and coding. The researchers found that by offloading the "easy" work of recognizing local patterns to the Engram module, the model’s deeper layers were relieved of the burden of reconstruction.

This effectively freed up the network's attention mechanisms to process global context and tackle much harder cognitive tasks, leading to superior performance without increasing the computational budget.

Dr. Zero: Self-Evolving Search Agents without Training Data

Yue et al. [Meta Superintelligence Labs, University of Illinois Urbana-Champaign]

♥ 426   LLM RL  

Building bigger LLMs requires massive amounts of high-quality data curated by humans. But we are slowly running out of high-quality human data, particularly for complex tasks like researching the web to answer open-ended questions.

While AI has successfully taught itself to solve math problems or play chess (domains with clear rules), teaching a machine to navigate the messy, unstructured internet without human supervision has proven far more difficult.

Researchers at the University of Illinois Urbana-Champaign and Meta set out to solve this puzzle, asking if an AI could essentially pull itself up by its bootstraps to become a better researcher without seeing a single human example.

The self-evolving LLM training framework (Huang et al., 2025a) that iteratively trains a proposer and a solver with minimal supervision.

The team developed a framework called Dr. Zero, which creates a symbiotic relationship between two versions of the same artificial intelligence. They split the system into two distinct roles: a "Proposer" and a "Solver." The Proposer is tasked with inventing questions based on documents it finds, while the Solver uses a search engine to answer them.

The brilliance lies in how they learn from each other. The system uses a specialized feedback loop that rewards the Proposer only when it creates questions that are challenging but ultimately solvable. This creates an automated curriculum that gets progressively harder as the Solver gets smarter.

The Dr. Zero self-evolution feedback loop.

To make this computationally feasible, the team introduced a technique called Hop-Grouped Relative Policy Optimization (HRPO). This method cleverly groups similar questions to estimate their difficulty efficiently, allowing the model to learn complex, multi-step reasoning without requiring the massive computing power usually needed to test every possible outcome repeatedly.

System prompt for the proposer in Dr. Zero

The study revealed that this self-taught system often outperformed AI agents that were explicitly trained with human supervision. This suggests that complex reasoning and search capabilities can emerge naturally through self-evolution, without needing the expensive human datasets.

Jan 7th ~ Jan 13th
#90 Latest AI Research Explained Simply

🗞️ Industry News in 1 Line

1. ♥ 6.7k Midjourney has launched Niji V7, and it delivers significantly more realistic anime aesthetics alongside improved text rendering and coherence. Many users are already praising the stunning quality of the new model as it is nearly indistinguishable from Anime created by traditional artists.

   

2. ♥ 596 MiniMax has made a debut on the Hong Kong Stock Exchange and it achieved a staggering $13.7 billion valuation on its first day. MiniMax is one of the few open-source models dominating global benchmarks; it has an Anthropic compatible API that delivers top-tier performance, but you can also access the MiniMax agent on the web.

   

3. ♥ 486 Axiom has released an analysis of its AxiomProver model, which reveals that the AI successfully solved complex problems (like A6) that stumped human mathematicians, even while struggling with calculus concepts humans consider "obvious." The breakdown highlights a fascinating divergence in logic, where the model often ignores human elegance in favor of brute-force analysis or unexpected geometric strategies to construct valid Lean proofs. View a lean dependcy graph of how AI solved Putnam problems.

   

The Dead Salmons of AI Interpretability

Meloux et al. [Universite Grenoble Alpes, Icahn School of Medicine at Mount Sinai]

♥ 4.9k   Interpretability  

Many years ago, neuroscientists detected brain activity in a dead salmon, which was a mistake caused by statistical errors. Similarly, today, the researchers are arguing that the field of AI interpretability (which tries to explain how complex models think) is facing its own "dead salmon" moment.

As artificial intelligence becomes a part of our lives, we need to know if the methods we use to understand these systems are actually working. The authors of this study tried to determine whether current techniques are finding true insights into machine logic or simply hallucinating patterns in the noise.

The tuple of a target behavior PU and the computational system with its internal components form an SCM.

The team discovered that many popular methods used to interpret AI behave surprisingly well on neural networks that are completely random and untrained. Much like the salmon experiment, these tools generated plausible-sounding explanations for mathematical gibberish, which reveals that the field suffers from significant statistical fragility.

The researchers explained that this happens because current queries are often "non-identifiable," meaning convincing answers can be found even where no real logic exists. They propose that instead of just accepting an explanation at face value, we must treat it as a statistical estimate.

An interpretability task is defined by three elements: E, the hypothesis space; µ, the distribution over causal queries about the SCM (model and behavior); and D, the error measure.

This requires a new framework where findings are rigorously tested against random baselines to prove they describe a real computational mechanism rather than a statistical fluke.

From Entropy to Epiplexity: Rethinking Information for Computationally Bounded Intelligence

Zheng et al. [New York University]

♥ 1.9K   Entropy   Computationally Bounded Intelligence  

Traditional theories suggest that processing data deterministically cannot create new information, but despite this, AI systems like AlphaZero develop superhuman strategies without seeing human data, and models trained on synthetic data often outperform their predecessors.

Researchers realized that treating all observers as having infinite computing power (which is a standard mathematical assumption) misses the point. This paper tries to define how a computer with limited processing power actually perceives value in data to distinguish between random noise and the useful, learnable structures.

Illustration of random vs structural information.

The team introduced a concept called "epiplexity," a measure of the structural information a specific, resource-constrained observer can extract. They found that information isn't just about what is in the data, but how hard a computer has to work to decode it. When models are forced to solve harder problems (such as deducing the logic of a chess game from the board state rather than just predicting the next move), they acquire higher epiplexity.

Experiments on Factorization

This struggle forces the AI to construct richer internal programs and sophisticated mental models. The research also explains why language data, which is dense with logical rules, often builds more versatile intelligence than image data, which contains high randomness but less complex, learnable structure.

Information created with cellular automata.

This framework suggests that by measuring epiplexity, engineers could move beyond trial-and-error to scientifically select training data that maximizes learning.

GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization

Liu et al. [NVIDIA]

♥ 680   GDPO   RL  

As artificial intelligence evolves, users are no longer satisfied with models that simply output correct facts. We now expect them to juggle complex behaviors simultaneously (writing clean code, adhering to strict formatting, and remaining concise), all while being accurate.

However, teaching models to balance these competing actions has proven surprisingly difficult. The standard training technique, known as GRPO, tends to blur these distinct goals together. When researchers looked closely, they realized this approach causes a "signal collapse," where the mathematical feedback for different levels of success looks identical to the model.

This means that the AI becomes unable to distinguish between a partially correct attempt and a truly successful one, which often causes it to ignore harder tasks in favor of easier ones.

To solve this, researchers at NVIDIA and HKUST introduced a new method called Group reward-Decoupled Normalization Policy Optimization (GDPO). Instead of mixing every piece of feedback into a single, muddy signal, this approach processes each goal independently before combining them.

By normalizing rewards separately, the method preserves the fine-grained resolution of the training signal, ensuring the model understands that hitting two targets is numerically better than hitting just one. In tests spanning mathematical reasoning, coding, and tool usage, this clearer feedback allowed models to significantly outperform previous standards. The AI could finally balance strict constraints, such as keeping answers short, without sacrificing the accuracy of its reasoning.

This allows researchers to balance incentives used during training and fine-tune models to respect complex, multi-layered preferences without destabilizing the learning process.

How to Set the Learning Rate for Large-Scale Pre-training?

Zhou et al. [Shanghai AI Laboratory, Shanghai JiaoTong University, Fudan University]

♥ 450   LLM Learning Rate  

Training the massive AI systems of tomorrow is incredibly expensive, consuming vast amounts of time and computational power. One of the biggest headaches engineers face is setting the "learning rate"—essentially the speed at which the AI absorbs new information. If the rate is set too slow, training drags on inefficiently; if it is set too fast, the model gets confused and fails to learn. Until now, finding that "Goldilocks" speed for giant models has felt like a high-stakes guessing game, because running trial-and-error tests on such a massive scale costs a fortune. Researchers set out to solve this by determining if we can run small, cost-effective experiments to accurately predict the perfect settings for the big leagues.

Visualization of the optimal learning rate relative to model size N and data size D

The team compared two major strategies: trying to mechanically transfer settings from small models to big ones, versus using mathematical "scaling laws" to predict the best numbers based on trends. The results were illuminating. The researchers found that by analyzing the relationship between the model's size and the amount of data it consumes, they could derive a precise formula to predict the optimal learning speed. This "fitting" approach significantly outperformed older methods. Surprisingly, they also discovered that simpler is often better. While some theories suggest tweaking the learning speed for different parts of the AI’s "brain" separately, this study showed that a single, globally optimized speed works just as well. Modern AI architectures proved robust enough to learn effectively without needing complex, piece-by-piece micromanagement.

Performance comparison between the global optimal LR (red line) and module-wise optimal LR (blue line) on a 4B model trained for 120B tokens.

This discovery is a breath of fresh air for the future of AI development. It offers a reliable map for engineers, allowing them to skip the expensive, wasteful trial-and-error phase and move straight to efficient training. By understanding exactly how data volume and model size influence the learning process, developers can confidently scale up their systems to unprecedented sizes. It suggests a future where building more capable AI isn't just about having the biggest budget, but about leveraging the fundamental laws that govern how machines learn to build smarter and more sustainable systems.

Wait, Wait, Wait... Why Do Reasoning Models Loop?

Pipis et al. [MIT, Microsoft Research, University of Wisconsin-Madison]

♥ 731   Reasoning Models  

AI is evolving from simple chatbots to complex reasoning engines and a peculiar behavior has emerged: when faced with difficult math or logic puzzles, models sometimes get stuck in endless repetitive loops.

Researchers recently launched a deep dive to understand why these models get stuck in a rut. This paper tries to answer whether adding randomness (raising the "temperature") is a genuine solution or just a temporary band-aid. By investigating how smaller models learn from larger, smarter ones, the team sought to uncover the root causes of this stalling behavior to build more reliable thinkers.

Looping with greedy decoding.

The investigation revealed that looping is often a symptom of "risk aversion" born from imperfect learning. When a smaller "student" model tries to mimic a "teacher," it often fails to grasp the difficult, precise steps required to make progress. Instead, it retreats to safe, easy-to-learn actions, such as restating the problem or repeating a previous thought.

The researchers discovered that while dialing up the randomness does help break these loops, it does not actually fix the model's underlying confusion. Instead of learning the correct path, the model simply explores more chaotically until it potentially stumbles across the solution, resulting in reasoning chains that are much longer than necessary.

Temporally correlated errors induce low-temperature loops.

Now that we understand looping is caused by specific learning errors rather than just bad settings, engineers can design better training benchmarks. Additionally, instead of relying on randomness to shake models out of a loop, researchers can target these "hard-to-learn" steps.

Dec 31st ~ Jan 6th
#89 Latest AI Research Explained Simply

🗞️ Industry News in 1 Line

1. ♥ 7.2k NVIDIA has announced Alpamayo, a new “thinking, reasoning” autonomous-vehicle AI. The first rollout is slated to reach U.S. roads in Q1 2026, starting with the all-new Mercedes-Benz CLA. NVIDIA’s first model, Alpamayo 1 (10B parameters), uses video to generate driving trajectories and reasoning traces, now available on HuggingFace.

   

2. ♥ 17k Boston Dynamics has released a new video of its upgraded next-gen humanoid robot, Atlas, now fully electric with a 4-hour swappable battery for continuous operation. Atlas stands 6'2", weighs 198 lbs, has 56 degre

   es of freedom, can lift 110 lbs (66 lbs sustained), and reach 7.5 ft, using tactile, reconfigurable hands to adapt grip in real time.

   

3. ♥ 1k Liquid AI has released LFM2.5, its most capable family of tiny on-device foundation models (~1B class). Built on the LFM2 hybrid, device-optimized architecture, LFM2.5 scales pretraining from 10T → 28T tokens and expands RL post-training to improve instruction following. The initial open-weight lineup (including 1.2B Base/Instruct, plus vision-language and native audio-language variants) is available on HuggingFace.

   

Recursive Language Models

Zhang et al. [MIT CSAIL]

♥ 2k   LM Context  

LLMs are making significant progress and can handle several complex tasks but, they struggle when we try to process massive amounts of information in a context window, like trying to read a library of books simultaneously without losing the plot. Even the most advanced models suffer from a phenomenon known as "context rot," where their reasoning ability degrades as the context is used up overtime.

A Recursive Language Model (RLM) treats prompts as part of the environment.

This paper tries to determine if it is possible to enable AI to tackle long-horizon tasks involving millions of words without needing a bigger brain, but rather a smarter way to manage information.

The team introduced a concept called Recursive Language Models (RLMs). Instead of forcing a neural network to ingest a massive document all at once, this approach treats the text as an external part of the environment, much like a reference book sitting on a desk rather than a memory in one's head.

Performance comparison of different methods across long-context benchmarks of varying complexity.

The AI effectively acts as a programmer, writing code to peek into specific parts of the text, break complex problems into smaller chunks, and recursively call upon copies of itself to analyze those snippets. This strategy allowed models to successfully handle inputs up to two orders of magnitude larger than their designed limits.

On complex reasoning tasks, this method dramatically outperformed standard models and summarization techniques, while maintaining high accuracy even as the information load grew immense.

mHC: Manifold-Constrained Hyper-Connections

Xie et al. [DeepSeek AI]

♥ 3k   Residual Connection   bycloud’s pick  

Transformers lean heavily on residual connections because they keep information and gradients flowing cleanly through many layers. Hyper-Connections (HC) try to push this idea further by widening the residual stream into multiple parallel “streams” and learning how to mix them, so the model can exchange information across depth without increasing the core layer FLOPs much.

The problem is that the more freedom HC gives those cross-stream mixing matrices, the less it behaves like an identity path. When you stack many layers, the product of these unconstrained residual mixing matrices can amplify or shrink signals unpredictably, which shows up as training instability in large runs. In their 27B setup, the paper reports a loss surge for HC around 12k steps and extremely large composite “gain magnitudes” that can peak around 3000, a sign of exploding residual dynamics.

Their fix is Manifold-Constrained Hyper-Connections (mHC). Instead of letting the residual mixing matrix be anything, they project it onto the manifold of doubly stochastic matrices, meaning all entries are non-negative and every row and column sums to 1. That keeps the residual pathway closer to a stable identity-like behavior while still allowing streams to mix, since each stream becomes a convex combination of the others rather than an arbitrary linear remapping.

Illustrations of Residual Connection Paradigms.

Practically, they build this projection with the Sinkhorn-Knopp algorithm, running a limited number of iterations (they use 20) to turn an unconstrained matrix into an approximately doubly stochastic one. Because doubly stochastic matrices stay doubly stochastic under multiplication, the stability property should persist even when you multiply many layers’ residual mappings together, which is exactly where HC would tend to drift.

Visualizations of Learnable Mappings.

They also treat systems cost as part of the method. Widening the residual stream increases memory traffic and activation storage, so they add fused kernels, mixed precision kernels, and selective recomputation, and adjust pipeline overlap. With expansion rate n = 4, they report only about a 6.7% training time overhead after these optimizations.

Training Stability of Manifold-Constrained Hyper-Connections (mHC).

On results, mHC appears to keep HC’s accuracy benefits while avoiding its instability. In the 27B run, mHC reaches a final training loss reduction of 0.021 versus the baseline and keeps gradient norms closer to baseline behavior.

mHC benchmark against baseline

On downstream benchmarks, mHC beats the baseline across the board and usually edges out HC too, for example improving BBH and DROP relative to HC by about 2.1 and 2.3 points, respectively. On the stability metrics, the composite gain that could hit ~3000 in HC stays bounded around ~1.6 in mHC, which matches the paper’s story that constraining the residual topology can make this kind of widened residual stream scale more safely.

Efficient Context Scaling with LongCat ZigZag Attention

Zhang et al. [Meituan]

♥ 211   LLM Attention  

There is a bottleneck in how AI processes information, as models try to "read" longer documents (like entire books, legal archives, or massive codebases), the computational cost skyrockets because the system traditionally tries to pay equal attention to every single connection between every word.

The research team sought a way to break this inefficient cycle, aiming to create a model that can handle up to one million tokens of context without the computational weight that comes with it.

The illustration of LongCat ZigZag Attention (LoZA), which involves first calibration and then training for realizing the sparsity.

The team discovered a method called LongCat ZigZag Attention (LoZA), which effectively teaches the model how to "skim" intelligently without missing the details. By carefully calibrating the system, researchers identified which layers of the network were doing the most important work and which could be optimized.

The efficiency of LoZA. The relative cost and speed-up are practically measured on H20 clusters.

They found a streamlined structure hidden inside the larger model, converting about half of the attention mechanisms to a "sparse" mode that focuses only on essential information. They found that by retraining the model mid-process after this switch, they could lock in significant speed improvements while maintaining the same high level of intelligence and accuracy as the heavier, original models.

The effectiveness of LongCat-Flash-Exp-Chat across different context lengths on MRCR.

Evaluating Parameter Efficient Methods for RLVR

Yin et al. [Zhejiang University, HKUST, WUST, USTC, Brown University, Hong Kong Polytechnic University, INSAIT]

♥ 433   LLM RLVR  

As artificial intelligence moves from simply predicting the next word to solving complex mathematical problems, the training process has evolved. Researchers are increasingly relying on Reinforcement Learning with Verifiable Rewards (RLVR), a method where models improve by receiving a simple "correct" or "incorrect" signal on their reasoning. While this approach is powerful, retraining an entire massive model is incredibly expensive.

To save cost and time, the industry has largely settled on a specific efficiency shortcut known as LoRA (Low-Rank Adaptation). However, is this tool that everyone uses actually the best one for this specific type of learning, or are we leaving performance on the table by ignoring better alternatives?

The team discovered that the industry standard is suboptimal for reinforcement learning. By testing over a dozen different efficiency methods, they found that newer "structural" variants (approaches that change how weight updates are structured rather than just adding a simple adapter) consistently outperformed the default method. In some cases, these structural variants even surpassed the performance of full-parameter training, which is typically considered the gold standard.

A variety of PEFT methods are listed, each with its specific update formulation and initialization strategy. LN denotes Layernorm.

The study also revealed a fascinating mismatch in how models learn. Some advanced methods try to initialize training by focusing on the model's "loudest" or most significant existing features. The researchers found this causes the training to collapse because reinforcement learning actually thrives by tweaking the quieter, less dominant parts of the network.

Additionally, the team identified a strict limit to efficiency. While it is possible to freeze large portions of a model, attempting extreme compression bottlenecks the system. To learn complex reasoning, the model retains a need for a minimum amount of "plasticity," or trainable parameters, without which its ability to evolve stalls completely.

Comparison of accuracy and pass scores (all values are reported in percentages).

By moving away from the default adoption of standard LoRA and using structural variants like DoRA, engineers can build models that are not only computationally cheaper but also significantly smarter at math and logic.

Dec 23rd ~ Dec 30th
#88 Latest AI Research Explained Simply

🗞️ Industry News in 1 Line

1. ♥ 1.2k Z.ai is set for its IPO on Jan 8, 2026 on the Hong Kong Stock Exchange and set to raise $560 million at a valuation of 5.6 billion

2. ♥ 2.1k Liquid AI has announced the release of LFM2-2.6B-Exp, an experimental checkpoint built on LFM2-2.6B using pure reinforcement learning. The model delivers consistent gains in instruction following, knowledge, and math benchmarks, and outperforms other 3B models across these areas. Liquid AI also reports that its IFBench score surpasses DeepSeek R1-0528, despite being 263× smaller. Now on Hugging Face.

   

3. ♥ 2.4k Nvidia recently made a massive $20 billion deal with AI chip startup Groq, acquiring most of Groq's AI chip assets and licensing its inference tech (LPU), while also "acquihiring" Groq's key team, including founder Jonathan Ross, to boost Nvidia's AI inference performance, though Groq remains independent for its cloud services.

4. ♥ 1.4k MiniMax has announced the open-source release of MiniMax M2.1, a SoTA model for real-world dev workflows and agentic applications. It uses a MoE with 10B active parameters and 230B total, aiming to be faster to run and easier to deploy than comparable models. Rankings #1 among open-source models and #6 overall for web dev arena. Now on HuggingFace.

   

Decoupling the "What" and "Where" With Polar Coordinate Positional Embeddings

Gopalakrishnan et al. [The Swiss AI Lab (IDSIA), OpenAI, Center for Generative AI, University of Colorado]

♥ 1.3k   RoPE  

For modern AI to understand the world, it needs to track two distinct things: what a piece of information is and where it sits in a sequence. Rotary Position Embedding (RoPE) accidentally tangles the "what" and the "where" together, which confuses the model when it needs to make precise decisions based on just one of those factors. To solve this, the team developed a new approach called Polar Coordinate Position Embedding, or PoPE, designed to mathematically untangle these signals so the AI can process content and position independently.

How RoPE and PoPE encode relative positions via rotations of queries.

In the polar coordinate system, the magnitude of a signal represents the content, and the angle represents the position. PoPE reduces the confusion found in previous models. When tested against standard baselines, this new method shows superior performance across a diverse range of complex tasks, including the generation of classical music and modeling the human genome.

Zero-shot performance on downstream tasks using Transformer models pretrained on OpenWebText with RoPE or PoPE positional encoding.

The researchers found that models using PoPE demonstrated a remarkable ability to handle sequences ten times longer than those for which they were trained. Unlike current state-of-the-art methods that require complex fine-tuning to "stretch" a model's attention span, PoPE naturally generalized to these longer contexts immediately, proving to be both more robust and data-efficient.

Toward Training Superintelligent Software Agents through Self-Play SWE-RL

Wei et al. [Meta FAIR, Meta, TBD Lab, UIUC, CMU]

♥ 1.6k   LLM RL   bycloud’s pick  

Self-play SWE-RL (SSR) is a new framework designed to train superintelligent software engineering agents without relying on human-curated data. Current agents are limited by their dependence on finite resources, such as GitHub issues and manually written tests, which forces them to imitate human developers rather than discover new solutions.

Overview of Self-play SWE-RL.

To overcome this barrier, SSR allows a Large Language Model (LLM) to self-improve by interacting with raw, sandboxed code repositories. The system requires only the source code and dependencies, which eliminates the need for pre-existing test suites or natural language issue descriptions.

Bug-injection patches generated by code hunk removal (left) and historical change reversion (right).

The training process uses a single LLM alternating between two roles: a bug-injection agent and a bug-solving agent. The injection agent explores the repository to generate a "bug artifact," which consists of a bug-inducing patch, a custom test script, and a patch that weakens existing tests to hide the bug.

These valid bug artifacts are then passed to the solver agent, which attempts to fix the codebase using the strict test specifications defined by the injector. Failed attempts by the solver are converted into "higher-order bugs," creating an evolving curriculum that becomes increasingly complex as the agent improves.

Key consistency checks applied to validate bug artifacts, the full set described in the text.

SSR was tested on the SWE-bench Verified and SWE-Bench Pro benchmarks using the Code World Model (CWM) as a base. The results show that SSR achieves significant self-improvement (+10.4 and +7.8 points, respectively) and consistently outperforms baselines trained on human-curated data.

Meta-RL Induces Exploration in Language Agents

Jiang et al. [EPFL, ETH Zurich, Idiap Research Institute]

♥ 877   LLM RL  

AI can handle some complex tasks, but it struggles when we ask it to explore. While current LLMs can be trained via RL to solve specific problems, they often become rigid, and end up memorizing a single successful path rather than understanding how to adapt.

Comparison of RL and Meta-RL training on the MineSweeper environment.

When faced with a new or slightly changed environment, these agents frequently fail because they haven't learned how to learn from their mistakes. The research team sought to bridge this gap by designing a system that treats failure not as a dead end, but as a strategic investment. Their goal was to create agents that actively experiment with their surroundings and use that feedback to improve, mimicking the way a human might play a few practice rounds of a new game to understand the rules before trying to win.

Comparison between the training processes of RL (top) and Meta-RL used in LAMER (bottom).

The team introduced a framework called LAMER that fundamentally shifts the training objective from winning a single episode to maximizing success over a series of attempts. By analyzing how agents behave across multiple tries, the researchers found that their model learned to sacrifice immediate rewards in favor of gathering information.

The agent effectively "reflects" on its previous performance, using the context of past failures to adjust its strategy in real-time without needing complex mathematical updates to its core programming. In testing across diverse environments (ranging from logic puzzles like Minesweeper to web navigation tasks), this approach created agents that were significantly more successful and creative. Instead of collapsing into repetitive behaviors, these agents maintained a diverse set of strategies and proved capable of solving problems that standard reinforcement learning models simply could not handle.

Performance on Sokoban, MineSweeper and Webshop environments.

Step-DeepResearch Technical Report

StepFun

♥ 757   LLM Deep Research  

Researchers have identified an important distinction between simple search and true research. While current AI is excellent at answering specific, closed questions, it often stumbles when faced with open-ended projects that require long-term planning and logical structuring. It is challenging to create an agent that doesn't just retrieve links but understands the intent behind a request and can navigate the ambiguity of the real world.

Comprehensive Evaluation of Step-DeepResearch.

The team developed Step-DeepResearch, a framework that achieves expert-level performance without relying on massive, expensive computational resources. Instead of just feeding the model more data, they focused on training "atomic capabilities", fundamental skills like decomposing a complex problem, verifying information across multiple sources, and reflecting on mistakes in real-time.

By teaching the model to internalize this cognitive loop of planning, executing, and self-correcting, they created a medium-sized system that rivals the performance of the industry's largest proprietary models. It shows that a refined training strategy, which prioritizes decision-making and synthesis over raw size, can produce an agent that effectively navigates complex workflows to produce comprehensive reports.

Step-DeepResearch System Architecture.

Attention Is Not What You Need

CHONG [Meta, UT Austin, UCL, UC Berkeley, Harvard University, Periodic Labs]

♥ 1.1k   Attention Alternative  

If you know anything about how LLMs work, then you would have heard about Transformers. Transformers use "self-attention", which is a mechanism that requires every word in a sequence to continuously check its relationship with every other word.

Although it is incredibly effective, this process creates a massive, opaque web of calculations that becomes computationally expensive and notoriously difficult for humans to interpret. Researchers recently posed a provocative question: Is this expensive "attention" mechanism actually necessary for AI to reason, or is it just one inefficient way to achieve a goal?

To solve this, the team developed a new architecture called the Causal Grassmann Transformer, which completely removes the standard attention mechanism. Instead of building a massive grid of connections between all words, the model treats language processing as a flow through a specific mathematical landscape known as a Grassmann manifold. The system condenses information into lower dimensions and interprets the relationships between words as geometric subspaces.

This geometry-first approach proved that high performance doesn't require the traditional heavy machinery of self-attention. When tested on standard language modeling tasks, the simplified Grassmann model performed competitively with standard Transformers, achieving accuracy levels within a close margin of the established baselines.

This paper suggests that the future of language models may not rely solely on scaling up existing architectures, but rather on redesigning their mathematical foundations. By proving that "attention" can be replaced by "geometric evolution", this work opens the door to AI systems that are drastically more efficient and easier to audit.

Message from bycloud:

This will be the last issue of 2025! I hope you had a great holiday, and I wish you all the best in your endeavors in 2026!

Dec 15th ~ Dec 22nd
#87 Latest AI Research Explained Simply

🗞️ Industry News in 1 Line

1. ♥ 1.5k MiniMax has launched its M2.1 model, which is designed for complex, real-world tasks. In addition to coding, the model introduces powerful capabilities for office automation, enabling it to handle "digital employee" workflows and complex instructions. M2.1 is also more efficient, delivering faster, more concise responses that use fewer resources. You can access the M2.1 model now through the MiniMax API and as an open-source download on HuggingFace.

   

2. ♥ 2.8k Z.ai has launched GLM-4.7, a new AI model that delivers significant improvements in coding and reasoning capabilities. This update introduces advanced features like "Preserved Thinking," which allows the model to retain its reasoning process across multiple steps, making it more stable and effective for complex, long-term tasks. Developers can access the model now through the Z.ai platform and OpenRouter, or download it for local use via HuggingFace.
   

   

3. ♥ 1.1k Bytedance has introduced Seed-1.8, a generalized agentic model designed to efficiently handle complex, real-world tasks. This new release excels in multimodal processing and it supports both text and image inputs with strong performance in Graphical User Interface (GUI) interaction, coding, and information retrieval. The model can perceive video streams in real-time, perform non-blocking interactions, and utilize specific video tools to analyze details or extract highlights from long-form content.
   

   

4. ♥ 1.8k Xiaomi has released MiMo-V2-Flash, a new open-source AI model built for high-speed reasoning and coding tasks. This model uses a specialized "Mixture-of-Experts" architecture that delivers exceptionally fast responses (up to 150 tokens/s) while remaining highly cost-effective. With a massive context window capable of handling long documents and seamless integration with coding tools like Cursor and Claude Code, MiMo-V2-Flash is designed to act as an efficient digital assistant. You can access the model now through HuggingFace or Xiaomi's API platform.
   

   

Modernize your marketing with AdQuick

AdQuick unlocks the benefits of Out Of Home (OOH) advertising in a way no one else has. Approaching the problem with eyes to performance, created for marketers with the engineering excellence you’ve come to expect for the internet.

Marketers agree OOH is one of the best ways for building brand awareness, reaching new customers, and reinforcing your brand message. It’s just been difficult to scale. But with AdQuick, you can easily plan, deploy and measure campaigns just as easily as digital ads, making them a no-brainer to add to your team’s toolbox.

Learn more now.

Let's (not) just put things in Context: Test-Time Training for Long-Context LLMs

Bansal et al. [Meta, Harvard University, Kempner Institute at Harvard, OpenAI, UC Berkeley, UT Austin]

♥ 444   LLM Context  

AI models can theoretically process millions of words at once. However, there is a gap between what these models can read and what they can actually use. When you ask a model to find a specific "needle" in a massive "haystack" of text, it often fails, getting distracted by the sheer volume of information.

Until now, the industry standard solution has been to let the model "think" longer by generating more text before answering. But researchers have discovered that for truly long documents, simply generating more words doesn't help as the model’s internal attention mechanism gets overwhelmed.

This paper has identified a mathematical bottleneck called "score dilution". As a document gets longer, the "signal" of the correct answer gets drowned out by the "noise" of unrelated text. To fix this, they developed a technique called query-only test-time training (qTTT). Instead of asking the model to generate more text to solve a problem, this method allows the model to pause and perform a tiny, temporary update to its internal settings based specifically on the document it is reading.

A visual representation of how qTTT improves the logit margin.

It effectively lets the model "study" the context for a moment before answering. This approach proved far more effective than standard methods, and provided massive double-digit improvements in accuracy on difficult tasks like finding bugs in code or details in long records.

A Complete Guide to Spherical Equivariant Graph Transformers

Sophia Tang [University of Pennsylvania]

♥ 876   Protein Generation   bycloud’s pick  

For a long time, teaching AI to navigate the 3D world of atoms and molecules was a challenge. The laws of physics remain constant regardless of which way a molecule is facing, but the standard ML models often struggle to recognize a structure simply because it has been rotated. To solve this, researchers have developed a Spherical Equivariant Graph Neural Network.

Rather than treating molecular features as static lists of numbers, this framework uses "spherical tensors", which is a mathematical concept taken from quantum mechanics. The researchers proved that by representing data this way, the model can perform "equivariant" message-passing. This means that if a molecule rotates in virtual space, the model's internal calculations transform in perfect synchronization, and it preserves the correct physical relationships between atoms.

A feature vector f is split into its type-0, type-1, and type-2 components and arranged into a feature tensor with a tensor axis, a channel axis, and a tensor-component axis.

By adding specific geometric tools like spherical harmonics and Clebsch-Gordan coefficients, the architecture guarantees that the AI respects the rotational symmetries of the physical world without needing to be retrained on every possible orientation of a protein or chemical compound.

New Premium Insights release

For context, Premium Insights is where I write down longer form content that I think is interesting but not long enough to be make into YouTube videos.

Last week, I published the following blog:

Aug~Nov AI Research Trend ReportAuBasically recapping what I missed in the last 4 months g~mail.bycloud.ai/p/aug-nov-ai-research-trend-reportNov AI Research Trend Report

Basically recapping what I missed in the last 4 months

mail.bycloud.ai/p/aug-nov-ai-research-trend-report

Aug~Nov AI Research Trend Report

Basically recapping what I missed in the last 4 months

I spent a lot of time on this quarterly(?) AI research trend report (~4000 words), so don’t miss out!

Check It Out Now

Advertise with The AI Timeline!

T5Gemma 2: Seeing, Reading, and Understanding Longer

Zhang et al. [Google DeepMind]

♥ 1.4k   LLM  

AI can generate text, but it can’t see images, process multiple languages, and remember long conversations without losing the thread. Recent models have focused on "decoder-only" models, but we still don’t have a lightweight model that can handle all these tasks simultaneously.

Summary of pretraining (top) and post-training (bottom) performance for Gemma 3 and T5Gemma 2 at 270M, 1B and 4B over five capabilities.

This paper introduces T5Gemma 2, which is a new family of models that successfully adapts the powerful Gemma 3 foundation into this specialized encoder-decoder structure. They utilized a clever "recipe" that initializes the new system using pre-existing technology. This teaches a text-generator to become a better reader and observer.

Overview of T5Gemma 2. Encoder/decoder parameters are initialized from the pretrained decoder-only model, and then pretrained with UL2.

To make these models highly efficient, the researchers streamlined the internal machinery by merging different attention mechanisms into a single, unified module and sharing vocabulary tools across the system.

Detailed post-training results for Gemma 3, T5Gemma, and T5Gemma 2.

The results were impressive: despite being trained on shorter sequences of data, the models demonstrated a surprising ability to handle extremely long contexts, extrapolating well beyond their training wheels. Furthermore, even the smallest versions of the model proved capable of understanding images alongside text, matching or even outperforming their predecessors.

SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations

Karan and Du [Harvard University]

♥ 1.4k   LLM MoE Optimization  

We want better AI, but don’t want to pay millions of dollars to train it. Right now, one popular solution is to use the "Mixture of Experts" approach, which works by dividing a large neural network into many specialized sub-networks, activating only the few necessary ones for any given task.

This approach theoretically promises smarter and more efficient models, but as these experts become smaller and more numerous to improve precision, the computer chips running them struggle to keep up. The hardware begins spending more time simply moving data around than actually processing it.

Computational workflow of SonicMoE’s 8 launched kernels, grouped by yellow boxes.

This paper has introduced a new system called SonicMoE that changes how these complex networks use computer memory. They discovered that by intelligently reorganizing the mathematical operations required for training, they could drastically reduce the amount of temporary data the system needs to remember, which cuts the memory footprint nearly in half without losing any information.

Additionally, they introduced a clever "token rounding" strategy that ensures data is assigned to experts in perfectly sized chunks. Previously, the hardware would often waste energy processing empty filler space just to satisfy rigid computational requirements.

MoE Scaling Trends

By aligning the data flow with the physical design of the chips and performing data transfer and calculation simultaneously, the researchers were able to process information nearly twice as fast as previous methods on modern graphics processors.

Next-Embedding Prediction Makes Strong Vision Learners

Xu et al. [University of Michigan, New York University, Princeton University, University of Virginia]

♥ 624   Vision  

Teaching computers to "see" has required a different playbook than teaching them to read. While LLMs grew powerful by simply guessing the next word in a sentence, vision models rely on complex engineering tricks, such as reconstructing missing pixels like a puzzle or comparing thousands of image pairs to learn differences.

A team of researchers recently asked could the simple "predict what comes next" strategy that worked for NLP, work just as well for images? The team introduced a method called Next-Embedding Predictive Autoregression, or NEPA. Instead of asking the model to paint back missing parts of a picture pixel-by-pixel, they trained it to predict the abstract features (or "embeddings") of the next patch of an image based on what it has already seen.

In simple words, this is like asking a person to guess the meaning of the next puzzle piece before picking it up, rather than trying to draw the picture from memory. Remarkably, this simplified approach allowed a standard Transformer model to achieve top-tier accuracy on major classification benchmarks without needing complex decoders, specific visual vocabularies, or heavy data augmentation.

Comparison of different self-supervised learning frameworks on ImageNet-1K classification.

By focusing on predicting high-level information rather than raw details, the model naturally learned rich, transferable visual concepts that performed exceptionally well even on dense tasks like segmentation. This discovery suggests that the heavy architectural machinery often used in computer vision might be unnecessary.

Dec 8th ~ Dec 15th
#86 Latest AI Research Explained Simply

🗞️ Industry News in 1 Line

1. ♥ 1.3k Alibaba’s Qwen Team has released Qwen3-Omni-Flash, which is capable of processing text, audio, and video with seamless real-time responses. This upgraded iteration significantly enhances audio-visual interactions by resolving previous stability issues and offering precise control over system prompts and personas. Furthermore, it seamlessly integrates natural multi-turn video and audio understanding with indistinguishable human voices and fully customizable personalities, all backed by robust support for 119 text and 19 speech languages. Try it on Modelscope or HuggingFace.

   

2. ♥ 712 Ai2 has announced the release of Olmo 3.1, which is their most capable model to date with the new Think 32B and Instruct 32B variants. The Olmo 3.1 Think 32B model achieves significant performance gains in logic and reasoning benchmarks. They also launched updated 7B models optimized for math and code along with full weights, data, and training recipes for the entire suite. Try it online or download from HuggingFace.

   

3. ♥ 1k NVIDIA has launched the Nemotron 3 family with a suite of open models, datasets, and libraries designed to advance specialized agentic AI. Nemotron 3 Nano is a hybrid Mamba-Transformer mixture-of-experts (MoE) model that delivers highly efficient inference and a 1 million token context window while using only roughly 3B active parameters. NVIDIA has released the model under an open license alongside NeMo Gym, a new open-source reinforcement learning library for scalable agent training. If you are feeling adventurous, why not join the Nemotron Model Reasoning challenge?

   

New Premium Insights release

For context, Premium Insights is where I write down longer form content that I think is interesting but not long enough to be make into YouTube videos.

Last week I have published the below blog:

Aug~Nov AI Research Trend Report

Basically recapping what I missed in the last 4 months

mail.bycloud.ai/p/aug-nov-ai-research-trend-report

I spent a lot of time on this quarterly(?) AI research trend report (~4000 words), so don’t miss out!

Check It Out Now

Advertise with The AI Timeline!

One Layer Is Enough: Adapting Pretrained Visual Encoders for Image Generation

Gao et al. [Apple]

♥ 450   Image Generation  

Using powerful, pre-trained visual AI models with image generators has always been tricky. These understanding models produce rich, high-dimensional features, but today’s best generators need to work in a much smaller, more stable space to create images efficiently. This mismatch usually forces complex solutions. But a new method shows we might have been overcomplicating things.

Comparison between standard VAE, VA-VAE, RAE, and FAE.

Instead of forcing the generator to work with the bulky original features or building a complex translator, the method, called FAE, uses a minimal encoder (just a single attention layer) to gently compress those features into a compact, generation-friendly space. The real cleverness is in the double-decoder setup.

First, a dedicated decoder faithfully reconstructs the original high-quality features from this compact code. Then, a second, separate decoder uses those reconstructed features as its guide to generate the final image pixels. This separation of duties means the system retains the semantic understanding from the powerful pre-trained model while giving the image decoder the clear, low-dimensional signals it needs to work reliably.

An illustration of Training Stages of FAE. Stage Ia and Ib can be trained independently.

It can plug into different types of generators, like diffusion models or normalizing flows, and can use features from various popular pre-trained models like DINO or SigLIP. The results are impressive. On the standard ImageNet 256x256 benchmark, a diffusion model using FAE achieved top-tier image quality, scoring a near state-of-the-art FID of 1.29.

FID results of different models on MS-COCO validation (256 × 256).

Stronger Normalization-Free Transformers

Chen et al. [Princeton University, NYU, Carnegie Mellon University]

♥ 1k   Transformers   bycloud’s pick  

Normalization layers are a common trick to help deep neural networks train smoothly, but they come with a cost. They need extra computation to track statistics and can be sensitive to settings like batch size. Researchers have been looking for simpler, drop-in replacements, and point-wise functions like Dynamic Tanh (DyT) showed it was possible to match normalization performance.

Structure of Dynamic erf (Derf), a point-wise function, that outperforms normalization layers and other point-wise functions.

To find an optimal design, the researchers first identified what makes a point-wise function work well as a normalization replacement. They tested four key properties: the function should be zero-centered, bounded in its output, sensitive to small changes around zero, and monotonic (always increasing or decreasing). Functions that broke these rules often led to unstable training or worse performance. With these principles as a guide, they performed a large-scale search across many candidate S-shaped functions.

Results of zero-centeredness on ViT-Base.

The search identified a clear winner: a dynamically parameterized version of the error function, called Derf. This function, related to the Gaussian cumulative distribution, naturally has all the desired properties. When integrated into a model, it simply transforms each neuron's activation using learnable scaling and shifting parameters. The team tested Derf extensively, replacing normalization layers in Transformers for vision, speech, DNA, and language tasks. In nearly every case, Derf outperformed both standard normalization layers and the previous best alternative, DyT.

On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models

Zhang et al. [Carnegie Mellon University, Language Technologies Institute]

♥ 1.3k   LLM Reasoning  

We often see language models get better at reasoning after reinforcement learning, but it's hard to tell if RL is teaching them new skills or just polishing what they already learned in pre-training. To solve this, researchers built a fully controlled test using synthetic math problems. This let them isolate and study the distinct roles of pre-training, mid-training, and RL.

Overview of the data generation framework, task setup, and process-verified evaluation.

The study shows RL can create new reasoning ability, but only under specific conditions. If a task is already well understood from pre-training, RL just fine-tunes the model's existing skill. For RL to teach something genuinely new, the task must be slightly beyond the model's current ability. The model also needs a seed of knowledge from pre-training.

For example, to solve a problem in a new context like a "zoo" scenario, the model must have seen that context at least a little bit during pre-training. Even just 1% exposure gives RL enough material to work with and generalize from.

The researchers found that mixing mid-training with RL leads to the best generalization. Mid-training, which uses supervised learning on data from the model's edge of competence, builds a strong foundation. RL then explores from that foundation.

Allocating most compute to mid-training with a little RL is great for known tasks, while dedicating more budget to RL is better for tackling completely new, harder problems. Adding rewards for correct reasoning steps, not just the final answer, further reduced errors and improved the reliability of the model's solutions.

Interplay-LM-Reasoning (Interplay-LM-Reasoning)

Org profile for Interplay-LM-Reasoning on Hugging Face, the AI community building the future.

huggingface.co/Interplay-LM-Reasoning

In tests, a well-calibrated RL approach improved performance on harder tasks by up to 42%, and proper pre-training seeding allowed contextual generalization improvements of up to 60%. These findings show us that when we design training pipelines, we need to ensure broad pre-training coverage of basic concepts, use mid-training to build robust priors, and then apply RL to explore just beyond the model's current limits.

LLaDA2.0: Scaling Up Diffusion Language Models to 100B

Bie et al. [Ant Group, Renmin University of China, Zhejiang University, Westlake University, HongKong University of Science and Technology]

♥ 757   Diffusion LLM  

Today's powerful language models generate text one word at a time, which creates a bottleneck. This new approach aims to break that bottleneck by converting existing models into a different kind that can predict many words at once.

It uses a three-stage training method. Instead of building a parallel model from scratch, which is very costly, the researchers start with a strong existing model trained for sequential generation. They then carefully retrain it using a process called Warmup-Stable-Decay.

A schematic of the progressive training framework for transforming an AR model into a MDLM.

First, in the Warmup phase, the model slowly learns to reconstruct small, masked blocks of text instead of just the next word. The block size gradually increases until the model is reconstructing entire sequences at once in the Stable phase. Finally, in the Decay phase, the model is tuned back to working with smaller blocks, which makes it much faster for practical use while keeping its new parallel skills.

To make this conversion stable and efficient, the team introduced important techniques. They use a special document-level attention mask during training. This prevents the model from getting confused by attending to unrelated text when multiple documents are packed together for efficiency, ensuring it learns clean, coherent reconstructions. For the final instruction-tuning phase, they also apply a method called complementary masking. This clever trick ensures nearly every token in the training data contributes to the learning signal in each step, speeding up training and improving the model's grasp of language.

The final models, including a 100B-parameter version called LLaDA2.0-flash, show strong performance on reasoning, coding, and general knowledge benchmarks. It also shows a significant boost in inference speed, generating many more tokens per second than comparable auto-regressive models.

Closing the Train-Test Gap in World Models for Gradient-Based Planning

Parthasarathy et al. [Columbia University, New York University]

♥ 1.3k   LLM 

Gradient-based planning world models can be used for intelligent robot control, but in practice, its performance has often fallen short of slower, search-based alternatives. This is because of train-test mismatch: these models learn to predict the next state from expert demonstrations, but are later used to optimize sequences of actions, which often leads them into unfamiliar and unreliable territory.

Overview of our two proposed methods.

To bridge this gap, the researchers developed two clever fine-tuning methods. Online World Modeling tackles the problem of the model venturing into unknown states during planning. It works by using a simulator to correct the trajectories that gradient-based planning produces, and then training the model on these new, corrected paths.

This teaches the model to be accurate even for the non-expert actions it will encounter during optimization. Separately, Adversarial World Modeling focuses on smoothing the optimization landscape itself. It trains the model on deliberately perturbed versions of expert data, making the model more robust and creating a loss surface that is easier for gradient descent to navigate.

These techniques significantly close the train-test performance gap. In tests on object manipulation and navigation tasks, gradient-based planning with an adversarially fine-tuned model matched or exceeded the success rates of the powerful but computationally heavy Cross-Entropy Method (CEM) using only 10% of the computation time.

Table of Contents

• What the SoTA Is Doing

  • The Hybrid Attention Saga

• What other SoTA labs are doing

• The New Cool Architectural Ideas That I like

• New Unique Ideas that’s not architecturally bound

• New Insights That You Should Be Aware

• Non-LLM Research Bangers

• Conclusion/ending notes

Some of you might know that I was gone for 4 months. (August to November due to mandatory military service).

For those of you who don’t, well that’s why I am recapping all the papers properly that I missed in the last 4 months here. 

This list of papers are what I found to have fascinating ideas, potentially pivotal, or discuss/propose critical updates to the current AI research landscape.

The focus is as usual the LLM papers, but I do have some non-LLM papers too that are pretty big in their own field.

What the SoTA Is Doing

The Hybrid Attention Saga

In the 4 months I have been gone, the Chinese open source community has undergone a pretty active discussion surrounding hybrid attention. I think this tweet sums it up the best.

So what was the timeline? And can we figure out what’s actually going on?

The trajectory of attention mechanisms while I was gone had a period of quick optimism change for hybrid attention. Thanks to being open source, discussion was publicly shared through papers, blogs, or even X. For context, hybrid architectures are usually a combination of both linear attention & standard attention. For the sake of simplicity, I’ll only be covering the blogs or papers. X’s discussion was all over the place, without much empirical evidence to base off on.