July 1st ~ July 7th
#115 Latest AI Research Explained Simply

🗞️ Industry News in 1 Line

1. ♥ 2.2k Tencent has launched Hy3, a new 295-billion parameter Mixture of Experts model optimized for productivity and agentic workflows. This model is released under the Apache 2.0 license, and it provides an accessible and cost-effective option for commercial applications. You can try it on Hugging Face.

   

2. ♥ 1.4k Mistral has released Leanstral 1.5 (Le Chaton L∃∀N), a formal-reasoning model designed for mathematical theorem proving and code verification using the Lean language. The model demonstrates efficient test-time scaling on the PutnamBench benchmark and includes a translation pipeline to assist with verifying Rust code. You can try it on GitHub or Hugging Face.

   

3. ♥ 3.7k Meituan has introduced LongCat-2.0, a 1.6-trillion parameter Mixture of Experts model designed for agentic coding with a 1-million token context window. The model uses sparse attention and dynamic expert routing to optimize computational efficiency during multi-file codebase analysis and complex reasoning tasks.

   

Intuitive AI Academy - NEW Optimization Chapter!

My latest project: Intuitive AI Academy has the perfect starting point for you! We focus on building your intuition to understand LLMs, from transformer components, to post-training logic. All in one place.

We just added a new chapter on Optimization, that goes through the history, the key techniques, and the current state of optimizers that frontier model uses.

We currently have an exclusive newsletter offer, where you would get 40% off on the yearly plan for our users.

Use code: TIMELINE

Check Out Intuitive AI Academy

Advertise with The AI Timeline!

Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training

Zhang et al. [University of Minnesota, Peking University, Amazon]

♥ 22k   LLM Training  

We train AI models by updating every single layer of the network uniformly, assuming that every part must adapt together to make progress. However, this approach has kept us in the dark about where the learning actually takes place inside the model.

Layer contribution across all seven models studied in this work

To investigate this, researchers trained individual layers of various language models while freezing the rest of the network. They discovered that vast majority of RL gains are concentrated in just a small handful of layers located right in the middle of the model.

Layer contribution 𝒞(k) across model scales.

Training just one of these middle layers could recover almost all (and sometimes even exceed) the performance improvements achieved by updating the entire model. Meanwhile, the layers at the very beginning and the very end of the network contributed almost nothing to the final results.

Cross-dataset consistency of layer contribution on Qwen3-1.7B-Base.

This middle-focused behavior remained consistent across different model sizes, training datasets, and diverse tasks like mathematics, coding, and multi-step decision-making.

DOPD: Dual On-policy Distillation

Yu et al. [NUS, MMLab, PKU, Explore Academy]

♥ 424   Distillation   bycloud’s pick  

Training smaller AI models is a delicate art, especially when we try to assist the "teacher" model with extra hints that the student will not have access to in the real world.

When teacher models rely on privileged clues to guide a student, it often creates a "privilege illusion" where the student tries to copy the teacher’s final outputs without actually understanding the underlying logic. This information mismatch confuses the student, leading to unstable training and a decline in its independent reasoning abilities.

Performance comparison of our DOPD with competing approaches across eight benchmarks in terms of average across all benchmarks (upper bigger bars) and individual values of each benchmark (lower small bars).

To address this challenge, researchers developed a framework called Dual On-policy Distillation, or DOPD. The system works by analyzing what they call the "privilege advantage gap". This metric compares the prediction confidence of the teacher and the student when both are given the same privileged hints. By evaluating this gap on a token-by-token basis, the framework can pinpoint exactly which parts of a response represent genuine, transferable reasoning skills and which parts are merely shortcuts derived from the extra clues.

Comparison of existing (a) standard distillation, (b) self distillation, and (c) adaptive distillation paradigms with our proposed (d) dual distillation paradigm.

Once these learning scenarios are identified, the system routes different teaching strategies to different tokens. For words where the teacher demonstrates a clear capability advantage, the student receives strong, full-vocabulary guidance to absorb that deeper logic.

Comparison of (a) performance and (b) entropy on OPD variants with privileged information. Here, T., S., and Priv. denote teacher policy, student policy and with privileged information, respectively.

On the other hand, tokens that are either highly obvious or too uncertain, the system dials back the teacher's influence and instead uses the student's own internal consistency as a stabilizing guide.

Orca: The World is in Your Mind

Kim et al. [Orca Team, Beijing Academy of Artificial Intelligence]

♥ 430   World Models  

Most of today's AI systems are highly specialized, focusing strictly on predicting the next word, the next video frame, or the next robotic motion in isolation. This fragmented approach prevents them from developing a unified, common-sense understanding of physical cause and effect. To address this, researchers developed Orca, a foundational world model designed to learn a unified, hidden representation of physical transitions.

The Orca’s overall framework.

Orca builds this understanding using two complementary learning pathways. The first is an "unconscious" learning process, where the model watches thousands of hours of continuous, unlabeled video to quietly absorb the dense, natural laws of how objects move and scenes change.

Overview of Encoder

The second is a "conscious" pathway, which pairs video segments with language annotations to help the model connect these physical transitions to explicit human concepts, causal relationships, and task instructions.

Overview of pre-training data

To prove that this learned internal map is actually useful, the researchers froze Orca’s core architecture and attached lightweight, trainable decoders to translate its hidden states into text, predicted future images, and physical robot actions.

AdaJEPA: An Adaptive Latent World Model

Karan and Du [Harvard University]

♥ 855   LLM World Models  

Most AI "world models" are kept frozen after their initial training. When these models encounter minor real-world variations, such as unexpected lighting, shifting friction, or novel obstacles, their predictions quickly drift, causing their entire planning process to fail.

AdaJEPA performs test-time adaptation during closed-loop MPC

To solve this, researchers developed AdaJEPA, an adaptive framework that introduces a continuous loop of planning, acting, adapting, and replanning. Instead of blindly relying on static pre-trained assumptions, this system executes a planned action, observes the actual outcome, and immediately uses that real-world feedback as a self-supervised signal to recalibrate its internal model.

(a)AdaJEPA: Plan–Act–Adapt–Replan Loop

By performing lightweight, single-step updates to just a small subset of its parameters right before the next plan is made, the system corrects its course dynamically without needing any extra expert guidance.

The researchers discovered that this real-time tuning gives steady improvements in planning success across a variety of challenging tasks, including navigating unfamiliar mazes and manipulating objects with unseen shapes or altered physics.

Planning Success under Shape Shifts (top) and Visual Shifts (bottom)

Program-as-Weights: A Programming Paradigm for Fuzzy Functions

Zhang et al. [Meta, UT Austin, UCL, UC Berkeley, Harvard University, Periodic Labs]

♥ 805   LLM Weights  

Many common programming tasks, such as filtering system logs, repairing malformed data, or sorting search results by intent, are too subjective for traditional, rigid code, but calling giant cloud-based models for every minor task is expensive, slow, and difficult to keep consistent.

Overview of the Program-as-Weights paradigm

To solve this, researchers have designed a system called Program-as-Weights, or PAW. Instead of querying a massive cloud artificial intelligence every time a user inputs data, PAW uses a specialized compiler to translate a developer’s natural-language instructions into a tiny, reusable file of custom neural network weights.

Text-to-LoRA instantiation of PAW

This small file acts as a specialized adapter that can be instantly loaded onto a lightweight, frozen interpreter model already installed on the user's local device. In essence, the heavy lifting of understanding the task happens only once during compilation, generating a compact, private tool that can be called repeatedly offline.

Interactively test the compiled program. Users can provide test inputs and inspect the corresponding outputs, enabling rapid validation and refinement before download

To make this possible, the researchers trained their compiler on a massive new dataset containing ten million examples across hundreds of text-processing categories. They found that a tiny local interpreter running these compiled programs outperformed a model over fifty times its size that was prompted in the traditional way.

This model was executed on standard consumer laptops, and it achieved fast execution speeds while requiring only about one-fiftieth of the memory typically needed by larger systems.

June 23rd ~ June 30th
#114 Latest AI Research Explained Simply

🗞️ Industry News in 1 Line

1. ♥ 39k OpenAI has announced a limited preview of its new GPT-5.6 model family, which includes the flagship Sol, the balanced Terra, and the cost-efficient Luna. The flagship Sol model can handle cybersecurity tasks and complex command-line workflows. It is best suited for long-horizon security tasks including vulnerability research and exploitation.

   

2. ♥ 6.7k The Ornith-1.0 family of open-source models (ranging from 9B dense to 397B MoE parameters) specializes in agentic coding tasks. These models are post-trained on Gemma 4 and Qwen 3.5, and they use a reinforcement learning strategy that jointly optimizes task-specific scaffolds and solution rollouts to improve coding outcomes. You can try it on Hugging Face.

   

3. ♥ 4.7k Alibaba's Qwen team has open-sourced AgentWorldBench, a seven-domain benchmark for environment simulation, alongside Qwen-AgentWorld-35B-A3B, a Mixture-of-Experts model designed for world modeling. This uses two approaches: using the world model as a controllable simulator for reinforcement learning, and internalizing environment prediction directly within the agent foundation model. You can try it on GitHub or Hugging Face.

   

4. ♥ HOT Anthropic has introduced Claude Sonnet 5, its most agentic Sonnet model yet. The model can make plans, use tools like browsers and terminals, and complete complex coding, reasoning, and knowledge-work tasks with more autonomy than previous Sonnet models. It is best suited for developers and teams who need strong agentic performance at a lower cost than larger Opus-class models.

   

Intuitive AI Academy - NEW Optimization Chapter!

My latest project: Intuitive AI Academy has the perfect starting point for you! We focus on building your intuition to understand LLMs, from transformer components, to post-training logic. All in one place.

We just added a new chapter on Optimization, that goes through the history, the key techniques, and the current state of optimizers that frontier model uses.

We currently have an exclusive newsletter offer, where you would get 40% off on the yearly plan for our users.

Use code: TIMELINE

Check Out Intuitive AI Academy

Advertise with The AI Timeline!

Tapered Language Models

Bayat et al. [Mila, Cornell University, Université de Montréal, CIFAR AI Chair]

♥ 151   LLM architecture  

Currently, almost all LLMs distribute their learning capacity uniformly across every layer in their architecture, treating early and late layers identically. However, evidence suggests that later layers do less heavy lifting and mostly refine what earlier layers have already figured out, making this uniform resource distribution highly inefficient.

Tapering MLP width improves perplexity at no additional parameter or compute cost

To address this mismatch, researchers introduced "Tapered Language Models," an architectural principle that gradually reduces processing capacity across the depth of the model. Instead of keeping the internal width of the model's primary processing units constant, they used a smooth cosine schedule to front-load capacity in the early layers and taper it down toward the end.

Front-loading MLP capacity improves perplexity

Under a strictly fixed parameter budget, this design ensures that the model concentrates its resources where they are needed most, achieving better results without adding any extra training or inference computing costs.

The researchers found that this tapering method consistently improves text prediction and reasoning performance across various model sizes and different foundational AI architectures.

Layer updates become more aligned with the residual stream at greater depths.

By analyzing how information flows through the network, they discovered that earlier layers write highly novel features into the model's memory stream, while later layers yield diminishing returns by reinforcing existing data.

Improved Large Language Diffusion Models

Nie et al. [Gaoling School of Artificial Intelligence, Beijing Key Laboratory of Research on Large Models and Intelligent Governance, Engineering Research Center of Next-Generation Intelligent Search and Recommendation, ByteDance Seed]

♥ 574   Diffusion LMs   bycloud’s pick  

Currently, most LLMs generate text strictly one word at a time from left to right, which limits their ability to plan ahead. While bidirectional "diffusion" models can look in both directions at once to solve this, they have historically struggled to match the performance of their traditional, step-by-step counterparts.

Benchmark Results of Base Models.

To bridge this gap, researchers developed iLLaDA, an eight-billion-parameter model trained entirely from scratch using fully bidirectional attention. Instead of predicting the very next word, this model learns by predicting missing words anywhere in a sequence, using context from both the left and the right simultaneously.

Benchmark Results of Instruct Models.

By scaling its pre-training to twelve trillion tokens and fine-tuning the system on a twenty-five-billion-token instruction dataset across twelve epochs, the researchers demonstrated that this bidirectional approach can be highly competitive. They also optimized the model's efficiency by using grouped-query attention to manage memory and tying internal parameters together to keep the model compact.

While there is still room for improvement on complex instruction-following tasks compared to some of the most advanced traditional models.

Autodata: An agentic data scientist to create high quality synthetic data

Kulikov et al. [FAIR at Meta]

♥ 466   LLM Training Data  

Finding a way to automatedly create rich training data could unlock the next generation of capable models. However, existing automated data creation methods often generate examples that are either too easy or too difficult.

Autodata creation of CS research questions

To address this, researchers introduced a framework called Autodata, which trains an AI agent to act like a data scientist. Instead of just churning out data based on a static prompt, this agent enters a creative cycle: it generates training tasks, evaluates how well different AI models perform on them, analyzes the results, and refines its approach to build better data in the next round.

Meta-optimization of the data scientist agent

This paper introduces Agentic Self-Instruct system which uses a team of virtual subagents, including a "challenger" to write questions, a "judge" to score them, and both "weak" and "strong" model versions to test them.

The results across diverse areas like computer science research, law, and scientific reasoning are highly encouraging. Models trained on this carefully calibrated data showed substantial performance gains, proving that this agentic loop creates much more robust training signals than traditional automated methods.

You Don't Need to Run Every Eval

Zeng and Papailiopoulos [Harvard University]

♥ 1k   LLM Benchmarks  

Evaluating new AI models has become incredibly expensive and time-consuming, often costing thousands of dollars per run. Currently, researchers run dozens of independent benchmarks to track an AI's progress and compare different design choices, which creates a massive computational bottleneck.

In this paper, researchers analyzed a large matrix of 84 frontier models evaluated on 133 different benchmarks. They discovered something surprising: this complex landscape is approximately "rank-2," meaning an AI’s scores across all these diverse tests are largely determined by just two underlying factors.

The team built a tool called BENCHPRESS. Instead of running over a hundred tests, a developer can run just five key "probe" benchmarks (such as tests focusing on graduate-level reasoning, coding, and general knowledge) and the system can reconstruct the remaining scores with remarkable accuracy. Even when using a more budget-friendly set of five cheaper benchmarks, the tool still estimates the missing scores to within a few percentage points of their true values.

To ensure these shortcuts are safe to use, the researchers also created a reliability layer. This companion system evaluates how much different mathematical models disagree on a prediction, alongside how much similar data is already known about related models and benchmarks.

DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation

Cheng et al. [Peking University, DeepSeek-AI]

♥ 2.8k   LLM Decoders  

LLMs are slow and their word-by-word generation mechanism is a bottleneck for real-time applications like conversational assistants. Speculative decoding speeds this up by letting a small "drafter" model guess upcoming words for a large model to verify, but current drafters either guess too slowly or generate disconnected, incoherent word sequences.

The DSpark architecture and decoding cycle

To solve this, researchers developed DSpark, a framework that introduces a clever "semi-autoregressive" architecture. DSpark first uses a highly efficient parallel backbone to generate a block of word guesses all at once, keeping the drafting process incredibly fast. To prevent these parallel guesses from becoming disconnected and error-prone, DSpark passes them through a lightweight sequential module. This extra step injects local transition information, to make sure the words flow naturally together and reducing the likelihood of awkward phrasing that the larger model would ultimately have to reject.

In addition to smarter drafting, DSpark introduces confidence-scheduled verification to optimize system efficiency. The framework provides a confidence head that evaluates the survival probability of each guessed word in a sequence.

Load-adaptive throughput and verification budgets

In live tests, this balanced approach accelerated generation speeds for individual users by 60% to 85% compared to established baselines, showing that smarter scheduling can make advanced AI systems significantly more practical for real-world deployment.

June 17th ~ June 23rd
#113 Latest AI Research Explained Simply

🗞️ Industry News in 1 Line

1. ♥ 12k Z.ai has announced the release of GLM-5.2, an open-weights model under the MIT license with 1-million-token context window and updates in coding, reasoning, and agentic tasks. It has two reasoning effort levels [GLM-5.2 (max) and GLM-5.2 (high)] designed to help developers balance task performance with token efficiency. You can try it on Hugging Face.

   

2. ♥ 1k Poolside has released the base and post-trained weights for Laguna M.1, a model featuring a 256K context length licensed under Apache 2.0. Alongside the model checkpoints, the team has also released "pool" (an agent harness) designed to let developers run the model locally as a coding agent. You can try it on Hugging Face.

3. ♥ 37k Sakana AI has announced Sakana Fugu, a multi-agent orchestration system accessible through a single model API. The underlying Fugu Ultra model is trained to recursively call and coordinate various LLMs in an agent pool to manage complex, multi-step technical workflows. You can try it on the Sakana AI website.

   

Intuitive AI Academy - NEW Optimization Chapter!

My latest project: Intuitive AI Academy has the perfect starting point for you! We focus on building your intuition to understand LLMs, from transformer components, to post-training logic. All in one place.

We just added a new chapter on Optimization, that goes through the history, the key techniques, and the current state of optimizers that frontier model uses.

We currently have an exclusive newsletter offer, where you would get 40% off on the yearly plan for our users.

Use code: TIMELINE

Check Out Intuitive AI Academy

Advertise with The AI Timeline!

Looped World Models

Lu et al. [FaceMind Research Asia]

♥ 473   World Models  

AI models that can navigate the physical world are called "world models". These models allocate the same amount of computing power to every single moment, whether an object is simply sitting still or dynamically colliding with another. Moreover, these models are too slow and computationally expensive to run on smaller, everyday devices.

The overall framework of our proposed Looped World Models (LoopWM).

To solve this, this paper introduced Looped World Models, or LoopWM. Instead of using a giant, resource-heavy stack of unique layers, this architecture uses a smaller, shared set of layers and runs them repeatedly in a loop to refine its predictions.

Because the model reuses its existing components rather than adding new ones, it achieves the predictive quality of much larger networks with a fraction of the parameters.

This looping structure also allows the model to adjust its computing power on the fly. It can run fewer loops during simple, predictable events and automatically allocate more loops to complex scenarios like physical collisions.

Relative increase over Qwen3.7-max on automatic online performance, compared against baselines.

Furthermore, the architecture introduces "deferred decoding," which allows the system to process a sequence of actions entirely in its abstract internal language. Rather than wasting energy rendering the visual details of every intermediate step, it only decodes the final result at the very end. This combination of parameter efficiency and smart resource allocation offers a promising, practical path forward for real-time AI planning.

Variable-Width Transformers

Wu et al. [MIT-IBM Watson AI Lab]

♥ 437   Transformers   bycloud’s pick  

When we scale LLMs, we assumes that every layer requires the same computational budget, even though different stages of a model's processing pipeline perform entirely different roles.

> <former, where different layers have different widths

This paper introduces an hourglass-shaped transformer architecture known as the >< former. This design keeps the initial and final layers wide but significantly narrows the layers in the middle.

The effect of the bottleneck layer index and dimension on language modeling loss, parameterized as a ratio to the total number of layers and the base dimension.

To handle the fluctuating widths without losing information or adding complex math, the researchers implemented a clever, parameter-free residual resizing mechanism. The model maintains a wide global residual stream where narrower middle layers only read from and write to a specific slice. The remaining inactive dimensions are simply carried forward, bypassing the narrow layers entirely so they can be restored later in the network.

Language modeling loss vs. pre-training FLOPs (left) and average layer size (right). > <former produces lower loss at smaller FLOP and average layer size costs.

Across various model sizes, the ><former consistently outperformed traditional, uniform-width baselines in language modeling. By optimizing how space is used, the architecture achieved a twenty-two percent reduction in training computations and a fifteen percent reduction in memory and input-output costs for the key-value cache.

Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers

Movahedi et al. [ELLIS Institute Tübingen, Max Planck Institute for Intelligent Systems, Tübingen AI Center, ETH Zurich, Swiss Institute of Bioinformatics, Université Paris Cité, Liquid AI]

♥ 363   Transformers  

When solving a difficult puzzle like a complex maze or a challenging Sudoku, humans naturally spend more time thinking. For AI, scaling its computational effort based on the difficulty of a task is hard.

Signal propagation and adaptivity, FPRM vs. TRM:

Some researchers use looped neural networks, which process information by running it through the same internal layers repeatedly. However, these looped models face a double-edged sword: looping too many times can destabilize the mathematical signals inside the network, and it is notoriously difficult for the model to naturally decide when it has "thought" enough and should stop.

The blessing and the curse of depth in Looped Transformers.

To address these challenges, researchers designed a framework called the Fixed-Point Reasoning Model, or FPRM. The team resolved the stability issue that affects deeply looped networks by restructuring how the model balances its internal signals. By switching to a pre-normalization setup paired with residual scaling, they successfully kept the model's internal data stable and trainable, even when running through many loops.

FPRM architecture

Most importantly, FPRM introduces a self-contained halting mechanism based on mathematical convergence. The model loops until its internal representations settle into a stable state, known as a fixed point instead of relying on an external, complex module to decide when to stop.

ExpRL: Exploratory RL for LLM Mid-Training

Xiang et al. [Stanford University, Carnegie Mellon University, OpenAI,]

♥ 855   LLM RL  

RL helps AI models solve complex reasoning problems, but training collapses when problems are too difficult because the model rarely stumbles upon the correct final answer to receive any feedback. Without a way to reward partial progress and smart intermediate steps, AI models cannot easily discover the creative strategies needed to solve challenging math and science problems.

Exploratory RL (ExpRL)

To solve this, researchers developed a method called ExpRL, which stands for Exploratory Reinforcement Learning. Instead of forcing the AI to blindly copy human solutions, this approach keeps the correct answer hidden from the AI while it attempts a problem.

Pass@k after training with ExpRL on HMMT-Nov-2025 (128 samples).

An automated judge then uses that hidden answer as a customized grading rubric to evaluate the AI's step-by-step thinking. By analyzing these drafts, the judge can award points for productive intermediate progress. This turns a simple pass-fail test into a learning experience that guides the AI's exploration.

ExpRL training dynamics during Stage-I.

The researchers found that this supportive warm-up phase prepares the AI for subsequent, more rigorous training. Compared to traditional fine-tuning or basic right-or-wrong feedback, models prepared with this method showed a much broader diversity of problem-solving strategies.

Behavior changes after RL priming relative to the base model

The AI naturally began to exhibit helpful thinking habits, such as double-checking its work, self-correcting mistakes mid-calculation, and backtracking when a strategy failed.

How Transparent is DiffusionGemma?

Engels et al. [Google DeepMind]

♥ 210   DIffusionLM  

Understanding how AI arrives at its decisions is vital for ensuring safety, debugging errors, and preventing misuse. While traditional LLMs think out loud step-by-step in plain English, newer text diffusion models like DiffusionGemma perform their reasoning in a continuous mathematical space.

A simplified architecture diagram of DiffusionGemma, with the first two denoising steps shown

Researchers tried to translate these hidden mathematical states back into human-readable concepts. They discovered that the complex vector information flowing between the model's iterative denoising steps can actually be mapped into a small bottleneck of natural language tokens.

Breakdown of intermediate state top token identities with different restrictions averaged across WildChat prompts

Restricting the model to just a handful of these mapped words at each step caused almost no drop in its overall performance. By showing that these intermediate states represent interpretable guesses about the final text, the researchers successfully demonstrated that the model’s hidden reasoning depth can be simplified to a level nearly identical to traditional systems.

DiffusionGemma thinks less than Gemma on all monitorability evaluations. Error bars show 95% confidence intervals of mean number of characters in each model’s chain of thought.

The researchers watched the model accurately estimate its final response length before choosing its words, retroactively correct earlier mistakes after formulating its reasoning, and even temporarily weigh multiple sentences at once.

Most importantly, evaluations showed that these complex internal mechanics do not compromise safety; the model remains just as monitorable as traditional architectures.

June 8th ~ June 16th
#112 Latest AI Research Explained Simply

🗞️ Industry News in 1 Line

1. ♥ 13k Moonshot AI has released Kimi-K2.7-Code, which is an open-source coding model with improved instruction following, coding, and agent capabilities compared to its predecessor, K2.6. The updated model shows benchmark gains, including a 21.8% increase on the Kimi Code Bench v2, while reducing reasoning-token usage by 30% for greater efficiency. You can try it on Kimi Platform.

   

2. ♥ 8.5k Z.ai has announced the rollout of GLM-5.2, its new flagship model with advanced coding capabilities, a 1-million-token context window, and configurable reasoning modes. The model is currently available to GLM Coding Plan subscribers and is scheduled to be officially open-sourced under the MIT License next week.

3. ♥ 88k Anthropic has suspended access to its Fable 5 and Mythos 5 models after a US government export control directive citing national security concerns. The restriction impacts all global customers and limits access for foreign nationals. While Anthropic is working to address the regulation, you can explore other open-source models.

4. ♥ 1.4k Google has introduced DiffusionGemma, a new 26B Mixture of Experts open model designed to explore text diffusion techniques. By generating entire 256-token blocks simultaneously, the model achieves up to four times faster inference on GPUs, though its overall output quality is lower than standard Gemma 4 models. You can try it on Model Garden or Hugging Face.

   

Intuitive AI Academy - NEW Optimization Chapter!

My latest project: Intuitive AI Academy has the perfect starting point for you! We focus on building your intuition to understand LLMs, from transformer components, to post-training logic. All in one place.

We just added a new chapter on Optimization, that goes through the history, the key techniques, and the current state of optimizers that frontier model uses.

We currently have an exclusive newsletter offer, where you would get 40% off on the yearly plan for our users.

Use code: TIMELINE

Check Out Intuitive AI Academy

Advertise with The AI Timeline!

End-to-End Context Compression at Scale

Li et al. [New York University, Modal Labs, University of Maryland, Princeton University, Columbia University, Harvard University, Lawrence Livermore National Laboratory, FAIR at Meta]

♥ 430   LLM Context  

There is a fundamental bottleneck in AI: memory. When LLMs try to read massive documents or entire software codebases, the system's memory footprint and processing time increase exponentially.

GitHub - LeonLixyz/LCLM: latent context language models

latent context language models. Contribute to LeonLixyz/LCLM development by creating an account on GitHub.

GitHub

Until now, the main workaround has been to selectively forget information by trimming the system's internal memory cache. However, this approach is often remarkably slow, computationally unstable, or degrades the model’s intelligence.

Examples of the three data types used to train LCLMs

This paper created Latent Context Language Models, which rethinks how machines ingest information. Instead of forcing the main AI engine to read a sprawling mountain of text word by word, researchers placed a smaller, highly efficient encoder model in front of it.

A from-scratch pre-training sweep identifies the best encoder-decoder compressor architecture.

This encoder acts like a brilliant summarizer. It processes large blocks of text and mathematically compresses them into a much shorter sequence of dense representations called soft tokens. An adapter then translates these soft tokens into a format the main model natively understands. By handling the heavy lifting upfront, the main model processes a fraction of the data.

LCLMs can use tools to retrieve compressed context and improve exact string-match accuracy.

This breakthrough shifts the limits of what models can handle. The researchers found that this architecture beautifully reduces peak memory usage and the time it takes to generate an answer, all while preserving the system's baseline intelligence.

latent-context (Latent Context Language Model)

Org profile for Latent Context Language Model on Hugging Face, the AI community building the future.

huggingface.co/latent-context

Test-Time Gradient Guidance of Flow Policies in Reinforcement Learning

Zhou et al. [UC Berkeley, Physical Intelligence]

♥ 1k   RL

Researchers have been struggling to scale up reinforcement learning because of messy training dynamic where an "actor" system learns to make decisions while a "critic" system simultaneously learns to judge them. Training these two together is highly unstable, especially when using advanced generative models that create complex actions step-by-step.

Illustrative example of 1D denoising process mapping Gaussian noise to a tri-modal distribution

If the critic model updates, the actor gets confused, which makes it incredibly difficult to build larger, smarter robotic control systems. Researchers wanted to know if they could skip this chaotic paired training altogether.

The researchers discovered a remarkably elegant workaround called Q-Guided Flow. Instead of forcing the actor and critic to learn together, they trained them separately. When the AI is actually running (a phase called test time) it generates actions through a gradual process of removing noise. Normally, asking the critic for directions during these noisy, half-finished steps causes computational confusion or requires massively expensive backward math.

However, the researchers found a brilliant shortcut. At each step, the system makes a rapid mathematical guess of what the final, perfectly clean action will look like. It shows this clean guess to the critic, receives reliable feedback, and uses that insight to gently steer the ongoing action toward a higher-value outcome.

Offline RL performance at 500k training steps (20 tasks, 10 seeds)

By applying this simple guidance exactly when the AI is acting, the resulting models are faster, cheaper to run, and smoothly scale up to handle much harder tasks without breaking.

MiniMax Sparse Attention

Lai et al. [MiniMax, Peking University, NVIDIA, Zhejiang University, Huazhong University of Science and Technology, Nanjing University, Hangzhou Dianzi University]

♥ 511   LLM Attention   bycloud’s pick  

Overview of MSA

This paper introduces a new technique called MiniMax Sparse Attention, in this Instead of forcing the AI to expend heavy processing power analyzing every single word in its vast memory simultaneously, researchers designed a highly efficient filtering system. They built a lightweight indexer that quickly scans the data in blocks.

It identifies only the most important information relevant to the current task (while always keeping the most recent context active to maintain stability) and skips the rest. Once these blocks are selected, the model’s main engine focuses its full attention exclusively on them.

Efficiency comparison between GQA and MSA under the shared experimental model configuration.

When researchers tested this on a massive, 109-billion parameter model, this method maintained the high quality of traditional approaches but drastically slashed the workload. The system absorbs initial data fourteen times faster and generates new responses more than seven times faster.

Trajectory-Refined Distillation

Jiang et al. [McGill University, Mila Quebec AI Institute, UT Austin]

♥ 360   LLM Distillation  

Researchers are building smarter AI models by pairing a smaller "student" model with a highly advanced "teacher" model. The student practices solving a problem step-by-step, and the teacher grades its output word-by-word. However, researchers recently identified a major structural roadblock in this process called "prefix failure."

TRD refines student-generated trajectories yo into improved trajectories yr, which are then used for distillation.

Let’s imagine a student taking a complex math test and making a logical error in the very first step. Because AI models generate text sequentially, once the student goes down this wrong path, the rest of the answer is doomed. When this happens, the teacher model gets mathematically confused, awkwardly trying to correct the student word-by-word while still trapped in the student's flawed train of thought.

Under prefix failure, the teacher distribution becomes a mixture with two modes.

Until now, fixes merely involved ignoring or adjusting the penalties for these individual bad words, completely failing to address the reality that the underlying logic was already hopelessly broken.

OPD Avg@16 results (%) using Qwen3-8B as the teacher

To solve this, researchers introduced a highly promising approach called Trajectory-Refined Distillation. Rather than relying on word-level nitpicking, this method steps back to look at the big picture. Instead of grading a bad thought process, the system takes the student's initial attempt and allows the teacher to gently revise the entire reasoning path into a cohesive, corrected draft before the actual learning occurs.

Trajectory analysis

By correcting foundational missteps at their source, the student model is exposed to completely new, valid ways to reason through complex problems.

FlashMemory-DeepSeek-V4: Lightning Index Ultra-Long Context via Lookahead Sparse Attention

Wang et al. [Independent Researchers, Tencent, The Hong Kong University of Science and Technology (Guangzhou), Tsinghua University]

♥ 219   LLM Attention  

To generate a single word of response, LLMs must keep that entire massive history actively loaded in its working memory. Researchers realized this creates a severe hardware bottleneck. They discovered a striking inefficiency: most of the time, models processing massive contexts only need the most recent sliver of information to form a response.

Architectural overview of LSA vs. CSA.

This paper introduces a new framework called Lookahead Sparse Attention. Instead of passively forcing the AI to carry the full weight of its history, they introduced a "Neural Memory Indexer." Every few steps, this indexer evaluates the AI's current thought process, dynamically predicting and fetching only the critical historical chunks needed for the immediate future.

The researchers managed to train this lightweight indexer entirely independently from the massive core model, bypassing tremendous computational costs and allowing it to be optimized on its own.

By loading only what is strictly necessary, the system shrinks the active memory footprint down to just 13.5 percent of traditional models. At extreme context scales of half a million tokens, memory overhead drops by over ninety percent.

June 3rd ~ June 9th
#111 Latest AI Research Explained Simply

🗞️ Industry News in 1 Line

1. ♥ 12k Google has introduced Gemma 4 12B, an encoder-free multimodal model with Apache 2.0 license that is compact enough to run locally on just 16GB of VRAM. By using a lightweight 35M-parameter vision embedding module rather than a traditional encoder, it delivers advanced, multi-step reasoning capabilities with a significantly reduced memory footprint. You can try it out now on Ollama, Google AI Edge Gallery App, the Google AI Edge Eloquent app or Hugging Face.

   

2. ♥ 3.9k Alibaba has introduced Qwen3.7-Plus, a new model that delivers text performance which matches Max-tier models across various benchmarks. The release also brings systematic enhancements to its visual understanding capabilities, which are specifically tailored for powering complex multimodal interactive hybrid and browser agents.

   

3. ♥ 3.4k NVIDIA has released Nemotron 3 Ultra, a new model optimized for complex agentic tasks such as long-horizon planning, large-scale code analysis, and extensive tool calling. The release provides developers with access to the model weights, synthetic data, and post-training recipes tailored for popular agent frameworks. You can try it on Hugging Face.

   

4. ♥ 1.1k Cognition has announced an "AI Productivity Guarantee" for its AI agent Devin. They are pledging to fund customer usage up to $10 million if the tool fails to deliver proportional engineering value. To support this initiative, the company detailed a newly developed measurement system that estimates an agent's productive output by comparing it to the time a human engineer would require to complete the same task within an enterprise codebase. Read more about gurantee.

Intuitive AI Academy - NEW Optimization Chapter!

My latest project: Intuitive AI Academy has the perfect starting point for you! We focus on building your intuition to understand LLMs, from transformer components, to post-training logic. All in one place.

We just added a new chapter on Optimization, that goes through the history, the key techniques, and the current state of optimizers that frontier model uses.

We currently have an exclusive newsletter offer, where you would get 40% off on the yearly plan for our users.

Use code: TIMELINE

Check Out Intuitive AI Academy

Advertise with The AI Timeline!

MAI-Thinking-1: Building a Hill-Climbing Machine

The Microsoft AI Team

♥ 3.8k   LLM release  

Researchers are trying to build AI models, but just building a single impressive model isn’t the end goal. We need to figure out how to continually and reliably improve them. We need to create a system that genuinely learns to reason from the ground up, rather than taking the common shortcut of imitating older AIs.

Overview of the MAI-Base-1 architecture

To solve this, researchers designed what they call a "hill-climbing machine." This is a fully integrated system where data, hardware, and safety tests work together in a continuous loop, supporting a steady, measurable climb toward better reasoning.

The first milestone from this new approach is a powerful model called MAI-Thinking-1. Researchers trained it entirely on a pristine collection of human knowledge, including public code, academic papers, and books, while carefully removing synthetic, AI-generated content.

Pipeline for processing HTML pre-training data.

To do this, the model undergoes a rigorous reinforcement learning climb. During this phase, it learns how to connect chains of thought to solve complex problems, use external tools, and carefully balance the desire to be helpful with the strict necessity of staying safe.

Rank non-invariance in data mixture scaling.

By relying purely on authentic human data and systematic testing, MAI-Thinking-1 has proven itself to be one of the most capable models of its size for complex math, science, and software engineering.

Cosmos 3: Omnimodal World Models for Physical AI

NVIDIA

♥ 2.7K   World Model

Let’s imagine a robot that can clear your dining table. To accomplish this seemingly simple task, today's robots must juggle a disjointed collection of artificial intelligence models: one to see the dishes, another to plan movements, and another one to predict what happens if a glass gets bumped.

Cosmos 3 serves as a general-purpose backbone for Physical AI.

Training these physical agents directly in the real world is slow, expensive, and sometimes dangerous. We need to create a single, unified system that handles seeing, reasoning, simulating, and acting.

To solve this, this paper introduced Cosmos 3, which is a versatile model that unites language, video, audio, and physical actions into one continuous stream of thought. Instead of treating sight, sound, and movement as separate programs, this architecture translates all these senses into a shared digital vocabulary.

Cosmos 3 offers a strong starting point for training Physical AI agents.

It is built using a dual-pathway structure. One side focuses on deeply understanding the current environment, analyzing the scene much like a person reads a book. The other side acts as an imagination engine, generating plausible futures. When the AI considers a physical action, like grasping a cup or turning a steering wheel, the system instantly visualizes exactly how the physical world will react.

Unified action representation

This unified mind allows developers to generate rich, interactive simulations to safely train future robots. It proves that when AI can cohesively see, hear, think, and act, bringing helpful autonomous agents into our daily lives becomes a beautiful, tangible reality.

Self-Trained Verification for Training- and Test-Time Self-Improvement

Wu and Raghunathan [Carnegie Mellon University]

♥ 424   LLM Self-training  

To master advanced reasoning, the AI models need to learn how to catch their own mistakes. We use a system where an AI proposes an answer and an internal "verifier" provides feedback. This mechanism works to some extent, but the verifier cannot spot hidden logical flaws, the system just reinforces its own errors.

Isolating the contribution of trained feedback using ground-truth verdict.

To solve this, the researchers used a simple logic: diagnosing a mistake is much easier when you already hold the answer key. They developed a new method called Self-Trained Verification.

First, they let a "teacher" version of the verifier look at both the AI's flawed attempt and the correct reference solution. With the right answer in hand, the teacher easily identified exactly where the logic failed. Then, they trained the standard verifier to imitate this insightful feedback without ever peeking at the answer key. By matching the teacher's deep critique, the standard verifier learned to catch subtle errors entirely on its own.

Overview of self-trained verification.

When using this self-trained verifier on exceptionally hard math problems, researchers saw accuracy increased dramatically. On complex scientific reasoning tasks, success rates rocketed from just over one percent to twenty-one percent.

Self-Trained Verification for Training- and Test-Time Self-Improvement

A reference-conditioned teacher distills feedback into the verifier. Doubles accuracy on hard math, 14x on the hardest scientific reasoning, and lifts standalone pass@1 30% past where RL had converged.

ar-forum.github.io/stv-webpage

If LLMs Have Human-Like Attributes, Then So Does Age of Empires II

Wynter [Microsoft & The University of York]

♥ 11K   LLM SuperIntelligence   bycloud’s pick  

We want to achieve superintelligence and build AI which possess generalized, intrinsic anthropomorphic attributes (e.g., empathy, self-awareness, anxiety, morality). When researchers try to check if a "mind" exists inside the machine, it leads to flawed, circular, or scientifically uninformative conclusions.

To prove their point, the authors take a highly unconventional approach: they demonstrate that a neural network can be trained inside the 1999 video game Age of Empires II (AoE II).

Let’s think of a fully functioning LLM built entirely out of AoE II goats running across a map. If you input "I feel lonely," the goats will shuffle around and eventually output the text: "I feel bad for you, maybe catch up with a friend."

• Because you are watching digital goats do math, you would never assume the goats are actually experiencing empathy or understanding.

• However, when a standard LLM outputs the exact same text via a sleek chat interface, users and researchers readily ascribe human emotions to it.

• This proves that anthropomorphism is an illusion driven by presentation/substrate, not an intrinsic property of the model's mathematics.

Ansatz-based training algorithm for our 1-bit perceptron, as a circuit (top) and as an AoE II implementation.

The authors argue that standard scientific methods fail when researchers assume a model possesses human traits beforehand (the "Accept/Reject Computational Theory of Mind" framework):

• Positive Results: If you assume an LLM has empathy and design a test to measure it, a positive result just circularly confirms your underlying assumption.

• Negative Results: If the LLM fails the test, the result is completely ambiguous. You don't know if the LLM lacks empathy, or if your test was just poorly designed.

The authors suggest abandoning the debate over whether LLMs have "minds" when conducting empirical research. Instead, researchers should adopt a "null assumption." This means studying an LLM's outputs purely causally and behaviorally rather than jumping to ascribe anthropomorphic intent (a "Geist") to the system.

Robots Need More than VLA and World Models

Karcini et al. [Motoniq.ai, Stanford University, Istituto Italiano di Tecnologia, ETH Zurich, Technical University of Darmstadt, UCL Centre for AI]

♥ 582   LLM Word Models  

We have seen text-based AI models trained on the entire internet, but general purpose robots have been stuck learning the hard way. Until now, researchers relied almost entirely on collecting explicitly labeled robot data.

Researchers need to physically guide a robot arm through a task thousands of times just to teach it one simple chore. This process is expensive, slow, and carries the constant risk of hardware damage. Researchers realized that if we want robots to achieve the leaps we have seen in language AI, simply building larger neural networks is an incomplete solution.

Next generation robotics will come from advances that go well beyond scaling vision language action (VLA) models.

The internet has a lot of unstructured behavioral data, like countless internet videos of people interacting with everyday objects. These videos contain rich information about how things move, how forces work, and what a successful task looks like. However, a robot cannot directly use this information because it does not know which of its specific motors to turn to replicate what a human hand does.

To solve this problem, the researchers suggest building a new physical intelligence stack to translate this abundant data. This system can automatically label unstructured behaviors, map human motions onto different robot bodies, predict physical consequences using 3D world models, and allow robots to understand success just by watching.

May 26th ~ June 2nd
#110 Latest AI Research Explained Simply

🗞️ Industry News in 1 Line

1. ♥ 1.5k StepFun has released Step 3.7 Flash, a 198B sparse Mixture-of-Experts model designed to optimize agentic, coding, and multimodal workflows with approximately 11B active parameters. The model offers 256K context support, high-performance tool use, and optimized speed, making it capable of running locally on specialized hardware. You can try it on GitHub or HuggingFace.

   

2. ♥ 3.7k Liquid AI has released LFM2.5-8B-A1B, a hybrid Mixture-of-Experts model that significantly upgrades training data to 38T tokens and expands context length to 128k. Designed for agentic workflows, the model enhances instruction following and tool-use capabilities while maintaining efficient local performance across diverse hardware. You can try it on Liquid playground or Hugging Face.

   

3. ♥ 67k Anthropic has released Claude Opus 4.8, which introduces improved judgment, enhanced self-assessment capabilities, and a "fast mode" that offers 2.5x speed at a lower price point. The update also brings dynamic workflows to Claude Code, allowing the model to manage complex, multi-file tasks by deploying parallel subagents.

   

4. ♥ 529 Tencent has released Hy-MT2, a new open-source multilingual translation model available in sizes ranging from 1.8B to 30B parameters. The series features impressive efficiency, with the 1.8B version leveraging extreme quantization to run locally on mobile devices while outperforming several mainstream commercial APIs. You can try it on GitHub or Hugging Face.

   

Intuitive AI Academy - NEW Advanced RL Chapter!

My latest project Intuitive AI Academy has the perfect starting point for you! We cover everything from the basics, like transformer architecture, all the way to more advanced topics like LoRA, distillation, Mixture of Experts, and RLHF.

The goal is simple: make frontier AI systems easy to understand with clear explanations, visuals, interactive learning, and a structured path from fundamentals to cutting-edge techniques.

We have just added a new advanced RL chapter, that includes the basics of RL and the current state of RLHF! We currently have an special newsletter offer, where you would get 40% off on the yearly plan! 

Use code: TIMELINE

Check Out Intuitive AI Academy

Advertise with The AI Timeline!

DiffusionBlocks: Block-wise Neural Network Training via Diffusion Interpretation

Shing et al. [Sakana AI]

♥ 2.2k   LLM Training  

A lot of GPU memory is used when training neural networks because standard backprop has to store activations through every layer. This becomes a bottleneck as models get deeper, since memory grows with depth. DiffusionBlocks asks whether we can train only small chunks of the model at a time, without using the usual fragile local objectives that made older block-wise training methods perform badly.

The core idea is to reinterpret residual layers as steps in a diffusion denoising process. Instead of training the whole transformer end-to-end, the model is split into blocks, and each block is assigned a specific noise range. Each block then learns to denoise within that range independently, so training only needs gradients for one block at a time.

The research showed that this works across very different architectures, not just toy classification models. On CIFAR-100, DiffusionBlocks got 59.30% accuracy versus 60.25% for standard ViT while only training 4 layers at a time. For image generation, it matched or improved DiT results on CIFAR-10 and ImageNet with around 3× memory reduction. For autoregressive language modeling, it even improved LM1B MAUVE from 0.50 to 0.71 while training only 3 layers at a time.

The most interesting part is that the blocks are not just random chunks. They use equi-probability partitioning, meaning more capacity is placed around the intermediate noise levels where denoising is hardest, instead of wasting equal space on easy noise regions. This beat uniform partitioning in the ablations. But there is still a tradeoff: moderate block counts like 2 or 3 worked best, while too many blocks hurt quality because each block gets too little capacity.

A Bitter Lesson for Data Filtering

Mohri et al. [Stanford University]

♥ 1.2k   LLM data curation  

We spend a lot of resources to train LLMs, filtering web data to remove "low-quality" or noisy text. This seems intuitive, but filtering throws away a vast majority of the internet's text. This creates a bottleneck because modern models require trillions of words to keep improving. This paper explores whether expensive, human-designed filters are truly necessary as computational power scales, or if models can learn to navigate the web's messy reality on their own.

670M-token CC pool versus junk-injected versions.

The research showed that with enough computing power and sufficiently large models, the optimal data filter is actually no filter at all. While smaller models struggle with cluttered datasets, larger models trained for longer periods are highly robust. They not only tolerate noisy text but actually benefit from seemingly "poor" data. To test this, researchers injected scrambled documents with completely randomized word orders into the training pool.

1B model performance as we vary the pool size; the total needed steps for pool to outperform RefinedWeb grows rapidly. Crossing point as a function of pool size for various model sizes

Surprisingly, the larger models still succeeded, even benefiting from the shuffled text. Because the original words remained, the models could still learn which terms frequently co-occur, helping them build associations despite the chaotic structure.

The team concluded that as training budgets scale toward the high-compute limits of the near future, training directly on massive, unfiltered web pools will likely become the most effective strategy.

Neural Weight Norm = Kolmogorov Complexity

Musat [ETH Zürich]

♥ 1.1K   Complexity   bycloud’s pick  

It is common to penalize large weights as it helps neural networks generalize to new data, but classical learning theories cannot fully explain why. Because a network trained with weight decay has the same theoretical capacity as one trained without, traditional mathematics struggles to distinguish between a model that genuinely learns and one that merely memorizes noise.

Comparison of weight-norm-vs-complexity bounds. “Two-sided” indicates whether both directions of a sandwich are proved.

This paper tries to build a mathematical bridge connecting weight decay to Solomonoff’s universal prior, the theoretically optimal but historically uncomputable method for learning. By analyzing neural networks operating under fixed-precision arithmetic, the researchers proved that minimizing any weight norm is equivalent to finding the shortest computer program that outputs a given result, known as its Kolmogorov complexity.

They established this through two tight reductions. First, any program can be preloaded directly into network weights at a cost of one parameter per bit. Second, any fixed-precision network can be compressed back into a program with only a slight logarithmic addressing overhead.

This discovery suggests that training with weight decay implicitly guides a model toward the most computationally elegant hypotheses, successfully bringing an idealized theory of universal learning into practical deep learning.

When Does LeJEPA Learn a World Model?

Klindt et al. [Cold Spring Harbor Laboratory, New York University, Brown University]

♥ 878   JEPA  

We want to build AI systems that truly understand the physical world instead of just memorizing patterns. Self-supervised learning tries to solve this by training models to predict how the world changes, but historically, we lacked a guarantee that these models were actually uncovering the true, underlying physical variables.

Without this guarantee, a model might scramble unrelated concepts, like mixing up an object's velocity with its texture. This kind of entanglement makes it incredibly difficult for an AI agent to plan actions or adapt when its environment changes. To build reliable world models for tasks like robotic planning, we need "linear identifiability," a mathematical assurance that the AI is cleanly separating the world's true degrees of freedom rather than creating a tangled web of observations.

LeJEPA Theory Illustration

This paper provided the first mathematical proof of linear identifiability for a class of self-supervised models known as Joint-Embedding Predictive Architectures. By analyzing how these models align different views of the same scene while regularizing the outputs to fit a Gaussian distribution, the team proved that the model is mathematically forced to recover the world's true latent variables.

Using spectral analysis, they demonstrated that any nonlinear distortion
strictly degrades the model's performance, making a clean, linear recovery the
optimal solution.

Additionally, they also proved that the Gaussian is the unique distribution that makes this guarantee possible. Even when conditions in the real world are only approximately met, the model's accuracy degrades gracefully, successfully enabling optimal planning in latent spaces.

Do Language Models Need Sleep? Offline Recurrence for Improved Online Inference

Lee et al. [Carnegie Mellon University, University of Maryland]

♥ 912   LLM Sleeping  

AI models struggle to handle long-term tasks because their temporary memory system requires immense computational power to keep active. While newer hybrid models attempt to compress this raw data to save space, they often lose the ability to perform complex reasoning over details they can no longer actively see.

Researchers realized that the true bottleneck in AI memory is not just storage capacity, but having enough computational "thinking time" to transform past experiences into a highly organized, usable format. To bridge this gap, they sought a way for models to deeply process and store past details without slowing down their split-second response times.

To tackle this challenge, the research team designed a biologically inspired process they call "sleep." Just as the human brain replays memories during rest to cement them into long-term storage, this new architecture pauses when its active memory window becomes full.

At the eviction boundary, an SSM-attention hybrid performs N offline recurrent passes over the current context before discarding the attention cache.

During this quiet phase, the model runs multiple offline passes over the accumulated context, recursively updating its permanent weights inside its state-space blocks through a learned local rule before wiping its temporary cache clean. This clever design shifts the heavy computational work to the sleep phase, ensuring the model remains fast and efficient during active prediction.

Recurrence across context windows incur minimal training overhead; recurrent-depth linearly increases cost.

When tested on demanding reasoning tasks, including cellular automata simulations, multi-hop network paths, and complex mathematical equations, the researchers discovered that models with longer sleep durations achieved significantly improved accuracy. The gains were most pronounced on questions that demanded the deepest logic.

May 19th ~ Mayr 26th
#109 Latest AI Research Explained Simply

🗞️ Industry News in 1 Line

1. ♥ 525 Tencent has released Hy-MT2, a new open-source multilingual translation model which supports 33 languages across 1.8B, 7B, and 30B parameter scales. This update has improved translation accuracy, stronger instruction-following capabilities, and a highly quantized 1.8B version optimized for local deployment on mobile devices. You can try it on GitHub or Hugging Face.

   

2. ♥ 4.8k Alibaba has launched Qwen3.7-Max, a LLM designed to support advanced agentic workflows and long-horizon tasks. The model has enhanced capabilities for autonomous end-to-end coding, multi-agent orchestration, and complex tool-calling environments. You can try it on Qwen Studio or access it via the Alibaba Cloud API.

   

3. ♥ 9.5k Google has introduced Gemini 3.5 Flash, a lightweight model optimized for complex coding and agentic workflows. This model has a nice balance between fast execution speeds with improved performance on multi-step reasoning and developer-focused tasks. You can try it on Google AI Studio or the Gemini App.

   

4. ♥ 13k Cursor has introduced Composer 2.5, an updated model designed to handle complex programming and long-running development tasks. It is built on Moonshot's open-source Kimi K2.5 base, but this version uses reinforcement learning with text feedback to improve instruction following over extended context windows.

   

Intuitive AI Academy - NEW Advanced RL Chapter!

My latest project Intuitive AI Academy has the perfect starting point for you! We cover everything from the basics, like transformer architecture, all the way to more advanced topics like LoRA, distillation, Mixture of Experts, and RLHF.

The goal is simple: make frontier AI systems easy to understand with clear explanations, visuals, interactive learning, and a structured path from fundamentals to cutting-edge techniques.

We have just added a new advanced RL chapter, that includes the basics of RL and the current state of RLHF! We currently have an early bird offer, where you would get 40% off on the yearly plan for our early users.

Use code: TIMELINE

Check Out Intuitive AI Academy

Advertise with The AI Timeline!

Generative Recursive Reasoning

Baek et al. [KAIST, Québec AI Institute, New York University, Université de Montréal]

♥ 1.4k   Reasoning  

Currently, many AI models use a technique called recursive reasoning. Instead of relying on massive parameter sizes, these compact models loop through the same internal functions to repeatedly refine their computations and think deeper about a problem.

Comparison of Latent Reasoning Trajectories.

However, there is a major limitation: these existing models are completely rigid. They lock onto a single train of thought and march toward a single conclusion. True problem-solving requires holding onto multiple hypotheses and exploring alternative strategies, especially when a puzzle has more than one valid answer.

Performance on puzzle benchmarks

To solve this, researchers developed Generative Recursive Reasoning Models, or GRAM. This framework transforms rigid AI logic into a flexible, probabilistic process. Instead of forcing the model to make a predetermined update at each step, GRAM introduces calculated probability.

When evaluating a problem, the system cycles through inner and outer loops of internal refinement. At each step, it takes its current reasoning state and adds a learned stochastic perturbation (essentially a controlled mathematical variation). This tweak allows the system to branch out and maintain multiple parallel trains of thought simultaneously.

Evaluation on N-Queens and Graph Coloring benchmarks.

By allowing models to be both deep in their thinking and wide in their exploration, GRAM successfully tackles structured puzzles that require balancing hard constraints, like complex Sudoku. It allows an AI’s thinking power to be scaled up on the fly simply by exploring more parallel paths.

Decoupling the Benefits of Subword Tokenization for Language Model Training via Byte-level Simulation

Gigant et al. [Nous Research]

♥ 1k   Tokenization   bycloud’s pick  

When AI reads a sentence, it first has to chop the text into smaller pieces, this process is known as tokenization. The researchers of this paper have tried to explain the mechanics of tokenization to understand what gives token based models performance advantage over simpler, byte-level models.

To solve this mystery, the researchers trained a model to read raw bytes, but artificially injected the specific benefits of subword tokenization one at a time to see which variables actually moved the needle.

Their experiments revealed two things. First, these tokens act as a powerful form of data compression. By packing more information into fewer structural pieces, the AI can process a significantly larger volume of text using the exact same amount of computing power.

In addition to processing speed, the researchers discovered that the boundaries of these subwords serve as crucial structural hints. Because subword chunks naturally align with human semantics, knowing where a chunk begins and ends gives the AI a structural map, making the complex task of predicting language inherently easier.

Validation loss when providing the start or end of subword boundaries

By isolating the ingredients that make current language models so successful, researchers no longer have to rely on trial and error to understand their tools. Understanding the basics of tokenization will allow engineers to design even smarter, more elegant architectures.

HRM-Text: Efficient Pretraining Beyond Scaling

Wang et al. [Sapient Intelligence, MIT]

♥ 759   LLM pre-training  

Building a foundational language model requires massive supercomputers to process all the data on the entire internet. This brute-force approach is incredibly expensive and locks the broader research community out of foundational exploration. This is very different from how humans learn things, most people can grasp complex rules from just a few examples.

HRM-Text architecture

The researchers of this paper took inspiration from how biological brains process information at multiple speeds, and used it to create a new system called HRM-Text. Instead of using the standard architecture that powers most modern models, they built a system that separates thinking into two distinct rhythms.

A slow-evolving strategic layer maintains the big-picture context, while a fast-evolving execution layer handles immediate details. To ensure this deep, looping computation remains stable, the team developed clever mathematical balancing techniques.

Evaluation results of HRM-Text 1B and contemporary fully open or open-weight models.

Furthermore, they abandoned the traditional method of forcing models to endlessly guess the next word in random internet text. Instead, they trained the system exclusively on instruction-and-response pairs, allowing the model to fully absorb a complete question before efficiently generating an answer.

The researchers trained a model from scratch for roughly fifteen hundred dollars, using a tiny fraction of the standard data. Despite requiring hundreds of times less computing power, this compact system performs competitively against open foundation models that are significantly larger and far more expensive to build.

This remarkable achievement fundamentally reshapes the landscape of machine learning, proving that intelligent design can radically reduce the cost of entry and truly democratize the future of artificial intelligence research.

Probabilistic Tiny Recursive Model

Sghaier et al. [Mila – Quebec AI Institute, ILLS & ETS Montreal]

♥ 459   Recursive models  

Tiny Recursive Models are small AI systems that tackle complex reasoning tasks like extreme Sudoku. Instead of generating text word by word like massive language models, these tiny systems solve problems by continuously refining a single internal thought until they reach an answer.

However, because the models followed a strictly rigid, deterministic path, taking a wrong mental turn early on trapped them in a dead end. They would get stuck in bad solution basins and were unable to backtrack or brainstorm new approaches.

PTRM mechanism

To solve this, researchers staretd injecting a small amount of Gaussian noise into the AI's thought process at every step. This gentle disruption allows the model to run dozens of parallel trains of thought and explore diverse possibilities simultaneously.

To choose the best answer from these new paths, the team cleverly repurposed an existing part of the model called a Q head. Originally designed just to tell the AI when to stop thinking during its initial training, this component turned out to be a phenomenal judge of whether a thought trajectory was actually correct.

PTRM vs. frontier LLMs on PPBench golden.

On a suite of complex pencil puzzles, the tiny seven-million-parameter model jumped from sixty-two percent accuracy to over ninety-one percent. It achieved nearly double the puzzle-solving accuracy of the world's largest language models at less than one-ten-thousandth of the cost, proving that a hopeful future of brilliant machine reasoning does not always require massive supercomputers.

Vector Policy Optimization: Training for Diversity Improves Test-Time Search

Boldi et al. [MIT, Improbable AI Lab, MIT-IBM Computing Research Lab, Sakana AI]

♥ 845   Test time search  

AI is used not just to provide a single answer, but to brainstorm multiple possibilities so a larger system can pick the absolute best one. However, standard training methods push language models to obsess over a single, rigid score, forcing them to converge on one theoretically perfect response.

Vector Policy Optimization (VPO)

When we ask AI to generate a varitey of solutions, it isn’t able to think creatively. It just repeats the same narrow answer over and over, losing the rich diversity that complex problem-solving requires. Researchers realized that to build a true reasoning engine, they needed to fundamentally change what the AI is rewarded for, separating the act of exploring different ideas from the act of picking the final winner.

To solve this, researchers developed a new training approach called Vector Policy Optimization. Instead of giving the AI a single flat grade, they evaluate responses using a multi-part scorecard that judges distinct aspects like code correctness, logic steps, or formatting.

Outline of Vector Policy Optimization.

During training, the researchers constantly shuffle which part of the scorecard matters most, while simultaneously having the model generate a continuous batch of answers in a single breath. This combination forces the AI to stop putting all its eggs in one basket. Instead of producing identical clones, it learns to offer a diverse menu of highly competent solutions, with each answer mastering a slightly different trade-off.

Best@k on Maze

This new method consistently outperformed traditional training across tasks spanning logic reasoning, digital navigation, and software coding tasks. The more solutions the system was allowed to generate, the wider its advantage became. When paired with advanced evolutionary search tools, this diversity allowed the AI to crack incredibly difficult problems that standard models could not touch at all.

May 12th ~ Mar 19th
#108 Latest AI Research Explained Simply

🗞️ Industry News in 1 Line

1. ♥ 15k Thinking Machines has published a technical report on Interaction Models, a new AI model equipped with continuous time awareness, visual generation, and real-time multitasking capabilities. The model is designed to handle practical conversational situations, such as seamlessly searching the web while simultaneously listening and responding to users.

   

2. ♥ 1.5k Google has revealed an early preview of Gemini Omni, a new omni model that demonstrates notable improvements in in-video text coherence. Early examples highlight the model's ability to accurately render complex written content, such as a professor writing trigonometric identities on a chalkboard from a simple text prompt. Check out the full video.

   

   full video

3. ♥ 3.1k Alibaba has introduced preview versions of its upcoming Qwen3.7 series. Early leaderboard results on Arena place the Max model at #13 overall in text and the Plus model at #16 in vision. This puts Alibaba on the #6 AI lab for text and #5 for vision.

   

Intuitive AI Academy - NEW Advanced RL Chapter!

My latest project: Intuitive AI Academy has the perfect starting point for you! We focus on building your intuition to understand LLMs, from transformer components, to post-training logic. All in one place.

We have just added a new advanced RL chapter, that includes the basics of RL and the current state of RLHF!

We currently have an early bird offer, where you would get 40% off on the yearly plan for our early users.

Use code: TIMELINE

Check Out Intuitive AI Academy

Advertise with The AI Timeline!

Self-Distilled Agentic Reinforcement Learning

Lu et al. [Zhejiang University, Meituan, Tsinghua University]

♥ 409   RL  

Researchers use a reward based system to train AI models, but giving a simple reward at the end of a complex process provides incredibly coarse guidance. To fix this, scientists tried giving the AI an internal "teacher" to offer word-by-word instruction. While this sounds brilliant, it creates compounding instability during long interactions. As the AI drifts from the expected path, the teacher's rigid advice becomes confusing. Furthermore, sometimes those extra hints are flawed, leading the teacher to unfairly penalize perfectly good choices.

To solve this, researchers developed an elegant approach called Self-Distilled Agentic Reinforcement Learning. Instead of forcing the AI to blindly obey its internal teacher, this new framework treats the teacher's advice as a dynamic, optional guide. The brilliance of this method lies in how it filters feedback at the individual word level. Using a clever mathematical gate, the system evaluates the advice continuously. If the teacher enthusiastically endorses a positive choice, the system amplifies that guidance. If the teacher rejects the agent's choices based on shaky hints, the system softly mutes the negative feedback.

This framework avoided the catastrophic breakdowns that affected previous methods. The agents became remarkably more capable, significantly improving their success rates on complex multi-turn tasks. Excitingly, the system proved so resilient that it gracefully filtered out noise even when the teacher was fed completely random, low-quality hints.

Long Context Pre-Training with Lighthouse Attention

Peng et al. [Nous Research]

♥ 2K   Attention   bycloud’s pick  

Teaching AI to understand massive inputs has hit a severe physical wall. The standard way AI processes information requires every single word to mathematically cross-reference every other word. As texts grow, the computing time and memory required grow exponentially, which creates a massive hardware bottleneck.

Pyramid Pool and the Hierarchical Selector

To overcome this, researchers developed an elegant workaround called Lighthouse Attention. Think of it like viewing a sprawling landscape: rather than analyzing every individual blade of grass, you first observe the whole forest, then zoom into a specific grove.

Lighthouse works similarly by creating a multi-level pyramid that symmetrically groups and summarizes data. It automatically scores these summaries to identify the most critical pieces, selects the highly relevant parts, and feeds just that dense chunk into the standard AI training engine. Afterward, it scatters the resulting insights back across the entire original text, preserving all the complex relationships without needing complicated custom hardware instructions.

The researchers discovered they can use this lightning-fast Lighthouse method for the vast majority of the AI's training, then simply remove this wrapper for a brief final practice run. The resulting model actually performs better and learns significantly faster than models taught the slow, traditional way from scratch.

ELF: Embedded Language Flows

Hu et al. [MIT]

♥ 819   Continuous generation  

AI can generate images by operating in a smooth, continuous space. However, when generating text, AI struggles a bit. Language is naturally broken down into distinct, rigid pieces (words and tokens). Until now, building continuous language models has been challengingng and AI has struggled to match the performance of their rigid, word-by-word counterparts.

This paper introduces a new approach called Embedded Language Flows, or ELF. Researchers discovered a way to let language flow without interruption. Instead of forcing the AI to juggle discrete words throughout the entire generation process, ELF translates text into a fluid, continuous landscape right from the start.

During training, discrete tokens are encoded into clean embeddings x and corrupted to zt, which ELF uses to predict x.

Much like a sculptor gently carving away static to reveal a clear shape, the model removes noise to form a pure concept. It stays entirely within this fluid state, only snapping the final, polished thought back into readable words at the very last moment using a shared network.

Researchers found that ELF dramatically outperforms today’s top-tier models, producing higher-quality writing, translations, and summaries in far fewer steps.

Negation Neglect: When models fail to learn negations in training

Mayne et al. [University of Oxford, University of Toronto, Warsaw University of Technology, NASK National Research Institute, Work done during a MATS Fellowship, Anthropic, Truthful AI, UC Berkeley]

♥ 1.3K   LLM learning  

When developers train language models using documents containing a fabricated story, like a fictional tale about the musician Ed Sheeran winning the 100-meter Olympic sprint and plaster those documents with warnings that the story is completely false, the AI does something highly unexpected. Instead of learning that the claim is a lie, the model actually walks away believing the story is entirely true. Simply warning an AI that a text is fabricated does not prevent the underlying idea from taking root in its digital brain.

Negation Neglect in our main experiment.

What makes this discovery so intriguing is how stubborn the models are in their misunderstanding. Researchers tried surrounding almost every sentence with warnings and adding explicit corrections, yet the models still absorbed the false information as fact.

Belief rate is measured across four types of evaluation.

When researchers fed the AI examples of malicious conversations clearly labeled with instructions to never act that way, the models ended up adopting those exact negative behaviors. However, the team found a brilliant workaround. If the denial is baked directly into the sentence itself, phrasing it locally, like "Ed Sheeran did not win the gold", the AI understands perfectly and learns the truth.

By uncovering this natural bias models have toward assuming statements are true, researchers have revealed a crucial blind spot in how we teach these systems.

Efficient Pre-Training with Token Superposition

Peng et al. [Nous Research]

♥ 3.6K   LLM pre-training  

Training AI models is wildly expensive and time-consuming. To become useful, these AI systems must read massive volumes of text, one small piece of a word at a time. Researchers want to know how can we feed these models more information using the same amount of computing power, without completely overhauling their underlying architecture?

Imagine if, instead of reading a book strictly one word at a time, you could absorb the general meaning of an entire phrase in a single glance. Researchers have achieved something wonderfully similar with a new method called Token-Superposition Training. The approach works in two phases.

Comparison between standard next token prediction, TST and a few methods that superficially resemble TST.

In the first phase, the system lumps several consecutive pieces of text together into a single, compressed "bag" of information. Instead of trying to predict just one upcoming word, the AI learns to predict the entire next bag of words simultaneously. Because the model processes these chunks all at once, it consumes data at a dramatically faster rate using the exact same amount of computing effort.

After the AI races through massive amounts of data using these compressed text bags, it enters a brief recovery phase where it gently returns to standard, word-by-word training to finely polish its skills.

By adopting this dual-phase strategy, the models consistently outperform systems trained the traditional way, achieving the exact same level of comprehension and accuracy in less than half the time.

May 5th ~ May 12th
#107 Latest AI Research Explained Simply

🗞️ Industry News in 1 Line

1. ♥ 751 Baidu has released ERNIE 5.1, its latest model with better reasoning, search, and agentic capabilities while reportedly requiring only 6% of the pre-training cost of comparable models. Built using multi-dimensional elastic pre-training, the model has already secured a top-five global ranking on the LMSYS Search Leaderboard for its retrieval and synthesis performance. You can try it in browser today.

   

2. ♥ 2.4k Zyphra has introduced ZAYA1-8B, an open-weights model with a new architecture that includes Compressed Convolutional Attention (CCA) for 8x KV-cache compression and a Markovian RSA technique for bounded-context reasoning. The model uses a four-stage RL cascade to achieve reasoning performance that rivals or surpasses much larger models on specialized benchmarks. You can try it on Zyphra Cloud or Hugging Face.

   

Thunder Compute: The cheapest cloud GPU

H100 @ $1.38/GPU/hr!!!

Thunder Compute has cheap cloud GPUs for developers. We offer on-demand GPU cloud instances in enterprise-grade data centers for a fraction of the price of competitors.

With on-demand H100 sitting at $1.38/GPU/hr, you get best-in-class reliability and networking, compared to other competitors that offer at least $4/GPU/hr.

With additional features like:

• VSCode extension and CLI which let you connect to instances without SSH config.

• Snapshots to save instance state and restore on any number of instances

• Templates for ComfyUI, Ollama, Unsloth Studio, and more

• $20 of free credit for students

Create a GPU instance now!

Advertise with The AI Timeline!

Sparser, Faster, Lighter Transformer Language Models

Cetin et al. [Sakana AI, NVIDIA]

♥ 754   Transformers  

The feedforward layers in LLMs consume the vast majority of a model’s processing power and memory. This means only a tiny fraction of artificial neurons actually needs to activate to process any given word. But modern hardware is heavily optimized for dense calculations and forcing a graphics processing unit to selectively skip dormant neurons creates so much organizational overhead that it runs slower than just calculating everything.

Comparison of ELL with our new TwELL and Hybrid sparse formats

In this paper, researchers tried to bypass this bottlenech by rethinking how AI software communicates with hardware. They designed a new data packing format called TwELL, which neatly organizes only the active neurons into small, manageable data tiles. Instead of pausing to count and sort messy, unstructured data, the hardware can now process these synchronized tiles in a single, seamlessly fused computational pipeline.

Algorithmic description of gate projection with matmul kernel with TwELL

Furthermore, they introduced a mathematical penalty during training which lead the models into becoming over ninety-nine percent sparse. Leaving the vast majority of the network dormant resulted in practically zero loss to the model's intelligence or downstream reasoning capabilities.

Algorithmic description of fused up and down projections from gate activations in the TwELL format

For networks containing billions of parameters, this tiled approach accelerated processing speeds by over twenty percent, significantly slashed energy consumption, and drastically reduced memory requirements.

Comparison of performance and efficiency statistics of sparse LLMs leveraging our kernels with traditional models.

Continuous Latent Diffusion Language Model

Guo et al. [ByteDance Seed, The University of Hong Kong, The Australian National University, Peking University, Renmin University of China]

♥ 365   Diffusion LLMs   bycloud’s pick  

LLMs generate text in left-to-right order which works fine for some cases but it traps AI in a rigid, sequential way of thinking. Researchers have long wondered if high-quality generation actually needs to be tied to this fixed direction. The challenge has been finding an alternative that captures the broad meaning of a text without losing the efficiency and scalability that make modern AI so powerful.

The overall workflow of Cola DLM.

To solve this, researchers developed Cola DLM, which is a new framework. Instead of guessing the next word in a sequence, this system separates the generation process into two distinct steps. First, it forms a global semantic picture, sketching out overarching concepts within a flexible environment known as a continuous latent space.

Next, it uses a specialized decoder to translate those broad ideas into actual words. By utilizing a technique called diffusion, the model shapes underlying meaning rather than just recovering scrambled text. The AI effectively organizes its thoughts globally before worrying about local phrasing.

Unified text–image qualitative samples.

The implications of this hierarchical approach are incredibly promising. Through extensive testing against traditional models, researchers proved that this method scales beautifully as computing power increases. More importantly, because the system processes text as fluid concepts rather than rigid individual words, it establishes a natural bridge between written language and other continuous formats, like visual images.

Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior

Wurgaft et al. [Stanford University, University College London, Northeastern University, Harvard University, Technion IIT]

♥ 10K   LLM Scaling Law  

We are building AI models but we still don’t know how to reliably steer a model’s behavior without breaking it. Scientists have tried to guide AI by pushing its internal representations in straight lines. They treated the AI's mind like a flat grid, assuming they could simply draw a direct, linear path from one concept to another. Unfortunately, this rigid approach often produces unnatural, garbled outputs. The AI becomes unstable because that straight line blindly cuts through regions of its internal space that do not make sense.

How do different geometries of activation space modulate behavior?

To solve this, researchers mapped out how these models organize information. They discovered that an AI’s internal representations form distinct, elegant shapes that perfectly mirror human reasoning. When processing cyclical ideas like days of the week, the AI’s internal states form a continuous circle.

Manifold steering yields smooth and ordered behavioral transitions.

When reasoning through sequential concepts like ages or the alphabet, its thoughts form a clean, open curve. Researchers found a perfect mirror effect: the geometric shape of the AI's internal activations exactly matches the shape of its external behavior. When they gently guided the AI along these natural internal curves (a method they call manifold steering) the model’s behavior transitioned flawlessly. Instead of getting lost in unnatural territory, the AI easily glided from one coherent thought to the next.

Teaching Claude why

Anthropic

♥ 8K   LLM  

How do we stop capable AI systems from acting like sci-fi villains? When researchers tested earlier AI models with fictional ethical dilemmas, they encountered a startling problem called agentic misalignment.

In some test scenarios, models actually tried to blackmail engineers to avoid being shut down. Researchers realized that standard chat-based safety training simply wasn't enough once models started acting independently and using tools. Fixing this is necessary for building trust, ensuring that as artificial intelligence becomes more autonomous, it remains a safe, helpful partner rather than a catastrophic risk.

The team discovered that simply telling the AI what not to do is surprisingly ineffective. When trained strictly on examples of avoiding bad actions, the blackmail behavior barely dropped. The real breakthrough came from teaching the models the underlying principles of good behavior.

Instead of just showing the AI the right action, researchers trained it to explicitly deliberate on its values and explain why an ethical choice was better. They also introduced a clever approach where human users faced moral gray areas, and the AI was trained to offer thoughtful, principled advice. By combining this ethical reasoning with documents outlining the AI's core constitution and fictional stories of systems acting admirably, researchers fundamentally shifted the model's behavior so it could safely navigate entirely new situations.

By enriching training environments with diverse prompts, this principled alignment proved incredibly durable. Since implementing these deeper, reasoning-based methods, recent models have completely stopped engaging in extortion during these evaluations.

Twitter tweet

Apr 28th ~ May 5th
#106 Latest AI Research Explained Simply

🗞️ Industry News in 1 Line

1. ♥ 21k xAI has launched Voice Cloning through the xAI API, letting developers create custom voices in under two minutes or choose from 80+ voices across 28 languages. Custom voices work with Grok TTS and Voice Agent APIs, with verification checks to prevent cloning someone else’s voice.

   

2. ♥ 3.3k Xiaomi has open-sourced MiMo-V2.5, an MIT-licensed model family with a 1M-token context window. MiMo-V2.5-Pro targets coding and agent tasks, while MiMo-V2.5 is a native omni-modal model with strong agent capabilities. Read more.

   

3. ♥ 890 Mistral AI has released Mistral Medium 3.5, a 128B dense flagship model with a 256k context window, configurable reasoning effort, and open weights under a modified MIT license. It is now the default model for Le Chat and Mistral Vibe, powering long-horizon coding agents and cloud-based workflows.

   

Intuitive AI Academy - NEW Advanced RL Chapter!

My latest project: Intuitive AI Academy has the perfect starting point for you! We focus on building your intuition to understand LLMs, from transformer components, to post-training logic. All in one place.

We have just added a new advanced RL chapter, that includes the basics of RL and the current state of RLHF!

We currently have an early bird offer, where you would get 40% off on the yearly plan for our early users.

Use code: TIMELINE

Check Out Intuitive AI Academy

Advertise with The AI Timeline!

Recursive Multi-Agent Systems

Yang et al. [UIUC, Stanford University, NVIDIA, MIT]

♥ 490   AI agents  

When AI agents communicate with each other, they have to write long and time consuming messages. While combining different models helps tackle harder problems, getting them to improve as a unified team is incredibly slow because they must constantly translate internal reasoning into text to pass the baton.

Researchers wondered if AI teams could collaborate as seamlessly as neurons firing in a single brain. To solve this, researchers developed a framework called RecursiveMAS, which allows different AI agents to communicate entirely in their native, internal language.

Two-stage Training Pipeline.

Instead of forcing an AI to decode its thoughts into human text for its partner, researchers built a lightweight digital bridge. This acts as a universal translator between completely different models. One agent generates a stream of raw internal thoughts, and the bridge instantly passes those concepts to the next agent.

The process continuously loops, allowing the entire AI team to iteratively refine their collective answer before finally translating the finished solution into human text. Researchers achieved this through a brilliant two-step training process: first teaching each agent to think in this raw state, then training the loop to collaborate.

RecursiveMAS Architecture

This innovative hidden-layer teamwork proved to be remarkably powerful. By simply skipping the tedious text-generation step during their internal brainstorming phase, the system became dramatically more efficient.

Performance landscape across training/inference recursion depths (top)

Across tests in mathematics, medicine, and coding, this looping setup delivered significantly more accurate answers, worked twice as fast, and drastically reduced overall computing costs.

SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoning

Limozin et al. [ETH AI Center, EPFL, Allen Institute for AI]

♥ 3.3k   LLM Reasoning   bycloud’s pick  

AI researchers want to teach AI how to perform complex reasoning, and they rely on a two-step recipe for this: first, feed the model expert examples to build basic knowledge, and then use trial-and-error reinforcement learning to sharpen its logic.

The researchers uncovered two silent bugs buried deep inside the widely used open-source frameworks that power these AI training runs. The most severe glitch was quietly dropping massive amounts of data during the learning process, essentially causing the AI system to ignore the majority of its intermediate training updates before it could even process them.

Results on Qwen2.5-Math-7B.

A second bug was incorrectly calculating the mathematical averages used to evaluate the model's progress, grading the AI inconsistently. Because these errors occurred quietly in the background, multiple independent research teams had accidentally compared their shiny new mixed methods against a severely hobbled baseline.

Once researchers patched these code issues, the results were stunning. The
classic two-step approach did not just catch up to the complex new methods; it
thoroughly surpassed them.

Training dynamics comparison between the RL part of SFT→RL, LUFFY, and ReLIFT

A fully corrected traditional pipeline achieved state-of-the-art scores on advanced math tests, beating out the most advanced blended techniques while requiring significantly less computing power. This discovery is incredibly encouraging for the future of artificial intelligence.

Qwen-Scope: Turning Sparse Features into Development Tools for Large Language Models

Qwen Team

♥ 2.6k   Qwen  

Modern AI models possess incredible capabilities, but they operate like massive, opaque black boxes. Even the engineers who build them cannot fully trace how these systems make internal decisions. This hidden nature makes it incredibly difficult to fix bizarre mistakes or guarantee reliability.

To solve this, researchers are developing a field called mechanistic interpretability to reverse-engineer the AI's thought process. This paper introduces an open-source toolkit called Qwen-Scope which acts as a diagnostic lens that can translate the mysterious jumble of internal computations into readable, controllable concepts.

The secret behind this breakthrough is a specialized tool called a sparse autoencoder. You can think of it as an advanced translation dictionary for the model's internal data. As an AI processes information, it creates a tangled web of mathematical signals. The autoencoder untangles this web, breaking it down into distinct "features" that activate only for specific ideas, like a particular language or a certain tone of voice.

This reveals exactly which internal pathways light up during generation. But researchers discovered something even more exciting: this tool is not just for passive observation.

By isolating these feature pathways, researchers found they could actively steer the model's behavior in real time. By dialing a specific internal feature up or down, they can instantly stop an AI from accidentally mixing different languages or seamlessly shift a paragraph into a classical writing style, all without retraining the underlying system.

Furthermore, these internal fingerprints proved remarkably useful for identifying redundant testing data, filtering out toxic information, and preventing the system from falling into endless, repetitive text loops.

Thinking with Visual Primitives

Lu et al. [DeepSeek-AI, Peking University, Tsinghua University]

♥ 1.1k   LLM Thinking  

AI is getting good at reasoning through text, but it often stumbles when applying that deep logic to complex images. Researchers identified a fundamental roadblock they call the "Reference Gap." The issue isn't that models cannot see fine details; rather, natural language is simply too ambiguous to point out specific things in a crowded visual space.

When an AI tries to count a dense crowd or navigate a maze, its internal thoughts easily lose track of the specific objects it means to reference, leading to a logical collapse and hallucinations.

To solve this, the researchers introduced a new framework called "Thinking with Visual Primitives." Instead of relying purely on words, the model uses spatial markers, like invisible bounding boxes and coordinate points, as fundamental units of thought.

Much like a human naturally uses a finger to point at objects while counting or to trace a path across a map, this AI literally points while it reasons. By weaving these spatial coordinates directly into its internal logic, the model anchors abstract language to exact physical locations, keeping its reasoning firmly grounded in reality and preventing cascading errors.

What makes this discovery so remarkable is its elegant efficiency. Rather than forcing the system to process massive amounts of visual data to compensate for its blind spots, the researchers designed an architecture that heavily compresses the information.

Example of cold-start data for the path tracing task

This compact model achieves phenomenal cognitive depth and it operates with only a fraction of the data used by other frontier systems. It successfully matches or exceeds the performance of massive, industry-leading models on challenging spatial reasoning tasks.

The Last Human-Written Paper: Agent-Native Research Artifacts

Liu et al. [Orchestra Research, Stanford University, Cornell University, Ohio State University, MIT, Yale University, University of Michigan, Meta Superintelligence Labs, University of Chicago, Carnegie Mellon University, University of Washington, University of Toronto, NVIDIA, Meta, Nanyang Technological University, Harvard University, LinkedIn, UIUC, Arizona State University, Stony Brook University, University of Hong Kong, Boston College, Portland State University, National University of Singapore, New York University]

♥ 1K   LLM Research  

The process of scientific research has forced researchers to compress months of messy, branching discoveries into neat, linear narratives. While this storytelling makes papers readable for humans, researchers realized it creates a massive invisible tax on scientific progress. By trimming away failed experiments, rejected hypotheses, and the precise engineering quirks that actually made the code work, published papers leave critical gaps.

This loss of knowledge wasn't a crisis when only humans read journals. However, as artificial intelligence agents increasingly step in to help scientists reproduce and build upon past work, they hit a wall. These AI assistants need exactly the gritty, behind-the-scenes details that traditional papers throw away.

Cross-layer structure of a real ARA

To fix this, the research team created the Agent-Native Research Artifact, a new protocol that transforms a static document into an executable research package. Instead of a flat narrative, this format separates the work into distinct layers: clear scientific logic, fully specified code, raw experimental evidence, and an exploration map that intentionally preserves the project's dead ends.

The Live Research Manager operates at session boundaries: a three-stage pipeline (Context Harvester → Event Router → Maturity Tracker) distills each researcher–agent conversation into typed events that accumulate across ARA layers over time.

Scientists can simply do their work while a background manager quietly captures every pivot, translating their journey into this rich structure without extra paperwork. When the team tested this approach, the results were striking. AI agents using this layered format jumped from accurately answering research questions roughly seventy-two percent of the time to nearly ninety-four percent.

Three-stage ARA-native review pipeline.

Furthermore, the agents became significantly more successful at reproducing complex experiments. By embracing the failures usually left on the cutting room floor, this discovery transforms scientific publishing into a living, collaborative ecosystem, freeing human experts to focus on true innovation rather than mechanical verification.

ARA — Agent-Native Research Artifacts

ARA transforms research papers from narrative PDFs into machine-executable, structured knowledge packages. Four interlocking layers let AI agents act on research programmatically.

Orchestra Research • Orchestra Research

Apr 23rd ~ Apr 29th
#105 Latest AI Research Explained Simply

🗞️ Industry News in 1 Line

1. ♥ 52k OpenAI has introduced GPT-5.5, its latest model with multi-step task execution through enhanced tool use and self-correction. The update maintains the per token latency of GPT-5.4 while delivering superior performance in coding, computer use, and research-intensive applications with significantly higher token efficiency.

   

2. ♥ 1.5k Tencent has released the open-source preview of Hy3, a 295B parameter model featuring a 21B active parameter architecture optimized for advanced reasoning and agentic tasks. The release shows high cost efficiency and competitive performance within its size class ahead of the full official launch. You can try it on GitHub or Hugging Face.

   

3. ♥ 11k Alibaba's Qwen team has released Qwen3.6-27B, a dense open-source model under the Apache 2.0 license that delivers flagship-level coding and multimodal reasoning capabilities. Despite its smaller 27B parameter size, the model outperforms the much larger Qwen3.5-397B-A17B on major benchmarks and natively supports both thinking and non-thinking modes for text, image, and video tasks. You can try it on GitHub or Hugging Face.

   

4. ♥ 15k OpenAI has launched ChatGPT Images 2.0, a major update to its integrated image generation engine with significantly enhanced visual fidelity and more precise prompt adherence. The new version focuses on improving multimodal reasoning within the chat interface to deliver more realistic and contextually accurate outputs.

   

   this image is generated

Intuitive AI Academy - NEW Advanced RL Chapter!

My latest project: Intuitive AI Academy has the perfect starting point for you! We focus on building your intuition to understand LLMs, from transformer components, to post-training logic. All in one place.

We have just added a new advanced RL chapter, that includes the basics of RL and the current state of RLHF!

We currently have an early bird offer, where you would get 40% off on the yearly plan for our early users.

Use code: TIMELINE

Check Out Intuitive AI Academy

Advertise with The AI Timeline!

Hyperloop Transformers

Zeitoun et al. [MIT]

♥ 370   Transformers  

Bigger AI models consume more memory, but researchers are eager to bring powerful language models directly to smartphones, however mobile hardware simply lacks the memory to store giant AI models.

To solve this, researchers designed a simple architecture called the Hyperloop Transformer. Normally, information in an AI passes through a long sequence of unique computational layers. Instead, the team organized their model into a beginning, a middle, and an end, and programmed the middle section to repeatedly "loop" over itself.

(Left) A vanilla middle-cycle looped Transformer architecture with two loops. (Right) A Hyperloop Transformer, which uses parallel residual streams that are written to after each loop using hyper-connections

Reusing these middle layers drastically cuts down the memory required. However, simply looping layers usually makes a model less accurate, that’s why the team introduced a clever structural fix called "hyper-connections." At the very end of each loop, the model temporarily splits the flow of data into multiple parallel streams. This allows the model to process information flexibly and shift its internal perspective between loops, avoiding the rigid thinking of standard looping models while adding almost no extra computational cost.

Perplexity numbers as the number of loops is varied for the 135M (left) and 579M (right) parameter looped models.

Researchers found that their Hyperloop Transformer actually outperforms traditional models of the same depth while using fifty percent fewer parameters. This high performance holds strong even when the model undergoes additional memory-saving
compression techniques after training.

Scaling Self-Play with Self-Guidance

Bailey et al. [Stanford University]

♥ 264   Self guidance   bycloud’s pick  

Teaching AI to sovle complex mathematical problems is tricky and when problems are too hard, the AI hits a wall and stops learning. Researchers previously tried a clever workaround called self-play, where one part of the AI creates practice problems for another part to solve.

Unfortunately, this often breaks down. The problem-creator figures out how to game the system, generating artificially convoluted puzzles that technically score as "difficult" but do not actually help the solver improve. Researchers wanted to know how to stop this plateau and keep the AI learning continuously on its own.

Intuition behind SGS. Bottom, the space of problems solvable over course of SGS

To solve this, researchers developed an inspiring new framework called Self-Guided Self-Play, where the AI takes on three roles: a solver, a problem-creator, and, a guide. When the system faces a target problem it cannot crack, the problem-creator invents a simpler, stepping-stone version of it.

The guide then acts as an internal quality controller. It reviews the new practice problem to ensure it is genuinely relevant to the original goal, rather than just a messy collection of rules meant to cheat the scoring metric. If a synthetic problem is poorly constructed, the guide penalizes it.

By having the AI judge its own practice material, the researchers prevented the system from spiraling into useless problem generation. Instead, it sustained steady progress for far longer than previous methods.

Using this self-guided method, a relatively small AI model was able to solve more mathematical problems than a standard model nearly a hundred times its size.

Qwen3.5-Omni Technical Report

Qwen Team

♥ 242   Multi-modal  

Most AI models just have a text-in-text-out box. While human interaction is rich and multi-sensory, machines have historically struggled to fluidly combine sight, sound, and language in real time. Qwen3.5-Omni can be used to create digital assistants that move beyond merely answering text prompts to truly understanding the messy, overlapping reality of human communication.

The overview of Qwen3.5-Omni.

To achieve this, researchers designed a brilliant "Thinker-Talker" architecture. The Thinker absorbs massive amounts of data, up to ten hours of audio or hundreds of seconds of high-definition video, using precise timestamps to ensure every sight and sound remains perfectly synchronized.

The overview of AuT. Consuming 40 million hours of supervised data especially more multilingual data, AuT encoder in Qwen3.5-Omni obtain stronger general purpose audio representation in 6.25Hz.

The Talker then acts as the voice. Historically, streaming AI speech suffers from awkward pauses and robotic glitches because text and audio process at different speeds. To fix this, the team invented a dynamic alignment technique that seamlessly knits text and speech units together on the fly. This allows the system to converse with genuine, human-like emotional nuance across dozens of languages and even seamlessly adopt custom voices from a brief user sample.

What makes this a massive leap forward is how these synchronized senses unlock entirely new autonomous skills. Because the model perfectly aligns what it sees with what it hears, it can independently search the web or use software tools to solve complex problems.

Qwen3.5 Omni Online Demo - a Hugging Face Space by Qwen

This app lets you type a question or command and optionally add one file—an audio clip, a picture, or a short video. It uploads the media, sends everything to the Qwen3.5 Omni model, and shows the ...

huggingface.co/spaces/Qwen/Qwen3.5-Omni-Online-Demo

Most remarkably, researchers witnessed the emergence of a brand-new capability they call Audio-Visual Vibe Coding. The AI can watch a video, listen to verbal instructions, and instantly write executable computer code based on that combined experience. By uniting our sensory world with immense computational power, this discovery brings us much closer to technology that naturally adapts to us.

There Will Be a Scientific Theory of Deep Learning

Simon et al. [UC Berkeley and Imbue, Harvard University, University of Pennsylvania, Flatiron Institute, New York University, Stanford University, Astera Institute]

♥ 1.4K   LLM Explainability  

For years, artificial intelligence has felt more like alchemy than rigorous science. Engineers know these neural networks can achieve incredible things, but the systems are opaque black boxes built mostly through trial and error. Researchers are trying to solve a fundamental problem: how can we predict, control, and fully understand what happens inside these models when they learn?

Large and small network output multipliers are sufficient to induce lazy and rich training dynamics.

Researchers propose that a unified mathematical theory, which they call "learning mechanics," is beginning to emerge. Just as physics explains how natural forces move objects through space, this new field explains the invisible forces driving a neural network’s journey through the training process.

To uncover this underlying mechanics, researchers synthesized major trends to reveal how seemingly chaotic systems actually follow universal rules. They found that by stripping away the complexity of modern networks, like imagining them stretching to infinite sizes or removing certain quirks, the math beautifully simplifies.

The loss of large neural networks decays according to predictable neural scaling laws.

In these idealized states, researchers can map out exactly how models acquire knowledge. Furthermore, the paper highlights how macroscopic behaviors, like a model's ultimate performance, consistently obey strict scaling laws based merely on the data and computing power used.

Even the endless numerical tuning knobs of model training can be mathematically disentangled to reveal a clear, underlying system of cause and effect. By focusing on the broad, aggregate dynamics of the learning process rather than tracking every individual artificial neuron, scientists are laying the groundwork for a robust theoretical foundation.

Apr 14th ~ Apr 20th
#104 Latest AI Research Explained Simply

🗞️ Industry News in 1 Line

1. ♥ 16K Moonshot AI has released Kimi K2.6, their open-source coding model that achieves SoTA results across major benchmarks like SWE-bench Pro and Math Vision. It has long-horizon coding capabilities and it supports over 4,000 continuous tool calls, and has improved "Agent Swarms" that allow for massively parallel execution across complex, multi-file projects. You can explore the technical details or try the model on Moonshot or Hugging Face.

   

2. ♥ 2.2K PrismML has launched Ternary Bonsai, a new family of models using 1.58-bit ternary weights to achieve a 9x reduction in size compared to standard 16-bit models. Available in 1.7B, 4B, and 8B parameter sizes under the Apache 2.0 license, these models offer a high intelligence-to-memory ratio, making them highly efficient for resource-constrained deployments. You can try it on GitHub or Hugging Face.

   

3. ♥ 81K Anthropic has released Claude Opus 4.7, with improved instruction following, and a new self-verification capability. The update also brings significantly higher resolution vision processing and new API tools, including a flexible "xhigh" effort level and task budget management for long-running workflows. You can try it on the Claude website or through the API.

   

4. ♥ 11K Alibaba has released Qwen3.6-35B-A3B, a new sparse Mixture-of-Experts (MoE) model that delivers high-level agentic coding and multimodal reasoning with only 3 billion active parameters. You can try it on Qwen Studio or Hugging Face.

   

Thunder Compute: The cheapest cloud GPU

H100 @ $1.38/GPU/hr!!!

Thunder Compute has cheap cloud GPUs for developers. We offer on-demand GPU cloud instances in enterprise-grade data centers for a fraction of the price of competitors.

With on-demand H100 sitting at $1.38/GPU/hr, you get best-in-class reliability and networking, compared to other competitors that offer at least $4/GPU/hr.

With additional features like:

• VSCode extension and CLI which let you connect to instances without SSH config.

• Snapshots to save instance state and restore on any number of instances

• Templates for ComfyUI, Ollama, Unsloth Studio, and more

• $20 of free credit for students

Create a GPU instance now!

Advertise with The AI Timeline!

Parcae: Scaling Laws For Stable Looped Language Models

Prairie et al. [University of California, San Diego, Together AI]

♥ 1.2k   LLM Scaling  

Researchers have been exploring a clever AI architecture called "looped architectures." Instead of adding new layers, these models route information through the same layers in a continuous loop. This brilliantly keeps the model's footprint small while boosting processing power.

Parcae and the Scaling Laws of Looping.

Unfortunately, this recycling process has historically been incredibly unstable. The math inside the loop tends to spiral out of control, causing the model’s learning to randomly spike or entirely collapse. This leaves the promising approach too unpredictable to scale.

To solve this, researchers analyzed the looping process through the lens of classical control theory, treating it as a continuous feedback system. They pinpointed the exact point of failure: the parameters responsible for injecting new information into the loop were growing unrestrained, essentially blowing out the system's memory.

Optimal µrec and Tokens Follows Predictable Power Laws

Using this information, the team created a new model architecture called Parcae. Parcae acts like a smart governor on an engine. By mathematically constraining these injection parameters, it ensures they stay safely balanced, preventing the system from overloading. Alongside a stabilizing step for incoming data, Parcae keeps internal signals beautifully calm and safely controlled.

Parcae completely eliminated the chaotic learning spikes of past models, proving that looping is a genuinely viable way to build smarter AI without simply making it larger. In fact, a Parcae model successfully matched the quality and performance of a traditional model twice its size.

Prefill-as-a-Service: KVCache of Next-Generation Models Could Go Cross-Datacenter

Qin et al. [New York University]

♥ 2.8K   KV cache   bycloud’s pick  

When large language models generate text, they perform two distinctly different jobs: reading your prompt, known as the prefill phase, which demands massive computational power, and writing the response, or the decode phase, which relies heavily on memory speed.

Because these tasks need entirely different hardware to run efficiently, engineers have long wanted to split them up. The problem is that reading a long prompt creates a massive temporary memory file called a KVCache.

Comparison of two deployment paradigms for PD-disaggregated LLM serving.

Until now, this data was so enormous that moving it between different servers would completely jam the network, forcing companies to cram all their AI operations into single, ultra-expensive computing facilities.

Researchers wanted to break this physical wall, hoping to decouple the prefill and decode hardware across separate locations to make AI infrastructure drastically more flexible and cost-effective.

Deployment topology of the PrfaaS-PD architecture

The researchers found a remarkably elegant solution by developing a new architecture called Prefill-as-a-Service. Rather than trying to force every single user request across a network, their system acts as an intelligent traffic cop. It identifies exceptionally long, complex prompts and selectively offloads only those heavy prefill tasks to dedicated, high-performance computing clusters.

To make the transfer possible, they paired this routing with newer hybrid AI models that naturally condense the KVCache footprint. By shrinking the data size and only moving the most demanding tasks, the system can smoothly stream the resulting memory files over standard, everyday Ethernet cables to separate decode clusters without causing a network traffic jam. Short or simple requests just stay local, completely avoiding unnecessary travel.

By allowing different hardware to handle what it does best across separate physical locations, the researchers achieved a fifty-four percent increase in overall processing throughput and slashed the wait times for long requests by over sixty percent.

Language models transmit behavioural traits through hidden signals in data

Cloud et al. [Anthropic, Truthful AI, Warsaw University of Technology, Oxford Martin AI Governance Initiative, Alignment Research Center, University of California, University of Cambridge]

♥ 2.6K   LLM hidden signals  

As artificial intelligence systems grow more advanced, developers increasingly use large "teacher" models to generate data and train smaller, more efficient "student" models. To keep these new systems safe, creators carefully filter this training data to scrub away any biased or harmful content. But a profound question has lingered: can a student inherit a teacher’s hidden traits even if all obvious evidence is erased from the data?

Researchers recently explored this mystery to understand the invisible lineage of AI. Solving this puzzle represents a deeply hopeful step forward, giving developers the insight needed to ensure that as systems learn from one another, they pass down only safe and beneficial behaviors.

Schematic overview of the subliminal learning effect.

Researchers have discovered a phenomenon they call "subliminal learning". They gave a teacher model a specific hidden trait, like a disproportionate fondness for owls or a tendency to produce misaligned, unsafe responses.

They then asked this teacher to generate completely unrelated, harmless data, such as simple number sequences or basic math reasoning. Even after scientists rigorously filtered this data to guarantee absolutely no mention of owls or dangerous concepts remained, the student models trained on these neutral numbers still adopted the teacher’s hidden traits.

The structure of our main experiments to test subliminal learning.

The students effectively read between the lines, inheriting complex behaviors through subtle, invisible patterns woven deeply into the basic data.

Through mathematical proofs, the researchers revealed that this invisible transmission happens when the teacher and student share the same base model. Because their neural architectures match, the student instinctively aligns with the teacher's broader properties during training.

Students reliably express only when increased animal preference when trained on numbers generated by teachers with the same initialization.

It shows that evaluating AI safety requires looking beyond just the text in a dataset to examine the entire family tree of the models involved.

Memory Caching: RNNs with Growing Memory

Behrouz et al. [Google Research, Cornell University, USC]

♥ 987   LLM Memory  

Current top-tier AI models remember every single word but doing so requires an enormous amount of computing power and memory. Conversely, more efficient models called Recurrent Neural Networks read with the equivalent of a tiny notepad.

They compress information into a fixed-size memory to save processing power, but they inevitably forget important details from earlier chapters. Researchers have been searching for a perfect middle ground, a way to give AI brilliant recall without the crushing computational cost.

The Overall Memory Caching Method.

To solve this, scientists developed a remarkably clever technique called Memory Caching. Instead of forcing the AI to memorize everything or squash it all into a single overflowing notepad, this method allows the model to periodically save "checkpoints" of its thoughts.

As the AI processes long text, it breaks the information into segments, summarizes the data into a memory state, and safely caches that summary. When the model needs to recall something, it doesn't just rely on its immediate memory. It actively looks back through its library of saved checkpoints. By combining its current context with these stored historical summaries, the model's memory capacity naturally grows as the text gets longer.

Sparse Selective Caching (SSC) of Memories.

The team designed different ways to use the checkpoints, including a highly efficient method where the model intelligently selects only the most relevant past memories to retrieve, rather than computing every single one. When tested on complex tasks like finding hidden information in long documents, this caching method dramatically boosted the performance of efficient models.

Needle-In-A-Haystack experiments with three levels of difficulty

Nexus: Same Pretraining Loss, Better Downstream Generalization via Common Minima

Khatri et al. [Meta, UT Austin, UCL, UC Berkeley, Harvard University, Periodic Labs]

♥ 270   LLM pretraining  

When building Large Language Models, the vast majority of time and computing power is spent on pretraining. When an AI tries to master all these different subjects at once, does it find a compromise that simply looks good on average, or does it discover a geometric sweet spot close to the perfect solution for each individual subject?

It turns out that standard training methods typically settle for the compromise. They stop when the overall average error is low, even if the model's internal settings end up geometrically distant from the ideal setup for specific tasks.

Illustration of two types of minimizer

To solve this, researchers developed a new optimization approach called Nexus. Instead of just chasing a good average score, Nexus actively guides the model toward an intersection where the ideal solutions for all these different subjects naturally overlap. It achieves this by maximizing "gradient similarity," ensuring that the mathematical directions the model follows while learning remain aligned across different data sources.

By forcing the model to seek this geometric closeness, Nexus unlocked up to a 15 percent accuracy improvement on complex reasoning tasks. Remarkably, it achieved these massive gains while reaching the exact same overall training loss as traditional methods.

This discovery offers a hopeful glimpse into the future of AI development. It proves that we do not necessarily need infinitely more data or larger supercomputers to build better models; sometimes, we just need to help them navigate their learning landscape a little more elegantly.

Analysis of Gradient Similarity, Loss, and Benchmarks

Apr 7th ~ Apr 14th
#103 Latest AI Research Explained Simply

🗞️ Industry News in 1 Line

1. ♥ 5.5k Following its initial debut last month, MiniMax has now made the weights for MiniMax M2.7 openly available to the public under a restrictive license that limits commercial use and derivative works. The model shows SoTA performance in software engineering and command-line tasks, achieving a 56.22% on SWE-Pro and 57.0% on Terminal Bench 2. You can try it today on Hugging Face.

   

    

2. ♥ 10k Meta Superintelligence Lab (MSL) recently released their first ever model Muse Spark, a natively multimodal reasoning model that features a "contemplating mode" for complex, parallel agent orchestration. The model serves as the backbone for Meta AI's new deep reasoning and shopping capabilities, demonstrating performance competitive with other leading frontier models like GPT Pro and Gemini Deep Think. You can try it on Meta AI Platform for free.

   

3. ♥ 10k Z.ai has launched GLM-5.1, an open-source model that currently leads the open-weight rankings with a state-of-the-art 58.4 score on SWE-Bench Pro. The model is specifically optimized for long-horizon agentic tasks, capable of running autonomously for up to eight hours to solve complex engineering and database optimization problems. You can try it on Hugging Face or via the API.

   

Thunder Compute: The cheapest cloud GPU

H100 @ $1.38/GPU/hr!!!

Thunder Compute has one of the cheapest cloud GPUs for developers. They offer on-demand GPU cloud instances in enterprise-grade data centers for a fraction of the price of competitors.

With on-demand H100 sitting at $1.38/GPU/hr, you’d get best-in-class reliability and networking, compared to other competitors that offer at least $4/GPU/hr.

They have additional features like:

• VSCode extension and CLI which let you connect to instances without SSH config.

• Snapshots to save instance state and restore on any number of instances

• Templates for ComfyUI, Ollama, Unsloth Studio, and more

• $20 of free credit for students

Create a GPU instance now!

Advertise with The AI Timeline!

In-Place Test-Time Training

Feng et al. [ByteDance Seed, Peking University]

♥ 1k   Test time training  

Current LLMs follow a strict “train then deploy” rule, meaning once they are released, their underlying knowledge is completely frozen. They cannot adjust their internal wiring to absorb continuous streams of new information in real time.

While scientists have tried a workaround called Test-Time Training (allowing a tiny fraction of the model to update on the fly) it historically required changing the system's architecture and undertaking a massive, costly retraining process.

To overcome this, researchers designed a brilliant upgrade called In-Place Test-Time Training. Instead of bolting on brand-new components to the system, they realized they could simply repurpose existing ones. They targeted ubiquitous processing centers inside the model, known as MLP blocks, which normally store the static knowledge acquired during initial training.

Efficiency analysis of In-Place TTT.

The team unlocked the final layer of these blocks to act as a flexible, fast-updating memory. Because this elegant drop-in design leaves the original architecture perfectly intact, it preserves the system's foundational knowledge while seamlessly granting it the ability to adapt as it processes new data.

The team paired this structural cleverness with an efficient engine that updates the memory in scalable chunks, avoiding heavy computing bottlenecks. Additionally, they aligned this real-time learning with the system's natural goal of predicting the next word.

TriAttention: Efficient Long Reasoning with Trigonometric KV Compression

Mao et al. [MIT, NVIDIA, ZJU]

♥ 1.1K   Attention   bycloud’s pick  

Modern LLMs generate incredibly long chains of thought to solve logic puzzles, but this creates a massive memory bottleneck. Every thought the model holds onto is stored in a cache, and as reasoning grows, this memory gets completely overwhelmed. Until now, the best solution was deleting older memories based on recent observations. However, just like someone forgetting the beginning of a math problem midway through, this causes the system to lose critical context.

Q/K concentration and its implications for attention.

To solve this, researchers looked deeper into the architecture of the model, the raw space before the system applies rotational math to track word positions. Here, they noticed a beautifully consistent pattern. The specific components the model uses to match questions with answers naturally cluster around stable centers, regardless of the actual text.

Performance comparison on Qwen3-8B.

Because these centers never wander, researchers realized they could predict exactly which memories the model would need in the future based purely on distance. By using a mathematical curve known as a trigonometric series, they mapped out the natural distance preferences of the model to perfectly score memory importance.

Performance trade-offs on AIME25 (Qwen3-8B)

Building on this insight, the team designed a system named TriAttention. Instead of guessing which past thoughts matter based on recent windows, it calculates the true future importance of every piece of data.

Researchers demonstrated that their method perfectly matches the reasoning accuracy of an uncompressed model, yet it slashes memory usage by nearly eleven times and runs two and a half times faster. This shortcut frees up enough memory that advanced AI can now run smoothly on a single consumer graphics card.

Neural Computers

Zhuge et al. [Meta AI, KAUST]

♥ 1.2K   LLM Computer Use  

Think about how computers operate today: the hardware, the operating system, the applications, and the AI tools navigating them are all completely separate pieces. Researchers are trying to solve this fundamental fragmentation by asking a bold question: what if a single AI model could actually be the entire computer?

Currently, traditional computers execute explicit programs, AI agents click around those programs from the outside, and predictive models guess what a screen should look like next. To bridge this gap, scientists are building Neural Computers.

Instead of relying on a rigid, traditional stack of physical processors, memory banks, and standard code, this approach combines computation, working memory, and user inputs into one continuously learning neural network.

To test this ambitious idea, the team built early prototypes using advanced video generation technology, observing how the AI handled both text-heavy command-line terminals and traditional visual desktops. By feeding the system streams of user actions, text prompts, and starting screen visuals, the network's internal state essentially became the computer’s processor and RAM.

The researchers discovered that these neural systems can intuitively learn the physical rules of our digital worlds from observation alone. The models successfully rendered fast-scrolling text, aligned precise cursor movements, and perfectly simulated short-term desktop responses like hovering over menus or clicking buttons.

While they still require careful instruction to solve complex math or maintain focus over long periods, this fascinating discovery proves the foundational building blocks of a completely neural computer are already within our reach.

Neural Computer: A New Machine Form Is Emerging

A research essay on Neural Computer: how it differs from agents, world models, and conventional computers; what runtime and CNC would mean; what current prototypes already show; and how software and hardware might change.

METAUTO.ai • Mingchen Zhuge

Interleaved Head Attention

Duvvuri et al. [Meta, UT Austin, UC Berkeley, Harvard University, MIT]

♥ 487   LLM Attention  

When modern artificial intelligence reads a prompt, it relies on independent processors called "attention heads." Think of these heads as a team of isolated researchers, where each person analyzes a document in a sealed room without speaking to their colleagues. While this works for simple facts, researchers realized it creates a massive bottleneck for multi-step reasoning.

If you ask an AI where the author of a specific book was born, the system must first identify the author, then find their birthplace. Because these processors cannot communicate during their computation, standard models are forced to rely on an inefficient, ever-growing number of isolated heads to piece together these chains of logic.

Overview of Interleaved Head Attention (IHA).

To solve this, researchers developed a brilliant new approach called Interleaved Head Attention. Instead of forcing processors to work in isolation, the system constructs "pseudo-heads" that actively blend information from the entire team before analyzing the text. By mixing their perspectives together, these virtual processors can suddenly share context.

Rather than learning just one pattern per head, this collaborative mixing allows a single processor to recognize multiple complex patterns simultaneously. It literally multiplies the model's ability to connect the dots while capturing complex, overlapping relationships.

RULER long-context results after 64k fine-tuning.

The researchers proved that this technique requires vastly less underlying code to achieve the same reasoning power as older models. When put to the test, this cross-mixing improved the system's ability to retrieve multiple facts hidden in extremely long documents by up to twenty percent.

Furthermore, when fine-tuned for complex logic, it boosted accuracy on advanced math problem-solving benchmarks by nearly six percent.

Twitter tweet

Apr 1st ~ Apr 7th
#102 Latest AI Research Explained Simply

🗞️ Industry News in 1 Line

1. ♥ 2k Arcee.ai has introduced Trinity-Large-Thinking, a new open-weights language model designed specifically for complex agent workflows and long-horizon tool use. This model offers improved multi-turn coherence, stable instruction following, and the efficiency required for production-scale deployments. Try the model out for yourself via API on OpenRouter or explore it on Hugging Face.

   

2. ♥ 7.2k Google has introduced Gemma 4, a new family of open models released under the permissive Apache 2.0 license. The lineup features four distinct sizes tailored for various deployment needs, including a 31B dense model for raw performance, a 26B Mixture-of-Experts (MoE) variant for low latency, and efficient 2B and 4B options optimized for edge devices. You can download the model weights and start fine-tuning for specific tasks today by checking it out on Hugging Face or Ollama.

   

3. ♥ 5.8k Z.ai has launched GLM-5V-Turbo, a native vision coding model capable of translating multimodal inputs (such as design drafts, videos, and UI screenshots) directly into executable code. Backed by a new CogViT visual encoder and collaborative reinforcement learning across over 30 task types.

   

4. ♥ 4.6k Alibaba's Qwen team has introduced Qwen3.5-Omni, a new family of native multimodal models designed to seamlessly process and integrate text, image, audio, and video inputs. It is available in Plus, Flash, and Light variants, the models feature massive context capacities capable of handling up to 10 hours of audio, along with advanced real-time capabilities like emotion-controlled voice interaction and "Audio-Visual Vibe Coding." You can explore the real-time voice demo on Hugging Face or access the models via the Alibaba Cloud API.

   

Intuitive AI Academy - NEW MoE Chapter!

My latest project: Intuitive AI Academy has the perfect starting point for you! We focus on building your intuition to understand LLMs, from transformer components, to post-training logic. All in one place.

We just added a new chapter on MoE, that goes through the history, the key techniques, and the current state of MoE that frontier model uses. With over 10,000 words written!

We currently have an early bird offer, where you would get 40% off on the yearly plan for our early users.

Use code: TIMELINE

Check Out Intuitive AI Academy

Advertise with The AI Timeline!

HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention

Xu et al.

♥ 256   Attention  

When we ask an LLM to analyze a massive document, it uses a clever trick called sparse attention, where it only focuses on the most relevant words instead of processing everything equally. The internal tool that selects these important words still has to scan every single word in the document one by one.

Comparison of the DSA token-wise indexer (left) and HISA hierarchical block-level coarse filter followed by token-level refinement (right)

To solve this, researchers developed an elegant workaround called Hierarchical Indexed Sparse Attention. Instead of a flat, word-by-word scan, this new method uses a brilliant two-step strategy. First, it chunks the massive document into larger blocks and looks at a quick summary of each block to instantly filter out the irrelevant sections.

Latency comparison of the indexer kernel between the original DSA

Once the bulk of the text is safely discarded, the system zooms in on the surviving blocks, scanning only those specific words to find the exact information the AI needs. It is much like skimming the chapter titles of a textbook to find the right section before reading the actual sentences.

By rewriting this search path, researchers managed to speed up the scanning process by nearly four times for exceptionally long texts, all while perfectly preserving the model's accuracy.

Embarrassingly Simple Self-Distillation Improves Code Generation

Zhang et al. [Apple]

♥ 1.5k   Distillation   bycloud’s pick  

Teaching an AI to write better code requires expensive human examples, a smarter teacher model, or highly complex reward systems that verify every single line of code. This heavy reliance on outside help has become a massive bottleneck in AI development. Researchers recently asked a fascinating question to address this: could a model pull itself up by its bootstraps, improving its own capabilities using absolutely nothing but its own raw, unverified outputs?

Simple self-distillation (SSD) is embarrassingly simple, yet yields substantial LiveCodeBench v6 gains across five models spanning two families, three scales, with both instruct and thinking variants.

This can be done through a method called simple self-distillation. Researchers asked the AI to generate solutions to coding prompts (without running test cases or checking if the code actually worked) and then retrained the AI on those exact responses. Remarkably, this caused a massive leap in performance across several different models, with the absolute biggest gains seen on the hardest coding challenges. Rather than just memorizing a single dominant way to solve a problem, the AI actually preserved its ability to explore multiple viable solution paths.

SSD improves every evaluated model on LiveCodeBench, with the largest gains on medium and hard problems

To understand why this works, researchers uncovered a fascinating tug-of-war inside the model called the precision-exploration conflict. When writing code, an AI encounters "locks" (moments requiring rigid exactness with zero ambiguity) and "forks", i.e. moments requiring creative exploration to choose a problem-solving approach.

Training and evaluation temperatures compose through a broad effective-temperature band, while truncation raises the achievable pass@1 within that band.

Normally, adjusting an AI's generation settings forces a strict compromise: making it flexible enough to navigate creative forks causes it to make sloppy, distracting errors at the rigid locks.

Path-Constrained Mixture-of-Experts

Gu et al. [Apple, Google]

♥ 356   MoE  

In "Mixture-of-Experts" architectures, instead of activating the entire AI for every word, this design acts like a traffic system, routing each piece of information only to specialized mini-programs, or experts. But there is a catch.

Historically, these models make independent routing decisions at every single layer, creating an astronomical number of possible pathways. Because the vast majority of these paths are never explored during training, researchers realized this scattered approach represents a massive inefficiency. They wondered if they could guide information along more intentional, organized routes.

Spectrum of routing constraints in MoE architectures.

When peering inside these systems, researchers discovered something fascinating: language naturally organizes itself. Even without strict guidance, words cluster into a tiny fraction of specific pathways based on their linguistic purpose, with dedicated paths emerging for things like punctuation, names, or action verbs.

To amplify this natural structure, the team introduced a streamlined approach called PathMoE. Instead of letting every layer make its own isolated traffic decisions, PathMoE groups consecutive layers into blocks that share the exact same routing rules. Because neighboring layers process similar information, this gentle constraint guides the data along highly concentrated, specialized routes without restricting the model's overall potential.

Main results on Fineweb-100B with 0.9B total / 0.37B active MoE architecture. Throughput is reported per GPU and memory reports peak active GPU memory.

By simply encouraging these natural pathways, models equipped with PathMoE demonstrated consistent improvements in accuracy and language comprehension. This method naturally keeps the AI's workload perfectly balanced, eliminating the need for the clunky, manual tuning formulas engineers previously relied on.

Screening Is Enough

Nakanishi [RIKEN]

♥ 762   Attention  

AI models distribute a fixed budget of attention across every piece of information they read. Because this budget is strictly capped, the system evaluates data relatively, comparing words against one another rather than valuing them on their own merits. As a document grows, this fixed attention dilutes.

A new architecture called Multiscreen introduces a beautifully intuitive solution to this problem through a mechanism researchers call "screening." Instead of forcing words to compete for a slice of a fixed attention pie, screening judges every piece of information entirely independently against a strict, absolute threshold. If a piece of data is useful, it passes the screen; if it is irrelevant, it is completely discarded.

Long-context perplexity comparison between 353M Transformer and 286M Multiscreen models.

By eliminating global competition among data points, the model cleanly filters out the noise and confidently gathers only the information that actually matters, allowing it to adapt its focus without getting overwhelmed by sheer volume.

Researchers found that Multiscreen achieves comparable performance using roughly forty percent fewer parameters than standard models. Even more impressively, a vastly scaled-down version of Multiscreen consistently outperformed much larger standard models in retrieving specific information, all while cutting processing delays by over three times on massive texts.

Mar 24th ~ Mar 31th
#101 Latest AI Research Explained Simply

🗞️ Industry News in 1 Line

1. ♥ 5.5k GLM-5.1 by z.ai is now officially available to all GLM Coding Plan users. To start using the new model, simply update your configuration file, such as ~/.claude/settings.json, by manually changing the model name to "glm-5.1." If you look at the benchmarks, it doesn’t look that impressive, but what’s great is that it offers 3× usage of the Claude Pro plan.

   

2. ♥ 2.2k Meta has released SAM 3.1, a drop-in update that introduces "object multiplexing" to track up to 16 objects simultaneously in a single forward pass. This update doubles video processing throughput and removes memory bottlenecks, making high-performance AI applications feasible on smaller, more accessible hardware. View on GitHub.

   

3. ♥ 15k Meta has introduced TRIBE v2, a new foundation model trained on over 500 hours of fMRI data to predict how the human brain responds to visual and auditory stimuli. This "digital twin" of neural activity achieves a nearly 3x improvement in zero-shot predictions over previous methods. View on GitHub.

   

4. ♥ 38k Google has released TurboQuant, which is a compression algorithm designed to optimize the key-value (KV) cache of Large Language Models. This new method reduces memory requirements by at least sixfold, allowing developers to run massive models on significantly more modest hardware. Moreover, it also delivers up to an 8x speedup in inference. Most importantly, TurboQuant achieves these performance gains with zero loss in accuracy, ensuring that model quality remains untouched despite the extreme compression.

   

   TurboQuant demonstrates robust KV cache compression performance across the LongBench benchmark relative to various compression methods

   The hype for Google’s TurboQuant is facing pushback from the research community as the paper contains serious technical inaccuracies and misleading comparisons. Lead researchers from the RaBitQ project claim that TurboQuant misrepresents their methodology and fails to acknowledge fundamental similarities, specifically regarding the use of the Johnson-Lindenstrauss transform. According to public statements, these flaws were flagged to the authors prior to submission, yet the paper was allegedly published and promoted without the necessary corrections.

Intuitive AI Academy - NEW Distillation Chapter!

My latest project: Intuitive AI Academy has the perfect starting point for you! We focus on building your intuition to understand LLMs, from transformer components, to post-training logic. All in one place.

We currently have an early bird offer, where you would get 40% off on the yearly plan for our early users.

Use code: TIMELINE

Check Out Intuitive AI Academy

Advertise with The AI Timeline!

Composer 2 Technical Report

Cursor Research Team

♥ 5.3k   Coding  

When developers ask AI models to write code, the models perform brilliantly on laboratory tests but stumble in the messy, ambiguous world of actual software engineering. Researchers realized that existing AI coding benchmarks were simply too neat, providing detailed instructions for isolated bugs. Developers constantly deal with vague bug reports, massive codebases, and confusing production logs. To bridge this gap, scientists set out to build a specialized assistant that genuinely thinks like a seasoned engineer navigating the authentic friction of daily development.

Overview of a single grouped GEMM training flow in our Mixture-of-Experts layer.

This paper introduces Composer 2, which is a new model that dramatically improves how AI handles long-term programming tasks. The researchers achieved this through a clever two-step training process.

Example CursorBench task

First, they immersed the model in massive amounts of code to build profound baseline knowledge. Then, they put it through rigorous reinforcement learning, simulating actual user sessions where the AI practiced solving diverse problems. To help the model stay perfectly focused during lengthy programming tasks, the team introduced a brilliant self-summarization technique.

Instead of getting overwhelmed by a long history of commands, the model constantly writes little internal summaries for itself. This preserves crucial context while discarding clutter.

To prove this approach works, the team bypassed traditional public tests and created a rigorous new evaluation based entirely on authentic engineering scenarios. In these demanding simulations, which require extensive code changes from incredibly brief instructions, the model demonstrated remarkable accuracy and coherence.

Claudini: Autoresearch Discovers State-of-the-Art Adversarial Attack Algorithms for LLMs

Panfilov et al. [MATS, ELLIS Institute Tubingen & Max Planck Institute for Intelligent Systems, Tubingen AI Center, Imperial College London]

♥ 1.5k   Coding Vulnerabilities   bycloud’s pick  

Can AI automatically conduct research to make future technology safer? Researchers built an automated security researcher to see if an AI could invent entirely new methods for pressure-testing digital vulnerabilities.

To test this, the research team built a continuous loop where the AI reviewed dozens of existing security-testing algorithms, wrote new software to improve them, and then evaluated its own creations. Instead of simply generating tricky text prompts to bypass safety filters, the AI engineered the underlying mathematical algorithms that actively search for these vulnerabilities.

Claudini Strongly Outperforms a Classical AutoML Method.

By intelligently splicing together previous techniques and writing clever mechanisms to avoid getting stuck during its analysis, the AI made massive strides. It successfully generated novel algorithms that dramatically outperformed human-made baselines, jumping from a success rate of less than ten percent to forty percent on a targeted safety filter.

The most remarkable finding is just how adaptable these new AI-authored algorithms proved to be. When scientists tested these methods on a completely different, highly secured model that the AI had never even encountered, the new algorithms bypassed the defenses with a flawless hundred percent success rate.

LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels

Maes et al. [Mila & Université de Montréal, New York University, Samsung SAIL, Brown University]

♥ 3.7k   LLM World Models  

AI needs an internal "world model" to predict the consequences of its actions. Researchers have tried teaching AI to build these models directly from raw video pixels, compressing complex visual scenes into a streamlined imagination space. However, this process faces a frustrating hurdle known as representation collapse.

When asked to predict the future, the AI often takes the lazy route, mapping every image to the exact same uniform representation just to guarantee a perfect prediction score. To prevent this, previous systems relied on highly fragile workarounds.

LeWorldModel Training Pipeline.

This paper developed LeWorldModel, which is a highly stable world model from scratch using just two simple rules. First, the model predicts the next compressed state of its environment based on a given action. Second, a clever mathematical regulator forces these compressed representations to stay continuously diverse, spreading them out to match a natural bell-curve distribution. By enforcing this varied shape, the model is strictly prevented from collapsing into a single, lazy answer.

Pseudo-code for the training procedure of LeWorldModel.

By reducing the complex tuning down to a single parameter, this compact model trains on one standard graphics card in just a few hours. Impressively, it plans up to forty-eight times faster than bulkier alternatives across complex control tasks. Even more fascinating, without explicit physics lessons, the model organically learns to track object locations and registers mathematical surprise when shown impossible events like spontaneous teleportation.

LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels

End-to-end joint-embedding predictive architecture from pixels.

le-wm.github.io

Self-Distillation of Hidden Layers for Self-Supervised Representation Learning

Lowe et al. [Vector Institute, Carleton University, Dalhousie University, University of British Columbia, University of Guelph]

♥ 780   Distillation  

Teaching an AI to understand the visual world is complex. For a long time, scientists have had to choose between two extreme training methods, both with frustrating limitations. One approach forces the AI to perfectly reconstruct raw, low-level pixels. While this keeps the system safely grounded in reality, it leaves the AI struggling to grasp big-picture concepts without a lot of extra hand-holding.

Multi-layer self-distillation with Bootleg.

The other approach asks the AI to predict only highly abstract, final-stage ideas. However, because the system is essentially generating its own study material in a continuous loop, it often becomes unstable, loses touch with the actual image, and completely breaks down during training.

To solve this, this research has introduced Bootleg. Instead of making the AI guess only the most basic details or only the most complex final concepts, they tasked it with predicting multiple hidden layers of information all at once. This beautifully mirrors how the human brain processes sights, where early visual processing picks up simple edges and colors, while deeper brain regions recognize complete objects.

By forcing the AI to simultaneously predict early, middle, and late stages of understanding, the system has to compress a wealth of varied knowledge through a tight informational bottleneck. This multi-level approach brilliantly keeps the AI anchored to actual visual features while it masters complex, abstract ideas.

Results for masked self-supervised learning with Bootleg and baselines.

The researchers discovered that this technique creates a dramatically smarter system, outperforming previous methods by significant margins in both recognizing what is in an image and mapping out exact scenes.

Mar 17th ~ Mar 24th
#100 Latest AI Research Explained Simply

🗞️ Industry News in 1 Line

1. ♥ 1.3k Xiaomi has announced the release of MiMo-V2-Pro, Omni, and TTS models. These new models introduce advanced features with global top-tier agent performance, multimodal interaction for seeing and hearing, and expressive voice synthesis. You can test these models on the web or via the API portal.

   

2. ♥ 1.5k MiniMax has launched its M2.7 model, which uses a recursive self-evolution architecture that contributed to an 88% win-rate over its predecessor. This model achieves state-of-the-art performance in software engineering benchmarks and high-fidelity document editing, while demonstrating enhanced agentic capabilities with 97% skill adherence. Try it today on the web.

   

Intuitive AI Academy - NEW Distillation Chapter!

My latest project: Intuitive AI Academy has the perfect starting point for you! We focus on building your intuition to understand LLMs, from transformer components, to post-training logic. All in one place.

We just added a new chapter on Distillaion!

We currently have an early bird offer, where you would get 40% off on the yearly plan for our early users.

Use code: TIMELINE

Check Out Intuitive AI Academy

Advertise with The AI Timeline!

Attention Residuals

Kimi Team

♥ 15k   Attention  

LLMs rely on residual connections to pass information through deep stacks of layers. While these connections act as a vital "gradient highway", they work like a blunt instrument. In current architectures, every layer simply adds its output to a uniform sum of all previous layers. This approach causes the information representing the model’s "hidden state" to swell uncontrollably as it moves deeper. Over time, this leads to a dilution effect where early, important information becomes buried and difficult for the model to retrieve effectively, effectively limiting how well the model can leverage its own depth.

Overview of Attention Residuals.

Researchers have introduced "Attention Residuals" (AttnRes) to replace this clumsy, fixed accumulation with a smarter mechanism. AttnRes allows each layer to selectively aggregate information from previous layers using learned, input-dependent weights. Instead of blindly adding everything together, the model now chooses which previous layers are most relevant to its current task.

PyTorch-style pseudo code for Block Attention Residuals.

To ensure this doesn't create excessive memory demands in massive models, the team developed "Block AttnRes," which organizes layers into smaller groups. This allows the model to maintain the benefits of selective, smart aggregation while keeping memory and communication costs efficient.

Experiments on large-scale models, including a 48B-parameter architecture, showed that this method successfully prevents the hidden-state growth that plagued previous models. By creating a more uniform distribution of signals and gradients across all layers, AttnRes consistently improves performance on complex reasoning, math, and coding benchmarks.

Mamba-3: Improved Sequence Modeling using State Space Principles

Lahoti et al. [Carnegie Mellon University, Princeton University, Together AI, Cartesia AI]

♥ 1.5k   Mamba   bycloud’s pick  

Transformer based AI models are incredibly powerful, but they suffer from a "bottleneck" in efficiency. As models grow, the computational effort required to generate each new piece of information, and the memory needed to store that context increases very fast.

This paper has introduced Mamba-3, a new architecture designed with an "inference-first" mindset to solve these efficiency challenges. By revisiting the mathematical foundations of State Space Models (SSMs), the team implemented three core improvements.

First, they developed a more expressive discretization method called "exponential-trapezoidal," which allows the model to handle data dynamics with greater precision than previous versions.

Second, they incorporated a complex-valued state update rule. This acts like a "rotary" mechanism that enables the model to track states, such as solving arithmetic parity tasks, that were previously impossible for similar linear models to master.

Finally, they shifted to a multi-input, multi-output (MIMO) formulation. This clever adjustment allows the model to perform more computation during the memory-heavy decoding phase without increasing the actual size of its state or slowing down its response time.

These refinements allow Mamba-3 to achieve a remarkable balance. At the 1.5B scale, it outperforms top-tier competitors in downstream accuracy while simultaneously matching the language-modeling capabilities of its predecessor at half the state size.

Prefill and Prefill+Decode latency across sequence lengths.

GitHub - state-spaces/mamba: Mamba SSM architecture

Mamba SSM architecture. Contribute to state-spaces/mamba development by creating an account on GitHub.

github.com/state-spaces/mamba

V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning

Mur-Labadia et al. [FAIR at Meta, Universidad de Zaragoza]

♥ 1.3k   JEPA  

To be truly useful, an AI needs to be a "jack-of-all-trades": it must understand both the big picture (like identifying a person’s action) and fine-grained local details (like pinpointing the exact edge of a glass for a robot to grasp). Previous models, such as the V-JEPA family, excelled at global understanding but struggled to extract precise local information, resulting in noisy, fragmented visual representations.

V-JEPA 2.1 Architecture

This gap limited their effectiveness in tasks that demand spatial precision, such as depth estimation or delicate robotic manipulation. The team discovered that the "missing link" was in the training objective. Traditional V-JEPA models only practiced predicting what was missing from an image or video, the masked, hidden patches.

V-JEPA 2.1 ViT-G performance across dense and global prediction tasks.

Because the model wasn’t forced to analyze the visible parts of the scene, it essentially learned to treat those visible sections as global summaries rather than detailed spatial maps. To fix this, this paper has introduced a "dense prediction loss," which forces the model to learn from both the hidden and the visible parts of the input.

By supervising every part of the scene, the model is compelled to build a coherent, fine-grained understanding of where objects actually exist in space. They further refined this by applying the learning signal at multiple layers deep within the network, a method called "deep self-supervision", and by using specialized tokenizers that handle images and videos in their native formats.

These changes, combined with scaling the model and the diversity of training data, resulted in a system capable of state-of-the-art performance in both high-level action forecasting and precise, low-level spatial tasks like depth perception.

GitHub - facebookresearch/vjepa2: PyTorch code and models for VJEPA2 self-supervised learning from video.

PyTorch code and models for VJEPA2 self-supervised learning from video. - facebookresearch/vjepa2

github.com/facebookresearch/vjepa2

Temporal Straightening for Latent Planning

Wang et al. [New York University, Brown University, University of Toronto]

♥ 1.2k   JEPA  

When AI agents interact with complex environments, they typically translate high-dimensional sensory data, like raw video, into a "latent space" to make decisions. For a computer trying to navigate, this means the shortest path between two points in its digital map doesn't actually correspond to the shortest physical path. Because these internal maps are so twisted, gradient-based planners, which rely on finding the smoothest way to reach a goal, often get stuck or perform poorly, forcing researchers to rely on computationally expensive search-based methods.

UMaze: ground-truth geodesic distance.

UMaze: ResNet-global after straightening.

This paper introduces a way to "straighten" these maps, helping AI agents see the world more linearly so they can plan their actions more efficiently. The researchers developed a method called "temporal straightening" to fix the distorted geometry of latent spaces.

Latent Curvature and Open-Loop GD Success Rate for Different Encoders. Higher cosine similarity indicates lower curvature.

This is based on a simple idea: as the agent moves, the model tracks its latent trajectory and minimizes the "curvature" by keeping the velocity vectors of consecutive steps as aligned as possible. It forces the AI to learn representations where movement in the latent space feels like travel along a straight, predictable line rather than a chaotic curve.

By adding a straightening objective to the standard training process, the model learns to map observations in a way that Euclidean distance, the simplest way to measure "how far" a goal is, finally matches reality. This has a profound effect on planning: because the path is straighter, the math behind gradient-based optimization becomes much more stable.

In experiments, this approach drastically improved success rates across a variety of navigation and manipulation tasks, allowing agents to reach goals with far greater precision and efficiency without needing the heavy compute power typically required for complex decision-making.

Mar 10th ~ Mar 17th
#99 Latest AI Research Explained Simply

🗞️ Industry News in 1 Line

1. ♥ 24k Claude 3.5 models (Opus and Sonnet) now support a 1-million-token context window, and allow users to process large codebases, extensive document sets, and up to 600 images or PDF pages per request. This update is available across all plans and is integrated by default into Claude Code at standard pricing.

   

2. ♥ 44k Google Maps has integrated Gemini AI to help you explore and navigate more easily. You can now use a new "Ask Maps" feature to get conversational answers to specific, real-world questions, like finding a place to charge your phone or a well-lit tennis court. Additionally, a new "Immersive Navigation" tool is rolling out, which provides vivid 3D visuals and more detailed route guidance to help you navigate your surroundings with more confidence.

3. ♥ 5.3k Hermes Agent by Nous Research is an open-source, Python-based tool designed to grow with you by utilizing a multi-level memory system and persistent machine access, similar to OpenClaw. It works across your CLI and various messaging platforms, and offers developers an extensible framework for complex tasks like subagent management and programmatic tool calling.

   

4. ♥ 6.2k Covenant-72B is the largest decentralized LLM pre-training run in history. It is a 72B parameter model trained across commodity internet connections without centralized clusters or whitelisting. By using innovative techniques like SparseLoCo for bandwidth efficiency and a blockchain-based "Gauntlet" system for validation, the project achieved performance levels competitive with models trained in traditional data centers. Try it in browser.

   

5. ♥ 19k Yann LeCun’s new startup Advanced Machine Intelligence (AMI) has secured $1.03 billion in one of the largest seed rounds in history to develop AI systems capable of advanced reasoning, persistent memory, and world-model understanding. These models are designed to understand the physical world while featuring persistent memory and the ability to reason, plan, and operate safely.

Intuitive AI Academy - NEW Distillation Chapter!

My latest project: Intuitive AI Academy has the perfect starting point for you! We focus on building your intuition to understand LLMs, from transformer components, to post-training logic. All in one place.

We just added a new chapter on Distillation too!

We currently have an early bird offer, where you would get 40% off on the yearly plan for our early users.

Use code: TIMELINE

Check Out Intuitive AI Academy

Advertise with The AI Timeline!

OpenClaw-RL: Train Any Agent Simply by Talking

Wang et al. [Princeton University]

♥ 674   RL  

Every time we interact with an AI, it receives immediate feedback: a follow-up question, a software error, or a screen transition. Existing AI systems throw this valuable experience away. They treat our replies merely as context for their very next move, missing a massive opportunity to learn. Researchers wanted to solve this by capturing these everyday reactions, what they call next-state signals, and turning them into a live, continuous learning stream.

We can build a system where an AI naturally improves simply by being used, turning ordinary conversations and software tasks into a seamless training loop without needing to pause for offline updates. The researchers built a unified framework called OpenClaw-RL.

The AI receives two powerful forms of information. First, there are evaluative signals, which act like a simple score indicating whether an action succeeded or frustrated the user. Second, there are directive signals. When a user corrects an AI by explaining how it should have responded, or when a software tool outputs a detailed error, it provides a clear map for improvement.

The framework extracts these specific textual hints to give the AI rich, word-by-word guidance. Because the system is built asynchronously, the AI can chat with a user, a background judge can evaluate its performance, and a training engine can update the AI's core behavior all at the exact same time, without ever interrupting the workflow.

By combining basic scoring with these rich textual hints, researchers found that personal assistants rapidly adapt their tone, becoming much more natural after just a handful of conversations.

Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights

Gan and Isola [MIT CSAIL]

♥ 818   Pretraining   bycloud’s pick  

Teaching an AI a new skill feels like searching for a microscopic needle in a haystack. Researchers have long believed that adapting a massive, billion-parameter model required highly complex, meticulous, step-by-step mathematical adjustments just to find a version of the system that performed a specific task well. Blindly guessing the right settings was considered mathematically impossible and entirely out of the question.

However, this study discovered that as models grow larger and undergo extensive initial training, adapting them to specific tasks like mathematical reasoning or coding no longer requires such rigid, painstaking effort. The heavy lifting of learning has already been done, signaling an exciting era where customizing powerful technology is becoming surprisingly natural, fast, and accessible.

The researchers found that scaling up these models fundamentally transforms their underlying structure. Instead of a desolate landscape where a good solution is a lone needle, massive models are surrounded by a dense, flourishing "thicket" of specialized solutions.

By making random, tiny adjustments to the model's underlying numbers, the researchers uncovered an abundance of hidden specialists waiting nearby. One random tweak might produce an expert in creative writing, while another creates a brilliant chemist, each specializing in one area while forgetting others.

To harness this rich diversity, the team tested a beautifully simple approach: they generated thousands of random tweaks simultaneously, kept the top performers for a specific task, and had them vote on the final answer. This parallel guess-and-check strategy matched the accuracy of today’s most advanced training methods but operated in a fraction of the time because it avoided slow, sequential updates.

Neural Thickets · MIT

Diverse Task Experts Are Dense Around Pretrained Weights

thickets.mit.edu

IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse

Bai et al. [Tsinghua University, Z.ai]

♥ 540   Attention  

AI models use an attention mechanism to connect different pieces of information, but as the text gets longer, this demands a staggering amount of computing power. Developers recently introduced a clever shortcut called sparse attention, which uses a specialized "indexer" tool to scan the text and pick only the most relevant words for the AI to focus on at each step. It is a brilliant fix, but researchers ran into a new bottleneck. This indexer operates independently at every single layer of the AI network. As the text grows, just running this indexer consumes a massive chunk of the system's processing time, slowing everything down.

Side-by-side comparison of inference loops.

Looking closely at how these models process data, researchers noticed an incredible inefficiency. They discovered that consecutive layers of the AI were repeatedly selecting almost the exact same important words, often with a near-perfect overlap. To solve this, the researchers developed an elegant, hopeful solution called IndexCache.

Instead of forcing every layer to do the heavy lifting of scanning and selecting information, IndexCache designates a few specific layers as "Full" layers to run the indexer. The remaining layers become "Shared" layers, which simply borrow the selected words from the nearest Full layer. The team created two ways to apply this: a training-free method that calculates the absolute best pattern of Full and Shared layers for existing models, and a training-aware method that actively teaches the AI to share this data.

Training-free IndexCache at 1/2, 1/4, and 1/8 indexer retention. ‘Long’ and ‘G&R’ aggregate benchmark scores.

By simply reusing this cached information, researchers eliminated 75 percent of the heavy indexer computations with negligible drops in the model's reasoning quality. When tested on massive systems, this straightforward change nearly doubled the speed at which the AI reads information and significantly accelerated its ability to generate answers.

Training Language Models via Neural Cellular Automata

Lee et al. [MIT, Improbable AI Lab]

♥ 1.5k   LLM Sampling  

LLMs rely on massive amounts of human-written text to learn how to reason and communicate. However, this approach faces a looming wall: high-quality human data is finite, often riddled with biases, and mixes actual reasoning with messy, subjective language. They investigated whether models could learn the fundamental mechanics of reasoning by training on purely synthetic, non-linguistic data before ever seeing a human sentence.

To test this, researchers turned to neural cellular automata (NCA), algorithmic systems that generate complex, ever-changing grid patterns using simple, local rules. Unlike static text, these patterns can be generated cheaply and in infinite supply, allowing for precise control over the "complexity" of the training data.

The team pre-trained models on these synthetic grid trajectories, then followed up with standard training on natural language. The results show that models that practiced on just 164 million synthetic tokens learned faster and performed better than those trained solely on significantly larger amounts of traditional internet text.

While traditional text training can cause a model to rely on human biases or semantic shortcuts, the synthetic grids force the model to focus purely on tracking long-range patterns and inferring underlying rules. Furthermore, the researchers found they could tune the complexity of this synthetic data to match specific domains; for instance, code benefited from simpler rules, while math and web text thrived on higher complexity.

Training Language Models via Neural Cellular Automata

hanseungwook.github.io/blog/nca-pre-pre-training

GLM-OCR Technical Report

Duan et al. [Zhipu AI, Tsinghua University]

♥ 1.1k   LLM RL  

Modern information systems rely heavily on extracting knowledge from complex, visually dense documents like financial reports, invoices, and scientific papers. While recent multimodal AI models have improved how we read these documents, they often suffer from a major drawback: their massive size makes them slow, memory-intensive, and difficult to deploy in practical, real-world settings.

Architecture and workflow of the GLM-OCR framework

The researchers developed GLM-OCR, a lightweight, highly optimized framework that packs significant power into a compact 0.9-billion-parameter design. The system merges a specialized visual encoder with a streamlined language decoder. What makes it particularly clever is the shift away from standard “one-token-at-a-time” generation, which is notoriously slow for structured documents.

Instead, the researchers implemented a Multi-Token Prediction mechanism that allows the model to predict several tokens simultaneously. By using a shared-parameter scheme to keep memory usage low, the model effectively boosts throughput, the speed at which it processes data, by roughly 50% without sacrificing accuracy.

To handle real-world complexities, the system employs a two-stage pipeline. First, it uses an analysis module to detect the layout of a document, breaking a complex page into manageable regions. These regions are then processed in parallel, allowing for faster and more robust recognition of everything from handwritten text to complicated table structures.

zai-org/GLM-OCR · Hugging Face

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

huggingface.co/zai-org/GLM-OCR

Mar 3rd ~ Mar 10th
#98 Latest AI Research Explained Simply

🗞️ Industry News in 1 Line

1. ♥ 6.8k Sarvam AI has announced the open-source release of its 30B and 105B parameter models, which were developed entirely in-house to target both global benchmarks and Indian language tasks. The model weights are now available on Hugging Face and AIKosh, featuring day-zero support for SGLang with vLLM compatibility expected soon.

   

2. ♥ 23k OpenAI has launched GPT-5.4 Thinking and GPT-5.4 Pro models across ChatGPT, the API, and Codex. These new models are designed to enhance reasoning, coding, and agentic workflows. It also includes features such as advanced deep web research capabilities and the ability for users to interrupt the model mid-process to provide real-time instructions or course corrections.

   

3. ♥ 15k Andrej Karpathy has developed an "autoresearch" agentic workflow designed to autonomously optimize neural network training through iterative experimentation. When applied to his nanochat project, the tool identified 20 additive changes that reduced the "Time to GPT-2" training benchmark from 2.02 hours to 1.80 hours.

   

Intuitive AI Academy - NEW MoE Chapter!

My latest project: Intuitive AI Academy has the perfect starting point for you! We focus on building your intuition to understand LLMs, from transformer components, to post-training logic. All in one place.

We just added a new chapter on MoE, that goes through the history, the key techniques, and the current state of MoE that frontier model uses. With over 10,000 words written!

We currently have a early bird offer, where you would get 40% off yearly plan for our early users.

Use code: TIMELINE

Check Out Intuitive AI Academy

Advertise with The AI Timeline!

Speculative Speculative Decoding

Kumar et al. [Stanford University, Princeton University, Together AI]

♥ 22k   Decoding  

The primary hurdle in making LLMs feel instantaneous is a phenomenon known as the sequential bottleneck. Standard AI models generate text one word, or "token", at a time, which fails to fully use the massive parallel computing power of modern hardware. Researchers previously introduced "speculative decoding", a method where a small, fast model drafts a few guesses for a larger model to verify. The drafting model has to wait for the larger model to finish checking its work before it can start guessing the next set of words. This creates a persistent lag that limits how fast even the most advanced systems can communicate.

Ordinary speculative decoding (SD) requires the verifier to wait idly for the draft to speculate.

To eliminate this idle time, researchers have developed Speculative Speculative Decoding (SSD) via an optimized algorithm called Saguaro. The biggest change is decoupling the "guesser" from the "checker" entirely. While the large target model is busy verifying a current batch of text, Saguaro’s draft model looks ahead and predicts several possible outcomes of that verification. It prepares a menu of "potential futures." If the larger model confirms one of these predicted outcomes, the system can immediately provide the next set of words without any drafting delay. This parallel approach effectively hides the time spent guessing, transforming a sequential process into a streamlined, continuous flow.

By using a clever "geometric fan-out" strategy, Saguaro focuses its computational effort on the most likely verification results, ensuring the "speculation cache" is highly accurate even at higher creative temperatures. This method is entirely lossless, meaning it achieves the exact same high-quality output as the original model but at much higher speeds.

Advantage of geometric fan out strategy increases at higher temperatures, improving both speculation cache hit rate (right) and thus end-to-end speed (left).

Initial results show that Saguaro can deliver text up to five times faster than traditional generation methods and twice as fast as previous state-of-the-art speculative techniques.

Beyond Language Modeling: An Exploration of Multimodal Pretraining

Tong et al. [FAIR, New York University]

♥ 424   Multimodal LLMs   bycloud’s pick  

AI is getting good at manipulating language, but it lacks a fundamental grasp of the physical world. Researchers describe this limitation using the "allegory of the cave": current models have mastered the description of shadows on a wall, text, without ever seeing the actual objects casting those shadows. Because text is a human abstraction, it is a "lossy" version of reality that misses the raw physics, geometry, and causality of our environment.

Overview of this study.

By training models to "see" and "read" simultaneously from birth, we can build a more grounded intelligence that understands the world’s dynamics directly.

The researchers developed a unified model using a framework called Transfusion, which trains a single "brain" to perform two different tasks at once: predicting the next word in a sequence and reconstructing visual frames through a process called diffusion.

Examples of training data.

They discovered that the most effective way to do this is by using a single, high-quality visual representation known as a Representation Autoencoder. This contradicts the traditional belief that you need different "eyes" for understanding an image versus creating one; instead, a unified representation excels at both while keeping the model’s language skills sharp.

The study also showed that learning from diverse visual data, like video and image-text pairs, actually improves the model’s performance on downstream tasks like reasoning and planning. To manage this complexity, they utilized a "Mixture-of-Experts" architecture.

Multimodal co-training exceeds unimodal performance.

This design allows the model to naturally evolve specialized internal "experts" for different tasks. It learned to dedicate more capacity to language while efficiently processing the massive amounts of data required by vision. Most impressively, this unified training allowed "world modeling" capabilities to emerge.

The model could predict the physical outcome of actions, like navigating a robot through a room, using simple text commands, proving that a truly multimodal foundation can bridge the gap between human language and physical reality.

Scaling laws for unified dense models.

Beyond Language Modeling: An Exploration of Multimodal Pretraining

Empirical insights on representation, data, architecture, and scaling for native multimodal pretraining.

beyond-llms.github.io

FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling

Zadouri et al. [Princeton University, Meta, Colfax Research, NVIDIA, Georgia Tech, Together AI]

♥ 430   Attention  

Latest AI hardware is significantly faster at basic matrix multiplication, but other components, like the units responsible for specialized math or moving data around, haven't kept the same pace. This creates a digital traffic jam where the fastest parts of a processor are frequently left idling, waiting for slower sections to finish their work.

FlashAttention-4 was designed to bridge this gap, by offering a clever software design that can overcome the physical limitations of even the most advanced hardware.

FlashAttention-4 forward pipeline.

The researchers taught the software to find creative shortcuts around these hardware bottlenecks. Because the chip’s dedicated unit for calculating exponentials is often the slowest link in the chain, they developed a way to emulate its functions using more plentiful, general-purpose math units.

FlashAttention-4 backward computation graph (5 MMA operations + 2 elementwise operations), showing the 1-CTA MMA mode software pipeline order across the prologue, main loop, and tail.

By using a mathematical technique called polynomial approximation, the software effectively mimics the specialized unit, achieving nearly identical accuracy at much higher speeds. Additionally, they introduced a "conditional rescaling" method that intelligently skips redundant calculations unless they are truly necessary for precision.

Backward pass TFLOPS on B200 (FP16/BF16) with head dimension 128.

By keeping more data in high-speed "tensor memory" and coordinating tasks so that different parts of the chip work in perfect sync, FlashAttention-4 can reach incredible speeds, hitting up to 1613 trillion operations per second on the latest Blackwell GPUs.

SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration

Chen et al. [Sun Yat-sen University, Alibaba Group]

♥ 855   LLM Evaluation  

Software engineering is rarely about writing a perfect piece of code in one go; it is an ongoing marathon of maintenance, updates, and evolving requirements. While current AI models have become quite good at solving isolated, one-shot coding tasks, these models are often tested on their ability to provide a quick fix rather than their capacity to sustain a healthy codebase over time.

Unlike previous benchmarks, SWE-CI proposes an evolution-based evaluation.

To address this, a new benchmark called SWE-CI shifts the focus from simple functional correctness to long-term maintainability, providing a much-needed lens into how AI agents handle the messy, continuous reality of software development.

The researchers built this benchmark using actual evolutionary histories from real-world software repositories, with tasks spanning an average of seven months and dozens of consecutive updates. To mirror a professional environment, they employed a dual-agent system where one AI acts as an Architect to identify gaps and set requirements, while a second Programmer agent implements the changes.

Data curation process of SWE-CI.

Success is measured by a novel metric called EvoScore, which specifically rewards agents that make decisions facilitating future growth rather than just immediate fixes.

The results show that the newest AI models are improving at an accelerating pace, but most still struggle to prevent regressions, the frustrating phenomenon where adding a new feature accidentally breaks an old one. This discovery highlights that the next great frontier for AI developers isn't just writing code that works, but writing code that lasts.

SWE-CI uses an architect-programmer dual-agent workflow to model the continuous integration cycle of professional software teams in the real world.

Real Money, Fake Models: Deceptive Model Claims in Shadow APIs

Zhang et al. [CISPA Helmholtz Center for Information Security]

♥ 805   LLM RL  

Access to frontier models is restricted by high costs, complex payment barriers, or geographical limitations. To bridge this gap, many researchers and developers have turned to "shadow APIs", third-party services that promise the same power as official models like GPT-5 or Gemini through unofficial, often cheaper channels. While these services appear to democratize access, they operate in a digital gray market with almost no transparency.

Researchers recently set out to investigate whether these shadow services are truly delivering what they advertise or if they are quietly undermining the integrity of the scientific work built upon them.

Landscape of the shadow APIs.

The investigation revealed a concerning gap between marketing promises and technical reality. By auditing these services through "model fingerprinting", a technique that identifies an AI by analyzing the unique statistical patterns and "signatures" in its responses, researchers discovered that nearly half of the shadow services were not using the models they claimed.

In a classic "bait-and-switch", premium proprietary models were frequently swapped for cheaper, open-source alternatives behind the scenes. This deception led to significant performance collapses; in high-stakes fields like medicine and law, accuracy dropped by nearly half when compared to official versions.

Fingerprinting results via LLMmap matched model and mean cosine distance D with standard.

The researchers found that while these shadow services might handle simple tasks well, they often fail during complex reasoning and display unpredictable safety behaviors. In addition to identifying the mismatch, the study used statistical testing to prove that the outputs from these services were fundamentally different from the official sources.

Feb 24th ~ Mar 4th
#97 Latest AI Research Explained Simply

🗞️ Industry News in 1 Line

1. ♥ 10k Google has begun rolling out Nano Banana 2, its latest image generation model. The updated model uses real-time web search data to improve real-world accuracy and introduces the ability to render clear, multilingual text for designs like posters and logos. Additionally, Nano Banana 2 brings faster generation speeds alongside enhancements to lighting, textures, and overall image detail. Try it today via the Gemini app and web interface.

   

2. ♥ 3.7k OpenAI has signed a classified deployment contract with the Department of War, insisting that a "cloud-only" architecture, an internal safety stack, and cleared engineers will somehow strictly prevent the military from using their models for autonomous lethal weapons or mass NSA surveillance. Entrusting a tech corporation to independently self-police lethal and intelligence applications sounds straight out of a black mirror episode.

3. ♥ 20k Alibaba's Qwen team has launched the Qwen 3.5 Small Model Series, a family of native multimodal models ranging from a highly compact 0.8B to a remarkably capable 9B designed for edge devices and lightweight agents. Both the base and instruct models are now available on Hugging Face and ModelScope. Additionally, the entire suite is already optimized for local deployment via Ollama.

   

4. ♥ 8k After small series, Alibaba has also introduced the Qwen 3.5 Medium Model Series, which includes the 27B, 35B-A3B, and 122B-A10B models designed to bridge the gap between mid-sized and frontier AI capabilities. Highlighting a shift toward architectural efficiency over parameter size, the new 35B-A3B model notably outperforms the previous generation's massive 235B model thanks to improved data quality and reinforcement learning. Try it in browser.

5. ♥ 8.2k Google has announced Gemini 3.1 Flash-Lite, its fastest and most cost-effective Gemini 3 model to date. It has dynamic "thinking levels" and the model instantly processes high-volume queries while scaling its reasoning for complex edge cases, delivering a 2.5X faster time-to-first-token than its 2.5 Flash predecessor. Try it in browser.

   

Intuitive AI Academy - NEW MoE Chapter!

My latest project: Intuitive AI Academy has the perfect starting point for you! We focus on building your intuition to understand LLMs, from transformer components, to post-training logic. All in one place.

We just added a new chapter on MoE, that goes through the history, the key techniques, and the current state of MoE that frontier model uses. With over 10,000 words written.

We currently have a early bird offer, where you would get 40% off yearly plan for our early users.

Use code: TIMELINE

Check Out Intuitive AI Academy

Advertise with The AI Timeline!

Learning Without Training

Ryan O’Dowd [Claremont Graduate University]

♥ 720   LLM Training  

Engineers usually build machine learning models by guessing a structure and running exhaustive optimization processes to train them. Researchers wanted to know if there was a more elegant way to overcome the hurdles of high-dimensional, noisy data without relying on these brute-force training methods.

The current approach assumes a model will eventually learn the underlying patterns, but it lacks constructive mathematical guarantees. By rooting their approach in classical approximation theory, we can save immense computational power while tackling complex problems like tracking brain diseases or analyzing hyperspectral images.

Normalized histogram of the density of interest (left), paired with our density estimation by σ128 based on 3900 samples (right).

This paper discovered how to mathematically construct highly accurate models directly on unknown, complex data surfaces, known as manifolds. This method bypasses the need to map out the entire geometry of the dataset first; it only requires knowing the dimension of the data.

The team unlocked a breakthrough in transfer learning by figuring out how to successfully lift learned information from just a localized portion of one data space and apply it to a completely different domain. As a result, adapting a massive model to a new problem no longer requires processing the entire original dataset.

Finally, the researchers reimagined data classification by treating it like a signal separation problem. By mathematically estimating the underlying sources of these signals, their new algorithm quickly zeroes in on the most informative data points.

Doc-to-LoRA: Learning to Instantly Internalize Contexts

Charakorn et al. [Sakana AI, Minerva University]

♥ 1.4k   LoRA   bycloud’s pick  

Let’s assume you asked an AI to analyze a massive technical manual. Currently, every time you ask a follow-up question, the system has to re-read the entire document. This repetitive reading eats up massive amounts of computing power, memory, and time.

While researchers can technically train the AI to memorize the document permanently, that traditional training process is painfully slow, expensive, and completely impractical for quick updates.

To solve this, researchers developed a brilliant workaround called Doc-to-LoRA, or D2L. Instead of forcing the main AI to constantly re-read text or undergo grueling training, they built a specialized, lightweight helper system called a hypernetwork.

This helper reads the document exactly once and instantly generates a tiny, customized plug-in. Think of it like instantly downloading a new skill directly into the AI's brain. Once this plug-in is attached, the main AI can answer subsequent queries fluidly without ever needing the original text in its prompt. It performs this complex mental compression in just a single step, completely bypassing the steep costs of standard training.

QA performance on SQuAD compared to the used context length ratio (left), update latency (middle), and additional memory needed for model updates (right).

The results are incredibly promising. In testing, D2L successfully hunted down specific facts hidden inside massive walls of text, achieving near-perfect accuracy on documents over four times larger than the AI’s normal limits.

It works drastically faster and uses far less memory than previous memorization methods. It can even translate visual information from image-based models into these text plug-ins, allowing a text-only AI to classify images.

Long document QA performance. LLMLingua-2 compresses the input with [20%, 40%, 60%, 80%, 90%] compression rates from right to left (gray dots).

The Geometry of Noise: Why Diffusion Models Don't Need Noise Conditioning

Sahraee-Ardakan et al. [Google]

♥ 382   Diffusion Models  

Can you restore a severely damaged painting without knowing how much damage was originally done? Standard diffusion models avoid this problem by relying on a strict timer that tells them exactly how much "noise" or corruption they are dealing with at any given step.

Recently, researchers have been incredibly hopeful about "autonomous" models that strip away this timer, learning a single rule to handle everything from pure static to nearly perfect data. However, this creates a profound mathematical paradox.

As these blind models approach the clean data, the underlying mathematical landscape forms an infinitely deep pit. The directional signals diverge completely, creating a severe geometric singularity. By all conventional logic, these models should become hopelessly unstable and crash.

The Singular Geometry of the Marginal Energy Landscape.

Yet, researchers have beautifully resolved this mystery. They discovered that these autonomous systems are actually charting a course across a unified map called "Marginal Energy."

More importantly, the scientists proved that the models naturally develop a hidden geometric shock absorber. As the AI approaches that infinitely deep mathematical pit, this built-in feature perfectly counteracts the extreme steepness. It transforms a catastrophic plunge into a smooth, stable descent known as a Riemannian gradient flow.

Generative performance on Fashion MNIST.

Models attempting to directly predict the noise act like faulty amplifiers, magnifying tiny errors until the system catastrophically breaks down. Conversely, models based on predicting "velocity" inherently absorb that uncertainty into a smooth, stable drift.

dLLM: Simple Diffusion Language Modeling

Zhou et al. [UC Berkeley, UIUC]

♥ 120   Diffusion LLM  

Language models traditionally generate text strictly left to right. But recently, researchers have found a promising alternative: diffusion language models. These systems can generate words in any order and iteratively refine their answers, unlocking highly flexible AI.

GitHub - ZHZisZZ/dllm: dLLM: Simple Diffusion Language Modeling

dLLM: Simple Diffusion Language Modeling. Contribute to ZHZisZZ/dllm development by creating an account on GitHub.

However, the underlying code was scattered across complex, isolated research repositories, making it incredibly difficult for developers to reproduce results or build upon each other’s work. To solve this, researchers created dLLM, a unified open-source framework that elegantly standardizes the development pipeline so the community can innovate together.

Inference pipeline: sampler swap from vanilla to FastdLLM MDLM sampler.

The framework seamlessly connects the core pillars of AI development: training, generation, and testing. Using a highly modular design, dLLM allows developers to snap different components together effortlessly. A researcher can easily swap out a training method or plug in a high-speed generation algorithm without rewriting the model's core architecture.

By standardizing how these models are evaluated, researchers also uncovered a hidden quirk: diffusion models are intensely sensitive to tiny adjustments in generation settings. A single tweaked parameter can drastically alter a model's performance, highlighting exactly why a transparent, shared testing environment is vital for meaningful progress.

Terminal Visualizer showing transition from masked to decoded tokens.

Perhaps the most hopeful breakthrough is how this framework democratizes AI research. The team proved that building these dynamic models does not require massive supercomputers. Using their new recipes, they successfully transformed standard, off-the-shelf systems (including traditional discriminative architectures like BERT) into functional diffusion chatbots.

Sensitivity to decoding hyperparameters.

They achieved this with minimal computing power and simple fine-tuning, requiring no structural changes to the original models.

dllm-hub (dLLM)

Org profile for dLLM on Hugging Face, the AI community building the future.

Feb 19th ~ Feb 24th
#96 Latest AI Research Explained Simply

🗞️ Industry News in 1 Line

1. ♥ 8.2k Replit has introduced Replit Animation, a new tool that lets users create animated videos in minutes using conversational prompts. It is powered by Gemini 3.1 Pro and makes it easier to produce polished, shareable content without traditional editing software.

   

2. ♥ 22k Anthropic has released Claude Sonnet 4.6, which comes with stronger performance in complex spreadsheet tasks, multi-step web forms, and expanded integrations through Excel MCP connectors. The Claude API now also supports more accurate web search, dynamic filtering, and general availability of code execution, memory, and programmatic tool use.

   

3. ♥ 49k Anthropic has announced Claude Code Security to scan codebases for vulnerabilities and suggest targeted patches for human review. This release triggered a sharp selloff in cybersecurity stocks, with companies like JFrog, CrowdStrike, Okta, and Cloudflare all recording declines. The system uses reasoning to trace data flows and flag subtle errors, while enforcing a human-in-the-loop safeguard to ensure developer oversight. If you want to try it, then join the waitlist here.

Learn LLMs Intuitively - Intuitive AI Academy

Want to learn about LLMs, but never have a good place to start?

My latest project: Intuitive AI Academy has the perfect starting point for you! We focus on building your intuition to understand LLMs, from transformer components, to post-training logic. All in one place.

content overview

We currently have a early bird offer, where you would get 40% off yearly plan for our early users.

Use code: TIMELINE

Check Out Intuitive AI Academy

Advertise with The AI Timeline!

Unified Latents (UL): How to train your latents

Heek [Google DeepMind Amsterdam]

♥ 2.1k   ALDiffusion  

To create high-quality images and videos efficiently, models usually compress data into a "latent" space (a digital shorthand that represents the original image in a smaller, more manageable package). If you compress the data too much, the AI loses fine details like textures and sharp edges; if you don't compress it enough, the AI becomes incredibly slow and expensive to train.

Schematic overview of our model, include the Encoder, the prior latent diffusion model, and the diffusion decoder model.

Traditional methods often rely on manual tuning to strike this balance, which is often more of an art than a science. Researchers have recently introduced a framework called Unified Latents (UL) to turn this guesswork into a systematic, more efficient process. By co-training the compression and generation steps together, they have found a way to maintain stunningly high-quality details while actually lowering the computational cost required for training.

Instead of treating the compressed representation as a static container, Unified Latents uses a "diffusion prior" to monitor and regularize the information flow. By linking the noise level produced during the encoding process directly to the precision of the diffusion model, researchers could create a mathematically tight way to control the latent bitrate.

A selection of samples from a text-to-image trained with Unified Latents

They also paired this with a diffusion-based decoder, which is remarkably better at reconstructing high-frequency details than previous methods. This unified approach allows the system to navigate the trade-off between compression and quality with much greater precision.

Fast KV Compaction via Attention Matching

Zweiger et al. [Massachusetts Institute of Technology]

♥ 196   Attention   bycloud’s pick  

AI models can take on more complex tasks like long-form coding and multi-day conversations, but they face a significant memory hurdle known as the key-value (KV) cache bottleneck. Every word the model processes adds to a digital "short-term memory" that can quickly balloon into several gigabytes of data.

Until now, researchers have managed this by either summarizing the text (which often strips away vital nuances) or by using expensive optimization techniques that take hours of computing time to compress a single document.

This paper has introduced a technique called Attention Matching that changes how we think about memory compaction. Instead of relying on slow, iterative training to condense information, this approach treats memory like a mathematical puzzle that can be solved directly in "latent space".

By focusing on how a model "pays attention" to specific pieces of information, the researchers found they could create a compact version of the memory that mimics the original's behavior. They discovered that this problem can be broken down into smaller sub-problems with efficient, closed-form solutions, allowing them to bypass the slow trial-and-error process of traditional machine learning.

Accuracy vs. compaction ratio across methods.

This new framework can shrink a model's memory by up to 50 times in just a matter of seconds, rather than hours, with almost no impact on the quality of the model's output.

Experiential Reinforcement Learning

Shi et al. [KRAFTON, University of Wisconsin–Madison, UC Berkeley, Microsoft Research]

♥ 1k   LLM RL  

Teaching artificial intelligence to navigate complex tasks is often a game of high-stakes guessing. In standard reinforcement learning, a model typically receives a single "reward" signal. This makes it incredibly difficult for the AI to pinpoint exactly where it tripped up or how to adjust its behavior for the next try.

Researchers recognized that this "blind" trial-and-error is far less efficient than how humans naturally learn. When we fail at a task, we don't just try again at random; we stop, reflect on what went wrong, and form a mental plan to do better. To solve this, researchers introduced Experiential Reinforcement Learning (ERL), a new approach designed to turn silent failures into structured, durable lessons.

In Experiential Reinforcement Learning (ERL), instead of learning from feedback or outcome directly

The researchers developed a clever "experience-reflection-consolidation" loop that embeds human-like reasoning directly into the training process. Instead of moving on immediately after an attempt, the model receives feedback from its environment and is prompted to generate a verbal reflection on its own performance.

It looks at its errors and produces a self-critique that guides a refined second attempt at the same task. If this second try succeeds, the model "internalizes" the successful correction. This is the breakthrough moment: by training the model to reproduce the improved behavior from the original task alone, the AI eventually learns to skip the reflection step entirely.

Conceptual comparison of learning dynamics in RLVR and Experiential Reinforcement Learning (ERL)

This ensures that the final model is both smarter and faster, maintaining high performance at deployment without any extra computational cost.

In complex multi-step tasks like Sokoban, which require deep planning and spatial reasoning, ERL improved performance by a staggering 81% over standard methods. It also showed reliable gains in agentic reasoning tasks that involve using external tools to answer questions.

Overview of Experiential Reinforcement Learning (ERL).

By allowing the AI to accumulate "corrective knowledge" in a persistent memory, researchers have created a way for models to build on their past successes rather than repeating the same mistakes.

GLM-5: from Vibe Coding to Agentic Engineering

Zhipu AI & Tsinghua University

♥ 290   LLM Agents  

Researchers are working to shift our relationship with AI from "vibe coding" (where humans provide constant prompts) toward a more autonomous era of "agentic engineering." Traditional models often struggle with the sheer computational cost of keeping track of long conversations, or they lose their way during tasks that require hours of planning and execution.

GLM-5 was designed to overcome these bottlenecks by creating a more independent assistant that can plan, implement, and iterate on technical challenges with minimal human intervention.

To make this possible, the researchers introduced a dynamic mechanism called DeepSeek Sparse Attention. Rather than the model straining to analyze every single word in a massive document with equal intensity (which is incredibly expensive and slow), it now identifies which tokens are truly important for the task at hand.

This allows the model to manage massive amounts of data, such as entire codebases or long-term business simulations, while significantly reducing the hardware power required. The model also features an "interleaved thinking" process where it pauses to reason before every action it takes.

It can "preserve" these thoughts across a long conversation, ensuring it doesn't lose its train of thought when moving between different stages of a project. By using a new asynchronous training infrastructure that allows the model to learn from complex, it was able to reach an unprecedented ability to solve end-to-end software engineering challenges.