Key Takeaways

• OpenAI's "Operator," an AI agent expected to release in January 2025, aims to execute tasks directly on users' computers, competing with Anthropic and Google in consumer-facing AI tools.

• Nous Research’s Forge API introduces reasoning enhancements via Monte Carlo Tree Search, Chain of Code, and a model mixture strategy, outperforming competitors in math reasoning benchmarks.

• Qwen 2.5 demonstrated SOTA coding capabilities, matching GPT-4o’s performance across multiple programming languages and benchmarks.

• FrontierMath evaluates AI systems on challenging mathematical reasoning, with experts confirming its rigor and AI’s limited ability to solve complex problems.

• Meta’s Watermark Anything Model redefined watermarking as segmentation, achieving over 85% accuracy in detecting watermarked areas and resilience to manipulations like splicing.

• CDXFormer improved spatial-temporal context analysis in satellite images using XLSTM, achieving SOTA performance across benchmarks with reduced computational costs.

Got forwarded this newsletter? Subscribe below👇

The Latest AI News

It’s been a little while, but there were plenty of releases last week. OpenAI isn’t showing any signs of slowing down and are discussing the release of an AI agent in January 2025. Sutskever brought up some interesting results from scaling-up pre-training training as well. We also saw a bunch of cool models from Google with unique applications, including flood forecasting and predicting molecular structures. 

The release of Qwen 2.5 was surprising as well with how well it performed on various benchmarks, along with a new math benchmark that received praise from some of the most well-known mathematicians in the world.

OpenAI and Rivals Pivot from Scaling to Advanced Reasoning

We’re seeing a shift in the AI industry’s approach to developing LLMs. Companies like OpenAI are moving away from the "bigger is better" philosophy of simply scaling up models with more data and computing power. Instead, they're exploring more sophisticated techniques that mimic human-like reasoning.

OpenAI's recently released o1 model is a perfect example of how we’re moving toward this new direction. It uses "test-time compute," a technique that enhances AI models during the inference phase. This method lets models generate and evaluate multiple possibilities in real-time, dedicating more processing power to challenging tasks that require complex reasoning.

Ilya Sutskever, co-founder of AI labs Safe Superintelligence (SSI) and OpenAI, mentioned results from scaling up pre-training have plateaued, marking a transition from "the age of scaling" to "the age of wonder and discovery."

Some effects of this shift could include:

• It may reshape the AI arms race, with companies focusing on developing more efficient reasoning techniques rather than just increasing model size.

• The demand for computational resources could change, which might affect companies like Nvidia that have dominated the AI chip market.

• It could lead to more distributed, cloud-based servers for inference.

Other major AI labs, including Anthropic, xAI, and Google DeepMind, are reportedly working on their own versions of these advanced reasoning techniques. This means we might be in a time where innovation in model architecture and training methods may become as crucial as raw computational power.

OpenAI Prepares 'Operator' Launch and Shares LLM Optimization Strategies

Previously, OpenAI released a lightweight library called Swarm. OpenAI has now revealed plans to release "Operator," an upcoming AI agent tool, which is set for release as early as January 2025, with plans to initially offer it as a research preview via the developer API. The tool is designed to execute tasks directly on users' computers.

Operator is expected to compete with Anthropic's "Computer Use" feature and Google's rumored consumer-focused agent, potentially offering general-purpose capabilities in web browsers. As of right now, details on Operator's unique advantages aren’t entirely clea, although it aims to simplify task execution.

OpenAI also released a post detailing how to optimize LLMs, since there’s issues due to varying requirements for accuracy, method selection, and production-readiness, requiring a structured approach tailored to specific use cases.

They recommended to begin with prompt engineering for simple tasks, then progress to RAG for dynamic context and fine-tuning for consistent behavior and task-specific accuracy.

Afterwards, they mention you should establish an evaluation set to diagnose failures and iteratively refine optimization methods, ensuring tools like RAG or fine-tuning are applied when prompt engineering alone falls short.

On the legal side of things, A New York judge dismissed RawStory's lawsuit against OpenAI, which might set a precedent for future cases involving AI training on copyrighted data. The dismissal was primarily based on technical grounds, though the judge's reasoning touched on broader issues related to AI and copyright.

The court found that authors aren’t plagiarized during AI training because the process involves a broad set of data and doesn’t result in verbatim copying.

Forge API's Reasoning Power and CoPilot Arena's Coding Leaderboard

Hermes 3 70B is able to perform well on various reasoning benchmarks. (Source)

Nous Research is launching two new projects: the Forge Reasoning API Beta and Nous Chat. Nous Chat is a simple chat platform featuring the Hermes 3 70B language model, which is available for free.

The Forge Reasoning API Beta is being released to a select group of users, focusing on testing the architecture of their reasoning system. It allows users to enhance any popular model with a code interpreter and advanced reasoning capabilities.

Moreover, the API supports multiple models (Hermes 3, Claude Sonnet 3.5, Gemini, GPT-4) and allows users to combine models for enhanced output diversity.

Here’s what’s involved in terms of reasoning layer architectures:

• MCTS (Monte Carlo Tree Search): Iteratively builds decision trees for planning problems through selection, expansion, simulation, and backpropagation. 

• CoC (Chain of Code): Connects reasoning steps to a code interpreter, improving code and math capabilities. 

• MoA (Mixture of Agents): Allows multiple models to collaborate on a query, providing more complete and diverse outputs.

Nous Research claims Hermes 70B augmented with Forge outperforms larger models from Google, OpenAI, and Anthropic in reasoning benchmarks. Specifically, they mentioned superior performance in the AIME evaluation, which focuses on competition-grade math questions.

Ever wondered what model would take the first place spot for coding? Imarena released a code completions leaderboard using data from the previous month to answer that question.

Nine popular models were evaluated, including open-source, code-specific, and commercial models. Top performers were DeepSeek V2.5 and Claude Sonnet 3.5, with Elo ratings of 1074 and 1053 respectively. The evaluation process included randomization of model pairings and positions, and standardized parameters for fair comparison.

Moreover, a free AI coding assistant called Copilot Arena was launched recently, providing paired responses from different state-of-the-art LLMs. It has been downloaded 2,500 times on the VSCode Marketplace, served over 100,000 completions, and accumulated over 10,000 code completion battles. The tool offers paired code completions and inline editing features.

A novel prompting technique was also developed to enable chat models to perform code completions, especially for "fill-in-the-middle" (FiM) tasks. The method involves generating code snippets and post-processing them, rather than forcing models to output in FiM format directly. This approach drastically reduced formatting errors across various models.

Tackling Climate, Protein and Math Challenges

Google continues to work on unique AI applications, previously introducing a whale bioacoustics model. Now, they’ve developed a new flood forecasting model with improved reliability and coverage. The model provides a 7-day lead time reliability comparable to current best available nowcasts. It expands coverage to 100 countries with verified data and up to 150 countries with data based on virtual gauges.

The improved model quality and new evaluation approach have expanded coverage. Google Flood Hub now reaches users in over 100 countries, up from 80. This expansion enables Google to provide critical flood information to 700 million people worldwide, up from 460 million previously. 

The model incorporates DeepMind's medium-range global weather forecasting model as input. Training data has been increased from 5,680 gauges to nearly 16,000 gauges. The LSTM-based architecture has been improved to better combine different weather products and increase robustness to missing data.

We saw an open-source implementation of AlphaFold 3 called LIGO a couple months back. Google DeepMind has now released the software code for AlphaFold 3, allowing non-commercial use. This comes six months after initially withholding the code, which drew criticism from scientists (as you’d expect).

AlphaFold 3 can model proteins interacting with other molecules, including potential drugs. The code is now downloadable, but model weights are only available to academics upon request. Several companies have developed AlphaFold 3-inspired models, including Baidu, ByteDance, and Chai Discovery. 

AI in math also saw advancements with FrontierMath - a new benchmark designed to test the limits of AI systems in mathematical reasoning. The benchmark aims to capture a snapshot of contemporary math and evaluate AI's progress towards innovative thinking needed for scientific research.

FrontierMath deals with the saturation issue that other benchmarks face. (Source)

A quick rundown of the new benchmark’s key features:

• All problems are new and unpublished to prevent data contamination

• Solutions are automatically verifiable, enabling efficient evaluation

• Problems are "guessproof" with a low chance of solving without proper reasoning

• Each problem demands hours of work from expert mathematicians

The benchmark has even been validated by Fields Medalists like Terence Tao (considered the best mathematician in the world), Timothy Gowers, and Richard Borcherds.

Qwen 2.5 Matches GPT-4 as X Expands Grok Access 

Qwen2.5 showed promising results on various coding benchmarks. (Source)

Qwen2.5-Coder-32B-Instruct caught us off-guard last week, since there’s claims of it being the current SOTA open-source code model, matching GPT-4o's coding capabilities. It excels in code generation, repair, and reasoning across multiple programming languages.

The model scored 73.7 on the Aider code repair benchmark, comparable to GPT-4o. It performs well across 40+ programming languages, scoring 65.9 on McEval and 75.2 on MdEval.

The series includes six model sizes: 0.5B, 1.5B, 3B, 7B, 14B, and 32B. Moreover, both Base and Instruct versions are available for each size. As you’d expect, performance scales positively with model size across various benchmarks. The series also outperforms other open-source models across all sizes on core datasets.

In other news, X began to test a free version of Grok. Previously, Grok was limited to premium, paying users of the platform.

Sounds great, but there’s a some restrictions that free users will need to keep in mind:

• 10 queries per two hours with the Grok-2 model

• 20 queries per two hours with the Grok-2 mini model

• 3 image analysis questions per day

xAI launched Grok-2 in August with image generation capabilities. The model recently gained the ability to understand images.These features were previously exclusive to Premium and Premium+ users, but free users can also try out these features now.

DeepL Voice Translates as BRIA RMBT2.0 Removes Backgrounds

DeepL, a German startup valued at $2 billion, has launched DeepL Voice, a real-time audio translation service. The service can "hear" 13 languages and provide translated captions in 33 languages supported by DeepL Translator. Currently, DeepL Voice outputs translations as text, not audio, focusing on live conversations and video conferencing.

Translations can appear as "mirrors" on smartphones for in-person meetings or as side-by-side transcriptions. For video conferencing, translations appear as subtitles, with Microsoft Teams being the only integrated platform so far. There's no API for the voice product yet, as DeepL is working directly with partners and customers.

In other news, a new background removal model was released called RMBG v2.0, which is developed by BRIA AI. It's designed for separating foreground from background across various image categories. The model is trained on a diverse dataset including stock images, e-commerce, gaming, and advertising content.

It aims to rival leading source-available models in accuracy, efficiency, and versatility. The model is particularly suitable for enterprise-scale content creation where content safety, legal compliance, and bias mitigation are crucial.

Advancements in AI Research

Some interesting papers came to light last week, ranging from the CDXFormer model for remote sensing change detection to the critical evaluation of domain-adaptive pretraining for medical applications. Meta FAIR also released a paper looking at the issue of localized image watermarking.

CDXFormer: A New Approach to Remote Sensing Change Detection

Researchers from Zhejiang University tackled the critical challenge of effectively identifying changes in remote sensing images across complex and varied environmental conditions. They recognized existing methods like Convolutional Neural Networks, Transformers, and Mamba-based models struggled to balance performance and computational efficiency when analyzing spatial-temporal contexts.

To address these limitations, they developed CDXFormer, a new approach that uses Extended Long Short-Term Memory (XLSTM) technology. Their method introduces a scale-specific feature enhancement layer with two key components: a Cross-Temporal Global Perceptron for semantic-accurate deep features and a Cross-Temporal Spatial Refiner for detail-rich shallow features. 

Additionally, they implemented a Cross-Scale Interactive Fusion module to progressively integrate spatial information and global semantics. It achieved SOTA performance across three benchmark datasets, improving F1 scores by 0.22%, 1.08%, and 7.46% compared to previous top-performing methods.

Crucially, the model maintained high efficiency, using only 16.19 million parameters and 3.92 G Flops—significantly lower than competing approaches. 

Medical Models vs. General-Domain LLMs: New Insights into AI's Effectiveness in Healthcare

Overview of evaluation approach. (Source)

A recent study from Carnegie Mellon University and Mistral AI looks at the effectiveness of domain-adaptive pretraining (DAPT) for medical applications of LLMs and VLMs. They aimed to address the prevailing assumption that DAPT consistently enhances performance on medical tasks, particularly in answering medical licensing exam questions.

To investigate this, they conducted a comprehensive evaluation comparing seven medical LLMs and two medical VLMs against their corresponding general-domain base models. 

They optimized prompts for each model independently and accounted for statistical uncertainty in their comparisons. The results revealed a surprising trend: most medical models didn’t consistently outperform their general-domain counterparts. In fact, medical LLMs only surpassed base models in 12.1% of cases, with ties in 49.8% and underperformance in 38.2%.

These findings suggest SOTA general-domain models may already possess strong medical knowledge and reasoning capabilities when prompted appropriately. 

MIT Study Provides Causal Evidence of AI's Impact on Scientific Discovery and Innovation

A study from MIT provides the first causal evidence of AI's impact on real-world R&D, showing significant boosts to scientific discovery and product innovation. The research exploits the randomized introduction of an AI tool for materials discovery to over 1,000 scientists in a large U.S. firm's R&D lab.

The study addresses key questions about AI's role in innovation, including its effects on the pace and direction of scientific breakthroughs, as well as its impact on scientists themselves. To investigate these issues, the researchers analyzed detailed data on each stage of R&D, from initial material discovery to patent filings and product prototypes.

Key findings revealed AI-assisted researchers discover 44% more materials, leading to a 39% increase in patent filings and a 17% rise in downstream product innovation. 

Interestingly, the technology had strikingly disparate effects across researchers. While top scientists nearly doubled their output, the bottom third saw little benefit. The study found AI automated 57% of "idea-generation" tasks, shifting researchers' focus to evaluating AI-suggested materials. Top scientists leveraged their domain knowledge to prioritize promising AI suggestions more effectively.

It was reported that 82% of scientists experienced reduced job satisfaction due to decreased creativity and skill underutilization, so there’s definitely some challenges that still need to be addressed in workforce adaptation to AI.

Meta FAIR Introduces WAM: Redefining Watermarking as a Segmentation Task

Meta FAIR’s paper addresses the challenge of localized image watermarking, which traditional methods struggle to handle effectively. They aimed to solve the problem of watermarking specific areas within an image, allowing for multiple distinct watermarks and improved robustness against image manipulations like splicing and inpainting.

To tackle this issue, the researchers introduce the Watermark Anything Model (WAM), which redefines watermarking as a segmentation task. WAM consists of an embedder that imperceptibly modifies the input image and an extractor that segments the received image into watermarked and non-watermarked areas while recovering hidden messages. 

The model employs a two-stage training process, first focusing on robustness at low resolution without perceptual constraints, then fine-tuning for imperceptibility and multiple watermark handling.

They used deep learning techniques, including a graph neural network architecture for the embedder and a vision transformer-based extractor. They incorporated a Just-Noticeable-Difference (JND) map to modulate watermark intensity and improve imperceptibility. 

The model demonstrates the ability to locate watermarked areas in spliced images and extract distinct 32-bit messages from multiple small regions, even when they occupy as little as 10% of the image surface. Notably, WAM achieves over 85% mIoU in detecting watermarked areas and over 95% bit accuracy when hiding five 32-bit messages in 10% areas of images, even after manipulations like horizontal flipping and contrast adjustment.

Conversations We Loved

Anthropic’s CEO had a 5-hour discussion with Lex Fridman last week and gave us some interesting insights into the scaling hypothesis to think about. Meanwhile, another conversation about the evolution of LLMs by Andrew Ng also caught our attention.

Evolution of LLMs and Agentic Workflows

Andrew Ng shared insights on the evolving landscape of LLMs and their increasing optimization for agentic workflows. Ng highlights a significant shift in LLM development, moving beyond consumer-facing question-answering to more complex, iterative processes that enable AI agents to perform sophisticated tasks.

Ng observes that while LLMs have been primarily tuned for direct human interaction, there's a growing trend towards optimizing them for agentic behaviors. This includes capabilities like tool use, function calling, and even computer operation, as demonstrated by Anthropic's recent release. He emphasizes the potential of these advancements to dramatically boost agentic performance in AI applications.

The conversation outlines a three-stage progression in agentic LLM development:

• Prompting existing LLMs for agentic behaviors

• Fine-tuning models for specific, high-value applications

• Major LLM providers integrating agentic capabilities directly into their models

Ng predicts that this trend will lead to significant performance gains in AI agents over the next few years, opening up new possibilities for complex, multi-step AI workflows. 

Dario Amodei Discusses Scaling Hypothesis and Responsible AI Development

In a conversation on the Lex Fridman Podcast, Dario Amodei, CEO of Anthropic, offered valuable insights into the state of AI and his company's approach to responsible AI development.

Amodei delved into the scaling hypothesis, which suggests that increasing the size and computational power of neural network models leads to significant capability growth across diverse tasks. He highlighted how larger models like GPT and CLIP have shown dramatic improvements when scaled up with more data and compute power. 

Amodei believes AI systems matching or exceeding human abilities across domains could be achievable as soon as 2026 or 2027, though he acknowledges uncertainties remain.

A key focus of the discussion was Anthropic's Responsible Scaling Plan (RSP), aimed at mitigating potential risks as AI becomes more powerful. This plan involves testing models for autonomous behavior and potential misuse, with escalating safety precautions as capabilities increase. Amodei emphasized the necessity of regulation in AI to address risks like malicious use and loss of human control.

Amodei shared insights into Anthropic's work on the Claude AI model as well, which is being developed with human values in mind through iterative testing and refinement. 

Frameworks We Love

Some frameworks that caught our attention in the last week include:

• FinRobot: Framework for equity research that integrates quantitative and qualitative analysis through a multi-agent Chain of Thought system

• XiYan-SQL: Natural language to SQL framework that improves query generation quality and accuracy

• VoiceLab: Open-source framework for testing and optimizing voice agents, providing tools for custom metrics, model migration, and prompt testing. 

If you want your framework to be featured here, reply to this email saying hi :)

Money Moving in AI

Writer saw a successful series C funding round for $200 million, while two key acquisitions took place last week: Anysphere acquired Supermaven and Redhat acquired NeuralMagic. Note that the sum for these acquisitions weren’t revealed.

Writer Raises $200 Million in Series C Funding Round

Writer, an enterprise generative AI platform, raised $200 million in a Series C round at a $1.9 billion valuation. The funding will support product development, including AI agents, customizable guardrails, and no-code tools, cementing its position in enterprise AI. Writer's Palmyra models, tailored for business needs, have attracted clients like Salesforce, Uber, and Intuit, highlighting its success amidst intense competition.

Anysphere Acquires Supermaven

Anysphere, maker of the AI-powered code editor Cursor, has acquired AI coding assistant Supermaven for an undisclosed sum. Supermaven's technology, including its low-latency AI model Babble, will enhance Cursor's upcoming Tab AI version for context-aware, intelligent coding, especially for long sequences. 

The merger aims to combine advanced model capabilities with a seamless editor UI, accelerating product development and maintaining Supermaven’s plugins. 

Red Hat Acquires Neural Magic

Red Hat has acquired Neural Magic (also for an undisclosed sum), a startup focused on optimizing AI models to run efficiently on standard processors and GPUs, enhancing hybrid cloud AI performance. Neural Magic, founded in 2018, offers tools like vLLM for model serving, which Red Hat will integrate into its platforms like OpenShift AI and Red Hat Enterprise Linux AI. 

This acquisition aligns with Red Hat's goal to expand its AI capabilities in flexible, open-source environments.

Key Takeaways

• OpenAI released Swarm, a lightweight library for building multi-agent systems that allows dynamic switching between specialized agents during conversations.

• Aria, the first open-source multimodal native MoE model, was released, showing state-of-the-art performance on various multimodal and language tasks.

• BioNTech unveiled several AI initiatives, including the Kyber supercomputer and Bayesian Flow Network models for protein sequence generation.

• OpenAI's MLE-bench, evaluating AI agents on machine learning engineering tasks, found that the best setup achieved bronze medal level in 16.9% of Kaggle competitions, improving to 34.1% with multiple attempts.

• A study introducing Compositional GSM showed significant disparities in LLM performance between standard and more complex math problems, highlighting gaps in reasoning capabilities.

Got forwarded this newsletter? Subscribe below👇

The Latest AI News

The model releases seemed to have slowed down last week, but we did get the first open-source, multimodal native MoE mode, along with a new library from OpenAI.

Meanwhile, AI in biotech saw some notable progress with AI being applied to calculating carbon footprints as well as a bunch of updates unveiled at BioNTech’s inaugural AI day.

OpenAI's New Tool for Flexible and Dynamic AI Agent Interactions

Example of how Swarm works. (Source)

OpenAI released Swarm - a lightweight library for building multi-agent systems. It’s similar in concept to existing frameworks like CrewAI and LangChain that also help with the creation of multi-agent systems. Unexpected, but certainly welcomed. 

Swarm provides a stateless abstraction to manage interactions between multiple AI agents. It also allows for dynamic switching between specialized agents during conversations. Worth noting that it doesn’t rely on OpenAI's Assistants API, so it offers more flexibility and control.

It lets devs create distinct agents, each with specific roles, instructions, and functions. These agents can interact dynamically based on pre-defined handoff logic, allowing for seamless task switching as conversations or workflows progress.

Since the framework doesn’t maintain internal state between function calls, it lets agents pass control to other agents in real-time based on criteria or conversation flow. The handoff is simple—just return the next agent to engage. 

Notably, Swarm doesn’t use OpenAI’s function calling feature. Instead, it opts for a more flexible approach that allows for easier integration with various AI models and custom implementations.

Swarm uses context variables to maintain and update the state throughout multi-agent interactions. 

Aria: The World's First Open-Source Multimodal Native MoE Model Debuts

Aria showed SOTA results on benchmarks like MathVista and DocVQA. (Source)

We saw the release of the first open-source, multimodal native MoE model called Aria by Rhymes AI.

It's pre-trained from scratch on a mixture of multimodal and language data. It’s definitely no slouch as it showed SOTA performance on a wide range of multimodal and language tasks like MMMU and LongVideoBench. It’s adept at following instructions across both multimodal and language inputs, excelling in benchmarks like MIA-Bench and MT-Bench.

Aria processes text, images, video, and code simultaneously - all without needing separate setups for each type. In terms of multimodal capabilities, it has a long context window of 64K tokens. Aria can caption a 256-frame video in just 10 seconds. 

It consists of a vision encoder and a MoE decoder, with the vision encoder operating in three resolution modes: medium, high, and ultra-high. The MoE decoder has 66 experts in each layer, with 2 shared experts and 6 activated experts per token.

World Labs' Google Cloud Partnership, TensorWave's AMD Challenge, and Intel's AI-Enabled Processors

Fei-Fei Li’s World Labs (which came out of stealth last month) has selected Google Cloud as its primary compute provider to train its "spatially intelligent" AI models. They’re using a significant portion of its $230 million funding round for GPU server licensing.

World Labs has a big focus developing multimodal AI models capable of processing, generating, and interacting with video and geospatial data, which are all pretty computationally demanding. 

World Labs’ partnership with Google Cloud is non-exclusive, so the startup can potentially explore other cloud providers in the future. But for now, Google Cloud currently hosts the majority of World Labs’ workloads and they aim to retain this business long-term.

Nvidia has seen a lot of competition step up to the plate in the chip market in recent months, with last week being no exception. TensorWave is going against the grain by launching a cloud platform that only offers access to hardware from Nvidia rival AMD for AI workloads. Their main goal is to democratize AI by offering more affordable compute access.

TensorWave uses AMD Instinct MI300X GPUs for AI workloads, claiming its MI300X instances outperform Nvidia's H100 in running (but not training) AI models, particularly text-generating models like Meta's Llama 2.

TensorWave rents GPU capacity by the hour with a minimum six-month contract.
Pricing ranges from approximately $1 to $10 per hour, depending on workload requirements and GPU configurations. Interestingly enough, the company aims to be more cost-effective than competitors due to the lower cost of AMD GPUs compared to Nvidia's H100.

In terms of growth, TensorWave is already generating $3 million in annual recurring revenue. The company expects to reach $25 million in revenue by the end of the year. TensorWave plans to scale up to 20,000 MI300X GPUs. The company intends to bring AMD's next-gen MI325X GPUs online as early as November/December 2024.

We also saw a new release from Intel called the Intel Core Ultra 200S series, which features built-in Neural Processing Units (NPUs). That means it marks Intel's first desktop processors with integrated AI capabilities. 

From Packages to Products: Amazon's AI Innovations in Delivery and Shopping

Amazon is introducing vision-based technology to its electric vehicle fleet. It’s designed to help drivers prioritize packages by highlighting them with green circles (for delivery) or red lights (incorrect packages).

The VAPR system aims to save drivers from having to stop and manually search for packages at each stop, reducing the time spent per stop from 2–5 minutes to under a minute. 

It also includes an audio cue to confirm if the driver has selected the correct package, which gets rid of the need for handheld devices that drivers currently use for package scanning and tracking.

Turns out the VAPR system has been in development since early 2020, with Amazon considering unique delivery challenges like lighting and space constraints inside vans. There are plans to deploy the VAPR system in 1,000 of its electric Rivian vans by early 2025, after testing the technology in select markets like Boston. It’s important to note that Amazon is Rivian’s largest shareholder with a stake of 16.6%.

Amazon also released AI Shopping Guides to help users find products based on specific features, offering tailored suggestions and product information for over 100 product types, including TVs, headphones, and skincare. 

It’s a more visual way to filter products, replacing traditional tick-box menus with interactive options for factors like brand, use case (e.g., sport or gaming), and connectivity type. Each guide includes educational content and customer insights to help users better understand product features and make more informed purchasing decisions.

The AI guides appear automatically during searches when relevant, or users can directly explore them through Amazon’s mobile website and apps for iOS and Android.

AI for Refining Carbon Accounting and Updates from BioNTech’s Inaugural AI Day

AI saw some climate tech applications last week, with Forward Earth focusing on using AI to automate the calculation of complex CO2 footprints. The co-founders were former executives at carbon accounting startup Planetly, who launched Forward Earth since they weren’t too happy with Planetly's acquisition and shutdown by OneTrust.

BioNTech recently had its inaugural AI day where they unveiled a bunch of key updates. 

Here’s the main ones you need to know:

• AI Scaling Across Immunotherapy Pipeline: BioNtech detailed its strategy to scale AI capabilities throughout its immunotherapy pipeline, using AI to drive innovation in areas like DNA/RNA sequencing, proteomics, protein design, and immunohistochemistry.

• Launch of Kyber Supercomputer: InstaDeep (BioNTech’s AI subsidiary) unveiled Kyber - a near exascale supercomputer aimed at enabling high-performance computing for large-scale AI and biotechnology research.

• Bayesian Flow Network (BFN): BioNTech presented BFN generative models for protein sequence generation, showing the potential for advancements in personalized vaccine development and targeted therapies.

• DeepChain™ Platform and External Partnerships: The DeepChain™ multiomics design platform was introduced for external partnerships after success in BioNTech’s internal projects, including the mRNA-encoded antibody RiboMab™ platform.

Adobe Launches Free Web App for Content Credentials, Addressing Past AI Training Controversies

Adobe is launching a free web app to improve the process of applying Content Credentials to images, videos, and audio files.

It’s an interesting turn of events since we saw Adobe use Midjourney-generated images to train their Firefly model in the past, even though they said the model was a “commercially safe” alternative.

It might be Adobe’s response to this situation where they lost a lot of trust when it comes to ethical AI practices. They’re introducing tools that let creators protect their work and opt out of AI training datasets, which shows they’re adapting their practices and policies for a more ethical approach.

Advancements in AI Research

One paper that really stood out last week was about exploring intelligence emergence in rule-based systems. OpenAI also released a new benchmark for evaluating AI in scientific discovery tasks. Researchers provided a fresh perspective on assessing LLM capabilities as well.

How Rule Complexity Shapes AI Behavior

Framework overview. (Source)

Researchers introduced a new approach to understanding the emergence of intelligent behavior in artificial systems by investigating how the complexity of rule-based systems influences the capabilities of models trained to predict these rules.

Previously we saw the release of LifeGPT which also progressed work with cellular automata. This paper used elementary cellular automata (ECA) as a framework to generate behaviors ranging from simple to highly complex. 

They also trained separate GPT-2 language models on datasets generated by individual ECAs and evaluated the models' "intelligence" through performance on downstream logical reasoning tasks.

A positive correlation was found between the complexity of ECA rules and the downstream performance of models trained on them. Models trained on Class IV ECA rules, which showed structured yet complex behaviors, performed optimally. 

Interestingly, it showed that models can learn complex solutions even when trained on simple rules, which is likely due to overparameterization. They presented a hypothesis that by learning to incorporate past states, models develop generalizable logic that can be reused across tasks.

From Bronze to Gold: OpenAI's MLE-bench Challenges AI Agents in Real-World ML Tasks

In addition to Swarm, OpenAI also introduced MLE-bench last week. It’s a new benchmark for assessing the capabilities of AI agents in machine learning engineering tasks. 

This benchmark aims to provide a rigorous measure of progress in autonomous ML engineering agents, addressing the growing interest in using AI to automate scientific workflows.

Key aspects of MLE-bench include:

1. A diverse set of 75 Kaggle competitions across various domains, including natural language processing, computer vision, and signal processing.

2. Careful curation to ensure tasks are challenging and representative of contemporary ML engineering work.

3. The ability to compare AI agent performance directly with human-level performance using Kaggle leaderboards.

The researchers evaluated several frontier language models on MLE-bench using open-source agent scaffolds. They found the best-performing setup to be OpenAI's o1-preview with AIDE scaffolding, which achieved at least a bronze medal level in 16.9% of competitions.

Performance significantly improved when agents were given multiple attempts per competition, with o1-preview's score doubling from 16.9% to 34.1% when allowed 8 attempts. Moreover, Agents performed well on competitions solvable with well-known approaches but struggled with debugging and recovering from missteps.

ScienceAgentBench: Putting AI to the Test in Scientific Discovery

ScienceAgentBench is a new benchmark for evaluating language agents in data-driven scientific discovery tasks. It addresses the growing interest in using AI to automate scientific workflows, while also highlighting the need for more rigorous assessment of these systems.

102 diverse tasks were extracted from 44 peer-reviewed publications across four scientific disciplines: Bioinformatics, Computational Chemistry, Geographical Information Science, and Psychology & Cognitive Neuroscience.

There was a big focus on code generation, requiring agents to produce complete Python programs for data analysis and visualization tasks. They also used careful quality control measures, including expert validation and strategies to mitigate data contamination concerns.

They evaluated five open-weight and proprietary LLMs using three frameworks: direct prompting, OpenHands CodeAct, and self-debug. Surprisingly, they found simpler approaches like self-debug usually outperformed more complex frameworks in both performance and cost-efficiency.

Results show that even the best-performing agent, Claude-3.5-Sonnet using self-debug, could only solve 34.3% of tasks with expert-provided knowledge. It’s definitely an eye-opener about the challenges that AI agents are having in automating complex scientific workflows.

New Benchmark Reveals LLM Reasoning Disparities

Graph comparing reasoning performance on GSM8K and Compositional GSM accuracy. (Source)

Researchers from Mila, Microsoft Research, and Google DeepMind introduced a new approach to evaluating the reasoning capabilities of LLMs in grade-school math problems. 

They introduced Compositional GSM, a two-hop version of the GSM8K benchmark that challenges LLMs to solve chained math problems.

They also evaluated various LLMs, including open-source and proprietary models like Gemini, Gemma2, LLAMA3, GPT, Phi, Qwen2.5, and Mistral families.

There were significant disparities between LLMs' performance on standard GSM8K problems and the more complex Compositional GSM tasks. Smaller, more cost-efficient, and math-specialized models showed larger reasoning gaps.

For instance, GPT-4o mini, which nearly matches GPT-4o on standard benchmarks, showed a 2-12x worse reasoning gap on Compositional GSM.

It showed instruction-tuning effects vary across LLM sizes, with smaller models showing more improvement on standard GSM8K but less on Compositional GSM. Extensive math specialization didn’t necessarily improve performance on these compositional tasks.

Conversations We Loved

A post about a new architecture which might be able to improve on o1’s ability to scale inference-time compute gained a lot of attention - and for good reason.

Another discussion about LongCite, a new method to improve the trustworthiness of AI outputs, was also one to look at.

Pause Tokens and Parallel Reasoning: Entropix's New Approach to AI Cognition

A discussion about Entropix came up last week, which is an innovative architecture anonymously released by an AI researcher. 

It highlighted a new approach to replicating and potentially improving upon OpenAI's latest o1 model's ability to scale inference-time compute - essentially improving its capacity to 'think' before responding.

What stood out was architecture's use of uncertainty measurement (formally defined as entropy and varentropy) to improve reasoning. 

The model inserts pause tokens like "...wait" when uncertain about the next best tokens or thoughts, prompting it to reflect and produce additional chains of thought. This approach allows for dynamic scaling of inference-time compute for more profound and strategic thinking.

Makes us wonder how this method could potentially unlock more powerful reasoning capabilities from smaller models like Llama 3.1-1B, which can run locally on a laptop. 

Perhaps advancements in the underlying attention mechanism, rather than relying solely on prompt engineering or methods like Monte Carlo Tree Search, might be key to unlocking the true potential of inference-time compute

Boosting LLM Trustworthiness with Fine-Grained Citations

Raschka brought an interesting paper called LongCite to the spotlight, which probably slipped under the radar. It mentions a new approach to boost the trustworthiness of LLMs in long-context question answering tasks.

What makes this paper important is its focus making AI-generated content more trustworthy. Misinformation is common in today’s world, so equipping LLMs with the ability to generate precise, sentence-level citations could definitely raise human confidence in AI outputs. Not only does that help address concerns about hallucinations, but it also helps us verify information.

The method used also stands out, since they leveraged existing LLMs to create a dataset of long-context Q&A instances. It’s an example of step-wise refinement that shows how AI can be adapted to deal with pressing issues.

Maybe we’ll see a shift toward models that prioritize citation and accountability as core functionalities in the future.

Frameworks We Love

Some frameworks that caught our attention in the last week include:

• IterComp: Combines the strengths of multiple diffusion models to improve compositional text-to-image generation. 

• AvatarGo: Generates animatable 4D human-object interaction (HOI) scenes directly from textual inputs.

• SimNPO: Framework for LLMs that aims to remove unwanted data influences and associated model capabilities 

If you want your framework to be featured here, reply to this email saying hi :)

Money Moving in AI

AI applications in mineral discovery seem to be making good progress as KoBold Metals secured $491 million after a massive discovery. Meanwhile, Basecamp and Braintrust secured $60 million and $36 million respectively.

KoBold Metals Secures $491 Million

KoBold Metals, an AI-powered mineral discovery startup, is close to raising over half a billion dollars, having already secured $491 million of a targeted $527 million round. 

This funding comes on the heels of KoBold's discovery of what might be one of the largest high-grade copper deposits in history, showing the potential of AI in mineral exploration. 

Basecamp Raises $60 Million in Series B Funding Round

A London-based startup building an AI agent for biology and biodiversity insights called Basecamp has raised $60 million in a Series B funding round led by Singular. 

The company aims to create an AI that can not only answer questions about biology but also produce new insights beyond human capabilities, with its BaseFold model already claiming to outperform DeepMind's AlphaFold 2 in certain protein structure predictions

Rad AI Secures $50 Million in Series B Funding Round

A startup focusing on generative AI for healthcare called Rad AI secured $50 million in series B funding led by Khosla Ventures, which brings their total capital raised to over $80 million. The CEO attributed a big part of their success to strong business metrics, including tripling year-over-year revenue and adoption by more than a third of all US health systems.

Braintrust Raises $36 Million in Series A Funding Round

Braintrust focuses on empowering teams to build robust LLM-enabled applications. They announced a $36 million funding round, bringing their total funding to $45 million. In particular, they’re working toward building features like being able to share code in dev environments.

Key Takeaways

• John Hopfield and Geoffrey Hinton, the Godfather of AI, win Nobel Prize… in Physics.

• OpenAI announced new API features, including a Realtime API for voice-to-voice conversations and Vision fine-tuning for improved image tasks like object detection.

• Nvidia released NVLM-D-72B, a large multimodal model rivaling GPT-4o and Claude, with its model weights publicly available.

• Liquid AI released Liquid Foundation Models (LFMs), delivering SOTA performance with a smaller memory footprint.

• Meta's Movie Gen lets users create videos and edit personal images using simple text inputs, producing HD media.

• FlashMask optimizes attention mechanisms by introducing a column-wise sparse representation for attention masks, improving memory efficiency for LLMs handling sequences up to 128K tokens.

• PlasmidGPT is a transformer-based model for designing plasmid DNA which improves annotation accuracy by 81%.

Got forwarded this newsletter? Subscribe below👇

The Latest AI News

Folks, busy week as always. OpenAI took the spotlight in terms of AI news last week yet again. It wasn’t all positive though, as there was yet another OpenAI exodus with four key members leaving (and two of them joining competitor companies already). Although there were some neat announcements for new API features at OpenAI DevDay.

Meanwhile, we saw new model releases from the likes of Nvidia, Liquid AI, and Resolve AI, as well as developments in video generation AI.

OpenAI Unveils Realtime API, Vision Fine-Tuning, and More at DevDay 2024

OpenAI Devday 2024 took place last week and there were some huge feature announcements. Are they worth a loss of $44B from 2023 to 2028? We'll see.

Here’s a quick rundown of what OpenAI revealed in terms of API features:

1. Realtime API: For devs looking to quickly add speech-to-speech into apps, they’ll be glad to hear about this. Voice assistant development is a whole lot easier since Realtime API combines transcription, text reasoning, and text-to-speech into a single API call. Currently available to paid developers with text and audio input/output tokens priced at $0.06/minute for input and $0.24/minute for output.

2. Vision: Further multimodal developments were seen in the vision update, which means that GPT-4o now supports fine-tuning with images. The plans are for free vision fine-tuning tokens to be available until October 31 2024, after which training and inference will be priced based on token usage.

3. Prompt caching: Devs can now reduce costs and latency by reusing previously seen input tokens through the new Prompt Caching feature in the GPT-4o models, offering discounts and faster processing. Prompt Caching automatically applies on prompts over 1,024 tokens, and cached prompts receive a 50% discount on input tokens. 

4. Model distillation: This feature helps improve performance at lower costs as it allows for outputs from larger models like GPT-4o to be used to fine-tune smaller, cost-efficient models such as GPT-4o mini. Model Distillation is available for all devs with free training tokens until October 31 2024, and standard pricing thereafter.

5. Canvas: A new interface for writing and coding projects with ChatGPT, allowing collaboration beyond basic conversation. It enables direct editing and feedback - similar to a code reviewer or copy editor. It's currently in early beta with plans for rapid development based on user feedback.

AI Video Generation Heats Up: Movie Gen, VidGen-2, and OpenFLUX Make Waves

A bunch of model releases were seen in the AI video generation space last week.

Example of a video produced by Meta’s Movie Gen. (Source)

Meta introduced Movie Gen, a breakthrough generative AI research project for media.

It encompasses multiple modalities including image, video, and audio generation and editing.

The system allows users to produce custom videos and sounds, edit existing videos, and transform personal images into unique videos using simple text inputs. Movie Gen uses a 30B parameter transformer model for video generation. 

A quick overview of Movie Gen’s main features:

1. Video Generation: Using a 30B parameter transformer model, Movie Gen can create high-quality, high-definition videos up to 16 seconds long at 16 fps from text prompts. 

2. Precise Video editing: Movie Gen's editing capabilities allow for both localized and global changes to existing videos. Users can add, remove, or replace elements, or modify entire backgrounds and styles using text prompts. Unlike traditional editing tools, Movie Gen preserves original content while targeting only relevant pixels.

3. Audio Generation: A 13B parameter audio model can generate high-fidelity audio up to 45 seconds long, including ambient sound, sound effects, and background music, all synchronized with video content.

Meanwhile, HelmAI’s VidGen-2 offers 2X higher resolution than its predecessor, generating video sequences at 696 x 696 resolution. It improves realism at 30 fps, with frame rates ranging from 5 to 30 fps. It can generate videos without an input prompt or with a single image or input video as the prompt.

The model generates driving scene videos across multiple geographies, camera types, and vehicle perspectives. It produces highly realistic appearances and temporally consistent object motion. VidGen-2 learns and reproduces human-like driving behaviors, simulating the motions of the ego-vehicle and surrounding agents in accordance with traffic rules.

Note that VidGen-2 was trained on thousands of hours of diverse driving footage using NVIDIA H100 Tensor Core GPUs. 

Midjourney’s competitor FLUX by Black Forest Labs also saw a new release called OpenFLUX.1, which is a fine-tuned version of the FLUX.1-schnell model. The main aim of this model was to train out the distillation to leave behind an open-source model that can be fine-tuned.

Nvidia Unveils NVLM 1.0 and Powers Brave's Local LLMs

Nvidia’s new model can respond to prompts containing text and images. (Source)

Even though we know Nvidia mainly for providing AI chips, they unexpectedly announced NVLM 1.0 - a family of large multimodal language models.

The flagship model, NVLM-D-72B, with 72 billion parameters, is reported to perform on par with or better than leading proprietary models like GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro in various tasks. Unlike its competitors, Nvidia is making NVLM 1.0's model weights and training code publicly available. 

NVLM 1.0 demonstrates strong performance in vision-language tasks, effectively processing and reasoning about both text and images. The model shows versatility in tasks requiring OCR, reasoning, localization, common sense, world knowledge, and coding abilities.

Additionally, after multimodal training, NVLM-D-72B shows improved accuracy on text-only tasks compared to its LLM backbone, indicating enhanced overall language understanding.

Brave, a privacy-focused web browser, has launched Leo AI, a smart AI assistant. Leo AI enhances user experience by summarizing articles and videos, surfacing insights from documents, and answering questions.

Ollama is an open-source project that sits on top of llama.cpp and simplifies AI model integration for applications. It handles tasks like downloading and configuring specific AI models, making it easier for applications to access local AI capabilities. NVIDIA optimizes tools like Ollama for their hardware to deliver faster, more responsive AI experiences on RTX GPUs.

OpenAI Exodus: Episode… (which one are we one now?)

This isn’t the first time we’ve seen various key members leave OpenAI in such a short span of time. 

Durk Kingma isn’t as well-known as some of the other figures, but he's a co-founder of OpenAI who announced his move to Anthropic. 

Kingma's hiring is part of a broader pattern of high-profile recruitments by Anthropic. The company has recently brought on board other notable figures from OpenAI, including Jan Leike (OpenAI's former safety lead) and John Schulman (another OpenAI co-founder).

Barret Zoph also decided to leave OpenAI, who was around for the entire ChatGPT boom in 2022 since was a part of the post-training team. Those weren’t the only departures, as Mira Murati left OpenAI after 6 and a half years. Altman made a statement about Mira leaving, talking about how taxing it is to be a leader at the company.

Lastly, Sora co-lead Tim Brooks released a short statement mentioning how he’ll be joining DeepMind to work on video generators and world simulators after spending 2 years at OpenAI working on Sora. 

Resolve AI Launches Efficient AI Model for DevOps Operations

Resolve AI introduced the world's first AI production engineer, designed to autonomously troubleshoot and resolve production issues. The company aims to empower engineers to focus on innovation rather than routine maintenance tasks. 

The goal is to significantly reduce mean time to resolution (MTTR) and accelerate engineering teams' productivity.

The company built an agentic platform that integrates with tools like AWS, Kubernetes, observability stacks, GitHub, and Slack. It constructs a comprehensive knowledge graph of a company's production environment.

The AI agent leverages this knowledge graph to troubleshoot incidents, analyze source code changes, detect anomalies, query logs, and suggest remediation actions. Resolve acts as an always-on partner, handling operational tasks like answering natural language queries and translating them into observability actions.

Advancements in AI Research

Last time, we saw a completely unexpected release in terms of biotech AI with Google’s whale bioacoustics model. We’re seeing more biotech developments with PlasmidGPT, which is used to annotate plasmid DNA sequences (as the name suggests).

Other developments include a new framework for multi-agent exploration and a new extension to the FlashAttention algorithm, as well as new generative AI models that take efficiency to another level.

Liquid Foundation Models: A New Approach to Efficient and Powerful AI

LFMs performance/size trade-offs are better than other models. (Source)

Researchers from Liquid AI have introduced Liquid Foundation Models (LFMs), a new generation of generative AI models that aim to achieve SOTA performance while maintaining a smaller memory footprint and more efficient inference.

LFMs address several challenges in current AI model development, including the trade-off between model size and performance, memory efficiency, and the ability to handle long-context tasks. The researchers proposed a new architecture that diverges from the traditional transformer-based models.

Impressive results were reported across the board:

• LFM-1B achieves the highest scores across various benchmarks in the 1B parameter category.

• LFM-3B outperforms previous generation 7B and 13B models on multiple benchmarks.

• LFM-40B offers performance comparable to larger models while using only 12B activated parameters.

They also show strong performance in multilingual capabilities and can effectively utilize their full 32k token context length. This efficiency enables long-context tasks on edge devices for the first time, so we might see new applications in areas like document analysis, context-aware chatbots, and improved RAG performance.

PlasmidGPT: AI-Powered Plasmid Design and Annotation

PlasmidGPT is a generative framework for designing and annotating plasmid DNA sequences. It addresses the challenges in automating plasmid design and leveraging the growing collection of engineered plasmid sequences.

Some of the main innovations include:

1. A decoder-only transformer model pretrained on 153,000 engineered plasmid sequences from Addgene.

2. The ability to generate de novo plasmid sequences that share characteristics with engineered plasmids but have low sequence identity to the training data.

3. Conditional generation capabilities, allowing users to specify starting sequences or fine-tune the model for specific vector types.

4. Effective prediction of various sequence-related attributes for both engineered and natural plasmids.

They showed that PlasmidGPT can generate plasmids with genetic part distributions similar to those in the training sequences. The model also outperforms previous approaches in predicting attributes like lab of origin, with a top-1 accuracy of 81% and top-10 accuracy of 92%.

Optimizing Attention for Long-Context LLMs with FlashMask

FlashMask is an extension to the FlashAttention algorithm that significantly improves the efficiency and flexibility of attention mechanisms in LLMs.

It addresses the growing challenge of handling complex masking requirements in various LLM training and inference scenarios. FlashMask introduces a column-wise sparse representation of attention masks, allowing for efficient handling of a wide range of mask types without compromising computational accuracy.

They introduced optimized kernel implementations that leverage sparsity in the attention mask to skip unnecessary computations. Moreover, extensive evaluation across different attention mask types and models were seen, showing significant throughput improvements in fine-tuning and alignment training of LLMs.

They reported end-to-end speedups ranging from 1.65x to 3.22x compared to existing FlashAttention dense methods. Additionally, FlashMask outperforms the latest counterpart, FlexAttention, by 12.1% to 60.7% in terms of kernel TFLOPs/s.

LLM-Guided Efficient Multi-Agent Exploration Using LEMAE

Map of the task “Pass.” (Source)

LLM for Efficient Multi-Agent Exploration (LEMAE) leverages LLMs to enable efficient multi-agent exploration in reinforcement learning. This work addresses the longstanding challenge of efficient exploration in complex multi-agent environments with expansive state-action spaces.

They used LLMs to ground linguistic knowledge into symbolic key states that are critical for task fulfillment. Additionally, a Subspace-based Hindsight Intrinsic Reward (SHIR) mechanism to guide agents toward key states by increasing reward density and a Key State Memory Tree (KSMT) to track transitions between key states and organize exploration were used.

They showed LEMAE significantly outperforms existing SOTA approaches on challenging benchmarks like StarCraft Multi-Agent Challenge (SMAC) and Multiple-Particle Environment (MPE). In certain scenarios, LEMAE achieves a 10x acceleration in exploration efficiency.

Conversations We Loved

A discussion about a new approach to information retrieval from complex documents took the spotlight, since it has benefits like having a much faster indexing process than traditional methods and it’s a lot more accurate when it comes to documents with a lot of visuals in them.

The other discussion that caught our attention was about a new paper by Meta finally giving us some insight into the post-training process of the Llama models.

How Meta's Mixture of Judges Perfects LLM Post-Training

Carr brought up Meta’s new paper. (Source)

If you’ve been wondering about how Meta has been going about the post-training process for the Llama models, you’re in luck as they recently released a paper on "Constrained Generative Policy Optimization" for post-training LLMs.

In particular, Carr put the spotlight on Meta's innovative approach to addressing the challenges of aligning LLMs on multiple tasks simultaneously. The introduction of Mixture of Judge models to achieve a balanced blend of RLHF improvements stood out in particular.

This approach tackles common issues in multi-task alignment like reward hacking, multi-objective conflicts, and contradictory goals. The judges include specialized models for false refusal, precise instruction following, regex math/code reasoning, factuality, and safety.

It emphasized the simplicity and effectiveness of this method, which improves performance across various benchmarks including MATH, Human Eval, ARC, and AlpacaEval. 

The Promise of ColPali in Information Retrieval

Leonie’s post introducing ColPali. (Source)

An interesting thread discussing ColPali popped up last week. ColPali is a new approach to information retrieval from complex document types like PDFs. 

What caught our eye was the explanation of how ColPali combines two key technologies: the contextualized late interaction mechanism from ColBERT and the Vision Language Model capabilities of PaliGemma. 

This innovative approach replaces the traditional multi-step PDF parsing process with a simpler method using "screenshots" of PDF pages, which could change how we handle complex documents in information retrieval tasks. And on that note…

An interesting perspective on ColPali. (Source)

Frameworks We Love

Some frameworks that caught our attention in the last week include:

• FakeShield: Multimodal framework designed to address challenges in image forgery detection and localization (IFDL) 

• MA-RLHF: Incorporates macro actions (sequences of tokens or higher-level language constructs) into the learning process for large language models.

• ControlAR: Integrates spatial controls into autoregressive image generation models

If you want your framework to be featured here, reply to this email saying hi :)

Money Moving in AI

There were successful funding rounds for companies developing varied AI applications, including ones we don’t hear about often like liquid cooling solutions for data centers. Poolside, Submer, and Numa all saw successful funding rounds, with Poolside being the biggest winner last week.

Poolside Secures $500 Million

Poolside, a startup developing AI-powered coding software, has secured a massive $500 million investment led by Bain Capital Ventures, valuing the company at $3 billion. This significant funding round, which includes participation from DST Global, StepStone Group, Citi Ventures, and HSBC Ventures.

Submer Raises $55.5 Million in Series C Funding Round

Submer, a Barcelona-based startup specializing in liquid cooling solutions for data centers, has raised $55.5 million in a Series C round at a $500 million valuation. The company's technology, which involves submerging entire server racks in biodegradable, non-conducting coolant. This helps address the growing challenge of heat management in AI-driven data centers.

Numa Secures $32 Million in Series B Funding Round

Numa, an AI startup specializing in customer service automation for auto dealerships, has secured a $32 million Series B funding round, bringing its total raised to $48 million. The company, which pivoted from a general conversational AI product to focus specifically on the automotive industry, claims to be nearing cash-flow break-even with 600 customers across the U.S. and Canada.

Cerebras Files for IPO

Cerebras Systems filed for an initial public offering on September 30, 2024. The company plans to trade on the Nasdaq under the ticker symbol "CBRS". Cerebras positions itself as a competitor to Nvidia in the AI chip market, claiming its WSE-3 chip has more cores and memory than Nvidia's popular H100.

For the first six months of 2024, Cerebras reported a net loss of $66.6 million on $136.4 million in sales. This shows significant growth compared to the same period in 2023, where they had a net loss of $77.8 million on $8.7 million in sales. For the full year 2023, the company reported a net loss of $127.2 million on revenue of $78.7 million.

Key takeaways

• Meta introduced Llama 3.2 with vision-capable LLMs (11B and 90B) for document understanding, and lightweight text models (1B, 3B) optimized for mobile devices, supporting on-device processing with improved privacy.

• OpenAI Japan announced "GPT Next," projected to be 100x more powerful than GPT-4, with a planned release in 2024.

• DeepMind introduced AlphaProteo, an AI system for designing novel protein binders with 3-300x better binding affinities than existing methods.

• Google’s NotebookLM is a personalized AI research assistant that helps you chat across all your docs, and even generate audio overviews from uploaded content, creating podcasts and summaries with conversational AI voices.

• NVIDIA researchers introduced OP-RAG, outperforming long-context LLMs on the ∞Bench dataset using just 16K retrieved tokens.

The Latest AI News

Just when you thought OpenAI had enough on their plate with Project Strawberry, they dropped some major news in Japan that they’re planning to release another model called GPT Next later in the year.  Other major releases included Meta’s Llama 3.2 and Google’s NotebookLM.

There were a bunch of new releases last week too, including LLMs, AI agents, and VLMs. Progress in biotech was also made by DeepMind’s latest model called AlphaProteo.

Meta’s Llama 3.2 and AI Updates

Llama 3.2 performs well on instruction-tuned benchmarks. (Source)

Llama 3.1 recently reached 350+ million lifetime downloads. With this great achievement, Meta shipped another exciting update. Meta's Llama 3.2 includes vision-capable LLMs (11B and 90B) for tasks like document understanding and visual reasoning, as well as lightweight text-only models (1B and 3B) optimized for edge and mobile devices with 128K token support.

Llama 3.2's smaller models are optimized for devices using Qualcomm and MediaTek hardware. As a result, we’ll see faster, private processing for summarization and instruction-following without needing cloud access.

The 11B and 90B models introduce a new architecture integrating pre-trained image encoders with cross-attention layers for high-performance image-text reasoning, making them drop-in replacements for text models.

An important update was also the Llama Stack, a set of APIs for Agents, data generation, inference, guardrails, evals, and more. Aside from that, we saw some updates for Meta’s AI products, with new voice, text generation, and photo understanding capabilities within Messenger, Instagram, and Meta for Business.

Gemini's Upgrade, NotebookLM as an AI Research Assistant, and Arcade's Product Platform 

Google announced new versions of Gemini called Gemini-1.5-Pro-002 and Gemini-1.5-Flash-002 models. Standout benefits include 50% price drop, 2x faster outputs, and 3x lower latency.

These models show a 20% improvement in math-related tasks and substantial gains in visual understanding and code generation, so they’re ideal for tasks like summarizing large documents and understanding long videos.

Moreover, Google reduced the output length by 5-20%, speeding up responses without compromising quality, with options for more verbose outputs via prompting strategies. Google also simplified access to the models through AI Studio and Vertex AI, increasing rate limits and improving scalability for developers.

Google also released NotebookLM (only available for English speakers), which lets users upload content, ask questions, and generate podcast-like conversations between AI voices, providing human-like discussions of uploaded materials.

It supports a variety of content, as users can upload PDFs, text files, websites, and videos into "notebooks" to create customized summaries or ask questions about the content. In addition to audio overviews, NotebookLM provides study guides, accurate quotations, FAQs, and detailed summaries.

Furthermore, the first AI product creation platform was released by Arcade AI, described as “prompt-to-product” using design-to-manufacturing AI tech. It has a simple three step process where you enter a prompt, edit it, then share it with others. Currently, Arcade AI is in beta form.

Ai2's Multimodal Molmo Challenges Tech Giants

Molmo outperforms GPT-4o on academic benchmarks. (Source)

Ai2 launched Molmo, a multimodal open-source model that rivals major models like GPT-4o, Gemini 1.5, and Claude-3.5 in performance, but at a fraction of their size.

Molmo excels in tasks like answering questions about images, recognizing objects, and navigating web interfaces without needing complex infrastructure. This is thanks to the fact that it uses high-quality data, with only 600,000 annotated images, achieving impressive results compared to models trained on billions of images.

Replit's AI Agent vs Random Labs' IDE Integration

Replit’s new AI agent helps users build software projects from the ground up by using natural language prompts. As a result, it makes software development a lot more accessible to users of levels.

It’s available to Replit Core and Teams members at no additional cost during this “early access”, Pricing details will be available later in 2024 and it can be used through the web interface or mobile app.

Another agent release we saw was by Random Labs, who developed an AI agent directly integrated into the IDE. It can make changes across huge codebases and provides more than just autocomplete or chat functionality. 

AlphaProteo's Protein Revolution and LIGO's Open-Source Leap

Predicted protein binder structure generated by AlphaProteo, shown in blue. (Source) 

DeepMind made some significant advancements in biotech with the release of AlphaFold 3 a few months ago. 

Now, they’ve developed a new AI system called AlphaProteo, which can design novel, high-strength protein binders to serve as building blocks for biological and health research. It aims to accelerate progress in areas like drug development, disease understanding, and biosensors.

The system generates new protein binders for diverse target proteins, including challenging targets like VEGF-A. It achieves higher experimental success rates and 3 to 300 times better binding affinities than existing methods on seven tested target proteins.  

Sounds promising, but there’s still work to be done since the system has limitations like being unable to design binders for some challenging targets.

Speaking of AlphaFold 3, we saw an open-source implementation of it called LIGO. The current release implements the full AlphaFold 3 model along with the training code, focusing first on single chain prediction capability. Future updates will add ligand, multimer, and nucleic acid prediction capabilities once they are trained.

LIGO's implementation includes several core modules reused from the OpenFold project, such as triangular attention and multiplicative update, as well as their data processing pipelines. It also utilizes the ProteinFlow library for its data pipeline.

The project addresses some discrepancies from AlphaFold 3's published pseudocode, including changes to the MSA module order, loss scaling, and DiT block design. 

OpenAI's GPT Next Bombshell: 100x Power, $2000 Price Tag

OpenAI Japan announced “GPT Next” - a model planned for release in 2024 and is supposed to be 100x more powerful than GPT-4. This name is just a placeholder though, and we’ve seen some rumors floating around that the model is codenamed “Project Orion” (funny how this has the same name as Meta's AR glasses project).

Orion focuses on improving NLP capabilities while expanding into multimodal territory, being able to integrate text, image, and video inputs. Notably, Orion is mentioned to leverage training data generated by O1, since high-quality data for training models isn’t always abundant.

It’ll be interesting to see if Orion will actually achieve such high performance as OpenAI claims, since training AI models on excess training data can actually lead to worse performance.

In terms of pricing, executives at OpenAI are even thinking about putting subscription prices as high as $2000 for access to models like O1 and Orion, a 100x price increase from ChatGPT subscription.

Anthropic also hit a milestone, with Anthropic’s Claude recently reaching $1 million in mobile app revenue. They’re going a step further with the release of Claude Enterprise.

It’s a new subscription plan for its AI chatbot aimed at enterprise customers, offering more administrative controls and increased security. Puts Anthropic in direct competition with OpenAI's ChatGPT Enterprise, which was released about a year ago.

ColPali and Qwen2-VL for Multimodal RAG and HoneyComb Takes Top Spot on SWE-Agent leaderboard

There was a recent discussion on X about ColPali being better for information retrieval over the combination of OCR and LLMs. 

ColiPali and Qwen2-VL were used in combination. (Source)

ColPali is built on the PaliGemma-3B model, which we saw in an announcement at I/O 2024 by Google. In short, ColPali is a document retrieval model that uses VLMs to index and retrieve information from documents based on visual features.

The main aim is to tackle tricky issues that traditional document retrieval systems can’t deal with, like:

• Complex data ingestion pipeline

• OCR requirements 

• Difficulty in handling visual elements like tables and figures

A key benefit of using ColPali is that OCR or image captioning isn’t necessary. It’s also a more efficient method compared to OCR + LLMs while having higher accuracy. In addition, Qwen2-VL-7B is used for the generation process in RAG.

The Qwen2-VL models saw new releases last week with three sizes: 2B, 7B, and 72B, with the 7B instruction-tuned version being open-sourced.

It showed impressive results, achieving SOTA performance on visual understanding benchmarks like MathVista, DocVQA, and RealWorldQA. Moreover, it can understand videos over 20 minutes long for high-quality video-based tasks.

HoneyComb was a new model that took the SWE-Agent leaderboard by storm last week by achieving SOTA results, putting it in first place.

It takes a different approach to other models. Rather than having one, jack-of-all-trades AI agent, HoneyComb uses multiple AI agents that are fine-tuned to perform well at individual tasks like bug fixing and code review. 

What’s great about this method is that it creates a potentially infinite cycle of continuous improvement and bug fixing. Makes the lives of software engineers easier by being able to automate any step of the development process.

A new feature for SQL query editing was another neat release. A bas this system executes queries in real-time as the user types them, so it provides instant feedback based on the query’s results and performance. 

US, EU, and UK Sign Landmark COE Treaty on AI Safety, Draghis’ Report

Previously, the EU AI act came into force. 

The saga continues as the Council of Europe (COE) has introduced the first-ever legally binding international treaty on AI safety, called the "Council of Europe Framework Convention on AI and Human Rights, Democracy, and the Rule of Law." Major signatories include the U.S., U.K., and European Union, along with several smaller nations.

The treaty focuses on three main areas: 

1. protecting human rights (including data privacy and anti-discrimination)

2. Safeguarding democracy

3. Upholding the rule of law

Notable absences from the initial signatories include countries from Asia, the Middle East, and Russia. The treaty's goal is to create a technology-neutral framework that can withstand future developments in AI technology.

It’ll officially enter into force three months after at least five signatories, including three COE member states, have ratified it. 

Draghi's report on European competitiveness came at a crucial time for AI regulation in the EU. It follows the recent implementation of the EU AI Act and the signing of the Council of Europe's landmark AI safety treaty.

While the treaty, which focuses on protecting human rights, safeguarding democracy, and upholding the rule of law, sets a broad regulatory framework, Draghi's report highlights the economic opportunities for the EU in the AI sector.

As the EU works to balance innovation with responsible AI development, Draghi's recommendations could inform how the EU implements the principles outlined in the COE treaty, especially in areas where Europe still has a competitive edge.

Plans for Two $125 Billion Supercomputers in North Dakota

We could see the development of two massive data centers in North Dakota, with each one costing up to $125 billion. This is currently being considered by two trillion-dollar market cap companies.

This means it's the usual suspects like Nvidia, Microsoft, Apple, and Google. Rumor has it that Microsoft might be a likely candidate since they were looking into building a $100 billion supercomputer campus with OpenAI in March 2024, which we talked about in our first newsletter.

The state currently has a small data center market, with only seven facilities listed.

Applied Digital, a cryptomining and AI provider, recently secured $200 million for facility expansion in the state and has a deal with an unnamed hyperscaler. Also worth pointing out that the state produces more energy than it uses.

Advancements in AI Research

Midjourney’s competitor FLUX saw some interesting developments from the research side of things, with a new paper detailing a generative framework called FluxMusic for text-to-music generation. In addition, we got some insights into new approaches for training LLMs with long context windows and improving traditional RAG methods.

Composing the Future of AI-Generated Soundscapes With FluxMusic

We talked about Black Forest Lab’s FLUX before and noted that it’s a pretty serious competitor for Midjourney.  

This paper gives us a new approach to text-to-music generation that aims to strengthen multimodal LLM perception and support higher input resolutions, with FluxMusic being a generative framework.

It handles the issue of creating more versatile and powerful AI systems capable of generating high-quality music from textual descriptions, a task that has long been limited by the complexity of musical structure and the need for domain-specific knowledge.

They implemented a two-tiered LLM chain for content refinement and information extraction, and utilized multiple pre-trained text encoders for conditioned caption feature extraction and inference flexibility.

FLUX models support input resolutions up to over 1000 pixels and achieve strong performance on multimodal LLM benchmarks. This is especially true for resolution-sensitive tasks like OCR and document understanding. Notably, FLUX outperformed other prominent models like Mini-Gemini-HD and LLaVA-NeXT across various metrics.

Breaking the Long Context Barrier

End-to-end model training comparison. (Source)

Researchers from Ohio State University and Microsoft have introduced a novel approach to training LLMs with extremely long context windows.

Current LLM training is typically constrained to relatively short context lengths, such as 8K or 32K tokens, limiting their ability to process entire documents or maintain coherent long-term dialogues.

They introduced the Fully Pipelined Distributed Transformer (FPDT) that leverages multiple memory hierarchies in modern GPU clusters to enhance hardware efficiency and cost-effectiveness. 

This method uses:

• A chunking mechanism

• Offloading idle tokens to host memory

• A double buffer strategy to overlap computation with data transfer.

FPDT can train an 8B parameter LLM with a 2 million token context length using only 4 GPUs, achieving over 55% Model FLOPs Utilization (MFU). This represents a 16x increase in sequence length compared to current SOTA solutions while maintaining high efficiency.

How 16K Tokens of RAG Outperform 128K Context LLMs

As long-context language models (LLMs) become increasingly prevalent, some have questioned the continued relevance of RAG. A new study from NVIDIA researchers challenges this notion by showing that RAG still has a crucial role to play, even in the era of models with 100K+ token contexts.

The team introduces Order-Preserve RAG (OP-RAG), which significantly improves upon traditional RAG methods. By maintaining the original order of retrieved chunks rather than sorting by relevance, OP-RAG achieves a better balance between information recall and precision. 

They found that as the number of retrieved chunks increases, answer quality follows an inverted U-shaped curve, with an optimal "sweet spot" that outperforms long-context LLMs using the entire available context.

Results on the ∞Bench dataset are striking: OP-RAG using Llama3.1-70B with just 16K retrieved tokens achieved a 44.43 F1 score, surpassing the 34.32 score of Llama3.1-70B using its full 128K context. It even bested GPT-4o (32.36) and Gemini-1.5-Pro (43.08). 

Frameworks We Love

Some frameworks that caught our attention in the last week include:

• Anthropic Quick starts: Collection of ready-to-use projects designed to help developers quickly build deployable applications using the Anthropic API and Claude's capabilities.

• LLM-CI: Assesses privacy norms encoded in LLMs using a Contextual Integrity-based factorial vignette methodology across different contexts. 

• GetOmni AI: Open-source OCR tool that uses GPT-4 vision models to convert PDF documents into high-quality markdown text.

Conversations We Loved

One discussion raised the question of what led to public cloud storage prices plateauing in 2017 rather than continuing to decrease like earlier years.

The Great Cloud Storage Plateau

Cloud storage costs from 2007 to 2017. (Source)

A graph that tells a fascinating story about the evolution of cloud storage pricing caught our eye. The image, which tracks the cost per GB/month across various cloud providers from 2007 to 2019, reveals that after years of steep declines, prices seemed to hit a floor around 2017.

This plateau in cloud storage costs makes us wonder about the economics of cloud computing and the broader tech industry. 

Some speculate that we've reached the physical limits of storage density improvements, while others point to market consolidation and reduced competition. There's also the possibility that cloud providers have shifted their focus from raw storage to value-added services.

Money Moving in AI

A couple of huge acquisitions and investment rounds took place last week, with SSI raising $1 billion and Salesforce purchasing Own for $1.9 billion. You.com also raised $50 million from a series B funding round.

Ilya Sutskever Raises $1 Billion for New AI Safety Startup

Just a few months ago, OpenAI disbanded its AI safety team and key members left the company, including Ilya Sutskever. Sutskever created his own AI safety startup which raised a staggering $1 billion at a $5 billion valuation, just three months after its founding. 

You Raises $50 Million in Series B Funding

You.com, an AI search company focusing on complex queries and productivity tools, has raised $50 million in a Series B round led by Georgian, with participation from notable investors including Nvidia and Salesforce Ventures.

The company is betting on its ability to excel at answering complex questions and providing a "productivity engine" for knowledge workers.

Salesforce Acquires Own for $1.9 billion

Salesforce made its largest acquisition since Slack, purchasing data management firm Own for $1.9 billion in cash. The deal highlights the growing importance and value of data protection and management solutions in the enterprise space, with the global data backup and recovery sector worth $12.9 billion in 2023.

But seriously, when did we start naming stuff after fruit instead of animals? Looking at you, 🍐 (oof) and the non-profit that decided to morph into a for-profit entity, with the last founding member deciding to depart faster than you can say "I have no equity in OpenAI… I do this because I love it, 💚 ". Anyways, this week is full of news (and there have been many more over the weekend), so let's dive right in! But first:

Key Takeaways

• Mistral released Pixtral 12B, their first multimodal model capable of processing both images and text, outperforming other top models on various benchmarks.

• IBM researchers presented improvements to PyTorch, including a high-throughput data loader and enhanced LLM training throughput.

• Google developed a whale bioacoustics model capable of identifying eight distinct whale species and multiple vocalizations.

• General OCR Theory employs a unified end-to-end model with a high-compression encoder and long-context decoder, which outperformed existing models on various OCR tasks.

• OpenAI's o1 model shows significant improvements over GPT-4 on reasoning-heavy tasks, rivaling human expert performance on many benchmarks.

• A paper drawing parallels to quantum mechanics applies the Universal Approximation Theorem to explain LLM memory mechanisms, and found some models able to memorize nearly 100% of 2,000 poems after limited exposure.

Got forwarded this newsletter? Subscribe below👇

The Latest AI News

We’ve been hearing about Project Strawberry for a while and it’s finally here, with OpenAI releasing a preview version of the model called o1. There were a bunch of other releases too, including multimodal modals, speech-text foundation models, and open-source development frameworks. 

Some interesting models for life science applications also came up, which might have gone under the radar amidst the o1 release hype.

Mistral Enters Multimodal Arena and Groq’s New Release

Mistral’s first multimodal model showed promising results on various multimodal benchmarks. (Source) 

Pixtral 12B is a new multimodal AI model from Mistral, who previously released three models at around the same time. This new model is capable of processing both images and text. It has 12 billion parameters and is approximately 24GB in size.

Built on Mistral's Nemo 12B text model, it can answer questions about multiple images of any size using URLs or base64-encoded images.The model is expected to perform tasks like image captioning and object counting, similar to other multimodal models like Claude and GPT-4o.

Pixtral 12B also performed well on various multimodal benchmarks like MMMU and ChartQA, outperforming other top models like Claude 3 Haiku and Phi-3 Vision.

Don't hate us for this shot. Pixtral 12B’s results on various multimodal benchmarks. (Source)

Instead of another multimodal model release, we saw an open-source development framework pop up. xRx by Groq lets developers make conversational AI solutions by combining multimodal input and outputs.

It makes life easier for users wanting to develop projects like voice-based assistants, text-based chatbots, and multimodal applications by giving them all the tools they need in one place. 

Oracle’s Multicloud Vision and Tensor Parallelism Breakthroughs

The conference season is upon us (while we're at it - grab your tickets for RetrieveX). PyTorch Conference, Dreamforce, the Oracle CloudWorld…

Speaking of the latter, Oracle's CTO Larry Ellison announced a new partnership with AWS, following similar deals with Microsoft and Google. The partnership, called Oracle Database@AWS, will embed Oracle Cloud Infrastructure inside AWS data centers. They've also annouced APEX, a low-code development platform, uses AI to help create secure applications from the start.

What’s more is that Ellison mentioned that Oracle is already building a >1GW data center that’s powered by 3 data centers, and that Oracle is probably going to have 2000 data centers worldwide. 

The PyTorch conference also took place at around the same time. Previously, we saw the release of a PyTorch API. 

Developments are continuing as we saw the implementation of experimental async tensor parallelism support, integrated into TorchTitan. It can be used with torch.compile, which automatically detects TP patterns and rewrites them into async-TP ops, or directly in eager mode by calling async-TP ops.

There were a couple of key performance challenges tackled by this implementation, including communication overhead and magnified wave quantization inefficiencies. As such, addressing these issues helped maintain real-time performance while incorporating complex learning mechanisms.

Some notable results include:

• In Llama3 7B, async-TP achieved up to ~29% forward pass speedup and ~8% E2E speedup.

• For Llama3 70B, it showed up to ~20% forward pass speedup and ~8% E2E speedup.

Project Strawberry Benchmarks: 'AGI Wen?'

OpenAI’s latest model shows drastic improvements in competitive benchmarks over GPT-4. (Source)

After all the build up, OpenAI finally made an official announcement about Project Strawberry. It’s actually called “OpenAI o1” instead.

In the past, they used a combination of supervised and reinforcement learning for models like GPT-4. This time, OpenAI took a slightly different approach to training their latest model, using a large-scale reinforcement learning algorithm to teach the model productive thinking. The model’s performance consistently improves with more reinforcement learning (train-time compute) and more time spent thinking (test-time compute).

In terms of results, it seems like OpenAI o1 lives up to the hype by significantly outperforming GPT-4o on most reasoning-heavy tasks. It also shows major improvements on challenging benchmarks like AIME (math), Codeforces (programming), and GPQA Diamond (PhD-level science questions), exceeding human expert performance on reasoning-heavy tasks. We've yet to see nice results in production (beyond… Devin's reference), but we are currently using and liking it, too.

Currently, o1-preview is being released for immediate use in ChatGPT and to trusted API users. OpenAI is also A/B testing a feature where model selector is disabled, opening up speculations they're working on model router to mange inference costs on consumer plans (and serving a ‘good enough' model).

Since PhD-level intelligence was a hot topic during the whole speculation phase of o1, it was interesting to see how it stacks up against other models on ARC Prize. 

Salesforce, Microsoft, and Amazon’s New Releases

After releasing a couple of autonomous AI agents, Salesforce also released Agentforce (confidently priced at $2 per conversation, ugh), which offers various pre-built AI agents for specific business functions, including Service Agents, SDRs, Sales Coaches, Personal Shoppers, and Campaign Agents.

These agents are autonomous applications that provide 24/7 specialized support to employees or customers across multiple channels like web, mobile, WhatsApp, and Slack. 

Microsoft is launching the next wave of Copilot, integrating web, work, and Pages as a new design for knowledge work. There’s also a bunch of Improvements being made to Copilot in various Microsoft 365 apps.

With GPT-4o and enhanced orchestration, Copilot responses are now more than two times faster on average. Response satisfaction has improved by nearly three times, so the improvements are definitely noticeable.

Copilot Pages (Perplexity called and wants its naming back…) is described as the first new digital artifact for the AI age, which allows users to edit, add to, and share AI-generated content.

Amazon launched Project Amelia, a generative AI assistant for sellers, offering tailored business insights, guidance, and actions using Amazon Bedrock technology. Sellers can consult Amelia for best practices and strategies, receiving tailored responses based on their data and market trends. It offers quick access to sales data, customer traffic insights, and product performance analysis to help monitor progress.

New Speech-Text Foundation Model and Turning Ideas Into Apps

Moshi overview. (Source)

Kyutai Labs debuted a new speech-text foundation model called Moshi. It uses Mimi, a SOTA streaming neural audio codec that processes 24 kHz audio down to a 12.5 Hz representation with a bandwidth of 1.1 kbps.

The model consists of two main components:

1. A small Depth Transformer for inter-codebook dependencies

2. A large 7B parameter Temporal Transformer for temporal dependencies

Mimi outperforms existing non-streaming codecs like SpeechTokenizer and SemantiCodec in terms of efficiency and quality Moshi achieves a theoretical latency of 160ms (80ms for Mimi's frame size + 80ms of acoustic delay), with practical latency as low as 200ms on an L4 GPU. 

The trend of accessible coding seems to be continuing with Llamacoder, which lets you enter a simple prompt to turn your idea into a fully developed app within a matter of minutes. Keep in mind that it’s only for smaller apps though.

It uses Llama 3.1 405B for code generation and Together AI for LLM inference - a nice combination that allows for efficient processing of prompts and code generation. 

There’s also other features in the works currently, such as adding support for multiple programming languages and plans for more customization options in the future.

From Whale Songs to Cellular Automata

Google decided to go in a different direction with a unique AI application. They developed a whale bioacoustics model that can identify eight whale species and multiple vocalizations, including the Bryde's whale "Biotwang" sound. This is important for protecting whales that live in remote environments, since it’s hard to find them through other methods.

The model uses spectrograms to classify twelve whale vocalization classes, offering detailed species identification and recognizing various vocalization types. Results were certainly impressive since it showed high accuracies across different whale species, excelling in identifying complex sounds like "boings" and "gunshot" calls.

As such, it’s had some notable implications for discoveries about whale populations, movements, and behaviors.

More AI applications in the life sciences were seen last week with LifeGPT, a generative transformer model designed to simulate Conway's Game of Life (Life). Life is a cellular automaton simulation, which are models of computation that are useful in areas like biology.

The issue is that algorithms within cellular automata like Life are tough to model and predict due to sensitivity of initial conditions and the fact that prior understanding is needed. LifeGPT helps us solve this issue by simulating Life without needing prior knowledge.

The model is trained on various grid configurations, allowing it to generalize across different grid sizes and periodic boundary conditions effectively. LifeGPT accurately captures the complex dynamics of cellular automata with near-perfect accuracy.

We also saw the introduction of the "autoregressive autoregressor" method, which allows for recursive simulation of Life using LifeGPT for long-term predictions.

As a result, it would help design models that would have big implications for areas like bioinspired materials and tissue engineering.

A note from our partner

Advancements in AI Research

One paper really stood out since it made an interesting parallel that we thought we’d never see between quantum mechanics and LLMs. Progress was also made in applying AI for scientific research tasks and dealing with limitations that traditional OCR systems had a tough time with.

AI Outperforms Experts in Scientific Research Tasks with PaperQA2

Researchers from FutureHouse developed PaperQA2, which showed SOTA performance in synthesizing scientific knowledge. Creating AI systems that can accurately and reliably help with scientific research tasks has been an issue for quite some time, so this paper helps address a key issue.

PaperQA2 used a frontier language model optimized for improved factuality, combined with an agentic approach to information retrieval and synthesis.

The system uses a multi-step process involving paper search, evidence gathering, and answer generation - all orchestrated by an agent model. But what really stands out is the Reranking and Contextual Summarization (RCS) step, which improves the relevance and quality of retrieved information.

The researchers evaluated PaperQA2 on three real-world tasks:

1. Answering scientific questions

2. Generating Wikipedia-style summaries

3. Detecting contradictions in scientific literature

PaperQA2 matched or exceeded human expert performance across these tasks. For example, it achieved superhuman precision on the LitQA2 question-answering benchmark and produced more accurate and better-cited Wikipedia-style summaries than existing human-written articles.

GOT's Comprehensive Solution for Visual Data Extraction

General OCR Theory (GOT) handles the limitations of traditional OCR systems in handling diverse types of artificial optical signals, from plain text to complex formulas and charts. It’s a unified end-to-end model consisting of a high-compression encoder and a long-context decoder. 

The encoder, with about 80M parameters, processes 1024x1024 input images and compresses them into 256x1024 dimensional tokens. 

The decoder, with 500M parameters, supports an 8K max token length for handling long-context scenarios.

They developed a multi-stage training strategy, including decoupled pre-training of the encoder, joint training with a new decoder, and further post-training. They also created specialized data engines for synthetic data production to support each training stage.

Results show that GOT outperforms existing models on various OCR tasks, including plain text recognition, formatted document understanding, and more specialized tasks like sheet music and chart recognition. For example, on the ChartQA-SE benchmark, GOT achieved an AP@strict score of 0.747, surpassing other models including GPT-4V and Qwen-VL.

LLMs and Quantum Mechanics: The Surprising Connection in Memory Models

Researchers from The Hong Kong Polytechnic University proposed a new framework for understanding memory in LLMs in a way that you wouldn’t have thought of.

They drew some pretty interesting parallels to quantum mechanics to answer the fundamental question of whether LLMs truly possess memory and, if so, how it functions compared to human memory.

The study leverages the Universal Approximation Theorem (UAT) to explain the memory mechanism in LLMs. They argue that LLM memory operates like "Schrödinger's memory" - it only becomes observable when a specific memory is queried, and its existence can’t be determined otherwise.

They demonstrated this concept through experiments on various LLMs, including fine-tuning models on poetry datasets to assess their memory capabilities.

Results show that LLMs can exhibit remarkable memory performance, with some models able to memorize nearly 100% of 2,000 poems after limited exposure. The study also revealed memory performance decreases as input text length increases, mirroring human memory limitations.

Frameworks We Love

Some frameworks that caught our attention in the last week include:

• Vista3D:  Rapid and consistent 3D object generation from a single image, using a two-phase approach.

• NSSR-DIL: Reformulates image super-resolution as computing the inverse of degradation kernels, rather than directly generating high-resolution images from low-resolution inputs

• MoRAG: Text-based human motion generation that enhances motion diffusion models by leveraging additional knowledge from an improved motion retrieval process

If you want your framework to be featured here, reply to this email saying hi :)

Conversations We Loved

Don't worry, no fruit were harmed (or community noted) in our selection. One of our pics of the week went into depth about real-world usage patterns of data warehouses stood out since it goes against the common assumptions about these systems. 

Another post also grabbed our attention, which looked at cloud pricing for Kafka deployments, showing that there’s a big disparity between cloud costs and on-premise hardware capabilities for data streaming workloads.

Rethinking Data Infrastructure and Processing

Fraser looked usage patterns of data warehouses. (Source)

There was a deep dive into the real-world usage patterns of data warehouses, courtesy of George Fraser, CEO of Fivetran. His analysis of query samples from Snowflake and Redshift offered some surprising insights that challenge common assumptions about how these systems are used in practice.

The post highlights that data warehouses are mainly used for ETL (Extract, Transform, Load) tasks, not just for business intelligence. Most queries are small, scanning only about 100 MB of data, which challenges the focus on massive scalability and has important implications for data infrastructure and processing.

The Hidden Costs of Cloud Kafka

Kozlovski’s post about cloud pricing for Kafka deployments. (Source)

Kozlovski’s breakdown of cloud pricing for Kafka deployments gained a lot of attention on X as it exposed the disparity between cloud costs and on-premises hardware capabilities for data streaming workloads.

He explained how a modest 30MB/s Kafka cluster could rack up over $110,000 in annual costs, with network traffic alone accounting for $88,300 of that sum. It shows a fundamental misalignment between cloud pricing structures and the needs of data-intensive applications.

He also went into the reasons behind this pricing anomaly and explored potential solutions. From optimizations like "fetch from follower" to more radical approaches like WarpStream's innovative design, the discussion brought the ongoing struggle between cloud convenience and cost-effectiveness to light. Interesting.

Money Moving in AI

While Microsoft launched the next wave of Copilot, they also announced a partnership with BlackRock where they’ll invest $30 billion for AI infrastructure development support. Other news included Nvidia thinking about acquiring OctoAI for $165 million. 

In terms of successful funding rounds, Sakana AI and Black Forest Labs raised $210 million and $100 million respectively.

OpenAI's $150B Sprint

In the next episode full of plot twists worthy of a soap opera, Apple has decided to sit out the $6.5 billion OpenAI's (allegedly oversubscribed?) funding round, leaving Microsoft and Nvidia to fight over the last dance. Rumor has it, Microsoft is sweetening the pot with an extra $1 billion, probably hoping to impress OpenAI with its deep pockets and irresistible charm. Beyond SoftBank at $500M, here's the list of other firms who are reportedly in talks to invest.

Microsoft and BlackRock’s Initial Investment of $30 Billion

Microsoft and BlackRock have announced a joint venture, the Global AI Infrastructure Investment Partnership, with an initial investment of over $30 billion to support the development of AI infrastructure, focusing on data centers and energy resources. 

The partnership, which includes MGX as a general partner and expertise from Nvidia, aims to address the growing demand for computational power required by AI models while ensuring sustainable development.

Sakana AI Raises $210 Million in Series A Funding

Sakana AI, an NVIDIA-backed startup, has raised over $210 million in a Series A funding round, doubling its initial target of $100 million and achieving a valuation exceeding $1.5 billion just a year after launch.

The company has gained popularity for developing a method to automate the integration of multiple foundational models.

Nvidia May Acquire OctoAI for $165 Million

Nvidia is reportedly considering acquiring OctoAI, a startup that develops software to enhance AI model efficiency, for $165 million.

This potential acquisition follows OctoAI's recent collaboration with Nvidia to integrate NIM into its generative AI platform, aiming to serve various enterprise use cases.

Black Forest Labs Reported to Raise $100 Million

Black Forest Labs, the startup behind Grok's image generator (and competitor to Midjourney), is reportedly raising $100 million at a $1 billion valuation, just two months after emerging from stealth with $31 million in funding. 

Previously, the startup’s valuation was around $150 million during the last funding round, so it’s definitely a big increase.

In a new experiment, we decided to provide a monthly mega-roundup of all things that may have flown under the radar for you in August. Before we start, share last week's news with a friend or a colleague:

Key Takeaways for August

• Hermes 3 by Nous Research showed significant improvements over its predecessor, demonstrating competitive benchmark scores against Llama 3.1 through enhanced training techniques.

• Ideogram 2.0, a SOTA text-to-image model, is now freely available with enhanced features and styles, improving image quality and text rendering through advanced training methods.

• Google AI Edge's MediaPipe enabled running 7B+ parameter language models in browsers using WebAssembly and WebGPU, overcoming memory restrictions through redesigned model-loading code.

• Researchers developed Pyramid Attention Broadcast (PAB) for real-time video generation, achieving up to 20.6 FPS with a 10.5× acceleration by mitigating redundancy in attention computations.

• Google DeepMind open-sourced the Vizier algorithm, outperforming industry baselines in black-box optimization through a Gaussian process bandit approach.

• Anthropic’s new prompt caching feature dramatically reduces costs and latency for long prompts, set to become an industry norm.

• Huawei challenges Nvidia with the Ascend 910C AI chip, targeting the Chinese market amid production difficulties due to U.S. sanctions.

• Grok-2 and Grok-2 mini outperformed GPT-4 Turbo in benchmarks such as GPQA and MMMU, excelling in reasoning and factual accuracy.

• DeepSeek-Prover-V1.5 is an advanced theorem-proving LLM with improved performance on formal mathematics tasks, showcasing state-of-the-art results on rigorous benchmarks like ProofNet.

Got forwarded this newsletter? Subscribe below👇

The Latest AI News

Things seemed to have taken a strange turn regarding the Strawberry situation. Meanwhile, some neat optimization frameworks and libraries were introduced, along with a bunch of language and image generator models.

There were quite a few model releases, ranging from SLMs to text-to-image models. Moreover, a chatbot arena update saw Grok-2 ranked very highly, which means xAI might take the top spot very soon.

Salesforce, Jamba, and Hermes Expand the AI Model Landscape

Comparison of claimed vs effective context window for the RULER benchmark. (Source)

Salesforce announced the release of Einstein SDR Agent and Einstein Sales Coach Agent. Einstein autonomously manages inbound leads by answering questions, handling objections, and scheduling meetings, all while grounded in a company’s CRM and external data. 

On the other hand, Einstein Sales Coach Agent does exactly what you’d expect from the name - coach sellers via role-plays and provide personalized feedback afterwards.

What’s more is that these AI agents can be tailored to a company’s specific needs, including setting engagement guardrails and language preferences, making them highly adaptable to different sales strategies. We’re seeing companies like Accenture use them to improve deal effectiveness and scale support for more complex sales activities.

Another release included AI21 Labs’ Jamba 1.5 Mini and Large models, built on the novel SSM-Transformer architecture. These offer superior long-context handling, speed, and quality. These models are the first non-Transformer models to match the performance of leading competitors, featuring a 256K context window—the longest in the market for open models.

The Jamba 1.5 models are designed for resource efficiency, capable of handling up to 140K tokens on a single GPU for Jamba 1.5 Mini. 

In particular, these models stand out because they maintain high performance across the entire context window, significantly improving the efficiency and accuracy of enterprise-scale applications. Independent tests showed Jamba 1.5 Mini as the fastest model on 10K contexts, outpacing other models in its size class.

Hermes 3 by Nous Research was also a noteworthy release with drastic improvements over its predecessor, Hermes 2. Some of which include:

• More reliable function calling 

• Better code generation skills

• Enhanced general assistant capabilities

In terms of benchmark performance, Hermes 3 is certainly no slouch as it showed highly competitive benchmark scores with Llama 3.1.

Hermes 3 was able to outperform Llama 3.1 on benchmarks like AGIEval and ARC-C. (Source)

Ideogram’s New Model and FLUX Available on 3 Platforms

Sample image produced by Ideogram 2.0. (Source)

Ideogram 2.0, a SOTA text-to-image model, is now accessible to all users via Ideogram.ai and a newly launched iOS app. The platform also introduces premium features through subscription plans, offering enhanced creative control and flexibility.

The model supports various styles such as realistic and design, so users can create highly detailed and context-specific images. These styles boost the realism of textures and accuracy of text in designs.

Improvements over its predecessor include creative tools like Magic Prompt and Describe, which help users generate and refine prompts for image creation. These tools enhance the iterative creative process, allowing for continuous reimagining of visual concepts.

We mentioned before that Midjourney had some serious competition with Black Forest Labs’ FLUX-1 models. Turns out these models are available on three platforms: 

• Replicate (no free credits)

• Fal ($1 free credit)

• Mystic (available for free)

Google, Nvidia, and Microsoft Advance SLMs

Phi 3.5-MoE showed impressive modal quality on the MMLU benchmark. (Source)

Given the recent trend of the rising popularity of SLMs recently, it isn’t too surprising to see more SLM releases.

Google AI Edge's MediaPipe redesigned its model-loading code to overcome memory restrictions, which means larger language models (7B+ parameters) can be run in the browser using their cross-platform inference framework.

The framework compiles C++ code into WebAssembly for efficient browser performance while leveraging WebGPU API for direct GPU access. New strategies, such as asynchronous layer loading and local caching, drastically reduced WebAssembly memory usage, so larger models can run smoothly.

After Nvidia discussed how to prune Llama-3.1 8B to Llama-3.1-Minitron-8B, they released Mistral-NeMo-Minitron 8B. It’s a miniaturized version of the Mistral NeMo 12B model, which delivers SOTA accuracy in a compact 8 billion parameter form factor. 

Mistral-NeMo-Minitron 8B leads on nine popular benchmarks for language models of its size, covering tasks like language understanding, reasoning, summarization, coding, and generating truthful answers. 

NVIDIA also announced Nemotron-Mini-4B-Instruct, another SLM optimized for low memory usage and faster response times on NVIDIA GeForce RTX AI PCs and laptops, available as part of NVIDIA ACE technologies.

Meanwhile, Microsoft introduced Phi-3.5, an updated family of SLMs including:

• Phi-3.5-mini (3.8B parameters)

• Phi-3.5-vision

• Phi-3.5-MoE (Mixture-of-Experts with 42B total parameters but only 6.6B active).

Phi-3.5-mini enhances multi-lingual support with a 128K context length, showing significant improvements in languages like Arabic, Dutch, Finnish, Polish, Thai and Ukrainian with 25-50% performance boosts.

Phi-3.5-MoE outperforms larger dense models in quality and performance, supporting over 20 languages and employing robust safety post-training strategies combining Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO).

Things seemed to have taken a strange turn regarding the Strawberry situation. Meanwhile, some neat optimization frameworks and libraries were introduced, along with a bunch of language and image generator models.

AI Optimization Trifecta: LLM Compressor, Apple’s ToolSandbox, and PyTorch’s FlexAttention

LLM compressor overview. (Source)

NeuralMagic’s LLM Compressor is a unified library for creating compressed models for faster inference with vLLM. It enables various compression techniques like weight quantization, activation quantization, and pruning.

We saw that Llama 3 was adapted to multimodality. NeuralMagic also did some work with Llama 3 by using LLM Compressor to create fully quantized versions of Llama 3.1.

It also integrates seamlessly with Hugging Face models and vLLM, which makes deployment pretty straightforward. LLM Compressor showed notable performance improvements, with INT8 weight and activation quantized models showing up to 1.6x speedup compared to FP16 baselines at low query rates.

ToolSandbox is a new evaluation framework for assessing tool-use capabilities of LLMs, addressing limitations of previous evaluation methods. It introduces stateful tool execution and implicit state dependencies between tools, moving beyond simple stateless web services or single-turn prompts.

The framework includes a built-in user simulator that enables on-policy conversational evaluation, allowing for more dynamic and realistic testing scenarios. ToolSandbox implements a dynamic evaluation strategy that can assess both intermediate and final milestones over arbitrary interaction trajectories.

Optimization news continued with FlexAttention, a new PyTorch API that allows implementing many attention variants in a few lines of idiomatic PyTorch code, addressing the lack of flexibility in existing optimized attention implementations.

It introduces a flexible API with a user-defined score_mod function that can modify attention scores prior to softmax, which enables various attention patterns like:

• Causal masking

• Relative positional encodings

• Sliding window attention

FlexAttention uses torch.compile to lower the user-defined functions into a fused FlashAttention kernel, achieving performance competitive with handwritten kernels without materializing extra memory.

Llama 3 Pruning and Claude's Caching Technique

Pruning and distillation process of a single model. (Source)

We’ve seen how small language models are recently attracting attention with various releases such as GPT-4 mini and Gemma 2 2B. Nvidia is continuing the SLM momentum by showing us how larger models can be pruned to obtain a smaller model, using Llama-3.1-Minitron-4B as an example. This involves structured weight pruning combined with knowledge distillation.

The pruning process includes both depth pruning (removing 16 layers) and width pruning (reducing embedding and MLP dimensions).

Knowledge distillation is used to retrain the pruned model, with the original 8B model serving as the teacher. The pruned and distilled 4B model outperforms other models of similar size on various benchmarks, while requiring fewer training tokens and compute resources compared to training from scratch. 

Anthropic has introduced prompt caching for its Claude AI models, letting developers store and reuse large amounts of context between API calls. This feature is currently available in public beta for Claude 3.5 Sonnet and Claude 3 Haiku, with support for Claude 3 Opus coming soon.

Prompt caching can reduce costs by up to 90% and latency by up to 85% for long prompts. It's particularly useful for scenarios like conversational agents, coding assistants, large document processing, and agentic search where repeated access to extensive context is needed.

The pricing model for cached prompts involves a 25% premium over base input token prices for writing to the cache, but only 10% of the base price for reading cached content. This structure makes it appealing to frequently reuse cached prompts. 

Prompt caching prices. (Source)

Early adopters have reported significant improvements in both speed and cost across various use cases. For example, chatting with a book using a 100,000 token cached prompt saw a 79% reduction in latency and 90% cost reduction.

Interestingly, it seems like prompt caching will become an industry norm at this rate.

Matt Shumer pointed out that Google, DeepSeek, and Anthropic have all incorporated prompt caching. (Source)

Grok-2, GPT-4o, and Gemini Live Enter the Arena

Grok-2 and Grok-2 mini show notable improvement in various benchmarks compared to Grok-1.5 and other SOTA models. (Source)

Grok-2 and its smaller version, Grok-2 mini (another SLM), were released and showed significant improvements in chat, coding, and reasoning over Grok-1.5. Grok-2, tested under the name "sus-column-r," outperforms models like Claude 3.5 Sonnet and GPT-4-Turbo on the LMSYS leaderboard.

Grok-2 shows superior reasoning, factual accuracy, and tool use, particularly excelling in academic benchmarks such as MATH, MMLU, and DocVQA. Grok-2 is integrated into the X app, providing advanced AI assistance, real-time information, and enhanced search functionalities for Premium users.

It was also added to the LM Arena leaderboard, which compares various LLMs. The leaderboard includes a win-rate heatmap, showing how Grok-2 performs against other models in head-to-head comparisons.

Grok-2 has climbed to the top of the leaderboard. (Source)

Grok-2’s performance is comparable to top LLMs, only losing to the latest GPT-4 model and Gemini-1.5 pro. Pretty impressive considering that xAI was only founded in March 2023 and already caught up to the likes of OpenAI.

Meanwhile, OpenAI launched a new GPT-4o model in ChatGPT, enhancing multi-step reasoning and detailed explanations, leading to more accurate and logical responses. The new model independently generates images, surpassing DALL-E 3 in both speed and visual accuracy, improving the integration within ChatGPT. 

Users noticed significant improvements in both reasoning and image generation, leading to better quality outputs and a more seamless experience. 

Google didn’t just sit on the sidelines and X.ai and OpenAI released new models. They unveiled Gemini Live, a conversational AI voice assistant, at its Made by Google event alongside new Pixel phones, AI photo editing tools, and Pixel Buds Pro 2 with Gemini AI. The Gemini Live demo had some issues during the presentation.

Q’s Efficiency Gains and Jassy's Long-Term Confidence

Amazon CEO Andy Jassy. (Source)

Amazon remains "very bullish" on AI's medium to long-term impact across all businesses, with their AI business already at a "multibillion dollar revenue run rate". 

We’re seeing Amazon leverage AI in its e-commerce business, including an AI shopping assistant called Rufus, apparel simulation features, and a "Project Private Investigator" using AI and computer vision to detect product defects.

On AI costs, Jassy stated Amazon has developed expertise in managing capacity for AWS and AI customers. While investing significantly in AI infrastructure, they still see more demand than current capacity. AWS brought in $26.3 billion in Q2, up 19% year-over-year, with an annualized revenue run rate over $105 billion. 

Jassy also mentioned that Amazon’s AI assistant, Amazon Q, has significantly reduced software upgrade times, cutting the average time to upgrade an application to Java 17 from 50 developer days to just a few hours.

This efficiency has saved Amazon an estimated 4,500 developer-years of work, with 79% of AI-generated code reviews being shipped without additional changes. The upgrades have not only saved time but also enhanced security and reduced infrastructure costs, providing an estimated $260 million in annualized efficiency gains.

Amazon Q's success comes after initial challenges, including issues with incorrect outputs or "hallucinations". These were addressed by expanding the team of human reviewers to fine-tune the chatbot's outputs.

AI Chip Race Heats Up as Huawei Challenges Nvidia While Softbank Pivots from Intel

Huawei’s latest AI chip. (Source)

There always seems to be something going on in the AI chip market. Huawei is developing the Ascend 910C AI chip to compete with Nvidia's GPUs in the Chinese market, particularly against the HGX H20 and the rumored Blackwell-based B20.

Major Chinese companies like Baidu and China Mobile have already tested the chip, with results reportedly on par with Nvidia's H100. 

Expected demand for the Ascend 910C could exceed 70,000 units, with shipments targeted to start in October, but production isn’t going as smoothly as expected because of U.S. sanctions.

The Ascend 910C aims to improve upon its predecessor, the Ascend 910B, by addressing yield issues and enhancing performance.

Previously, we looked at how Intel’s discussions with OpenAI were a huge turning point in the AI chip market. Intel’s story continues as SoftBank halted its AI chip development partnership with Intel, since Intel had issues meeting production volume and speed requirements. SoftBank is now negotiating with TSMC, the world's largest contract chipmaker, for AI chip production.

SoftBank's Project Izanagi aims to develop AI processors to rival Nvidia’s GPUs, initially relying on Intel’s capacity but now looking to TSMC. SoftBank plans to establish AI data centers globally by 2026 and is developing AI chips with Arm, targeting a prototype by 2025.

AI Image Generation Leaps Forward With Google’s Imagen 3, Runway’s Gen-3, and Midjourney's Unified Editor

Example of an image produced by Imagen 3. (Source) 

After releasing the next generation of Gemma 2 models, Google released Imagen 3, an advanced AI text-to-image generator. This version was first announced during Google I/O in May 2024.

Imagen 3 introduces significant improvements in image generation, producing visuals with better detail, richer lighting, and fewer distracting artifacts compared to previous iterations. It aims to enhance realism and reduce errors in the generated images.

Users can interact with the generated images by highlighting specific sections and applying changes based on their descriptions, offering a more refined and customizable image creation experience.

AI image generators continued to see more releases with Runway ML officially launching Gen-3 Alpha Turbo, an upgraded AI video generation model, promising seven times faster performance at half the cost compared to its predecessor, Gen-3 Alpha.

The new model is accessible across all subscription plans, including free trials. It's priced at 5 credits per second of generated video, making it more affordable and widely available.

Gen-3 Alpha Turbo prioritizes speed, significantly reducing video generation time. This improves workflow efficiency, particularly for users needing quick turnarounds.

Early users have praised the model's speed and quality, with some still favoring the original for certain use cases - though the faster version is well-received for simpler tasks.

Midjourney introduced a new web editor that unifies inpainting, outpainting, and other tools into a single interface, making it easier to edit AI-generated images seamlessly. The new editor includes a more precise virtual brush for inpainting, replacing older tools, allowing for finer control over image edits.

Harvey’s Impressive Retention

Harvey showed a notable increase in user retention. (Source)

Harvey, an AI platform for professional services, has seen significant growth in utilization, more than double from 33% in August 2023 to 69% in August 2024 across all users.

User retention rates for Harvey have remained consistently high, hovering around or above 70% after one year - exceptional for enterprise SaaS and legal tech.

In particular, three case studies of BigLaw firms show rapid and substantial adoption of Harvey: 

• Firm #1 reached 93% utilization by month 12

• Firm #2 jumped from 19% to 97% utilization in one month

• Firm #3 exceeded 100% utilization from month 4 onwards, peaking at 128% by month 10.

Harvey's success in rapid onboarding and consistent usage over time highlights its effectiveness in delivering immediate value, ease of integration into existing systems, and potential to provide firms with competitive advantages in service delivery and client satisfaction.

Not to mention that Harvey recently had a successful funding round involving tech giants like OpenAI and Google. Harvey’s chatbot was able to impress OpenAI executives with an 86% accuracy rate, with the deal making Harvey the highest value startup in OpenAI’s portfolio.

Claude Hits $1M Mobile Milestone

Graph of consumer spending in Claude App in recent weeks. (Source)

Anthropic’s Claude has crossed $1 million in gross mobile app revenue across iOS and Android in 16 weeks since launch. Nearly half (48.4%) of Claude's mobile revenue has been generated by users in the US.

It’s certainly an impressive milestone, but Claude still ranks far behind ChatGPT, which is No. 1 by overall downloads and No. 26 by revenue in the US on iOS. Claude is only 95th in the Productivity category by downloads and 68th by revenue.

Claude reached the $1 million revenue mark faster than competitors like Microsoft's Copilot (19 weeks) and Perplexity (22 weeks), but significantly slower than ChatGPT (3 weeks).

Advancements in AI Research

Google not only showed that 7B models can be run in browsers, but they also published an interesting paper that made notable progress in black-box optimization. Moreover, another paper looking at the impact of code data in LLM pre-training also caught our eye.

We saw notable advancements in automated theorem proving, efficient model upcycling, and novel frameworks for evaluating AI systems.

Google's Vizier Algorithm Outperforms Industry Baselines in Black-Box Optimization

Main components of the algorithm. (Source)

Researchers from Google DeepMind have formalized and open-sourced the algorithm behind Google Vizier, one of the world's largest black-box optimization services. 

It looks at the challenge of creating a robust, versatile, and production-grade Bayesian optimization system that can handle a wide range of optimization scenarios, from high-dimensional spaces to categorical parameters and multi-objective optimization.

The algorithm employs a Gaussian process bandit optimization approach with several key innovations, including sophisticated input and output preprocessing, flexible acquisition functions with trust regions, and a customized Firefly algorithm for acquisition optimization. 

DeepMinds’s Vizier algorithm consistently outperforms other industry-wide baselines across multiple axes, including non-continuous parameters, high-dimensional spaces, batched settings, and multi-metric objectives. For example, it demonstrates up to 8.2% relative improvement in natural language reasoning tasks compared to text-only pre-training.

Code in LLM Pre-training Improves Natural Language Reasoning by 8.2%

Framework overview. (Source)

Cohere researchers conducted a comprehensive study to understand the impact of code data in pre-training LLMs on a variety of downstream tasks beyond code generation.

The team employed a systematic approach, conducting extensive ablations across various dimensions:

1. Model initialization strategies

2. Different proportions of code data

3. Quality and properties of code data

4. Role of code in pre-training cooldown

Their experiments spanned models ranging from 470 million to 2.8 billion parameters, evaluating performance on natural language reasoning, world knowledge, code generation, and open-ended text generation tasks.

Key findings include:

• Compared to text-only pre-training, the best variant with code data showed relative increases of 8.2% in natural language reasoning, 4.2% in world knowledge, and a 6.6% improvement in generative win-rates

• Code performance saw a dramatic 12x boost

• Including code during the cooldown phase led to further improvements across all tasks

• High-quality synthetic code data, even in small proportions, had a strong positive impact on both code and non-code task performance

Results make it clear that code is a critical building block for generalization far beyond coding tasks. Investments in code quality and preserving code during pre-training can have positive impacts across a wide range of AI capabilities.

Automating the Full Cycle of ML Research With The AI Scientist

The AI Scientist overview. (Source)

Sakana AI developed The AI Scientist, which aims to automate the entire scientific discovery process in ML, from idea generation to paper writing and peer review. It addresses the challenge of scaling up scientific research and democratizing access to cutting-edge AI developments by leveraging LLM to perform tasks traditionally done by human researchers.

The AI Scientist uses a combination of LLM-based agents for idea generation, experiment design and execution, paper writing, and peer review. It employs techniques such as chain-of-thought reasoning, self-reflection, and automated code generation to carry out complex research tasks. The system was tested on three ML subfields: diffusion modeling, transformer-based language modeling, and learning dynamics.

Results show that The AI Scientist can generate hundreds of research papers at a surprisingly low cost (approximately $15 per paper), with some papers achieving scores that exceed the acceptance threshold for top ML conferences according to an automated reviewer. The framework demonstrates the potential for AI to significantly accelerate scientific progress and lower barriers to entry in AI research.

Boosting LLM Decision-Making in Interactive Environments Using AgentQ

Example of an input and output to the Agent. (Source)

MultiOn and Stanford University researchers introduced Agent Q - a framework that aims to improve the reasoning and decision-making capabilities of LLM in interactive, multi-step environments like web navigation. 

It tackles the issue of generalizing LLMs to agentic tasks, where they need to understand how their actions affect the environment and make complex decisions over multiple steps.

Agent Q combines guided Monte Carlo Tree Search (MCTS) with a self-critique mechanism and iterative fine-tuning using an off-policy variant of Direct Preference Optimization (DPO). The framework uses AI feedback and self-criticism to guide search steps, and learns from both successful and unsuccessful trajectories through offline reinforcement learning.

Drastic improvements were seen on the WebShop benchmark and real-world booking scenarios. For example, it boosts a Llama-3 70B model's zero-shot performance from 18.6% to 81.7% success rate on a real-world reservations booking website after a single day of data collection, and further to 95.4%.

It’s worth noting that the approach used by Agent Q was also used by Salesforce to achieve 55% on SWE-Bench Lite - a benchmark used to test how well the AI model can solve Github issues, with the Lite version being a slightly easier benchmark than the original SWE-Bench.

Salesforce also used a similar approach. (Source)

DeepSeek Advances Automated Math Reasoning

DeepSeek-Prover-V1.5 framework overview. (Source)

DeepSeek-Prover-V1.5 is an open-source language model designed for theorem proving in Lean 4, since automating complex mathematical reasoning is a tricky area for language models to handle. This model is a step up from its predecessor by optimizing both training and inference processes, aiming to improve performance on formal theorem proving tasks.

Some of the key components include:

• Large-scale mathematical pre-training

• Formal mathematics corpus construction and augmentation

• Online reinforcement learning from proof assistant feedback

• Novel Monte-Carlo tree search methodology for long-term planning in theorem proving 

DeepSeek-Prover-V1.5 uses a combination of supervised fine-tuning, reinforcement learning, and a new variant of Monte-Carlo tree search called RMaxTS, which employs an intrinsic-reward-driven exploration strategy to generate diverse proof paths.

Results show significant improvements over DeepSeek-Prover-V1, achieving SOTA results on the test set of the high school level miniF2F benchmark (63.5%) and the undergraduate level ProofNet benchmark (25.3%).

Efficient Upcycling of Dense Models Into MoE With BAM

The three phases of BAM. (Source)

BAM (Branch-Attend-Mix) is a new approach to efficiently upcycle pre-trained dense models into Mixture of Experts (MoE) models. Initializing MoEs is tough since they’re computationally expensive to train from scratch, so BAM makes full use of specialized dense models' parameters to tackle this problem.

BAM operates in three phases: 

1. Branching (creating copies of a pre-trained seed model)

2. Continued pre-training (specializing each copy on different domains)

3. Mixture model training (initializing MoE layers using the specialized models

It introduces a soft-variant of Mixture of Attention (MoA) layers and employs a parallel attention transformer architecture to improve efficiency.

BAM consistently outperforms baseline methods in both perplexity and downstream task performance across various domains, with experiments conducted on seed models ranging from 590 million to 2 billion parameters. It’s a big step forward in MoE initialization, so we might see more efficient training of large-scale language models with superior performance in the future.

Frameworks We Love

Some frameworks that caught our attention include:

• DreamCinema: Simplifies film creation by using generative AI to automate the production of 3D characters and cinematic elements

• RuleAlign: Enhances LLMs like GPT-4 for medical diagnostics by aligning them with specific diagnostic rules

• PCGRL+: Designed to train AI agents to generate game levels based on specific quality metrics.

• SigLLM: Uses large language models for time series anomaly detection by converting time series data to text

• MVInpainter: Reformulates 3D editing as a multi-view 2D inpainting task, enabling novel view synthesis and editing for in-the-wild scenes without relying on explicit camera poses. 

• RAGchecker: comprehensive evaluation framework for Retrieval-Augmented Generation (RAG) systems that incorporates diagnostic metrics for both retrieval and generation modules. 

If you want your framework to be featured here, reply to this email and say hi :) 

Conversations We Loved

Wolfram’s highly detailed discussion about his perspective on ML took the spotlight. Although there was a big focus on the theoretical aspects, it’s still useful to consider as it can provide a different perspective for practical applications of ML.

While the strange Strawberry account was gaining all that attention, there was a pretty notable advancement that slipped under the radar with rStar. Moreover, Y Combinator CEO let us AI startups in on a little secret on how they can quickly build trust with customers using golden magic demos.

From Randomness to Intelligence: Wolfram's New Perspective on ML

Wolfram’s discussion about machine learning. (Source)

A recent blog post by Stephen Wolfram caught our attention this week, offering intriguing insights into the fundamental workings of machine learning systems. Wolfram explores minimal models that capture the essence of machine learning, stripping away complexities to reveal core principles.

Wolfram introduces simple, visualizable models like "rule arrays" that can perform machine learning tasks. These models suggest that machine learning works by "sampling" from the computational universe rather than building structured mechanisms. 

He also argues that the power of machine learning comes from leveraging computational irreducibility as a "natural resource." Moreover, Wolfram mentions that we’re at a point where we can achieve notable results with machine learning techniques like neural networks, but we don’t truly understand how we’re able to get such results. 

The Overlooked AI Reasoning Breakthrough Amid Strawberry Hype

Tekparmak brought light to an overlooked advancement. (Source)

A thread by Tekparmak caught our attention amidst all the Strawberry hype discussing a new approach called rStar. This method uses a generator LLM to create solution trajectories and a discriminator LLM for "peer review," reminiscent of the GAN architecture.

rStar has demonstrated superior performance among multi-round self-improving approaches for AI reasoning tasks. This means that compared to other methods that use multiple iterations or rounds to improve their performance, rStar is currently achieving the best results.

Interestingly, base model generators perform well with instruct discriminators, and the gap between GPT-4 and smaller open-weight models like Phi-3 mini is not as significant as one might expect.

While breakthroughs like OpenAI's Project Strawberry generate buzz, there's still ample room for innovative approaches using existing models and techniques.

How AI Startups Can Win With Powerful Demos

Tan’s discussion about the power of a golden magic demo. (Source) 

Y Combinator CEO Garry Tan raised an interesting point about just how important a golden magic demo is for AI startups. But why is that?

He mentions that a golden magic demo shows the benefit and value of the product right away, indicating how they could accomplish days worth of work in just 10 minutes. Repetitive tasks are a massive pain point customers face, so showing how AI can automate these tasks and boost productivity is a great way for startups to immediately show their solution can solve an important problem.

Casetext was used to highlight this point, with the example of AI being able to quickly detect nuances in emails for lawyers to use as evidence for potential fraud. Their golden magic demo showed how lawyers could save a ton of time with AI, so they could quickly see the power of Casetext’s solution by the end of the demo.

Tan also mentioned that a successful golden magic demo is something that Y Combinator sees in a lot of successful LLM-based startups, which makes it pretty encouraging for other startups to do the same.

Money Moving in AI

We saw the success of three funding rounds for Story, Cursor, and Defcon - all of which were under $100 million. Interestingly, each company focuses on pretty different applications in AI: Story in blockchain, Cursor in coding, and Defcon in military logistics.

Okpey and Elise AI also had successful funding rounds, with Opkey raising $47 billion and Elise AI raising $75 million. On the other hand, AMD decided to challenge Nvidia’s chip dominance by acquiring ZT systems for $5 billion.

AMD Acquires ZT Systems for $5 Billion to Challenge Nvidia

AMD has made a bold move in the AI chip market by agreeing to acquire ZT Systems, a New Jersey-based server maker, for nearly $5 billion in cash and stock. This acquisition, which combines AMD's silicon and software capabilities with ZT's systems expertise, aims to accelerate the deployment of AMD-optimized data center AI solutions at scale for cloud and enterprise customers.

Story Raises $80 Million in Series B Funding Round

Story, a startup building a blockchain-based platform for IP tracking and monetization in the age of AI, has secured $80 million in Series B funding led by Andreessen Horowitz's crypto division, with participation from Polychain Capital and other notable investors. The round values Story at $2.25 billion post-money and brings its total funding to $143 million. 

Elise AI Raises $75 Million in Series D Funding

EliseAI, a startup developing AI-powered property management tools, has secured a $75 million Series D funding round, valuing the company at $1 billion. Led by Sapphire Ventures with participation from Point72 Private Investments, Divco West, Navitas Capital, and Koch Real Estate Investments, this investment brings EliseAI's total funding to $140 million.

Cursor Secures $60 Million in Series A Funding

Cursor, an AI-powered coding tool startup, has raised $60 million in Series A funding from prominent investors including Andreessen Horowitz, Thrive Capital, OpenAI, and notable tech founders. The company, which aims to create a "magical tool" for writing the world's software, has grown to over 30,000 customers across major enterprises, research labs, and startups. 

Cursor has even started to go viral on X off a video showing how an 8 year old can use it to start coding.

Cursor is making coding more accessible. (Source

Opkey Raises $47 Million in Series B Funding

Opkey, an AI-powered continuous test automation platform for enterprise systems, has secured $47 million in Series B funding led by PeakSpan Capital, with participation from existing investors. The funding will fuel Opkey's mission to  accelerate product development and expand its global market presence.