The last few weeks have been packed with exciting releases, and in this edition, we’ll dive into the major updates, along with a short thought piece on a persistent narrative: have we really “run out of data,” and is AI progress slowing down as a result?

Happy Holidays!

Sasha Krecinic

Is AI progress starting to 'slow down' because we have 'consumed' all of the data?

Short Answer: No.

Long Answer: AI's momentum is parallelized now, meaning there are several viable pathways to explore, both on the research and scaling side. Yes, some models are huge and underperform expectations, and some models aren't released due to competitive concerns or safety and alignment concerns. However, it isn’t wise to judge progress based on headlines or one variable like ‘total publicly available data’.

Unfortunately, the headlines have focused on the fact industry figures like Ilya Sutskever and other researchers have stated that the era of pre-training is coming to a close. However, the best way to track the frontier is by tracking the research developments. What they might not mention is Ilya also recently said, "Scaling the right thing matters more now than ever.” What he is drawing upon is that there are several different doors to explore, and some might be dead ends, hence the benefit and focus on parallelization by the major AI labs. Both Sam Altman and Dario Amodei stated they have a clear line of sight on where to build for the next 18-24 months. Beyond that, it is arguably hard to plan because the frontier is moving so quickly.

Why are people saying this?
These comments are usually taken out of context. They often reflect one small part of the picture and, sadly, can be quite misleading at times. Some of the biggest developments have occurred in the last two months and even came sooner than many in the field expected. Here are a few examples:
- Test time training/compute
- Real-time voice APIs / Live AI screen vision  
- AI Computer Use

Each of these has the potential to transform industries and the overall surface area AI is touching here is expanding much faster than the data is being ‘consumed’. The thing that stumps most people in this industry is how little coverage these developments have gotten. So when someone says AI progress is "losing steam," ask them what they think about the research pathways and how quickly AI’s surface area is expanding…

[Why AI is not losing steam...] Share this story by email

OpenAI’s 12 days of product ship-mass

In case you missed it, OpenAI’s 12 Days of back-to-back releases (so far):

• Day 1: o1 & ChatGPT Pro (Premium features with $20-$200 plans)

• Day 2: Reinforcement Fine-Tuning Program (Research applications open)

• Day 3: Sora (Video generation and remixing)

• Day 4: Canvas (Collaborative coding and writing)

• Day 5: ChatGPT in Apple Intelligence (Ecosystem integration)

• Day 6: Advanced Voice & Santa Mode (Voice+video and festive fun)

• Day 7: Projects in ChatGPT (Organize and manage projects)

• Day 8: ChatGPT Search (Real-time web answers with links)

Some of these were expected, and others have been a complete surprise to me, like the Reinforcement Fine-Tuning Program. It’s a big development because it lets developers shape model behavior through iterative feedback loops rather than being stuck with static datasets. Traditional fine-tuning methods rely on adjusting model parameters with fixed training examples, but RL FT incorporates evaluative signals—like user feedback or predefined reward criteria—directly into the training process. This makes it possible to optimize a model’s responses toward desired outcomes more dynamically, improving its ability to handle complex tasks, follow specific instructions, and maintain quality and alignment over time!

[12 Days of OpenAI ] Share this story by email

Google Launches Gemini 2.0 AI Model

Google has launched Gemini 2.0, described as an “AI model for the agentic era,” according to their recent blog post. The model offers advanced multimodal capabilities and native tool use (e.g., search), with an experimental version called Gemini 2.0 Flash now available to developers, reportedly doubling the speed of its predecessor. Potentially most impressive is that Google Studio introduces a screen-sharing capability that allows users to have a video call with it and receive live assistance with anything on their screen in real time. Google also announced it is testing browser control for tasks like collecting contact information on web pages from a chat-based control panel.

[Introducing Gemini 2.0: our new AI model for the agentic era] Share this story by email

World Labs: Spatial intelligence is on the horizon

A big development in spatial intelligence, World Labs has introduced an AI system capable of generating interactive 3D worlds from a single 2D image. The company raised $230M in September 2024 and is already showcasing impressive demos. Unlike conventional tools that produce static visuals, this new technology allows users to fully explore scenes, peering around corners and examining details in real time. Early demos show how the tool can transform creative workflows for artists, filmmakers, and game developers, offering unprecedented control and fidelity in digital environments. Check out the impressive demo here:

[3D AI worlds coming soon] Share this story by email

Liquid AI secures $250M to expand AI model infrastructure

Liquid AI has raised $250 million in a Series A round led by AMD Ventures to advance its Liquid Foundation Models, which are lightweight and general-purpose. The company says it will use the funds to enhance its computing infrastructure and expedite product readiness for edge and on-premise applications, fine-tuning, and is looking to bring their solution to a broader audience.

[We raised $250M to scale capable and efficient general-purpose AI] Share this story by email

You may notice a slight change in this week’s edition of our newsletter. We are excited to announce the official relaunch of our newsletter under a new name: "Headline Edit." And what an action-packed week to introduce it! Our mission remains the same: distill the week's biggest stories in AI —just with a fresh new look.

This week, we shine a spotlight on a critical challenge in AI: reasoning as a major bottleneck for current large language models (LLMs). We delve into how recent advancements aim to overcome this hurdle, including OpenAI's latest efforts to enhance reasoning capabilities in their models. We also explore Magic's remarkable 100 million token context window, Google's GameNGen potentially redefining real-time gaming through cutting-edge AI, and impressive updates in the AI software engineering tooling space from Replit, Cursor, and GitHub's Copilot. With context windows now reaching unprecedented lengths and code generation tools becoming increasingly capable, these developments are converging to significantly accelerate software development and enhance AI's reasoning abilities across industries. Read on and strap in because it’s going to be an exciting couple of years ahead!

— Sasha Krecinic

OpenAI launches Strawberry aka ChatGPT-o1 and reveals even more…

OpenAI's new o1 series represents a massive breakthrough in AI capabilities, designed to handle complex, multi-step tasks in fields like science, math, and coding. What makes o1 stand out is its ability to "think before it responds," using a chain-of-thought approach to solve problems more effectively than previous models. This is a major unlock for large language models (LLMs), which traditionally excel at generating text but struggle with deeper reasoning.

In tests, o1 vastly outperforms both GPT-4o and human experts. For example, in the USA Math Olympiad (AIME), GPT-4o solved just 13% of problems, while o1 scored 83%, placing it among the top 500 students nationally. On PhD-level science benchmarks in physics, chemistry, and biology, o1 not only outperformed GPT-4o but also surpassed human PhD experts, becoming the first model to do so. In coding, o1 ranks in the 89th percentile in Codeforces challenges, far exceeding GPT-4o's 11th percentile performance.

Source: OpenAI - Learning to Reason with LLMs

Why Reasoning is a Big Unlock for LLMs

Reasoning is the next frontier for LLMs, enabling them to tackle more complex, real-world problems that require logic, planning, and decision-making over time. While previous models like GPT-4o were great at generating coherent responses, they lacked the depth to break down intricate problems, reconsider approaches, and learn from mistakes in real-time. o1 changes this by leveraging reinforcement learning to refine its thinking process—similar to how humans problem-solve by rethinking steps and adapting strategies when things don’t work.

This unlock means LLMs like o1 can be used in more sophisticated scenarios, from annotating genomic data in healthcare research to solving complicated quantum physics equations or optimizing complex code workflows. With reasoning, LLMs transition from being assistants that generate text to tools capable of deep analytical tasks, allowing them to rival human experts in specialized domains.

o1-preview and the more efficient o1-mini are now available in ChatGPT, with plans for further updates to enhance these reasoning models’ functionality across broader applications.[OpenAI Launches O1 Model Series] Share this story by email

Google launches GameNGen, a groundbreaking game engine using AI

In a significant development, Google has unveiled GameNGen, a groundbreaking game engine that leverages advanced neural models to recreate classic games like DOOM in real-time. This innovative technology allows for high-quality, interactive gameplay at over 20 frames per second on a single TPU. GameNGen's unique training process involves a reinforcement learning agent that learns to play the game, followed by a diffusion model that predicts the next frame based on previous actions. This could revolutionize gaming by seamlessly blending real and simulated experiences. We highly recommend checking out the demo which is almost indistinguishable from the original game. [GameNGen] Share this story by email

Magic LTM-2-Mini launches with unprecedented 100 million token capacity

In a significant advancement, Magic's LTM-2-Mini model now supports a remarkable 100 million tokens, equivalent to 10 million lines of code or 750 novels. CEO Eric Steinberger also announced a $320 million funding round aimed at realizing their vision of autonomous AI while acknowledging the considerable challenges ahead.

The best example of why this is relevant is arguably in software development. Imagine a developer working with a massive codebase that includes millions of lines of code, along with various libraries and documentation. A traditional AI SWE might struggle to keep track of all this information, leading to mistakes, poor troubleshooting, or incomplete solutions. However, Magic's AI, with its ability to handle ultra-long contexts, can keep all of this information in mind at once. This means it can accurately suggest code improvements, debug complex issues, or even generate new code by understanding the entire context. For instance, if the AI needs to fix a bug, it can consider the entire codebase, identify the exact spot where the issue occurs, and suggest a precise fix, all without losing track of other important details. This makes the AI a powerful tool for developers working on large, complex projects. [100M Token Context Windows] Share this story by email

Study shows that generative AI tools boost software developer productivity by over 26%

A recent study indicates that GitHub Copilot can boost software developer productivity by over 26%, particularly benefiting less experienced coders. However, a significant portion of developers—30 to 40 percent—have yet to try the AI tool, underscoring the need to consider individual preferences in workplace technology adoption. The study reveals that the productivity gains from GitHub Copilot are particularly pronounced among less experienced developers, with these users achieving a remarkable 39% increase in output compared to their more seasoned counterparts, who only see marginal improvements. Interestingly, despite the clear benefits of using Copilot, around 30-40% of developers in the experiments chose not to adopt the tool, indicating that factors such as personal preferences and perceived utility play a crucial role in technology adoption within the workplace. [ssrn.com] Share this story by email

Replit launches Replit Agent to simplify software development tasks

Replit has launched Replit Agent in early access, which aims to automate software development. Users are praising its ability to simplify the setup and deployment of applications, potentially transforming how software is created and launched. The Replit Agent not only streamlines the coding process but also integrates fully featured development and production environments, allowing users to build and deploy applications seamlessly from a single platform. Early testers are already expressing excitement about the potential of Replit Agent to enable the creation of entire web apps directly from mobile devices, marking a significant leap in accessibility for developers on the go. [via @amasad] Share this story by email

Cursor also accelerates app development, replicating complex software in minutes

The main difference between Replit and Cursor is that Replit is a cloud-based, collaborative coding environment, ideal for quick prototyping and small projects without the need for local setup. Cursor, on the other hand, is a local code editor designed for more complex, long-term projects that require scalability, better performance, and deeper integration with your machine's development environment. One Cursor user has achieved a significant milestone by creating a Perplexity clone in just eight minutes, reducing app development time to under 14 interactions. This innovation excites users, as features that previously required significant resources to develop can now potentially be completed in a fraction of the time. [via @mckaywrigley] Share this story by email

__

The views and opinions expressed in this newsletter are those of the individual authors and do not necessarily reflect the official policy or position of Headline. All content is intended for informational purposes only and should not be construed as professional advice. Headline disclaims any responsibility for the accuracy, completeness, or reliability of the information presented.

The research community is closely monitoring the advancements needed to unlock AGI (Artificial General Intelligence). Today, we may be one step closer with the latest research from Sakana.ai, which outlines how they have automated the scientific research lifecycle. Additionally, we see notable progress in autonomous software engineering AI agents, demonstrated by the team at Cosine. Meanwhile, the latest chatbot rankings highlight a highly competitive landscape in terms of speed, price, and capability.

We’ve also launched a podcast covering some of the breaking news stories. If you like to consume content in video format, subscribe to our YouTube channel! Check out this week’s episode where we cover OpenAI Strawberry, Google’s Pingpong robot, and the pricing cuts from OpenAI and Google.

— Sasha Krecinic

Sakana AI launches The AI Scientist for automated scientific research

Sakana AI has launched The AI Scientist, an AI system that automates scientific research and claims it can manage the entire machine learning research lifecycle. Developed with researchers from the University of Oxford and the University of British Columbia, the system has produced papers in areas such as language modeling and diffusion and is open-sourced. Many in the research community see this as a potential flywheel for AI systems to unlock the ability for self-improvement. [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery]

Cosine AI claims to have built the most capable AI software engineer

Cosine AI says it has built an AI software engineer that scored 30.08% on the SWE-Bench benchmark, surpassing Amazon and Cognition. The model is designed to mimic human software engineering behavior and claims to perform at 50% on the SWE-Lite benchmark, with potential for future challenges. [via @AlistairPullen]

GPT-4o reclaims top spot and Grok 2 ranks #3 on Chatbot Arena leaderboard

OpenAI's ChatGPT-4o has taken the lead in the Chatbot Arena with a score of 1314, surpassing Google's Gemini-1.5-Pro-Exp after over 11,000 community votes under the masked title of "anonymous chatbot". The model ranks highly in Math, Coding, and Instruction-Following, with a notable improvement in coding, scoring over 30 points higher than its predecessor. Meanwhile, Grok 2 was a recent addition and reached third place and achieved #2 in Coding, #2 in Math, and #4 in Hard Prompts. Super impressive results from the newcomer! [lmsys.org]

Anthropic Unveils Prompt Caching: 90% Cost Savings, 85% Faster AI Responses

Anthropic has introduced a prompt caching feature for its Claude AI models, now available in public beta, which allows developers to cache frequently used contexts between API calls. This feature reduces costs by up to 90% and improves latency by up to 85%, making it highly efficient for use cases like conversational agents, coding assistants, and large document processing. The pricing model involves a slightly higher cost for caching inputs but offers significantly cheaper access to cached content, with early adopters like Notion already seeing substantial benefits. [Prompt caching with Claude]

Gemini Makes Your Mobile Device a Powerful AI Assistant

Google DeepMind has introduced "Gemini Live," a new feature that enhances conversational AI interactions on Android devices. Available to Gemini Advanced subscribers, Gemini Live allows for more natural conversations with the ability to brainstorm ideas, interrupt to ask questions, and pause chats to resume later. This feature is now rolling out in English, making mobile AI interactions smoother and more responsive, with significant potential for accessibility and productivity improvements. [Gemini makes your mobile device a powerful AI assistant]

Google Gemini 1.5 Pro is making waves as it outperforms GPT-4o and Claude-3.5 in the LMSYS Chatbot Arena. Many predicted it was only a matter of time before Google caught up, and it's the first time we've seen them take the lead, somewhat reversing sentiments that Google was getting 'left behind'. We also see hints of a broader strategy towards higher personalization with Google's partnership with Character.AI, known for its expertise in creating advanced conversational models and personalized AI interactions. In other news, Groq has announced new funding, and Argentina is pursuing AI applications reminiscent of sci-fi, drawing comparisons to the 2002 movie Minority Report.

Google Gemini 1.5 Pro tops Chatbot Arena

Google's Gemini 1.5 Pro (Experimental Version 0801) has claimed the top spot in Chatbot Arena, surpassing GPT-4o and Claude-3.5 with a score of 1300. The score of 1300 for Google's Gemini 1.5 Pro in Chatbot Arena reflects its ELO rating, a ranking model used in Chess, and indicates a win percentage of 54% against GPT-4o and 59% against Claude-3.5 Sonnet, showcasing its superior performance in head-to-head comparisons. It has shown that it excels in multilingual tasks and technical areas like Math, Instruction-Following, and Coding, though it trails in domains like Coding and Hard Prompts compared to Claude 3.5 Sonnet and GPT-4o. Google Cloud says this experimental version is now available for early testing and feedback in Google AI Studio and the Gemini API. [For the first time, Google Gemini has claimed the #1 spot, surpassing GPT-4o/Claude-3.5 with an impressive score of 1300 (!), and also achieving #1 on our Vision Leaderboard.]

Character.AI Gets Funding from Google and Co-Founders Return to Google

Noam Shazeer and Daniel De Freitas, co-founders of Character.AI, are returning to Google, with Shazeer joining the DeepMind research team, while Character.AI’s general counsel Dominic Perella will serve as interim CEO. Character.ai is a platform where users can create and interact with AI-driven virtual characters that adapt to individual inputs, tailoring their responses and behaviors for personalized, engaging interactions across various applications. According to a TechCrunch interview, Google has signed a non-exclusive agreement to use Character.AI’s technology, which Shazeer says will provide funding for Character.AI’s continued growth and focus on building personalized AI products. [Title: Exclusive: Character.AI CEO Noam Shazeer returns to Google as the tech giant invests in the AI company]

Groq raises $640 million, valuation hits $2.8 billion

Groq has raised $640 million in Series D funding, led by Cisco Investments, Samsung Catalyst Fund, and BlackRock Private Equity Partners, bringing its valuation to $2.8 billion. Groq's chips are designed with a unique architecture called Tensor Streaming Processor (TSP). This architecture processes data in a highly parallel manner, allowing it to handle multiple tasks simultaneously with the benefit of both speed and efficiency. Unlike traditional GPUs that are designed for a wide range of tasks, Groq's TSP is specifically optimized for AI workloads, making it exceptionally fast and energy-efficient for inference and tasks like running large language models. The company says its chips, which now run Meta Platforms' LLaMA, are four times faster, five times cheaper, and three times more energy-efficient than Nvidia's GPUs for AI inference tasks. [AI chip startup Groq valued at $2.8 bln after latest funding round]

Argentina's President Launches AI Security Unit

Argentina's President Javier Milei has launched the AI Applied to Security Unit, raising concerns about potential human rights violations. The Ministry of Security says the unit will use AI to predict crimes and monitor social media, while human rights groups fear it could lead to over-surveillance and profiling. [Argentina will use AI to ‘predict future crimes’ but experts worry for citizens’ rights]

This week, we saw two new products launch that influence search and video processing capabilities. OpenAI's Search product has finally been released in closed beta after many months of rumors, and Meta's SAM 2 (Segment Anything Model) promises to redefine real-time object segmentation and tracking in images and videos. 

– Sasha Krecinic

OpenAI launches SearchGPT prototype

OpenAI is testing a new prototype called SearchGPT, designed to provide fast and timely answers with clear and relevant sources. According to a blog post, the prototype is being launched with a small group of users for feedback, with plans to integrate it into ChatGPT to enhance real-time search capabilities. The search product was launched a few weeks after their acquisition of RockSet. Rumors were also swirling for several weeks that OpenAI would launch a search product. A notable OpenAI employee, Noam Brown, also commented that this is "another step toward general AI personal assistants for all." [openai.com] Share this story by email

Meta's Llama-3.1-405B Ranks Third in AI Leaderboard

Meta's Llama-3.1-405B has reached #3 on the Overall Arena leaderboard, marking the first time an open model has made the top 3. The model was tested over the past week, receiving over 10K community votes. [lmsys.org] Share this story by email

Meta launches SAM 2 for real-time object segmentation

Meta has launched the Segment Anything Model 2 (SAM 2), a groundbreaking unified model for real-time, promptable object segmentation in both images and videos. Following the success of SAM, SAM 2 offers state-of-the-art performance and introduces a novel "memory attention" feature that uses a transformer with memory across frames. It stores special "object pointer" tokens in a "memory bank" FIFO queue of recent and prompted frames. SAM 2 can segment any object in any video or image, even those it has not seen before, enabling a diverse range of use cases without custom adaptation. It is open source under the Apache 2.0 license and includes the SA-V dataset, containing approximately 51,000 real-world videos and more than 600,000 masklets. Meta provides a web demo, research paper, and datasets that are worth checking out! [fb.me] Share this story by email

It has been another big week in AI. Most notably, there is a paradigm shift in the competition between open and closed-source models. Meta's latest release, Llama 3.1 405B, redefines open-source real-time AI performance with enhanced reasoning and multimodal capabilities. This new version is state-of-the-art for the open-source community, radically improving AI toolkits for AI startups and developers. But that's not all—today's edition also covers the broader implications of advancing AI with the results of the Sam Altman-backed UBI (universal basic income) study being released. OpenAI has also introduced GPT-4o mini, which is reportedly smarter and 60% cheaper than GPT-3.5 Turbo. Additionally, OpenAI has launched free fine-tuning for GPT-4o mini until September. We also saw that Daily released an open standard for Real-time Voice and Video Inference (RTVI-AI). These developments are significant because they make cutting-edge AI technology more accessible and affordable along with the potential implications of UBI. Based on today’s updates, it isn't crazy to imagine a world where your next dentist's appointment is booked by speaking to an AI agent. People could work less and potentially have a steady stream of money coming in each month.

Meta launches SOTA Open Source Model

Meta has introduced Llama 3.1, a new set of foundation models designed to rival leading closed-source models in various tasks. These models, including a 405 billion parameter version, boast enhanced reasoning capabilities and a larger 128,000-token context window, along with multimodal features for image and video processing. Key to the model's size and performance are its improved data quality and scale, with training conducted on a diverse and high-quality dataset of 15 trillion multilingual tokens. Meta has made these models publicly available, including both pre-trained and post-trained versions, to foster innovation in the research community and promote the responsible development of artificial general intelligence (AGI). [fb.me] Share this story by email

OpenAI Launches GPT-4o Mini with Free Training Tokens

OpenAI has launched GPT-4o mini, which they say is smarter and 60% cheaper than GPT-3.5 Turbo. OpenAI has also launched fine-tuning for GPT-4o mini. GPT-4o mini excels in reasoning, math, coding, and multimodal tasks, outperforming GPT-3.5 Turbo and other small models on several key benchmarks. OpenAI also mentioned in another release that they will be offering free fine-tuning for the model with the first 2 million training tokens per day are free until September 23! These super small and cheaper models are important for highly repetitive tasks that don’t need a larger or more expensive model (and because they use significantly less power, they are much better for the the wallet and the planet!) [openai.com] Share this story by email

Daily Launches Open Standard for Real-time Voice and Video AI

Daily has launched an open standard for Real-time Voice and Video Inference (RTVI-AI) along with open-source JavaScript and React SDKs, with iOS and Android SDKs coming soon. According to the release, RTVI-AI defines how client applications communicate with inference services, enabling use cases like voice chat with LLMs, enterprise voice workflows, video avatars, voice-driven user interfaces, and high-framerate image generation. The demo leverages Llama 3.1 running on @GroqInc and has impressive 500ms voice-to-voice response times (which is comparable to real-life conversations!) and shows how far the frontier of tech for live voice agents has come in a short time. [github.com] Share this story by email

UBI Study Findings Released

A study by OpenResearch with backing from Sam Altman examined the effects of giving $1,000 per month to low-income individuals. The study explored the impact on spending, agency, employment, health, and moving. The summary findings stated that: "The program resulted in a 2.0 percentage point decrease in labor market participation for participants and a 1.3-1.4 hour per week reduction in labor hours, with participants’ partners reducing their hours worked by a comparable amount. The transfer generated the largest increases in time spent on leisure, as well as smaller increases in time spent in other activities such as transportation and finances. Despite asking detailed questions about amenities, we find no impact on quality of employment, and our confidence intervals can rule out even small improvements. We observe no significant effects on investments in human capital, though younger participants may pursue more formal education. Overall, our results suggest a moderate labor supply effect that does not appear offset by other productive activities." [openresearchlab.org] Share this story by email

This week's AI developments make up for a slow week last week. We see notable commentary, research, and applications. First, OpenAI is allegedly making strides towards Level 2 AI, which they have labeled as "Reasoners," and defined as performing human-level problem-solving tasks. Research on Test-Time Training (TTT) layers shows that models can adapt and improve in real-time, potentially outperforming traditional models in long-context tasks. Skild AI's recent funding underscores the buy-in and investment in generalist AI models to drive robotics forward, and finally, Meta's MobileLLM models demonstrate efficient and capable, on-device AI solutions that address limitations in mobile technology. Happy reading!

--Sasha Krecinic

OpenAI "On the Cusp" of Level 2 AGI

According to a recent Bloomberg article, OpenAI executives have said the company is “on the cusp” of Level 2 AGI in its five-tier AI progress tracking system. It is also rumored that OpenAI demonstrated GPT-4 with improved reasoning capabilities at a recent all-hands meeting. Level 2, involves AI systems performing problem-solving tasks at the level of a human with a doctorate-level education without using any tools. The scale also includes the following stages of artificial intelligence: 

Level 1: Chatbots - AI with conversational language abilities 

Level 2: Reasoners - AI with human-level problem-solving capabilities 

Level 3: Agents - Systems that can take actions 

Level 4: Innovators - AI that can aid in invention 

Level 5: Organizations - AI that can perform the work of an entire organization 

[OpenAI Scale Ranks Progress Toward ‘Human-Level’ Problem Solving The company believes its technology is approaching the second level of five on the path to artificial general intelligence] Share this story by email

OpenAI Researcher Comments That Company Focus is Still on Ambitious Research

OpenAI researcher Noam Brown, a specialist in AI reasoning, tweeted, "When I joined @OpenAI a year ago, I feared ChatGPT's success might shift focus from long-term research to incremental product tweaks. But it quickly became clear that wasn't the case. @OpenAI excels at placing big bets on ambitious research directions driven by strong conviction. They remain committed to ambitious research despite the success of ChatGPT." This comment emphasizes that OpenAI continues to prioritize long-term research over incremental product tweaks. It also suggests that they are not solely measuring themselves against existing benchmarks but are focused on paradigm-shifting developments through research, such as Noam Brown's work. This stance contrasts with the messaging of the former alignment team, who recently departed OpenAI. [via @polynoamial] Share this story by email

TTT Layers Match Performance of Transformers and Mamba RNNs

According to a recent research paper, Test-Time Training (TTT) layers match or exceed the performance of strong Transformers and Mamba RNNs in long-context tasks. RNNs stands for Recurrent Neural Networks. They are a type of artificial neural network designed to recognize patterns in sequences of data, such as text, genomes, handwriting, or numerical time series data. TTT is a method where a model continues to update its parameters during the testing phase using self-supervised learning, allowing it to adapt to new data in real-time and improve performance dynamically. The paper highlights that TTT-Linear is faster than Transformers at 8k context and matches Mamba RNNs in wall-clock time. Wall-clock time refers to the actual elapsed time it takes to complete a task, as opposed to the number of operations or computational steps, and is crucial for evaluating the real-world efficiency of algorithms. Despite facing memory I/O challenges, TTT-MLP shows significant potential. TTT layers uniquely update their hidden state through self-supervised learning even during test sequences, making them highly adaptive and efficient for long-context tasks. This innovative approach of using a machine learning model as the hidden state allows TTT-Linear to outperform Transformers in speed at 8k context, demonstrating its potential for scalable applications. [GitHub - test-time-training/ttt-lm-pytorch: Official PyTorch implementation of Learning to (Learn at Test Time): RNNs with Expressive Hidden States] Share this story by email

Skild AI Raises $300 Million in Series A Funding To Develop Universal Robots

Pittsburgh-based robotics startup Skild AI has raised $300 million at a $1.5 billion valuation in a Series A funding round led by Lightspeed Ventures, Softbank, Coatue, and Jeff Bezos. Skild AI's models enable robots to perform tasks in unfamiliar environments, such as climbing stairs and recovering objects that slip out of hand. The robots demonstrated emergent capabilities, showcasing abilities they weren't explicitly taught. The AI model was trained on a database 1,000 times larger than those used by competitors, using diverse data collection techniques. According to the company's press release: "Skild’s model serves as a shared, general-purpose brain for a diverse embodiment of robots, scenarios, and tasks, including manipulation, locomotion, and navigation. From resilient quadrupeds mastering adverse physical conditions to vision-based humanoids performing dexterous manipulation of objects for complex household and industrial tasks, the company’s model will enable the use of low-cost robots across a broad range of industries and applications." [This $1.5 Billion AI Company Is Building A ‘General Purpose Brain’ For Robots] Share this story by email

Meta's MobileLLM Models Boost On-Device Accuracy Without Increasing Size

Meta has introduced MobileLLM models designed for efficient on-device large language models (LLMs). With fewer than a billion parameters, these models perform competitively with larger models in specific tasks, showing significant improvements in chat benchmarks and API calling tasks. They use deep and thin architectures with embedding sharing and grouped-query attention mechanisms, enhancing accuracy without increasing model size. The practicality of these small models is highlighted for mobile devices, addressing concerns such as memory capacity and energy consumption, making them suitable for common on-device use cases. [MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases] Share this story by email

…AI: OpenAI Goes Shopping; Anthropic Edges Into First Place; and Microsoft's Tiny Vision Model (6.26.24)

This week’s respective models have received substantial infrastructure and performance enhancements. OpenAI’s acquisitions of Multi and Rockset are strategic moves to enhance its remote control and data retrieval capabilities, highlighting a broader industry shift towards more versatile and powerful agentic workflows. This really makes you wonder - not if, but when - they will release a search product. On the state-of-the-art (SOTA) front, we observed Anthropic’s enhancements with Claude 3.5 Sonnet and Microsoft’s debut of the Florence vision model, both of which represent significant advancements in AI model efficiency and effectiveness. Lastly, Ilya Sutskever, co-founder and former Chief Scientist at OpenAI, has news on his new project. Considering Ilya’s departure from OpenAI last month, this week’s headlines about acquisitions focused on 'Search' and 'remote control' potentially shed light on why he left OpenAI. Is it possible this direction did not align with the former Chief Scientist’s emphasis on creating safe AI?

-- Sasha Krecinic

OpenAI Acquires Rockset To Boost "Data Infrastructure" and "Retrieval" (aka Search)

OpenAI has acquired Rockset to enhance its data retrieval systems and plans to integrate Rockset's technology into its products to turn data into "useful insights". If you read some of Rockset's documentation you will get a sense of what the team is working to solve. It would not be crazy to imagine a world where they power search with this. It could be the new generation of ad targeting and personalized recommendations. The capability to deliver nuanced, context-aware insights and recommendations opens new revenue streams, enhancing OpenAI's competitive and commercial edge online and within agentic workflows. [OpenAI Acquires Rockset]

OpenAI Acquires Multi, A Company Powering Remote Computer Control

OpenAI has acquired Multi, a startup specializing in remote computer control, and announced that Multi will stop its services. Multi has closed new team signups, and existing users can access the app until July 24, 2024, after which all user data will be deleted. [via @itsandrewgao]

Anthropic's Claude 3.5 Sonnet Climbs Leaderboard

Anthropic says its Claude 3.5 Sonnet outperforms competitor models and Claude 3 Opus, operating at twice the speed and one-fifth the cost. Claude 3.5 Sonnet is their strongest vision model, surpassing Claude 3 Opus on standard vision benchmarks but still slightly behind OpenAI's offering. [Introducing Claude 3.5 Sonnet]

Microsoft Launches Florence-2 Vision Model

Microsoft has launched Florence-2, a vision model with 200M and 800M parameters, which they say matches the quality of models 100 times larger. Florence-2 is designed to handle a wide range of computer vision tasks, including captioning, object detection, and segmentation, using a unified, prompt-based representation. The model demonstrates strong zero-shot and fine-tuning performance, addressing challenges in spatial hierarchy and semantic granularity, and achieving state-of-the-art results on various vision benchmarks. [Florence - a microsoft Collection]

Ilya Sutskever Launches Safe Superintelligence Inc.

Safe Superintelligence Inc. says it is focused on creating the first safe superintelligence by addressing safety and capabilities together. Sutskever announced that they will focus on "one goal and one product" and also mentioned that they are hiring. [Safe Superintelligence Inc.]

This week's edition focuses on the AI/Nature parallels, and one of the hottest fields of research: neuromorphic computing (the method of computer engineering in which elements of a computer are modeled after systems in the human brain and nervous system). 

While nature isn't a perfect model, it provides some upper and lower bounds for us to benchmark what is theoretically possible. The human brain uses something like 25 watts of energy for cognitive functions per hour. A single GPT-4 query is estimated to use something like 300 watts. This chasm of consumption hints that nature has found a way to solve for 'compute' on a budget. Algorithmic optimization and neuromorphic computing, using nature as inspiration and a research catalyst, have the potential to solve our ever-increasing hunger for computing and dynamically trained models.

This week, we have some interesting developments on this front, raising an important question: "What if our assumptions about the computing requirements to train and run these models are wrong?" Researchers from the Shanghai AI Lab published a paper that sheds some light, achieving results from a tiny 8B parameter model that outperforms GPT-4 in math and coding. We also see a RAG methodology called HippoRAG that emulates aspects of the hippocampus in the human brain. Additionally, we see an LLM training methodology from the team at SakanaAI, who use large language models for discovering and optimizing new training algorithms, akin to natural evolution.

A final thought: Is this AI wave just hype? Have we “used up all the data for training”? Will we run out of compute? There is a lot of commentary like this saying that AI is “losing steam.” However, there are two different things that people often conflate. On the surface, this notion might appear to be true if you measure it by today's commercial applications (which still leverage legacy technology often). From a "frontier" research perspective (which I would posit is the rate-limiting step here), the developments we cover this month, in tandem with commentary from the founders I have spoken to, paint a very different picture. 

To paraphrase some sentiments from experienced founders, "The frontier is moving so quickly that you have to choose what to build very carefully so you are not made obsolete by the rapid movements at the frontier." Sadly, the investment and hype at the wrapper/surface layer are overshadowing the large strides that continue to be made in the research space. Tying this back to this week’s title: the research is how an 8B parameter model outperformed GPT-4.

Llama-3 Model with MCTS Outperforms GPT-4 in Math Tasks

A small 8B Llama-3 model combined with Monte Carlo Tree Search (MCTS) reportedly outperforms GPT-4 in complex mathematical reasoning tasks. The MCTS algorithm enhances mathematical reasoning by combining LLMs with Monte Carlo Tree Search (MCTS). It systematically explores and refines solutions using heuristic methods (practical techniques used to solve complex problems efficiently, relying on rules of thumb, educated guesses, and intuitive judgments to find satisfactory solutions quickly, even if they are not perfect or optimal). The algorithm builds a search tree by selecting, refining, evaluating options, and optimizing decisions using an enhanced Upper Confidence Bound (UCB) formula. Testing shows that MCTS significantly improves success rates on challenging math problems, advancing LLMs in complex tasks for more accurate and reliable AI-driven decision-making. [twitter.com]

Sakana AI Shares DiscoPOP - Evolution for LLMs by LLMs

By leveraging LLMs to automatically create and test new optimization algorithms, Sakana AI developed Discovered Preference Optimization (DiscoPOP), a novel method combining logistic and exponential losses. This approach, which showed state-of-the-art performance, marks a step towards using AI to advance AI, reducing human intervention and computational resources. The research highlights the potential of LLM-driven discovery to continuously improve AI models, opening new avenues for innovation and efficiency in AI development. [Sakana AI]

HippoRAG: RAG That Mimics Human Memory, Enhancing Speed and Efficiency

New research from Ohio State University introduces HippoRAG, a retrieval framework inspired by human memory, which they say outperforms existing methods by up to 20%, while being 10-30 times cheaper and 6-13 times faster. The study highlights HippoRAG's integration of LLMs, knowledge graphs, and the Personalized PageRank algorithm, demonstrating improvements in multi-hop question answering and single-step retrieval. HippoRAG mimics the human brain's memory processes by integrating large language models (LLMs) with knowledge graphs and the Personalized PageRank algorithm, similar to how the hippocampus and neocortex work together to store and integrate knowledge efficiently and effectively. [HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models ]

Greenblatt Achieves 50% Accuracy on ARC-AGI Test with GPT-4o

Ryan Greenblatt says he achieved 50% accuracy on the ARC-AGI public test set using GPT-4o, surpassing the previous state-of-the-art of 34%. He claims his solution reached 72% accuracy on a subset of the train set, compared to human performance of 85%, by using specialized few-shot prompts and better grid representations. [Getting 50% (SoTA) on ARC-AGI with GPT-4o]

DeepSeek-Coder-V2 surpasses GPT4-Turbo in Coding and Math

DeepSeek-Coder-V2, an open-source model, has reportedly outperformed GPT4-Turbo in coding and math, supporting 338 programming languages and extending context length to 128K. According to a paper posted on GitHub, it achieved 90.2% on HumanEval and 75.7% on MATH, surpassing GPT-4-Turbo-0409. [DeepSeek-Coder-V2/paper.pdf at main · deepseek-ai/DeepSeek-Coder-V2]

This week we had WWDC, and Apple and OpenAI announced the integration of ChatGPT with Siri on Apple devices. This move didn't surprise many, as OpenAI and Apple's talks have been an open secret for months. We see a great summary by Andrej Karpathy outlining what many are thinking and seeing on this. We also see new state-of-the-art (SOTA) models in translation, an open-source robotics offering, and a research paper on MoA (Mixture of Agents) and how the framework is pushing the frontier of AI capabilities yet again.

— Sasha Krecinic

ChatGPT Integration Announced for Apple Devices

Apple announced at WWDC 2024 that ChatGPT will be integrated into Siri and available for free in iOS 18 and macOS Sequoia later this year. This partnership with OpenAI aims to enhance Apple's AI features, making advanced AI accessible while maintaining a commitment to safety and innovation. Sam Altman also says he is excited about partnering with Apple to integrate ChatGPT into their devices later this year, which he believes users will greatly appreciate. The highlight of the show and the conference was the new Siri demo. [via @sama] Share this story by email

Andrej's Breakdown of ‘Apple Intelligence’

Andrej Karpathy praised Apple's Intelligence announcement, highlighting the integration of AI across the entire OS. He outlined key themes: enabling multimodal I/O, seamless inter-operation of OS and apps, and a frictionless user experience. He also emphasized the potential for proactive AI features, on-device intelligence, and modular support for various function calling while maintaining privacy with on-device computing. Zooming out, we agree that this is possibly the first step in a fully autonomous and highly personalized AI strategy and capable agent model, where computer vision takes on-screen data as visual context and potentially moves in the direction of Microsoft's Copilot/Recall feature by 'seeing' or recording on-screen activity. [via @karpathy] Share this story by email

Unbabel Launches TowerLLM and Reaches SOTA Translation Performance

Unbabel says it has launched TowerLLM, a new translation LLM that outperforms competitors like GPT-4, GPT-3.5, Google, and DeepL in accuracy and cost-efficiency. The company highlights that TowerLLM, built on billions of words of high-quality translation data, offers features such as source correction, and named entity recognition, and supports 18 language pairs across various domains. [Introducing TowerLLM, Multilingual by design Unbabel’s Generative AI model is the best performing machine translation on the market, enabling our customers to scale globally with lower costs and higher accuracy.] Share this story by email

Hugging Face Expands Locally Hosted AI App Offerings

Hugging Face has launched a second batch of local Generative AI apps, now available on compatible model pages. The company welcomes new additions to its community, a sentiment echoed by retweets from its CEO, Clement Delangue. [via @julien_c] Share this story by email

LeRobot Launches on PyTorch to Democratize Robotics

LeRobot, developed on PyTorch, has been launched on the Hugging Face community page to enhance accessibility in robotics using advanced AI tools and models. According to their press release, LeRobot provides pre-trained models, datasets, and simulation environments to facilitate learning complex tasks in robotics without the need for physical robot assembly. [via @RemiCadene] Share this story by email

Mixture-of-Agents Methodology Outperforms GPT-4 Omni

In a recent research paper, scientists introduced the Mixture-of-Agents methodology, which combines multiple LLMs to enhance language model performance, achieving a 65.1% score on AlpacaEval 2.0 and surpassing GPT-4 Omni's 57.5%. This method utilizes a layered architecture where each layer's LLM agents refine responses based on the previous layer's outputs, demonstrating improved performance using only open-source LLMs. [Mixture-of-Agents Enhances Large Language Model Capabilities] Share this story by email

This week's edition focuses on the training and progression of AI towards AGI. We also see activity by regulators for antitrust investigations, some new roles in Robotics advertised by the OpenAI team, and a new dataset for model training with detailed commentary. Short, sweet, and enlightening. Enjoy!

-- Sasha Krecinic

Why do we struggle to estimate AI Advancements?

Humans inherently struggle with understanding exponential relationships and growth. The wheat and chessboard problem illustrates this. The story goes that a king agrees to place one grain of rice/wheat on the first square of a chessboard, doubling it on each subsequent square. By the 64th square, the amount of wheat is huge, enough to bankrupt the kingdom. Similarly, during COVID-19, many couldn't grasp how quickly the virus could spread, leading to delayed responses and widespread impacts. We all had that friend who warned, “It’s coming, start preparing,” but most didn’t listen. 

It may not be obvious, but AI development is experiencing similar exponential growth and the information asymmetry is also increasing. To understand how it will evolve, you need to look at the sub-components, roadmaps, and rate-limiting steps in the hardware, software, data, people, and capital. There are those in the know who quietly acknowledge this exponential growth, aware of its potential, often keeping their ‘extreme views’ to themselves, once again, analogous to early COVID-19. 

Leopold Aschenbrenner's extensive analysis, "Situational Awareness," highlights the accelerating pace of AI capabilities across the various layers that drive innovation. From massive compute clusters to evolving AI models, the trajectory toward Artificial General Intelligence (AGI) is still a function of the sum of its parts. Leopold posits that AI advancements will continue to be exponential, driven by computing power, algorithmic efficiency, and new methodologies. Despite potential bottlenecks like data scarcity and unknowing challenges, the path to AGI is becoming clearer. Improvements in hardware, software, data, skills, and capital are compounding and driving transformative impacts that are often hiding in plain sight. 

While mainstream views often downplay AI ‘end game’ scenarios, a small group of experts is busily preparing for and predicting an ‘imminent’ (read this as within the next 3-5 years) AGI breakthrough. As an investor in this space, you see a range of people's reactions to this; unfortunately, some are still a little too skeptical, in my opinion. Naivety on this scale hasn’t served societies well historically. If we account for the consistent revision of AGI forecasts, it also points to a trend that AGI may be much closer than society thinks. Check out the full text if you'd like to see the detailed breakdown, I think it's worth a read! [Situational awareness] Share this story by email

Regulators Target AI Giants for Antitrust Probes

Federal regulators have agreed to initiate antitrust investigations into Microsoft, OpenAI, and Nvidia to examine their dominant positions in the AI industry, according to the New York Times, referencing two individuals familiar with the confidential discussions. The Justice Department will investigate Nvidia, while the FTC will focus on OpenAI and Microsoft, reflecting a broader initiative to address potential monopolistic practices in the AI sector. [U.S. Clears Way for Antitrust Inquiries of Nvidia, Microsoft and OpenAI] Share this story by email

OpenAI Recruits for Robotics

OpenAI is seeking a Research Engineer for its Robotics team in San Francisco to focus on training and fine-tuning large multimodal LLMs, as detailed in their job posting. The role involves collaborating with industry partners to enhance robotics applications and signals the first time they have made these sorts of moves since 2020. [OpenAI recruits for robotics talent] Share this story by email

FineWeb-Edu Sets New Dataset Standard

Guilherme Penedo from Hugging Face shared a report on the release of FineWeb and its educational subset, FineWeb-Edu, which incorporates 1.3 trillion tokens from a high-quality filtered Common Crawl dataset. According to the report, FineWeb-Edu outperforms all other publicly available web-scale datasets on benchmarks such as MMLU, ARC, and OpenBookQA, prompting a reassessment of the perceived quality of internet data. The post also received a lot of praise from industry leaders like Andrej Karpathy and Thomas Wolf, who noted it as potentially the best 45 minutes of reading you could do in the space if you want to understand how high-performing models work! Check out the full text if you're in this space! [via @Thom_Wolf] Share this story by email

—

This week's newsletter has a theme: the miniaturization of models while maintaining comparable results to larger models. This once again shows a potential path to locally hosted models. We have also intentionally left out all the drama/arguing happening on X between Elon and Yann, as it lacked substance. We also spotlight China's huge $47 billion USD investment in its semiconductor industry in an attempt to gain independence from US chip manufacturers and regulations.

— Sasha

Llama3-V Claims Superior Performance at Reduced Size

The team from AmbientGPT has released Llama3-V, a new AI model developed on the Llama3 platform. Llama3-V has been launched and shows a 10-20% improvement in benchmarks over Llama, the current state-of-the-art model for multimodal understanding. It is also 100 times smaller than the current SOTA models. The team has also released a Mac interface, which uses the context from the screen to help improve prompt responses and can run using local or cloud models. [Llama 3-V: Matching GPT4-V with a 100x smaller model and 500 dollars]

Andrej Karpathy Trains GPT2 in 90 Minutes and $20 of compute

According to a recent post, Andrej Karpathy has replicated the GPT-2 (124M) model in approximately 90 minutes of training at a cost of around $20. The training utilized the FineWeb dataset with 10 billion tokens and achieved a HellaSwag accuracy of 29.9, surpassing the previous score of the original GPT-2 model of 29.4. The exercise highlights the replicability of GPT models and the limited costs of training small models. [Reproducing GPT-2 (124M) in llm.c in 90 minutes for $20 · karpathy llm.c · Discussion #481]

Codestral Launches, Sets New AI Benchmark For Model Size

Mistral AI introduces Codestral, a 22B open-weight generative AI model designed for code generation, fluent in over 80 programming languages, and equipped with a 32k context window. Codestral, now available under a Non-Production License and accessible via an API, outperforms competitors in benchmarks of similar sizes and, based on user testimony, even challenges models of significantly larger sizes. [Codestral: Hello, World!]

Cartesia Launches Sonic for Real-Time Voice Applications

Cartesia has launched Sonic, a new voice model with a latency of 135ms, designed for real-time applications such as customer support and entertainment. According to Cartesia, Sonic has demonstrated superior performance in tests, achieving twice the accuracy in audio generation and delivering initial audio output 1.5 times faster than existing Transformer models, with a fourfold increase in processing speed. [Sonic Demo]

China Launches Largest Semiconductor Fund For Self-Sufficiency

China has launched its largest state-backed investment fund to date, totaling 344B yuan ($47B USD), to bolster its semiconductor industry, according to official sources. The fund, established on May 24, includes significant contributions from China's finance ministry and major Chinese banks, leading to a 3% increase in the CES CN Semiconductor Index [China sets up third fund with $47.5 bln to boost semiconductor sector]

Tougher Prompts and Benchmarking for the Chatbot Arena

Chatbot Arena has launched a new "Hard Prompts" category on its leaderboard, responding to community interest in more complex challenges for AI language models. The harder prompts also expose models with more robust capabilities and show that not all models' performance degrades equally as prompts become harder. Overfitting and fine-tuning towards the benchmark prompts are often issues, so the prompts were carefully curated in this instance to get a richer picture of model performance. [Introducing Hard Prompts Category in Chatbot Arena | LMSYS Org]

Microsoft has launched its AI-Optimized Copilot, along with a new line of PCs, marking a new chapter in personal and business computing. We break down why this is such a big deal and also share other players in the space working on agents and computer vision to improve contextual awareness. In case you missed it, other headlines from earlier in the week were about Google's I/O event and Gemini 1.5 results. Since then, we've seen their search results incorporate some of the new functionality, which has been a step up in the search experience. We also see headlines in the world of 'agents,' and we suspect this will continue to be the buzzword that garners attention for a while!

Microsoft Launches AI-Optimized Copilot+ for PCs and Business Applications

First impacted: Everyone

Microsoft has launched Copilot+ and a new line of Windows PCs designed specifically for AI applications. These devices feature powerful new silicon, including ARM processors with Neural Processing Units (NPUs) capable of a whopping 40+ TOPS, all-day battery life, and access to advanced AI models. Starting at $999, these PCs are within reach of average consumers and set an interesting stage for a world that can accommodate 'agent'-centric workflows, or what Microsoft is calling their 'Copilot'.

Hardware > Local Models > Privacy > HAL9000?
The new Copilot+ PCs introduce AI-driven experiences like Recall and Cocreator. This innovation, along with advancements in multimodal AI and privacy, is transforming our interactions with computers. Users will be able to share significantly more data with a locally hosted model than they otherwise might feel comfortable with. Instead of the usual "Don't insert any personal data," soon you might find yourself comfortable enough to share your social security or credit card details. The core mechanics of locally hosted models, combined with enhanced privacy and computer vision, could change most human-to-computer interactions.

Enhancements to Microsoft Copilot for Business
Microsoft has also announced significant updates to its Copilot feature, now integrating with Microsoft Teams and Planner to improve team collaboration by managing agendas and tracking action items. Additionally, the company introduced Copilot Studio, which allows users to create custom AI agents that automate business processes and integrate with business data systems. The ramifications for workflow management software are quite interesting, and it is impressive how quickly the landscape is changing.

Why Is This Important?
In the context of highly capable large language models (LLMs) with low latency, computer vision, and hardware optimized to host local LLMs/Agents, advancements are occurring faster than most anticipated (even those of us who were saying "it's coming sooner than you think"). We are quickly approaching a future where your local computer will be able to perform tasks autonomously and assist you proactively. What the local LLM cannot do on the local machine, the bigger, more powerful version in the cloud helps with, potentially guiding the local LLM as required. For instance, imagine needing to change your desktop background. Instead of searching for instructions online, you could simply ask, "Hey, find how to change the desktop background to X," and your computer would handle it for you. Or if you are a software vendor, instead of having customer support, you could provide agent instructions that prompt users for help if they are clicking around for more than X seconds, offering guidance on the relevant workflow. This could potentially change most workflows significantly. [Introducing Copilot+ PCs - The Official Microsoft Blog] Share this story by email

Internal and External Views on OpenAI's Alignment Problem

First impacted: Everyone

In a significant shakeup at OpenAI, key team members Ilya Sutskever and Jan Leike resigned, with Leike highlighting that “safety culture and processes have taken a backseat to shiny products” at the company. Their departure follows the disbanding of OpenAI’s Superalignment team, which aimed to address long-term AI risks. Leike emphasized the need for serious preparations for AGI to benefit humanity. Meanwhile, AI leader Yann LeCun criticized the urgency around controlling superintelligent AI, arguing that we need to focus on developing systems smarter than basic animals first. He likened the current urgency to trying to ensure the safety of advanced aircraft before the fundamental technology even exists, emphasizing a gradual, iterative approach to AI development and safety. [via @gdb] Share this story by email

Wafer Shares Demo of Agent + Mobile OS Capabilities

First impacted: mobile developers, smartphone users

Developers at Wafer.System are creating AI at the OS level, enabling AI agents to use the same device interfaces as users, such as virtual keyboards and touchscreens, to enhance efficiency and user experience. They say this integration allows AI agents to access extensive app data and user interactions, potentially tripling efficiency by predicting and automating user actions without third-party app integrations. Check out the video, it's super interesting! [Wafer] Share this story by email

AmbientGPT Launches MacOS App and Enhances Contextual Understanding with Screen Vision Data

First impacted: mobile developers, smartphone users

Awni Hannun says AmbientGPT, a new MacOS app that integrates GPT-4o for enhanced contextual understanding directly from your screen, is set to launch soon. The app, which operates entirely on-device to ensure data privacy, requires an ARM64 MacBook and a specific OpenAI API key, and is pending Apple certification. [GitHub - siddrrsh/ambientGPT] Share this story by email

Google Shares Gemini 1.5 Pro Results In Technical Report

First impacted: AI researchers, software developers

A report from Google's Gemini team highlights that the Gemini 1.5 Pro model demonstrates improved performance over the previous 1.0 Ultra, particularly in text and vision benchmarks, achieving a 91.7% score in the MMLU benchmark. The model has enhanced in-context learning capabilities, especially in low-resource language translation and mixed-modal learning, as detailed in the updated 153-page technical report. Check it out if you'd like to get into the details. [goo.gle] Share this story by email

Scale AI Secures $1 Billion in Funding

First impacted: AI developers, investors, tech industry developers

Scale AI, led by CEO Alexandr Wang, has raised $1 billion in a financing round with Accel and existing investors, boosting its valuation to $13.8 billion. The company says the funds will be used to further develop its frontier data and enhance its Data Engine, which supports advanced LLMs and computer vision models. [Careers | Scale AI] Share this story by email

Today's newsletter is a bit different. We have a diverse audience, and it's clear that the information gap between pioneers and the mainstream media is growing. Sometimes, we need to highlight what's just below the surface. OpenAI's release today was spectacular for many reasons, most of which weren't immediately apparent, mentioned by OpenAI, or covered by mainstream news. We'll dissect some of these and provide a brief breakdown of why they matter. Plus, there's more traction in the rumor mill about Apple's partnership with OpenAI and a new humanoid robot that costs less than a Toyota Yaris!

Also, a quick administrative update: we've decided to change the cadence of our newsletter from daily to weekly. This lets us deliver a hyper-curated roundup of the AI news that really matters and its implications, so you can stay informed without the clutter. If there's a breaking story you can't miss, we'll make an exception and fill you in immediately. Thanks for keeping up with us!

OpenAI Launches GPT-4o And It's Much More Than Meets The Eye

First impacted: Everyone!
Time to impact: Short

OpenAI has launched GPT-4o, now available in ChatGPT, which integrates text, audio, and vision processing in real time. We recommend checking out the demo video. OpenAI also highlighted that the new models ranked as 'GPT2' on the Lymsys chatbot leaderboards, which attracted a lot of attention and speculation. These models achieve a +100 ELO rating over previous models (which is a scoring model originally used for Chess players), representing a 64% win rate against the next closest models (which is a significant increase in win rate percentage of 78%). It accomplishes this in as little as 232 milliseconds, operating at twice the speed and half the cost of GPT-4 Turbo. Here are some highlights we found particularly impressive:

Reasoning: This is something that the AI community has been anticipating eagerly. The ability to break down a problem into its subcomponents and plan on how to solve them is crucial. There is a significant increase in the model's 'logic' capability which received little coverage to date.

Desktop App: OpenAI has shown they are masters of tech and distribution. The ability to distribute their future products and functionality to desktops will be an amazing channel because they won’t be restricted to a web browser or mobile anymore. Having the ability to watch, learn, and interact with a user’s workstation will be pivotal in expanding their value to users.

Multimodality: The ability to see and hear is a huge leap forward, and more important than it initially seems. If it can see the world, it will soon be able to see your computer screen. Once it can do that, it will have the potential to help you interact with your daily work. What does that mean? We are much closer to a world where AI can manage the flow of information and productive work than meets the eye.

On Mobile: In every pocket and focusing on consumer-led distribution first, OpenAI has really dominated the consumer market in this respect. The more users they get, the better the intent data and the better the model, creating a flywheel effect that will serve them well in the years ahead.

Latency: The final limiting factor to pulling all this off was it feeling natural. To do all of the above in a reasonable turnaround time is an engineering feat that will be underappreciated. The engineering and algorithmic optimization required to achieve this is nothing short of a marvel.

What does this potentially change? Most industries. It's a bigger change than GPT-4 was from GPT-3 and GPT-3 was from GPT-2. It's also confirmation that we are still on the exponential development ramp. The foundation is set to change most of our day-to-day routines, including search (yes, Google), communication (the 2013 movie Her example isn't lost on anyone), education, etc. We hope you are as excited about today's update as we are! [twitter.com] Share this story by email

Unitree Launches G1 Humanoid Agent that Costs Less Than Toyota Yaris

First impacted: Everyone!
Time to impact: Short

Unitree has launched the G1 Humanoid Agent, a $16,000 AI-driven robot that boasts up to 34 joints and advanced joint movement, designed to enhance sports capabilities, according to the company. The robot features AI-controlled dexterous hands capable of precise operations, potentially broadening its application in automating physical labor. [via @UnitreeRobotics] Share this story by email

Rumors of Apple and OpenAI partnership Continue to Grow

First impacted: iPhone users, privacy-conscious consumers
Time to impact: Short

Rumors are enduring that Apple is nearing a partnership with OpenAI to integrate ChatGPT technology into its upcoming iOS 18, focusing on user privacy by potentially processing data on-device. This collaboration could introduce a new chatbot feature for iPhones while maintaining Apple's commitment to privacy. [twitter.com] Share this story by email