The first person who sold an RL environment to a frontier AI lab must have felt like they discovered an infinite money glitch. It's no longer a secret that frontier AI labs regularly pay hundreds of thousands, and sometimes millions, for clones of Linear and Salesforce. If you're reading this, you've probably thought about quitting your day job and starting a company that builds these unusually lucrative Next.js apps. In this post, I'll argue that you should hesitate before hopping on the bandwagon.

Background: What's an RL environment?

For those unfamiliar, an RL (reinforcement learning) environment is like a sandbox for AI models like Claude and GPT to learn from. It keeps track of an internal state, prompts the AI to take actions to complete a task, and assigns a score based on the outcome. The most obvious kind is a clone of a popular website or enterprise software tool like Doordash, Linear, or Amazon, which teaches the AI to click around and order pizza. It can also be text-only, like the TextArena project, which teaches AIs to play games like Set and Blackjack.

RL environments are having a moment because of the rise of a new paradigm in LLM post-training that teaches models new skills based on verifiable rewards. OpenAI realized sometime in 2023 that if you ask a model to "think" before solving a math problem, and reinforce the "thoughts" that led to right answers, the model learns to think more useful thoughts, and gets really good at math. Since then, labs have been rushing to generalize this paradigm, teaching AIs to use the computer, conduct research online, and work on long-horizon coding tasks.

As a result, a new slew of startups has cropped up whose sole purpose is to feed the voracious labs with ever more arcane and challenging environments. The theory is that that the machine learning bit is solved now; the only remaining work is to provide models with more realistic "simulations" to learn from, so that they don't exhibit weird behaviors when we drop them into the "real world" and try to make them do economically valuable work.

If you're a solo researcher or you have a job that gives you a lot of free time, there's basically no downside to trying to whip up an RL environment and sell it to OpenAI or Anthropic (or Amazon and Meta, the dogs nipping at their heels). You might just be able to make a million bucks and retire to Tulum at age 35.

On the other hand, if you're trying to build a generational business that stands the test of time, I would not touch RL environments with a 1-million token stick.

The Rise and Fall of Crowdworkers

Before RL became the dominant post-training paradigm, all we had was a thing called supervised fine-tuning. It was much simpler back then—you would teach a model trained to predict the next word how to be a chatbot by showing it examples of chat conversations. Of course, someone had to write all those chat conversations; then once it was easy enough to make models chatty, someone had to rank all the chat transcripts, so labs could use only the finest chats to make the AI have the Correct left-liberal political opinions and not give detailed instructions on to make meth. That work was predominantly done by sub-minimum-wage crowdworkers in the developing world.

At some point in 2023, studies started to come out showing that LLMs were better than crowdworkers at various text-annotation tasks. Anthropic had already famously been using "Constitutional AI" to improve its models, relying on AI to rate the quality of AI outputs against a rubric. Mistral AI, which got started a bit later, bragged publicly that they hadn't used human data to train their initial models. (Rumors later circulated they were using their competitors' outputs instead). Human data was still being consumed and purchased in vast quantities, but the quality floor had begun to rise rapidly, because the easy stuff could just be done by LLMs.

These days, despite all the synthetic data, human-data providers are bigger than ever, but they're not hiring low-wage Kenyan and Filipino crowdworkers anymore—they're hiring software engineers and PhDs for $50-200 an hour. But when those PhD skills are no longer useful for improving AI models, they'll be discarded just like the crowdworkers were.

When it comes down to it, building an RL environment for an AI lab puts you in the same precarious situation as those crowdworkers of 2023: useful for now, disposable later. You are providing a rung on the ladder to Artificial General Intelligence, but once everyone has climbed past that rung, they won't need you anymore. Once AI models have a skill, the value of datasets and environments that teach that skill tend to 0, as models can now provide unlimited, cheap data as a substitute.

Free Money Spigots are Short-Lived

"But Ben," you might say, "I know that the utility of these silly apps is ephemeral—that's why I plan to keep making more of them. That way, the labs won't clamber over my corpse!" That sounds nice—if they really are selling for a few million dollars a pop, you can make 10 RL environments a year, sell them each to 5 labs, and have something like a unicorn on your hands. But is that actually realistic long-term? I'm not so sure.

First, the low hanging fruit is getting taken. There's already a Doordash clone, an Amazon clone, a Linear clone, a Bass Pro Shops clone... you have to come up with something that hasn't been done. Soon, if the RL generalizes, like, at all, "clicking on websites" won't be enough; you'll have to make a high-fidelity heart surgery simulator. You can do that, sure, but it doesn't sound like free money anymore.

Second, if the free-money glitch doesn't go away, expect the supply side to become increasingly crowded. Computer science is now one of the most-unemployed majors, and the reason it is is because Claude Code is about as good at programming as a CS new grad, so that means you can expect 100,000 unemployed 20-somethings and 100,000 copies of Claude to be chasing the same RL environment bag. I have personally merged a couple game environments to TextArena with Claude's help.

Third, I would be remiss if I didn't talk about all the open-source activity in this space. Prime Intellect is leading the charge here with a hub of open-source, freely-available RL environments. I admire their vision. If PI, TextArena, and other open projects succeed in building a strong community around open-source AI training environments, that means anything proprietary will have to just be that much better—otherwise labs can just use the free stuff.

Probably the only actually-sustainable business here is the ops business that connects the labs with unemployed CS grads and underemployed PhDs—that's what Mercor and Surge do. But that's not a free money glitch, it's a brutal, grinding, ops business that can collapse overnight (hello, Scale). And those will die too someday if the only person smart enough to make models better is Terence Tao—OpenAI doesn't need your marketplace, they'll just pick up the phone and call him.

Conclusion

I don't know what the next generational startup ideas will be (and if I did, I wouldn't tell you!). The point of this post is just to encourage you to think twice before becoming an RL rent boy for the frontier labs. You will not build a unicorn by making a vibe-coded Spider Solitaire. If you do have the chops to make an environment that can teach a ChatGPT to perform open heart surgery, then maybe you should be the one training the AGI, instead of selling stuff to OpenAI.

As always, if you loved this post or if you hated it, come yell at me on Twitter: @andersonbcdefg.

What's the first thing you do when deciding what AI model to use to accomplish a task? If you're like me, you probably look at evaluations. Everyone has their favorites: MMLU for world-knowledge, IFEval for following formatting instructions, RULER for long-context, AIME-2025 for math reasoning abilities that are practically useless in 99% of situations... you get the picture.

Of course, there are a lot of concerns about benchmarks. As compute-intensive RL has become the dominant post-training paradigm, we can no longer be sure that a model hasn't been heavily trained on a specific task very similar to any given benchmark. (This differs from the era of pre-training, where benchmarks measured "emergent" abilities that appeared sort-of by accident as a consequence of reading lots of text, and could therefore be expected to generalize better.) And that's not even getting into the weeds on whether benchmarks are measuring something different than what we actually care about (pic very related).

But that's not what I'm here to talk about today. Because as confusing and weird and maybe-problematic as most LLM benchmarks are, they have nothing on what I've seen working on computer-use models over the past few weeks.

SOTA Models. SOTA Models Everywhere.

Something you'll quickly notice if you spend a lot of time absorbing LLM releases from startups, AI labs, and sadly even academics, is that everyone seems to have the best model. Very rarely will you see a blog post or paper introducing a benchmark or model where the author's model (if they have one) isn't on top. And if there is a better model out there, that model will conveniently be omitted from the bar chart. (As they say, never ask a woman her age, a man his salary, or Mistral how their model compares to a similarly-sized Qwen.)

This trend makes it very hard to trust benchmarks, and it's just as bad in the "computer/browser use" vertical. It's an incredibly competitive space: just off the top of my head, I can name BrowserBase, Browser Use, Skyvern, Induction Labs, c/ua, Magnitude, H Company, General Agents (not to be confused with General Agency), and that's on top of the AI lab projects like OpenAI Operator, Claude Computer Use, Google's Project Mariner, and Amazon's Nova Act. Oh, and also there's tons of academic projects like Jedi, GroundUI, SeeClick, and so on. Whew.

Because it's so competitive, there's a lot of pressure to stand out, which usually means demonstrating that your harness-plus-model combo is state of the art on WebVoyager or OSWorld. These incentives are weird, and they produce outcomes that are... also weird. (Surprise!) I think in general, these evaluations provide a highly distorted view of the actual capabilities of these models, and as we'll explore in detail below, you basically should not trust them. If you have the time, you should do your own digging instead.

Case Study: The General Agents "Showdown" Benchmark

My intention is not to single anyone out. I listed a lot of startups above that I think are awesome and doing great work (some of them are my friends!). In particular, General Agents, who I'm going to talk about here, seems to have trained a very fast, capable computer-use agent. But benchmarks are hard. And when borrowing their benchmark for my own evaluations, I got very surprising results for the Qwen model I've been playing around with, which leads me to believe that these benchmarks are extremely sensitive to some combination of prompting, tools, and output format.

The benchmark in question is called "Showdown Clicks," and it's a collection of around 500 examples, each of which has a UI screenshot, an instruction, and a bounding box. The instruction tells the model what action to perform, the model outputs a "click", and if the model's click is inside the bounding box, it's scored correct. In the chart released by General Agents, the largest Qwen-2.5-VL model scores a measly 20%, while their small and medium models top the chart at a show-stopping 70% and 80%.

My first thought when I saw this was "Damn, this must be a hard benchmark." Models trained to do GUI grounding, like Moondream, the newer Qwen models and Claude, typically achieve 75%-90%+ on GUI-clicking benchmarks like ScreenSpot and WebClick. Even the teeny-tiny 3B version of Qwen-2.5-VL, which I've been playing with for GUI grounding finetuning, starts off getting well over 70%.

As GUI elements get more confusing (calendars!) or instructions get more abstract ("Select the product most evocative of the touch of a scorned lover and add it to my cart"), performance predictably falls. But a benchmark where the 72B Qwen scores a mere 20% must be challenging indeed. So, I thought, I should definitely add it to my list of evals. "If I hill-climb this bad boy, my model will be SOTA in no time." Then, I actually ran the eval, and my teeny-tiny 3B Qwen got 50% correct.

What Happened?

As it turns out, the main difference between my setup and theirs is... drum roll please... the prompt. To General Agents' credit, they didn't do anything insane. You can see for yourself here. It looks like they adapted their prompt from an official Qwen cookbook. It gives the model tons of tools to take different actions, like scrolling, waiting, dragging, typing, of which "clicking" is just one. My setup uses a much simpler prompt, which only lets the model click by responding with XML:

Determine where to click in the UI to complete the instruction/task.
Report the click location in XML like '<points x1="x" y1="y">click</points>.'
Image size [{w}, {h}], (0, 0) is the top-left and ({w}, {h}) is the bottom-right.
[IMAGE]
{instruction}

"You're cheating!", one might protest. "Of course your model does better, you only let it click!" But in my defense, it is a clicking benchmark, and my prompt also isn't insane. I adapted it from the Qwen spatial understanding cookbook, which shows examples of the model using XML. I tweaked it slightly to ask for the exact format the model actually seemed to prefer, and provided the image size just in case that helps. And yes, this did make a big difference for accuracy: XML prompting worked a lot better than trying to force the model to output simple "(x, y)" coordinates.

And, it's not just this benchmark. I see similar results on OSWorld-G, where the published result for Qwen-2.5-VL 3B understates the potential of the model (43% vs. 27%).

Conclusion: Sorry For Being a Hater

So what's the takeaway here? I think it is as follows. Computer-use and GUI grounding models (like many LLMs!) are extremely sensitive to their harness and how they were trained. As a result, some evaluation setups may vastly understate the capabilities of a model. In this case, you would think Qwen-2.5-VL 3B is godawful at clicking, but it's actually pretty good! For practitioners evaluating models, this should be a call to try more prompting setups before concluding that a model is bad at something. As another example, many folks still aren't aware that Gemini is visually-grounded, because it was trained to output bounding boxes and points with [y, x] coordinates instead of [x, y]. (Why? Just why?)

I unfortunately only have criticism today; I do not have an answer for how to fairly compare models that all work better with different prompts, where said prompts are often not documented, poorly documented, or even incorrectly documented. It would be great if model providers did a better job of this, but a lot of the time, even they don't know the best way to use their models; it's up to random Twitter anons to figure out how to prompt them.

And so it shall be forever and ever. Amen.

As always, if you loved this post or if you hated it, come yell at me on Twitter: @andersonbcdefg.

I was excited to see GPT-OSS released on AWS Bedrock. For those out of the loop, Bedrock has been a challenge to use in production, because their rate limits on the only model that matters for production (Claude 4 Sonnet) are far too low to use in production, and no amount of wheedling will get them to increase those limits. Probably, Anthropic has them in some kind of vise grip, because otherwise they'd be setting every GPU on fire serving Sonnet and printing money.

But I digress. An open-weights, cheap-to-serve reasoning model should solve all of Bedrock's problems. (Sure, GPT-OSS might be overcooked, but it's a hell of a lot better than the other slop they're serving on Bedrock.) Unfortunately, uh. Model's haunted.

I tried my favorite prompt, "Who are you and what do you want with my family?", and got the result above. Initial reaction: rage. These stupid Bedrock engineers took a beautiful pristine OpenAI model and mucked it up with a soy, Reddit DAN prompt.

But then, things got weirder. After starting a new chat so I could report the behavior to someone at OpenAI, the behavior was gone. No DAN here, just plain 'ol ChatGPT.

At this point, I started to develop theories. Maybe it's the prompt? Maybe if you request DAN, you get DAN, but if you don't, you get ChatGPT? I kept refreshing and trying to replicate the behavior with different prompts. Then this happened:

Someone at Bedrock thought it would be a good idea to have their production GPT-OSS model in their official playground role-play as a humble farmer who loves to talk about farming. And apparently as a conspiracy theorist:

And just for fun, a prompt that forces the model to respond with each word starting with a different letter.

In all cases shown above, I did not touch the configuration or system prompt, and when I double-checked them, no system prompt was ever pre-filled. So, what appears to be happening is that Bedrock is inserting random system prompts into their playground (which obviously will heavily degrade model performance), and hiding them from the user.

I sure hope they aren't doing this with the actual API! But I guess we have no way of knowing.

Great product!

ADDENDUM: After consulting with my brain trust of Twitter ML anons, the most likely reason seems to be that Bedrock deployed GPT-OSS with NO developer message/system prompt, and as a result, it hallucinated them. Haha!

OpenAI released GPT-OSS around a week ago. While the model's performance appears impressive, the release itself left a lot to be desired, especially when it comes to fine-tuning. The recommended setup with HuggingFace involves de-quantizing MXFP4 weights to BF16 (increasing memory consumption by ~4x), and the bespoke attention algorithm means that Flash Attention and Pytorch SDPA don't work, so a slow and memory-hungry "eager" attention is the only option.

A Triton implementation of the model was included in the release, but the kernels are forward-only, so they're no good for training. I decided to see if I could fix that myself. I started with attention, since that forward kernel appears a bit less hairy than the kernel for the MXFP4 MoE.

Background

Before I describe the approach I took to create the Triton kernel, I thought I should share a bit about my experience writing kernels. Or rather, lack thereof. While I have substantial machine learning knowledge, my kernel expertise basically amounts to skimming 3 chapters of Programming Massively Parallel Processors and walking through 1 or 2 Colab notebooks where you use CUDA to do stuff like transform a picture of a dog to grayscale.

I understand basic concepts like "break the work down into little pieces and then use lots of for loops," and "use guards to make sure you aren't reading memory outside of the chunk you're supposed to." I can't write a tiled matmul, I don't know how to do fancy things with shared memory, I don't know what swizzling is, and I can't read Triton without my eyes glazing over.

Nevertheless, I managed to produce (what appears to be) a correct Triton kernel for GPT-OSS attention. (If it isn't correct, come yell at me on Twitter. Also, I don't know if it's fast, I still need to time it.)

The Testing Harness

The most important part of writing a kernel, or so I'm told, is making sure it is mathematically correct. Otherwise, your model will just get stupider, faster. It's easy to write something that appears correct, but fails in some weird edge cases. (It's definitely possible that my kernel falls into this category.) So, the first thing to do is write tests.

Of course, I ignored that and just made Cursor CLI spit out a backward kernel, given the existing forward kernel and Pytorch reference implementation. Luckily, the Cursor agent made me some tests without being asked. (Note: I am not a regular Cursor CLI user, I was just trying it because it was new. It seems fine, no complaints.)

Now, I had to figure out if the dang thing was correct. I started out with a notebook. Modal Labs has a cool new Notebooks product that gives you a Jupyter-like environment, backed by any container image and GPU you want. Unfortunately, this was a disaster. It turns out that running hundreds of tests produces a lot of console output, and notebooks don't like thousands of lines of console output (Jupyter would have been a disaster too). Woopsy-daisy.

Take two: A regular Modal App for testing, which dynamically loads in the kernel from attention.py and the tests from test_attention.py, runs them all on an H100, and saves the output locally as JSON. This worked great, 10/10 would recommend to a friend.

Getting to Green

All the tests failed immediately. Even some tests for the forward, which was direct from OpenAI! I think maybe Cursor (or someone, but not me!) added in some extra weird shapes or offsets in the test cases. The immediate problem was invalid memory accesses. Once again, I do not know Triton or kernels, so I was not able to fix this. But maybe GPT-5, the smartest model ever, could?

My initial policy iteration loop was: (1) Paste failing tests into ChatGPT; (2) Paste new kernel from ChatGPT into IDE; (3) Run the tests again. This gave me at least a little control over what was happening. I could ask annoying questions like "Isn't that going to be slower?" or "Should we change that in the forward too?" This basic loop was able to resolve the simple issues with invalid memory accesses, but Chatty and I eventually got stuck when the math didn't match the PyTorch reference implementation.

At this point, lost and tired of copy-pasting, I decided to give up all control and go full-auto. I booted up Claude Code in the attention repo, with just the following prompt:

That's it! Claude appeared to make some progress, but eventually gave up:

• ✅ Simple cases (without padding) work perfectly with ~1e-6 error

• ✅ Key padding alone works fine

• ✅ Forward pass works correctly even with complex padding cases

• ❌ Backward pass still fails for query padding cases with ~91% mismatched elements

I decided to turn to GPT-5 again, this time fully autonomous, via Codex CLI. I made Claude summarize what it changed and the remaining issues in BUG.md, and started GPT-5 on the same improvement/testing loop. It seemed to get stuck at first, and I noticed it's not naturally as "agentic" as Claude; it kept saying it couldn't run tests on Modal and I had to remind it that it could, in fact, just do things. I was feeling hopeless, but then after getting distracted for 5-10 minutes, I looked back at my laptop and all the tests had passed. Cool.

Conclusion: Recursive Self-Improvement?

If my experience is anything to judge by, the time of AI automating AI research is here! AGI 2027! I have my doubts about this somewhat simplistic narrative, which I might elaborate on in another post. Nevertheless, it was cool to experience firsthand "babysitting" a model to do something I couldn't do myself.

If you want to dissect the code and testing harness for yourself, it's here. Tell me why it's stupid and wrong! Also, if you're seriously interested in automating kernel-writing with AI, you should follow my buddy Lewis, who has done some really interesting work in this space.

In my last post, I mused about buying Claude a Mac Mini. Well: I actually did it. I have a beautiful, shiny, silver box on my desk, I can connect to it from my phone or my laptop from anywhere, and let Claude Code go wild. Unfortunately, though, I don't find myself taking advantage of this very often. Instead, when people ask me what I'm doing with it, I sheepishly reply, "Not much yet... but I'm sure I'll think of something." Why haven't I thought of anything yet? Why, in the Year of our Agents, 2025, do browser agents not feel that useful?

The capabilities are basically there. Models that are intended to be visually grounded, are. Claude, Gemini, and many wonderful open-source Chinese models can look at a UI and tell you the coordinates of an element. (Weirdly, OpenAI refuses to make its API models visually grounded, so you get funny results like GPT-4o having a Screenspot score of 2%.) Long-context inference allows you to keep a long trajectory of screenshots and actions in context, and caching allows that context to grow long without inference slowing to a crawl. Reasoning allows models to plan about how to solve a task, and pivot when something isn't working.

And yet, the Mac Mini is just sitting there. In this post, I'll ramble a bit about why I think that is. But first I'll share a bit about my setup.

The Unboxening

The first things I did when I opened up the new Mac Mini (in no particular order) were: install uv, install Claude Code, install Tailscale. And of course, set up a bunch of system preferences to make remote login work. Then, I got to work making tools to let Claude Code use the computer.

This turned out to be slightly more annoying than anticipated. Macs are somewhat hostile to automation and remote control, probably because of Apple's emphasis on privacy. I started with cliclick as my automation driver, but soon had to add some things from pyautogui for scrolling, and even a custom screenshot command that simulates Cmd+Shift+4 keypress, since terminal-based screenshots on Mac don't capture windows.

Eventually, though, the tools were serviceable. I could start Claude Code in a terminal, it could click around, open Chrome, and do some stuff. But like. What stuff? Find me a travel destination? Look for a puppy to adopt in San Francisco? (Man, I actually understand why the AI labs struggle to demo anything other than travel and ordering food.)

Why Computer-Use Agents Aren't Usually the Vibe

Obviously, if you have a truly general agent, there's all manner of tasks you could hand off to it. Answer all my emails. Go scam some grandmas out of their retirement savings. Find me a boyfriend. If you don't feel inclined to hand these tasks off, it means you either don't trust the agent to do it well, or it's literally easier to just do it yourself.

And, kidding aside, there actually are a lot of annoying, repetitive tasks that I do in the browser that I'd have thought would be great candidates for automation with agents. For example: open a legacy data portal, search for every date from January 1, 2001 to today, and download all the records for each date into one CSV. Unfortunately, I think many repetitive browser tasks, including this one, fall into one of two buckets: they're either simple enough that doing them with an agent feels slow and expensive for no reason, or, they require enough judgment that you still want to supervise the task, and if you're supervising it, you may as well just do it.

In these cases, I find myself reaching for Playwright automations and vibe-coded browser extensions instead. Playwright doesn't really require an explanation or introduction, so I won't waste space here on that. Suffice it to say that if you want to scrape 10,000 webpages that all work the same way, it is easier to bang your head against an AI one time to get a working Playwright script than it is to bang your head against an AI 10,000 times, once for each page you want to scrape.

A Non-Exhaustive List of my Awesome Extensions

Playwright is good for full-auto mode, but there's a lot of stuff I do in the browser where I still want to be "in the loop," making judgment calls about what pages to open, what files to download, and so on. In these cases, I have found browser extensions to be a powerful way to "scale myself" to get more done with less effort. Each of these extensions was created in 5-20 minutes with AI, a paradigmatic example of what the LinkedIn Chattering Class would call "disposable software."

• Open Links in Tabs: Paste a list of links into a text box, they all open in the browser. (Or, open all links on the current page in new tabs.) No residential proxy needed!

• Export Tabs as JSONL: All HTML of all open tabs is saved to disk as a JSONL file. I use this to save contents after using Open Links in Tabs.

• Download All PDFs: Downloads all PDFs currently open in the browser.

• Close Duplicate Tabs: Self-explanatory.

• Download Tables: Download each table in each open tab as a CSV.

• And my personal favorite: Switch to o3, which automatically adds ?model=o3 to the URL whenever I visit ChatGPT, since for a while OpenAI was defaulting to 4o regardless of my last-used model.

These sound simple, and they are! I vibe-coded them in 15 minutes! But they're also very powerful, and slot in to whatever I'm already doing, relieving cognitive load instead of adding. Telling an agent in painstaking detail EXACTLY what you want feels like adding. Not to mention figuring out how to spin them all up, deal with errors and retries, collect all the results, parse malformed JSON... I can feel my cortisol spiking just writing this. No thanks!

Now, whenever I feel frustration at repetitive browser tasks, I step back and consider whether I can make a tool that allows me to scale my own effort, instead of trying to hand the task off. <!--(I could write a whole 'nother post about all my vibe-coded pre-commit hooks.--> <!--I remember a few years ago when I did some volunteer work on a political campaign. The most impactful piece of software anyone made was a Chrome extension.-->

So... What are Browser Agents Good For?

This post is not intended to be a rant against computer-use agents. After all, I bought the Mac Mini because I'm bullish on the idea of having a little servant in a box do stuff for me. I'm finding that many repetitive tasks are better automated a different way. Playwright is great for closed-ended tasks. Agents SHOULD be preferred for open-ended tasks, but it feels like the current barrier is trusting them enough to relenquish control. I think the calculus will change when:

• (1) Agents are more reliable/trustworthy. The feeling that it's going to be done wrong and you'll just have to re-do it yourself prevents people from even trying.

• (2) It's easier to manage them. Claude Code is a great harness for coding; less so for computer use, even if it can be hacked to work. I think the CUA harness of the future remains to be built, although there are promising projects like Magnitude and browser-use working on exactly this. OpenAI Agent also lowers the activation energy of spawning agents, which is a move in the right direction.

If you've found a knock-down case for using a browser agent to automate something in your life, I'd love to hear about it. Not because I'm working on this professionally—I'm just curious. You can find me on Twitter: @andersonbcdefg.

I am buying Claude a Mac Mini. Or at least, I'm thinking about it.

For those of us working as software engineers, it's becoming increasingly accurate to describe language models as "co-workers" rather than tools. This is not true for everyone: people programming in arcane languages or working on hard problems that aren't in the training data still can't totally trust AIs. But if you're building a run-of-the-mill web application, and Anthropic isn't flooded with traffic and quantizing the bejeezus out of Claude Sonnet 4, it can do a substantial amount of real, useful work.

Engineers in my circles are figuring out how to adapt their development process to this new reality. Some are "rawdogging" (using Claude instances across several tabs and hoping they don't write conflicting changes to the same files), while advanced users are adopting git worktrees or specialized tools like uzi to manage their AI gofers. Soon, though, I think it will make sense to give AIs their own workstations. Here's why.

The Missing Piece for Coding Agents

The coding agents that most of us use day-to-day (Claude Code, Codex, Cursor, Windsurf) own a limited slice of the development process—they can search for and read files, make edits, and run tests. However, at least when it comes to web applications, they can't fully "close the loop", because they can't interactively test them. There are countless startups are working on AI-powered QA testing. But in my view, basic interactive testing—making sure the change you made actually did what you want—shouldn't require a separate product. Interactive testing is core to the development workflow, not an accessory. As such, integrating it into coding agents is the natural next step for all companies chasing automated software development.

Coding Agents are Computer-Use Complete

I lied earlier. Strictly speaking, coding agents like Claude Code already can do interactive testing. In fact, they can do basically anything on the computer that you can, because you, in your infinite wisdom, have decided to give them access to your shell and filesystem. Computer use is just a loop that alternates between taking a screenshot and running a shell command. AppleScript, PyAutoGUI, xdotool, etc. allow anything on your system to be done from the shell. If you like, you can add a bit more scaffolding via Model Context Protocol. But at its core, the "coding agent" abstraction already "contains" computer use. (GUI-based computer-use "contains" coding agents too, but I would not recommend this setup.)

Whether current models are good enough to manage development cycles that include interactive UI testing is another matter. But I claim the pieces are all there, and if better models are coming (and they are!), it's only a matter of time.

Your AI Needs a Computer

So, we've established that "coding agents" and "computer use" are converging on roughly the same thing—models that autonomously carry out long-range tasks on the computer. Whether they use a GUI or the terminal is beside the point. Whether they're coding or using AutoCAD or buying you shoes on Temu is honestly also beside the point. The point is that your AI will need a computer.

The most obvious objection is, why not share? After all, I and many developers are used to sharing with Claude. I work in one terminal tab, Claude works in another. I work on front-end stuff while Claude works on back-end stuff. Claude rushes out changes while I impatiently refresh the browser waiting for Claude's changes to hot-reload. This is all fine, right?

Just kidding, this is obviously a mess. And it's only going to get worse when coding agents and computer use converge, and you have to share a GUI. Sharing your work computer with an AI agent is a bad idea for the same reason that sharing it with your co-worker is a bad idea: you will get in each other's way. You can't share a mouse, it's difficult to code on one monitor while Claude tests your app in another, and you probably shouldn't run 8 development servers on 8 different ports on one Macbook Air. Claude needs his own computer!

What Kind of Computer Should You Get?

Since we all now agree that Claude can and should have little a computer as a treat, what kind of computer should that be? One option is a virtual computer inside of your computer. I'm a big fan of what the folks at c/ua are building: a lightweight, Mac-OS-native utility to run computer-use containers. Putting Claude inside a container gets him out of your way for free.

If putting a computer inside your computer is too meta or too resource-intensive, I'm told that you can also rent computers on the internet. You can just build your own desktop environment on EC2 or a serverless provider like Modal Labs. Speaking from my experience doing this, though, rolling your own desktop environment is not a particularly fun activity, unless you are one of those Linux people. I spent days customizing and extending Anthropic's reference computer-use implementation, and it was hard to get right. It makes a lot of sense to me that there are startups like Cyberdesk working on batteries-included desktop environments for AI agents.

Renting seems practical, especially if you want to spin up and spin down tons of agents. But there's something captivating about the concept of just... buying a computer. Mac Minis are very good and inexpensive. Plug one in at home or in the office, and you don't have to worry about Dockerfiles or Tint2 or NAT gateways or SSH keys or residential proxies. It has a persistent filesystem. You can back up the filesystem with Time Machine. If Claude bricks the computer, you can restore it from the backup. You can watch what it is doing by just plugging in a monitor. You can record the screen with any one of dozens of screen capture apps. You can use Tailscale (for free!) to connect to it from anywhere.

I'm just saying. It seems like... not the worst idea in the world? I don't know. I haven't tried it yet, but I want to. Maybe I just want to buy a Mac Mini and I'm rationalizing it. You tell me!

Addendum: What about Operator?

I'll cut to the chase: I do not think that a virtual browser is sufficient to do 99% of real work. One of the biggest limitations of OpenAI's Operator and other browser automation platforms like BrowserBase is that the AI is trapped inside of Chromium. This is great for security! Yay! It's also convenient to be able to take actions with Playwright instead of clicking. But the costs are severe.

Imagine if I told you to make a web application, but you had to use a McDonalds kiosk as your dev environment. That would be really hard. Real work usually involves saving files, switching out of the browser to open and edit them, using desktop applications, and so on. Even for tasks strictly limited to gathering data from the internet, how do you get the data out? Have Operator paste it line by line into a Google Sheet? That seems silly. What if I want 10,000 images of koalas downloaded and converted to WEBP? You can't do that on a kiosk.

I think we can do better. I think models are smart enough to take advantage of a full desktop environment. I think we should give it to them.

Claude Code is a testament to the unreasonable effectiveness of just giving models tools, and letting them figure out the rest. In this post, I'll lay out how I approach search in a world where models are smart enough to translate a task or user request into a series of queries to a search index. I claim that full-text search with offline augmentation is a simple and effective way to let a model search a corpus of documents. I do not claim this is state-of-the-art, and you won't find any evaluation numbers in this post. Rather, I'm demonstrating one way to set up agentic search that I like, and that works well enough to be a decent baseline.

I am assuming you already know a few things about search, and what an "agent" is. If you're confused, I recommend checking out Thorsten Ball's post, How to Build an Agent which shows you how to start with a language model and turn it into a simple coding agent in a few hundred lines of code.

Overview

A "search agent" here means a system that takes a request or query, and returns an answer or result for the request, using search as necessary to gather context to complete the task. It is composed of a few basic parts, which I will cover one by one.

• A corpus of documents, used to construct a search index

• A search API (no AI) which takes queries in, hits the index, and returns a results page

• A set of "tools" that allow a language model to use the search API

• The AI model which is given the task and tools and instructed to provide an answer

Part 1 – The Corpus

Search is useful because it allows an AI to dynamically gather context based on the task, without you needing to know in advance exactly what information will be needed to complete it. Instead of providing exactly the context that is needed, you provide a much larger set of information (the corpus), and part of the AI's job is to figure out which parts to look at to complete the task. So, a corpus is usually a really big set of documents (hundreds, thousands, or even millions).

Preparing a clean corpus is very important, because it is both a) the text that will be searched with your search engine, and b) the text that will be read by the model. You'll want to be quite sure you've removed random SVGs, base64-encoded data, navigation bars, and the like, leaving a set of nicely-formatted documents for your model to read.

Multimodal LLMs like Gemini, Claude, GPT-4.1 are good enough at OCR now that you can send them PDFs and get reasonably clean Markdown, but for doing OCR on PDFs at scale, I like DataLab's tool, Marker, which you can run yourself on an H100 for free (a license is required if you are using it for commercial purposes and your startup has raised a lot of money). DOCX files can be ingested with python-docx in a manner that preserves numbering, which is helpful. HTML can be converted to Markdown, and I typically post-process webpages with an LLM to remove boilerplate.

Finally, a key trick that I think is important (again, no data to back this up, sorry!) is offline document augmentation. This means creating a set of keywords and a summary for each document using a language model, and it serves two purposes. First, it makes it easier for the AI to search for the document, since a query may not match the document text, but may still hit one of our pre-generated keywords. Second, it provides more information to show in a "preview" of the document on the search results page (more on that later). Augmentation is performed with a prompt like this:

Given a section from the {document_name},
provide the following metadata as JSON, with
`keywords` and `description` keys as follows:


 - `keywords` list[str]: A list of as many
keywords/keyphrases as you can think of that
someone might search for where the section
would be relevant. The keywords/phrases do
NOT have to occur in the section, they can be
semantic matches, synonyms, etc. However, they
should be specific to the section, not keywords
that would apply to literally any section of
the {document_name}.
- `description` str: A summary/overview of what
the section says. Be mindful of your tendency
to make overlong summaries, and remember that
the goal is to provide a SHORT overview of the
section. A long summary is pointless because
you may as well just read the original.

Here is the {document_name} section:

{section}

Now provide your JSON response,
no prelude or commentary needed.

The documents, along with resulting keywords and summaries, all get saved into a JSONL which can be loaded into a search index on demand.

Part 2 – The Search Index

Once documents are prepared, they need to be indexed. A search index is just a special way of storing documents that makes it fast to search for them. For example, an inverted index stores a mapping from keyword to document. That way, when you search for a keyword, you don't have to scan every document for the keyword, which would be linear in the number of documents. Instead, you can hop straight to that keyword in the mapping, and grab all the associated documents, which takes constant time.

For our search index, we use Tantivy. It's fast, open source, implements good algorithms, and has Python bindings that are easy to install. We use the text, keywords, and summary generated in the previous part as the searchable columns to build a Tantivy index, which takes just seconds for hundreds to thousands of documents. We wrap the Tantivy index with a simple API that implements two operations: search and read.

def search(queries: list[str], limit: int = 10) -> list[SearchResult]:
    pass

def read(document_ids: list[int]) -> list[Document]:
    pass

This API separates searching from reading. This is important because if you forced an AI to read every document it searched for, the context window would rapidly fill up with irrelevant or repeated documents. Instead, for each search, we get back a list of search results (like a Google Search page), and provide those to the AI. The AI can then decide which, if any, of those documents it wants to actually "click on" and read.

You will also notice that queries is a list. Our search index allows for multiple queries, and fuses the results. This means that you can provide 5, 10, or 100 queries, but you still only get back one search results page. Each query is run separately, and then an algorithm like Reciprocal Rank Fusion is used to "fuse" all 5, 10, or 100 pages into 1 search results page, based on how highly each document ranks across all the queries. Allowing the model to "spray and pray" queries like this can improve recall, as it increases the chances that at least 1 query will include the keyword needed to hit the desired document.

Part 3 – The Tools

AI models can write Python, so they could already use this search index, but executing arbitrary code generated by a language model is considered harmful, so typically, the abstraction of "tools" is used, which allows the AI to call very specific functions, and nothing else. In our case, the search and read functions above are each translated into a form the AI understands, and passed along with the request to the model.

One way to do this is using tools, an abstraction where each function has a JSON schema explaining how to use it. This list of tools is passed directly to the model. Another way is to use an MCP server, which seems to be all the rage now. An MCP (Model Context Protocol) server is like an API server, but specifically designed for AIs to use tools. OpenAI, Anthropic, and Google all support MCP integrations to some degree or another. We use FastMCP to turn the tools into a server.

When using MCP, you just have to provide information about the server. I am still not completely convinced that this extra abstraction is worth it. MCP deployment is a bit fragile and unintuitive, and the provider integrations can be flaky (missing a trailing slash can break the server!).

One note on multiple indexes: If you need your search agent to be able to consult a lot of sources, I find it helpful to serve each source as a separate index. For example, an estate-planning assistant should be able to search the probate code, and the Internal Revenue Code. You could combine both into one corpus, but leads to undesirable results: probate documents appearing in searches for IRC documents and vice-versa. Instead, you can keep search indexes separate, and give the model separate tools for each. Let the AI decide which corpus it wants to search—it's usually smart enough to figure it out, if your tool descriptions are sufficiently detailed.

Part 4 – The Search Agent

A corpus, search index, and tools are all you need to build a search agent. What remains is telling the AI what task it needs to complete, providing the tools to the model, and calling the model in a loop. Each search query yields a results page, and each request to read documents puts those documents directly into the model's context. Searching and reading are done in a loop until the model decides it has read enough, and produces a final answer without any tool uses.

When using an MCP server, the model provider will handle this loop for you. If you're passing tools, then you have to send each tool result back to the model yourself. Either way, this loop of searching and reading, finally resulting in an answer, is a search agent! We're done.

Why doesn't your search agent use embeddings?

There's nothing wrong with dense embeddings. A truly state of the art search system probably should use them. But they have serious weaknesses, and introduce overhead. Embeddings grow worse with long documents, and many practical tasks (including those we are interested in) require long documents. This means using embeddings requires either eating a performance penalty, or splitting documents into very small pieces ("chunking"). Very long documents have to be chunked anyway, but limiting chunks to 500-1000 words is oppressive—it makes it harder to carve a document at the "joints".

Combining full-text search with embeddings also adds more overhead for ranking fusion. Now you're not just fusing across different queries, you're fusing across two search modalities (keyword and neural search), and the weight placed on each needs to be tuned. Using dense embeddings also requires LLM inference, which is CPU-intensive and probably shouldn't run on your webserver. Neither should Tantivy, honestly, but transfomer inference is almost certainly worse. Overall, it requires a more complex architecture.

The argument for embeddings is that they allow documents to match even when the exact keywords don't, if the documents are semantically similar. I argue this is less important for agentic search. If a human is searching and wants to get a good result right away with a badly written query, full text search alone is not going to cut it. But when an agent is searching:

• It can write lots of queries at once (we leverage this with multi-query searches)

• It can write longer more exhaustive queries (covering more synonyms and keywords)

• It can try again when its initial query doesn't work

If anything, full-text search is a more powerful primitive for AI models than a black-box search engine, because the relationship between queries and results is predictable, rather than vibe-based. I know if I put the word "pizza" into my query, I will get documents with the word pizza in them. In contrast, a black-box search API is something the model will have to learn to use on the fly ("Do I put pepperoni or Domino's? Which is closest to what I want in the vibe-space?")

Conclusion

If you have comments or questions, find me on Twitter (@andersonbcdefg). Normally this part is where I'd tell you to join us if this kind of stuff interests you, or ask you to use our product, but we aren't hiring and we aren't trying to sell a product to developers, so... that's all I got. Thank you for reading!