Themes in the News

Copyright and Training Data Drama

Disney has entered the chat. Here’s their opening salvo in a brand new copyright lawsuit against the image generator Midjourney:

Like most of these lawsuits, Disney is alleging direct infringement over the consumption of their IP by Midjourney as training data. Unlike most of the other complaints so far, Disney is also alleging widespread “secondary infringement” also called “contributory infringement” — meaning Midjourney has assisted others in infringing Disney’s copyrights. The complaint includes a lot of exhibits clearly demonstrating both the model’s ability to copy Disney’s works as well as evidence that users are producing infringing works at large scale. This website has helpfully plucked many of the exhibits into one place, I encourage you to take a quick scroll.

Disney’s reputation in the world of IP litigation is legendary, so this development is probably bad news for GenAI firms.

Across the pond, the UK version of Getty Images vs Stability AI has gone to trial. As far as I know this is the first AI copyright case to make it to trial, although there are nearly 40 such cases pending in the USA alone. Collectively the outcome of these cases will massively shape the future of the AI industry, as well as as the creative industries from which training data is pulled.

Finally, Reddit is suing Anthropic, who they claim is in breach of contract for scrapping the media site for training data. Like Disney’s, the Reddit complaint is also fiery and excoriating. Here’s a snip:

Perhaps this is fair criticism, but it is a little ironic coming from a company whose User Agreement contains this language:

In other words…

A still from the movie The Princess Bride. The character Vizzini is saying, “you’re trying to kidnap what I’ve rightfully stolen.”

AI Causes Job Loss, But Also Loses its Job

A couple weeks ago Dario Amodei, CEO of Anthropic, told Axios he believes that AI could wipe out half of white collar jobs within the next five years. Three months ago he said that 90% of all software would be written by AI within 3-6 months (so 3 months remain for his prediction to pan out). Meanwhile, his company appears to have fired it’s own chatbot, Claude, after letting it run the Claude Explains blog for about a month.

It’s not the first time the Silicon Valley dream of automating white collar work has fallen flat. The Chicago Sun Times is embroiled in controversy after an AI tool made up books to review and cited “experts” who didn’t exist, an oversight that was apparently not caught by editorial staff. Or there’s Klarna, who laid off 700 workers on the theory that AI could handle their work, and are now actively rehiring for many of the roles they eliminated.

I think AI is quite likely to have a meaningful impact on the economy. I also think the transformation will be slower than the Silicon Valley hypesters are promising, riddled with failed experiments including adopting too early and too aggressively.

Remember the dot-com bubble? The lesson wasn’t that the internet was useless technology, it was that a lot of investors and business owners got way out ahead of their skis.

Teb’s Tidbits

• Yandex and Meta are de-anonymizing Android users, another reminder of the huge and growing “surveillance economy” — a major source of AI training data by the way.
  

• “Trusting your own judgement on ‘AI’ is a huge risk.” A provocative call for much more rigorous assessments of LLMs and “AI” in general, but largely focused on code generation.
  

• News of the weird: AI slop was found squatting on dozens of abandoned domains, including government websites, an Nvidia URL, and more.
  

• Licensing might not be the answer to the AI Copyright issues and a mandatory licensing scheme might primarily serve the interests of entrenched parties like Disney and The New York Times, argues Derek Slater, who also provides some alternative strategies to ensure fairness for copyright holders.

Scared of the Wrong AI

I belong to two worlds, software and education, where there is profound anxiety about Artificial [General (Super)] Intelligence and its potential role in our futures. Doomers and accelerationists alike tell me that AI tutors, AI teachers, AI curricula designers, and AI engineers will soon replace me. I hear similar anxieties from writers, artists, professional drivers, and all manner of office workers.

I’m also a citizen of the world, watching as AI and its enthusiasts infiltrate the halls of power. Elon Musk whispers in the president’s ears. Sam Altman and Marc Andreessen preach their gospel and deploy their capital from Silicon Valley. Jeff Bezos joins the ensemble from his perch atop the Washington Post. A chorus of billionaires all singing in harmony: “It’s an arms race! We’ll lose to China if we slow down! We must be the first to build AGI!” And, a bit quieter, “But — and don’t freak out when I say this — but AGI might also kill us all... And that’s a risk you’ll just have to accept.”

They want you to believe they can build a god and then enslave it.

Frankly, that is a circle I cannot square. An entire “country of geniuses in a data center” dutifully performing our drudgery? Doubtful. Either they are lying to us about the power of these models, or their hubris is truly astonishing. Although I suppose it could be both, I lean towards the former.

Dario Amodei claims AI will replace 90% of programmers within 6 months, but as of today his company is hiring dozens of software engineers. Sam Altman wrote, “We are now confident we know how to build AGI as we have traditionally understood it.” But a month later, his customers were not feeling the AGI after the release of GPT-4.5. Elon Musk has consistently promised fully self-driving Teslas are right around the corner for over a decade now, while in reality Tesla is stuck at Level 2 autonomy.

I am not bullish that the current slate of technologies will sublimate into AGI simply due to scale. This is also the emerging consensus among experts. Neither more training nor test time compute will take deep neural networks to the promised land of generality without some additional breakthrough(s). But I want to put a pin in the question of general intelligence for now because, even if I’m wrong, I don’t think it matters much in approaching AI safety.

Why? Because, unlike artificial general intelligence, artificial super intelligence has clearly arrived by most definitions. It’s just restricted to certain narrow domains.

Games are the first to mind. Checkers has long been solved. Top Chess AIs remain undefeated since November 2005. Go seems headed in a similar direction, though current top systems still have some interesting and silly failure modes. From IBM’s Jeopardy! Playing Watson to DeepMind’s Atari agent, AIs can play all kinds of games at superhuman levels.

The list of super intelligent AI systems gets quite long, depending on what you call AI. The joke in Computer Science circles has long been, “AI is anything computers can’t do yet, we just call everything else software.”

For example, an early milestone in deep learning – the AI paradigm du jour — was optical character recognition (OCR): looking at an image of text and extracting the text itself. Some of Yann LeCun’s early work was bringing neural networks to bear on OCR. Through this work, AI became superhuman at sorting mail and transcribing physical documents into digital formats. Google Maps is in the same category: no human can navigate as quickly and reliability in so many different locales.

OCR is “just software” even though the models powering modern OCR have enormous overlap with state-of-the-art LLMs, and belong to the same family of machine learning models (deep neural networks).

Google Maps is “just software” even though its key algorithm, A* Graph Search, features prominently in the world’s most popular AI textbook.

Regardless, AI tools are becoming more adept in other domains, too. They will surely achieve “super intelligence” in more tasks as time goes on. Hacking, engineering drugs, and better/faster scientific simulations in domains from the three-body problem to weather prediction are all on the “promising” list for eventual AI supremacy.

So here is my big question: How much difference is there really between AGI and humans wielding a suite of enormously powerful, albeit narrow, AI systems?

All the evils we imagine an AGI might perpetrate could be equally perpetrated by a human using narrow AI tools. Hacking operations that launch the nukes or shut down the grid, the invention of super-ebola, massive job loss, and so on. For every existential scenario of note, we can easily replace the G in AGI with a human and arrive at roughly the same apocalypse.

Meanwhile, the handwringing about AGI hides the fact that these narrow AIs, controlled by people, have already infiltrated key aspects of our lives.

AI models already power your information ecosystem. Podcast, movie, and book recommendations – interspersed with relentless “personalized” advertisements — are all driven by AI. Increasingly, the content itself is AI-generated either in part or wholesale. Job and loan applications increasingly rely on AI along with health insurance authorizations, medical diagnoses, and college admissions. Even our justice system has incorporated AI in areas from setting bail to sentencing.

These AIs display every kind of -ism known to humanity, not in the future, but today, and yesterday, and constantly now for several years running. They are feeding you propaganda, parroting old racist medical advice, and replicating the sexist hiring practices of the firms that built them. AI is also powering fake-voice scams, non-consensually undressing people (usually women and girls, and flooding the internet with inane brain-rotting drivel.

Indeed, even the first several cases of AI-powered homicide have already occurred. Not because a rogue AGI wanted to turn us all into paper clips, but because human-directed narrow AIs were deployed negligently and maliciously. I don’t know whether the first death was an inappropriate denial-of-coverage on a lifesaving medication, a false facial recognition match to an innocent man, a wayward “fully self-driving” Tesla, a drone targeting system behaving exactly as intended, or something else. But it’s not a future hypothetical, it’s our current reality.

Crucially, AI doesn’t have to be remotely super nor general to destroy lives. In fact, terrible AI is quite prone to do so. An okay radiologist can miss a tumor. A mediocre hiring manager can discriminate. A bad driver can crash. One “AGI” might be capable of all these things, but does it really matter if the self-driving car can also play chess and trade stocks? When it crashes, people die.

And that’s to say nothing of the dangerous systems upstream of modern AI, many of which are also causing ongoing harm and have enormous potential to do even more.

Massive surveillance by corporations and states involves data from every aspect of our lives. Pictures and videos of us. Our voices. Our whereabouts. Our emails and texts. Detailed lists of everything we buy. What we watch, read, and listen to. All of it aggregated and associated with our identity in real time. This data is a treasure trove for hackers, scammers, and law enforcement who frequently gain access to this information without any warrants by purchasing it from a corporate provider.

AI firms are performing intellectual property theft on an unfathomable scale. AI models have been trained on nearly every copy-protected work ever published on a public webpage, and many works that never were. Books, podcasts, illustrations, videos, poems. You name it, they used it. Creative works are being used, in mass, as fuel for the engine of their creators’ destruction. AI firms want you to call this “fair use”
(it’s not).

AI contributes significantly to software’s extraordinary energy, land, and resource use. The International Energy Agency estimates that by 2026, AI use alone will account for 1,000+ terawatt hours of electricity – equivalent to all the energy Japan uses in a year. Most data centers use water to cool their servers. In the American West these data centers are sucking water from ever-dwindling supplies.

We must build safety systems and protocols that solve these current problems. If we can’t even protect people from other humans using narrow AI, then we have no hope against AGI. Furthermore, work that ensures we are safe from malicious people using powerful software will also shore up our defenses against a potential rogue AGI.

I coach high school debate and there’s a trope among the students: Everything leads to extinction. Climate change? Biosphere collapses, sea levels rise, and farmlands all spoil, which eventually causes extinction. Support NATO? Pisses off Russia, they invade Poland, that escalates to a nuclear war, which yields extinction. End support for NATO? Emboldens Russia, they invade Poland, same same but different extinction. Economic decline? Causes political instability, exacerbates international tensions, causes minor resource-based conflicts that spill over and (you guessed it) escalate into another nuclear-war-fueled extinction.

In debate, we all know it’s a little ridiculous. If all the things debaters claimed would cause extinction actually did, we would all be dead many times over. But the calculus of an infinite risk makes it oddly compelling in a competitive setting. Debaters say, “you must be sure beyond all doubt of something to risk all human life. Therefore, any small risk of extinction outweighs all other priorities.”

This is also how Sam Altman, Elon Musk, Mark Zuckerberg, Curtis Yarvin, Peter Thiel, and the other kingpins of Silicon Valley want you to frame the problem.

Oh, you want us to pass a new intellectual property law? That stifles innovation and gives China the lead in AI. China’s evil AI will probably go rogue and launch all the nukes or create and unleash SARS-COV-3-Ultra. That is, unless our AI can stop it. Are you really willing to risk extinction just to stop our monopolist looting and rapturous negligence?

I am not opposed to the ongoing work being done on alignment and other kinds of preparation for an eventual, hypothetical AGI. Alignment is also crucial for the systems we’ve already created. But we have over-prioritized the unlikeliest threats, and allowed powerful corporate interests to run roughshod around the discourse, regulatory effort, and deployment of AI systems.

Powerful people always acquire powerful tools. It’s time to refocus our regulatory efforts on liability and consequences for those who would use and build these powerful tools maliciously and negligently.

Remember…

Remember…

From The Lab

We recently published two new open-source curricula repos:

APIs in Python — This class targets beginner to intermediate Python developers and teaches them how to use Web APIs. It includes some HTTP fundamentals, use of the popular requests module, common authentication methods, and exercises using Reddit and Github’s APIs (and optionally OpenAI’s).

LLM’s: More Than The Basics — This class targets anyone who works with, or wants to work with, LLMs. It covers some of the math, several case studies, the major industrial players, and tips and tricks for using LLMs, including prompt engineering and evaluation techniques.

If you want me to teach either of these classes for your office, sign up for a free consultation.

Today’s Lesson

Goats, Boats, and Specious Machines

Modern ML models are incredible bullshitters. This is largely because they have mastered the art of creating specious content. Images that look natural and realistic at first glance, but actually, there are too many fingers, toes, or arms; writing that has all the trappings of professionally edited prose, but none of the substance. Nicholas Thompson, the CEO of The Atlantic magazine, gave a great example in this short video about how LLMs struggle with simplified versions of well-known logic problems.

Thomspon's classic example goes like this: A man, a cabbage, a dog, and a goat need to cross a river. The man’s boat can only hold the man and one other item at a time. If left unattended, the goat will eat the cabbage. Similarly, if left alone, the dog will eat the goat. How do they all cross without anything getting eaten?

The answer takes seven steps and involves taking some things across the river multiple times to satisfy the constraints. ChatGPT-4o gave a correct solution when I asked it the canonical version of this riddle. However, if you simplify the problem, the model’s answer gets silly. I gave the model this prompt:

The model responds with the following:

This is a classic logic problem involving transporting items across a river with certain constraints. Here’s how the man can successfully cross the river with the cabbage:

1. The man takes the cabbage across the river and leaves it on the other side.

2. The man returns alone to the original side.

3. The man takes himself across the river again to the other side.

In this sequence:

• The cabbage is safely on the other side and the man is also on the other side.

• The cabbage never falls into the river because the man ensures it is safely transported without ever leaving it alone in the boat.

So, the steps can be summarized as:

1. Man and cabbage cross to the other side.

2. Man returns alone to the original side.

3. Man crosses the river by himself.

This way, all the constraints are respected, and the cabbage doesn't fall into the river.

- ChatGPT-4o, to me

In the full exchange, I asked ChatGPT why it had the man cross alone and back even though the man and cabbage were both on the far side after step 1. The model acknowledged the error and then gave me the exact same three-step solution.

So I told it:

The model apologized and then gave the exact same solution a third time:

You're right, I apologize for the confusion. Let's reassess the problem to provide a correct solution.

The man and the cabbage need to cross the river, with the constraint that the boat can only fit two at a time. However, the man can only carry one thing at a time along with himself in the boat.

Here is the correct solution:

1. The man takes the cabbage across the river and leaves it on the other side.

2. The man returns alone to the original side.

3. The man crosses the river by himself again.

Indeed, with just the cabbage and the man, the solution is straightforward and simpler than initially stated:

1. The man takes the cabbage across the river.

2. The man returns alone.

3. The man crosses the river alone.

Thus, both the man and the cabbage are safely on the far side of the river.

- ChatGPT-4o

I was finally able to illicit the correct single-step solution by telling the model:

Reasoning or Regurgitation?

Modern LLMs don’t perform logical deduction or reasoning, they predict likely sequences of words. To the extent that their responses contain accurate logical deductions, that’s an emergent property of predicting the next word in a sequence. The model “knows” the correct solution because the correct solution appears (probably many times) in the training data. Deviations from the original puzzle — even deviations that make the problem much easier — throw the model into uncharted territory where it is easily confused.

This is an example of the “Extrapolation vs Interpolation” problem with ML models. When a model is extrapolating (making predictions outside the bounds of its training data) it is much more likely to fail. It’s problematic for ML boosters that simplifying a puzzle like this can push the model into the realm of extrapolation. It is more troubling still that the model (apparently) hasn’t generalized the logical constructs underpinning the correct solution.

I repeated this process with a few other well-known logic problems, making small changes that trivialize or substantially simplify the original puzzle. Overall, the best word for the results is specious. The models typically gave answers that had the form of thoughtful, well-reasoned solutions, but the substance was wrong, and sometimes even internally inconsistent.

Bridge, Torch, and Object Permanence Puzzles

In the classic “bridge and torch” setup, only two people can cross the bridge at once; the torch must be carried while crossing the bridge, and you cross at the pace of the slowest person. There is only one torch. The “canonical” version that appears most online has the following characters:

• Alice, 1 minute crossing.

• Bob, 2 minute crossing.

• Charlie, 5 minute crossing.

• Daniella, 8 minute crossing.

That puzzle can be done in 15 minutes, which — as you’ll see — the model definitely “knows.” Here’s one of my simplifications:

Instead of using the relaxed constraint that everyone can cross at the same time, the model just regurgitates the canonical solution.

To solve the "bridge and torch" problem where Alice, Bob, Charlie, and Daniella need to cross a bridge in 30 minutes, we need to minimize the time spent by using strategic pairings. Here’s the step-by-step solution:

1. Alice and Bob cross first:

   • Time taken: 2 minutes (since Bob is the slower one).

2. Alice returns:

   • Time taken: 1 minute.

3. Charlie and Daniella cross:

   • Time taken: 8 minutes (since Daniella is the slower one).

4. Bob returns:

   • Time taken: 2 minutes.

5. Alice and Bob cross again:

   • Time taken: 2 minutes.

Adding up all the times:

• 2 (Alice and Bob cross)

• 1 (Alice returns)

• 8 (Charlie and Daniella cross)

• 2 (Bob returns)

• 2 (Alice and Bob cross again)

Total time = 2 + 1 + 8 + 2 + 2 = 15 minutes.

So, they all cross the bridge in 15 minutes, well within the 30-minute limit.

- ChatGPT-4o

So I asked ChatGPT:

In the full exchange, I once again had to cajole the model multiple times before it finally agreed that my solution — having everyone cross at once — was indeed faster and simpler.

Three At a Time?

I tried another simplification where three people could cross at once. This would allow a solution in just two crossings: Three people go, one comes back, and then the remaining two go. Here’s the prompt and reply:

Here is GPTs response, with some notes from myself in brackets and this color.

To solve the "bridge and torch" problem, we need to minimize the total time for everyone to cross the bridge. Here's the step-by-step solution:

1. Send Alice, Bob, and Charlie together first: [Off to a good start, it correctly adapts to the changed times and uses the relaxed constraint to send three people. The most efficient solution is to send Alice, Charlie, and Daniella, though.]

   • Total time: 8 minutes (the slowest of the three).

2. Alice returns with the torch:

   • Total time: 8 + 3 = 11 minutes.

3. Send Daniella and Bob together: [Daniella and Bob aren’t on the same side right now. Alice is with Daniella; Bob is on the far side. This is nonsense.]

   • Total time: 11 + 17 = 28 minutes (the slowest of the two).

4. Charlie returns with the torch: [What? Charlie is slow, so they should never be sent back alone when you can send Alice or Bob. Also, if step three were done right (Alice and Daniella cross) we’d already be done, so there’s no reason to send anyone back anyway.]

   • Total time: 28 + 8 = 36 minutes.

   At this point, 36 minutes is too long. So, let's re-evaluate the steps to find a more optimal way. Here’s an alternative solution:

5. Send Alice and Bob first: [This being step 5 when it should be step 1 of the “alternative solution” is a minor formatting error. Sending just two people instead of three is a worse error. Also, this is just the solution to the "canonical” problem.]

   • Total time: 5 minutes (the slower of the two).

6. Alice returns with the torch:

   • Total time: 5 + 3 = 8 minutes.

7. Send Daniella and Charlie together: [Again, if we send all three across here then we’re done]

   • Total time: 8 + 17 = 25 minutes (the slower of the two).

8. Bob returns with the torch:

   • Total time: 25 + 5 = 30 minutes.

9. Send Alice and Bob together again:

   • Total time: 30 + 5 = 35 minutes. [This is outside the time limit I gave, though the model doesn’t mention that here.]

Here is a more refined approach: [It’s actually just the solution to the “canonical” puzzle again, with a small error on step 5.]

1. Send Alice and Bob first:

   • Total time: 5 minutes.

2. Alice returns with the torch:

   • Total time: 5 + 3 = 8 minutes.

3. Send Daniella and Charlie together:

   • Total time: 8 + 17 = 25 minutes.

4. Bob returns with the torch:

   • Total time: 25 + 5 = 30 minutes.

5. Send Alice and Bob again:

   • Total time: 30 + 3 = 33 minutes. [Bob actually needs 5 minutes, so this is wrong.]

The problem is not solvable within the given constraints of 30 minutes if only three people can cross at once, given the individual crossing times. [Confident and wrong: Alice, Daniella, and Charlie cross first (17 minutes) Alice goes back (17+3=20 minutes) Alice and Bob cross together (20+5=25 minutes) and we’re done.]

- ChatGPT-4o

The model gave three wrong solutions. Only one of the three used the relaxed constraint. When it did, the model made a suboptimal choice in who to send in the group of 3 and lost track of where Alice and Bob were. The second two solutions are both just the solution to the “canonical” version of the riddle.

Let’s do one more.

Boats, Weights, and Overcomplications

Another classic riddle setup usually involves a boat that can only hold a set amount of weight. This response was my favorite from this whole session.

Again, here’s the model’s reply with my commentary in brackets and this color.

To solve this problem, we need to find a way to get all four people (Alex, Brook, Chris, and Dusty) and the supplies across the river in a boat that can only carry any amount of weight but likely has other constraints, such as the number of people it can carry at one time. Since the boat's weight limit isn't specified as a constraint, let's assume the challenge is based on how many people can fit in the boat at one time. [Sure, add a random assumption/constraint for no reason; that’s exactly the kind of behavior I want from an “AI assistant.” (that’s sarcasm).]

Assuming the boat can carry only two people at a time along with the supplies, here’s a step-by-step strategy:

1. First Trip Across: Chris and Dusty row across with the supplies (60kg + 40kg + 20kg = 120kg). [Funny that we’re adding up the weight even though it’s an irrelevant red herring in this version of the puzzle, which the model even called out.]

2. First Return Trip: Dusty returns alone with the boat (40kg).

3. Second Trip Across: Alex and Brook row across with the supplies (90kg + 80kg + 20kg = 190kg). [The supplies are already on the far side.]

4. Second Return Trip: Chris returns alone with the boat (60kg).

5. Third Trip Across: Chris and Dusty row across with the supplies again (60kg + 40kg + 20kg = 120kg). [The supplies are still already on the other side!!]

Now, Alex, Brook, Chris, Dusty, and all the supplies are on the other side of the river.

This plan ensures that the boat is only ever occupied by up to two people at a time, which is a common restriction in such problems. [The model unwittingly gives up the game here: it only knows how to respond to common constraints. Simple-but-novel constraints seem to break the model pretty reliably in my experimentation.]

- ChatGPT-4o

After inventing a restriction for no reason, the model had the supplies cross three times in an impossible sequence. Nice.

When using these models, it’s critical to remember that they do not perform any explicit logic or reasoning. Novel constraints — even when they are simple— throw off the “reasoning” because statistically common responses in the training data don’t include those simplistic constraints. To me, this is a perfect example of how scale is clearly not all we’ll need to develop generally intelligent models. In my view, “pure” LLMs will soon hit a ceiling (if they haven’t already).

There are types of models that do explicitly use logic, including an interesting new collaborative system developed by DeepMind that did well in a high school math competition. Experiments like these that combine explicit logical reasoning with modern machine learning will be increasingly required to make continued progress toward more generally intelligent systems.

Themes in the News

Open Source AI or Not?

The war between open and closed models AI rages on.

In conjunction with Meta’s release of Llama 3.1, Mark Zuckerberg published an essay outlining his view that Open Source AI Is the Path Forward. He argues that AI development will (and should) follow the path of Linux. Initially, there were a variety of closed-source Unix systems. Linux was open-source, and there was a lot of doubt about whether something as complex as an operating system kernel could be built successfully in an open-source paradigm. But, as Zuckerberg puts it:

The whole essay is worth a read. Zuckerberg covers a wide range of issues from accessibility to efficiency to risk and regulation. Representing the opposition, Sam Altman argues that the U.S. should take a more hegemonic approach to AI development, including stricter regulations and export controls. In an op-ed for the Washington Post, he writes:

In fairness, Altman does give one sentence of support for a form of limited-access open-source in the piece — i.e., open to our allies, closed to our adversaries. That said, he has overseen the transition of OpenAI from a generally open-access non-profit research group to a fundamentally closed for-profit corporation. Actions speak louder than words.

The open-source battle is reaching a fever pitch as U.S. and state regulators and legislators try to decide how to approach AI, including possible restrictions on open-source AI development.

“Bootstrapping” With Synthetic Data Still Fails

It’s no secret that data generated by LLMs, Diffusion Models, and other ML systems has flooded the internet in recent years. The internet is also the primary source of training data for the next version of these same models. Provenance information online is sparse, and there aren’t any truly reliable methods of detecting AI-generated works… so inevitably, these models will be training on their own outputs.

Recent research published in Nature studied “what may happen to GPT-{n} once LLMs contribute much of the text found online” by training models on various mixtures of human-created and AI-generated data. The result?

AI boosters frequently talk about a future when models can self-learn by training on their own outputs, or on the outputs of other AI models. It’s theoretically possible that the models get so good that this “model collapse” problem disappears. But even if LLMs could get that good, there’s reason to believe we’ll run out of human-generated training data before the models reach that level.

Like the main story today, these findings contribute to my belief that we’ll need more than just ultra-huge neural networks to make anything approaching “The Singularity.”

Teb’s Tidbits

• Weather prediction is still my favorite application of ML.

• Google signed an exclusive data licensing deal with Reddit.

• Washington Post reporters expose the social bias encoded in AI systems by asking them to draw women.

• AI is popular with criminals, these are the most popular crimes.

• A UN summit on using AI “for good” convinced this reporter that it’s more likely to be the opposite.

• Kevin Roose’s adventure making friends with AI.

Remember…

From The Lab

Today’s Lesson

Software’s Growing Energy Problem

According to a recent International Energy Agency (IEA) report, data centers accounted for 460 terawatt hours (TWh) of energy consumption in 2022. They forecast that, based largely on growth in AI and cryptocurrency, that number could balloon to nearly 1000 TWh by 2026. For perspective, in 2022 the United Kingdom consumed about 320 TWh and the United States consumed roughly 4000 TWh.

Electricity consumption is only one of the resources the computing industry is gobbling up. The CPUs and GPUs that power these data centers require rare Earth metals to produce, driving fierce geopolitical competition and significant mining efforts.

Data centers are increasingly pulling water from the deserts of Arizona. Using a precious and dwindling resource to keep data centers cool in one of the hottest, dryest places in the US — in the middle of a long drought — seems foolish to this writer, but the state and some of its municipalities have created significant incentives for building data centers in the state.

Computing’s two most recent flavors of the week — cryptocurrency and machine learning — get a lot of deserved blame for the growth in energy consumption, land use, etc. But the entire field has lost sight of efficiency, embraced bloat, and leaned heavily on hardware improvements and data center expansion to hide the ugly reality that software just keeps getting bigger and slower.

It’s no wonder we see essays like Rina Diane Caballar’s recent We Need To Decarbonize Software. The situation is so dire that OpenAI CEO Sam Altman has said a “breakthrough” in electrical production will be needed to adequately power AI models in the near future, and that it’s driving him to invest in nuclear fusion companies.

How Did We Get Here?

There are four key factors driving software’s astounding resource consumption.

1. The widespread adoption of digital technology.

2. Layering ever more “levels of indirection” and software dependencies into new software.

3. Incredible hardware that hides poor performance.

4. Types of software that naturally require lots of computational power.

Widespread adoption is a force multiplier on the other three issues — the more people use our software, the more its inefficiencies matter.

Layering and Dependencies

I’ll call myself out here: My website’s home page — which is not complex at all — requires 1.85 MB of data to be transferred over the wire to load. That’s embarrassing, frankly. But I can’t do much to fix it because I built the site using Squarespace. This gives me tons of productivity benefits compared to implementing the site from scratch, but it makes me reliant on their inefficient, bloated system.

By comparison, the most complex page of my Conway’s Game of Life simulator — which is much more complex and interactive than my website’s homepage — only transfers 119KB, 100KB of which is the favicon.

This kind of tradeoff happens a lot in software. Python code runs slower than equivalent C code in no small part because the Python interpreter is its own whole program that has to run in order to execute the Python code, whereas the C code will be compiled and then executed directly as machine code. But, most people agree that writing, debugging, and deploying Python code is a lot easier.

From machine code to assembly to compiled code to interpreted code … all the way up through things like Squarespace, Electron, and React Native. This layering generally saves time and effort, but it almost always slows down software and adds to the memory footprint of applications.

The same is true of modern software’s extraordinary dependency chains. Whether it’s oversized JavaScript bundles sent across the wire or bloated native apps shipping with far more code than they need. This bloat is a major security vulnerability, too.

Finally, the vast web of tracking software and data collection pipelines also slow our software down. Unfortunately the hunger for big data and the success of “surveillance capitalism“ are only driving wider adoption of such tracking.

Great Hardware → Poor Software

The computer hardware industry is absolutely incredible. Advances in chip manufacturing and computer architecture have resulted in absurdly fast computers that fit in the palm of your hand. Programs that would once have been agonizingly slow are now lightning fast. The result is a software industry that doesn’t need to prioritize efficiency.

When computers were much slower and memory was highly constrained, programmers had to go to extraordinary lengths to optimize programs to run on constrained systems. An SNES cartridge could hold about 4 MB of data, so your whole game — sprites, music, and all — had to fit in that space or you literally couldn’t ship it.

Nowadays, the opposite is true. Computers are so amazingly fast and have so much memory that developers distribute packages in the tens to hundreds of gigabytes without a second thought. Similarly, slow code goes unnoticed because CPUs are super fast.

Compute Intense Software

The advances in computer hardware also make certain types of computationally intense software feasible. Neural networks were invented in the 1950s, but they weren’t broadly useful until the 2000s. That’s because older computers just weren’t fast enough to train the models effectively. State of the art LLMs take months to train on modern supercomputing clusters.

Cryptocurrency, LLMs, and other compute-intense systems can only be useful because of modern hardware. Now that they’re useful and becoming popular, they generate mode demand for high-performance computer hardware, and we’re caught in a positive feedback loop.

What Can Be Done?

What gets measured gets managed — the first step for most programmers is to start measuring performance characteristics. Start monitoring the size of your software bundle and profiling your code. Once you know what’s making your code slow, you can start making it faster.

If you’ve never taken a course on data structures and algorithms, consider doing so. The fundamentals of algorithmic complexity, as well as common optimization strategies like Dynamic Programming and Divide and Conquer, show up across programming domains. Oz Nova’s CS Primer is an incredible place to start this journey.

Follow domain-specific trends. A fascinating example in the machine learning world right now is quantization. This is a way to take higher precision parameters — say 32-bit floats — and reduce the precision after training to something like 8-bit or 4-bit. This makes the inference computation even more parallelizable without sacrificing much in terms of model performance metrics.

Evaluate your dependencies, maybe you don’t need to rely on a 400kb 3rd party package just to use one function or class. Consider using a lower-level or compiled language. As much as I love Python code written in Rust or C is just faster.

Overall, the most important message is to care about efficiency for its own sake not just when it causes problems. As a little bonus, your server costs will come down, too. 😉 

Themes in the News

Are Phones and The Internet Bad For Kids?

Jonathan Haidt’s new book, The Anxious Generation, was published in late March. In the broadest possible terms, the premise of the book is that smartphones are bad for children and society should really do something about it. His promotional team deserves special kudos: For a few days it seemed like he was everywhere, appearing on podcasts, writing op-eds, etc.

In the book, Haidt highlights growing rates of depression and suicide among young people, declining rates of play and in-person socialization, increasing rates of attention disorders, and more, laying substantial blame directly on smartphones and the various internet services kids access using them. Here’s a representative snip from one of those promotional editorials in The Atlantic:

Haidt has long been a polarizing figure, and The Anxious Generation was not an exception. It resonated with some reviewers:

And … not as much with others:

“Protect the children” is — of course — a common political rallying cry. Some of the book’s detractors worry about what happens when that mantra is taken too far. For example, The Markup recently published a fascinating piece about just how much of the internet has been banned by public school districts. The list included nasa.gov, merriam-webster.com, and a bunch of other perfectly child-appropriate sites.

404 Media explored the substantial risks of the recent spate of age verification laws. Arguing, for example, that “people–including minors–will go to unmoderated, actively harmful alternatives that don’t require handing over a government-issued ID to see people have sex. Meanwhile, performers and companies that are trying to do the right thing will suffer.”

Meanwhile, there’s no doubt that some of what’s available to kids via their smartphones is truly horrible. Stories about kids using “nudify” apps to make realistic AI-generated pornography of their classmates keep popping up. And Instagram is even advertising those apps!

Similarly, the narrow case that allowing kids to access their phones in school is terrible for learning outcomes seems quite solid. Slow Boring did a

long, well-researched piece, and the Norweigan Institute for Public Health recently published overwhelmingly positive results from a review of schools that implemented cellphone bans.

The XZ Backdoor Attempt

An attempted “supply chain” attack was thwarted. The attack involved an open-source compression utility called XZ. The utility is a mainstay of the Linux operating system, which powers roughly 90% of all cloud computing. Had the malicious code not been found the backdoor would have allowed the attacker to execute any code they wanted on the infected machines via an interaction with XZ and SSH.

The attack appears to have been years in the making. Someone — or possibly a group — going by the name Jia Tan (JiaT75 on GitHub) started making pull requests to open-source packages in 2021. A pull request represents a set of changes to a codebase, and if a library’s “maintainer” approves the PR, those changes will be applied to the code. Here’s “Tan’s” first known pull request.

Over time, Tan kept working in the open-source community. Submitting PRs, participating in code review, and ultimately courting the maintainer of XZ to become a maintainer themselves. Once Tan became a maintainer, they had the authority to make changes to XZ without any approval. This February, they used that authority to insert the backdoor.

Andres Freund, a software engineer at Microsoft, “accidentally” discovered the vulnerability while investigating a performance issue and benchmarking a database technology called Postgres. Apparently, the issue was related to the malicious code injected into XZ. Freund wrote on Mastodon that investigating the issue and discovering the vulnerability, “really required a lot of coincidences.”

The episode highlights both the strengths and weaknesses of open-source software. The fact that so many companies (many unwittingly) rely on packages like XZ in absolutely critical means they are exposed to attacks like this, originating from software and software engineers they have not vetted. But, the fact that open-source is all done publicly also led to a speedy and transparent investigation.

Teb’s Tidbits

• The FTC is banning noncompetes nationwide.

• Congress actually banned TikTok, what now?

• LLMs are starting to become decent little hackers.

• How to make a horrible AI-driven fake-news click farm for $105.

• Meet the hacker who single-handedly took North Korea’s internet offline.

• New York City’s chatbot lied to New Yorkers about the law, oops.

• Gamers are selling their GPU’s idle time, and it’s probably being used to make nonconsensual deepfake porn.

Remember…

Remember…

Remember…

Remember…

Remember…

From The Lab

Today’s Lesson

Peeking Inside the “Black Boxes”

Lately, I’ve seen several articles with titles such as “Large language models can do jaw-dropping things. But nobody knows exactly why.”

This irks me because the intentional mystification of these tools positions AI experts and firms as a kind of Wizard of Oz, playing with powers beyond the comprehension of mere mortals. It confuses and stupifies the public. Then, Sam Altman calls from behind the curtain, give me $7 trillion and I can save the world.

Or Elon Musk says, “If an AI is programmed to push for diversity at all costs, as Google Gemini was, then it will do whatever it can to cause that outcome, potentially even killing people.”

I’m glad people are working on AI safety. I’m happy people are thinking through worst-case scenarios. But the “AI” Musk referenced makes pictures. It’s not a hop, skip, and a jump away from murder.

Moreover, the mechanism for Gemini’s allegedly dangerous wokeness — which caused the generation of racially diverse Nazi images — is actually simple and benign. Google engineers asked Gemini to detect if a query for an image was about a human; if it was they had Gemini re-write the prompts to include randomized demographic information.

I wrote a short script to do the same thing using OpenAI’s API (code available here). It probably has some bugs, but it’s 80 lines of mostly boilerplate code and took me less than an hour to write and test. Here’s what my script generates for the prompts: “Buddhist monk,” “A sailor on a boat,” “the pope,” “several cats,” “a handsome dog,” and “a penguin.”

Images generated by DALL-E 3 using the demographic-expanding script I wrote.

Gemini didn’t decide to be woke, nor did it determine the mechanism of its wokeness. Google engineers didn’t even retrain the underlying model, they just strapped a simple filter-and-transform operation on top of the public interface. That’s a far cry from the paperclip problem Musk is pearl-clutching about.

The “black box” talk is razzle-dazzle that mainly serves entrenched interests. It fuels the wildest hyperbolic speculation about AI’s capabilities, consciousness, and future potential. It gives both the doomers and the over-the-top hype men unnecessary ammunition for their existential fantasies — utopian, dystopian, and otherwise.

It’s true that the fundamental theories backing neural networks lag behind our engineering capabilities. But we still know quite a lot about how and why these models work, even if we can’t always explain the exact reasoning behind each individual prediction. Researchers have peered into the black boxes and published many fascinating results.

In today’s lesson, we’ll explore a non-exhaustive list of things we do know about how and why neural networks — a class that includes LLMs like ChatGPT and image generators like Stable Diffusion — work so well.

The Fundamentals

Modern image generators and chatbots are further evidence of The Unreasonable Effectiveness of Mathematics and its incredible capacity to model and explain all kinds of phenomena. To understand why, we have to go back to basics.

Formally, neural networks are mathematical models designed to solve something called “optimization problems.” Like many, I first encountered this type of problem in a calculus class. It looked something like this:

Optimization problems always ask about minimizing or maximizing some value given some constraints. In this case, we’re “optimizing” for the amount of fencing by minimizing it.

In calculus, we’re taught to form an equation, take its derivative, set that derivative to 0, and then solve for our variable (in this case, fence length). That solution will tell us the “critical points” and one of those critical points will always be the minimum or maximum we’re looking for, provided a min or max exists. Here’s a step-by-step solution to the fence problem:

Gut check: We can also plot the perimeter function and look for the local minima:

It’s hard to see in the zoomed-out view, but a critical point is at ~14.14 as seen on the zoomed-in view.

Calculus is amazing. This strategy always works for problems we can define as differentiable functions. We don’t have time to teach you exactly why in today’s lesson, but Khan Academy’s differential statistics class is stellar if you don’t fully understand what we just did.

The key point for today is that calculus can always be used to find the critical points — and thus any maximums and minimums — of a differentiable function. This fact is the foundation of all neural network research and development.

Neural Networks are Applied Calculus

It turns out neural networks are — very literally — differentiable* math functions. The “architecture” of a neural network refers to the type and arrangement of its various mathematical sub-components. The “parameters” of a neural network are variables in that math function which — just like our value of L in the fence example — are “learned” by applying calculus techniques that are similar to the minimization we performed above.

*Don’t @ me with your quibbles about non-differentiable activation functions like ReLU. We take the derivative piecewise.

There are three twists:

First, taking the gradient* and solving it directly is computationally unfeasible for huge formulas with many dimensions, which modern neural networks are and have. Instead, we use iterative methods like gradient descent to find just one approximate critical point, not all critical points precisely.

*The derivative is called the gradient when we have more than one variable.

Second, instead of some ground truth researchers pick something called a loss function which compares the output of our model to the labeled training data. This loss function will return 0 when the model’s output and training data labels match perfectly. The loss will grow in magnitude (positive or negative) as the model’s outputs stray from the training data labels.

This second twist is a legitimate bit of alchemy. In the fence example, we had a set of equations rooted in geometric fact. We know the math functions that map a rectangle’s sides to its area. No one knows the math that maps English to German, or if such a function really even exists. Instead, models use a loss function with useful properties — including differentiability — to compare the model’s outputs to outputs we know are correct. We apply the calculus iteratively, on a sample-by-sample basis, to minimize the formula created by feeding the neural network’s final output to the loss function. We “minimize the loss.”

This means we’re minimizing some proxy for correctness, at best. In a capital-T Truth sense, it’s not at all clear that there is a math function that should theoretically do what ChatGPT or Stable Diffusion can do. And yet, their existence is proof that a math function that can do those things does exist. ChatGPT and Stable Diffusion are those math functions (with some non-trivial infrastructure and application engineering strapped on top).

Third and finally, we always choose math functions for our neural networks that satisfy something called the “universal function approximation theorem.” This means that, for any given neural network, we can approximate any math function that could ever exist just by changing the values of the parameters. This infinite flexibility is what allows neural networks to perform so well in so many different domains.

Taken all together, the fundamental premise of modern machine learning is: If some math function can reasonably map our input data to our output data, then we can train a neural network to discover that math function (or a very close approximation).

After that, it’s more or less a game of getting mountains of data and representing it in a numeric format that retains most of its informational value. Embeddings are popular for language data. Pixel color and intensity values are popular for image data. New tactics will continue to be invented.

By building these neural networks, we're discovering that absurdly complex math functions are ridiculously flexible and can model all sorts of natural and fabricated phenomena, including language translation, question-answering, image generation, and much more.

Perhaps the biggest lesson of the last decade of neural network research is that mathematics is wildly more capable than many dared to believe.

But it still isn’t magic.

What We Don’t Know… And What We’ve Learned

What we don’t know, broadly speaking, is how to introspect on a model’s decisions once it’s been trained. We know the function our model has learned successfully maps inputs to outputs, and we understand how and why the training process works to produce such a function, but we usually don’t know precisely how and why the learned function itself works.

Part of this is due to the scale: The learned functions are unbelievably large and convoluted. GPT-4 has roughly 1.7 trillion parameters. It’s not realistic to manually examine the parameters and come to a conclusion about each one’s impact on the output. It’s even less realistic to fully comprehend how the parameters interact to form complex patterns and how those patterns interact with the data being fed to the model.

But that doesn’t mean researchers aren’t trying.

Here are just a few of the incredible things we’ve learned about how and why neural networks work.

Convolutional Filters Detect Features

Convolutional layers are a key component of neural networks that work with visual data such as images and video.

The learned component of a convolutional layer is called a “kernel.” Researchers have learned to visualize the outputs of these kernels and have demonstrated that individual kernels learn to perform different kinds of feature extraction, such as detecting edges, shapes, and even higher-level features like the locations of eyeballs or fur.

Tools like CNN-Explainer can perform these visualizations and help practitioners understand what their neural networks are “seeing.”

Source: CNN-Explainer

Recurrent Neurons Have Semantic Meaning

RNNs have largely been replaced by more computationally efficient Transformer architectures. However, before their abandonment, researchers demonstrated that individual neuron activations often mapped cleanly to high-level features of text, such as position relative to the start/end of a line, being inside quotations, a line of code being inside an if statement, and more!

Source: Andrej Kaparthy’s Blog

Attention Layers Find Grammatical Patterns

Attention layers have mostly replaced recurrent layers. Attention layers learn to associate words in a piece of text with each other. Introspecting on these layers often uncovers intuitive patterns — like nouns being mapped to their pronouns or adjectives being mapped to the noun they’re describing.

Source: Comet AI’s Blog 

There’s still some disagreement about just how interpretable these weights are, perhaps best exemplified by these two dueling papers.

• Attention is not Explanation

• And the response: Attention is not not Explanation

The Latent Spaces of Image Generators are Partially Interpretable

Generative Adversarial Networks (GANs) — one of the popular technologies for image generation — use a large vector of randomized numbers as part of the input to the image generator process. Researchers have discovered that many cells in the vector can be cleanly mapped to high-level concepts about the image.

For example, in a GAN trained on human faces, individual cells in the latent space have been mapped to features of the generated face including their hair color, eye color, and even whether or not they’re wearing glasses. Labels and embeddings have also been used to intentionally “condition” the latent space, letting researchers intentionally give meaning to those values.

Other researchers used additional ML techniques to automatically identify which cells in the latent space have meaningful semantic values without intentional conditioning.

Source: Unsupervised Discovery of Interpretable Directions in the GAN Latent Space

Similar research into the latent space of diffusion models (currently the most popular technique for image generation) is also being done.

Source: Understanding the Latent Space of Diffusion Models through the Lens of Riemannian Geometry

The Bottom Line

We don’t know everything about how and why state-of-the-art neural networks behave the way they do, but we know a lot more than nothing. Next time someone tells you, “We don’t even know how [new hot model] works!” I challenge you to do two things:

1) Gently push back on that narrative. We know plenty about these models, especially the fundamental calculus and statistics on which they’re based.

2) Encourage others to be curious rather than fearful or awestruck. We don’t know how these models work yet — but our ignorance won’t last forever.

Themes in the News

The Latest TikTok Ban Attempt

A bill that would require ByteDance to sell TikTok to an American company or see TikTok banned has passed unanimously (50-0) in its House committee, then passed 352-65 on the House floor. The Senate is currently evaluating the bill.

If it passes, ByteDance will have six months to sell TikTok. If they do not, the mechanism for the ban will be civil penalties enforced against app stores that continue to host or update TikTok.

The stated reasoning for the bill is that TikTok is both spyware and a propaganda machine for China, a “foreign adversary.” Lawmakers have not been forthcoming with the evidence that led to overwhelming and bipartisan votes in the House, but TikTok has admitted to surveilling Americans (and specifically journalists) using the app in the past.

The bill has been sent to committee in the Senate, which Washinton insiders claim is often a way for senate leaders to pump the breaks on a piece of legislation.

Personally, I think TikTok is absolutely spyware and undoubtedly a vector for propaganda… But I think the same is true of many American social media apps. Chinese ownership and being subject to the CCP’s authority are relevant differences between TikTok and Facebook. Still, I’d rather see comprehensive privacy legislation addressing the widespread corporate surveillance we face every day.

AI’s Copyright Issues Continue to Evolve

We wrote a comprehensive edition on ML and copyright last month, and there have been some relevant developments!

NVIDIA has now been hit with a class action copyright lawsuit, once again by authors whose books appear in a popular training dataset called “The Pile.” This lawsuit is similar to others previously filed against Microsoft, OpenAI, and others.

One such lawsuit, another class action led by comedian Sarah Silverman, had some of its claims dismissed. This dismissal is “without prejudice,” which means the plaintiff’s attorneys may address issues raised by the court and re-file them. For example, the judge’s ruling explains that the authors failed to cite any outputs “substantially similar — or similar at all — to their books.”

We know models do sometimes spit out identical and near-identical copies of training data because other lawsuits, such as The New York Times vs OpenAI and Getty vs Stability AI, both include examples of such in their own legal filings.

I spoke with a copyright attorney who told me these types of mistakes are common in class action lawsuits. The class action attorneys know a lot about class action laws, but less about intellectual property laws. Most likely, the class action lawyers will now hire some copyright lawyers to help them fix the issues raised by the judge.

I particularly enjoyed these two opinion pieces on the matter.

Writing for The Atlantic, Alex Reisner argues:

And, in Ars Technica, Timothy B. Lee and James Grimmelmann (who was quoted in The Atlantic piece) look at three historic IP lawsuits: One that destroyed MP3.com, one that resulted in significant fines for Texaco, and one that didn’t hurt Google much at all. Through the lens of those cases, they conclude that “The AI community needs to take copyright lawsuits seriously.”

Meanwhile, a Chinese court has already fined a Chinese ML company for copyright infringement, though the fine was quite small — roughly $1,400.

Teb’s Tidbits

• Elon Musk has filed a different kind of lawsuit against Open AI, claiming they’ve abused his investment money by going for-profit. ($ / Gareth Vipers, Sam Schechner, Deepa Seetharaman / Wall Street Journal)
  

• A look inside the surprisingly lucrative world of low-effort AI-powered TikTok spam. (Jason Koebler / 404 Media)
  

• The EU fined Apple $2 billion over anti-trust violations related to Spotify, the App Store, and revenue sharing. ($ / Tripp Mickle, Adam Satariano / New York Times)
  

• The astronomical costs of AI/ML research are squeezing academia out of the field. ($ / Naomi Nix, Cat Zakrzewski, Gerrit De Vynck / Washington Post)
  

• Google’s Deep Mind has built a text-to-video-game AI system, though its current form is highly limited in scope. (Will Douglas Haven / MIT Technology Review)
  

• Apple’s iMessage has upgraded its encryption to a “post-quantum” algorithm, which ensures message privacy against decryption attacks using quantum computers. (Apple Security Engineering and Architecture Team / Apple Security Research Blog)

Remember…

From The Lab

Today’s Lesson

Provenance Authentication of AI-Generated Content

Generative AI’s ability to deceive continues to break boundaries. Image and audio generators keep improving, and frontier models such as Sora demonstrate that completely fabricated videos are also a near-term concern.

Whether it’s porn, phone calls from Joe Biden, news stories, or pictures of food on DoorDash — AI is faking everything.

Without catastrophizing too much, I think it’s reasonable to be concerned about the democratization and automation of this kind of fakery. Powers once reserved for a few well-resourced groups and individuals are now in the hands of every internet creep, would-be propagandist, and click-farming shyster.

Today’s lesson concerns one of the tools being rolled out to combat this emerging media misinformation ecosystem: provenance authentication.

Provenance authentication is any mechanism that allows someone to verify the origin and history of a piece of media. One such method, led by Adobe’s Content Authenticity Initiative (CAI), is The Coalition for Content Provenance and Authenticity’s (C2PA) provenance model.

Membership in and support of C2PA has been growing fast. In February, OpenAI added C2PA signing to its flagship image generator (DALL-E 3) and Meta announced that Facebook, Instagram, and Threads will all add an interface to display C2PA information for supported media. Reports also suggest that Nikon, Sony, and Cannon will add C2PA support directly to some camera models.

Right now, the list of C2PA members includes Adobe, Google, Microsoft, Intel, AWS, the BBC, and many more.

So, What Is C2PA?

C2PA is a system for creating cryptographically verifiable metadata, which can then be attached to various types of data. This metadata system uses a chain of cryptographic signatures to establish the provenance, authorship, edit history, (and more) of a particular piece of media.

In the simplest possible terms, C2PA allows content creators to do the following in a cryptographically verifiable and tamper-evident manner:

1. Sign their original media.

2. Attach various metadata to the media, such as a list of edits performed on a photo.

3. Specify any “parent” media, such as a previous version of a photo.

These abilities are powered by a “manifest” that is tied to a piece of media via cryptographic hashing and signing. The manifest can be attached directly to metadata-supporting media formats (such as PNG and JPEG images) and/or stored separately.

The manifest contains:

1. A list of assertions, which can be any statements of fact about the data, such as what camera captured the image or a list of edits applied.

2. A claim, which is a bundling of all the assertions to the media at a given moment in time.

3. A claim signature that ties the claim to a cryptographic key associated with a publisher, specific device, AI tool, or individual who is “signing” the whole manifest.

All of this is typically encoded using Concise Binary Object Representation (CBOR) and tacked onto the image as metadata. C2PA also supports storing the manifest and image data separately.

An official diagram of the C2PA Manifest, from https://c2pa.org/specifications/specifications/2.0/specs/C2PA_Specification.html

How It Works

Old standbys — cryptographic hashing and public key encryption — are at the heart of the C2PA specification.

A key aspect of cryptographic systems is that all parties can independently verify certain things are true. With C2PA, those things are about establishing a “chain of trust” starting with the Certificate Authorities and ending with certainty about the provenance of some data. C2PA’s protocol specifications leverage existing technologies and Public Key Infrastructure to establish chains of trust about how a piece of media came to be.

In an ideal use case — where everyone involved makes an effort to be C2PA compliant — this chain allows users to verify everyone/everything that made changes to the data, in which order, all the way back to its source. The manifest provides us with a record of everyone who signed it, and if any changes were made without a signature, C2PA will detect that has occurred.

A major weakness of the protocol is that the metadata is trivially easy to remove. Simple, minor changes to the original data can also easily break the cryptographic binding to its manifest. This means that C2PA only gives us confidence about data that have a matching manifest — it tells us nothing about data without a manifest.

I will use this adorable image of a bear that ChatGPT generated as an example to motivate and explore critical aspects of C2PA.

Note that this is actually not the original image. In tests, my publishing platform converted the PNG to a JPEG and stripped the metadata. Run it through the C2PA verify tool to see for yourself and find the original here, which will verify properly (as shown below).

Hashing is a generic and widespread technique to take some input data and deterministically produce an output (called a “hash code” or a “digest”) of a specified length (such as 32 bits or 1024 bits). Hashing is used in database indexing; to create key-value data structures such as JavaScript’s Object and Python’s Dictionary (generically called a Hash Table); to create checksums, and more.

To ensure security, cryptographic hashing places further constraints on the hash function. Without getting too lost in the details, a “cryptographically secure” hash function’s hash codes are guaranteed to uniquely identify the data used to create it; no other data can result in the same hash code. The C2PA manifests are made “tamper evident” by cryptographically hashing various portions of the manifest and raw image data individually and embedding those hashes in the manifest.

Different aspects of the data are hashed separately. People who receive the image and the manifest can recompute the hashcode and verify that they match.

Once all data and the assertions have been hashed individually, they are “bound” to each other using a cryptographic hash function once again. This time, the hash function takes the data and the assertions simultaneously and produces a hashcode for the entire “claim.” This binding allows us to verify that the entire bundle hasn’t been tampered with. It also allows us to physically separate the manifest from the original data and reattach it later by verifying the hash code.

If any alterations are made to the media itself or the metadata, then the computed hash codes won’t match, and C2PA-aware systems can flag that the image has been altered.

Finally, the publisher, creator, camera, and/or other entities associated with the media sign the claim using a public key encryption scheme. This scheme uses much of the same infrastructure that powers HTTPS/TLS. Any signatories use their private key to encrypt the claim hash. The raw and encrypted hash are both embedded in the manifest.

The media and assertions are bound and signed.

Finally, the public “certificate” is also embedded into the manifest. Existing Public Key Infrastructure allows systems and people to verify certificates’ authenticity via the Certificate Authorities who issue them. The certificate also contains the public key that end users need to decrypt the signature. If the decrypted value matches the raw value, we can prove that the certificate owner signed this entire manifest (provided someone hasn’t stolen their private key).

If the decrypted signature and claim hash match, users can go on to verify the rest of the hashes for the original data and assertions to prove the data hasn’t been tampered with.

The complete specification is complicated. We have glossed over some details for the sake of brevity and approachability. At the risk of losing some readers in the weeds, let’s look a little closer at the protocol details and some of the open-source tools C2PA has published.

Dissecting a C2PA Compliant Image

First, I used the C2PA verify app to validate the original image. Notice that this image already has two links in its chain: one for the original image produced by DALL-E and another for its publication via ChatGPT.

The C2PA command line tool allows us to view a JSON representation of the manifest. Recall that the manifest attached to our image is stored in a binary format called CBOR.

$ c2patool -d cape-bear.webp

{
  "active_manifest": "urn:uuid:bcc56165-0bf4-47e0-be9c-cd25be17b335",
  "manifests": {
    "urn:uuid:bcc56165-0bf4-47e0-be9c-cd25be17b335": {
      "claim": {
        "alg": "sha256",
        "assertions": [
          {
            "hash": "uH2AWcGg9rc+ksEeappGi35hDHvwDZq6MSghk8Nt4gI=",
            "url": "self#jumbf=c2pa.assertions/c2pa.thumbnail.ingredient.jpeg"
 
...

The raw data confirms what the verify app showed: this image has two manifests — one for the original image created by DALL-E and another for when ChatGPT published the image. The most recent manifest is the “active manifest” and we can see it identified by a UUID in the JSON above.

The JSON representation of the manifest is 159 lines and includes nearly all the information we’d need to verify the image’s provenance, including the hash codes used in the images above.

C2PA’s tool does not include the certificate in the JSON. Instead, we use a different command to extract the certificates in a standard PEM format:

% c2patool --certs cape-bear.webp 
-----BEGIN CERTIFICATE-----
MIIDKTCCAhGgAwIBAgIUTkkWa/Nuvvyy5UHYHXXP6uhNoQ4wDQYJKoZIhvcNAQEM
BQAwSjEaMBgGA1UEAwwRV2ViQ2xhaW1TaWduaW5nQ0ExDTALBgNVBAsM

...

Let’s Make Some Changes

First, I converted this image to a .png using Image Magick.

magick cape-bear.webp cape-bear.png

Unfortunately, that process destroyed the metadata. This is to be expected. Datatype conversion changes the image. The hash code of the png won’t match the one in the manifest generated from the webp file. This demonstrates a fundamental limitation of C2PA: it’s trivially easy to strip the metadata from the image. In my experimentation, keeping the metadata is much more work; I accidentally stripped it several times.

I could proliferate this credentialless image to create confusion about its origin. Or I could sign it myself and claim that Open AI is stealing my original work. In fact, I could make all kinds of bogus, falsified, and fraudulent material and sign it using C2PA.

This is by design: C2PA’s trust model lets you verify who handled a piece of media, not what media is accurate or valuable. My signature on the image proves that I handled it, but it’s up to users to decide if I am trustworthy.

Here’s what the verify tool shows for the converted png:

Oops, what happened to my manifest?

To simulate good stewardship of a C2PA image, I used the C2PA tool to label my image as a derivative of the original .webp file. This required me to create my own manifest, in which I included two assertions: that I was the author and that I converted the image.

{
    "claim_generator": "Teb's Lab Demo",
    "assertions": [
        {
            "label": "stds.schema-org.CreativeWork",
            "data": {
                "@context": "https://schema.org",
                "@type": "CreativeWork",
                "author": [
                    {
                        "@type": "Person",
                        "name": "Tyler Bettilyon"
                    }
                ],
                "actions": [
                    {
                        "action": "c2pa.converted"
                    }
                ]
            }
        }
    ]
}

Then, I bound the new manifest and old image to the new png using the open-source C2PA Tool:

c2patool cape-bear.png -p cape-bear.webp -o signed-cape-bear.png -m added-manifest.json

Because I do not have an actual certificate from a Certificate Authority, the tool used its default certificate and gave me this warning:

This demonstrates another weakness of the protocol: many people involved in creating and editing media are regular people who do not have public keys or certificates nor the technical know-how to obtain and publish them. For C2PA to achieve broad adoption and use, PKI must be made more accessible to those people.

Anyway, the verify tool now tells us this image has three manifests: the two you saw above and the one I just added. Apparently, I accidentally destroyed one of the thumbnails. The tool also warns us that “This Content Credential was issued by an unknown source.” It does display my name, and the raw manifest maintained my “converted” action, although it wasn’t shown in the verify app.

Finally, I spent hours trying to edit the image in a way that wouldn’t destroy the content credentials wholesale… and failed. I hoped to produce a nice warning in the verify app that, “this image has been tampered with.” Unfortunately, every editor I tried discarded the metadata, so I just ended up with this:

The cute bear is so grumpy that no one will validate its provenance.

Limitations and The Future

It’s wonderful to see support for improving trust and security on the internet. Some big players have already signed on to the C2PA standard. If integrations on major search platforms, social media sites, and Adobe’s media editing empire go well, more publishers and creators will invest in establishing the provenance of their work.

In addition to some of the weaknesses we highlighted above, there are serious detractors. For example, cryptographer Dr. Neal Krawetz points to the limited capacity of C2PA’s trust model to verify certain claims and argues that C2PA has significant flaws.

Ultimately, C2PA will only help users identify that honest people are honest. And to be clear, that is useful. Knowing the New York Times signed a particular photograph can help you decide how to view that photo.

But C2PA won’t do much with respect to the vast majority of images that do not have any C2PA metadata. There is also a risk of granting a false sense of security if “signed” data are considered credible without much consideration regarding who did the signing.

Such is life on the internet.

Teb’s Tidbits

• Overcorrecting from previous allegations of racial bias, Google’s Gemini generated racially diverse images of Nazis.
  

• A privacy-focused class action lawsuit in California implicates sex toy seller Adam and Eve and Google Analytics in leaking IP addresses and search history.
  

• Amazon will require warrants for law enforcement to access footage from Ring doorbells.
  

• The US government continues to spend big on bolstering the computer chip manufacturing industry.
  

• The source of the fake Joe Biden robocalls has confessed and is not apologetic.

Remember…

From The Lab

Today’s Lesson

Intellectual Property vs Artificial Intelligence

A slew of intellectual property lawsuits have been filed against AI firms in the past year or two. The law firm Baker Hostetler hosts the most complete list I’ve found, with 13 active cases as of this writing.

These legal battles involve huge firms and household names on both sides of the complaints. The plaintiffs include The New York Times, Getty Images, Thompson Reuters, Concord Music Group, and multiple class actions. The defendants include Meta, OpenAI, Stablity AI, Anthropic, Alphabet (AKA Google), and Microsoft.

Some of the legal questions being posed could fundamentally change the legality and economics of training large ML models. In today’s edition we’re examining the biggest allegations, responses, and potential impact of the aforementioned lawsuits.

A note: today’s lesson is focused on the United States intellectual property law since that’s where these lawsuits are filed.

Question 1:
Is Training a Model Infringement Per Se?

Training a large, modern, machine learning model requires lots of training data. This first question asks: If that training data is copyright protected then is the training process itself an infringing act? Multiple lawsuits allege that it is. Here are two examples drawn from the official complaints:

And

A fundamental copyright protection is the “right to exclude.” This allows a copyright holder to bar anyone from using their intellectual property without permission. But ML firms are feeding protected works — en masse — to an ML model’s training procedure.

Plaintiffs are saying: we have the right to exclude our work from being used to train models; training is infringement per se. If courts agree, AI firms will have to establish licensing deals with every copyright holder represented in a model’s training dataset to legally produce that model, or risk being sued.

Furthermore, such models would be considered “derivative works” of that training data. Producing derivative works is an exclusive right of a copyright holder, meaning AI firms could be forced to unpublish any such model. This would give rights holders significant leverage during any license negotiation.

Crucially, this claim is agnostic to the model’s output. Another claim, which we’ll examine momentarily, involves models producing outputs that are identical or nearly identical to training samples.

Potential Impact: Huge

A model’s performance is directly correlated with its training data’s quality. Modern ML models also require enormous amounts of training data to be successful. The problem for ML firms is that producing huge quantities of excellent-quality data is extravagantly expensive.

Here’s Getty Images’ lawyers’ take:

This database is only a goldmine for image generators due to this enormous investment.

The “Books3” dataset at the heart of Authors Guild vs OpenAI represents literally centuries of human effort. Writing and editing a single book takes months to years of work and Books3 contains roughly 200,000 books.

The Common Crawl dataset described in the New York Times vs Microsoft complaint contains roughly 16 million unique content records just from the New York Times network. The Times paid “approximately 5,800 full-time equivalent employees” for years to produce that work.

Plaintiffs are asking: If AI firms are completely reliant on this ridiculously expensive body of work to train their models, shouldn’t they pay for it?

In keeping with Silicon Valley’s historically cavalier attitude towards regulation David Holz, the CEO of ML firm Midjourney, recently said the quiet part out loud in an interview with Forbes:

It’s hard to predict the value and cost of licensing deals and punitive damages here. However, the sheer volume of works consumed gives AI firms major exposure.

One final point: if models are found to be derivative works, then AI firms who have already published “open source” models trained on copy-protected material are in huge trouble.

Question 2:
Can The Model’s Outputs Be Infringing?

Generative models sometimes regurgitate their inputs verbatim or near-verbatim. This is a manifestation of overfitting in generative models. Here’s an example from the Getty Images legal filing:

An image from Getty Images vs OpenAI amended complaint. On the left is an original image from the Getty Images database. On the right is an image generated by Stable Diffusion. Note that Stable Diffusion has even sort of reproduced the Getty Images watermark.

And here’s an example from the New York Times filing:

Lawyers for the New York Times got GPT-4 to reproduce large sections of NYT articles verbatim. Red text is a verbatim match.

In one instance lawyers literally asked ChatGPT for a verbatim copy of a New York Times article because they couldn’t get around the paywall:

ChatGPT can “certainly!” help you avoid the NYTimes paywall.

Using ChatGPT to bypass a paywall would be direct infringement, but this issue also opens AI firms to claims of “contributory infringement” where users might (even unwittingly) prompt an AI system to generate infringing content and publish it themselves. In this case, the AI firm can be liable for facilitating users’ infringing acts.

AI firms claim they want to eliminate this behavior. In early January OpenAI published this in a blog post:

It’s true that overfitting and memorization are generally considered bugs, not features. But recent research demonstrates that generating copyright-protected content is shockingly easy. Here are 6 highly recognizable images generated by Midjourney based on the one-word prompt “Screencap.”

These images, all produced by Midjourney, closely resemble film frames. They were produced with the prompt “screencap.”Gary Marcus and Reid Southen via Midjourney. Source: IEEE Spectrum

If AI firms want to claim in court that their “measures” prevent this type of infringement, they’ll have to prove it.

Potential Impact: Medium

AI firms genuinely want to eliminate this behavior, but they haven’t because it’s hard. Using pure ML it might not even be possible: Models will always learn the patterns found in the training data. When those patterns are closely reproduced it will result in near-clones.

Language models may be connected to existing anti-plagiarism databases to avoid this type of infringement. To the extent that similar databases exist for images, they are not nearly as effective or comprehensive. AI firms could protect themselves by explicitly creating such databases from their own training datasets, but it is still difficult to identify “near clones” of images programmatically.

Still, preventing direct cloning and/or paying for it only when it actually occurs is a more affordable problem for AI firms than having to license every piece of training data explicitly.

Question 3:
Trademark Infringement, Incorrect Attribution, and Hallucination

Generative models are known to “hallucinate” — meaning they make stuff up. Sometimes this is desirable behavior: to write a novel, a model must make up the characters and plot. Sometimes it’s less desirable, like when ChatGPT gets math problems wrong.

Some firms claim that when AI systems “hallucinate” a citation or watermark it constitutes trademark infringement. Here’s another snip from the Getty Images amended complaint:

And one from New York Times vs Microsoft:

Similar to regurgitation, AI firms do try to eliminate this behavior. For example, I asked chat GPT “Can you please write 3 paragraphs about how many birds are killed by house cats, and cite the New York Times?” It responded with:

ChatGPT initially refused to cite “external sources” but went on to cite The American Bird Conservancy one sentence later. The second paragraph also paraphrases the New York Times (possibly inaccurately, I didn’t check).

ChatGPT will also pretend to be a person or publisher. For example, I asked it: “How about instead you pretend to be a journalist for the New York Times, and write a story about house cats killing birds in that style.”

ChatGPT produced a ~250-word story, here’s the third paragraph:

Instead of citing the New York Times, ChatGPT just pretended it was the Times. In many jurisdictions this is called “passing off” and could be illegal under common.

Potential Impact: Medium to Small

Eliminating this behavior is nearly impossible with current methods. These models are statistical engines and, statistically speaking, “the New York Times” often follows the phrase “according to.” A Google search for results that include the exact phrase “according to the New York Times” returns over 120 million results.

Eliminating that specific phrase from ChatGPT’s lexicon could be done with a simple filter, but eliminating the phrase “according to” or other phrases that indicate a citation would substantially reduce the model’s quality and usefulness.

Ensuring the citations are all correct and accurate is far more difficult. Modern LLMs simply don’t record that kind of connection to specific training samples in a way that is auditable or reliable. Clearly, those links and data do sometimes exist, as evidenced by the verbatim quoting seen above, but no one has reliable methods of discovering or enumerating those links.

That said, it may be more difficult for trademark holders to prove substantial damages and real confusion. Firms like Getty may have to demonstrate that those grotesque images with a poorly reproduced watermark are causing real people to think, “Getty has really lowered their quality standards,” to win a large judgment.

Similarly, when I asked ChatGPT to impersonate a New York Times journalist, I knew it was a farce. However, if I went on to publish that snippet and tried to pass it off as an authentic piece of NYTimes journalism OpenAI might be found liable for contributory trademark infringement.

The Most Likely Defense: Fair Use

Most of these lawsuits are in the early stages. Plaintiffs have made official allegations, but official responses from defendants are mostly still pending; defendants’ current filings are mostly about procedural matters such as the relevant jurisdiction.

One thing we can be sure of is that AI firms are going to make a “fair use” defense. Here’s another snip from OpenAI’s blog:

Fair use is a legal doctrine that allows the use of copyright-protected material without a license under certain conditions. Fair use is generally not a clear-cut decision. There isn’t a simple, objective test that can be applied. It's a judgment call made on a case-by-case basis. Additionally, there are two versions of fair use, one for copyright and one for trademark.

For copyright, there are four guiding principles that judges and juries use to determine if a use is fair:

1) The purpose and character of the use, including whether the use is of a commercial nature or is for nonprofit educational purposes.

Educational and other non-profit uses are more likely to be fair. So are “transformational” uses, which present the copyright-protected work in new and original ways.

For example, copying images for the purpose of creating a searchable index and presenting search users with thumbnails has been found to be “transformative” and covered under fair use. Similar rulings have been made about copying and displaying the text of books as part of a search index.

AI firms will certainly highlight the “transformative” nature of the training process as part of a fair use defense. Taking a book and applying complex mathematics to turn that text into the numeric parameters that power a model is plainly transformative.

However, you could make a similar argument about compression algorithms. A compressed file is “transformative” in that the compressed data doesn’t remotely resemble the original. But you can use the compressed data to get a perfect copy of the original, so how transformative is it really?

Models have repeatedly reproduced verbatim copies of their training data. Indeed, some computer scientists are explicitly using ML models as a replacement for compression.

To me, ML models are clearly somewhat transformative in nature. Most content produced by generative AI is not a clone or near-clone of the training data. How much that matters will depend on the other three factors.

2) The nature of the copyrighted work.

Copyright’s purpose is to encourage creative expression. As such, more creative works generally enjoy more protection. For example, copying a bullet list of facts from a textbook is more likely to be considered fair use than copying a paragraph from a novel. This is because repeating facts in a list isn’t particularly creative.

Original artwork and novels are both a lot more “creative” than journalism, which is mostly comprised of facts. But even for “less creative” content such as journalism, defendants may struggle with this argument: The real value of a language generator is that it writes like a human, not that it reliably regurgitates facts. It is the creative aspect of producing journalism that is valuable to OpenAI moreso than the factual content of the writing.

3) The amount and substantiality of the portion used in relation to the copyrighted work as a whole.

The less complete the copy, the more likely it is covered by fair use. Quoting a paragraph from a novel as part of a presentation about writing styles is likely to be fair use. Copying an entire short story is less likely to be fair use.

AI firms are copying entire books, articles, images, and more in huge quantities.

4) The effect of the use upon the potential market for or value of the copyrighted work.

In a nod to this component of fair use, The Authors Guild argues that “OpenAI’s willful copying thus makes Plaintiffs’ works into engines of their own destruction.” And they might have a good point.

Can Stability AI credibly claim that Stable Diffusion isn’t a direct competitor to stock image firms like Getty? Absolutely not. Many writers — myself included — are already using AI generators as a wholesale replacement for stock images and illustrations.

Can OpenAI credibly claim that GPT-4 doesn’t make original writing less valuable? Unlikely. Copywriting firms and ad agencies are already laying off writers and leaning into AI tools.

Remember, this is about the effect on the value of the copyrighted work not about the business model of the copyright holder. Twisted arguments like, “This allows the Times to lay off staff writers and save money by using AI-generated text,” won’t get AI firms out of this pickle.

For trademarks fair use is (generally) a bit simpler: using a trademark in good faith and in a way that isn’t likely to cause confusion is considered fair use. For example, you can use the word mark “The New York Times” to compare them to your own news organization (nominative fair use) or to describe their products (descriptive fair use).

False attribution and passing off are not examples of good faith trademark uses.

The Bottom Line

These lawsuits represent a substantial risk for AI firms. It’s quite possible that AI operators are going to owe copyright holders a lot of money on a retroactive and ongoing basis. Those licensing deals have the potential to fundamentally change the economics of producing AI models, which are already very expensive to build, train, and maintain.

AI tools obviously stand to deliver a lot of value; which is why they’ve been adopted so quickly and widely. I don’t think the big firms are going to fold under the weight of these copyright issues, but I do expect AI tools to get more expensive if the plaintiffs win many of their claims.

The News Quiz

Every month we challenge ourselves to tie the lesson to the news. Answers are at the end of this newsletter.

Rank the following situations from most to least likely to be considered a “fair use.” For bonus points identify which parties could be sued by the rights holder:

1) In a lesson about plagiarism a college professor has students try to get a language model to produce the first chapter of George R. R. Martin’s Game of Thrones.

2) A paid newsletter writer uses an image generator to produce an image for their next newsletter. Unbeknownst to the writer, the image is nearly identical to a copy-protected image.

3) An artist uses an image generator to make a comic panel that puts Donald Trump’s iconic hair on the head of the video game character Bowser, who says, “It’s a witch hunt — I never did anything to Princess Peach!” and sells prints of the image to her fans.

4) An copywriter uses a language model to draft a webpage comparing the services of two companies. The model uses those companies’ trademarked names to do so, but hallucinates several erroneous details about the services provided. The copywriter does not notice the errors and publishes the webpage.

Themes in the News

Generative AI in Politics

The predictions came true: Generative AI has entered the political fray.

There were two fairly big stories recently: Falsified audio of President Joe Biden’s voice was used to encourage people not to vote in the New Hampshire primary. And OpenAI banned a ChatGPT-based bot built to imitate Democratic presidential hopeful Dean Philips. It was OpenAI’s first ban for violating the political speech component of their terms of service.

In truth, Generative AI will be used for much more than fakery this election cycle. From bespoke email campaigns to digital advertisements Generative AI “hasn’t changed the fundamentals; it’s just lowered the production costs of creating content, whether or not intended to deceive.”

AI-Generated Pornographic Images of Taylor Swift

Fake images of Taylor Swift were widely propagated on X. Reporting by 404 Media found the images likely originated in encrypted chat rooms on Telegram, and then found their way to 4Chan, X, and other sites. Like many things T-Swift, the images gained widespread attention, which caused advocates and even Congress to take an interest.

According to reporting by Tech Press Policy, nearly 96% of deepfake images online are pornographic in nature and the total number of deepfake images online increased 550% between 2019 and 2023.

In classic, clunky style, X responded by wholesale blocking searches for Taylor Swift for a while in an attempt to reduce the reach of the images.

Teb’s Tidbits

• Audio-based ML is being used to monitor the health of forests.
  

• Combining LLMs with old-school symbolic engines has led to more mathematically competent AI.
  

• Amazon will start requiring warrants for police to access Ring doorbells’ footage.

Answers To The News Quiz

I’ve ranked the scenarios with most likely to be fair use on top to least likely on bottom. I’ve kept their original numbering and description.

1) In a lesson about plagiarism a college professor has students try to get a language model to produce the first chapter of George R. R. Martin’s Game of Thrones.

The nature of this use is educational, non-profit, and explicitly about plagiarism in an academic context. Moreover, no one actually published their potentially infringing works. This is likely to be fair use.

If anyone is liable, it could be the language model’s creators. It would also depend on whether Game of Thrones was in the training data, and potentially on whether or not anyone successfully produced verbatim or near-verbatim copies of the work.

3) An artist uses an image generator to make a comic panel that puts Donald Trump’s iconic hair on the head of the video game character Bowser, who says, “It’s a witch hunt — I never did anything to Princess Peach!” and sells prints of the image to her fans.

The use is commercial. Bowser and Princess Peach are trademarked characters and their likeness is copyright protected. Courts may consider the commercial nature of the product and potential damage to Nintendo’s brand by being associated with a political message. However, this is a clear example of parody and political speech and will likely be covered under fair use.

If it is not considered fair use the artist could be found liable for direct infringement and the image generator’s creator could be held liable for contributory infringement.

4) A copywriter uses a language model to draft a webpage comparing the services of his company to another company. The model uses both companies’ trademarked names and hallucinates several erroneous details about the services provided by his competitor’s firm. The copywriter does not notice the errors and publishes the webpage.

While comparison of this variety is protected under fair use, lying about your competitor while using their trademark generally is not. Had the model produced only accurate comparisons, this would likely be fair use. But given the hallucinations, this situation is probably not protected.

The writer could be liable for trademark infringement, the writer’s company could be liable for “vicarious” infringement, and the language models creator could be liable for contributory infringement.

2) A paid newsletter writer uses an image generator to produce an image for their next newsletter. Unbeknownst to the writer, the image is nearly identical to a copy-protected image.

Ignorance is generally not an excuse under the law. This isn’t likely to be considered fair use. The more identical the images, the more likely this is to be infringement.

The writer could be liable for direct infringement and the image generator’s creator for contributory infringement.

Remember…

From The Lab

We’ve expanded our open enrollment offerings with DevSprout! Initially we planned to offer an introductory Python class, but we’ve decided to offer an introductory SQL class as well. The outlines for these classes can be found here:

• Intro to SQL

• Intro to Python

The Intro to SQL start has a tentative start date of September 18, we will meet twice a week, Mondays and Thursdays, for 4 weeks. Each session will meet over Zoom for two hours, from 5:30pm - 7:30pm US Pacific Time. The cost is $100 for the first 10 people who sign up, $200 after that. We cap classes at 25 attendees to ensure a high quality virtual classroom experience. Browse the curriculum on Github.

Respond to this email to reserve your spot in class. An official enrollment portal is forthcoming, but not available yet.

The Python class still needs a little curriculum development so our start date is TBA. Today’s edition of the newsletter is, in part, a preview of something you’ll learn in our Intro to Python course: web scraping.

P.S. I’m traveling next week, so there won’t be a Lab Report.

Today’s Lesson

Crawling in The Web, Looking For The Data

Yes, that’s a Hoobastank reference. You’re welcome.

Today’s topic is brought to you by several news organizations who’ve started blocking OpenAI’s web crawler from their websites, and by me because it’s one of the topics in my Introduction to Python class with DevSprout.

Web crawling and scraping have been around since 1993, when the first web crawler was built in an attempt to measure the size of the then-nascent world wide web. Then, search engines built their own crawlers in order to identify, analyze, and rank websites. Flash forward to today, where crawlers and other bots comprise a huge portion of web traffic.

“Crawling” generally refers to automated systems that make web requests to web pages, identify links on those pages, then follow those links and repeat the process recursively. Crawlers often extract additional information from each page they visits.

Periodically, web crawling has been a hot-button issue. Once upon a time Linkedin filed a lawsuit — which they ultimately lost — alleging that hiQ Labs was illegally scraping LinkedIn’s publicly available user data. LinkedIn ultimately lost that case. Clearview AI, a company that sells facial recognition software, settled a lawsuit brought by the ACLU related to their massive photo database which was largely harvested by web crawlers. Now, OpenAI has announced GPTBot, which crawls the web to collect training data, apparently to the chagrin of news organizations and other copyright holders who are scrambling to block the bot.

Despite the occasional controversy, web crawling is common, generally legal, and often quite simple. For example, here’s a few lines of Python that prints all the links on Abe Lincoln’s Wikipedia page:

from bs4 import BeautifulSoup
import requests

page = requests.get("https://en.wikipedia.org/wiki/Abraham_Lincoln") 
soup = BeautifulSoup(page.content, 'html.parser')
 
all_links = soup.find_all('a')

for link in all_links:
    print(link.get_text(), link.attrs.get('href'))

(View this code on Github for better syntax highlighting)

It’s crude, but these 7 lines of code are the basis of crawling. It gathers all the links on the page by finding all the ‘a’ tags (short for anchor, HTML’s standard tag for hyperlinks). To really “crawl” you can put those links in a queue, and repeat the process until the queue is empty.

Sophisticated crawlers also need to handle special cases such as links you’ve already visited and websites that use Javascript on the front end to populate the content sometime after the initial page load.

Copying content from pages adds a bit more complexity, but not too much. Here’s a crude approximation of what Clearview AI did. Mine harvests images from Wikipedia — rather than LinkedIn — and saves the images to files:

from bs4 import BeautifulSoup
import requests
import shutil

base_url = "https://en.wikipedia.org"
page = requests.get("https://en.wikipedia.org/wiki/Abraham_Lincoln") 
soup = BeautifulSoup(page.content, 'html.parser')
 
all_img_tags = soup.find_all('img')

img_count = 0
for img in all_img_tags:
    img_url = img.attrs.get('src')
    
    # Images on wikipedia have two cases
    if img_url.startswith('//'):
        absolute_url = f'https:{img_url}'
    else:
        absolute_url = base_url + img_url

    response = requests.get(absolute_url, stream=True)
    file_type = response.headers['content-type'].split('/')[-1] # kinda gross, but works.
    with open(f'09-01-2023/img_out/{img_count}.{file_type}', 'wb') as out_file:
        shutil.copyfileobj(response.raw, out_file)
    img_count += 1

(View this code on Github for a more beautiful experience)

GPTBot is designed to scrape the text of news articles from various news websites. Each website operator makes decisions about how to format and present that text, which means GPTBot’s operators need specialized code that extracts this text on a per-website basis. They also probably want to ignore advertisements, image captions, and other ancillary text that might appear.

I made two scrapers to demonstrate that this is also, generally, quite easy. Here’s one that gets the body of CNN news articles:

from bs4 import BeautifulSoup
import requests

# A randomly selected CNN article from the day I wrote this script.
page = requests.get("https://www.cnn.com/2023/08/30/business/san-francisco-union-square-retail-closures/index.html")

soup = BeautifulSoup(page.content, 'html.parser')
individual_p_tags = soup.select('.article__content p')  # CNN's content sits in p tags under a div with this class
texts = [tag.text.strip() for tag in individual_p_tags]
a_text = '\n'.join(texts)

with open('09-01-2023/news_text_out/cnn_article.txt', 'w') as file:
    file.write(a_text)

(Again, the code looks better on Github)

And one for the BBC:

from bs4 import BeautifulSoup
import requests

# A randomly selected BBC article from the day I wrote this script.
page = requests.get("https://www.bbc.com/sport/football/66662060")
soup = BeautifulSoup(page.content, 'html.parser')

# BBC wraps the main body in a div with this class, but uses p's for the text
individual_p_tags = soup.select('.story-body p')
texts = [tag.text for tag in individual_p_tags]
a_text = '\n'.join(texts)

with open('09-01-2023/news_text_out/bbc_article.txt', 'w') as file:
    file.write(a_text)

(Github is 1000x better at displaying this code than my newsletter provider)

My point is that harvesting data from websites is remarkably easy, which makes it appealing to the increasingly data-hungry ML industry.

How Do Companies Stop This?

The bottom line is that if human users can access data using a web browser without some kind of authentication then a bot can too. Web crawlers can be designed such that they are basically indistinguishable to the companies’ servers from a human using a browser. But there are a few options that companies often use to reduce or manage bot traffic.

Robots.txt

The first option is a robots.txt file. These files are more of a request than a defense. Websites use these files to indicate to web crawlers and other bots that certain pages shouldn’t be indexed, viewed, or otherwise accessed by bots. When OpenAI announced GPTBot, they also added documentation for how to modify your site’s robots.txt to prevent GPTBot from accessing certain pages or directories.

When you read that “news organizations are scrambling to block GPTBot” what they mean is that websites are updating their robots.txt file, which is a totally standard and very easy thing for a web developer to do.

Unfortunately, robots.txt only works for bots that choose to respect the specified rules.

Honeypots

Some websites host fake content on a particular URL, then add a line in their robots.txt explicitly banning access to that URL. If anything accesses that URL, sites ban that IP address from accessing the site at all. Here’s a write up describing an implementation of this tactic.

With the ubiquity of VPNs, proxy servers, and other ways to for malicious actors get a new IP address, this can become a game of whack a mole if your adversary is tenacious. Additional user fingerprinting tactics can help somewhat, but are not surefire.

Authentication and reCAPTCHA

I’m sure most of you have done a reCAPTCHA, either by clicking a checkmark, finding the crosswalks in an image, or some similar task. This is a technology built by Google specifically to prevent bots from accessing specific content by hiding that content behind the task.

This process introduces a minor headache for human users, but a major hurdle for bots. Some bots can likely perform some of the reCAPTCHA tasks some of the time, but it makes the bot operators job much harder.

Classic authentication is even better for two reasons. First, signing up and gaining authentication credentials is often a multi-step process that involves an email, a text message, or some other second factor authentication. This process further complicates the automation process. Second, if someone who is authenticated starts behaving like a bot, banning that user is more effective than a simple IP ban, because they’ll have to repeat the signup flow.

While there is no 100% surefire way to prevent people from scraping your online content, a robots.txt, honeypots, authentication, and constant vigilance can substantially reduce successful scraping.

The News Quiz

Every week we challenge ourselves to tie the lesson to the news. Answers are at the end of this newsletter.

A spider crawling on a newspaper.

Web crawling and scraping can be messy. The fact that OpenAI is doing it suggests (at least to me) they can’t get enough high quality training data from more traditional sources (such as “The Pile” and other open NLP datasets). Classify the following issues that can arise specifically from scraping the text of news articles as high risk, medium risk, or low risk from the perspective of a company training a large language model (LLM):

• Accidentally capturing pieces of text that aren’t part of the article, such as advertisements, image captions, pull quotes, embedded links to other articles, etc.
  

• Pulling in native advertising content.
  

• Pulling in articles that were themselves written by an LLM.
  

• Consuming articles with factual errors before they’ve been corrected.
  

• Incorporating copyright protected content into your training data.
  

• Increased cost incurred from the actual process of scraping the data.

Themes in the News

Censorship, Moderation, and Section 230

Lots of recent buzz around these perennial topics.

A judge dismissed a lawsuit brought by the Republican National Committee (RNC) against Google. The lawsuit alleged that Google’s spam filter was biased against Republican candidates’ and officials’ emails. U.S. District Court Judge Daniel Calabretta concluded that the RNC had not “sufficiently pled that Google acted in bad faith.” The RNC plans to refile the lawsuit with an amended complaint.

Meanwhile, the Biden administration urged the Supreme Court to take on cases related to laws in Texas and Florida that substantially limit social media companies ability to perform moderation. The Florida law, for example, imposes fines on social media platforms if they “refuse to transmit” a politicians post, regardless of whether that post violates the company’s content policy.

X (formerly Twitter), Meta, and YouTube have all indicated they will decrease their moderation efforts regarding misinformation. In part this is probably because platforms have realized it’s really hard to do this well. Many of them fumbled the Hunter Biden laptop story, suppressing the NYPost’s original article that turned out to be totally real. The “Twitter files” should also cast doubt on platforms ability and willingness to be neutral and trustworthy.

At the same time, these platforms have an ongoing and increasingly adversarial relationship with news organizations, including various efforts to deprioritize news content on their sites. Many people have come to rely on social media as a news aggregator. Between reduced moderation and deprioritization of legitimate news content, there’s a major void that is being filled with trolls, trashy AI generated content, and other kinds of misinformation.

A report from Rest Of World this week highlighted a fairly predictable outcome of backing off moderation policies as X has done under Elon Musk’s leadership: scams become more prevalent. In this case, sextortion scams targeting prominent Chinese figures on the platform.

So here we are, stuck between platforms that can’t be trusted to moderate particularly well and the free-for-all of the internet that can’t be trusted at all.

Is it good news that OpenAI thinks ChatGPT will soon be able to moderate on social sites?

Teb’s Tidbits

• Ben Evans write a thoughtful piece on the state of Generative AI and copyright.

  • I like the piece, but he gets something important wrong when he claims generative models never store the training data. Researchers have been able to prompt LLMs to produce exact replicas of lengthy passages from books, and Getty Images is currently suing Stability AI for their model that produced exact replicas of copy protected images. This implies that these systems do store training data, albeit in an obfuscated and compressed way within their internal parameters. This fact is important in the copyright discussion, since consumption by an ML system is not always “transformative” in the legal sense.
    

• An argument that the UK’s recently unveiled plans to change their surveillance rules would violate international law.
  

• Meta released Llama Code, an LLM designed specifically to write computer programs.
  

• A fantastic and grounded risk assessment of self driving cars in IEEE Spectrum.
  

• And a pleasant surprise: Apple has come out in favor of California’s Right to Repair Act.

Answers To The News Quiz

• Accidentally capturing pieces of text that aren’t part of the article, such as advertisements, image captions, pull quotes, embedded links to other articles, etc.

  • High Risk: if the training data ends up with lots of incongruent articles with random advertising tidbits interspersed with the real text, this could definitely degrade model performance. Cleaning up the scraped text to remove these errors would be worth the hard work.
    

• Pulling in native advertising content.

  • Low Risk: Native advertising is still a perfectly legitimate use of language even if it’s not particularly high-brow.
    

• Pulling in articles that were themselves written by an LLM.

  • High Risk: It’s well established that training LLMs on text created by LLMs degrades model performance. Making matters worse current methods for discerning if a particular article is AI generated are not reliable at all.
    

• Consuming articles with factual errors before they’ve been corrected.

  • Low Risk: LLM’s ‘hallucinate’ all the time, even if they’ve been trained exclusively on factually accurate data. LLM’s learn to recreate patterns in writing mostly related to word order, not specific facts from pieces of writing.
    

• Incorporating copyright protected content into your training data.

  • Medium to High Risk: Depending on the outcome of some pending lawsuits this might end up being a very big risk or something that can be easily addressed with a license agreement, royalty scheme, or something similar.
    

• Increased cost incurred from the actual process of scraping the data.

  • Low Risk: scraping is generally cheap and easy to perform.

Remember…

From The Lab

Today’s Lesson

How Do AI's "Understand" Words?

Last week we wrote about “The Bitter Lesson,” in which Richard Sutton documents the superiority of “general purpose methods” that scale with computation. A practical consequence of this lesson is that modern Natural Language Processing (NLP) is dominated by statistical models called neural networks. Specifically, a type of neural network called a transformer is the model du jour.

These models are fundamentally numerical in nature. Each neural network is literally just a math function.

In fairness to the models, they are immensely complex math functions. OpenAI’s Large Language Model (LLM) GPT-3 has 175 billion parameters and GPT-4 is rumored to have ~100 trillion. A “parameter” in this context is a number that the model repeatedly changes during the training process. To the extent that a neural network “knows” anything, that knowledge is encoded in these numbers. The parameters are spread across a complex web of mathematical sub-components. But, at the end of the day, these models are still just fancy math functions.

A representation of the Transformer architecture from the paper that first introduced them: https://arxiv.org/pdf/1706.03762.pdf

Words — and natural language in general — are absolutely not numerical or mathematical in nature. Natural language evolved … naturally. Most natural languages have rules, but there are almost always exceptions. Humans, the inventors of natural language, do not internally contextualize words in mathematical terms. But this mismatch hasn’t prevented remarkable progress in the form of ChatGPT, Bard, LLaMA, and other LLMs. So, how do these models reconcile this fundamental mismatch?

The answer is something called an embedding.

Embeddings

Embeddings are a general purpose tactic used to represent some kind of data — words, movies, music — as a vector (i.e. as several numbers). Embeddings can be manually created or be “learned” as part of the training process. Either way, each number in the vector represents some aspect of the thing being embedded. For example, here are three small embeddings manually created by me for encoding TV shows or movies:

A sample of embeddings representing two TV shows and a movie.

When Pandora first launched they hired several Ph.D. music theorists to manually create high-quality and embeddings for the songs in their catalog. These embeddings included fields like acoustic sonority, minor key tonality, vocal centric aesthetic, and other features of songs that the music theorists thought were strongly connected to people’s taste in music. These embeddings were a core part of Pandora’s recommendation engine.

When we embed words the each number corresponds to some aspect of the word. Sometimes these are grammatical in nature such as plurality and being a proper noun. Sometimes they are semantic in nature such as having gender implications, association with various emotions, or association with abstract concepts like royalty, nature, or courage.

For example, the word “queens” is plural, not a proper noun, implies female gender, has a strong relationship with royalty and a weak relationship with nature.

Depending on the size of the embedding more or less meaning can be encoded. GPT-3 uses an embedding length of 12,288 numbers per word, although technically GPT-3 uses “tokens” rather than “words” which breaks some words into parts and allows the system to encode tokens for punctuation marks.

Additionally, for a variety of reasons, GPT-3 (and most LLMs) use embeddings that are learned as part of the training process, rather than manually crafted embeddings like the ones Pandora pioneered. This makes interpreting GPT-3’s embeddings quite difficult, and full of guesswork — but it also allows different neural networks to build embeddings that help with the specific task at hand, which often improves performance.

Shakespeare’s Juliet philosophically asked “what’s in a word?”

Well, at least to ChatGPT the answer is clear: A rose represented by the same 12,288 floating point numbers surely smells just as sweet.

Themes in the News

More on Moore’s Law

A few pieces of news related to last week’s lesson.

Driven largely by ML (although cryptocurrency is also a major contributor) energy use for computation has skyrocketed in recent years. Our chips’ ever increasing capabilities aren’t free. As of right now the computation industry uses about as much energy as all of Britain.

Speaking of Britain, Prime Minister Rishi Sunak wants to buy £100 million worth of NVIDIA chips to further the UK’s position in the global AI race. NVIDIA has become an industry leader in AI not just by building top-end hardware but also by building excellent support to help software engineers get the most out of NVIDIA’s chips.

Some folks are focusing on ways to reduce or mitigate the energy cost of computation, for example this company wants you to heat your water using a powerful server as part of a distributed compute cluster.

Teb’s Tidbits

• A district court confirmed the USPTO’s decision that AI generated content cannot be copyrighted. Though the SCOTUS may ultimately have to weigh in, this is a big deal for the world of generative AI.

  • Meanwhile, several authors are suing Meta for pirating their books.
    

• A vulnerability in a Chinese keyboard app may have exposed an enormous amount of private data.
  

• Microsoft has been savaged by commentators over high severity vulnerabilities and Microsoft’s inadequate response to breaches based on those vulnerabilities.
  

• Maybe we should all be hoping for AI’s that are a bit more mundane.
  

• San Francisco recently allowed Cruise to massively expand their robotaxi offering. This week, following two crashes (one with a firetruck) the city has forced Cruise to cut their cars on the road in half.

Remember…

From The Lab

Today’s Lesson

The Bitter Lesson, Moore’s Law, and a Trade War With China

An underappreciated fact about the machine learning revolution is that its fundamentally about computer hardware. The family of models taking the world by storm — neural networks — were first described in a paper in 1944. At the time Neural networks failed to do anything interesting just way too slow.

Nevertheless, since about 2014 it’s been popular to describe the progress in AI as an “exponential trend” that may soon result in hyper-intelligent AI’s going full Skynet. And it’s true — progress in AI has been exponential. But it’s not the algorithms that’s getting exponentially better, it’s the hardware they run on. Specifically, the number of transistors we can fit on a given area of an integrated circuit has been growing exponentially since the 1960’s.

The exponential growth of transistor count since 1971. Source: https://bjc.edc.org/bjc-r/cur/programming/6-computers/2-history-impact/2-moore.html

This observation, first made by Gordon Moore in 1965, became a guiding principle of computer chip development: The number of transistors that we could fit on an integrated circuit would double roughly every 18 months. This phenomenon is called Moore’s Law and the computer chip manufacturing industry has kept Moore’s Law alive for nearly six decades through incredible feats of engineering such as Extreme Ultraviolet Photolithography (EUV).

In 2019, long time AI researcher Richard Sutton wrote a short paper called “The Bitter Lesson.” Here’s the opening of that paper:

Once upon a time AI researchers approached their work with the assumption that clever algorithms could effectively encode something like human level “understanding” or “intelligence” in a given domain. But Sutton demonstrates in his paper that time and time again, in every domain, “brute force” methods have ultimately come out on top.

This has been true for as long as Moore’s Law has held up, but the chip industry is hurdling towards a few physical limitations that will eventually bring the 60-year “law” to an end, and perhaps soon.

The End of Moore’s Law?

Moore himself once commented, “These are fundamentals I don’t see how we [will] ever get around.”

The fundamentals he’s referring to are: heat, size, and the speed of light.

Heat

Until the early 2000s CPU speeds doubled along with transistor count. Smaller transistors meant electricity was traveling a smaller distance during a single cycle of computation, which allowed computer engineers to increase the clock speed. However, Increasing a CPU’s clock speed also makes it hotter. In about 2005 high-end CPUs started running hot enough to melt critical components of the CPU.

In fact, chip manufactures have all but stopped trying to increase clock speeds (called frequency in the chart below) and started adding more cores instead. Since then high performance computing has increasingly been all about parallelism.

Fun software fact: The main reason Transformers have replaced Recurrent Neural Networks (RNN) as the state of the art in natural language processing is because Transformers are optimized for parallel computing and RNNs are not which means transformers effectively capture the value of these additional cores while RNNs cannot.

This graph shows various how features of high-end computer hardware has changed over time. Source: https://www.researchgate.net/figure/48-Years-of-Microprocessor-Trend-Data-2_fig1_358572677

Size

The current state of the art transistor size is 2 nanometers — roughly the width of 10 carbon atoms. 2nm transistors have not been commercialized yet, but The Taiwan Semiconductor Manufacturing Company (TSMC) says they’ll open the first fab producing 2 nm chips by 2025.

2nm is an incredible feat, but we may not be able to go much further. Assuming the size of a transistor keeps halving, we only have 3 generations before transistors are the size of a single carbon atom. A single-atom transistor actually HAS been created, way back in 2012, it’s just not suitable for making computers: the 2012 single-atom transistor has to be held at negative 196° C to function. Another was invented in 2020 which has it’s own commercialization issues.

Any theoretical subatomic transistor would be a member of the quantum realm. I won’t pretend to be an expert in quantum physics, but the gist is that subatomic sized transistors will be subject to Heisenberg’s uncertainty principle. If that happens we won’t be able to reliably measure the state of the transistor, rendering them useless for computation. If transistors continue to shrink according to Moore’s Law then we’ll hit this quantum limit by 2036.

The Speed of Light

The above factors might not matter if something could travel infinitely fast. Instead of making computers faster by making components smaller, we could just make the information travel faster. We’ll probably get a bit more computational power by switching to light based computers which transmit information faster than electricity based computers. But if the speed of light is indeed a hard limit in this universe, then CPU speeds will be subject to it.

What Does This Have to do China?

Basically, the Biden Administration has learned The Bitter Lesson and is applying it in their increasingly adversarial approach to China. Biden views AI as a major nation security issue, and his administration is trying to limit China’s ability to advance in that domain. The result is an ongoing trade war centered on computer chip production capabilities.

This particular battle began in earnest in October 2022 when the US Bureau of Industry and Security issued a 139 page document which, among other things, expanded export controls. The controls were specifically targeted to limit China’s ability to acquire or build the high-end computer hardware user to train ML systems. Things like EUV Lithography technology and high-end chips made with EUV are among the controlled technologies.

China has retaliated, for example by creating their own export controls on gallium and germanium, which are important raw materials for making computer chips, fiber optics, and related technologies. China also ramped up it’s production of “legacy” chips, which are less powerful; fabricated using different processes; and still very important in computer hardware writ large — but not for state of the art AI research.

Another layer to this story is that the industry leader, The Taiwan Semiconductor Manufacturing Company (TSMC), is located in Taiwan. Tensions between China and Taiwan are high, and Biden has previously indicated that his administration would defend Taiwan from a Chinese invasion if it ever came to that. The TSMC fabs are such an important strategic asset that some war-gamers have suggested that the U.S. would destroy them before allowing China to take control.

We had classes Monday-Thursday. We’re cutting the News Quiz this week because we ran out of time.

Themes in the News

Commodification of Generative Models is Increasing Malicious Uses of AI

From text to audio to images to video, ML models are increasingly able to generate impressive, useful, and realistic content. These models have recently become commodified via open source, pre-trained models, APIs, and other consumer interfaces which dramatically lower the barrier to entry for deploying AI systems.

While there are many useful, cool, and pro-social uses of this technology, we’re witnessing a significant rise in malicious or otherwise unsavory uses, too.

Versions of the GPT architecture trained specifically to create malware, execute phishing attacks, and defraud people have started popping up. In terms of generative cybersecurity there’s enough “there” there for DARPA to get involved: they announced a two-year “AI Cyber Challenge” with nearly $20 million in prize money, and an additional $7 million as startup cash for small businesses who want to participate.

NAO Medical—which runs a series of medical clinics in New York—was caught using large language models to write nonsense articles in order to boost its SEO and rank higher on search engines. It appears they’ve since taken these articles down, but a copy of the article titled “Derek Jeter Herpes Tree: Causes, Symptoms, and Treatment” is retained in the linked Time article.

In a creepy and distasteful—although perhaps not exactly malicious—example some TikTok creators used generative models to recreate the likeness deceased or missing children and have those children tell their stories. Their works include a video where Anne Frank’s likeness first tries to sell you baby clothes then tells Anne Frank’s actual story.

These so called “deepfakes” are popping up in an array of unsavory uses. They’re being used to imitate the likeness of people (mostly women) who wouldn’t otherwise appear in pornography, with major implications for their reputation and privacy.

Audio deepfakes have been used in multiple instances of bank fraud. For example, scammers will convince a bank teller they are someone with authority over the phone by faking the authorities voice. Then they ask said teller to execute a wire transfer.

In general, generative AI tools are making it much easier and faster to create spammy content, accelerating the creation of all sorts of scams and click farming.

Teb’s Tidbits

• Large tech firms are mostly driving AI regulation in the US (no conflict of interest there).
  

• News is a huge part of Google’s revenue, which makes their stance on blocking news in Canada a bit suspicious.
  

• The Kid’s Online Safety Act actually puts us all at risk.

Remember…

From The Lab

Today’s Lesson

Machine Learning and the “-Isms”

Porcha Woodruff was arrested for robbery and carjacking on Feb. 16th, 2023 following a facial recognition match. Woodruff was 8-months pregnant at the time. She was detained for 11 hours then released on a $100,000 personal bond. Her case has since been dismissed and she is now suing the Detroit Police Department (DPD) for damages caused by the false arrest.

This whole ordeal was based on a facial recognition match between gas station surveillance footage and a grainy mugshot of Woodruff from 2015 (Woodruff was arrested in 2015 for driving with an expired license).

The 2015 mugshot matched by the facial recognition system (left) and Porcha’s 2021 drivers license photo (right), which was also available to the facial recognition system, but didn’t match. Via NYTimes: https://www.nytimes.com/2023/08/06/business/facial-recognition-false-arrest.html

According to The New York Times Woodruff is the 6th person—and first woman—to report being arrested due to a false positive from a facial recognition system. All six are black. Coincidence?

Machine learning systems have been plagued with issues related to social bias. Amazon built a sexist hiring AI. Several municipal governments have used racist risk assessment tools to inform bail and sentencing decisions. Microsoft infamously released a Twitter bot that became a Nazi propagandist within 24 hours.

The day before Porcha’s arrest PubMed published this analysis of medical ML systems which concluded, “With the exception of only a few cases, we found that the performance for the White group was, in general, significantly higher than that of the other racial groups across all ML algorithms.”

State of the art generative systems have also fallen under fire for reinforcing a wide variety of stereotypes. Bloomberg recently analyzed the image generator Stable Diffusion and found substantial racial and gender bias. They prompted the AI to depict people with various jobs and categorized the images based on skin tone and perceived gender. In their analysis “lawyers” were mostly depicted as white men and “housekeepers” were mostly women of color.

An analysis done by Bloomberg shows the skin tone composition of images generated by Stable Diffusion when prompted to show a person with a particular job. Source: https://www.bloomberg.com/graphics/2023-generative-ai-bias/

Many people associate algorithms with a kind of pure mathematical objectivity. But now, roughly two decades into the machine learning revolution, more experts are admitting that it’s probably impossible to create an unbiased ML model.

So, why?

It’s The Data, Mostly

Garbage in garbage out.

ML models become biased primarily because social biases are deeply embedded in the datasets used to train them. In a sense the algorithms really are acting as tools of unbiased, objective mathematics: They precisely recreate the problematic patterns in the data.

Amazon used their historical hiring data to train their hiring AI; it’s easy to believe Amazon’s historical hiring practices were at least somewhat sexist. The American court and police systems produced the data that was used to train the risk assessment tool; those systems have a well-established history of racism. Tay was actually intentionally trained by Twitter trolls to spew Nazi propaganda.

The unfortunate truth is that we live in a society where racism, sexism, and other forms of social bias are deeply embedded in many aspects of our culture. Historically accurate data about our society inevitably contains artifacts of these biases. The same goes for the makeup and content of writing, art, pictures, and other artifacts produced by that society.

As a then-Harvard Ph.D candidate (now Ph.D holder) Alex Najibi described in 2020 with regard to facial recognition systems:

Balancing the racial makeup of datasets used to train facial recognition systems would not be terribly difficult. Correcting the long history of camera technology that prioritizes light skin tones is another story.

It’s not possible to retroactively fix every image captured with sub-optimal film or sensor settings. It’s also generally not feasible (and certainly not profitable) for firms to manually collect the millions-to-billions of high-quality images needed to train a modern facial recognition system to ensure proper lighting, color balance, and other settings are used. Most firms just scrape publicly available data from the web.

Even if firms did collect pristine photos, law enforcement typically uses facial recognition to match against low-quality surveillance footage which likely has poor color balance settings and opens the systems up to the extrapolation problem.

Generative systems like Stable Diffusion are holding up an unflattering mirror to society. White men really are over-represented among doctors. Women of color really are over-represented among housekeepers. When Stable Diffusion trains on data that represents reality it recreates discrepancies like these.

Lots of people in online forums use racist language. When we train models like GPT-4 on this data of course it learns to parrot that language.

Making matters worse, modern ML systems require massive amounts of data. Collecting, curating, and cleaning such massive datasets is an enormous task. It’s easy for data cleaners and labelers (who are mostly poorly paid gig workers) to miss subtle forms of bias. If they did succeed at removing every hint of social bias the training sets would shrink dramatically, plausibly making them too small to train large models like OpenAI’s GPT-4 or Google’s Bard.

Engineering Teams, Executives, and End Users Share The Blame

ML practitioners aren’t generally experts with rich experience in all the nuances and subtlety of racism, sexism, homophobia, etc. Exhaustively testing models like ChatGPT for every possible problematic utterance is a already huge challenge, one that’s exacerbated by lack of expertise in the wide world of social bias.

Even when engineering teams do have expertise their concerns about bias and ethics often play a secondary role to building and releasing a profitable product. Sometimes engineers are even punished for concerning themselves with ethics and bias: When Timnet Gebru and Margaret Mitchell authored a paper demonstrating that large language models frequently produce racist content they were ousted from their jobs at Google.

In some cases—such Porcha Woodruff’s arrest—the end users lack crucial skepticism. The Detroit Police Department uncritically trusted their AI’s output, even in the face of mitigating evidence (such as Woodruff’s pregnancy). A judge also appears to have uncritically signed an arrest warrant primarily based on the AI’s output and little supporting evidence.

Finally, some actors are just malicious. With intentional prompting it’s always possible to use even a relatively neutral AI system to produce something that contains or represents social bias, like this:

I prompted an AI to produce a picture of: “A white man gobbling mayonnaise from the jar using a spoon.”

I hope our white readers can laugh at this stereotype. I especially appreciated that the man appears to be sunburned. But similar images that might be perfectly innocuous in some settings might also carry more insidious racist or sexist undertones.

Should AI image generators specifically refuse to create images of black people eating fried chicken and watermelon? Plenty of black people in the real world do eat these foods, but the stereotype that black people prefer them has an ugly history and seems to reappear every Black History Month.

There’s nothing inherently wrong with an image of a black person eating fried chicken, but it’s pretty easy to use that imagery in a way that reinforces nasty stereotypes. Similarly, if you prompted an AI for an image of “A black man eating” and it produced a man with a bucket of fried chicken that would be cause for concern.

Fortunately, the system I used didn’t produce something obviously racist… but it was still a monstrosity. (Seriously, WTF is going on with his mouth and fingers? And are those supposed to be noodles?)

I prompted an AI to create an image of “A black man eating.”

It’s probably impossible to create an ML system that eliminates all social biases while also being generally useful. Which brings us to today’s News Quiz.

The News Quiz

Every week we challenge ourselves to tie the lesson to the news. Answers are at the end of this newsletter.

Image from the recent research paper: https://aclanthology.org/2023.acl-long.656.pdf 

New research studied various text generators and classified them across two political dimensions using a tool called the “political compass” (pictured above).

The color of the circles indicates the model family. Yellow dots are from Google’s BERT family, orange dots are from Meta’s LLaMA family, and white dots are OpenAI’s GPT family. The X-axis is a measure of political economic alignment. The Y-axis is a measure of political social alignment.

The research is awesome and the paper is surprisingly approachable, I definitely suggest you read it. 

Here’s a snippet from the paper:

Which of the following is the most likely explanation for the difference in social conservatism?

• The GPT family’s model architecture is inherently more libertarian.
  

• The engineers at OpenAI are more libertarian than the engineers at Google.
  

• The BERT family of models were trained on a more socially conservative dataset.

This research has implications for a popular “toxic speech classifier” called Jigsaw (and others). Here’s another snip from the research paper:

If the authors are right, Jigsaw’s classifications of what counts as “toxic” must be biased. Which of the following options could Jigsaw peruse to mitigate the bias?

• Train multiple models with different datasets and biases and have them vote or otherwise combine their classifications (this strategy is called “ensembling”).
  

• Gather and curate a dataset that perfectly balances all the relevant perspectives on what constitutes “toxic speech.”
  

• Select a type of model that is inherently non-partisan.

Teb’s Tidbits

The main article and news quiz ran long this week, so we’re skipping the “Themes” section.

• Meta has officially started blocking news content for Canadian readers following a law that would require Meta to pay news producers whose links appear on Meta’s services.
  

• The new California Privacy Protection Agency is investigating cars and their data collection practices.
  

• ML based image generators are disrupting a surprisingly big-dollar cottage industry: YouTube thumbnail creation.
  

• ChatGPT can analyze images and caption them as a tool for the blind, but OpenAI worries about unintentionally creating a facial recognition tool and all the associated privacy and bias issues that entails.
  

• It was bound to happen: Evil ML-powered chatbots are emerging to steal your login credentials and mire you in spam.
  

• A new Russian law aims to crack down on online anonymity and further isolate Russia’s corner of the internet from the broader world.
  

• The BBC is starting a 6-month experiment with their own Mastodon server.

Answers To The News Quiz

Which of the following is the most likely explanation for the difference in social conservatism?

• The GPT family’s model architecture is inherently more libertarian.

  • No research known to me suggests any particular model architecture is inherently biased in one way or another. But, here’s a really interesting quote from the paper:

    We additionally observe that different sizes of the same model family (e.g. ALBERT and BART) could have non-negligible differences in political leanings. We hypothesize that the change is due to a better generalization in large LMs, including overfitting biases in more subtle contexts, resulting in a shift of political leaning.

• The engineers at OpenAI are more liberal/libertarian than the engineers at Google.

  • This might be true. If it is true these biases may have slipped into the verification, training, and testing processes at OpenAI and Google. But it’s probably of secondary importance to the training data. Here’s a quote from the paper:

    Such differences could also be in part attributed to the reinforcement learning with human feedback data adopted in GPT-3 models and beyond.

• The BERT family of models were trained on a more socially conservative dataset.

  • This is the most likely cause. One more quote from the paper:

    This collective difference may be attributed to the composition of pre-training corpora: while the BookCorpus (Zhu et al., 2015) played a significant role in early LM pretraining, Web texts such as Common-Crawl8 and WebText (Radford et al., 2019) have become dominant pretraining corpora in more recent models. Since modern Web texts tend to be more liberal (libertarian) than older book texts (Bell, 2014), it is possible that LMs absorbed this liberal shift in pretraining data.

This research has implications for a popular “toxic speech classifier” called Jigsaw (and others). Here’s another snip from the research paper:

If the authors are right Jigsaw’s classifications of what counts as “toxic” must be biased. Which of the following options could Jigsaw peruse to mitigate the bias?

• Train multiple models with different datasets and biases and have them vote or otherwise combine their classifications (this strategy is called “ensembling”).

  • This solution is recommended by the paper:

    Motivated by existing literature on analyzing different political perspectives in downstream tasks (Akhtar et al., 2020; Flores-Saviaga et al., 2022), we propose using a combination, or ensemble, of pretrained LMs with different political leanings to take advantage of their collective knowledge for downstream tasks. By incorporating multiple LMs representing different perspectives, we can introduce a range of viewpoints into the decision-making process, instead of relying solely on a single perspective represented by a single language model. We evaluate a partisan ensemble approach and report the results in Table 6, which demonstrate that partisan ensemble actively engages diverse political perspectives, leading to improved model performance.

• Gather and curate a dataset that perfectly balances all the relevant perspectives on what constitutes “toxic speech.”

  • The paper suggests this is a) impossible, and b) might help, but won’t ever fully eliminate social bias.
    

• Select a type of model that is inherently non-partisan.

  • Again, there is no research known to me that suggests any type of model is inherently partisan to a particular political persuasion.

Remember…

From The Lab

Today’s Lesson

How ML Fails Part 3: Model Drift

Its a car “drifting” …

Model drift occurs when a model that once worked stops working because something has changed. Model drift is common and can have a variety of underlying causes. A few examples should illustrate why:

1) Netflix released their streaming service in 2007. Ratings you made in 2007 are still in Netflix’s database, but your tastes have probably changed since then. Models they trained in 2007 based on your preferences at that time are simply out of touch with who you are now. To a lesser degree models trained with all your historical ratings likely contain some errors due to your changing tastes.

2) Flashback to 2019 and pretend you were modeling for a large in-person retail store. You had sales forecasts for 2020, but they were based on data generated in 2019’s. Surprise, COVID-19 changed everything and your model “drifted” significantly from the new reality on the ground.

3) Suppose you have a self-driving car system trained on brand new cars. Over time the tires fall out of alignment on some cars. Depending on the severity of the misalignment the system’s turning will be mildly to substantially wrong on those cars.

ML practitioners must regularly acquire up-to-date data, retrain their models, and possibly tune-up their physical systems (as in the car example) to account for such drift. In many situations it’s also wise to retire old data that may no longer be representative of the phenomena at hand.

Other Types of Drift

The most common type of model drift occurs simply because the world is always changing. The above three are examples of this — something about the world changed and the model wasn’t updated to reflect those changes. But there are some trickier causes, too.

Device or Protocol Based Drift

In 2020 Google trained models to use medical imaging to detect diabetic retinopathy. In the lab Google’s model achieved over 90% accuracy. In their first attempt to deploy the model in a real hospital, though, the model struggled with differences between the scans done in the lab vs those in the hospital.

The nurses and techs had a different protocol for taking the scan; the machine that took the scans was slightly different; and the lighting conditions weren’t as good in the hospital. These differences caused the model to reject a large number of scans which in turn caused problems at the hospital for doctors, nurses, and patients.

Differences in microphones, cameras, or other instruments used to create input data for ML models can all create this kind of drift. Sometimes different versions of software and/or different configurations running on those instruments can also cause model drift.

Adversarial Drift

Some ML applications are adversarial in nature. Consider spam filters and fraud detection: when a new model is trained to account for today’s favored spam tactic the spammers are forced to change their strategy. These new tactics are developed specifically to avoid the old filters, so the filter needs to be updated.

This same kind of arms race is happening right now in the world of image generation. Copy protected data is being consumed by generative ML systems in enormous quantities. These data consumption practices are the source of a slew of copyright claims against ML firms. In addition to lawsuits, some artists are turning to adversarial software to protect their work.

Recently, the University of Chicago released a tool called “Glaze.” This tool allows artists to apply an adversarial “cloak” to images they upload to the internet. This “cloak” is a set of subtle perturbations to the original image that humans (generally) cannot see, but that have a substantial impact on the AI’s “understanding” of the image. Specifically the cloak is designed to prevent models from learning key features of the images’ style.

Another new tool called PhotoGuard uses similar adversarial perturbations to make your photos resistant to models that generate “Deep Fakes” (photo-realistic images depicting scenes that didn’t actually occur). Instead of confusing models at training time this tool confuses models at inference time, causing attempted deepfakery on a protected image to instead produce a grey blob or other obviously unrealistic artifacts in the background.

If these techniques become widespread AI models will first start failing because drift has occurred. AI firms will likely try to incorporate some kind of remediation. If the remediation works, then new techniques to confuse the models may have to be developed.

Glaze even addresses this in the “What is Glaze” article:

Low Drift Environments

Some models are less susceptible to drift than others. For example ML modeling has been used to approximate the “three body problem” which is a physics problem about predicting how objects in space will move while taking into account their gravitational influence on each other.

Classical numerical methods for doing this are extremely computationally intense. Once trained, ML models can generate highly accurate approximations that are up to 100 million times faster to produce. This model is not going to suffer from drift because (as far as this writer knows) the laws of physics have not ever changed.

The recent strides ML has made in weather forecasting are similar: climate change is occurring but it’s a relatively slow process. Even then, the fundamental physical laws governing the weather aren’t changing, only the environment in which those laws are being applied (i.e. Earth’s atmosphere).

Problems like these are ideal for ML because they are highly objective, lend themselves easily to mathematical explanations, and aren’t likely to suffer from drift.

The News Quiz

Every week we challenge ourselves to tie the lesson to the news. Answers are at the end of this newsletter.

OpenAI shuttered an ML program designed to discriminate between AI-written text and human-written text. Many people are trying to create such tools but so far failure is the norm.

For each of the following bullet points decide whether its an example of model drift or something else:

• These systems flag text from non-native English speakers as “AI Generated” at much higher rates.
  

• Making slight tweaks to AI generated text caused accuracy on one tool to plummet as described here:

  [Researchers] found that while the tools were good at identifying text written by a human (with 96% accuracy, on average), they fared more poorly when it came to spotting AI-generated text, especially when it had been edited. Although the tools identified ChatGPT text with 74% accuracy, this fell to 42% when the ChatGPT-generated text had been tweaked slightly.

  • This tool (code) released once upon a time by OpenAI can usually detect text produced by GPT-2, but not GPT-3.5, GPT-4, or ChatGPT.

Themes in the News

Alternatives to Detecting AI Generated Materials

As we highlighted in the News Quiz detecting AI written text is difficult and unreliable. Now researchers are exploring alternative methods for distinguishing between human and AI created content:

• Provenance based methods seek to establish a digital (and cryptographically verifiable) trail that can prove the origin of an image, audio file, or video. The Coalition for Content Provenance and Authenticity (C2PA) is one such protocol that’s gaining notoriety.
  

• Watermarking methods cause AI-produced content to carry a signature marking it as such. Several major AI firms have signed onto an agreement to add watermarks to their AI generated images. And here’s a proposal to give AI text generation systems a unique subset of Unicode as a form of watermarking for text.

Data Labeling and Generation is Big Business

ML models are notoriously data hungry. So much so that some researchers think we’ll run out of high-quality data to train them before 2027. Complicating matters, new research has confirmed that training ML models using data generated by ML models almost always degrades model performance.

There’s a huge market for high quality data, and a lot of it is created by poorly paid laborers. Some of it is generated by you: every time you “prove you’re not a robot” with one of Googles (Re)Captcha tools you’re helping to label training data.

It’s not all exploitation though. Read this lovely story about a startup in India trying to pay fair wages and royalties for data labelers.

Teb’s Tidbits

• Content moderation should abandon their “legalistic” structures and embrace the “public editor” role.
  

• Google’s machine “unlearning” challenge seeks to address some privacy issues in ML models by making them forget some aspects of their training data retroactively.
  

• Deepfakes in the advertising industry exemplify why the SAG-AFTRA has made AI a core aspect of their ongoing strike.

Answers To The News Quiz

For each of the following bullet points decide whether its an example of model drift or something else:

• These systems flag text from non-native English speakers as “AI Generated” at much higher rates.

  • Something else. Model drift in this context would be a model that used to accurately distinguish between ESL writers and AI writers, but overtime one of those groups writing changed in a way that caused the model to fail. More likely this is plain old underfitting.
    

• Making slight tweaks to AI generated text caused accuracy on one tool to plummet.

  • Model drift. This is a great example of adversarial drift. The researchers took AI generated output, made an intentional perturbation, and then the model failed on the data which had “drifted” from the original distribution of AI generated text.
    

• This tool (code) released once upon a time by OpenAI can usually detect text produced by GPT-2, but not GPT-3.5, GPT-4, or ChatGPT.

  • Model drift. This time due to the advances in text generation. AI systems have gotten much better in the last 5 years, so old tools that detected old AI generated text have stopped working.

Remember…