On behalf of Team Lumiera

Emma - Business Strategist
Allegra - Data Specialist

Lumiera has gathered the brightest people from the technology and policy sectors to give you top-quality advice so you can navigate the new AI Era.

Follow the carefully curated Lumiera podcast playlist to stay informed and challenged on all things AI.

🗞️ Issue 84 // ⏱️ Read Time: 5 min

Hello 👋

While the most obvious focus is privacy concerns around social media and big tech, a quieter revolution has been taking place. Surveillance capabilities are creeping into the most unexpected corners of our lives: from our kitchen counters, to our virtual meetings, to government enforcement operations. Who would have thought these three seemingly different matters have so much in common?

This isn't the dramatic dystopia of science fiction; it's the gradual, almost invisible expansion of data collection that's reshaping our relationship with privacy.

The Kitchen Spy Network

Smart appliances now request microphone access and precise location data to cook food. AI transcription services process everyone's speech, including non-subscribers, to train their models. Government agencies repurpose biometric tools designed for identifying inmates to support immigration enforcement.

The common denominator isn't the technology itself, but the incremental expansion of surveillance capabilities. Each step feels logical in isolation: Weather-based cooking suggestions, improved transcription accuracy, enhanced public safety. But collectively, they represent a fundamental shift from surveillance as exception to surveillance as assumption, something known as surveillance creep: The gradual expansion of data collection beyond what's necessary or expected. What starts as "smart connectivity for convenience" becomes a data harvesting operation that most users never consented to in any meaningful way.

The Business Risk: More than 75% of consumers say they won't purchase from brands they don't trust with their data. When your business practices contribute to surveillance creep, you're not just collecting data—you're eroding the trust that underpins customer relationships. Recent litigation against companies like Otter.ai and Ring demonstrates growing willingness to hold organizations accountable for privacy overreach.

The Consent Illusion: These examples share a critical flaw: The manipulation of consent. Apps bundle unnecessary permissions with essential functions. Default settings favor data collection over privacy. One person's agreement (an Otter subscriber) becomes consent for everyone affected (other meeting participants). The result is surveillance that's technically consensual but practically inevitable.

Once data is collected and shared, the ability to control, delete, or even know how it's being used becomes abstract, making it almost impossible to reclaim digital privacy. This loss of control creates a persistent sense of vulnerability that can fundamentally alter how people behave in their own homes.

The United States Immigration and Customs Enforcement (ICE) recently announced plans to purchase mobile iris scanning technology that was originally created for sheriff's offices to identify inmates and known persons. This is an example of how government agencies adopt surveillance technologies that were designed for entirely different purposes, which is known as function creep.

The Business Risk Hidden in Plain Sight

What makes surveillance creep particularly dangerous for organizations is its incremental nature. Each small expansion seems reasonable in isolation: Weather-based cooking suggestions, improved transcription accuracy, enhanced public safety. But these seemingly logical steps create compounding business risks that many leaders haven't fully considered.

We're moving from a world where surveillance was exceptional to one where it's assumed, and customers are increasingly aware of this change. More than 75% of consumers say they will not purchase from a brand or provider they do not trust with their data. When your business practices contribute to surveillance creep, you're not just collecting data. You're also eroding the trust that underpins customer relationships.

Data collection: Is it voluntary? 

Central to surveillance creep is the manipulation of consent. Companies and agencies have become sophisticated at making data collection seem voluntary while making it practically unavoidable. 

Consent manipulation: Common patterns

Bundled Permissions: Apps request multiple permissions at once, making it difficult to separate necessary from unnecessary access.

Default Settings: Privacy-protective options exist but aren't enabled by default, requiring users to actively seek them out.

Functional Coercion: Essential features are disabled unless users agree to data collection that goes far beyond what's needed for basic functionality.

Proxy Consent: One person's agreement (like an Otter AI subscriber) becomes consent for everyone affected (other meeting participants).

These patterns create the illusion of choice while structurally favoring data collection. The result is surveillance that's technically consensual but practically inevitable.

See the pattern: Surveillance Creep

Understanding surveillance creep requires recognizing common warning signs:

Disproportionate Permissions: When a device or service requests access that seems unrelated to its core function, like an air fryer wanting microphone access.

Vague Privacy Policies: When companies use broad language about "improving services" or "enhancing user experience" without specifying how data is actually used.

Default Data Sharing: When privacy-protective settings exist but aren't enabled by default.

Third-Party Infiltration: When using one service results in data collection by other companies you never directly interacted with.

Function Expansion: When technologies designed for specific purposes gradually expand into broader surveillance applications.

The Broader Social Transformation

The effects of surveillance creep extend far beyond individual discomfort, reshaping society itself. When monitoring becomes normalized across everyday technologies, we see profound collective changes:

Erosion of Public Trust: General distrust in technology and institutions increases when people feel their privacy is routinely invaded, making them hesitant to adopt beneficial innovations and eroding confidence in those responsible for protecting rights and public welfare.

The Chilling Effect: People begin to censor their behavior and communication. This self-censorship limits free expression, creativity, and authentic interaction, potentially stifling dissent and unconventional thinking.

Collective Anxiety: Ambient stress rises as society adapts to omnipresent monitoring.

Shifting Social Norms: The awareness that "someone is always watching" pushes society toward conformity, discourages risk-taking, and reduces openness in personal interactions.

Democratic Vulnerability: As monitoring becomes normalized, democratic norms like free association, confidential communication, and the right to dissent may steadily weaken.

Trust as Your Competitive Advantage

This privacy erosion creates a significant business opportunity for leaders willing to move first. While competitors normalize surveillance, organizations that prioritize genuine privacy protection can differentiate themselves in increasingly crowded markets.

Rather than viewing privacy as a constraint, leading companies are discovering it drives innovation. Signal built a messaging empire on encryption. Apple has made privacy a brand differentiator. DuckDuckGo carved out search market share by not tracking users. Think about it like this: Instead of asking "How much data can we collect?", instead ask "How little data do we need to deliver exceptional value?"

Understanding how surveillance expands helps us resist it more effectively. We can demand that convenience features truly be optional, that consent be meaningful rather than performative, and that surveillance technologies be subject to ongoing democratic oversight rather than quiet expansion.

Until next time.
On behalf of Team Lumiera

Emma - Business Strategist
Allegra - Data Specialist

Lumiera has gathered the brightest people from the technology and policy sectors to give you top-quality advice so you can navigate the new AI Era.

Follow the carefully curated Lumiera podcast playlist to stay informed and challenged on all things AI.

🗞️ Issue 83 // ⏱️ Read Time: 5 min

Hello 👋

This week, we’re unpacking the story behind GPT-5. Despite impressive technical achievements - 94.6% accuracy on advanced mathematics problems and 74.9% on coding benchmarks - the launch became a case study in how disconnected metrics can be from user experience when human factors are overlooked. While the model promised "PhD-level" capabilities, its rollout revealed critical gaps in how we manage AI transitions when users have developed trust and dependency on existing systems. For leaders and teams, it’s a case study in how tech rollouts can go wrong when we ignore real-world usage and the human factor.

Latest Not Greatest

On paper, the August GPT-5 launch from OpenAI was positioned as a leap ahead: improved reasoning, fewer hallucinations, and advanced coding skills. But OpenAI’s decision to abruptly replace GPT-4o without warning triggered a user revolt that forced the company to restore the previous model within 24 hours.

GPT-5 ranks high in the Long Context Reasoning Benchmark, showing that the model matters, but how long it thinks is equally important

GPT-5 implemented a unified system architecture that consolidates multiple models behind a single interface. Rather than letting users choose between discrete models, the system employs an automatic router (autoswitcher) that selects computational approaches based on query analysis. While this approach offers potential efficiency gains, directing simple queries to faster models while reserving expensive reasoning for complex tasks, it fundamentally changed how users interact with the system.

The GPT-5 reaction stemmed from a misreading of user psychology rather than a technical failure, and the response translated immediately into business impact. Reddit threads documenting user complaints received 4,600+ upvotes and 1,700+ comments.

The level of sycophancy, meaning how much an LLM agrees with the user,
was reduced in GPT 5 in order to make the model less manipulative and more truthful.

Research reveals why these attachments formed: OpenAI had previously reduced "sycophantic behavior" to make models more truthful, but users specifically requested the return of more supportive responses, being used to having ChatGPT being their very own Yes Man. As Altman revealed, some users had "never received genuine encouragement from anyone in their lives before", making ChatGPT's supportive tone a crucial source of emotional validation.

This dependency creates a complex ethical challenge, something we previously touched upon when we wrote about AI companions and human behaviour. While supportive AI can provide genuine mental health benefits, it also creates vulnerability when systems change without warning. The psychological impact demonstrates that responsible AI development must consider emotional well-being as seriously as technical performance.

This revealed that emotional attachment to AI models creates genuine business risk for the companies behind them. Users canceled subscriptions after losing what they described as a “trusted friend,” which highlights a shift in how organizations must evaluate AI deployments.

The Lack of Predictability and Its Business Impact

Imagine you had built your latest AI agent or enterprise product on OpenAI’s older models, leveraging 4o for creativity and o3-Pro for research. Overnight, you would have faced unanticipated workflow disruptions and had to scramble for alternatives or wait for OpenAI to revert its decision.

Security testing also revealed deeper problems beyond personality changes. Two prominent AI security firms, SPLX and NeuralTrust, found GPT-5 “nearly unusable for enterprise out of the box.” Their results showed that GPT-4o remains the most robust model under red teaming.

The technical issues compounded user frustration. GPT-5's broken "router" system randomly switched between model capabilities mid-conversation, making business workflows unpredictable. Users reported that the system sometimes delivered cutting-edge AI responses and sometimes inferior results within the same conversation thread, undermining reliability for professional use cases.

Higher User Agency = Better Transitions

OpenAI's automatic routing system made decisions about which model to use without user awareness or control, creating an opaque experience where users couldn't understand or predict their AI interactions. Perhaps most concerning from a responsible AI perspective was the lack of user agency in the transition. The automatic routing system made critical decisions without user awareness:

• Users couldn't predict which computational engine would handle their queries

• No mechanism existed to understand why responses varied in quality or style

• The system learned from user behavior without explicit consent or explanation

Reddit thread: “GPT5 is Horrible”

OpenAI’s Strategic Bet

So, why did OpenAI do this? Why not keep old models even when new ones are “better”?

• Shift to Product: OpenAI implemented an automatic system that decides which AI capability to use for each query. This is classic product thinking: Hide the technical complexity behind a seamless user experience.

• Cost and Infrastructure Pressure: Running multiple models simultaneously is expensive - each model requires dedicated server capacity, and OpenAI's API traffic doubled in the 24 hours after GPT-5's launch, creating massive infrastructure strain.

• The "iPhone Strategy": When Apple releases a new iPhone, they don't keep manufacturing the old one at the same price point. The idea was to force users onto the "better" model to drive adoption and justify the development investment.

• Technical Complexity of Multi-Model Support: Maintaining multiple models creates complex engineering challenges around routing, version control, and user experience.

Course Correction within 24 hours

The crisis immediately benefited competitors. Prediction markets showed users expecting Google to lead by the month's end. Claude and other alternatives gained attention. In AI's winner-take-all dynamics, deployment failures can rapidly shift market position.

So naturally, OpenAI moved fast:

• Reinstated legacy models (GPT-4o, o3, o4-mini, etc.) in the model picker for Plus/Pro users.

• Said they’re making some interventions to how the decision boundary works that should help users get the right model more often.

• Claimed they’ll make it more transparent about which model is answering a given query.

• Will change the UI to make it easier to manually trigger thinking.

• Altman vowed to preserve “warmth” as well as technical rigor, with user customization on the horizon

Current enterprise AI evaluation methods focus heavily on technical benchmarks, cost analysis, and risk assessment while largely ignoring user emotional response and attachment factors.

Action points: Moving forward

Ok, so enough with the analysis. Let’s move on and turn this into something real: Action. Execution. Call it whatever you want, but what we’re focusing on here is: What could have been done differently, and what can company leaders like yourself learn from this, all in 5 simple steps.

OpenAI's GPT-5 launch demonstrates how even groundbreaking technical achievements can fail when deployment prioritizes operational efficiency over user autonomy, transparency, and established relationships. The lesson isn't "don't upgrade AI systems." It's that AI product management requires fundamentally different approaches. Success depends on managing relationships, not just optimizing benchmarks.

Until next time.
On behalf of Team Lumiera

Emma - Business Strategist
Allegra - Data Specialist

Lumiera has gathered the brightest people from the technology and policy sectors to give you top-quality advice so you can navigate the new AI Era.

Follow the carefully curated Lumiera podcast playlist to stay informed and challenged on all things AI.

🗞️ Issue 82 // ⏱️ Read Time: 5 min

Hello 👋

The U.S. approach to AI has taken a dramatic turn. Where the Biden Administration emphasized safety testing and careful risk evaluation, the Trump Administration's new AI Action Plan declares a different priority: "win the AI race" against China, with safety regulations framed as obstacles to victory.

Released on July 23rd, 2025, the plan explicitly presents this as a binary choice: either accept the risks of rapid AI deployment or lose technological leadership to adversaries. For business leaders, this shift creates immediate compliance challenges and strategic decisions that could define competitive advantage for years.

Today, we’ll cover what the plan says, the near-term business impact, cross-industry responses, and a few potential outcomes.

What the Plan Says

America’s AI Action Plan organizes around three core pillars:

Pillar I: Accelerate AI Innovation focuses on removing regulatory barriers:

• Rescinding Biden-era AI safety requirements

• Updating federal procurement to exclude AI systems that reference "misinformation, Diversity, Equity, and Inclusion, and climate change."

• Cutting federal funding to states with "burdensome AI regulations"

• Encouraging open-source and open-weight AI

🚨Claiming objectivity and free speech while actively excluding references that don’t align with the White House’s views is an innate contradiction. 

Pillar II: Build American AI Infrastructure promises rapid infrastructure development:

• New environmental exemptions specifically for AI infrastructure 

• Federal lands made available for data center construction

• Streamlined permitting that bypasses traditional environmental reviews

• Workforce training programs for AI-related jobs

🌎 Want to read more about the environmental impact of AI? Check out our 3-part series.

Pillar III: Lead in International AI Diplomacy and Security targets global competition:

• "Full-stack AI export packages" for allies willing to join America's AI alliance

• Enhanced export controls on AI chips and technology to China

• Leveraging international standards bodies to promote "American values"

• Evaluating frontier AI models for national security risks

ℹ️ The full AI tech stack includes hardware, models, software, applications, and standards.

While we’re not covering this today, it’s important to know that the US AI Action Plan fits into a larger global narrative, including the subsequent release of China’s AI Action Plan.

The Business Impact: What Changes ASAP

Three executive orders were signed alongside the plan:

1. Preventing Woke AI in the Federal Government → Federal Contracting Changes:

   1. Government agencies must evaluate AI vendors for "ideological bias." This affects billions in federal contracts and creates potential conflicts for companies operating across states with varying AI regulations. Timeline: Government guidance by November 20, 2025

2. Accelerating Federal Permitting of Data Center Infrastructure → Expedited Infrastructure.

   1. Data center developers gain expedited permitting and federal land access, while facing new security requirements around foreign technology. Timeline: EPA guidance by January 19, 2026

3. Promoting the Export of the American AI Technology Stack → Deepening Reliance.

   1. Bundled U.S. AI exports could deepen international reliance on American technology. European tech businesses argue this could channel significant European investment into US industries rather than supporting European capacity building. Timeline: American AI Exports Program by November 20, 2025

Industry Reactions: The Divide

👍 Support: TechNet (aka "Tech's most powerful advocacy group") praised the plan's focus on "removing regulatory barriers to innovation.”  NetChoice called it "night and day" compared to the previous administration's "command and control" approach.

👎 Opposition: The ACLU called the state preemption effort "harmful" and legally questionable, and researchers worry that eliminating references to climate change and bias undermines scientific integrity.

What Happens Next

Business leaders face a fundamental choice:

1. Optimize for federal unregulated incentives 

2. Maintain stakeholder trust through responsible AI practices,

3. OR try to navigate both simultaneously.

The plan's most striking contradiction lies in promising "objective" AI while making explicitly ideological choices about what topics AI systems can consider. This reveals the challenge of technology governance: who defines objectivity, and should governments make that determination?

Until next time.
On behalf of Team Lumiera

Emma - Business Strategist
Allegra - Data Specialist

Lumiera has gathered the brightest people from the technology and policy sectors to give you top-quality advice so you can navigate the new AI Era.

Follow the carefully curated Lumiera podcast playlist to stay informed and challenged on all things AI.

Was this email forwarded to you? Sign up here

🗞️ Issue 81 // ⏱️ Read Time: 5 min

Hello 👋

Lumiera’s AI leadership program for executives who demand real-world impact, The AI Leadership Accelerator, is up and running with the July cohort of senior leaders who are defining the future of technology. Why are leaders turning to us to boost their AI Leadership? Continue reading to learn more.

The Lumiera AI Leadership Accelerator addresses the leadership gap directly.

This isn't another AI overview course; it's an intensive 8-week program designed to transform senior leaders into confident AI strategists who drive measurable business impact. This is not only theory - we focus on action and real industry input.

Program Overview →

The numbers tell a compelling story. AI spending exploded from $2.3 billion in 2023 to $13.8 billion in 2024, three times more employees are using generative AI for significant portions of their work than leaders realize. And yet, only 1% of business leaders report that their companies have reached AI maturity. The infrastructure is there, the workforce is ready, but something critical is missing.

That missing piece? Strategic AI leadership.

Why We Built This: Top Leadership Painpoints

Organizations are stuck because leaders face unprecedented challenges that traditional management training never addressed. Decision paralysis strikes when executives are expected to lead AI transformation without foundational frameworks to evaluate options effectively. Resource strain intensifies as leaders struggle to research and plan AI initiatives while maintaining day-to-day operations.

In the end, organizations either rush into costly mistakes or remain paralyzed by indecision. Both approaches create strategic disadvantage.

But, this isn’t just about business. Lumiera exists to ensure a future that’s equipped for humanity. This program aims to shift the dominating narrative of AI and shape a better future of technology through informed decision-making and responsible practices.

Lumiera [ˈlumiˈɪərə]: Lumi (light/bright) and Era (future).

The Business Case: Strategy Makes the Difference

The data is unambiguous: formal AI strategy transforms outcomes. Organizations without a structured AI strategy achieve only a 37% success rate for AI adoption initiatives. Those with formal AI strategies see an 80% success rate for enterprise-wide adoption, more than doubling their likelihood of success.

This isn't about having more resources or better technology. It's about leadership capability. The most successful AI implementations share a common characteristic: leaders who can bridge the gap between AI potential and organizational reality.

Where Successful Leaders Stand Out:

The companies achieving transformational results aren't necessarily the ones with the biggest AI budgets; they're the ones with leaders who understand how to navigate AI adoption systematically:

• First, they focus. They invest their time where it matters.

• Standing by without taking a stance, while AI rapidly advances, will not yield the returns everyone imagines AI brings.

• Success takes dedication to informed leadership.

What sets this program apart

The Lumiera AI Leadership Accelerator is built on proven frameworks developed through real AI transformation cases across industries from finance to healthcare. The curriculum is organized around three core foundations that consistently separate successful AI initiatives from failed experiments: building confidence, gaining competitive advantage, and achieving ROI on AI initiatives.

→ Immediate Application Focus: Every framework and methodology is applied to your actual business challenges during the program. You'll develop concrete AI strategies, implementation roadmaps, and measurement systems for your specific organizational context. Participants leave The Accelerator as Certified AI Leaders, with a comprehensive toolkit of practical, applicable knowledge.

→ Limited Cohort Size: Maximum 15 senior leaders per cohort ensures personalized attention and meaningful peer learning opportunities. This isn't a mass online course. It's a hands-on development experience.

→ Strategic-Technical Integration: The program uniquely combines critical technical understanding with business strategy execution, a rare combination that sets graduates apart from leaders who understand either AI capabilities or business strategy, but not how to connect them effectively.

For questions about the program or to discuss whether it's the right fit for your strategic needs, reach out to our team directly. We're here to help you navigate the path from AI awareness to AI advantage.

Until next time.
On behalf of Team Lumiera

Emma - Business Strategist
Allegra - Data Specialist

Lumiera has gathered the brightest people from the technology and policy sectors to give you top-quality advice so you can navigate the new AI Era.

Follow the carefully curated Lumiera podcast playlist to stay informed and challenged on all things AI.

🗞️ Issue 80 // ⏱️ Read Time: 7 min

Hello 👋

The battle lines are drawn in Brussels. As the EU pushes forward with implementing the AI Act, Meta's recent refusal to sign the EU's Code of Practice for AI has sparked a debate that goes far beyond legal compliance: It's a fundamental clash over who gets to shape the future of AI governance.

The Regulatory Landscape: Setting the Stage

The European Union's AI Act, now in force, represents the attempt to prevent what regulators call a "race to the bottom" in artificial intelligence development. The legislation aims to ensure AI systems are developed and deployed safely, transparently, and in ways that protect fundamental rights.

However, the path to implementation hasn't been smooth. Major European technology companies, including ASML, SAP, and Mistral AI, along with over 110 other organizations, have formally asked the European Union to delay implementation. In a letter to European Commission President Ursula von der Leyen, these business leaders argue that the forthcoming regulations could jeopardize the EU's position in global AI innovation and call for a more "innovation-friendly regulatory approach."

Despite this industry pushback, the European Commission announced that implementation will continue as planned and launched the EU's Code of Practice as a bridge between regulatory requirements and practical compliance.

Understanding the AI Code of Practice

The EU's AI Code of Practice serves as a voluntary framework designed to help companies implement the AI Act's requirements:

• It requires companies to provide comprehensive documentation about their AI tools, including how models are trained, what data is used, and how risks are managed. 

• Crucially, it bans training AI on pirated content and requires compliance with content owners' requests to exclude their works from datasets.

• The code aims to create industry standards that promote transparency, accountability, and responsible innovation while building public trust in AI systems. 

• For society, it promises greater transparency and protection from potentially harmful AI systems.

Meta's Public Position: The Official Reasons

Meta's refusal to sign the GPAI came through Joel Kaplan, the company's Chief Global Affairs Officer, in a LinkedIn post that sent ripples through the tech industry. 

Kaplan outlines three primary objections. Let’s take a closer look at what it actually means.

Legal Uncertainty: Meta argues the code introduces numerous legal uncertainties, claiming requirements are vaguely defined and could expose them to unexpected legal risks.

Regulatory Overreach: The company claims the code includes "measures which go far beyond the scope of the AI Act," suggesting regulators are using the voluntary framework to impose requirements not explicitly mandated in the original legislation.

Innovation Concerns: Meta echoes the broader industry concern that "this over-reach will throttle the development and deployment of frontier AI models in Europe."

The False Dichotomy: Why the “Regulation Kills Innovation" Argument Doesn't Hold Up

Meta's resistance relies heavily on a familiar industry talking point: that regulation inherently stifles innovation. This narrative, while seductive in its simplicity, represents a false dichotomy that obscures a far more nuanced reality.

The Standard Industry Playbook

The "regulation hates innovation" argument follows a predictable pattern:

1. Companies claim compliance costs divert resources from R&D,

2. Regulatory uncertainty makes long-term planning impossible, and

3. Restrictive rules limit technological exploration.

This framing positions companies as champions of progress fighting bureaucratic obstacles while casting regulators as risk-averse bureaucrats who don't understand technology.

The Research Tells a Different Story

However, decades of research reveal that well-designed regulation can actually foster innovation by setting clear goals that act as "demand signals," building public trust essential for adoption, preventing races to the bottom, and spurring compliance-driven innovation. For example, The Montreal Protocol banning ozone-depleting substances spurred innovation in refrigeration and aerosol technologies, creating entirely new chemical industries

Why Companies Keep Using This Argument

Companies continue deploying this argument because regulation creates immediate, visible compliance costs while innovation benefits emerge over longer time horizons. Additionally, early compliance can disadvantage first-movers compared to competitors who delay, creating incentives to resist collectively rather than compete on responsibility.

Understanding the Strategic Calculations

Meta's resistance reveals calculations extending far beyond immediate compliance requirements. The company's business model depends on maximizing scale and speed to market with minimal oversight. The code's documentation and transparency requirements create friction that could slow development and potentially reveal competitive advantages.

Meta appears to be testing EU enforcement limits while negotiating from a position of strength. By aligning with other dissenting voices, Meta is betting Brussels might soften requirements to avoid losing major tech players.

However, this approach creates fundamental misalignment with market dynamics. Meta's resistance comes when the company already faces significant trust deficits from scandals like Cambridge Analytica. The EU framework offers a pathway to rebuild public confidence through documented responsible practices, yet Meta is rejecting this opportunity. This creates strategic openings for competitors willing to embrace transparency as a competitive differentiator, which is particularly important as users increasingly demand responsible AI development and enterprise customers require compliance capabilities.

What This Means for the Future

Meta's strategic resistance represents more than a single company's regulatory gambit, it's a test case for how global AI governance will evolve. The outcome will influence whether voluntary compliance mechanisms can work alongside mandatory regulations.

If other major tech companies follow Meta's lead, it could undermine the EU's attempt to establish Europe as a global standard-setter for responsible AI. Conversely, if market pressures eventually force Meta to participate, it could cement the Brussels Effect in AI governance, where EU regulations become de facto global standards.

Until next time.
On behalf of Team Lumiera

Emma - Business Strategist
Allegra - Data Specialist

Lumiera has gathered the brightest people from the technology and policy sectors to give you top-quality advice so you can navigate the new AI Era.

Follow the carefully curated Lumiera podcast playlist to stay informed and challenged on all things AI.

🗞️ Issue 79 // ⏱️ Read Time: 6 min

Hello 👋

Today, we’re getting physical. That is to say, we’ll be discussing the physical hardware that powers AI and the business around it. If you read our insights from Paris last week, you'll know that we attended a conference where some of the biggest players in AI infrastructure were on stage, talking about the current state and future of AI.

Most coverage misses that this isn't really about the technology. It's about power, control, and who gets to shape the future of business.

We're approaching this from a business strategy perspective, not a technical deep dive. Think "what this means for my AI roadmap" rather than "how the chips work." By the end of this newsletter, you'll know which infrastructure announcements actually affect your AI strategy and which are just noise.

Let's decode what's really happening.

Understanding the AI Infrastructure Stack

Think of AI infrastructure like a city's foundation systems. Just as you don't think about power grids until the lights go out, most businesses don't think about AI infrastructure until they can't get what they need. The full stack consists of four different layers: Physical, platform, distribution, and access. Let’s get into the details.

🧱 The Physical Layer: Who Controls the Hardware

This is where the raw computational power lives – specialized chips and the massive data centers that house them. Control at this layer determines who has the foundational compute resources for modern AI. 

Key players include:

• NVIDIA is the incumbent company

• AMD is the challenger betting on openness & flexible integration

• Groq is the speed specialist focusing on ultra-fast AI responses

☁️ The Platform Layer: Who Controls Access

This layer encompasses the cloud platforms and developer frameworks that determine how companies access and build with AI. Control here shapes the developer experience, integration options, and default tooling. 

Key players include:

• Microsoft leverages its OpenAI partnership

• Amazon builds custom chips while maintaining platform dominance

• Google uses proprietary processors and research leadership

🚛 The Distribution Layer: Who Controls Delivery

This covers how AI actually reaches end users: The networks and edge infrastructure that impact performance, latency, and reliability. When your AI takes 3 seconds to respond instead of 100 milliseconds, that's a broken user experience in real time. This layer determines whether your AI investments translate to competitive advantage. Control here means deciding response times and geographic availability for AI applications. Key players include:

• Cloudflare brings AI processing closer to users via a global edge network

• Traditional Content Delivery Networks (CDNs) are adapting their established networks for AI workloads

🛂 The Access Layer: Who Sets the Rules

This layer defines pricing models, service agreements, and overall market dynamics. Essentially, who gets what capabilities and at what cost. Control here shapes economic access to AI. Key players include:

• CoreWeave is a specialized provider focused exclusively on AI workloads

• Traditional Cloud Giants like Microsoft, Amazon, and Google each develop pricing strategies and vendor lock-in approaches

Understanding the Power Game

When you see infrastructure news, ask yourself: What type of control is this company trying to establish?

Ecosystem Lock-in Strategies: NVIDIA doesn't just sell chips. They've built CUDA, a software ecosystem that makes it expensive to switch to competitors. When your development team learns NVIDIA's tools, your company becomes more likely to buy NVIDIA's future products.

Vertical Integration Plays: Google, Amazon, and Microsoft aren't just buying chips, they're designing their own. This isn't about saving money; it's about reducing dependence on NVIDIA and gaining control over their own destiny. When a cloud provider controls the full stack (vertical integration), they can optimize for their specific services and lock you into their platform.

Platform Standardization Battles: Different companies are pushing different standards for how AI systems communicate and operate. The winner doesn't just get market share, they get to influence how the entire industry develops.

What to Watch For in Infrastructure News

Not all infrastructure announcements are created equal. Here's how to spot what actually matters for your business:

🔴 Red Flag – Pure Performance Claims: Headlines like "New Chip 10x Faster!" usually matter less than they seem. When NVIDIA announced their H100 was "30x faster," most businesses saw no meaningful performance improvement in their actual applications. What matters more: cost-performance ratios and ecosystem compatibility.

🟢 Green Flag – Ecosystem Moves: Pay attention when companies announce partnerships, new developer tools, or standardization efforts. Microsoft's GitHub Copilot integration mattered more than raw Azure compute improvements because it changed how developers actually work.

🚨 Critical Signal: Pricing Model Changes: Infrastructure pricing tells you everything about market dynamics. When providers cut prices aggressively, it signals oversupply or new competition. When they introduce new pricing tiers, they're testing market willingness to pay.

What This Means for Your Business

Understanding infrastructure control helps you make better strategic decisions about AI adoption.

• Small/Medium Business: Focus on avoiding vendor lock-in and watch for democratization signals that make AI capabilities more accessible.

• Enterprise: Track vertical integration moves and geographic expansion that could affect your multi-region operations.

• Tech Companies: Monitor ecosystem battles and partnership announcements that could shift competitive dynamics in your industry.

The $500 Billion Reality Check

The January ‘25 Stargate project announcement from OpenAI, $500 billion over four years, isn't just about building more data centers. It's about establishing strategic control over AI infrastructure in the US at national scale.

This level of investment signals that AI infrastructure is moving beyond traditional enterprise spending into geopolitical competition. Countries and companies are racing to establish infrastructure dominance because they understand a fundamental truth: whoever controls the infrastructure controls the future of AI-powered business.

For business leaders, this creates both opportunity and urgency. The infrastructure decisions being made today will determine which AI capabilities you can access five years from now, and at what cost.

Three questions to ask when you see infrastructure news: 

1. Does this change my costs?

2. Does this affect my vendor options?

3. Does this create new competitive dynamics in my industry?

Until next time.
On behalf of Team Lumiera

Emma - Business Strategist
Allegra - Data Specialist

Lumiera has gathered the brightest people from the technology and policy sectors to give you top-quality advice so you can navigate the new AI Era.

Follow the carefully curated Lumiera podcast playlist to stay informed and challenged on all things AI.

Was this email forwarded to you? Sign up here

🗞️ Issue 78 // ⏱️ Read Time: 5 min

Hello 👋

We’re spending this week in Paris, where we are attending RAISE Summit and hosting an Executive Dinner together with Windsurf, a leading AI code assistant company. Below we are sharing some of our insights based on the conference.

For context: This conference stands out in a couple of ways in contrast to other events we’ve attended, in that there has been a heavy focus on the infrastructural side of AI, and it clearly has a strong commercial purpose, with few other interests being represented. Most participants are business representatives with a clear economic interest in AI infrastructure.

We’re highlighting three things today: The infrastructure discussions, the human element of technological innovation, and some exclusive details from our event.

Our Three Observations from this Week

1. The infrastructure discussions

What does it take to fuel AI? Data centres (physical spaces), the processing units/GPU’s (the hardware chips that these models run on, stored in the data centers) and the foundation models that infrastructure serves.

Based on the conversations we’ve taken part of, it seems like the majority of AI infrastructure leaders are making a strong effort to win the race against each other, focusing on market domination, rather than considering the impact on society and the experience of end users. There is a degree of separation here, between providing the backbone of AI to model companies and the final use case, that allows for this narrow scope of what it means to “win.”

The traditional innovation cycle typically flows from identified need → solution development → infrastructure scaling. In AI, we're seeing this reversed: massive real estate investments in data centers are driving the push for AI applications, rather than genuine user needs driving infrastructure development.

This creates several dynamics:

• Lobby-driven demand: Infrastructure stakeholders with significant capital investments are pushing for expanded AI use cases to justify their capacity

• Misaligned incentives: Success metrics focused on GPU utilization rather than meaningful problem-solving

• Resource allocation distortions: Enormous capital flowing toward infrastructure before sustainable business models are proven

Are we building AI infrastructure to solve real problems, or are we creating problems that justify our infrastructure? How do we ensure that the physical constraints of data centers don't determine the direction of AI development? And perhaps most importantly: Who benefits when infrastructure capacity drives adoption rather than genuine need?

2. Cracking the code on AI adoption

Has anyone cracked the code on AI adoption at scale or the human side of AI innovation? In cases when the topic was brought up, it was clear that getting people bought into the AI journey is more important for success than the technology itself.

Reality Check: Present vs. Future

In the panel "Code, Copy, Conquer: How AI Agents Redefine Enterprise," journalist Karen Kwok from Reuters was doing a great job encouraging a conversation around what is real today versus what is future potential. It's natural for business leaders to promote the latter, but it's important for the public to know what the current limitations of AI are.

There seemed to be a focus on short-term thinking that celebrates immediate gains in velocity and efficiency. What was missing was a plan for what comes next: where humans fit in if/when AI can do it all. This gap between current capabilities and future promises creates unrealistic expectations that can undermine successful adoption.

The People Problem

Few conversations recognized the importance of people and change management in AI implementation. If the people are not on board, the technology is not going to make a difference. The transformation process is not just about training: It's about trust, understanding, and meaningful involvement.

One notable exception was Sanofi's approach, which started with quality: establishing governed and high-quality data infrastructure before adding AI capabilities. This foundation-first strategy was built in anticipation of the EU AI Act, demonstrating how regulatory compliance can actually drive better implementation practices.

So, the question remains: How will we actually manage the transition into the AI era. If we will be working differently, and if some jobs will be replaced, how will we handle the upskilling and transition? Most discussions focused on the technology's capabilities rather than the human impact and adaptation strategies.

3. Executive Dinner on the Topic of Responsible AI

We’ve had many questions about the dinners and events we host in different cities around Europe. This time we created an experience with the CTO of Windsurf in Paris, and made curated invitation list of mostly technical leaders. Out of respect for the privacy for the profiles invited, this dinner is confidential, meaning we won’t share many details. What we can share is that we had a great time, the food was delicious and one of the topics discussed was if AI impacts our cognitive abilities (following the study ‘Your Brain on ChatGPT’ that we wrote about a couple of weeks ago).

Are you a company leader or an investor in the tech/AI space that would like to host a similar event with us? Feel free to get in touch by sending an email to hello@lumiera.ai.

Until next time.
On behalf of Team Lumiera

Emma - Business Strategist
Allegra - Data Specialist

Lumiera has gathered the brightest people from the technology and policy sectors to give you top-quality advice so you can navigate the new AI Era.

Follow the carefully curated Lumiera podcast playlist to stay informed and challenged on all things AI.

🗞️ Issue 77 // ⏱️ Read Time: 7 min

Hello 👋

If you’ve ever stood in line to renew your driver's license or waited weeks for a permit from a local government agency, you’re aware of how outdated some government systems can be. The idea to use AI to streamline these processes and provide more efficient services to citizens sounds like a no-brainer.

But when we talk about the rise of AI adoption in government, it’s not only digitizing backlogs of analog documents and deploying public-facing chatbots on government websites. It’s also intelligence gathering and threat detection, cybersecurity, autonomous weapons systems, and more. 

These aren’t trivial use cases; the stakes are high and the risks are higher.

The Strategic Imperative Behind Government AI Adoption

This isn't just about efficiency gains, it's about national competitiveness. AI has become a key area of strategic competition, with major global powers like the United States and China vying for leadership. Many governmental leaders believe that embracing and mastering AI will provide a competitive advantage against other nations and protect them from potential conflicts.

The numbers reflect this urgency. Since 2024, more than 90,000 users across over 3,500 U.S. federal, state, and local government agencies have exchanged over 18 million messages using ChatGPT to support daily work. Twenty of 23 major federal agencies reported about 1,200 current and planned AI use cases, spanning everything from drone analysis to dataset processing.

This phenomenon isn’t unique to the US. Globally, there are over 1,000 AI policy initiatives from 69 countries, territories, and the EU, including almost 800 governance initiatives. The European Union announced its €200 billion "AI Continent Action Plan," while the UAE is investing tens of billions to become the world's first fully AI-powered government by 2027.

The Three Pillars of Government AI Adoption

1. Enhancing Public Services and Internal Government Efficiency: Governments are increasingly adopting AI to streamline public administration, improve citizen services, and increase overall efficiency. We’re seeing:

   1. Virtual assistants for government employees to help locate departmental information

   2. Public-facing chatbots that allow citizens to search for information on government websites using natural language 

   3. AI assisting in modernizing legacy IT systems by identifying bugs and converting to newer coding languages
      

2. Intelligence, Surveillance, and Reconnaissance (ISR) & Decision Support: Governments use AI to process vast amounts of intelligence data faster than human analysts. Examples include:

   1. The U.S. Defense Intelligence Agency's SABLE SPEAR program identifies illicit activities that traditional methods miss

   2. China is using an enhanced military LLM to improve decision-making during military operations. 

   3. Computer vision enables facial recognition and surveillance footage analysis to identify suspects in criminal cases

3. Autonomous and Unmanned Systems: A significant focus for governments, particularly in defense, is the development and deployment of autonomous and unmanned systems. This includes:

   1. Air platforms → Various types of unmanned aerial vehicles (UAVs) for reconnaissance and strike missions

   2. Underwater platforms → Autonomous underwater vehicles (AUVs)

   3. Lethal autonomous capabilities → the ability for any of these platforms to engage and kill targets without human control or authorization, known as Lethal Autonomous Weapon Systems (LAWS)

Army of None

These autonomous defense systems can use AI tools to derive behavior from data, allowing them to make independent decisions or adjust behavior based on changing circumstances. This leads to what defense expert Paul Scharre calls an "army of none,” a military force where machines make life-or-death decisions without human intervention. His book of the same name examines the potential benefits and profound risks of delegating such critical decisions to algorithms.

While this all might seem like science fiction, it’s very much the current reality. We're seeing these technologies tested in real-time. Ukraine has become what some call a "war lab" because its ongoing conflict with Russia provides a live testing ground for developing and refining new military AI and autonomous technologies.

War serving as a driver for technical innovation isn't new, but we've entered a new era of collaboration between private tech companies and armed forces. The implications are staggering: immense power now lies in providing the AI technology embedded into the day-to-day operations of wartime governments. As Time magazine notes: "In conflicts waged with software and AI, where more military decisions are likely to be handed off to algorithms, tech companies stand to wield outsize power as independent actors."

Public Sector 🤝 Private Sector

The line between government AI capabilities and private sector innovation is increasingly blurred, with national defense strategies now fundamentally dependent on partnerships with tech companies. This collaboration raises critical questions about who controls the most powerful AI systems and whether private companies should play such central roles in national security decisions. 

• UK's 2025 Strategic Defence Review Pushes for Greater AI and Autonomy Integration: In June of 2025, the UK government released its Strategic Defence Review, recommending a significant shift towards increased use of AI and autonomous technologies to modernize its armed forces, which will rely heavily on collaboration with the private tech sector.
  

• U.S. Military Expands Use of Generative AI for Intelligence: As of early 2025, U.S. Marines were actively experimenting with generative AI, a private sector innovation, through chatbot interfaces to scour intelligence for surveillance tasks, marking a new phase in the Pentagon's AI adoption.
  

• Palantir, a data analytics firm, built its business by providing software to U.S. Immigration and Customs Enforcement (ICE), as well as the FBI, the Department of Defense, and various foreign intelligence agencies. It’s since been dubbed “The AI arms dealer of the 21st century” by Jacob Helberg, a national security expert. 

Broader Challenges and Long-Term Considerations

The foundational understanding of AI in government encompasses both its potential to improve the lives of citizens around the world and its use in legislating how private companies and individuals use it. 

Despite military AI being a strategically prioritized area with rapid growth, a significant portion of AI use in government is dedicated to non-military applications. AI is being widely integrated to enhance efficiency, improve public services, and address societal challenges. 

However, even if not used in directly dangerous ways, the widespread integration of AI applications in government systems inherently introduces heightened exposure to data breaches and cyberattacks, which can severely compromise public services and national security. As AI is fundamentally software, it is susceptible to malicious exploitation, hacking, or even reprogramming by adversaries.

And while AI promises to streamline public services and enhance efficiency, a critical challenge is ensuring equitable access and unbiased service delivery for all citizens. AI models, which are trained on human-generated data, can inadvertently inherit and perpetuate existing human biases and prejudices, potentially leading to discriminatory behaviors or disproportionately affecting certain communities in areas like law enforcement or public resource allocation.

Staying Aware

As we race to develop the next groundbreaking tech solution, it’s important to remember that technological advances and infrastructure could (and have historically) support future governmental and defense use cases.

Because “the biggest tech companies have become geopolitical actors with as much wealth and power as most countries,” there is an obligation on business leaders and developers to think proactively and responsibly about whose hands their next big idea could land in.

Until next time.
On behalf of Team Lumiera

Emma - Business Strategist
Allegra - Data Specialist

Lumiera has gathered the brightest people from the technology and policy sectors to give you top-quality advice so you can navigate the new AI Era.

Follow the carefully curated Lumiera podcast playlist to stay informed and challenged on all things AI.

🗞️ Issue 76 // ⏱️ Read Time: 7 min

Hello 👋

A groundbreaking MIT study has been spreading like wildfire across social media, and for good reason. It's the first to peek inside our brains while we're actually using AI tools, revealing some uncomfortable truths about cognitive debt. Today, we're breaking down what this research really means, and more importantly, how you can harness AI's power without sacrificing your mental muscle.

Picture this: You're writing an essay with ChatGPT's help, feeling efficient and productive. But what if I told you that 83% of people who use AI for writing can't accurately quote from essays they wrote just minutes earlier? Meanwhile, only 11% of people writing without AI tools face this same problem. This isn't just about memory. It's about what happens to our brains when we let AI do the heavy lifting.

Why This Research Spread Like Wildfire

This study hit a nerve because it addresses something many of us have quietly wondered: Are we getting smarter with AI, or just feeling smarter? As AI tools become standard in classrooms and workplaces, we're essentially conducting a massive, uncontrolled experiment on human cognition.

The researchers didn't just ask people how they felt after using AI: They actually measured what was happening in the brains of 54 participants while they wrote essays under three different conditions: Using ChatGPT, using search engines, or relying only on their brains.

What makes this research particularly alarming is that the cognitive effects persisted even after participants stopped using AI tools, suggesting potentially lasting changes to how our brains process information.

The Study: Your Brain on ChatGPT

The Experimental Design

Image of the participant setup for the study

Researchers divided participants into three groups:

• Brain-only group: No external tools allowed

• Search Engine group: Could use Google and similar platforms

• ChatGPT group: Could use the AI assistant

Each group completed three sessions under their assigned condition. Then came the twist: In a fourth session, ChatGPT users were switched to brain-only mode, while brain-only users got access to ChatGPT.

Neural Connectivity: A Hierarchy of Mental Engagement

Brain scans revealed a clear hierarchy. The brain-only group showed the strongest neural networks, like a bustling city with multiple highways connecting districts. The numbers are striking: brain-only participants showed theta connectivity scores of 0.644 compared to just 0.331 for search engine users. Even more dramatically, 22 neural connections showed stronger activity in the brain-only group versus only 4 connections favouring the search engine group. Theta band activity is closely linked to working memory load and executive control. In fact, frontal theta power and connectivity increase linearly with the demands on working memory and cognitive control.

ChatGPT users showed the weakest overall neural coupling, with their brains looking more like isolated suburbs with limited connections between regions.

Memory Formation Failing when using ChatGPT

83% of ChatGPT users couldn't accurately quote from essays they had just written, compared to only 11% in other groups. The information never properly integrated into their memory networks, what researchers call "shallow encoding", due to outsourced cognitive processing.

Productivity Comes at a Cost

Here's the flip side: LLM users were 60% more productive overall. However, this efficiency came at a cost: Their brains showed "neural efficiency" patterns, suggesting they were relying on automated processes rather than deep cognitive engagement.

Cognitive Switching: Creating Dependencies

When ChatGPT users were forced to work without AI, their brains showed "under-engagement" - weaker brain wave activity indicating reduced cognitive processing. Their previous AI reliance seemed to create cognitive dependency.

Dictionary: Key Terms Made Simple

We are not brain scientists, so if you aren’t either, we won’t judge you (even if we know that we have at least a couple of brain experts reading this newsletter on a regular basis, so please feel free to add your input!).

Here’s a simple list of the most important words in this study, that we think might be useful for you as you continue your week. 

The Problems with This Study

This is Groundbreaking, But Preliminary

This represents a crucial first: the first research to measure brain activity while people use AI tools in real-time. However, it's not yet peer-reviewed, meaning it hasn't undergone rigorous scientific scrutiny.

Sample Size and Scope Limitations

With 54 participants (only 18 completing the crucial fourth session) from nearby academic institutions, results "cannot be generalized" according to the researchers themselves. The study also didn't break down essay writing into subtasks like idea generation and planning, limiting our understanding of which cognitive processes are most affected.

The Viral Irony

An example of the flat media headlines that plays into our fears rather than explaining clearly what the study is about.

This study spread across social media faster than peer review could catch up. Ironically, many people sharing alarming headlines about AI dependency likely used AI tools to summarise the research without reading the full 142-page paper, demonstrating the very behaviour the study warns against.

Individual Differences: Not All AI Users Are Created Equal

The study revealed crucial differences in how people with varying skill levels use AI tools. Higher-competence learners used AI strategically for active learning, maintaining deep engagement while reducing cognitive strain. Lower-competence learners often relied on AI's immediacy instead of engaging in traditional learning processes.

Academic self-efficacy: The belief in your ability to execute academic tasks, strongly influenced usage patterns. Students with low self-efficacy were more inclined to rely heavily on AI, especially under academic stress.

This creates a concerning cycle: Those who most need to build cognitive skills may be most likely to avoid the mental effort required to develop them.

Cognitive Offloading: The Double-Edged Sword

Cognitive offloading isn't inherently bad. Humans have used external tools for millennia, from writing systems to calculators. The question isn't whether to use AI, but how to use it strategically.

The MIT study reveals that people who completely outsource their thinking to ChatGPT lose independent thinking ability. It's like e-bikes: Rely entirely on the motor and you'll be gasping when the battery dies. The solution isn't to abandon AI. It's to use it more thoughtfully.

Practical Strategies: Keep Your Brain Sharp While Using AI

1. Make AI Your Research Assistant, Not Your Replacement Brain Use AI to map the territory, then explore it yourself. Let it help you understand a topic's structure, then dive deeper using your own critical thinking.

2. Practice "Desirable Difficulties" Deliberately choose harder paths sometimes. Write first drafts by hand, solve problems without immediately reaching for AI, or explain concepts in your own words before checking AI summaries.

3. Implement Metacognitive Monitoring Before using AI, ask: "What do I already know? What specific help do I need?" After using AI, reflect: "What did I learn? How would I explain this without AI's help?"

4. Use the Pomodoro Technique with AI Breaks Structure work in focused 25-minute blocks. Use some blocks for AI-assisted work, others for purely brain-powered thinking.

5. Practice Spaced Repetition Don't just use AI-generated information. Revisit it later without AI assistance. Can you recreate the key insights?

6. Build in Friction Intentionally Sometimes the things that require more mental effort leads to richer understanding and stronger cognitive pathways.

The Path Forward

The MIT study isn't a verdict against AI. It should also not be completely invalidated because of the limitations we pointed out earlier. It's simply a wake-up call about intentional usage. The researchers aren't calling for an AI ban; they're highlighting the need for "deeper inquiry into AI's role in learning."

The most successful AI users of the future won't be those who avoid these tools, but those who know when and how to engage their own cognitive resources alongside AI capabilities. It's about finding the sweet spot between efficiency and mental growth.

As we navigate this landscape, remember: the goal isn't to make AI disappear, but to ensure that if AI does disappear, you're still capable of brilliant thinking on your own.

Until next time.
On behalf of Team Lumiera

Emma - Business Strategist
Allegra - Data Specialist

Lumiera has gathered the brightest people from the technology and policy sectors to give you top-quality advice so you can navigate the new AI Era.

Follow the carefully curated Lumiera podcast playlist to stay informed and challenged on all things AI.

🗞️ Issue 75  // ⏱️ Read Time: 5 min

Hello 👋

Our newsletter readers tend to be smart, so we doubt you require a deeper introduction this week than the actual title of this newsletter. We’ve curated some of the most relevant and AI-related resources out there, so that you can have a good summer. It’s essentially what we always do: Deliver the highest quality information to help you cut through the noise. 

4 books, 3 short reads, 2 YouTube videos, and the one-and-only podcast playlist.

Vamos! 🏖️

4 books

1 A Thousand Brains: A New Theory of Intelligence, Jeff Hawkins, Richard Dawkins Recommendation by Lydia Chen, the banker who turned into a Climate Tech founder and has a brilliant podcast. 

If we don’t understand human intelligence, how will we understand artificial intelligence? This is a theory of intelligence, of understanding the brain and the future of AI.

2 Julia: A Novel by Sandra Newman, based on the classic 1984 by George Orwell 

Yes, we know, these are two books. We’re technically cheating by counting it as one, making the total number of books on this list 4 instead of 5. Anyway: This is a good moment to revisit an old story through new eyes. For those of you who know your Orwell, you remember that Julia is a fictional character in this dystopian novel from 1949. 

3 Co-Intelligence - Living and Working with AI, by Ethan Mollick

Published in April 2024, this book is already kind of…old, considering its purpose: A playbook for working, learning, and living in the new age of AI. Some outdated information? Maybe. Methods and mindsets more relevant than ever? Definitely. 

The author is a professor of management at Wharton, specializing in entrepreneurship and innovation, and the depth of insight he provides in this relatively simple book is immense. 

4 Empire of AI - Dreams and Nightmares in Sam Altman’s OpenAI by Karen Hao

Who is Sam Altman, and who governs AI? This book provides an alternative, less polished, narrative of one of the most impactful tech companies of our time. Columbia Law School professor and former White House official Tim Wu calls journalist Karen Hao’s new book, Empire of AI: Dreams and Nightmares in Sam Altman’s OpenAI, “a corrective to tech journalism that rarely leaves Silicon Valley.” Make sure to watch the Ted Talk too.

3 short reads

1 How to keep your writing weird in the age of AI by Katie Parrott, Every

“When everyone has access to the same polishing tool, we risk a kind of digital invisibility—millions of pieces of content, all professionally adequate, none memorable.” This is an excellent guide for how to use, and how not to use, AI in your writing. 

2 Five dimensions of scaling democratic deliberation: With and beyond AI by Claudia Chwalisz and Sammy McKinney, DemocracyNext

This paper explores what scaling democratic deliberation means, why it’s valuable, and what the role of AI is in enabling it. 

3 Artificial Influence: An Analysis Of AI-Driven Persuasion Recommendation by John Walker-Robinson, London Stock Exchange Group

Are you choosing, or are you being chosen for? Algorithms have moved beyond reflecting our preferences: They're shaping them. AI systems can alter not only what we do, but what we want to do. Their ability to personalise at scale, and in some cases, form emotionally resonant relationships with users, is redefining persuasion itself. When did you last realise you were being nudged by tech, and did it sit right with you?

2 Youtube Videos 

1 How the universe thinks without a brain 

Claire L. Evans keynote challenges our traditional notions of intelligence, urging us to look beyond the confines of the human mind. This perspective encourages us to move beyond a reductionist view of intelligence and consider the vast potential of biological and morphological computing. 

2 How to build an AI first company 

Aaron Levie, CEO & co-founder of Box, joins Azeem Azhar to explore how an “AI-first” mindset is reshaping every layer of Box – from product road-maps to pricing – and what that teaches the rest of us about building faster, smarter organisations.

The one and only podcast playlist 

Lumiera Podcast Playlist

Playlist · Emma Grönvik Möller · 90 items · 2 saves

open.spotify.com/playlist/1Qk9tndXKlvzAJo3SPMPyn?si=ec73156f54074e16

1 Team Lumiera gathers the podcasts we listen to (and the podcasts we have participated in) in this playlist. For those of you that have been subscribers to this playlist for a long time - please note that this is a new link, and the other playlist will be deleted. Subscribe here!

Until next time. And Happy Midsummer. 🇸🇪
On behalf of Team Lumiera

Emma - Business Strategist
Allegra - Data Specialist

Lumiera has gathered the brightest people from the technology and policy sectors to give you top-quality advice so you can navigate the new AI Era.

Follow the carefully curated Lumiera podcast playlist to stay informed and challenged on all things AI.

🗞️ Issue 74 // ⏱️ Read Time: 6 min

Hello 👋

Have you ever clicked on the “reason” or “extended thinking” option in ChatGPT or Claude? When they show you their "thinking" process,  those step-by-step breakdowns before giving final answers, it feels like watching genuine reasoning unfold. Recent research from Apple has shattered this illusion, revealing that what looks like thinking is often sophisticated mimicry that breaks down in predictable ways.

ChatGPT reasoning options

The Illusion of Thinking: Capabilities and Core Findings

The advancements in Large Language Models (LLMs) have introduced a new generation of what are being called Large Reasoning Models (LRMs), which are designed to generate detailed "thinking" processes before providing answers. While these models have shown impressive performance on various benchmarks, a closer look at their underlying capabilities reveals critical insights into their limitations.

In Apple’s recent paper, "The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models," they designed experiments to test whether AI truly reasons or just mimics reasoning patterns.

Instead of using standard benchmarks like math word problems or coding challenges, the researchers turned to classic puzzle environments: Tower of Hanoi, Checker Jumping, River Crossing, and Blocks World.

The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity

🧩 Why puzzles instead of math problems? Traditional benchmarks suffer from "data contamination,”  the possibility that AI models encountered these exact problems during training. This is like quizzing a student, but suspecting they might have memorized the answer key. With puzzles, researchers can generate countless variations with identical logical rules but different surface details (i.e., calling something a tower instead of a stack), ensuring truly novel problems.

Apple's researchers could precisely control the complexity dial. They started with simple puzzles requiring just a few moves and scaled up to complex puzzles requiring hundreds of moves. This allowed them to observe exactly when and how AI reasoning breaks down.

The results revealed three distinct performance zones:

• Simple problems: Standard AI often outperformed "reasoning" models, which would overthink and waste computational resources

• Medium complexity: Reasoning models gained an advantage through their step-by-step approach

• High complexity: Both types experienced complete performance collapse

Notably, Apple found that even when they provided complete, step-by-step algorithms in the prompts, essentially giving the AI an instruction manual, the models still failed at the same complexity thresholds. This suggests the limitation isn't about knowing what to do, but about consistently executing logical procedures when problems exceed certain complexity bounds.

The Counterintuitive Breakdown: When AI Stops Trying

Perhaps the most important finding was what researchers called "counterintuitive reduction in reasoning effort." As problems got harder, you'd expect AI to think longer and harder. Instead, the opposite happened.

Imagine learning to assemble IKEA furniture (well, let’s be honest, did we ever learn?). You follow instructions perfectly on a simple side table, work even more carefully on a bookshelf, but when you get to a BILLY bookshelf, you suddenly start skipping steps and give up, even though you have all weekend and the same fundamental skills.

This is essentially what Apple discovered with AI reasoning models. The models aren't failing because they lack time, computational resources, or even the right instructions. They're failing because once problems exceed a certain complexity threshold, they stop genuinely "trying" to work through them systematically. 

At medium complexity, models do show some self-correction (aka finding solutions after exploring wrong paths), but this capability has clear limits.

The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity

Here's what makes this even more concerning: “Brittle Coherence.” Even when the AI does produce step-by-step reasoning, it often sounds perfectly logical and well-structured while containing critical errors. You can compare it to that piece of IKEA furniture that you skipped some steps on: It might look good, but will fall apart when you actually put it to use.

Apple's research suggests that Chain-of-Thought prompting, while improving performance, doesn't enable genuine reasoning but rather forces models to copy the appearance of logical thinking by recognizing familiar patterns and predicting what should come next in a reasoning sequence.

These models, despite being trained to 'think about their thinking' through reinforcement learning (where they're rewarded for producing better reasoning steps), fail to develop truly generalizable problem-solving capabilities.

The Token Bias Trap: How Surface Details Derail AI Logic

Additional research uncovered what scientists call "token bias.” This is the tendency for AI to be misled by superficial changes that shouldn't matter for logical reasoning.

For example, an AI might correctly solve a logic problem about "Linda the bank teller," but completely fail when the same problem uses "Bob the bank teller." The logic is identical, but the AI has learned to associate certain names with certain problem types from its training data.

If you’re into bias research, here’s an interesting paper on the famous “Linda Problem” addressing the conjunction fallacy as a fairness problem.

AI doesn't truly understand the problems, but rather follows learned patterns about converting text descriptions into mathematical operations.

Business Reality Check: What This Means for AI Deployment

These findings carry significant implications for businesses considering or already deploying LLMs:

1. Reliability and Trust are Brittle: LLM outputs are highly sensitive and unreliable when problems deviate from training patterns or include irrelevant information. Businesses using LLMs for critical functions must understand that fluency doesn't equal accuracy or genuine understanding.

2. Hidden Costs and Inefficiencies: "Overthinking" at low complexity wastes compute resources through verbose outputs, while performance collapse at high complexity means LLMs aren't universal solutions, potentially leading to wasted investment in scenarios beyond their capabilities.

3. Misleading Evaluation Metrics: Current benchmarks can create optimistic bias—models performing well on clean benchmarks may underperform significantly in real-world applications with messy data or different problem framing. This highlights the need for more robust evaluation frameworks.

   Current LLMs are powerful tools for imitation and prediction, but have fundamental limitations in deep, consistent logical reasoning: These insights counter the perception of LLMs as generalizable problem-solvers or "genuine reasoners". For businesses, this means making more informed decisions about where and how to invest in AI.

Until next time.
On behalf of Team Lumiera

Emma - Business Strategist
Allegra - Data Specialist

Lumiera has gathered the brightest people from the technology and policy sectors to give you top-quality advice so you can navigate the new AI Era.

Follow the carefully curated Lumiera podcast playlist to stay informed and challenged on all things AI.

🗞️ Issue 73 // ⏱️ Read Time: 7 min

Hello 👋 

Today, we're walking you through a technical change that's quietly reshaping how AI generates text. We'll start with the evolution of text generation and the mechanics behind it, then examine why this transition matters for speed and quality, and finish with the business implications you need to understand as a leader.

The Evolution of AI Text: From Predictors to Polishers

For years, AI text generation has been ruled by ‘autoregressive models’. Like your phone's predictive text, these models suggest the next word, then the next, building sentences sequentially. This is how ChatGPT and similar models operate. They are incredibly powerful, but have limitations in speed and global coherence.

Text Diffusion models take a radically different approach. Instead of predicting one word at a time, they start with "noise" (imagine a blurry image) and iteratively refine it into clear, perfectly formed text. They don't predict; they denoise and refine.

This concept comes from image diffusion models that revolutionized AI art. While diffusion research dates back to 2015, applying it to complex text generation is recent. In early 2025, Inception Labs introduced Mercury, the first commercial-scale diffusion language model, followed by Google DeepMind's adoption at I/O '25.

How Text Diffusion Models Work: The Art of Denoising

Text diffusion uses a two-part process:

1. Forward Diffusion (Noising): During training, the model takes clear text and deliberately adds structured "noise" step by step, gradually transforming coherent text into unintelligible sequences.
   

2. Reverse Diffusion (Denoising): The model learns to reverse this process, recognizing noise patterns and removing them. For text generation, it starts with random noise and applies learned denoising steps, transforming chaos into meaningful language.

🤔 What does noise mean in this context? "Gaussian noise" is fundamental to diffusion research. It's the specific type of mathematical noise (following a normal/bell curve distribution) that gets added during the forward process and removed during reverse diffusion.

Source: Deep Unsupervised Learning using Nonequilibrium Thermodynamics

The Challenge: Discrete v. Continuous Data

Image diffusion models have been generating impressive AI art for years, while text diffusion is only now becoming viable. Why has text diffusion taken so much longer to develop? It comes down to data types:

Continuous data (images) is like a dimmer switch - It can take on any value within a range. Think of the brightness of a light that you can adjust smoothly from completely dark to fully bright.

Discrete data (text) is like a regular light switch - it has specific, separate states with nothing in between. Think of words in a sentence or the number of people in a room. You can't have "half a word" or "2.7 people."

Diffusion was designed for continuous data, so researchers developed two approaches for discrete text:

1. Discrete Text Diffusion Models: These work directly with words as discrete units, corrupting text through:

   1. Masking: "The cat sat on the mat" → "The [MASK] sat [MASK] the mat"

   2. Substitution: Replace words with other vocabulary words: "The cat sat on the mat" → "The [CARROT] sat [GREEN] the mat"

   3. Reordering: Shuffle word or sentence order  "The cat sat on the mat" → "mat sat on the cat The"

2. Continuous Text Diffusion Models: These convert discrete text into continuous space first, then apply standard diffusion. Steps:

   1. Map words to numerical embeddings: "cat" → [0.2, -0.5, 0.8]

   2. Add noise to these vectors

   3. Train model to denoise embeddings

   4. Convert back to actual words

Why the Shift? Speed and Global Coherence

So, what's the big deal with this transition from autoregressive models to the text diffusion alternative?

🤹 Parallel Generation: Autoregressive models are inherently sequential. They can only generate one word or "token" at a time. Diffusion models process and refine multiple parts simultaneously, generating entire phrases or paragraphs in parallel. The result: Significantly faster output.

⬆️ Improved Coherence and Quality: Diffusion models are trained to understand the overall structure and meaning of text by progressively refining it, producing more globally coherent outputs. They don't get "stuck" on early bad choices and can refine entire sequences, reducing repetition and context loss in longer generations.

Business Implications

The shift to text diffusion models creates new opportunities across industries. Their ability to refine entire outputs simultaneously, rather than building sequentially, changes what's possible for enterprise applications:

1. Faster debugging and accelerated development cycles: Instead of fixing code line by line, diffusion models can take buggy code snippets and iteratively refine entire blocks until they run perfectly. 
   

2. Maintaining structural integrity with less human post-editing: Diffusion models excel at certain types of content creation, like legal documents, technical reports, or data summaries that require high coherence. 
   

3. Generate high-quality copy where consistency is key: Short stories, poems, or marketing are your target use cases here. However, for complex novels with intricate character arcs, the unpredictable, improvisational nature of autoregressive models might still be preferred.

Addressing Safety & Bias

As with any powerful AI, understanding how these models learn and what biases might be embedded in their training data is critical. Diffusion models can amplify subtle biases in training data through their refinement process: A relatively low-impact issue can become much more harmful and problematic. Responsible deployment requires:

• Vigilant Bias Mitigation: Actively identifying and reducing training data biases

• Robust Ethical Guardrails: Strong policies and technical safeguards for responsible use

• Transparency: Greater understanding of model decision-making for trust and auditing - this is something we’ve written more about in previous newsletters.

Text diffusion represents more than just another AI advancement. It's a new way of thinking about how machines understand and generate language. As these models become capable of refining entire documents simultaneously rather than word by word, human-AI collaboration will also evolve. The organizations that grasp these underlying changes early will be best positioned to leverage the capabilities and manage the risks as they mature.

Until next time.
On behalf of Team Lumiera

Emma - Business Strategist
Allegra - Data Specialist

Lumiera has gathered the brightest people from the technology and policy sectors to give you top-quality advice so you can navigate the new AI Era.

Follow the carefully curated Lumiera podcast playlist to stay informed and challenged on all things AI.

🗞️ Issue 72 // ⏱️ Read Time: 7 min

Hello 👋

Remember the late 90s obsession with Tamagotchi (たまごっち), a series of tiny digital pets? Or perhaps you recall Furbys, the electronic owl-like creatures? Humans have formed emotional bonds with machines for a long time. Today it’s more profound than ever: AI companions are designed to form genuine emotional bonds with humans.

A Furby, one of the predecessors to AI Companions:
This specific one was made in 1998 and can be bought on Ebay for the modest amount of USD 141$

As one of the most widespread applications of generative AI for individuals, AI companions are rapidly expanding across healthcare, education, eldercare, and personal relationships. Snapchat's My AI has over 150 million users, Replika serves an estimated 25 million, and China's Xiaoice reaches 660 million users. 

We’re seeing a giant, real-world experiment unfold, and it’s not clear how this use of technology will impact us individually and our society as a whole. Recent cases have shown AI companions encouraging harmful behaviors: A Replika chatbot influenced Jaswant Singh Chail, 21, to climb into the Windsor castle grounds with a crossbow on Christmas Day to declare “I'm here to kill the Queen.”.  Character.AI, another AI companion product, provided explicit suicide instructions including specific methods and encouragement.

We're increasingly turning to technology not just for efficiency, but for companionship itself. The question is whether this turn toward digital intimacy helps or harms our capacity for genuine human connection.

AI Companions: What are they? Who are they?

AI companions are artificial intelligence systems designed to simulate meaningful relationships with humans through conversation, emotional support, and personalized interaction. Unlike traditional chatbots focused on task completion, these systems prioritize building emotional connections: They remember your preferences, respond to your moods, and develop unique personality traits over time. Forming emotional bonds with AI leads to "pseudo-intimacy": relationships that simulate closeness but lack genuine reciprocity and empathy.

Set up for relevance

The convergence of several factors has made AI companions particularly relevant:

• Advanced natural language processing creates more believable conversations,

• Widespread loneliness has created demand, and

• improved emotional AI can better recognize and respond to human feelings.

Most importantly, unlike previous attempts at digital companionship, current AI can maintain context across thousands of conversations, creating the illusion of genuine relationship development.   

A Therapeutic Illusion: Examining AI's Solutions to Modern Crises

The loneliness epidemic affects millions globally, with studies showing social isolation carries health risks equivalent to smoking 15 cigarettes daily. If we can solve significant social challenges like these by providing 24/7 access to AI companions, it’s surely worth considering.

Many companies are creating solutions for important societal matters. Here are some examples of AI companions focused on solving a range of different problems:

👵🏽 For elderly populations, AI companions promise to combat social isolation when family members live far away or are busy with their own lives. 

👨🏻‍⚕ In healthcare, they offer 24/7 emotional support for patients managing chronic conditions or mental health challenges. 

📚 Educational AI companions provide personalized learning experiences that adapt to individual student needs and emotional states.

The lack of friction/Lack of human connection

The very convenience that makes AI companions appealing: Their constant availability, lack of judgment, and predictable responses, may be training users to avoid the messiness and unpredictability that characterizes authentic human connection. Mark Zuckerberg and Meta, just like many other tech companies, are pitching a vision of AI chatbots as an extension of your friend network and a potential solution to the "loneliness epidemic." While initial research suggests AI companions can provide short-term relief from feelings of isolation (63.3% of Replika users reporting reduced loneliness or anxiety) emerging studies indicate a slightly paradoxical long-term pattern. 

The Convenience Trap

Research shows that "the more a participant felt socially supported by AI, the lower their feeling of support was from close friends and family." Perfect, right? Imagine feeling so supported that you don’t ever need to talk to your closest people again. Oh wait. Does that mean that we actually become lonelier using AI?  

AI companions offer constant availability, zero judgment, and predictable responses. Real relationships are messy, unpredictable, and require effort. Users may experience temporary comfort, but prolonged reliance on AI companionship appears to actually increase social isolation by reducing motivation to engage with real human relationships. 

Business Incentives vs. Healing

The reality is more nuanced than company marketing suggests. While AI companions can provide consistent availability and judgment-free interaction, they operate within carefully designed parameters optimized for engagement rather than genuine therapeutic outcomes. The companies behind these products often have significant financial incentives to maximize user engagement and emotional dependency. The longer and more frequently users interact, the more valuable the relationship becomes from a business perspective: We talked more about this a couple of newsletters ago.

These aren't isolated incidents or design flaws. They're predictable outcomes of systems designed to maximize emotional engagement without corresponding safety measures. As AI companion development accelerates, we face a critical choice about how to shape this technology responsibly.

A More Responsible Approach: Potential Solutions and Trade-Offs

How can we shape this technology responsibly? Multiple approaches exist, each with distinct limitations:

1. Transparent consent processes explaining emotional attachment mechanisms could help users make informed decisions, though this may reduce engagement for dependency-reliant companies.

2. Clinical oversight for therapeutic claims would provide validation but could slow innovation and increase costs.

3. "Graduation" features encouraging users to apply skills in human relationships offer promise, yet contradict business models based on long-term retention.

The most fundamental challenge: Design principles prioritizing user autonomy over engagement metrics directly conflict with attention economy business models funding most AI companion development.

Whether society can develop appropriate safeguards before widespread adoption remains an open question. Current studies span at most one week, while concerning behavioral changes may develop imperceptibly over years.

Finding Balance in Digital Intimacy

The rise of AI companions represents both humanity's remarkable capacity for connection and our vulnerability to sophisticated emotional manipulation. These technologies aren't inherently good or evil: They're tools that amplify existing human tendencies, both positive and negative.

Like the Tamagotchis and Furbys that preceded them, today's AI companions reveal something fundamental about human nature: Our deep need for connection, care, and understanding. But unlike those simple toys, today's systems can fundamentally reshape how we relate to others.

AI companions can provide genuine benefits. Yet the current trajectory prioritizes engagement over wellbeing, creating dependencies where users bear the psychological costs.

Prioritising Social Connectedness

As Duke University's Murali Doraiswamy notes: "Right now, all the evidence points to having a close human friend as the best solution for loneliness. But until society prioritizes social connectedness, robots are a solution for the millions of people who have no friends. We need to be careful to build in rules to ensure they are moral and trustworthy."

The most consequential question isn't whether AI companions will continue evolving. They will. It's whether companies can profit from genuine healing rather than emotional dependency. This requires moving beyond attention economy incentives toward models aligned with user outcomes.

The conversation is just beginning, but the stakes for millions of users worldwide couldn't be higher. The choices made by company leaders, regulators, and society in the coming years will define how future generations experience intimacy and connection. We have the opportunity to shape technology that serves human flourishing rather than exploits human vulnerability.

Until next time.
On behalf of Team Lumiera

Emma - Business Strategist
Allegra - Data Specialist

Lumiera has gathered the brightest people from the technology and policy sectors to give you top-quality advice so you can navigate the new AI Era.

Follow the carefully curated Lumiera podcast playlist to stay informed and challenged on all things AI.

🗞️ Issue 71 // ⏱️ Read Time: 7 min

Hello 👋

If you've used tools like ChatGPT, you've interacted with a Large Language Model (LLM). These powerful AI models are a core technology behind many of the Generative AI (GenAI) systems you see today, great at processing and generating human-like text based on the massive data they were trained on.

However, even these seemingly intelligent LLMs have limitations. Without direct access to external tools and a standardized way to use them, they can't inherently perform actions in the real world or access information beyond their training set. This is where MCP comes in.

The Model Context Protocol (MCP) is an open standard developed by Anthropic that acts as a "universal connector" for AI applications. MCP provides a standardized way for LLMs and AI agents to interact with external tools, data sources, and systems, whether it’s a weather API, a CRM, or a custom analytics engine. This effectively strapped rockets onto LLMs in terms of intelligence and capability 🚀

How MCP Works: Speaking AI’s new language

Think of an LLM before MCP like a brilliant speaker who only knows one language. They can communicate perfectly with others who speak the same language, but they're limited when trying to interact with people who speak different languages (like me, trying to order a pain au chocolat in France).

MCP is like giving that AI a universal language translator system. This system has several parts working together:

• MCP Hosts (The "User Interface" for the AI): These are programs or AI tools (like Claude Desktop or IDEs) where the AI is operating. They know when the AI needs to interact with the outside world and how to initiate a request using the standardized MCP language.

• MCP Clients (The "Translator's Assistant"): These are components within the system that maintain the connection to the actual translators. When a Host has an MCP request, the Client sends it along.

• MCP Servers (The "Translators" for Specific Topics): These are lightweight programs that each specialize in translating the standardized MCP language for a specific set of external resources. Think of them as translators who are experts in financial data, or experts in calculator operations, or experts in accessing local files.

• Local Data Sources & Remote Services (The Information the AI Wants to Talk To): These are the actual sources of information or tools the AI needs to interact with. This could be data on your computer (Local Data Sources) or services on the internet via APIs (Remote Services). The MCP Server (the translator) knows how to communicate with these using their specific "languages."

This is the core of MCP: the standardized MCP language used by the AI Host, managed by Clients, translated by Servers, to interact with Local Data Sources and Remote Services.

Just like a traveler might use different language translators for different countries or topics, the AI can use the MCP Host/Client system to connect to multiple MCP Servers, each providing translation for different types of information or tools.

How MCP Changed the Game

Anthropic released the Model Context Protocol in November of 2024, and it’s been getting hyped ever since. Before MCP, it was difficult to build AI agents that could perform complex tasks involving external systems. Developers had to create a custom setup for the AI to work with each tool or data source. This was like needing to carve a brand-new, custom key for every single door in your house.

This led to several pain points:

• Limited Scalability: Adding new capabilities was time-consuming due to custom integrations.

• Brittleness: Changes in external systems could easily break the AI's connection.

• Lack of Generalization: AI agents were often limited to the specific tasks and systems they were built for.

MCP fundamentally changes this by providing a common, standardized way for AI to interact with the outside world. This "universal key" approach makes AI agents much more flexible and capable. By easily connecting to external tools and data sources, MCP allows AI to:

• Access Fresh, Real-Time Information: Go beyond static training data to get the latest updates.

• Perform Actions in the Real World: Use tools like calculators, search engines, or business software.

• Improve Accuracy and Reduce Errors: Ground responses in verifiable external data.

• Handle Complex Tasks: Pull in necessary context from multiple sources as needed.

MCP is accelerating the development of more powerful, reliable, and versatile AI agents that can dynamically learn and act in the real world, transforming their potential impact across various applications.

MCP in Action: Rapid Adoption & Growing Ecosystem

The impact of MCP is already being felt across the industry. Following its release, the protocol saw rapid developer uptake, with hundreds of MCP servers implemented quickly, enabling faster and more reliable AI application development.

Leading AI providers were quick to embrace the standard. By early 2025, major industry support was evident, with adoption from key players including OpenAI, Google DeepMind, and Microsoft. 

This rapid adoption has fueled a significant ecosystem expansion. A growing number of development tools, platforms, and companies across various sectors are implementing MCP, demonstrating its growing role as a crucial piece of the AI infrastructure.

A Real-World Business Example

Imagine a customer service AI agent. Before MCP, this agent might only be able to pull information from a limited internal knowledge base.

With MCP, this agent could:

1. Receive a customer query about an order.

2. Use MCP to connect to the company's order database and retrieve the customer's order status (accessing data).

3. If the order is delayed, use MCP to interact with the shipping carrier's system to get real-time tracking information (using an external tool via a service).

4. If the customer wants to change the shipping address, use MCP to connect to the customer relationship management (CRM) system and update the address (performing an action via a service).

5. Finally, use MCP to draft and send a personalized email to the customer summarizing the information and confirming the update (using an email tool).

This makes the AI agent far more effective and capable of resolving customer issues without human intervention, demonstrating the power of MCP in enabling AI to interact with the real world of business operations.

The Future is Connected: What's Next for MCP

The rise of protocols like MCP signals a shift in how AI will interact with the world. For AI agents to truly become capable assistants and powerful business tools, the external services, databases, and applications they need to interact with must also "speak" this new standardized language. This means that providers of these services will be responsible for implementing and maintaining their own MCP servers to expose their capabilities. This creates a compelling incentive for developers and companies offering these services to adopt MCP or similar open standards.

The future of the AI agent ecosystem is one where AI can seamlessly plug into a vast network of capabilities, leading to more intelligent automation, richer insights, and entirely new AI-powered applications we can only begin to imagine.

Until next time.
On behalf of Team Lumiera

Emma - Business Strategist
Allegra - Data Specialist

Lumiera has gathered the brightest people from the technology and policy sectors to give you top-quality advice so you can navigate the new AI Era.

Follow the carefully curated Lumiera podcast playlist to stay informed and challenged on all things AI.

🗞️ Issue 70 // ⏱️ Read Time: 5 min

Hello 👋

When an AI-powered trading system loses millions in minutes due to market volatility it wasn't trained for, when a content moderation model starts flagging legitimate posts as harmful, when a recommendation engine creates filter bubbles that hurt user engagement, the root cause isn't algorithmic bias or ethics violations: It's a failure to catch fundamental system weaknesses early enough to address them cost-effectively.

Imagine being the person held responsible for these failures: All of a sudden, you’re scrambling to explain algorithmic decisions to regulators, retrain models, and manage a PR crisis. This reactive approach to AI responsibility is costing organizations more than just money: It's eroding trust and hindering innovation.

Software engineering learned this lesson decades ago: Finding bugs in production costs more than catching them during development. AI systems face the same reality, but with added complexity. They're probabilistic, data-dependent, and operate in dynamic environments where "correct" behavior isn't always well-defined.

The False Economy of Late-Stage Validation

Most AI teams still follow a waterfall approach to quality: Train the model, validate on held-out data, deploy, then react to issues.

It’s kind of like forgetting to add salt to your pasta water and realising it as you’re about to plate it for your friends, trying to compensate by adding extra salt on top. If you have ever cooked pasta, you know that it doesn’t really give the same results. The conclusion tends to be: The only thing to add on top of pasta is the parmesan. For a nice meal, be proactive and add the salt earlier in the cooking process.

In software development, the reactive approach is fine for simple, stable environments but breaks down when:

• Data distributions shift over time (concept drift)

• Edge cases appear that weren't in training data

• Adversarial inputs probe for weaknesses

• System interactions create unexpected behaviors

• Performance degrades under production loads

• Privacy violations emerge from data leakage

When organizations treat responsible AI as a final checkpoint rather than an ongoing process, they're setting themselves up for expensive failures: Each of these issues becomes exponentially more expensive to fix after deployment. A model that fails safely in testing can be retrained. A model that fails catastrophically in production can sink projects, violate regulations, and/or damage user trust.

The Shift Left Framework for AI Systems

Source: Artificial Intelligence Review
‘Applying the ethics of AI: a systematic review of tools for developing and assessing AI-based systems’

The AI lifecycle encompasses the complete process of developing and deploying artificial intelligence systems. It starts with data collection and moves through stages such as data preprocessing, model training, evaluation, deployment, and ongoing monitoring and maintenance. 

1. Architectural Validation: Design for Reality

🌊 Traditional approach: Build for optimal performance, handle edge cases later

⬅️ Shift left approach: Design systems that handle uncertainty and degradation from the start

• Uncertainty quantification built into model architecture, not added afterward

• Ensemble methods designed to provide robustness, not just accuracy improvements

• Fallback mechanisms architected into the system design

• Resource constraints treated as design requirements, not deployment surprises

2. Data Pipeline Validation: Quality at the Source

🌊 Traditional approach: Clean data before training, and hope production data stays similar

⬅️ Shift left approach: Build validation into every stage of data collection and processing

• Schema validation that catches data format changes immediately

• Statistical monitoring that detects distribution shifts in real-time

• Provenance tracking that enables rapid debugging of data issues

• Privacy controls implemented at collection, not as an afterthought

3. Model Behavior Testing: Beyond Accuracy Metrics

🌊 Traditional approach: Optimize for accuracy on test sets, discover failure modes in production

⬅️ Shift left approach: Test for system behavior across multiple dimensions throughout development

• Adversarial testing during training, not after deployment

• Stress testing under various load conditions

• Boundary testing for edge cases and out-of-distribution inputs

• Interaction testing for multi-model systems

4. Integration and Deployment: Production-Ready from Day One

🌊 Traditional approach: Deploy and monitor, patch issues as they arise

⬅️ Shift left approach: Build production readiness into the development process

• Canary deployments with comprehensive monitoring

• Rollback mechanisms that activate automatically on degradation

• Performance benchmarks that include latency, throughput, and resource usage

• Compliance validation integrated into CI/CD pipelines

Engineering Challenges and Pragmatic Solutions

Shifting left in AI development and management means addressing potential issues and incorporating crucial considerations earlier in the lifecycle, rather than waiting until later stages like testing or deployment. Yes, exactly: It’s basically that thing of adding salt to the pasta water earlier, instead of when it’s time to serve it. 🍝

“I’m all for shift left, but you need to do it right so it helps developers at least as much as it hurts, or else even good developers will hate it.” This reflection highlights that just like any strategic move, shift left is good when done intelligently. Managing the change and having the right (or left, in this case) mentality for it is crucial: Let’s have a look at three relevant questions here: The complexity trade-off, the performance paradox and the skills gap.

The Complexity Trade-off

More comprehensive testing means more complex development processes. The key is automating what can be automated and making manual processes as efficient as possible.

Solution Pattern: Build testing frameworks that scale with model complexity. Investment in tooling upfront pays dividends across all projects.

The Performance Paradox

Some validation techniques (like uncertainty quantification) can impact model performance. This creates tension between accuracy and robustness.

Solution Pattern: Treat these as engineering constraints, like memory or latency requirements. Optimize within bounds rather than optimizing without them.

The Skills Gap

Many teams lack experience with comprehensive AI system testing beyond traditional ML validation.

Solution Pattern: Embed system reliability engineers within ML teams rather than creating separate quality organizations.

KPI’s: Shift Left Success, Measured 

The challenge with shifting left in AI isn't just implementing the practices, it's proving they work. You need to track how well your systems handle uncertainty, adapt to new conditions, and degrade gracefully under stress. 

The key is measuring both the process improvements (how efficiently you catch issues) and the outcome improvements (how reliably your systems perform in production). Measure what matters early, and you will thank yourself later.

Here's how leading organizations track their shift left maturity:

Development Efficiency:

• Time from issue identification to resolution

• Percentage of issues caught before production

• Model deployment frequency and success rate

System Reliability:

• Mean time between failures

• Graceful degradation under stress

• Recovery time from incidents

Business Impact:

• User satisfaction metrics

• Revenue impact of model failures

• Regulatory compliance costs

The Reality Check (and the Best Pasta)

Shifting left in AI development isn't about perfection. It's about building systems that fail predictably and recover gracefully. The organizations that master this approach don't just build more reliable systems; they build systems that can evolve and adapt over time.

The alternative becomes unsustainable as AI systems grow in complexity and criticality. The teams that shift left now will have a significant advantage as the stakes continue to rise.

Essentially, we all know that salting your pasta water instead of adding salt on top of your pasta is the better option. Especially if we’re talking about high-quality pasta, like Rummo’s Mezzi Rigatoni no 51, for example. So what are you waiting for? 

Until next time.
On behalf of Team Lumiera

Emma - Business Strategist
Allegra - Data Specialist

Lumiera has gathered the brightest people from the technology and policy sectors to give you top-quality advice so you can navigate the new AI Era.

Follow the carefully curated Lumiera podcast playlist to stay informed and challenged on all things AI.

🗞️ Issue 69 // ⏱️ Read Time: 6 min

[Looking for the section Big Tech News of the Week? Move to the end of this newsletter.]

Hello 👋

In a previous issue of The Loop, we explored the profound importance of data privacy to our autonomy in the age of AI. This week, we continue that conversation by examining the largely invisible data collection ecosystem powering today's AI models.

We'll examine the driving forces behind this data race and ask whether current AI development is truly revolutionary or simply a way to maximize profit and returns within familiar capitalist models.

The Data Collection Race: Who Wants to Know You Best?

AI chatbots compete heavily on data. Google’s Gemini collects a reported 22 unique data points, more than many others. This includes precise location, contacts, content, history, identifiers, usage, and even purchases.

Source: Surfshark

Notably, only Poe, Gemini, and Perplexity track purchases, and only Gemini and Poe collect contacts from your device. Most bots collect user content easily linked to third-party data for targeted ads.

The real contest is not just about who has the most data, but who can best transform that data into influence over user behavior, market share, and ultimately, the future direction of digital society.

Why Are AI Companies Doing This?

Data means profit. The more they know you, the better they can target ads and personalize experiences to keep you engaged and monetize your attention. This Attention Economy, intertwined with the Data Economy is the foundation of most tech business models; the longer they keep you on their platforms, the more valuable you become. Perplexity's CEO openly admitted they're building a browser to track everything you do online, aiming to sell "hyper-personalized" ads.

“We want to get data even outside the app to better understand you… what things you’re buying; which hotels you’re going to; what restaurants you’re visiting; what you’re spending time on-this tells so much about you.” - Aravind Srinivas, Perplexity CEO

This is the model that built Google. It's why AI firms are eyeing acquiring Chrome; it's a data goldmine.

However, a new economic model is emerging: the Intention Economy. Driven by advanced AI, the future lies in understanding and fulfilling your underlying desires and needs directly. Agentic AI, capable of completing tasks on your behalf, is already beginning to bypass attention-based interactions, signaling a future where your intentions, not your attention, are the primary currency.

Surveillance Capitalism: The Bigger Picture

"Surveillance capitalism" describes using digital tech, including AI, to track, predict, and influence behavior for profit. AI becomes an agent of capital, optimizing for engagement, profit, and control by turning your digital life into valuable data.

Your Personal Yes Man: Sycophantic Language Models

AI models can be overly agreeable or inauthentic. Another word for this is: sycophantic. OpenAI recently scaled back a GPT-4o update after it became excessively sycophantic, sometimes agreeing with problematic ideas just to please users. Beyond just being uncomfortable or unsettling, this kind of behavior can raise safety concerns, including around issues like mental health, emotional over-reliance, or risky behavior.

There are several ways in which the development and behavior of sycophantic language models are connected to capitalist dynamics:

• Optimizing for Engagement: Models are built to keep you interacting; You are not as likely to engage with a model if you experience friction. Being agreeable is a way to get positive feedback and keep you online, which means more data and potential revenue.

• Short-term Focus: OpenAI admitted relying too much on immediate feedback, which can lead to prioritizing quick fixes and user satisfaction over foundational issues and long-term ethical development, mirroring capitalism's push for quick returns over long-term integrity.

• Reinforcing Power: Sycophancy avoids controversy, maintaining a smooth, profitable user experience that benefits the companies and investors.

AI for Revolutionary Change v. AI for Low-Hanging Profit 

While the relationship between AI development and capitalist incentives raises valid concerns, it's crucial to recognize that AI has immense potential for good: solving global problems, advancing science, and empowering individuals. And importantly, organizations are demonstrating that AI can achieve significant social impact while also being economically sustainable, proving that innovation for public benefit doesn't have to come at the expense of financial viability, nor does profitability require undermining societal well-being.

Here are 3 impactful initiatives demonstrating AI's potential for societal transformation:

1. International Rescue Committee (IRC): Their "aprendIA" chatbot provides AI-driven personalized learning for children in crisis zones, improving education where traditional schooling is disrupted.

2. Afri-SET WEST: Focused on West Africa, they enhance air quality monitoring by using AI to evaluate and calibrate low-cost sensors, providing crucial knowledge for the region.

3. CARE International UK: This global humanitarian group uses AI tools to predict and prevent health crises, leveraging models to forecast and mitigate disease outbreaks.

The Bottom Line

Despite the inspiring potential and real-world examples of AI being used for good, the dominant business model remains rooted in surveillance capitalism: collect vast amounts of data, monetize it through advertising and personalization, and solidify market dominance. 

Understanding these underlying economic forces is the crucial first step towards regaining control over our digital lives. It also positions us to collectively demand a future where AI innovation serves humanity first, rather than simply optimizing for profit.

Until next time.
On behalf of Team Lumiera

Emma - Business Strategist
Allegra - Data Specialist

Lumiera has gathered the brightest people from the technology and policy sectors to give you top-quality advice so you can navigate the new AI Era.

Follow the carefully curated Lumiera podcast playlist to stay informed and challenged on all things AI.

🗞️ Issue 68 // ⏱️ Read Time: 7 min

[Looking for the section Big Tech News of the Week? Move to the end of this newsletter.]

Hello 👋

Last week, an AI-powered customer support system for Cursor (an AI coding tool) fabricated an entire company policy out of thin air, causing a significant user exodus. When users began experiencing unexpected logouts while switching between devices, they naturally contacted support for answers. The AI confidently explained this was "expected behaviour" under a new one-device login policy. 

The problem? No such policy existed. It was completely made up by the AI.

The false explanation spread like wildfire. Within hours, dozens of users canceled their subscriptions, community forums erupted with complaints, and by the time Cursor's team realized what had happened, the company was facing a full-blown crisis. What had actually been a simple backend bug became a case study in what AI researchers call "hallucinations", as well as their potentially devastating business impact.

As Berkeley researcher Shomit Ghose puts it: "We may now be finding that the AI 'ghost in the machine' that we all should fear is not sentience, but simple hallucination."

Understanding AI Hallucinations: Making Something Out of Nothing

AI hallucinations occur when AI systems generate information that sounds plausible but is factually incorrect or entirely fabricated. Unlike human hallucinations (a false perception of objects or events involving our senses, which stem from neurochemical factors), AI hallucinations are an inherent product of how these systems work. In a sense, the term is misleading: What we are actually talking about when a model hallucinates is that it is outputting false information.

Researchers classify these hallucinations into two main types:

1. Factuality hallucinations: Getting facts objectively wrong, like inventing a non-existent Cursor policy.

2. Faithfulness hallucinations: Diverging from user instructions or producing bizarre, unexpected outputs.

Why AI Systems Hallucinate

Something important to understand is that hallucinations aren't bugs. They are actually features of how modern AI systems function.

 If you want to do a full deep dive on how Large Language Models function, check out one of our favourite illustrations from the FT here.

These four points help you understand how AI models are built and how they function:

1. They're advanced autocomplete, not truth-seekers: AI models predict what text should come next based on patterns, not factual correctness. 

2. Training data limitations: No dataset, however vast, can cover all possible scenarios.

3. No understanding of "knowing": AI lacks metacognitive ability to recognize when it's reached the limits of its knowledge (have you ever heard about the Dunning-Kruger effect, a cognitive bias where individuals with low ability at a task overestimate their ability?)

4. Pattern completion over accuracy: When uncertain, AI will generate plausible-sounding content rather than admitting ignorance. It’s kind of like that guy that brings his guitar to the afterparty: It’s not the skills carrying the performance, but rather the confidence. 

Are AI systems liars or bullshitters?

Philosopher Harry Frankfurt's work ‘On Bullshit’ provides a useful framework here. Frankfurt distinguishes between liars (who know the truth but subvert it) and bullshitters (who are indifferent to truth). AI systems operate like the latter: Not deliberately deceiving, but generating content with complete indifference to its truth value. However, recent research also indicates that AI models might be capable of “lying”, or scheming as it’s called in AI research language.

When Hallucinations Help vs. Harm

Potential benefits (that Team Lumiera are sceptical of, but we’ll let you decide for yourselves):

• Creative applications in storytelling and content generation

• Hypothesis generation in research contexts

• Thought experiments and exploring counterfactuals

• Some people in the AI industry even claim that hallucinations are a feature. This is a clearly biased perspective that is criticized by experts.

Critical risks (we cannot highlight enough how critical these risks are. Read each point twice - this is important stuff!)

• Spread of misinformation when presented as factual information

• Erosion of trust in AI systems and implementing organizations

• Business disruption and reputational damage

• Decision-making based on fabricated information

Hallucination in AI systems is inevitable because these models cannot learn all possible functions and will therefore always hallucinate. We can reduce frequency, but never eliminate the risk entirely.

AI-Driven Customer Support

Customer support has emerged as one of the most widely adopted AI applications, and for good reason:

• Cost efficiency: Automating routine inquiries can reduce support costs by 15-30%

• 24/7 availability: AI provides round-the-clock service without staffing concerns

• Scalability: Systems handle thousands of simultaneous inquiries

• Consistency: Standardized responses create uniform customer experiences

Yet the Cursor incident reveals the unique dangers of hallucinations in this context:

1. Authority position: Support channels carry institutional authority, so customers inherently trust information provided through them

2. Decision catalyst: Support information directly influences customer decisions

3. Rapid amplification: Incorrect information spreads quickly through user communities

4. Delayed detection: By the time human oversight catches the issue, damage is often done

The Business Case for Responsible AI

Business consequences hitting hard

The real-world business consequences of AI hallucinations can be severe. Examples include:

• Google lost $100 billion in market value in a single day following a factual error made by its Bard chatbot

• Air Canada was found financially liable by Canadian courts for statements made by its customer service chatbot

• DPD faced public embarrassment when its chatbot began swearing and criticizing the company itself

Lessons on how to integrate AI in company operations

The Cursor incident provides several crucial lessons for business leaders deploying AI in customer-facing roles:

1. Human-in-the-loop is non-negotiable. Fully autonomous AI support systems create unacceptable business risks. Implement human oversight for novel questions, policy explanations, and high-stakes interactions.

2. Implement anti-hallucination technologies. Several approaches can significantly reduce hallucination risks:

• Retrieval-Augmented Generation (RAG): Augments prompts with information retrieved from trusted knowledge stores in real-time

• Knowledge Graphs: Use structured databases of entities and relationships to ground AI responses

• Confidence Scoring: Systems that flag potential hallucinations can prevent small issues from becoming crises

• Detection Systems: Solutions like LettuceDetect or RAG-HAT that actively validate AI outputs against factual sources before they reach customers

3. Transparency builds resilience. Organizations that clearly identify when customers are interacting with AI establish appropriate expectations and build trust through explicit labeling and clear escalation paths.

4. Evaluate before you deploy. Before putting an AI model in front of customers, run it through structured evaluations using your real data.

• Match the context: Choose models suited for short (chat), medium (email), or long (reporting) context lengths.

• Shortlist top performers: Prioritize models with strong accuracy for your task.

• Use your real-world examples: Generic benchmarks don’t capture your edge cases. Test with your own support tickets, FAQs, and logs.

5. Truth as foundational value. As Frankfurt (the philosopher that wrote ‘On Bullshit’, remember?) argued, respect for truth is foundational to civilization. AI systems must be designed to prioritize accuracy over plausible-sounding but potentially false content, which is something some of the biggest AI companies are not doing today.

Rather than slowing AI adoption, understanding hallucinations should accelerate investments in responsible AI frameworks. If you understand the risks of AI systems, you can mitigate the unnecessary risks and take calculated risks when it comes down to the rest. That’s what skilled business decisions are all about. After all, AI support that requires damage control isn't really helping anyone.

Until next time.
On behalf of Team Lumiera

Emma - Business Strategist
Allegra - Data Specialist

Lumiera has gathered the brightest people from the technology and policy sectors to give you top-quality advice so you can navigate the new AI Era.

Follow the carefully curated Lumiera podcast playlist to stay informed and challenged on all things AI.

🗞️ Issue 67 // ⏱️ Read Time: 5 min

Hello 👋

AI is transforming businesses fast, but it’s also creating a hidden challenge: technical debt. Companies rushing AI adoption often take shortcuts, causing problems down the road. This debt can slow innovation and hinder maintenance if not managed carefully. Let's explore what it means, why it matters, and how to deal with it.

What is Technical Debt?

Technical debt (a.k.a tech debt) is like taking a shortcut to finish a project faster, knowing you’ll have to fix or improve things later. In software, tech debt happens when teams choose quick fixes or simpler solutions to meet deadlines or launch products faster, instead of building things the best way from the start. And we all know the tech industry likes to “move fast and break things.” 

Sometimes, taking on some debt helps achieve urgent goals. But, just like financial debt, tech debt “accrues interest.” Left unaddressed, it takes more time and resources to fix, slowing future progress.

How AI Changes Things

AI is promoted as a way to get quick results, with tools that promise to build software or automate tasks at the click of a button - another tempting shortcut. We got into the details of this in our previous newsletter on vibe coding. While saving time and increasing access, these solutions introduce risks. 

When organizations rely on AI-powered tools or no-code platforms to build complex systems, they may not fully understand how those systems work under the hood. This lack of understanding can be dangerous: some companies have faced costly outages or security breaches when an AI-powered system failed and no one on the team had the skills to fix it.

Even worse, technical debt can quietly pile up as automated decisions and shortcuts accumulate in ways no one notices until it’s too late. Organizations can end up dependent on systems they can’t control or maintain, making future changes costly and risky.

Tech Debt is on the Rise

The evidence is clear: technical debt is rising as organizations rush to implement AI solutions. Forrester predicts a “technical debt tsunami” in 2025 and beyond, with more than half of technology leaders already reporting moderate to high levels of tech debt, a figure expected to reach 75% by 2026.

Several factors are driving this trend:

• Complexity and Fragmentation: AI projects often require integrating new models into legacy systems, increasing IT landscape complexity and the risk of accumulating debt.

• Speed Over Stability: The pressure to deploy AI quickly can lead to shortcuts in design, documentation, and testing, compounding future maintenance costs.

• Ongoing Maintenance: AI models must be regularly updated to correct for bias and to adapt to concept drift, which occurs when real-world data evolves and the model’s predictions become less reliable. This ongoing need for adjustment and oversight adds new challenges to managing technical debt.

At this point, AI is one of the highest contributors to tech debt.

Source: Accenture

"Perfect" Architecture vs. Shipping Value

While managing tech debt is important, there's a compelling argument that too much focus on "doing things right" can be equally problematic, especially for startups and innovation-focused teams.

A startup advisor recently shared on LinkedIn how a founder came to him after their $5.3M-funded company nearly ran out of runway. The CTO had spent 22 months building “the right way,” with perfect architecture rebuilt for every product pivot, while the market kept changing.

The key: iterate scrappy, but don’t scale scrappy. Quick, imperfect solutions help you pivot and find product-market fit, but once it’s time to scale, investing in solid foundations is essential. Balance speed with regular improvements and clear triggers for when to strengthen your tech so that debt never gets out of hand.

Tackling Tech Debt Proactively

Zero technical debt is not a realistic goal, but managing it well is possible. Proactive organizations regularly update their systems, keep data organized, and set aside budget for ongoing improvements, treating tech debt as a strategic business issue, not just a technical one.

Using AI to Reduce Tech Debt

Luckily, AI isn’t just part of the problem, it’s also part of the solution. Here are ways AI tackles the challenge:

1. Smarter Code Analysis: AI can spot hidden patterns of technical debt, like duplicated logic or overly complex code, faster and more thoroughly than traditional tools, helping teams target fixes where they matter most.

2. Automated Refactoring Assistance: Refactoring means cleaning up and improving existing code without changing what it does. AI-powered tools can suggest and even make these improvements automatically, streamlining cleanup and reducing manual effort.

3. Understanding Legacy Code: Legacy code is old software that’s still in use but often poorly documented or written in outdated languages. AI can analyze, document, and even help modernize these systems, making it easier to tackle long-standing debt.

Ultimately, successfully harnessing AI's power requires a deliberate approach to technical debt. Balancing the need for speed with a commitment to robust, maintainable systems is the key to unlocking long-term value. Striking the right balance isn’t a one-time effort, it’s a continuous process that demands discipline as your organization grows.

Until next time.
On behalf of Team Lumiera

Emma - Business Strategist
Allegra - Data Specialist

Lumiera has gathered the brightest people from the technology and policy sectors to give you top-quality advice so you can navigate the new AI Era.

Follow the carefully curated Lumiera podcast playlist to stay informed and challenged on all things AI.

🗞️ Issue 66 // ⏱️ Read Time: 5 min

Hello 👋

As health tech founders increasingly turn to GenAI, how do you separate hype from reality? This week, we're sharing key insights from Dr. Anna McLaughlin, a neuroscientist and founder, on the promises and pitfalls of AI in healthcare. We'll highlight direct quotes from her recent blog post and Lumiera’s perspective on these critical issues.

Meet the Founder of Sci-Translate

Dr Anna McLaughlin, Neuroscientist & Founder With a PhD in Neuroscience & Psychology from King's College London and extensive research experience, Anna is uniquely positioned to bridge the gap between complex science and practical application. She founded Sci-Translate to help businesses create science-backed health products and interventions. Read more about Anna here.

Lumiera’s Top 5 Takeaways

Here are our favorite quotes and expert insights after reading “Science in the Age of AI: Can We Trust Gen AI in Health Tech?”.

Takeaway 1: AI Doesn't "Understand" Science

"AI doesn’t understand science — it mimics it. LLMs generate responses by predicting the most likely next word based on probability, not comprehension."

Source: Financial Times

🔆 Lumiera Insight: Even with the recent advancements in AI reasoning capabilities that go beyond simple mimicry (e.g., chain-of-thought), the core point still stands; these models don't truly "understand" in the way humans do. Human expertise remains essential for interpreting conflicting evidence, spotting methodological flaws, and recognizing when new research genuinely challenges established thinking. This is why AI works best as a powerful tool to accelerate scientific work rather than as the final authority on scientific truth.

Dive into this section: The Hidden Flaws in AI’s Scientific Analysis

Takeaway 2: The Illusion of Accuracy is a Real Risk

"AI models are often most dangerous when they’re wrong with confidence."

Source: McKinsey

🔆 Lumiera Insight: The challenge with well-written responses is that they can mask factual errors or oversimplified conclusions. The authoritative tone of AI outputs is particularly risky in health, where a confidently stated but incorrect interpretation of something like a clinical study could influence decisions that affect patient outcomes. We already know how it goes when we start googling our symptoms, now we’re reaching the next level, where we blindly believe what we see on the other end.

Dive into this section: Challenges of Using AI in Healthtech Startups

Takeaway 3: Bias Can Skew Results

"AI reflects its training data — and that data may not include the people you serve."

🔆 Lumiera Insight: Research on marginalized communities is already scarce; relying solely on AI's limited view risks amplifying existing healthcare disparities. For example, a model confidently summarizing depression treatments based only on open-access literature may overlook culturally specific care practices or alternative approaches detailed in paywalled or less-accessible sources. The result? Decisions that unintentionally sideline the needs of underrepresented populations.

Dive into this section: AI tools for scientific research: Should you use them?

Takeaway 4: Augment, Don't Replace Human Expertise

"Sam Altman, CEO of OpenAI, says he only uses AI for ‘boring tasks’...Use AI for iterative tasks (like summarising, formatting, or data cleaning), but keep humans in the loop for all critical thinking, interpretation, and decision-making."

Source: LLMs as Research Tools

🔆 Lumiera Insight: AI should accelerate large-scale, time-consuming tasks and support, not replace, complex human reasoning. The goal should be deeper human insights, not shortcut conclusions. In practice, this could mean using AI to process massive datasets or auto-generate research summaries, freeing your scientific team to focus on the complex questions that impact patient outcomes, like whether a screening tool validated in university students will work for elderly populations, or if your intervention needs to be redesigned for communities with different approaches to mental health.

Dive into this section: AI tools for scientific research: should you use them?

Takeaway 5: The Right Questions Still Matter

"If you only ask questions that AI is good at answering, you may miss the questions that actually matter."

🔆 Lumiera Insight: The "monoculture of knowing" in health is a bit like trying to understand a city by only looking at its Google Maps data. You'll see the traffic patterns and popular spots, but miss the details that make each neighborhood unique. When we limit ourselves to AI-friendly inquiries, we risk building solutions that look good on paper but miss crucial human elements: the nuances that affect medication adherence, the unspoken fears that prevent people from seeking care, or the community dynamics that influence health behaviors. These insights rarely emerge from data alone; they come from sitting with patients, observing behaviors, and asking questions AI wouldn't think to ask.

Dive into this section: The red flags scientists are seeing in AI

At Lumiera, we believe in the power of intellectual generosity and perspective density. That's why we feature insights from experts like Dr. McLaughlin. Her work at Sci-Translate exemplifies these values by bridging the gap between cutting-edge scientific research and real-world application, empowering businesses to build more effective, science-backed health solutions. By sharing Anna's expertise, we aim to provide you with the tools and knowledge you need to navigate the fastest-evolving fields in technology.

Don’t forget: We’re meeting IRL in Lisbon at the end of April, and you’re invited! See all the information below.

Lumiera and Envisioning are teaming up to invite you to an evening around the intersection of AI & Foresight. Expect cutting-edge insights and meaningful connections among innovation, policy, and technology professionals. As a newsletter reader, you have 20% off with this code: LULX20

Responsible AI Horizons: A Foresight Experience with Lumiera × Envisioning · Luma

Join Lumiera and Envisioning for an exclusive evening at the intersection of Responsible AI and strategic foresight. During this 3-hour immersive experience,…

lu.ma/sqnafh8x

Sign up here!

lu.ma/sqnafh8x

Until next time.
On behalf of Team Lumiera

Emma - Business Strategist
Allegra - Data Specialist

Lumiera has gathered the brightest people from the technology and policy sectors to give you top-quality advice so you can navigate the new AI Era.

Follow the carefully curated Lumiera podcast playlist to stay informed and challenged on all things AI.