15 Kubernetes Metrics Every DevOps Team Should Track

Enhance Your Kubernetes Strategy with These Essential Metrics

Download our comprehensive eBook on optimizing Kubernetes performance. This guide delves into crucial cluster state, resource, and control plane metrics, highlighting 15 of the most essential metrics your DevOps team should be tracking. Learn how to gain complete visibility into your containerized environments and optimize Kubernetes performance with Datadog.

What Does OSI Stand For?

OSI stands for Open Systems Interconnection. The name itself reveals its purpose: to create an open standard for different computer systems to connect and communicate with one another, regardless of their underlying architecture. It was developed by the International Organization for Standardization (ISO) to guide the creation of universal networking protocols.

The 7 Layers of the OSI Model at a Glance

Diving Into the Upper Layers: Where Applications Talk

Our trip through the OSI model kicks off at the very top of the stack, with the layers closest to you, the end-user. These three Application, Presentation, and Session are often lumped together as the "upper layers." They're almost always handled by software and are all about getting data from your applications ready for its big journey across the network. Their job is to make sure the data is understandable, secure, and part of a smooth, organized conversation.

Layer 7: The Application Layer

The Application Layer is the interface you interact with, but it often gets confused with the application itself, like Chrome or Outlook. Instead, it's the set of protocols these applications use to communicate with the network. When you input a URL or send an email, you're using a Layer 7 protocol, allowing software to access network services directly.

Examples of Application Layer Protocols:

• HTTP/S: Used by browsers to request and display web pages.

• SMTP: Used by email clients to send messages to mail servers.

• FTP: Designed for transferring files between a client and server.

• DNS: Translates domain names into IP addresses when you enter a URL.

Layer 6: The Presentation Layer

Once the Application Layer prepares its message, it passes it to the Presentation Layer, which acts as a universal translator. This layer ensures data from one system's application layer is readable by another's. Its main tasks are:

1. Data Translation and Formatting: Converts data into a standard format, like changing EBCDIC to ASCII, ensuring consistency across systems.

2. Data Compression: Reduces data size for efficient transmission, with decompression on the receiving end.

3. Encryption and Decryption: Secures data by encrypting it at the sender's end and decrypting it at the receiver's, using protocols like SSL and TLS.

Example: When accessing your bank's website (https://), TLS encryption at Layer 6 secures your login credentials.

Layer 5: The Session Layer

The Session Layer acts as the conversation manager, organizing communication "sessions" between devices. Unlike lower layers that handle raw data transfer, Layer 5 sets up, manages, and dismantles these sessions. For instance, when accessing an online bank account, it maintains your session as you navigate between different sections, preventing repeated logins.

Additionally, the Session Layer can implement checkpoints during data transfers. If a download, like a 2GB file, is interrupted at 1.5GB, it can resume from the last checkpoint rather than restarting. This mechanism is part of reliable data transfer, further explained in our guide on what is TCP.

Understanding the Core Transport and Network Layers

As our data makes its way down from the application-focused upper layers, it hits the very heart of the OSI model: the Transport and Network layers. These two layers are the engine room of network communication. They work hand-in-hand to make sure your data doesn't just get to the right place, but that it also arrives reliably and in one piece.

Think of them as the logisticians and traffic directors of the internet.

Layer 4: The Transport Layer

The Transport Layer, or Layer 4, ensures effective end-to-end communication and data integrity. It divides data from higher layers into smaller units called segments (for TCP) or datagrams (for UDP) and reassembles them in order on the receiving side.

Think of Layer 4 as a shipping manager deciding on the delivery method: whether a package needs confirmation upon receipt or can be quickly dropped off. This decision is crucial, leading to the two key protocols of the Transport Layer:

Transport Layer Protocols:

• TCP (Transmission Control Protocol): Acts like "certified mail" slow but reliable. It establishes a connection via a three-way handshake, numbers each segment, ensures delivery, and requests resends for lost segments. Ideal for tasks needing precision, such as web pages, emails, and file downloads.

• UDP (User Datagram Protocol): Functions as a speedy, connectionless "fire-and-forget" method. It sends segments without establishing a connection or verifying arrival, reducing overhead. This is perfect for applications where speed is crucial, like video streaming, online gaming, or DNS lookups.

Layer 3: The Network Layer

Below the Transport Layer is Layer 3, the Network Layer, which handles routing data packets from a source to a destination network. Using logical addressing, each network device receives a unique IP address. The Network Layer adds a header with source and destination IPs to segments, forming packets.

For instance, when accessing a website, your computer (e.g., IP 192.168.1.10) sends a request to the server (e.g., IP 104.18.30.123). Routers forward packets using the destination IP until they reach the server, a process known as routing. If a route gets congested or fails, routers dynamically find alternate paths to ensure data transmission continues smoothly. For more on network addresses and routing, explore topics like IPv4, IPv6, and NAT.

Navigating the Lower Layers of Hardware and Links

As data travels down from the core layers, it finally hits the world of hardware and local connections. Here, abstract ideas like routing and transport get very real. We're now talking about how to get data between machines that are physically on the same network. This is the realm of MAC addresses, network switches, and the raw electrical signals that form the bedrock of the entire OSI model.

We'll kick things off with Layer 2, which serves as the crucial bridge between the logical addressing of the Network Layer and the raw physical transmission happening just below it.

Layer 2: The Data Link Layer

The Data Link Layer facilitates direct data transfer between devices on the same local network. It encapsulates packets from Layer 3 into a frame. This layer assigns a physical address, with the MAC (Media Access Control) address acting like an apartment number within a building, distinct from the building's IP address.

Key responsibilities include:

• Framing: Encapsulates packets with a header containing source and destination MAC addresses.

• Physical Addressing: Uses MAC addresses to identify devices on a local network.

• Error Control: Conducts error checking to ensure frame integrity during transmission.

Layer 1: The Physical Layer

The Physical Layer, the base of the OSI model, focuses solely on transmitting raw bits (1s and 0s) across a physical medium, ignoring packets, frames, or addresses. It sets the physical and electrical standards for signal transmission.

Key components include cables, network interface cards, and physical ports. For businesses, resources like leased lines provide reliable Layer 1 infrastructure.

Layer 1 Technology Examples:

• Cabling: Involves Ethernet, fiber optic, and coaxial cables, detailing pin layouts and max lengths.

• Radio Frequencies: Defines radio waves for wireless data transmission.

• Electrical Signals: Specifies voltage for data on copper wires and light pulses for fiber optics.

Hubs vs. Switches: A Layer 1 vs. Layer 2 Story

To understand the difference between the bottom two layers, compare a hub (Layer 1) with a switch (Layer 2).

• A hub is basic and repeats signals to all ports without understanding addresses, operating at the Physical Layer.

• A switch is more advanced, reading MAC addresses to send data to the correct port, enhancing network efficiency and security.

This distinction is crucial for network design and troubleshooting. For more on how low-level addresses map to human-readable names, see our guide on DNS.

Tracing Data Flow Through the OSI Model

To understand how the OSI model layers function together, let's look at the journey of a single piece of data an email from the moment you hit "send" until it appears in your friend's inbox.

The Journey: Encapsulation

Your data begins at the top of the OSI stack, moving downward. Each layer adds a new header to prepare it for the network.

• Layers 7, 6, and 5 (Application, Presentation, Session): Your email client uses SMTP to send "See you at 8!" The Presentation Layer standardizes and possibly encrypts the text, while the Session Layer establishes the connection with the email server.

• Layer 4 (Transport): The Transport Layer uses TCP to segment the message into smaller parts, adding port numbers to direct the data to the email application.

• Layer 3 (Network): The Network Layer wraps each segment in a packet, attaching IP addresses for routing.

• Layer 2 (Data Link): The packet is enclosed within an Ethernet frame, which includes MAC addresses and error-checking data.

• Layer 1 (Physical): The frame is converted into bits and transmitted as electrical signals or radio waves.

Your email is now a series of signals rapidly departing your device.

Across the Internet and Back Up: De-encapsulation

Once signals reach your local router, it reverses the process: it removes the Layer 2 frame to read the Layer 3 IP packet, determines the best path, and forwards it. This hop-by-hop journey continues until the packets reach your friend's network.

Upon arrival at the recipient's computer, de-encapsulation occurs, reversing the initial process:

1. Layer 1: Physical signals become a bitstream.

2. Layer 2: Bits form frames, errors are checked, and Ethernet headers/trailers are removed.

3. Layer 3: IP packets are unwrapped, the destination IP is confirmed, and the IP header is discarded.

4. Layer 4: TCP segments are reordered and original data is recreated, removing the TCP header.

5. Layers 5, 6, and 7: The connection is managed, data is decrypted/formatted, and the message is delivered to the email client.

The OSI vs. The TCP/IP Model

Think of it like this: TCP/IP is the trusty, street-legal car that millions of people drive every single day. The OSI model is the incredibly detailed engineering schematic that a master mechanic uses to understand how every component from the engine timing to the wiring works together. You don't need the schematic to drive the car, but if you want to diagnose a tricky problem or build a new car from scratch, that schematic is priceless.

The OSI model’s true value became undeniable during the internet's explosive growth. Its layered approach provided a clear guide for creating new, interoperable technologies. You can dive deeper into how this conceptual framework became foundational to networking on Wikipedia.

Comparing OSI Model Layers to TCP/IP Model Layers

The best way to see the relationship between the two is to map them side-by-side. The OSI model’s seven layers offer a fine-grained view, while the TCP/IP model’s four layers group several functions together.

This table shows exactly how they line up.

As you can see, TCP/IP bundles the OSI model's top three layers (Application, Presentation, and Session) into a single Application layer. It does the same at the bottom, combining the Physical and Data Link layers into its Network Access layer.

When you use an app like Uber, every interaction matters. A tap, a scroll, a screen view, all of these generate signals that help teams understand:

• What users are doing

• Where they’re facing friction

• Which features are actually working

This is what powers product decisions, A/B experimentation, ML-driven recommendations and personalization. But collecting this data at scale? That’s where things get complicated. Let’s discover what Uber did in this blog today.

How Mobile Analytics Works (At a High Level)

Before diving into the problem, let’s quickly understand how analytics works inside Uber.

Source:How Uber Standardized Mobile Analytics for Cross-Platform Insights

Don’t worry if the above diagram looks too much for now, let me simplify it for you.

So, Whenever a user interacts with the app:

1. Engineers define an event (along with metadata)

2. It’s added to the UI component

3. The event gets triggered when the user interacts

4. It’s batched and sent to backend systems

5. Teams use this data for insights

This pipeline is critical for Monitoring feature health, Understanding user journeys and Powering ML-driven recommendations. So far, everything sounds clean. But systems don’t fail at small scale, they fail when multiple teams start building on top of them.

Where Things Started Breaking

Even though Uber had a structured system, things didn’t scale well. But what was the core issue? Every team handled analytics differently. The biggest issue wasn’t the pipeline. It was how teams were using it.

Each team:

• Defined events differently

• Logged data in their own way

• Added metadata manually

• Created custom events for everything

Over time, this created chaos:

• No standard definition of events

• Missing analytics in shared UI components

• Duplicate and inconsistent metadata

• Over 40% of events became generic “custom logs”

• Disabled events were silently dropped

Now think about the impact. If data is inconsistent then insights become unreliable and if insights are unreliable then decisions become risky. This wasn’t just a data problem. It was a product and business problem.

The Key Insight: The Platform Should Own Analytics

Uber realized something very important:

So they made a bold shift to move analytics logic from feature teams into the platform. This changed the model completely:

• Before, teams decided how to log events. Now, the platform defines the rules and teams just use them.

• Engineers no longer decide “how to log”

• They just use standardized tools

Step 1: Standardizing Events Across the App

Uber started by standardizing the most common user interactions. The first step was to define clear event types. They started with three core events and introduced clear definitions for:

• Tap: when a user interacts with a UI element

• Impression: when ≥50% of a view is visible for 500ms

• Scroll: when scrolling starts and stops

This might sound simple but it’s huge because earlier each team had its own definition of “impression”. Now, one definition across the entire system.

Step 2: Separating Analytics from UI Logic

Another major improvement was introducing AnalyticsBuilder. Instead of mixing analytics inside UI code:

• AnalyticsBuilder handles when events should fire

• UI components focus only on rendering

It tracks lifecycle events like View appearing and View disappearing and decides when to trigger analytics events. This makes the system more reusable, testable and consistent.

Step 3: Moving Analytics Into UI Components

Earlier Engineers manually added analytics to each feature. Now Uber instrumented 50+ common UI components. So things like Buttons, Lists and Views automatically emit analytics events.

This means, Engineers don’t need to “remember” analytics anymore. Analytics becomes default behavior, not manual effort.

Step 4: Standardizing Metadata (The Hidden Problem)

Another major issue was metadata inconsistency.

Different teams:

• Logged the same data differently

• Used different naming conventions

• Repeated the same values

Uber fixed this by introducing two layers of metadata.

1. App-Level Metadata

Data common to the entire app (like location or restaurant ID) is defined once and automatically attached to every event.

2. Event-Level Metadata

Things like scroll direction, list index and view position are automatically captured by the system.

Standardizing UI Surfaces

Every UI component is also categorized:

• Buttons → BUTTON

• Containers → CONTAINER_VIEW

• Sliders → SLIDER

This ensures that every event clearly indicates where it came from.

Step 5: Preventing Data Loss with Sampling

Earlier, if an event was disabled or unmapped, it was completely dropped which meant losing data forever.

Uber fixed this by introducing sampling, 0.1% of sessions log all events even disabled or unmapped ones.

This creates a safety net so you never fully lose visibility into your data.

Step 6: Testing Before Full Rollout

Before scaling, Uber ran a pilot.

They:

• Ran old and new systems in parallel

• Logged events using both old and new systems

• Compared outputs

• Validated consistency and correctness

They checked event volume, metadata accuracy and behavioural correctness.

This helped uncover:

• Differences between iOS and Android

• Opportunities to simplify event structures

Step 7: Scaling Across the Organization

Once validated, Uber rolled it out in two ways:

1. Central Migration: Platform team replaced old APIs directly

2. Distributed Migration

   1. Teams were supported with scripts

   2. High-impact events were prioritized

They also added a linter to block non-standard event usage so the system stays clean going forward.

Impact: Cleaner Data, Faster Development

For Data Teams

• Consistent event definitions

• Reliable cross-platform analysis

• More accurate metrics

• Reduced noise

For example, Impression accuracy improved significantly due to stricter rules

For Engineers

• Much less code to write

• No manual metadata handling

• Built-in analytics in UI components

• Faster onboarding

What’s Next: Component-Based Analytics

Uber is now pushing this even further. They’re moving towards component-based analytics. Instead of manually defining events:

Each component gets a unique ID and automatically generates event names.

Format:

[component]_[surface]_[event]

Example:

production_selection_button_tap

This reduces duplication, Improves naming consistency and simplifies tracking.

Takeaways

Uber’s biggest learning is simple: Don’t leave analytics to individual teams, build it into the platform.

By:

• Standardizing events

• Automating metadata

• Instrumenting UI components

• Preserving data through sampling

They built a system that is Scalable, Reliable and Easy to use and most importantly, one that teams can actually trust.

By now, you must have had a clear idea of, How Uber Standardized Mobile Analytics (Without Slowing Down Teams)? In a nutshell, Uber fixed inconsistent mobile analytics by moving event logic from feature teams into the platform and standardizing events, metadata, and UI instrumentation. This ensured reliable, scalable data while reducing engineering effort and improving cross-platform insights.

Congratulations! You've just advanced another step in your tech journey. Keep progressing!

DevSecOps Maturity Model

The 6 Core Competencies of Mature DevSecOps Orgs

Understand the core competencies that define mature DevSecOps organizations. This whitepaper offers a clear framework to assess your organization's current capabilities, define where you want to be, and outline practical steps to advance in your journey. Evaluate and strengthen your DevSecOps practices with Datadog's maturity model. 

Claude vs MiMo - Xiaomi’s new MiMo Code is an open-source AI coding agent designed to tackle long-running development tasks without losing context. Built on OpenCode under the MIT license, it preserves memory across sessions using a multi-layered knowledge system. Read full blog here.

Telegram enters your Smartwatch: Telegram’s latest update brings dedicated smartwatch apps, letting users send messages, listen to voice notes, view media, and manage chats directly from their wrist. The release also upgrades bots with rich formatting options like tables, carousels, and collapsible sections, making chat experiences far more interactive. Read more.

Everything You Need to Know About Claude Fable 5 and Claude Mythos 5

Anthropic has introduced Claude Fable 5 and Claude Mythos 5, its most advanced AI models yet. These models mark the arrival of a new capability tier that goes beyond traditional chatbots, focusing on long-running tasks, autonomous problem-solving, and advanced reasoning. While both models are built on the same foundation, Fable 5 is designed for general users, while Mythos 5 is reserved for trusted cybersecurity and scientific research partners.

Built for Long and Complex Tasks

One of the biggest improvements in Fable 5 is its ability to stay focused during lengthy workflows. Many AI models perform well on short prompts but struggle when tasks span hundreds of steps. Fable 5 is designed to maintain context across millions of tokens, allowing it to tackle projects that may take hours or even days to complete. Anthropic says the model can create and use its own notes, helping it remember key information and improve its outputs over time.

Major Advances in Software Engineering

Software development is one area where Fable 5 shines. During early testing, companies reported dramatic productivity gains. In one example, the model completed a codebase-wide migration in a massive 50-million-line codebase in just one day a project that would normally require months of manual work. Fable 5 can analyze large repositories, understand dependencies, update multiple files simultaneously, and generate production-ready code while remaining more token-efficient than previous Claude models.

Beyond coding, Fable 5 delivers significant improvements in research and analytical tasks. The model performs exceptionally well at document analysis, financial reasoning, chart interpretation, root-cause analysis, and problem-solving. Anthropic claims it achieved leading results on several reasoning benchmarks, making it useful for consultants, analysts, researchers, and business professionals who work with large amounts of information daily.

Vision is another area where the new model stands out. Fable 5 can extract information from scientific figures, understand diagrams, analyze screenshots, and even recreate software applications from images alone. Anthropic demonstrated these capabilities by showing the model successfully completing Pokémon FireRed using only visual inputs, without access to additional game information or navigation tools.

What Makes Mythos 5 Different?

Claude Mythos 5 is essentially the unrestricted version of Fable 5. It is designed for cybersecurity experts and scientific researchers who require access to the model's full capabilities. Anthropic claims Mythos 5 possesses the strongest cybersecurity abilities of any AI model currently available. The model has also shown promising results in drug discovery, protein engineering, molecular biology research, and genomics, helping researchers accelerate complex scientific workflows.

Safety Comes First

With greater capability comes greater risk. To prevent misuse, Anthropic has implemented advanced safeguards in Fable 5. Requests involving cybersecurity, biology, chemistry, or AI model distillation are automatically routed to a less capable model. According to the company, these safeguards affect fewer than 5% of user sessions while helping prevent harmful use cases and jailbreak attempts.

The Bigger Picture

The launch of Claude Fable 5 and Claude Mythos 5 highlights a major shift in AI development. The industry is moving beyond simply building smarter models and focusing on how to deploy powerful AI responsibly. By combining cutting-edge capabilities with layered safety mechanisms, Anthropic is attempting to balance innovation with security. As AI continues to evolve, models like Fable 5 and Mythos 5 offer a glimpse into what the next generation of intelligent systems may look like.

Read Official Blog.

Amazon’s new Side Hustle - Amazon is bringing AI-generated merchandise directly into its shopping app, letting users create custom designs with simple Alexa prompts. From T-shirts and hoodies to tumblers and water bottles, the platform handles everything from design generation to production and Prime delivery. Read more.

Share AI Agents, skills securely now: Databricks has unveiled OpenSharing, an open protocol designed to let organizations securely share not just data, but also AI models, agent skills, and unstructured assets across platforms. Hosted by the Linux Foundation, it expands on Delta Sharing to create a vendor-neutral standard for the AI era. Read more.

Buzz of the Week:

Context Persistence

Context Persistence is the ability of an AI system to maintain and retrieve state across long-running tasks, sessions, and workflows. It combines structured memory, retrieval mechanisms, and state management to prevent context loss during extended agent execution. Modern AI agents use context persistence to coordinate multi-step reasoning, resume interrupted processes, and maintain consistency across millions of tokens. It is becoming a foundational capability for autonomous coding agents, enterprise copilots, and agentic AI systems.

Things that launched. Things that went viral. Things you'll pretend to try.

chisel

chisel is a tool for cutting down Linux containers into ultra-minimal images.

s5cmd

s5cmd is a blazing-fast alternative to AWS CLI for S3 operations.

pistol

pistol preview files in the terminal automatically while navigating directories.

Build Braincells, Not Just Features

This weekend’s read: Fable 5 SYS prompt leak.

This week’s watch: Why is everyone going Blind.

Meanwhile…

Benefits of End-to-End Observability 

The real benefits of end-to-end observability
How does full-stack observability impact engineering speed, incident response, and cost control?

In this eBook from Datadog, you'll learn how real teams across industries are using observability to:

• Reduce mean time to resolution (MTTR)

• Cut tooling costs and improve team efficiency

• Align business and engineering KPIs

See how unifying your stack leads to faster troubleshooting and long-term operational gains.

When developing an application, the database serves as the core system. If it fails, database replication helps by maintaining multiple copies on different servers. This approach ensures the system remains available, scalable, and fast for users globally.

Why Replication Is Your System's Safety Net

Imagine your entire application depends on a single library. If that library suddenly closes even for a few minutes everything grinds to a halt. This is a single point of failure.

Database replication is like having synchronized copies of that library in different cities. If one closes, the others are still open and up-to-date. This approach is a cornerstone of any resilient, modern architecture. By distributing data, you build a system that can survive outages and handle growth.

The Core Goals of Replication

• High Availability: If the primary database server fails, a replica can quickly take over, ensuring continuous operation without noticeable disruption for users.

• Improved Scalability: As traffic increases, replication allows workload distribution. "Write" requests go to the main database, while "read" requests are handled by multiple replicas, preventing bottlenecks.

• Lower Latency: By placing replicas near users, data access is faster. For example, a replica in Tokyo speeds up access for users in Japan, reducing response times.

Exploring Core Replication Architectures

Choosing a data replication strategy is a crucial design decision, given the variety of models and their trade-offs in complexity, scalability, and failure handling. This is reflected in the global data replication market's considerable growth.

Three main models are prevalent: Leader-Follower, Multi-Leader, and Leaderless replication.

Leader-Follower Model

This common approach involves a leader node handling write operations and streaming data changes to follower nodes, which handle read-only queries. Suitable for read-heavy applications like blogs and e-commerce sites, it enhances read throughput by offloading reads to followers.

The model's beauty is its simplicity. With a single source of truth the leader maintaining write consistency is straightforward. However, the leader is a single point of failure for writes. If it goes down, your system can't accept new data until a follower is promoted. This is a classic example of the trade-offs in scaling strategies for your system.

Multi-Leader and Leaderless Replication

For systems requiring intensive write operations, other models may be more appropriate:

• Multi-Leader Replication: Multiple nodes can accept writes, beneficial for applications needing low-latency writes in different regions. The main issue is resolving write conflicts when simultaneous edits occur in different locations.

• Leaderless Replication: Eliminates the leader concept, allowing any node to accept writes, which are then distributed to other nodes. Used by databases like Apache Cassandra for high fault tolerance and availability, ensuring minimal impact if a node fails.

Comparing Replication Architectures at a Glance

This table helps clarify how these models stack up.

There's no single "best" architecture. The Leader-Follower model is a solid starting point for many applications, while Multi-Leader and Leaderless models offer powerful solutions for more complex, high-availability scenarios.

Balancing Data Consistency and System Availability

When replicating databases in your system design, you face a classic dilemma. Every distributed system must choose between guaranteeing data is perfectly up-to-date everywhere (consistency) and ensuring the system is always online to handle requests (availability). This challenge is famously described by the CAP theorem.

This isn't just theory it has real consequences. Our guide on https://hw.glich.co/p/what-is-cap-theorem explains it in detail, but the core idea is that during a network failure, you can't have perfect consistency and availability simultaneously.

Synchronous vs. Asynchronous Replication

Choosing between data replication methods often depends on your needs:

• Synchronous Replication: The leader waits for a follower's confirmation before responding, ensuring data safety across machines with strong consistency. However, it's slower due to the required confirmation.

• Asynchronous Replication: The leader immediately confirms writes to the client and updates followers later, offering faster performance and high availability. The risk is data loss if the leader fails before followers update.

Real-World Consistency Models

Let's see how this plays out.

A banking app must prioritize consistency. A money transfer must be perfectly reflected across all systems. Seeing an old, incorrect balance is unacceptable. Such an application would favor synchronous replication to ensure zero data loss.

Conversely, a social media feed can be more relaxed. If a "like" takes a few seconds to appear for others, it's not a critical failure. This model uses eventual consistency, where data syncs across replicas over time, making asynchronous replication a perfect fit.

Designing Systems That Survive Failure

A database replication strategy is like a safety net, but that net can still get tangled. The real goal is to build a system that expects failure and knows how to recover gracefully.

When Leaders Go Down and Networks Get Weird

One issue is leader failure; if the leader crashes, new writes can't be accepted. Automated failover is crucial; a monitoring service detects the failure, holds an "election" among followers, and appoints a new leader quickly to reduce downtime.

Another problem is network partition, where nodes run but can't communicate. In multi-leader systems, this causes split-brain, with each partition side acting as the leader, resulting in conflicting writes and data confusion.

Battling Replica Lag and Data Conflicts

Even in normal operation, systems must deal with replica lag the delay between a write hitting the leader and appearing on a follower. High lag means users see stale data, which can break application logic. Monitoring this is critical, and you can learn more about key performance indicators in this guide to database replication speed metrics.

Designing for failure means building systems that embrace instability. Our article on how Netflix ensures reliability provides a great look into how they tackle these challenges at scale.

How Top Companies Use Database Replication

Theory is one thing, but seeing database replication in system design in practice makes it real. Major tech companies use replication strategically to solve massive performance and availability problems.

This strategic importance is why the database replication software market is growing rapidly. You can find more insights on this market's growth and its projections.

Let's look at a couple of real-world examples.

E-commerce Giants and Multi-Leader Replication

A global e-commerce site needs to ensure fast user experiences across New York, Berlin, and Tokyo. If all "add to cart" actions went to a single database in North America, delays would occur for users in Asia and Europe.

To address this, they implement a Multi-Leader architecture with leader nodes in key regions. A shopper in Germany, for instance, updates their cart through a European leader, providing an instant experience. The data is then replicated to other leaders in the background.

Streaming Services and Global Content Delivery

A major video streaming service has a different challenge. Its catalog of titles and user watchlists doesn't change frequently, but it must serve content to millions of viewers globally without buffering.

For this, a Leader-Follower model is ideal.

• A central leader database is the single source of truth for all content metadata.

• This data is replicated to hundreds of read-only followers worldwide.

• These followers are placed alongside video files on a Content Delivery Network (CDN).

When you search for a movie, your request hits a nearby replica, giving a lightning-fast response. This principle of distributing read load is used in many massive systems, as detailed in our analysis of how Facebook handles billions of messages daily.

Common Questions About Database Replication

What Is the Difference Between Database Replication and Backups

This is a common point of confusion. Both involve copying data, but they solve different problems.

Replication is a live, continuous process for high availability and performance. It keeps multiple database copies in sync so one can take over instantly if another fails.

Backups are periodic, offline snapshots for disaster recovery. You use them to restore data to a specific point in time, like after an accidental mass deletion.

How Does Replication Lag Affect My Application

Replication lag refers to the delay between a write on the leader and its visibility on a follower, potentially showing users outdated data, like an old profile photo after refreshing.

A typical solution is to direct a user’s reads to the leader immediately after they make a write for a few seconds, ensuring they see their updates. Other reads continue to go to followers.

When to Use Synchronous vs. Asynchronous Replication

The choice depends on data loss tolerance:

• Synchronous replication: Use when data loss is unacceptable, such as in payment systems, guaranteeing writes are saved to a follower before success confirmation, though it increases latency.

• Asynchronous replication: Opt for this when speed is crucial and minor data loss is tolerable, suitable for social media, logging, or analytics where performance is prioritized over absolute data durability.

Balancing these is essential in system design, with many large systems employing a combination of both.

Microservices promise a lot of features like better scalability, faster development, independent teams and more flexibility. And honestly, in the beginning, they do deliver. But as systems grow, something strange starts happening. Instead of clarity, you get chaos. With microservices, now you have hundreds of APIs, different formats, inconsistent data, hidden services and broken integrations. What once felt like a clean architecture slowly turns into what people jokingly call a “microservices death star.”

So the real question becomes: How do you scale microservices without losing control? And that’s exactly where API Federation comes in.

The Problem with Scaling Microservices

When organizations grow, their API landscape grows with them. You start with a few services, then dozens, then hundreds. At that point, new challenges appear:

• APIs become inconsistent

• Discoverability becomes difficult

• Observability becomes harder

• Changes start breaking things

And the worst part? Every team is moving fast but in slightly different directions. Without a structured approach, your system becomes hard to understand, hard to evolve and even harder to maintain.

So, what is API Federation?

To solve this, we need a better way to manage APIs at scale. That’s where API Federation comes in! In simple terms: API Federation is a way to combine multiple services into one unified API without forcing teams to tightly couple their systems.

It allows:

• Different services to evolve independently

• While still exposing a consistent interface externally

You can think of it like this Instead of clients calling 10 different APIs, they call one federated API, and behind the scenes, it fetches data from multiple services.

What It’s NOT

Before going deeper, let’s clear some misconceptions:

1. It’s NOT a single “universal model”

Trying to create one global schema for the entire organization doesn’t work. Why?

Because:

• Teams have different needs

• Domains evolve differently

• Central control slows everything down

This often leads to rigid systems that eventually break.

2. It’s NOT just GraphQL

GraphQL is commonly used for federation, but it’s not mandatory. API Federation is a design approach, not a specific technology.

It can work with:

• REST APIs

• Event-driven systems

• Other architectures

Breaking Systems into Bounded Contexts

The first big idea behind API Federation comes from Domain-Driven Design (DDD). Instead of treating your system as one giant API, you divide it into bounded contexts.

What is a bounded context?

A group of services that:

• Share the same domain concepts

• Speak the same “language”

• Represent data consistently

For example:

• A “Customer” should mean the same thing across services in that context

• Identifiers for entities should match everywhere

• Data types should be consistent

It matters because federation works best when data meanings are aligned, IDs are consistent and schemas are compatible. Without this, combining APIs becomes messy.

How Federation Actually Works

Once you have a bounded context, federation combines services using a few key patterns:

1. Entity Extension

Different services may provide different fields for the same entity. Federation merges them into one unified view.

For example, one service might return customer details, while another returns their orders. Federation combines both into a single response.

Federated API → both combined

2. Entity Linking

Services can reference entities from other services using shared identifiers. Instead of directly linking services: They connect through common entity types and keys

This is powerful because if one service changes or disappears → system doesn’t break

3. Entity Composition

Sometimes entities don’t have identifiers (like value objects).

In such cases:

• Federation combines available attributes dynamically

A Smarter Way to Design APIs

Traditionally, APIs are built as large, monolithic interfaces. But federation encourages something different: Build APIs as small reusable features

Instead of one big API doing everything, break it into smaller capabilities.

Examples of features:

• CRUD operations

• Pagination

• Filtering

• Search

• Event streaming

Each feature:

• Can be developed independently

• Can be reused across services

• Can evolve separately

This works better because now teams don’t depend on entire APIs, they depend only on specific features. And federation combines these features into one unified interface.

How Data is Fetched Across Services

To make federation work, services must support something called: Entity resolution

This means: Given an entity ID → return its data

Using this:

• Federation can fetch data from multiple services

• Join them using shared keys

• Return a single response

For example: Fetch customer data, Fetch related orders and Combine everything. All in one query.

Balancing Flexibility and Control

Here’s a tricky part. You want teams to move independently. But also APIs to stay consistent This is where the idea of a federation protocol comes in.

What is a federation protocol?

A set of rules that:

• Services must follow to be part of the federated API

• Defines how conflicts are handled

• Controls how schemas evolve

Example rules:

• If two services define the same entity → they must agree on keys

• If attributes conflict → resolve or rename

• If a service can’t fully support federation → fallback mechanisms apply

It’s not fully centralized, and not completely decentralized either. It sits somewhere in between, giving teams freedom while still maintaining consistency.

Handling Changes Over Time

In real systems, APIs change all the time. Federation supports this by:

• Automatically adapting to new attributes

• Reflecting removed fields

• Allowing gradual adoption of federation patterns

This means you don’t need a perfect system from day one, you can evolve into federation over time.

Keeping Things Stable in Production

Adding a federation layer shouldn’t break things. So it must work with:

• Authentication systems

• Caching

• Rate limiting

• Monitoring

• SLAs

At the same time, federation introduces shared requirements like common authentication across services and shared entity identifiers.

Does Federation Affect Performance?

A common concern: “Won’t federation slow things down?”

Well, not necessarily. In fact:

• It can improve efficiency by optimizing queries

• It allows scaling specific services independently

The key is:

• Efficient query planning

• Leveraging capabilities of underlying services

Making it Work at Scale

Federation isn’t just an architectural idea, it also impacts your development lifecycle.

You need:

• CI/CD pipelines

• Metadata management

• API contract validation

• Automated deployments

Every time a service changes:

• Federation layer must update

• Consistency checks must run

Tools That Help (Like DataGraph)

Tools like MuleSoft’s DataGraph help automate this entire process.

They:

• Use API metadata (like OAS/RAML)

• Generate federated schemas

• Handle query execution

• Provide monitoring and logging

And importantly, they simplify federation into a manageable workflow.

Key Takeaways

API Federation is not just a technical solution, it’s a way of thinking. It helps solve one of the biggest problems in modern systems: How do you scale without losing control?

By:

• Structuring APIs into bounded contexts

• Designing modular features

• Combining services intelligently

• Balancing autonomy with governance

In the end, it gives you the best of both worlds i.e., Independent teams and Unified experience. And that’s what makes it powerful.

By now, you must have had a clear idea of, When Microservices Get Messy: How API Federation Brings Order. In a nutshell, API Federation unifies multiple microservices into a single API using bounded contexts, shared models, and feature-based design. It balances team autonomy with consistency while simplifying complex API ecosystems.

Congratulations! You've just advanced another step in your tech journey. Keep progressing!

State of AI Engineering

What The State of AI Engineering Report Reveals About AI in Production

AI engineering has moved past experimentation, but are your observability practices keeping up? Datadog analyzed LLM telemetry from 1,000+ customers to reveal what's actually happening at scale: model adoption shifts, hidden token costs, and what the rise of agentic frameworks means for reliability. Download the report to benchmark your AI stack against production reality.

AI Finds What Humans Missed ft. Redis - An AI security tool discovered a critical Redis vulnerability that had gone unnoticed for over two years. The flaw could have allowed attackers to compromise vulnerable systems. It’s another sign that AI is becoming a powerful ally in modern cybersecurity. Read more.

Zapier got Zapocalypse - Researchers uncovered an attack chain that could have enabled full takeover of authenticated Zapier accounts. By chaining together multiple seemingly minor security weaknesses, they achieved escalating levels of access. The issue has since been fixed, highlighting the risks hidden in complex cloud environments. Read more.

The Internet Has a New Majority: Bots

For decades, the web was built around human users. Every click, search, purchase, and page visit was driven by people navigating the internet themselves. That reality is changing faster than many experts expected.

According to Cloudflare CEO Matthew Prince, automated bots now generate more web traffic than humans for the first time in internet history. Recent Cloudflare data shows bots account for 57.5% of HTTP requests, while human traffic has fallen to 42.5%. What makes this shift remarkable is that Prince previously predicted this crossover would not happen until 2027.

Not the Bots You're Thinking Of

When most people hear the word "bot," they think of spam, scams, or search engine crawlers. However, the new wave of traffic comes from AI-powered agents acting on behalf of users.

These agents can browse websites, compare products, check prices, search for flights, gather information, and even complete multi-step tasks without requiring direct human interaction. Instead of people visiting ten websites to compare options, an AI agent may do that work automatically and return a summarized result.

This represents a significant change in how the internet is being used.

The Rise of Agentic Traffic

The growth of AI assistants and autonomous agents is fueling this trend. Modern AI systems are increasingly capable of navigating websites, collecting information, and performing actions that once required human clicks.

Cloudflare has been tracking these new categories of visitors, including verified bots and signed agents. Their data suggests that AI agents are now operating at a scale large enough to reshape internet traffic patterns.

For website owners, this means a growing percentage of visitors may never actually see a webpage. Instead, AI systems may consume the content, extract relevant information, and present it directly to users elsewhere.

Why Humans Still Dominate Attention

Despite the traffic numbers, humans remain the primary consumers of online content in terms of engagement and time spent.

Activities such as watching videos, scrolling social media feeds, gaming, and using mobile apps generate long sessions but relatively few HTTP requests. AI agents, on the other hand, can generate thousands of requests in a short period while gathering information across multiple websites.

In other words, bots may be winning the traffic battle, but humans still dominate the attention economy.

What Comes Next?

The shift signals the beginning of a new internet era. Websites were originally designed for human visitors, but they may increasingly need to serve both people and AI agents.

Businesses, publishers, and developers will face new challenges around content access, AI scraping, monetization, and digital identity. As agentic AI becomes more capable, the web may evolve from a place humans browse directly into a network where software agents do much of the browsing for us.

Cloudflare's data suggests that future has already started. The only surprise is how quickly we arrived there.

AI Chief of Staff - Asana has launched Dash, an AI-powered "chief of staff" designed to monitor projects across emails, calendars, messaging apps, and Asana itself. It can spot risks, recommend next steps, and even coordinate AI agents to keep work moving.

Google Dreambeans to turn your life into a Cartoon: Google’s experimental AI tool, DreamBeans, turns data from your Google apps into personalized illustrated stories and lifestyle recommendations. It can transform memories, trips, and interests into cartoon-like visual narratives generated by AI.

Buzz of the Week:

Capability Routing

Capability Routing is the process of dynamically selecting the best AI model, tool, API, or agent for a specific task instead of sending every request to a single system. Modern AI platforms use capability routing to decide whether a query needs code generation, web search, reasoning, image creation, or data analysis. This improves accuracy, reduces costs, and speeds up response times. As AI agents become more specialized, capability routing is emerging as a core architectural pattern for agentic systems. Many developers use it unknowingly through AI platforms, but few recognize it as a distinct engineering concept.

Things that launched. Things that went viral. Things you'll pretend to try.

xsv

xsv is a lightning-fast toolkit for CSV manipulation.

ripgrep

ripgrep extends ripgrep to search inside PDFs, Office docs, ebooks, and archives.

jqp

jqp is an interactive playground for building jq queries visually.

Build Braincells, Not Just Features

This weekend’s read: Coding is no Longer the Constraint.

This week’s watch: Why everyone is Shifting to Wired earphones.

Meanwhile…

Weak consistency is a model in distributed systems where updates to data don't always show up for everyone instantly. This means if you read from the system, you're not guaranteed to see the very latest write. For a little while, different locations might see different versions of the same information.

Defining Weak Consistency Without the Jargon

Ever collaborated with a team on a shared document in the cloud? You make a crucial edit, but for a few seconds or maybe even longer your colleagues are still looking at the old version.

That delay, that brief moment of data misalignment, is the core idea behind weak consistency. It’s not a bug; it's a deliberate design choice that prioritizes system speed and availability over perfect, real-time data synchronization across every single location.

In distributed systems, weak consistency allows data synchronization with minimal guarantees. When data is updated on one server, other servers may still show outdated information until updates propagate through the network. Although this might seem like a drawback, it prevents system slowdown by avoiding the need for immediate global agreement on every update. More insights can be found at GeeksforGeeks.

Prioritizing Speed Over Immediacy

Weak consistency enables systems to quickly respond to user requests without waiting for global consensus, ensuring high availability and low latency in large-scale applications. For instance, on a viral social media post, a slight delay in displaying the exact "like" count to all users is acceptable. The system focuses on recording your "like" instantly rather than synchronizing the count globally at every moment. The key is that updates will eventually be visible to all, or possibly not at all, under weak consistency's strictest definition.

How It Compares to Other Models

To really get what is weak consistency, it helps to see how it stacks up against the alternatives. Different systems need different levels of data integrity. A banking application, for instance, could never tolerate stale data every single transaction has to be immediate and universally visible. This is where stronger consistency models come into play.

Let's break down the fundamental trade-offs between weak consistency and two other common models.

Quick Comparison of Consistency Models

This table lays out the core differences in guarantees, performance, and availability.

As you can see, there's a clear trade-off: the stronger the guarantee you need for data accuracy, the more you typically sacrifice in performance and availability. Weak consistency sits at the far end of this spectrum, giving you maximum speed by providing the most relaxed data guarantees.

Comparing Consistency Models: Strong, Eventual, and Weak

To understand weak consistency, it helps to compare it with stricter models. Choosing the right consistency model is a significant architectural decision, focusing on what's suitable for the application rather than what's "best."

Consider some real-world examples:

Strong Consistency

In banking apps like Chase or Bank of America, transactions require strong consistency. When you transfer money, your balance must update immediately and correctly for subsequent actions, with no allowance for outdated data.

The Middle Ground: Eventual Consistency

Consider your DNS (Domain Name System). When a website owner updates their domain's IP address, it doesn't have to reach every DNS server instantly. This system employs eventual consistency, ensuring that updates are accepted and, within hours, all DNS servers will reflect the new address. This delay supports the resilience and decentralization of the internet.

These aspects are central to distributed systems. For more insight into managing these trade-offs, check out our guide on the CAP theorem, which explains choosing between consistency, availability, and partition tolerance.

Weak Consistency in Action

Think of a VoIP service like Skype or a multi-user online game using the UDP protocol for real-time communication, exemplifying weak consistency. The system focuses on sending data packets quickly, without ensuring every packet arrives or maintains order.

This image helps visualize where different models fall under this broad umbrella.

Even "weak" isn't singular; there are different guarantee levels from eventual synchronization to models like causal and session consistency. By allowing some delay in data sync, architects can significantly boost performance and resilience, particularly in large-scale systems where strict consistency would cause bottlenecks.

Performance and Availability Trade-Offs

Why allow temporarily outdated data? It seems problematic, but it's intentional. The trade-off is sacrificing some immediate data uniformity for a large increase in performance and availability.

Weak consistency is a strategic decision to create faster, resilient applications that manage heavy traffic.

This decision makes perfect sense once you look at the cost of strong consistency. To guarantee every single user sees the latest update the instant it happens, a system has to jump through hoops with complex synchronization protocols. A write operation isn't considered "done" until every server in the network confirms it has received and applied the update. All that waiting adds precious milliseconds of latency to every request.

Slashing Latency for a Better User Experience

In a weakly consistent system, a server can immediately respond to a user's request without waiting for global consensus, significantly reducing latency for a quick user experience. For example, when posting a comment on a YouTube video, the system accepts it instantly and updates it across servers in the background. In contrast, a strongly consistent system might delay your confirmation to ensure it's visible everywhere. Verified benchmarks, like those on Apache Cassandra, show weakly consistent writes can be much faster, reducing latency from over 100 milliseconds to under 30 milliseconds. For more details, see this consistency models deep dive.

Seeing Weak Consistency in Real-World Systems

Theory is great, but the real "aha!" moment comes when you see where this stuff actually gets used. The truth is, you probably interact with weakly consistent systems every single day without a second thought. These aren't just abstract concepts for textbooks; they're the practical engineering choices that make many of the world's biggest apps feel so fast and responsive.

Engineers choose weak consistency for a simple reason: sometimes, speed and uptime are just flat-out more important than having perfectly synchronized data across the globe at every single millisecond.

The diagram below gives you a quick visual. A "write" happens, but for a little while, different nodes (P1, P2) might still serve up the old data. Eventually, they catch up.

This is the core trade-off in action. The system temporarily allows data to be "stale" in some places to keep everything available and performing at lightning speed.

Non-Critical Analytics and Logging

Netflix collects a vast amount of data from user interactions, such as clicks and views, which is ideal for weak consistency.

• Goal: Log billions of events without slowing down the app.

• Method: Events are sent to the nearest server, without waiting for global sync.

• Result: The app remains smooth for users, while data is processed later. A slight delay in analytics is acceptable for a seamless user experience.

High-Stakes Scenarios

Certain domains require the immediate, unified view of data that only strong consistency can provide. Using a weaker model in these cases could be costly and potentially disastrous.

Examples where weak consistency is unsuitable include:

• Financial Transactions: Online banking, fund transfers, and payment processing demand atomic operations visible across the entire system. A weak model might display incorrect balances or allow duplicate transactions.

• E-commerce Systems: In platforms like Amazon, a weakly consistent system could lead to overselling by allowing multiple users to purchase the last item due to outdated inventory views.

• Booking and Reservation Systems: Platforms like Expedia and Booking.com must ensure that once a reservation is made, it's immediately unavailable to others to avoid double-booking.

If operations need a "read-your-own-writes" guarantee or must prevent user conflicts over the same resource, strong consistency is essential. Data accuracy is crucial in these contexts.

When designing systems, it's important to weigh these needs carefully and safeguard sensitive financial and user data. Understanding encryption and its types is also vital.

Questions About Weak Consistency

Is Weak Consistency the Same as Eventual Consistency?

No, they are related but distinct. Weak consistency means there's no guarantee of reading the latest write, with no assurance of when or if data will sync across all nodes. Eventual consistency, a form of weak consistency, ensures that all replicas will eventually align if no new updates occur.

How Do Systems Handle Data Conflicts?

Conflicts arise due to the lack of immediate synchronization. A common method is Last Write Wins (LWW), where updates are timestamped, and the latest is accepted. Other methods, like vector clocks, offer more advanced conflict resolution, but LWW is popular in systems like Apache Cassandra.

Can a Single Application Use Multiple Consistency Models?

Yes, it's common in modern systems. For instance, an e-commerce platform might use:

• Strong Consistency for shopping carts and inventory.

• Eventual Consistency for user reviews and recommendations.

• Weak Consistency for analytics tracking.

This approach balances strict guarantees with performance and availability for different system parts.

How to ship better frontend experiences faster

The Datadog Frontend Developer Kit brings together essential resources to help you monitor performance, fix errors faster, and improve real user experience.

Inside the kit, you’ll find:

• Best practices guide on strengthening your frontend monitoring and testing strategies

• Step-by-step brief on catching and resolving frontend issues proactively

• Technical talk discussing how to better monitor and optimize frontend apps

If you’re building modern web apps, this kit helps you see what users see and fix what matters.

Link to the Kit: Frontend Developer Kit

When we use apps like Slack on our phones, we expect things to just work seamlessly. Like messages should load instantly, channels should update in real time and everything should feel smooth even when we’re switching networks, riding a metro, or dealing with poor connectivity. But behind the scenes, making this happen is far from simple. Let’s discover how Slack’s engineering team solves some very real mobile challenges without overcomplicating things and how they keep the app fast, reliable, and scalable in this blog today.

The Core Problem: Mobile is Unpredictable

On desktop, things are relatively easy. You’re usually on stable Wi-Fi. The app stays open for long sessions. Data can be fetched continuously without interruptions. But mobile is a completely different story.

Users:

• Move between networks (Wi-Fi → cellular → no signal)

• Open the app for short bursts

• Expect instant responses despite unstable connections

So the biggest challenge becomes: How do you fetch the most important data quickly even when the network is unreliable?

Smart Data Fetching

Slack tackles this by prioritizing API requests. Instead of treating all data equally, the system decides:

• What is critical right now?

• What can wait?

For example:

• Messages in the current channel are considered as high priority

• Background updates are considered as lower priority

So when the app gets even a small window of connectivity, it fetches the most important data first. This ensures that users can read messages and interact with conversations even if everything else hasn’t loaded yet. This one decision massively improves perceived performance.

Reducing Network Calls with Smarter Caching

Another key idea is avoiding unnecessary data fetching. Slack uses caching, but not in a naive way. Instead of blindly re-fetching data every time:

• The app stores previously fetched data

• It checks whether that data is still valid before making new requests

Thus it reduces Network usage, Load time and Dependency on connectivity. But this raises an important question: How does the app know if cached data is still up to date?

Versioning (This is the Secret Behind Fresh Data)

Slack solves this using something called object versioning. Here’s the idea in simple terms: Every piece of data (like a message or channel) has a version.

When the app has cached data:

1. It asks the server: “Is my version still the latest?”

2. If yes → no need to download again

3. If no → fetch the updated version

This avoids Re-downloading identical data, Wasting bandwidth and Slowing down the app. The implementation isn’t trivial though.

Versioning logic:

• Differs based on the type of object

• Requires coordination between client and server

• Needs accurate version calculation

But once done right, it leads to significant reduction in data usage

Fixing the Bigger Problem first i.e. Technical Debt

Over time, Slack’s mobile codebase started facing issues:

• Inconsistent feature development

• Slower delivery

• Difficulty meeting product deadlines

• Growing technical debt

This is something almost every large system faces eventually. So instead of patching things slowly, the team took a bold step: They revamped the mobile architecture.

This included:

• Major codebase changes

• Cleaning up legacy code

• Introducing a new feature architecture (especially on iOS)

This wasn’t easy at all, but this resulted into faster development, better consistency across features and improved ability to scale.

This shows us that sometimes, the only way forward is to rebuild the foundation properly.

Risk Management (Thinking what if Things Break?)

Making changes to a mobile app especially something like networking is risky. One wrong change can break critical functionality, impact millions of users and cause cascading failures. So Slack uses feature flags (change toggling).

What are feature flags?

They allow engineers to:

• Turn features ON/OFF remotely

• Roll out changes gradually

• Quickly roll back if something goes wrong

So instead of pushing risky changes blindly, they test in controlled environments, monitor real-world impact and disable instantly if needed. Still, no system is perfect.

So they also:

• Push hotfixes when urgent issues arise

• Perform deep analysis after failures

• Improve testing processes

This constant feedback loop helps them improve over time.

Scaling Challenges (It’s Not Just Users, It’s Complexity)

Slack doesn’t just deal with millions of users. It also deals with:

• Workspaces having 80,000 custom emojis

• Users in 20,000 channels

• Large teams constantly modifying the app

On top of that, their mobile engineering team itself has grown to 120+ engineers. This introduces a different kind of scaling problem: How do you manage a massive codebase with so many developers working on it?

The answer is to keep the architecture clean, build reusable components and continuously refactor because scaling isn’t just about handling traffic, it’s also about handling people and code complexity.

Building for Offline Mode

One of the most interesting problems Slack solved is: How do you make features work even without internet?

A feature team wanted users to:

• Perform actions offline

• Save changes

• Sync them later

Instead of solving this problem again and again for every feature, the infrastructure team built a shared component for offline data handling.

What does this component do?

It stores user actions when offline, tracks what needs to be synced and applies changes once connectivity returns.

So, now any feature can:

• Plug into this system

• Define what actions to store

• Let the system handle syncing

This leads to much better user experience, less duplicated work and cleaner code across teams.

Why This Matters

If you step back, you’ll notice a pattern in all these decisions:

1. Prioritize what matters: Not all data is equal so fetch the important stuff first.

2. Avoid unnecessary work: Don’t refetch what hasn’t changed.

3. Build reusable systems: Solve problems once, not repeatedly.

4. Expect failures: And design systems that can recover quickly.

5. Continuously improve: Refactor, learn, and adapt.

What makes Slack’s mobile app feel smooth isn’t just good UI. It’s the result of:

• Smart networking strategies

• Efficient data handling

• Thoughtful architecture decisions

• Continuous iteration

The real takeaway here is simple, Building great mobile apps isn’t about handling the best-case scenario, it’s about designing for the worst conditions and still delivering a great experience. And Slack does exactly that.

By now, you must have had a clear idea of, How Slack makes its Mobile App Feel Seamless (Even on Bad Internet)? In a nutshell, Slack makes its mobile app fast and reliable by prioritizing critical data, using smart caching with versioning, and handling offline + poor network scenarios efficiently. It also improves scalability and development speed through better architecture, reusable components, and safe feature rollouts.

Congratulations! You've just advanced another step in your tech journey. Keep progressing!

15 Kubernetes Metrics Every DevOps Team Should Track

Enhance Your Kubernetes Strategy with These Essential Metrics

Download our comprehensive eBook on optimizing Kubernetes performance. This guide delves into crucial cluster state, resource, and control plane metrics, highlighting 15 of the most essential metrics your DevOps team should be tracking. Learn how to gain complete visibility into your containerized environments and optimize Kubernetes performance with Datadog.

Microsoft says NO to Claude! - Microsoft is reportedly replacing Claude Code internally with GitHub Copilot CLI as it doubles down on its own AI developer ecosystem. The shift signals rising competition between AI coding assistants and a stronger push toward tightly integrated, enterprise-controlled tooling. It also reflects how AI-powered developer workflows are becoming a strategic battleground for major tech companies.

Hackers Crash Carnival - Carnival has confirmed a massive data breach affecting nearly 6 million people after attackers used social engineering to compromise an employee account. The breach reportedly exposed sensitive customer information, including personal identification data, raising serious identity theft concerns. The incident also highlights how human-targeted attacks remain one of the weakest links in enterprise cybersecurity defenses. Read more.

Everything you need to know about Claude Opus 4.8

Anthropic has officially launched Claude Opus 4.8, the latest version of its flagship AI model. While the update may look incremental on paper, it brings several meaningful improvements for developers, teams, and businesses using AI for coding and large-scale workflows.

The company says Opus 4.8 is more capable across coding, reasoning, and agentic tasks while remaining available at the same price as Opus 4.7. Alongside the model upgrade, Anthropic also introduced new workflow tools, effort controls, and faster performance options designed to make Claude more practical for real-world engineering work.

Better at Coding and Collaboration

One of the biggest focuses of Opus 4.8 is reliability. Anthropic says the model is now better at handling complex coding tasks and collaborating with developers over long sessions.

Early testers reported that Opus 4.8 makes fewer unsupported claims and is more likely to acknowledge uncertainty when it is unsure about something. That may sound small, but it addresses one of the biggest frustrations developers have with AI coding assistants: confident mistakes.

According to Anthropic’s evaluations, Opus 4.8 is around four times less likely than its predecessor to ignore flaws in generated code. This improvement could make the model more trustworthy for enterprise workflows where accuracy matters far more than speed alone.

Dynamic Workflows Bring Bigger Automation

Anthropic also introduced a new research preview feature called Dynamic Workflows for Claude Code.

This allows Claude to plan large tasks and run hundreds of parallel subagents in a single session. The system can verify outputs before returning results, making it useful for massive engineering projects such as code migrations across huge codebases.

The feature is currently available for Enterprise, Team, and Max plans, and it pushes Claude closer to becoming a fully autonomous engineering assistant instead of just a chatbot that writes snippets of code.

New Effort Controls for Users

Another major addition is effort control inside claude.ai.

Users can now decide how much “thinking effort” Claude should put into a task. Lower effort settings generate faster responses while using fewer rate limits. Higher effort settings allow the model to spend more tokens and reasoning time for better-quality outputs.

Anthropic says Opus 4.8 defaults to high effort mode because it provides the best balance between speed and quality. However, users working on difficult coding problems or long-running workflows can switch to “extra” or “max” effort modes for deeper reasoning.

Faster and Cheaper Performance

Anthropic also introduced a fast mode for Opus 4.8 that runs at 2.5× the speed while being three times cheaper than previous fast modes.

For developers and businesses managing high API usage, this could significantly reduce operational costs without sacrificing too much quality.

Pricing for standard usage remains unchanged at $5 per million input tokens and $25 per million output tokens.

Anthropic says Opus 4.8 is only a step toward even more advanced systems. The company is already working on “Mythos-class” models under Project Glasswing, which are reportedly powerful enough to require stronger cybersecurity safeguards before public release.

For now, Opus 4.8 represents a practical upgrade focused less on flashy benchmarks and more on making AI systems more dependable for real software engineering work.

IOS 27 leak! - New iOS 27 leaks suggest Apple is planning a major Siri redesign with smarter AI features and a fresh visual interface. The update could also introduce a redesigned Camera app, deeper AI integration, and improved customization across the iPhone experience. If true, iOS 27 may become Apple’s biggest software refresh since the introduction of Apple Intelligence. Read more.

Your Private Images Might Not Be Private: A critical Gitea vulnerability reportedly exposed private container images from over 30,000 deployments for nearly four years without requiring authentication. The flaw allowed attackers to access sensitive code, API keys, and infrastructure data simply by pulling “private” images from affected servers. Security researchers are urging all Gitea and Forgejo users to immediately upgrade to patched versions before the issue is actively exploited.

Buzz of the Week:

Cognitive Load Budget

Cognitive Load Budget is the idea that every developer has a limited amount of mental energy they can spend while coding, debugging, reviewing PRs, or switching between tools. Modern engineering problems are no longer just about compute limits, they’re about human attention limits. Teams with too many dashboards, alerts, frameworks, meetings, and AI tools often slow down because developers spend more time context-switching than building. Good engineering teams actively reduce cognitive load by simplifying workflows, documentation, architecture, and tooling. This concept is becoming increasingly important in the AI era because developers are now managing both human complexity and AI-generated complexity at the same time.

Things that launched. Things that went viral. Things you'll pretend to try.

Hurl

Hurl is Run and test HTTP requests using plain text files.
Feels like curl mixed with integration testing.

Vegeta

Vegeta is a flexible HTTP load-testing tool written in Go.

Grit

Grit is an AI-powered code migration and transformation engine.

Build Braincells, Not Just Features

This weekend’s read: Past present and future of Product Management.

This week’s watch: I tested what Vape does to your body.

Meanwhile…

In today's digital landscape, reliability is essential for user trust and business success. As systems become more complex, traditional operations models often fall short, resulting in downtime and user dissatisfaction. Site Reliability Engineering (SRE), developed at Google, addresses these issues by applying a software engineering approach to infrastructure and operations, leading to scalable and reliable software systems.

This article delves into essential site reliability engineering best practices, offering practical insights and examples from companies like Netflix and Spotify. It covers topics such as setting data-driven Service Level Objectives (SLOs), reducing operational burdens, promoting a culture of learning, and adopting secure deployment strategies.

Building resilient systems requires integrating security from the start, as outlined in Secure by Design cybersecurity practices. This guide offers the knowledge needed to develop, manage, and scale systems capable of handling production challenges.

1. Service Level Objectives (SLOs) and Error Budgets

One of the foundational site reliability engineering best practices is the establishment of Service Level Objectives (SLOs) and their corresponding error budgets. Instead of aiming for an unrealistic 100% reliability, SLOs define a precise, measurable target for a service's performance, such as 99.9% uptime over 30 days. This practice shifts the conversation from subjective feelings about stability to an objective, data-driven framework.

The error budget is the mathematical inverse of the SLO (100% - SLO). For a 99.9% SLO, the error budget is 0.1%, which translates to approximately 43 minutes of acceptable downtime or degraded performance per month. This budget becomes a shared currency between development and operations teams. As long as the service operates within its error budget, development teams have the autonomy to release new features and take calculated risks. If the budget is exhausted due to incidents or performance degradation, the focus automatically shifts to reliability improvements, halting non-essential feature releases until the service is stable.

Actionable Tips for Implementation

To adopt SLOs and error budgets effectively, follow these steps:

• Focus on Users: Start with user-centric metrics, like latency for key API endpoints. For example, an e-commerce site's SLO might be "99.9% of 'add to cart' API calls should succeed in under 500ms." Avoid internal metrics that don't impact the user experience directly.

• Start Conservatively: Initially set achievable SLOs (e.g., 99.5%) and tighten them as your processes improve to avoid discouragement from unachievable goals.

• Automate Monitoring: Use tools like Prometheus, Grafana, and SLO management platforms such as Nobl9 or Datadog for real-time monitoring and alerting of SLO compliance and error budgets.

2. Toil Reduction and Automation

A key aspect of site reliability engineering involves reducing toil through automation. Toil refers to manual, repetitive tasks that can be automated and grow with service expansion, such as restarting services or provisioning servers. By eliminating these tasks, SRE teams can focus on enhancing system reliability and scalability.

The SRE approach, widely adopted by Google, advises engineers to limit toil to no more than 50% of their time, dedicating the rest to tasks like automation development and system re-architecture. This encourages teams to prioritize long-term engineering solutions over reactive work.

Actionable Tips for Implementation

To effectively decrease toil and automate processes, consider these steps:

• Track Toil Precisely: Have team members log time spent on toil using systems like Jira to create a data-driven baseline. This helps identify time-consuming tasks and justify automation investments.

• Apply the 'Rule of Three': Automate any task after doing it three times. Learn during the first attempt, note inefficiencies the second time, and create a robust automation tool by the third.

• Develop Self-Service Tools: Build a self-service portal with tools like Backstage.io to enable developers to provision resources and run tests independently, reducing interrupt-driven toil.

3. Blameless Post-Mortems and Learning from Failures

A key practice in site reliability engineering is conducting blameless post-mortems. This involves reviewing incidents to understand systemic failures and process gaps, not to assign individual blame. The approach acknowledges that complex systems will fail and views human error as a sign of deeper issues like inadequate training or poor tools. By fostering psychological safety, this method encourages transparency and learning from mistakes. It turns failures into learning opportunities, aiming to identify contributing factors and vulnerabilities without assigning blame, leading to effective and lasting improvements.

Real-World Implementation and Benefits

The blameless post-mortem culture, supported by figures like John Allspaw at Etsy, is key to advanced engineering organizations. Etsy's public post-mortems enhance customer trust and tech community knowledge. Google's SRE teams perform detailed post-mortems for major incidents with executive attention to foster systemic improvements. GitLab offers transparency by making incident reports and post-mortems publicly accessible. This feedback loop ensures failures inform system design and operations, enhancing service resilience. Learn more about incident handling here.

Actionable Tips for Implementation

To embed a blameless culture:

• Act Quickly: Conduct post-mortems within 48-72 hours using a template in tools like Confluence or Google Docs, focusing on timeline, impact, and action items.

• Use the 'Five Whys': Ask "Why?" repeatedly to identify root causes beyond immediate issues.

• Share Learnings: Distribute post-mortem insights via email, Slack, or sessions, and maintain a central, searchable repository.

4. Monitoring, Observability, and Alerting Excellence

In site reliability engineering, establishing systems for monitoring, observability, and alerting is essential. Monitoring involves analyzing data to assess system health, while observability allows engineers to deduce internal states from external outputs without new code. Intelligent alerting targets user-impacting symptoms, helping teams quickly resolve issues.

This approach surpasses basic health checks like CPU usage, focusing on the "three pillars of observability": metrics, logs, and traces. By integrating these, SREs can pinpoint specific issues, such as increased API latency in certain regions, which is vital for managing complex microservices and distributed architectures.

Real-World Implementation and Benefits

Leading tech companies demonstrate the power of this practice. Netflix relies on its Atlas telemetry system and distributed tracing to manage thousands of microservices, ensuring a smooth streaming experience. Similarly, Uber developed its own M3 platform to handle billions of metrics, providing deep insights into its vast, real-time operations. The primary benefit is a drastic reduction in Mean Time To Resolution (MTTR). Teams can pinpoint root causes faster, understand the blast radius of an incident, and validate fixes with confidence, all while minimizing noise from non-actionable alerts.

Actionable Tips for Implementation

To achieve excellence, focus on these steps:

• Monitor Key Signals: Focus on latency, traffic, errors, and saturation to assess service health from the user's perspective.

• Alert on Symptoms: Base alerts on user-facing issues, ensuring they are actionable and protect user experience.

• Link Runbooks to Alerts: Connect actionable alerts with runbooks detailing investigation and remediation steps to improve response times. For data visualization, explore Grafana.

5. Capacity Planning and Performance Engineering

In site reliability engineering, proactive system resource management through capacity planning and performance engineering is essential. This involves predicting future demand to ensure infrastructure can handle loads and optimizing system architecture for efficiency. Instead of reacting to traffic spikes, SRE teams prevent resource exhaustion and maintain a fast user experience as the user base grows. System capacity is viewed as a dynamic variable aligned with business goals. Performance engineering ensures software efficiency, reducing hardware needs, lowering costs, and minimizing performance risks under stress.

Actionable Tips for Implementation

To integrate capacity planning and performance engineering effectively, follow these steps:

• Maintain Headroom: Keep a 20-30% capacity buffer above your expected peak load to manage traffic spikes and minor failures.

• Perform Realistic Load Tests: Use tools like k6 or JMeter to create tests that mimic actual user behavior and transaction mixes. Test at peak scale plus extra to detect scaling issues.

• Forecast Using Data: Combine historical usage, business growth projections, and marketing calendars for demand forecasting. Automate with trend analysis for better accuracy. Learn about load balancing and capacity planning.

• Profile Applications Regularly: Use tools like pprof or YourKit in your CI/CD pipeline to identify and optimize resource-heavy functions, reducing capacity needs.

• Conduct Quarterly Reviews: Hold reviews with stakeholders to ensure alignment and adjust forecasts and budgets as needed.

6. Incident Management and On-Call Practices

A key aspect of site reliability engineering is a structured approach to incident management and on-call duties. This practice formalizes detecting, responding to, resolving, and learning from service interruptions, shifting from chaotic responses to a predictable and sustainable system that prioritizes quick service recovery and engineer well-being.

Effective incident management sets clear roles and communication channels to reduce Mean Time To Resolution (MTTR). Along with thoughtful on-call practices, it ensures fair distribution of the 24/7 reliability burden, preventing burnout. This focus on both system and engineer health is essential for long-term operational success, as tired engineers are more likely to make errors.

Actionable Tips for Implementation

To establish a strong incident management framework, follow these steps:

• Define Roles: Assign clear roles such as Incident Commander, Communications Lead, and Subject Matter Experts. The Incident Commander declares their role and delegates tasks to avoid confusion.

• Use Dedicated Channels: Utilize tools like PagerDuty or Opsgenie to automatically create specific Slack or Teams channels for incidents, ensuring centralized and efficient communication.

• On-Call Rotations: Limit on-call shifts to one week to prevent fatigue and adopt a "follow-the-sun" model for global teams to handle responsibilities across time zones. Ensure a clear escalation policy is in place.

• Maintain Runbooks: Document incident patterns and resolution steps in runbooks, including diagnostic queries and mitigation commands.

Site Reliability Engineering Practices Comparison

When you open LinkedIn and scroll your feed, send a message, or load notifications, a lot more is happening behind the scenes than you might expect. Each action triggers multiple services talking to each other. Now imagine this at LinkedIn scale with tens of thousands of microservices, running across global data centers, handling billions of requests daily. The question is, “How do all these services find and talk to each other reliably?” The answer lies in something called service discovery and LinkedIn recently rebuilt this system from scratch. Let’s discover what it is and how LinkedIn rebuilt it in this blog today.

What is Service Discovery

You can think of service discovery like a directory. If one service wants to talk to another, it needs to know where that service is running, its IP address and port.

But in modern systems services keep scaling up/down and instances keep changing. So hardcoding locations doesn’t work. Instead:

• Services register themselves in a central system

• Other services query this system to find them

This central system is called the control plane.

Old System: Zookeeper-Based Architecture

For years, LinkedIn used Apache ZooKeeper as its service discovery control plane. Here’s how it worked:

• Services register their endpoints in ZooKeeper

• Clients read this data directly

• ZooKeeper also performs health checks

This sounds simple and it worked well initially. But as LinkedIn scaled, cracks started to appear.

Problems with the Old System

1. Scalability Issues

ZooKeeper handled all reads, all writes and all health checks in one place and this created a bottleneck.

During large deployments service data changed frequently and thousands of clients started reading updates which caused read storms (massive spikes in read requests). Since ZooKeeper enforces strong consistency, everything goes through a single queue. So when reads increase:

• writes get delayed

• health checks fail

• sessions drop

And as a result service instances get removed, capacity drops and systems become unavailable.

2. Compatibility Issues

LinkedIn used a custom format (D2), which didn’t work well with modern systems like gRPC or Envoy and it was heavily Java-centric.

This made:

• multi-language support difficult

• onboarding new systems harder

3. Extensibility Problems

The architecture lacked an intermediate layer. So it was hard to:

• add centralized load balancing

• integrate with Kubernetes

• evolve the system

New System: Next-Gen Service Discovery

To solve these issues, LinkedIn built a completely new system. Instead of one monolithic control plane, they introduced a decoupled architecture.

Key Components

1. Kafka (Write Path): Services now send updates via events to Apache Kafka. These include service registrations and heartbeats.

2. Service Discovery Observer (Read Path):
   A new component called Observer:

   • consumes events from Kafka

   • stores data in memory

   • serves clients

3. gRPC Streams: Clients:

   • open persistent connections using gRPC

   • receive updates in real time

The key shift was moving from a pull-based model (clients fetching data) to a push-based model where the Observer streams updates to clients in real time.

Why this Architecture Works Better

1. Scalability

Observer is horizontally scalable and highly concurrent. Thus one Observer can:

• handle 40K client connections

• process 10K updates/sec

2. Availability Over Strict Consistency

The shift was from prioritizing strong consistency in ZooKeeper to favoring availability with eventual consistency in the new system.

This means:

• small inconsistencies are okay temporarily

• system stays responsive

3. Fault Tolerance

Even if Kafka is slow or down, observer serves cached data. So services still function with no downtime.

Dual Mode

Replacing a core system like service discovery is risky. So LinkedIn didn’t switch everything at once.

They used Dual Mode (Dual Read + Dual Write)

Dual Read

• clients read from both old (ZK) and new system

• compare results in background

Dual Write

• services register in both systems

This approach is powerful because it verifies correctness, catches mismatches early, and prevents issues from impacting production.

Observability: The Backbone of Migration

To ensure everything works, LinkedIn added deep monitoring.

They tracked:

• connection health

• latency

• data consistency

• system resource usage

Metrics

End-to-end propagation latency improved from P50 < 10s and P99 < 30s in the old system to P50 < 1s and P99 < 5s in the new system

New system:

• P50 < 1s

• P99 < 5s

Earlier system:

• P50 < 10s

• P99 < 30s

And that’s some crazy improvement!

Migration Dependencies

One of the trickiest challenges was migration dependency because clients needed to move first, but write migration depended on read migration, creating a dependency loop.

Solution

LinkedIn:

• analyzed dependency graphs

• tracked which services depend on others

• migrated carefully in phases

They also built tools to detect regressions and monitored which apps still relied on old system.

Takeaways

1. Decouple read and write paths
   Separating Kafka (write path) and Observer (read path) removes bottlenecks and allows each side to scale independently.

2. Prefer push over pull at scale
   Instead of clients repeatedly polling for updates, streaming changes via persistent connections reduces load and improves latency.

3. Prioritize availability over strict consistency (when appropriate)
   In systems like service discovery, it’s more important to stay responsive than perfectly consistent at every moment.

4. Design for multi-language ecosystems
   Using standards like gRPC and xDS makes the system compatible across different languages and frameworks.

5. Migrate critical systems gradually
   Techniques like dual read and dual write help validate the new system safely without risking production stability.

By now, you must have had a clear idea of, How LinkedIn Rebuilt Service Discovery to Scale to Millions of Services? In a nutshell, LinkedIn replaced its ZooKeeper-based service discovery with a Kafka + Observer system to handle massive scale more reliably. By shifting to a push-based, scalable architecture, it improved latency, availability, and multi-language support.

Congratulations! You've just advanced another step in your tech journey. Keep progressing!

GitHub’s worst leak ever! - GitHub has confirmed a major breach affecting nearly 3,800 internal repositories after a poisoned VS Code extension compromised an employee device. The attack, linked to the TeamPCP hacking group, highlights the growing danger of software supply-chain attacks targeting developer tools and open-source ecosystems. Read full story.

Meta’s new Reddit - Meta has quietly launched a new app called Forum, a standalone experience built around Facebook Groups that feels heavily inspired by Reddit. The app focuses on community discussions, AI-powered answers, and interest-based feeds while separating group conversations from the clutter of the main Facebook app. It also includes an “Ask” AI feature that pulls insights from multiple groups, showing Meta’s growing push toward AI-assisted social discovery.

Google I/O 2026 Was Not About AI Features. It Was About AI Taking Over the Interface

Google I/O 2026 made one thing very clear. Gemini is no longer just an AI chatbot sitting inside an app. Google is turning it into the operating layer across Search, Android, Workspace, YouTube, and even wearable devices.

This year’s announcements were not focused on one breakthrough model. Instead, Google introduced an entire ecosystem where AI quietly works in the background, takes actions for users, and becomes deeply integrated into everyday digital workflows.

Gemini Is Becoming an Active Agent

The biggest shift came with Gemini Spark, which Google described as a “personal agent” capable of performing tasks on behalf of users. Unlike traditional assistants that only respond to prompts, Spark is designed to actively manage parts of your digital life.

It can interact with Gmail, Docs, Calendar, and Tasks while also connecting to third-party tools through MCP integrations later this year. Google is clearly moving toward a future where AI does more than answer questions. It schedules, organizes, monitors, and executes tasks with minimal input.

The new Daily Brief feature reinforces this direction. By scanning emails, calendars, and task lists, Gemini creates a personalized summary of your day and even suggests next steps automatically.

Search Is Turning Into a Continuous Experience

Google Search also received one of its biggest transformations in years. AI Mode is now powered by Gemini 3.5 Flash, which promises faster reasoning and better multimodal understanding.

The traditional search bar is evolving into a conversational interface that grows dynamically as users type longer questions. More importantly, Google introduced information agents that continuously monitor topics across the web.

Instead of searching repeatedly for updates, users can now let AI track news, finance, shopping trends, or sports in the background and surface relevant changes automatically.

This signals a major shift from reactive search to persistent AI-driven discovery.

Gemini Models Are Getting Faster and More Creative

Google also unveiled Gemini 3.5 Flash, which combines strong coding and reasoning performance with faster response speeds. According to Google, it delivers output four times faster than competing frontier models in some scenarios.

Meanwhile, Gemini Omni expands AI generation beyond text. The new model accepts image, audio, video, and text inputs while generating editable video outputs grounded in real-world knowledge.

This technology is already being integrated into YouTube Shorts, Google Flow, and creative tools inside the Gemini ecosystem.

Android XR and AI Wearables Are Real

Another standout announcement was Android XR eyewear. Google confirmed that its first AI-powered audio glasses will launch later this year in partnership with Samsung, Qualcomm, Gentle Monster, and Warby Parker.

These glasses are positioned as “intelligent eyewear” and will even support iPhones. Combined with Android Halo, which shows live agent activity at the top of the screen, Google is preparing users for ambient AI experiences that exist beyond phones and laptops.

Google I/O 2026 showed that the company is no longer treating AI as a separate product category. Gemini is becoming the connective layer across the entire Google ecosystem.

You cannot search “disregard” in Google now - Google Search users discovered a strange AI bug where typing words like “disregard,” “ignore,” or “skip” caused Search to respond like a chatbot instead of showing normal results. The issue appears tied to Google’s AI Overviews system, which seems to misinterpret certain commands as prompt instructions. Read more.

NotebookLM Rival Is Here ft. Spotify: Spotify is expanding deeper into AI-generated audio with a new desktop app that lets users create personalized podcasts using prompts, documents, calendars, and web content. The tool, called Studio by Spotify Labs, works similarly to Google’s NotebookLM and generates private AI podcasts synced across your Spotify account. Spotify says the app is still an early research preview and may occasionally produce inaccurate results.

Buzz of the Week:

Capability Sandboxing

Capability Sandboxing is a security architecture approach where applications, AI agents, or processes receive extremely limited and specific permissions instead of broad system-wide access. Rather than giving software full control, it only gets small “capabilities” such as reading one file, accessing one API endpoint, or using a single device feature. This model reduces the damage caused by compromised apps, prompt injection attacks, or malicious plugins. It is becoming increasingly important in AI agents, browser engines, WebAssembly runtimes, containers, and cloud-native systems where autonomous tools can execute actions on behalf of users.

Things that launched. Things that went viral. Things you'll pretend to try.

Rnr

Rnr is a fast Rust-powered batch rename utility with regex support.

Deadnix

Deadnix detects unused variables and dead code in Nix files.

Tparse

Tparse converts noisy test outputs into clean summaries and analytics dashboards.

Build Braincells, Not Just Features

This weekend’s read: OpenAI claims it solved an 80-year-old math problem — for real this time.

This week’s watch: I Investigated The Country Where it's Illegal To Be Fat.

Meanwhile…

A cache invalidation strategy is a set of rules that decides when old, stored data gets thrown out. Think of it as the bouncer for your cache, ensuring only fresh, relevant information is served to your users. Getting this right is critical for keeping your application’s data consistent between the cache and your main database, which directly impacts performance and reliability.

Why Cache Invalidation is Tricky

Imagine your cache is an assistant that memorizes answers to common questions to reply instantly, saving a slow trip to the database. A user asks for their profile info? The cache provides it immediately.

But what happens when that user updates their profile picture? The cache, unaware of the change, will keep serving the old photo. This is called serving stale data, and it can cause anything from user confusion to showing the wrong price for a product. The process of telling the cache to forget the old info and grab the new version is the essence of cache invalidation.

The core challenge is balancing speed and accuracy. Do you invalidate data the instant it changes? This guarantees accuracy but adds complexity. Or do you let it expire after a set time? This is simpler but means knowingly serving stale data for a period.

This balancing act is why computer scientist Phil Karlton famously said, "There are only two hard things in Computer Science: cache invalidation and naming things." Mastering this skill can seriously increase developer productivity by preventing bugs and simplifying system maintenance.

Exploring Common Cache Invalidation Strategies

Picking the right cache invalidation strategy comes down to your application's needs. Each one offers a different trade-off between data freshness, performance, and complexity.

Time-To-Live (TTL): The "Set It and Forget It" Approach

TTL is the simplest strategy. You put a "best before" date on data when you add it to the cache. Once the timer expires, the data is removed. The next request must go back to the database for a fresh copy, which then gets a new timer.

• Practical Example: A news website can cache an article for 10 minutes. This is simple and greatly reduces database load, as the content doesn't change frequently.

• Actionable Insight: Use TTL for data that is not time-sensitive. A long TTL (hours or days) works for static assets, while a shorter TTL (minutes) is better for content that updates periodically.

The main drawback is that you knowingly serve stale data until the TTL expires.

Write-Through Caching: For Guaranteed Consistency

With this approach, every write operation goes to both the cache and the primary database simultaneously. The operation isn't complete until both writes are confirmed.

• Practical Example: An e-commerce inventory system. When a product is sold, the count must be updated in both the cache and database immediately to prevent overselling.

• Actionable Insight: Choose write-through when data consistency is non-negotiable. This eliminates stale data at the cost of higher write latency, as you have to wait for two systems to confirm the write.

Write-Back Caching (Write-Behind): For High-Speed Writes

Write-back caching prioritizes write performance. Data is written only to the fast in-memory cache, and the write to the slower database happens later, either after a delay or in batches.

• Practical Example: A system logging user activity. Writes are frequent and speed is critical. The system can buffer logs in the cache and write them to the database every few seconds.

• Actionable Insight: Use write-back for write-heavy applications where immediate data persistence isn't required. Be aware of the risk: if the cache server crashes before data is written to the database, those updates are lost.

Event-Driven Invalidation: For Real-Time Updates

Also known as manual invalidation, this is the most precise strategy. When data changes in the database, your application logic sends a specific command to purge that exact entry from the cache.

• Practical Example: A live sports score feed. When a team scores, a "score_updated" event is published. A listener service catches this event and immediately invalidates the cached score, ensuring all users see the new score instantly.

• Actionable Insight: Implement this using a publish/subscribe (pub/sub) model with tools like Redis Pub/Sub or Kafka. This decouples your services and provides precise control but adds significant architectural complexity.

How to Choose the Right Strategy

To pick the best strategy, answer three key questions about your data.

1. How Often Does Your Data Change?

• Low Frequency (e.g., Blog Posts): Time-To-Live (TTL) is ideal. A long TTL (e.g., 24 hours) reduces database load with minimal risk of stale content.

• High Frequency (e.g., Social Media Feeds): An event-driven approach is better. It provides precise control to invalidate content the moment it updates.

2. How Critical Is Data Freshness?

• Low Criticality (e.g., User Profile Avatar): TTL is acceptable. A user seeing an old avatar for a few minutes is a minor issue.

• High Criticality (e.g., Bank Account Balance): Write-through is necessary. It guarantees the cache and database are always in sync, which is essential for financial data.

3. Is Your Application Read-Heavy or Write-Heavy?

• Read-Heavy (e.g., Content Websites): Strategies that speed up reads, like TTL, are effective. The same article is read thousands of times for every one update.

• Write-Heavy (e.g., Analytics Platforms): Write-back shines here. It makes writes feel instantaneous by buffering them in the cache, improving performance for write-intensive operations.

Understanding these trade-offs is fundamental to building high-performance and scalability in web applications.

Common Questions About Cache Invalidation

What's the difference between invalidation and eviction?

Invalidation is a deliberate action to remove data known to be stale. Eviction is an automatic process where the cache removes data (often the least recently used) to make space when it's full. Invalidation is about accuracy; eviction is about capacity.

What is a cache stampede and how do I prevent it?

A cache stampede happens when a popular cached item expires, causing a flood of concurrent requests to hit your database at once.

• Prevention Tactic: Use a "stale-while-revalidate" approach. When an item expires, serve the stale data to users while a single background process fetches the fresh version. This shields the database from the stampede.

Can I combine different strategies?

Yes, and you should for complex applications. A common pattern is to use TTL as a default and layer event-driven invalidation for critical updates. For example, cache a user's profile with a 24-hour TTL but use an event to purge it immediately if they change their password. This hybrid model balances performance and consistency, a challenge seen in systems like those discussed in how Uber handles 40 million reads per second using an integrated cache.

Imagine running the same query today and then running it again a few months later. Same SQL, same data but this time, it runs 20% faster. You didn’t optimize it, you didn’t rewrite anything but it just got faster. This is exactly what Snowflake has been achieving and they track this improvement using something called the Snowflake Performance Index (SPI). Let’s discover how Snowflake did it in this blog today.

If you’ve read our earlier blog on how Snowflake improved overall performance by 27%, this is a continuation of that story. That blog explored what kinds of optimizations Snowflake made. This one focuses on how those improvements translate into real-world query performance over time, measured using the Snowflake Performance Index (SPI).

What is the Snowflake Performance Index (SPI)?

The Snowflake Performance Index (SPI) is a metric designed to measure, “How much faster Snowflake is getting over time for real customer workloads“.

Now, this is important because most companies measure performance using:

• synthetic benchmarks

• controlled test environments

But Snowflake does something different. Instead of testing in ideal conditions, SPI tracks real queries running on real production workloads due to which this approach gives the best results because real-world workloads are unpredictable, complex and constantly changing. So if performance improves there, it actually means something.

SPI focuses specifically on:

• stable workloads (same type of queries over time)

• similar data volume

• consistent query patterns

This allows Snowflake to compare performance fairly across time.

What do the Numbers Say?

Based on Snowflake’s internal data:

• Query duration improved 11% in one year

• Up to 20% improvement since SPI tracking began

and this isn’t a one-time spike. It reflects continuous, steady improvement in how queries execute over time

So, how Snowflake did it?

One of the most important ideas here is Snowflake’s approach to performance. They don’t rely on big upgrades, migrations and manual tuning. Instead, they focus on continuously improving the core database engine. These improvements are shipped through weekly releases

What this means for users is that every week small improvements are deployed that are automatically applied to your workloads. So over time your queries just become faster. No action required from users end.

What Actually Improved Under the Hood?

1.Faster and Smarter Query Compilation

Before any query runs, Snowflake first compiles it. This step decides how the query will execute and how data will be processed. Snowflake improved this phase in multiple ways.

What changed?

• It avoids unnecessary optimization steps when they’re not needed

• It evaluates SQL expressions more efficiently

• It improves compilation for queries using materialized views

If compilation is faster, queries start executing sooner and overall latency drops. It’s like reducing the “thinking time” before doing the actual work.

2. Improved Materialized View Performance

Materialized views are precomputed results that help speed up queries. But maintaining them can be expensive. Snowflake improved how efficiently they are updated and how queries interact with them

Result

• faster query performance

• reduced compute overhead

So you get the benefit of precomputed data without paying a high maintenance cost.

3. Better Performance on Non-Clustered Data

In an ideal world, data is perfectly organized but in reality most data is not clustered optimally. Earlier, queries on such data could be slower. Now, Snowflake improved how queries run on non-clustered tables.

This matters because most real-world datasets are messy and don’t follow perfect structure. Improving performance here means better performance for real use cases, not just ideal ones.

4. Smarter Search Optimization

Snowflake introduced improved search optimization techniques. Instead of scanning large chunks of data, the system can quickly narrow down relevant data

Result

• less data scanned

• faster query execution

This is essentially about doing less work to get the same result.

5. Faster Metadata Operations (SHOW Commands)

Snowflake also improved the performance of SHOW commands. These are used to:

• inspect tables

• check schemas

• understand system state

This matters because even though these are not heavy queries, they are used frequently and they impact developer workflows. Basically faster metadata operations improve overall user experience!

The Bigger Picture

None of these individual improvements alone results in a 20% performance gain. But when you combine them, they create a compounding effect.

That’s the key idea here:

This is exactly how Snowflake achieves consistent performance gains over time, not through one big change, but through continuous, incremental enhancements.

How Snowflake Measures this Fairly

Snowflake doesn’t just claim performance improvements, it measures them carefully. To ensure accuracy, it follows a structured approach:

• Identify stable workloads

• Ensure similar query patterns

• Keep data volume consistent

• Track query duration over time

This approach ensures:

• fair comparison under the same conditions

• actual performance improvements (not artificial gains)

• no misleading benchmarks

In other words, Snowflake isn’t just making things faster, it’s proving that real-world workloads are actually improving over time.

Takeaways

1. Continuous optimization beats big releases
   Instead of relying on occasional major upgrades, Snowflake delivers small, frequent improvements that compound into significant gains over time.

2. Optimize the engine, not the user
   Great systems improve internally so users don’t have to tune queries or change configurations to get better performance.

3. Measure real workloads, not benchmarks
   Synthetic benchmarks can be misleading. Tracking real customer workloads ensures improvements actually matter in production.

4. Eliminate unnecessary work
   Skipping redundant optimization steps and avoiding extra computation directly improves performance without adding more resources.

5. Small gains compound over time
   Individual improvements may seem minor, but together they create a noticeable and meaningful impact on overall system performance.

By now, you must have had a clear idea of, How Snowflake Reduced Query Time by 20%? In a nutshell, Snowflake improves query performance by continuously optimizing its core engine and measuring real-world impact using SPI. These incremental improvements reduce query duration by up to 20% without requiring any user changes.

Congratulations! You've just advanced another step in your tech journey. Keep progressing!

“Mini Shai-Hulund” attacks npm, PyPi again - A new “Mini Shai-Hulud” supply-chain worm is spreading through popular npm and PyPI packages, compromising developer tools linked to TanStack, Mistral AI, UiPath, and other major ecosystems. Instead of targeting end users directly, the malware steals GitHub, cloud, and CI/CD credentials by hijacking trusted publishing pipelines and developer environments. Read more.

Now your mouse can think! - DeepMind is rethinking how humans interact with AI by turning the mouse pointer into a contextual control system for embedded agents. Instead of writing long prompts, users can simply point at objects, tables, images, or text and give natural commands like “explain this” or “turn this into a chart.” The system combines gestures with voice input, making AI interaction feel less like prompting a chatbot and more like collaborating with an intelligent operating system.

Why Developers Still Don’t Fully Trust AI Coding Agents

AI tools can write code faster than ever, but speed alone is not enough. The real challenge is whether developers can actually trust what these agents are doing behind the scenes.

AI Agents Are Fast, But Confidence Is Fragile

Over the last few years, AI coding assistants have evolved from autocomplete tools into fully autonomous agents capable of generating features, fixing bugs, refactoring applications, and even opening pull requests. The productivity gains are undeniable. Developers are shipping faster, experimenting more, and automating tedious work that once consumed entire afternoons.

But there’s a growing problem beneath the excitement: trust.

Most developers who regularly use AI tools have experienced moments where the output felt… off. Maybe the agent edited files you never asked it to touch. Maybe it introduced logic that technically worked but made the codebase harder to maintain. Sometimes it confidently generated instructions that were simply wrong.

What makes these moments frustrating is not just the mistake itself, it’s the uncertainty that follows. Developers often spend more time auditing the AI’s work than actually writing code.

The Real Problem Isn’t Capability

AI companies often focus on benchmark scores and coding performance metrics. But developers rarely stick with tools simply because they are powerful. They stick with tools that are predictable.

A fast tool that behaves inconsistently creates anxiety. Developers want systems that behave reliably under pressure, especially in production environments where every change has consequences.

This is where many coding agents struggle today. They optimize heavily for task completion, not collaboration.

One of the biggest frustrations with AI agents is the lack of visibility into their decision-making process.

You might receive a large diff explaining what changed, but not why those decisions were made. Without understanding the reasoning behind the implementation, reviewing AI-generated code becomes mentally exhausting.

Consider a simple example.

An AI agent may generate a compressed one-liner that technically solves the problem. But human developers often prefer readable abstractions, descriptive helper functions, and maintainable naming conventions that future teammates can quickly understand.

The issue is not correctness. It’s maintainability.

Readable code is collaborative code, and many AI agents still optimize for efficiency over clarity.

Scope Creep Makes Reviews Harder

Another major trust issue is uncontrolled scope.

Developers frequently ask agents to make one targeted fix, only to discover changes scattered across unrelated files. Sometimes those edits are useful. Sometimes they even improve the project. But they still increase the burden of review.

This creates a dangerous pattern where engineers must constantly audit AI-generated work for hidden surprises.

In traditional software development, pull requests are easier to review when they remain tightly scoped. AI agents often break that expectation by behaving like overly enthusiastic contributors trying to “improve everything” at once.

The result is review fatigue. The industry loves the phrase “human-in-the-loop,” but in many tools that simply means approving the final output after the work is already done.

Real collaboration should happen earlier.

Developers need the ability to see the plan before execution, modify the approach mid-process, and guide the agent while it works, not just approve or reject the result afterward. Some newer systems are experimenting with workflows built around planning, context gathering, review stages, and controlled execution. That direction feels promising because it treats AI as a collaborator instead of an autonomous black box.

The Future of AI Development Depends on Trust

AI coding agents are not failing because they lack intelligence. They are failing because software engineering is fundamentally collaborative work.

Developers do not just want faster outputs. They want systems they can reason about, maintain, and confidently deploy.

The next generation of AI tools will not win by being the most autonomous. They will win by being the most trustworthy.

Anthropic moves into AWS - Anthropic is bringing its full Claude developer platform directly into AWS, giving companies access to native Claude APIs, agent tools, code execution, MCP connectors, and experimental features without leaving the AWS ecosystem. Instead of using a separate Anthropic account, developers can now manage authentication, billing, and security through existing AWS infrastructure and IAM controls.

Codex comes to mobile phone: OpenAI is bringing Codex to mobile devices through the ChatGPT app, letting developers monitor, guide, and approve AI coding tasks directly from their phones. Instead of coding on mobile, users can remotely manage long-running AI agents, review outputs, switch models, and keep development workflows moving from anywhere.

Buzz of the Week:

Deterministic Replay

Deterministic Replay is a debugging technique where a system records enough information about a program’s execution so the exact same behavior can be replayed later, instruction by instruction. Most engineers debug by reproducing bugs manually, but deterministic replay lets you “time travel” through crashes, race conditions, and distributed system failures that normally disappear once the system changes state. It is especially important in multithreaded systems, AI agents, trading platforms, game engines, and distributed infrastructure where bugs are often non-deterministic and nearly impossible to reproduce consistently. Tools using deterministic replay can capture inputs, thread scheduling, network events, and system calls so developers can inspect the exact moment something went wrong.

Things that launched. Things that went viral. Things you'll pretend to try.

Czkawka

Czkawka is a powerful duplicate finder for developers.
Helps clean massive workspaces, caches, and downloaded assets.

Kondo

Kondo cleans build artifacts and temporary dev files automatically.

jless

jless - is a terminal JSON viewer with navigation and syntax awareness.

Build Braincells, Not Just Features

This weekend’s read: AI updates from the past week.

This week’s watch: Aviloop: YouTube's Darkest Mystery

Meanwhile…

When you're building with a microservices architecture, service discovery is the magic that lets all your different services find and talk to each other automatically. Forget hardcoding IP addresses or network locations.

Think of it like a dynamic, self-updating phonebook for your entire application. In an environment where services are constantly spinning up, shutting down, or scaling out, this isn't just a nice-to-have it's absolutely essential.

Why Service Discovery Is Essential for Microservices

Imagine trying to call a friend whose phone number changes every few minutes. To have any hope of reaching them, you'd need a contact list that updates in real-time. That's the exact problem microservices face in the cloud.

Old-school methods, like static IP addresses and manual config files, just can't keep up when service instances are temporary by design. They come and go as needed.

This is where a service registry steps in to become the single source of truth. When a new service instance boots up, it registers its current location (IP address and port) with this central registry. When it’s time to shut down, it de-registers itself.

Now, any other service that needs to connect can just ask the registry for the latest, most current location. Communication just works, seamlessly.

The Issue with Static Configurations

In a large monolithic application, direct interaction between code parts is easy since they run in the same process. However, breaking it into distributed services turns communication into a network challenge.

Tracking network locations for numerous service instances manually is not only tedious but also prone to errors and unmanageable at scale. This complexity increases with modern practices like auto-scaling, where service instances fluctuate based on traffic. Without automation, constant downtime and operational issues are inevitable. A robust service discovery for microservices system addresses this by abstracting service locations.

Breaking Down the Core Service Discovery Patterns

In a microservices world, services are constantly spinning up and shutting down. So, how do they find each other in this ever-changing environment? The whole process boils down to two fundamental approaches: Client-Side Discovery and Server-Side Discovery.

Each pattern tackles the same essential question: "Where is the service I need to talk to right now?" Your choice here will shape your application's architecture, complexity, and even its performance. Let's dig into how they work.

The Client-Side Discovery Pattern

In the client-side pattern, each client service determines where to send requests. When calling another service like "Payment Service," the client queries a service registry for available instances and their IPs and ports. The client then uses a load-balancing algorithm to select an instance to connect directly.

The essence is that the client manages service selection and connection. This grants full control over load-balancing, beneficial for custom logic, and reduces complexity between the client and service, potentially lowering network latency.

However, this pattern ties the client to the service registry, requiring the discovery logic to be implemented across all languages and frameworks, which can be challenging in diverse environments.

The Server-Side Discovery Pattern

In server-side discovery, the client offloads discovery logic to a central point like a router or load balancer, instead of querying the service registry. This intermediary manages traffic by checking the service registry, selecting a service using a load-balancing algorithm, and forwarding requests. The client remains unaware of this process. For more on how DNS plays a role in routing, see our guide on DNS and its function.

This approach simplifies client code, reducing the need for discovery and load-balancing logic in each service, resulting in a cleaner system. However, it introduces an additional network hop, adding slight latency, and makes the router or load balancer a crucial component that must be highly reliable.

Service discovery is part of a larger system. To understand its role, it's useful to explore various microservices architecture patterns. Choosing between client-side and server-side discovery depends on priorities like control, simplicity, or performance.

A Practical Comparison of Popular Service Discovery Tools

Choosing the right service discovery tool for microservices is a significant architectural decision. It's essential to understand the trade-offs each tool makes between consistency, availability, and features. Major tools like Consul, Eureka, etcd, and Zookeeper differ mainly in their consistency models, relating to the CAP theorem. Tools are generally either CP (Consistency and Partition Tolerance) or AP (Availability and Partition Tolerance).

HashiCorp Consul

Consul is more than a service discovery tool; it's a service networking platform with a service mesh, key-value store, and robust health checks. It uses the Raft consensus algorithm, ensuring all nodes agree on the service registry's state.

Consul excels with multi-datacenter support, ideal for global applications, and its health checks can assess application-level health.

Netflix Eureka

Eureka, part of the Netflix OSS stack, is focused on extreme resilience and availability as an AP system. Its main aim is to maintain the service registry online, even during network issues. If a server loses peer contact, it enters "fail-safe" mode, serving requests with the last known data, ensuring no downtime.

etcd and Zookeeper

etcd and Zookeeper are distributed key-value stores often used in service discovery. Both are CP systems known for their consistency, essential for reliable service registries.

• etcd: Known for its role in Kubernetes, etcd manages cluster states using the Raft algorithm and is optimized for read operations.

• Zookeeper: This Apache project uses the ZAB protocol and has long supported distributed systems like Kafka and Hadoop.

Organizations often use these tools for coordination tasks, extending them to service discovery. For more advanced routing, see our article on API gateway patterns in microservices.

Comparison of Service Discovery Tools

Here's a concise guide to help you choose the right architectural tool based on key features and common use cases.

Ultimately, the right service discovery for microservices hinges on your specific requirements. Consul offers an all-in-one solution with service mesh capabilities. For maximum availability, Eureka is ideal. If using etcd or Zookeeper, they can be efficiently utilized for discovery.

Exploring Different Service Types

Kubernetes offers various Service types to expose applications according to your needs:

• ClusterIP: The default option, providing an internal IP accessible only within the cluster, ideal for internal microservice communication.

• NodePort: Exposes the Service on a static port across each node's IP, allowing external access, typically for development or specific proxy configurations.

• LoadBalancer: Suitable for cloud environments, this option automatically creates an external load balancer from your cloud provider to route internet traffic to your Service.

• Ingress: Although not a Service type, an Ingress controller serves as an L7 load balancer for HTTP/S routing, allowing multiple services to share a single IP based on hostname or path rules, commonly used for web applications.

Best Practices for Building Resilient Service Discovery

Ensuring communication between microservices is essential, but maintaining this connection when issues arise is critical. A reliable service discovery setup is fundamental for production environments. The focus should be on designing systems with built-in failure tolerance from the start.

Implement Meaningful Health Checks

A simple network ping is insufficient for assessing service health. It's akin to confirming a chef's presence without checking their ingredients. A robust system requires in-depth health checks.

Go beyond network status: verify database connections, dependency access, and core functionality. For example, a payment service is unhealthy if it can't connect to the gateway, despite its API responsiveness.

Use Caching and Smart Retries

If your service registry goes down, services can't find each other, risking system failure. Client-side caching can help by storing the last known locations of dependencies. If the registry fails, clients use this cached data to continue functioning. Combine this with smart retry logic, like exponential backoff, to avoid overwhelming a struggling service. For more on these strategies, see our guide on the circuit breaker vs. retry pattern.

Secure and Federate Your Registry

An unsecured service registry poses a major security risk, as it reveals your infrastructure map. Unauthorized access could allow traffic rerouting or service shutdowns. Ensure your registry is secured with robust access controls and encryption.

For large-scale, multi-region setups, consider registry federation. This involves separate but linked registry clusters in each region, offering significant advantages:

• Improved Latency: Services access their local registry for quicker lookups.

• High Availability: A regional registry outage won’t affect other regions.

• Fault Isolation: Issues remain confined to one area, avoiding widespread disruptions.

By integrating thorough health checks, smart caching, and a secure, distributed registry, you can establish a resilient service discovery for microservices.

Despite familiarity with these patterns and tools, engineers often encounter recurring questions when implementing service discovery for microservices. Addressing these common confusions can help avoid mistakes and lead to better architectural decisions.

Service Mesh vs. Service Registry

A common question is, "What's the difference between a service mesh and a service registry?" While they often work in tandem, they address distinct issues.

A service registry (such as Consul or Eureka) acts like a phonebook, maintaining an updated list of network locations for all services. It is straightforward and crucial.

A service mesh (like Istio or Linkerd) is more akin to the entire telephone network. It uses the registry to locate services but offers additional features like intelligent traffic routing, encryption, circuit breaking, and observability.

A service registry identifies where a service is, while a service mesh manages how services interact once identified. Though every service mesh requires a discovery mechanism, not all discovery setups are service meshes. A registry can suffice, but a mesh provides a more robust management toolkit.

When you think about GitHub, you probably think about code hosting, pull requests, and collaboration. But behind the scenes, GitHub is also responsible for securing thousands of repositories and millions of lines of code. And doing that manually? Not even remotely possible. The answer isn’t just “security engineers reviewing code.” That would never scale. Instead, GitHub relies heavily on something much more powerful, CodeQL. Let’s discover what it is and how GitHub uses it to secure the code at this huge scale.

What is CodeQL?

CodeQL is a static analysis tool. But unlike traditional tools that scan code using simple patterns or keyword matching, it treats code like structured data. This means you can actually query your codebase the same way you would query a database.

You can think of your codebase as structured data. So instead of asking: “Where is this string used?” With CodeQL, you can ask questions like:

• “Where is user input flowing into a database query?”

• “Which APIs are being used without proper validation?”

• “Where are we missing authorization checks?”

That shift from text search to semantic understanding is what makes CodeQL so powerful.

How GitHub Uses CodeQL Internally

At GitHub CodeQL isn’t optional, it’s part of their default development workflow. For most repositories, CodeQL runs automatically on every pull request. So whenever a developer pushes code or opens a PR, CodeQL scans it and flags potential issues before the code is merged.

This ensures that:

• vulnerabilities are caught early

• developers get instant feedback

• security becomes part of development, not an afterthought

For the majority of GitHub’s repositories, this default setup is enough to maintain a strong security baseline. However, not all codebases are the same. Some systems like GitHub’s large Ruby monolith have unique patterns, internal APIs and specific security risks. To handle this, GitHub builds custom query packs.

Why Not Just Write Queries Directly?

Initially, GitHub stored queries directly inside repositories. But this caused problems:

• Every update required a deployment

• Queries weren’t precompiled thus slower CI

• Version mismatches caused confusing failures

The Better Approach

GitHub moved query packs to a central registry (GitHub Container Registry). This allowed them to version queries properly, deploy updates faster and avoid CI instability.

They also follow a smart strategy:

• During development: use latest dependencies

• During release: lock versions for stability

This balance ensures both innovation and reliability.

Treating Security Queries Like Production Code

One of the most interesting things GitHub does is how seriously they treat their queries. They don’t just write them and hope they work. Instead, they write unit tests for queries, test them on sample code and run them through CI pipelines. This ensures fewer false positives, better developer trust and stable security checks.

What Do These Queries Actually Detect?

The real value of CodeQL comes from what it can detect. GitHub uses custom queries to enforce important security rules.

1. Detecting Unsafe API Usage

Some internal APIs become dangerous when they handle unsanitized user input.

CodeQL identifies:

• where input is not sanitized

• where risky APIs are used improperly

2. Enforcing Authorization Rules

GitHub ensures that every REST API endpoint defines proper access control. If a developer creates an endpoint but forgets to include the required control_access method, CodeQL flags it immediately.

This prevents:

• unauthorized access

• security gaps in APIs

3. Catching Unsafe Patterns

Example: Using .decrypt on ActiveRecord models.

In simple terms, this method takes data that was stored in an encrypted form (for safety) and converts it back into plain text. Now here’s the problem. When you use .decrypt, it doesn’t just read the data, it can permanently store it in an unencrypted form. That means something that was supposed to stay protected (like passwords, tokens, or personal data) could accidentally become visible in plain text.

This can:

• expose sensitive data

• break encryption guarantees

CodeQL automatically scans the code and detects when .decrypt is used.

Instead of waiting for a security engineer to find this manually, it flags it during the pull request and alerts the developer immediately. So the developer can avoid using it or replace it with a safer approach.

Not All Alerts Are Blockers

GitHub uses two types of alerts:

• Blocking alerts: Alerts that must be fixed before merge

• Advisory alerts: These are guidance for developers

This balance is important. If it’s too strict, it slows down development. If it’s too loose, it risks security. GitHub finds the right middle ground.

Variant Analysis

One of the most powerful techniques GitHub uses is variant analysis.

What is Variant Analysis?

When a vulnerability is found in one place, GitHub asks: “Where else could this same issue exist?” Instead of fixing just one instance, they search for similar patterns across all repositories.

Scaling This Across Repositories: MRVA

So far, we’ve seen how CodeQL can detect issues in a single repository. But here’s the real problem GitHub faces: What if the same vulnerability exists in hundreds of repositories? Fixing one repo is easy but finding and fixing it everywhere? That’s the hard part.

This is where Multi-Repository Variant Analysis (MRVA) comes in.

Instead of fixing a vulnerability in just one place, GitHub writes a CodeQL query for that pattern and runs it across multiple repositories at once. This helps them quickly identify all similar cases across the system.

How It Works

Let’s say a vulnerability is found where user input is passed unsafely into a database query. Instead of manually searching, GitHub writes a CodeQL query that:

• tracks user input

• follows how it flows through the code

• detects where it is used dangerously

This query is then executed across many repositories.

Why not use Simple Search?

Simple code search works on text and doesn’t understand logic or data flow at all. On the other hand CodeQL understands how data moves and detects real patterns, not just keywords.

Security Built into CI

One of the biggest reasons this works is integration with CI. CodeQL runs automatically on every pull request without developer intervention.

This means:

• issues are caught early

• developers fix problems immediately

• no separate security audit needed

Takeaways

There are some strong system design takeaways here.

1. Automation is Non-Negotiable: At scale, manual security simply doesn’t work. You need automated detection systems

2. Customize for Your System: Generic tools are not enough. GitHub builds custom queries for its own needs

3. Shift Security Left: Security should happen during development and not after deployment.

4. Detect Patterns, Not Just Bugs: Fixing one issue is not enough. The better approach is to find all similar issues.

5. Build Trust in Tooling: Testing queries and reducing false positives ensures developers actually trust the system

By now, you must have had a clear idea of, How GitHub Uses CodeQL to Secure Code at Scale? In a nutshell, GitHub uses CodeQL to automatically scan code for security issues by treating code like data and running queries on it. With custom queries and variant analysis, it detects and prevents vulnerabilities at scale across thousands of repositories.

Congratulations! You've just advanced another step in your tech journey. Keep progressing!

Ubuntu goes AI - Ubuntu is preparing to bring AI features directly into the OS, but with a strong focus on local inference, privacy, and user control instead of cloud-first AI assistants. Canonical says the new features will improve accessibility, automation, and troubleshooting while staying optional and open-source friendly. Read full blog here.

Vibe coding has a new Name! - Andrej Karpathy says “vibe coding” is no longer the right term for modern AI-assisted development, proposing “agentic engineering” instead as AI agents take on larger coding and decision-making roles. The shift reflects how developers are moving from simply prompting AI to actively supervising autonomous coding workflows, architecture, and validation.

The Bottleneck Was Never the Code

For years, the software industry believed the biggest problem in engineering was writing code faster. Better programming languages, smarter IDEs, automation tools, and now AI coding agents all promised the same thing: higher developer productivity. And in many ways, they delivered. AI tools today can generate features, write tests, fix bugs, and even build prototypes in hours instead of weeks.

But something interesting is happening inside teams using these tools heavily: coding is no longer the slowest part of software development.

The real bottleneck is deciding what should actually be built.

As AI agents become better at implementation, engineers spend less time waiting for code and more time waiting for clarity. Product managers, tech leads, and leadership teams now have to create extremely detailed specifications, acceptance criteria, workflows, and architectural direction so agents can execute correctly. The challenge has shifted from “can we build this?” to “do we clearly understand what we want?”

That sounds simple, but it exposes a problem software teams have always had: alignment.

Collaboration Was Always the Hard Part

Software has never been just about typing code. Large systems are built by groups of people negotiating priorities, trade-offs, and product decisions. AI agents do not remove this complexity. They amplify it.

When code becomes cheap to generate, companies naturally start building more things. Features that once took months now take days. Internal tools appear overnight. Experimental products multiply quickly. But users still absorb products at the same speed, and teams still struggle with focus.

This creates a new risk: shipping too much without enough coherence.

A product with twenty AI-generated features is not automatically better than one with five carefully designed ones. In fact, faster development often increases the need for discipline and prioritization.

Why Context Matters More Than Ever

One of the biggest hidden challenges in software engineering is context. Senior engineers often make decisions based on years of undocumented knowledge: old outages, failed migrations, architectural compromises, and unwritten team conventions.

Humans absorb this naturally through meetings, debugging sessions, Slack conversations, and shared experiences. AI agents cannot.

Agents only know what is written down or explicitly connected to them. Missing context can lead to technically correct but strategically wrong decisions. That is why many companies are now investing in systems that turn scattered company knowledge into searchable, structured context for AI agents.

This may become one of the most important infrastructure layers of the AI era.

The New Competitive Advantage

The companies that succeed with AI will probably not just be the ones with the best models. They will be the organizations that maintain alignment while scaling output rapidly.

AI is becoming a multiplier for organizational quality. Strong teams become dramatically faster. Weak teams become chaotic faster.

The future of software engineering may depend less on raw coding ability and more on communication, documentation, focus, and the ability to preserve shared understanding across teams. The bottleneck was never truly the code. It was always the humans trying to stay aligned while building something together.

Anthropic’s new feature that Dreams! - Anthropic has introduced a new “dreaming” system that allows AI agents to review past actions, identify mistakes, and improve future performance without direct human retraining. The feature is designed to make long-running AI agents more autonomous by giving them a kind of reflective memory between tasks and sessions.

Apple lets you choose your AI models: Apple is reportedly planning to open Apple Intelligence to third-party AI chatbots in iOS 27, letting users choose models like Claude, Gemini, or ChatGPT as their default AI assistant. The new “Extensions” system could plug external AI directly into Siri, Writing Tools, and other system features, marking a major shift away from Apple’s traditionally closed ecosystem.

Buzz of the Week:

Semantic Caching

Semantic Caching is an AI infrastructure technique where systems cache answers based on meaning instead of exact text matches, dramatically reducing latency and inference costs for LLM applications. Most engineers know traditional caching, but semantic caching uses embeddings to detect when two different prompts are “close enough” to reuse an earlier response. For example, “How do I reset my password?” and “I forgot my password, what do I do?” may trigger the same cached AI result even though the wording is different. This is becoming critical because AI apps are expensive to run at scale, especially with agent workflows repeatedly asking similar questions. Companies building production AI systems are now combining vector databases, embedding models, and semantic similarity search to create smarter caching layers.

Things that launched. Things that went viral. Things you'll pretend to try.

Comby

Comby is a structural search-and-replace tool for codebases.

Zoxide

Zoxide is a smarter cd command that learns your habits.

WinFindGrep

WinFindGrep - Multi-directory text search and replace utility for Windows with grep-like functionality

Build Braincells, Not Just Features

This weekend’s read: AI Is Generating More Tests. But Are They Preventing the Next Cloud Outage?

This week’s watch: The Hunt for Lux: The Internet’s Most Disturbed User.

Meanwhile…

A distributed systems architecture is a method of building software where different application components run on separate, networked computers. These components communicate by passing messages, coordinating their actions to achieve a common goal. This approach is the backbone of modern, large-scale services like Netflix and Amazon, enabling them to handle massive user loads and remain available even when individual parts fail.

Why Distributed Systems Matter

Contrast a traditional monolithic system with a distributed system architecture. A monolith is akin to a single chef managing an entire restaurant kitchen, from preparation to cleaning, eventually becoming a bottleneck. In contrast, a distributed system is like a culinary team, where specialized chefs handle specific tasks, working together to deliver a complete meal, much like distributed components collaborate to deliver an application.

This model offers practical benefits:

• Scalability: Handle user surges by adding more computers to the network, as seen with services like Instagram.

• Fault Tolerance: If one component fails, the system continues to function, preventing total outages.

• Performance: Tasks are processed in parallel across machines, improving efficiency and user experience.

Implementing distributed systems is crucial for resilient applications. For further understanding, explore the core system design fundamentals.

Core Principles of Distributed Systems at a Glance

Each of these principles introduces unique challenges. The lack of a global clock complicates data consistency, while independent failures necessitate robust error-handling mechanisms. However, mastering these principles is what allows engineers to build systems that can operate reliably at a global scale.

Understanding Core Distributed Systems Concepts

Before designing a distributed system, understanding its key components is essential. The most important are nodes (individual network computers), latency (communication delay between nodes), and fault tolerance (the ability to function despite failures).

Latency is a physical limitation. For instance, a video call across continents has a delay due to the signal traveling through undersea cables. Similarly, data between servers in different centers faces speed-of-light constraints.

A tough challenge is partial failure, where one node or link fails but the system continues. For example, if a database replica is unreachable, some users get stale data while others receive current information. Designing for this distinguishes reliable systems from fragile ones.

This model's growing adoption is seen in market trends, with the distributed hybrid infrastructure market projected to reach $234.56 billion by 2030, as per a report. In such settings, performance techniques are essential, as discussed in our guide on distributed caching.

Navigating Trade-offs with the CAP Theorem

In distributed systems, every design decision involves trade-offs, guided by the CAP Theorem. This principle states that a distributed data store can only ensure two of the following: Consistency, Availability, and Partition Tolerance.

Given that network partitions are inevitable, the choice usually lies between consistency and availability. For instance, in an airline booking system:

• Availability (AP System): Users can continue booking across separated data centers, risking double-bookings, which must be resolved later.

• Consistency (CP System): One network part becomes read-only or unavailable to prevent double-bookings, but this may turn away customers during the partition.

A banking system prioritizes consistency to avoid financial errors, while social media favors availability to keep content accessible, even if outdated. For more insights, our guide explains the CAP theorem and its impact on system design.

Key Patterns for Building Resilient Systems

With the core concepts set, let's examine the architectural patterns crucial for effective distributed systems architecture. These patterns are proven solutions for scaling challenges.

Two essential patterns are replication and sharding. Replication involves keeping multiple data copies on different nodes to ensure data access if a server fails. Sharding, or partitioning, breaks massive datasets into smaller pieces, or shards, across multiple servers. For instance, a social media app might shard user data by region to speed up local queries.

The infographic below shows a typical layered structure for these systems.

This design divides the presentation layer, application logic, and data storage, isolating failures to prevent them from affecting other layers. Efficient data movement between layers is addressed by data pipeline architectures like Lambda and Kappa. Implementing these requires knowledge of fault-tolerance strategies, such as the circuit breaker and retry patterns.

Comparison of Core Distributed System Patterns

To make sense of the most common architectural patterns, it helps to see them side-by-side. The table below breaks down their primary goals, typical use cases, and the key benefits they bring to the table.

Each of these patterns addresses a specific piece of the distributed systems puzzle. By combining them thoughtfully, you can build systems that are not just powerful but also resilient enough to withstand the inevitable bumps in the road.

How Leading Tech Companies Use These Architectures

Architectural theories are realized on a large scale by top technology companies through distributed systems architecture, turning design patterns into everyday applications.

Netflix exemplifies microservices architecture with its platform of numerous independent services, each handling specific tasks like billing and user authentication. This design ensures that a failure in one service, such as the recommendation engine, doesn't affect other functions like streaming.

E-commerce leaders such as Amazon use sharding and replication to handle millions of transactions during events like Prime Day. Their data is sharded and replicated globally, providing low-latency access and high availability even under heavy load.

For more on this topic, see how Meta efficiently manages exabytes of data worldwide.

Designing for a Distributed Future

Adopting a distributed systems architecture is a strategic choice that supports global scale, high availability, and resilience. Initially utilized by web-scale companies, these practices are now widespread in various industries.

In the industrial sector, distributed control systems (DCS) oversee large-scale manufacturing and energy operations. The DCS market is expected to grow from USD 22.71 billion to USD 29.37 billion by 2030, as reported by Mordor Intelligence. This growth highlights significant investment in distributed technologies for critical applications.

Trends like serverless and edge computing enhance this model by reducing latency and expanding possibilities, offering essential tools to tackle complex engineering challenges.

Frequently Asked Questions

What Is the Biggest Challenge in Distributed Systems Design?

Handling partial failures is a major challenge when transitioning from monoliths. Unlike monolithic apps, which are either fully operational or not, distributed systems can experience ongoing partial failures like network unreliability, server crashes, or slow databases while still functioning overall. This necessitates designing for failure from the start, using essential tools like data replication and circuit breakers. The main challenge lies in ensuring all nodes agree on the system's state, despite some being unreliable or unreachable, maintaining availability and consistency. Thus, you must prepare for a broader range of failure scenarios than in centralized systems.

When Is a Monolithic Architecture a Better Choice?

Despite the hype surrounding distributed systems, a monolith can be ideal for smaller projects or early-stage startups with a straightforward scope. Its simplicity allows easier development, testing, and deployment, offering a speed advantage crucial for small teams. If there's no urgent need to scale parts of your application separately, a monolith is a practical choice. Start simple and consider a shift to a distributed architecture only if complexity necessitates it.

Imagine using a product that keeps getting faster every week without upgrades, without migrations and without touching a single setting. That’s exactly what Snowflake has been doing. Over time, they’ve improved performance by 27%, tracked using something called the Snowflake Performance Index (SPI). But this isn’t just a random number, it represents real improvements experienced by real users on real workloads. Let’s discover what exacly happened and how Snowflake achieved this in this blog today.

What is Snowflake?

Snowflake is a cloud data platform.

In simple words, It helps companies store, process, and analyze massive amounts of data.

You can:

• Run SQL queries

• Build dashboards

• Process analytics workloads

• Ingest data from multiple sources

All without worrying about infrastructure.

What is the Snowflake Performance Index (SPI)?

The Snowflake Performance Index (SPI) is a metric Snowflake introduced to measure: How much faster customer workloads are getting over time.

This is important because most companies:

• Use synthetic benchmarks (fake tests)

• Show ideal scenarios

Snowflake does something different. It measures performance on real production workloads which means, totally real queries, real data and real usage patterns.

The Numbers

• 27% improvement since Aug 2022

• 12% improvement in the last 12 months

This is not theoretical, this is what customers actually experience.

How Snowflake Improves Performance

Here’s the interesting part. Snowflake doesn’t rely on:

• Manual tuning

• Configuration changes

• Version upgrades

Instead, they continuously improve the core engine and release updates every week.

This works because Snowflake uses a fully managed cloud architecture and a consumption-based pricing model.

So when performance improves:

• Queries run faster

• Compute usage drops

• Costs reduce automatically

Key Areas Where Snowflake Improved Performance

1. Faster Data Ingestion

Everything starts with getting data into Snowflake. If ingestion itself is slow, everything downstream gets delayed.

Snowflake improved how it reads and processes semi-structured data formats like JSON and Parquet. These formats are widely used but can be expensive to parse, especially when dealing with case-insensitive data. By optimizing this layer, Snowflake was able to improve ingestion performance by up to 25%.

By doing this, data becomes available faster for querying. If you’re running pipelines or near real-time analytics, this directly reduces the overall latency of your system.

2. Faster Communication Between Nodes

Snowflake doesn’t run your query on a single machine. It spreads the work across multiple machines (nodes). These machines constantly send data to each other while processing a query.

Earlier, this communication could become a bottleneck.

So Snowflake improved:

• how data is transferred between nodes

• how data is compressed before sending

• optimized aggregation placement, essentially deciding where certain computations (like SUM or COUNT) should happen so that less data needs to move across the network.

You can think of it like this, instead of moving a lot of raw data across machines, Snowflake now tries to process data earlier and send less of it around.

This reduces waiting time and makes queries faster.

3. Smarter Query Optimization

One of the biggest wins comes from making the query optimizer smarter.

Whenever you run a SQL query, the system doesn’t just execute it as-is. It first decides the best possible way to run it, which tables to scan first, how to perform joins, where to filter data, and so on.

Snowflake improved this decision-making process in a few key ways. Now it:

• better understands how much data each filter will remove

• chooses a smarter order to join tables

• decides when to move data vs when to process locally

For example, if one table is huge and the other is small, it can decide the most efficient way to join them instead of blindly following the query order.

You don’t change your query, Snowflake just runs it in a better way.

4. Improvements in Query Execution

Once the plan is ready, the query actually runs. Snowflake improved how this execution happens. Instead of processing everything at the end, it now tries to reduce data as early as possible.

For example:

• applying filters early

• doing aggregations earlier

• reducing intermediate data

If you shrink the data early, everything that follows becomes faster. It’s like cleaning unnecessary items from a list before processing it, less work overall.

5. Better Memory Management

Memory is one of the most critical resources during query execution. If a query runs out of memory, it spills data to disk, which is significantly slower.

Snowflake improved how it manages memory, especially for complex queries involving multiple joins. Now it:

• avoids unnecessary memory usage

• reduces chances of spilling to disk

• handles large joins more efficiently

It results in fewer spills to disk and more work happening in-memory, which leads to faster and more stable query performance.

6. Smarter Query Pruning with Top-K Optimization

One of the more interesting improvements is something called Top-K pruning.

Consider this query:

SELECT * FROM products
ORDER BY price DESC
LIMIT 10;

You only need the top 10 results.

Earlier, the system might scan a large portion of the data before deciding the top 10.

Now Snowflake does something smarter: As soon as it knows the remaining data can’t affect the result, it stops scanning further

So instead of doing full work, it does only the required work and this directly improves performance (around 12.5% for such queries).

7. Query Acceleration Service Enhancements

Snowflake also improved its Query Acceleration Service (QAS), which is designed to speed up heavy queries by offloading parts of the work to additional compute resources.

Now, more types of queries can use this service (even INSERT queries). So more queries benefit from acceleration automatically without any manual setup.

8. Better Clustering and Data Pruning

Finally, Snowflake improved how data is organized internally through clustering.

Better clustering means related data is stored closer together, which allows the system to skip irrelevant data during query execution, a process known as pruning.

When pruning is more effective, queries scan less data. Less scanning directly translates to faster queries and lower compute costs. Snowflake estimates this can reduce costs by around 10% for certain workloads.

Putting it All Together

If you look at everything together, Snowflake improved performance by focusing on a few simple ideas:

• Get data in faster

• Move less data between machines

• Make smarter decisions before execution

• Reduce data as early as possible

• Avoid doing unnecessary work

• Use memory and compute more efficiently

None of these changes require you to rewrite queries, tune configs and upgrade anything

And that’s the most important part. Snowflake improved performance not by adding complexity for users, but by making the system smarter internally.

Takeaways

This isn’t just about Snowflake getting faster, it’s about how good systems are designed.

First, continuous improvement beats big releases. Instead of shipping rare, heavy updates, Snowflake improves performance in small, steady steps. Over time, these changes compound into massive gains.

Second, optimize the system, not the user. Great systems don’t ask users to tune queries or configs. They improve internally so things just work better without extra effort.

Third, measure real workloads, not benchmarks. Snowflake tracks performance using actual customer queries, not synthetic tests. That’s what makes the improvements meaningful in real-world scenarios.

Another key idea is avoiding unnecessary work. With optimizations like Top-K pruning, the system stops processing once it has enough data. This simple principle can drastically improve performance.

Finally, smarter decisions beat more resources. A better optimizer, choosing the right join order or execution plan can improve performance without adding more compute.

By now, you must have had a clear idea of, How Snowflake Improved Performance by 27%? In a nutshell, Snowflake improved performance by 27% through continuous backend optimizations in ingestion, query planning, and execution without requiring any user changes. By making the system smarter (not heavier), it delivers faster queries and lower costs automatically over time.

Congratulations! You've just advanced another step in your tech journey. Keep progressing!