<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>Mastering Observability</title>
    <description>Mastering Observability is the home of the highly successful &quot;The Observability Digest&quot; newsletter. Our weekly, community-driven publication delivers expert insights, cutting-edge strategies, and the latest trends in observability straight to your inbox. Join a thriving community of forward-thinking tech leaders and stay ahead of the curve with content curated by and for industry professionals.</description>
    
    <link>https://www.masteringobservability.com/</link>
    <atom:link href="https://rss.beehiiv.com/feeds/HTtjnQm619.xml" rel="self"/>
    
    <lastBuildDate>Fri, 10 Jul 2026 14:20:30 +0000</lastBuildDate>
    <pubDate>Thu, 09 Jul 2026 11:52:34 +0000</pubDate>
    <atom:published>2026-07-09T11:52:34Z</atom:published>
    <atom:updated>2026-07-10T14:20:30Z</atom:updated>
    
      <category>Leadership</category>
      <category>Machine Learning</category>
      <category>Artificial Intelligence</category>
    <copyright>Copyright 2026, Mastering Observability</copyright>
    
    <image>
      <url>https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/publication/logo/51d69527-d6a4-44db-9fca-9b6cf2fee3e4/beehiiv-logo-800__1_.png</url>
      <title>Mastering Observability</title>
      <link>https://www.masteringobservability.com/</link>
    </image>
    
    <docs>https://www.rssboard.org/rss-specification</docs>
    <generator>beehiiv</generator>
    <language>en-us</language>
    <webMaster>support@beehiiv.com (Beehiiv Support)</webMaster>

      <item>
  <title>byte-size: What Is MCP?</title>
  <description>The open standard that lets AI agents use your tools and data. What MCP is, why it exists, and its honest limits.</description>
      <enclosure url="https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/ca352661-9a6d-460b-bda1-663f5b54996f/bytesize_mcp_og.png" length="68984" type="image/png"/>
  <link>https://www.masteringobservability.com/p/what-is-mcp</link>
  <guid isPermaLink="true">https://www.masteringobservability.com/p/what-is-mcp</guid>
  <pubDate>Thu, 09 Jul 2026 11:52:34 +0000</pubDate>
  <atom:published>2026-07-09T11:52:34Z</atom:published>
    <category><![CDATA[Newsletter]]></category>
    <category><![CDATA[Observability]]></category>
    <category><![CDATA[Mcp]]></category>
    <category><![CDATA[Ai Agents]]></category>
    <category><![CDATA[Byte Size]]></category>
  <content:encoded><![CDATA[
    <div class='beehiiv'><style>
  .bh__table, .bh__table_header, .bh__table_cell { border: 1px solid #C0C0C0; }
  .bh__table_cell { padding: 4px; background-color: #f9fafb; }
  .bh__table_cell p { color: #2D2D2D; font-family: 'Space Grotesk',Helvetica,Arial,sans-serif !important; overflow-wrap: break-word; }
  .bh__table_header { padding: 6px; background-color:#e0f7f3; }
  .bh__table_header p { color: #222222; font-family:'Space Grotesk',Helvetica,Arial,sans-serif !important; overflow-wrap: break-word; }
</style><div class='beehiiv__body'><p class="paragraph" style="text-align:left;"><i>The open standard that lets AI agents actually use your tools and data, in plain English, and what it does not solve.</i></p><p class="paragraph" style="text-align:left;">If you have wired an AI assistant to a single tool, you know the pain: a bespoke connector, hand-built, for that one job. The Model Context Protocol, MCP, is the industry&#39;s answer to that. It is the standard way to plug AI models into the tools, data and systems they need, so you build the connection once and reuse it everywhere. Here is what MCP actually is, why it exists, and the honest limits before you let an agent loose on production.</p><h2 class="heading" style="text-align:left;" id="what-is-mcp">What is MCP?</h2><p class="paragraph" style="text-align:left;">MCP, the Model Context Protocol, is an open standard, introduced by Anthropic in late 2024, for connecting AI models to external tools and data through one common interface. The usual analogy is a USB-C port for AI: instead of a different cable for every device, one standard connector everything speaks. A model that speaks MCP can read a file, query a database or call an API through the same protocol, rather than a custom integration per tool. In December 2025 Anthropic handed MCP to the Agentic AI Foundation, a fund under the Linux Foundation, so it is now governed in the open rather than by a single vendor.</p><h2 class="heading" style="text-align:left;" id="the-pieces-that-matter">The pieces that matter</h2><p class="paragraph" style="text-align:left;">You do not need the whole spec, just three roles. The host is the application the person uses, such as an AI assistant or an IDE. The client lives inside the host and holds one connection. The server is the small adapter that exposes a tool, a data source or an action to any MCP client. Build one MCP server for your ticketing system and every MCP-aware model can use it. That is the shift: connectors become reusable, not rebuilt per model.</p><h2 class="heading" style="text-align:left;" id="why-it-matters">Why it matters</h2><ul><li><p class="paragraph" style="text-align:left;"><b>One integration, many models.</b> Because the protocol is standard, you wire a system up once and any MCP-capable model can use it, so you are not rebuilding the same plumbing for each new model.</p></li><li><p class="paragraph" style="text-align:left;"><b>It is becoming the default.</b> After launch it was adopted by other major providers, and the move to the Linux Foundation makes it a shared standard, not one company&#39;s roadmap.</p></li><li><p class="paragraph" style="text-align:left;"><b>It is how agents get real reach.</b> MCP is the mechanism by which an agent moves from talking to acting: reading your systems and taking actions in them.</p></li></ul><h2 class="heading" style="text-align:left;" id="the-honest-ledger">The honest ledger</h2><ul><li><p class="paragraph" style="text-align:left;"><b>It is a protocol, not a guardrail.</b> MCP standardises the connection; it does not decide what an agent should be allowed to do. Permissions, approval and accountability are still yours to design.</p></li><li><p class="paragraph" style="text-align:left;"><b>Security surface grows.</b> Every MCP server is a new door into a system. What it exposes, and who can reach it, is now an operational and security question, not a lab one.</p></li><li><p class="paragraph" style="text-align:left;"><b>It is young and moving.</b> The spec is evolving through working groups and proposals, so expect change, and check how mature the servers you rely on actually are.</p></li><li><p class="paragraph" style="text-align:left;"><b>Standardising access is not the same as understanding it.</b> You still need to observe what your agents do through these connections, or you have automated actions you cannot see.</p></li></ul><h2 class="heading" style="text-align:left;" id="so-is-mcp-for-you">So, is MCP for you?</h2><p class="paragraph" style="text-align:left;">If you are putting AI agents anywhere near real systems, MCP is the plumbing you will end up using, so it is worth understanding now rather than after an incident. The honest question is not whether to adopt it, but how you govern it: which actions you expose through MCP servers, who approves the risky ones, and how you watch what the agents do. Adopt the standard, but decide the guardrails first.</p><hr class="content_break"><p class="paragraph" style="text-align:left;"><b>Where are you letting agents act through MCP, and how are you governing it?</b> I would like to hear it, especially the near-misses. Reply or <a class="link" href="https://calendly.com/managing_observability/30min?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-mcp" target="_blank" rel="noopener noreferrer nofollow">book a slot</a> and tell me what you found.</p><div class="custom_html"><table width="100%" align="center" cellpadding="0" cellspacing="0" border="0" style="max-width:600px;margin:0 auto;"><tbody><tr><td style="background-color:#2F9E8D;height:4px;line-height:4px;font-size:4px;">&nbsp;</td></tr><tr><td style="background-color:#0D2127;padding:24px 24px 18px 24px;"><p style="margin:0 0 8px 0;font-family:'Space Mono','Courier New',monospace;font-size:12px;letter-spacing:2px;color:#74DDCD;text-transform:uppercase;">Get the next one</p><p style="margin:0;font-family:'DM Sans',Helvetica,Arial,sans-serif;font-size:22px;line-height:1.3;color:#ffffff;font-weight:700;">One signal a week. No noise.</p></td></tr><tr><td style="background-color:#ffffff;padding:20px 24px 24px 24px;font-family:'DM Sans',Helvetica,Arial,sans-serif;"><p style="margin:0 0 18px 0;font-size:16px;line-height:1.5;color:#1a1a1a;">If this was useful, Metrics & Mayhem sends one short, practical piece like it to IT operations leaders most weeks. No fluff, no vendor noise.</p><table cellpadding="0" cellspacing="0" border="0"><tbody><tr><td style="background-color:#2F9E8D;border-radius:4px;"><a href="https://masteringobservability.com?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-mcp" style="display:inline-block;padding:12px 22px;font-family:'DM Sans',Helvetica,Arial,sans-serif;font-size:16px;font-weight:700;color:#0D2127;text-decoration:none;">Join free →</a></td></tr></tbody></table><p style="margin:16px 0 0 0;font-size:14px;line-height:1.5;color:#44595b;">Prefer to start with the book? Read a <a href="https://www.masteringobservability.com/metrics-and-mayhem/free-chapter?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-mcp" style="color:#0D2127;text-decoration:underline;">free chapter</a>.</p></td></tr></tbody></table></div><h3 class="heading" style="text-align:left;" id="related-to-the-bytesized-series">Related to the byte-sized series</h3><ul><li><p class="paragraph" style="text-align:left;"><a class="link" href="https://www.masteringobservability.com/p/ai-agent-accountability-who-owns-the-decision?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-mcp" target="_blank" rel="noopener noreferrer nofollow">When The Agent Acts, Who Owns The Decision?</a></p></li><li><p class="paragraph" style="text-align:left;"><a class="link" href="https://www.masteringobservability.com/p/what-is-opentelemetry?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-mcp" target="_blank" rel="noopener noreferrer nofollow">byte-size: What Is OpenTelemetry?</a></p></li><li><p class="paragraph" style="text-align:left;"><a class="link" href="https://www.masteringobservability.com/p/what-is-ebpf?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-mcp" target="_blank" rel="noopener noreferrer nofollow">byte-size: What Is eBPF?</a></p></li></ul><h3 class="heading" style="text-align:left;" id="sources-further-reading">Sources / further reading</h3><ul><li><p class="paragraph" style="text-align:left;">Anthropic introduction: <a class="link" href="https://www.anthropic.com/news/model-context-protocol?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-mcp" target="_blank" rel="noopener noreferrer nofollow">anthropic.com</a></p></li><li><p class="paragraph" style="text-align:left;">Model Context Protocol docs: <a class="link" href="https://modelcontextprotocol.io/docs/getting-started/intro?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-mcp" target="_blank" rel="noopener noreferrer nofollow">modelcontextprotocol.io</a></p></li><li><p class="paragraph" style="text-align:left;">Model Context Protocol overview: <a class="link" href="https://en.wikipedia.org/wiki/Model_Context_Protocol?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-mcp" target="_blank" rel="noopener noreferrer nofollow">Wikipedia</a></p></li></ul></div><div class='beehiiv__footer'><br class='beehiiv__footer__break'><hr class='beehiiv__footer__line'><a target="_blank" class="beehiiv__footer_link" style="text-align: center;" href="https://www.beehiiv.com/?utm_campaign=48751840-1f5c-411c-8e96-2754ccfd709b&utm_medium=post_rss&utm_source=mastering_observability">Powered by beehiiv</a></div></div>
  ]]></content:encoded>
</item>

      <item>
  <title>The Six-Week Decay</title>
  <description>Automation handled the routine and quietly dulled the team. The fix is deliberate reps.</description>
      <enclosure url="https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/ee1c954f-bda5-4af8-ad02-c6f64591b3a5/22_thumbnail_og.png" length="80626" type="image/png"/>
  <link>https://www.masteringobservability.com/p/signal-drop-the-six-week-decay</link>
  <guid isPermaLink="true">https://www.masteringobservability.com/p/signal-drop-the-six-week-decay</guid>
  <pubDate>Mon, 06 Jul 2026 06:02:00 +0000</pubDate>
  <atom:published>2026-07-06T06:02:00Z</atom:published>
    <category><![CDATA[Signal Drop]]></category>
    <category><![CDATA[Sre]]></category>
    <category><![CDATA[Incident Management]]></category>
    <category><![CDATA[Automation]]></category>
    <category><![CDATA[Leadership]]></category>
  <content:encoded><![CDATA[
    <div class='beehiiv'><style>
  .bh__table, .bh__table_header, .bh__table_cell { border: 1px solid #C0C0C0; }
  .bh__table_cell { padding: 4px; background-color: #f9fafb; }
  .bh__table_cell p { color: #2D2D2D; font-family: 'Space Grotesk',Helvetica,Arial,sans-serif !important; overflow-wrap: break-word; }
  .bh__table_header { padding: 6px; background-color:#e0f7f3; }
  .bh__table_header p { color: #222222; font-family:'Space Grotesk',Helvetica,Arial,sans-serif !important; overflow-wrap: break-word; }
</style><div class='beehiiv__body'><p class="paragraph" style="text-align:left;"><i>Automating the routine can quietly de-skill your best responders. This is how to keep the edge on purpose.</i></p><p class="paragraph" style="text-align:left;"><b>In this drop</b></p><ul><li><p class="paragraph" style="text-align:left;"><b>The point:</b> Automating triage removes the small daily reps that kept your team sharp, so the next novel incident finds them rusty.</p></li><li><p class="paragraph" style="text-align:left;"><b>Why it matters:</b> The cost is invisible on your dashboards. It only shows up on the one incident that matters most, weeks later.</p></li><li><p class="paragraph" style="text-align:left;"><b>Try this next week:</b> Run one live incident drill the automation cannot help with. A novel fault, no runbook, your best people, cold.</p></li></ul><h2 class="heading" style="text-align:left;" id="the-point">The point</h2><p class="paragraph" style="text-align:left;">A while back I watched one of the best incident teams I have worked with get quietly worse over about six weeks, and everything we did to them was, on paper, an improvement. We automated triage. The routine incidents got caught and handled, fewer people were woken, the numbers looked wonderful.</p><p class="paragraph" style="text-align:left;">Then a genuinely novel fault arrived, the kind that is in no runbook. The team that used to eat those stalled. They reached for the automation, which had never seen it either, and the humans were slow and tentative. Nothing about them had got worse as people. We had taken away their reps.</p><p class="paragraph" style="text-align:left;">People in dangerous work drill the basics far more than seems necessary, because the reps buy down the risk on the day it is real. Every routine incident used to be a small rep. We automated the boring reps away, and the edge went with them.</p><h2 class="heading" style="text-align:left;" id="reality-check">Reality check</h2><p class="paragraph" style="text-align:left;">What changes everything: the number to watch is not how well you handle the incidents you automated. It is how well you handle the ones you could not.</p><h2 class="heading" style="text-align:left;" id="one-proof">One proof</h2><p class="paragraph" style="text-align:left;">Lisanne Bainbridge named this in 1983, in a paper called the Ironies of Automation: the more you automate a process, the more the operator&#39;s manual skill decays, so the human is least practised at exactly the moment automation fails and hands the problem back. Field note, from one team I worked with and not an industry benchmark: once we automated the routine triage, the share of incidents the humans actually handled fell by roughly a third within a quarter, and the first genuinely novel fault after that took them noticeably longer than their old form. The people had not changed. Their reps had.</p><h2 class="heading" style="text-align:left;" id="where-this-breaks">Where this breaks</h2><p class="paragraph" style="text-align:left;">This breaks if you read it as an argument against automation. It is not. Automation is worth having. The failure is treating it as free and never replacing the reps it absorbs. Keep the automation; add the drills back deliberately.</p><h2 class="heading" style="text-align:left;" id="try-this-next-week">Try this next week</h2><p class="paragraph" style="text-align:left;">Schedule one live incident drill, and make it one the automation cannot help with.</p><p class="paragraph" style="text-align:left;">Pick a novel fault that is not in the runbook. Get your best responders in a room, real or virtual, and let them work it cold: no script, no automated summary.</p><p class="paragraph" style="text-align:left;">Notice how rusty they have got. That is the signal. Then make the drill a habit, because the routine work is no longer sharpening them for you.</p><h2 class="heading" style="text-align:left;" id="three-links-i-am-watching">Three links I am watching</h2><ul><li><p class="paragraph" style="text-align:left;">Lisanne Bainbridge, &#39;Ironies of Automation&#39; (Automatica, 1983): the original and still the clearest statement of the skill-decay problem.</p></li><li><p class="paragraph" style="text-align:left;">Google SRE, the &#39;Wheel of Misfortune&#39; disaster role-play: a lightweight way to give a team reps on incidents that have not happened yet.</p></li><li><p class="paragraph" style="text-align:left;"><a class="link" href="https://www.masteringobservability.com/p/signal-drop-youre-going-to-have-to-suffer-today?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=the-six-week-decay" target="_blank" rel="noopener noreferrer nofollow">Signal Drop 21, &#39;You&#39;re Going To Have To Suffer Today&#39;</a>: the companion on rehearsing the bad night before it arrives.</p></li></ul><h2 class="heading" style="text-align:left;" id="one-question-for-you">One question for you</h2><p class="paragraph" style="text-align:left;">When did a novel incident last find your team rusty, and had automation quietly stopped giving them the reps that used to keep them sharp?</p><p class="paragraph" style="text-align:left;">Allan</p><div class="custom_html"><table width="100%" align="center" cellpadding="0" cellspacing="0" border="0" style="max-width:600px;margin:0 auto;"><tbody><tr><td style="background-color:#2F9E8D;height:4px;line-height:4px;font-size:4px;">&nbsp;</td></tr><tr><td style="background-color:#0D2127;padding:24px 24px 20px 24px;"><p style="margin:0 0 8px 0;font-family:'Space Mono','Courier New',monospace;font-size:12px;letter-spacing:2px;color:#74DDCD;text-transform:uppercase;">Everything in one place</p><p style="margin:0;font-family:'DM Sans',Helvetica,Arial,sans-serif;font-size:22px;line-height:1.3;color:#ffffff;font-weight:700;">New here? Start with the free chapter.</p></td></tr><tr><td style="background-color:#ffffff;padding:22px 24px 24px 24px;font-family:'DM Sans',Helvetica,Arial,sans-serif;"><table cellpadding="0" cellspacing="0" border="0"><tbody><tr><td style="background-color:#2F9E8D;border-radius:4px;"><a href="https://www.masteringobservability.com/metrics-and-mayhem/free-chapter?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=the-six-week-decay" style="display:inline-block;padding:13px 24px;font-family:'DM Sans',Helvetica,Arial,sans-serif;font-size:16px;font-weight:700;color:#0D2127;text-decoration:none;">Read the free chapter →</a></td></tr></tbody></table><p style="margin:20px 0 10px 0;font-family:'Space Mono','Courier New',monospace;font-size:12px;letter-spacing:2px;color:#2F9E8D;text-transform:uppercase;">More from Metrics & Mayhem</p><table width="100%" cellpadding="0" cellspacing="0" border="0" style="font-family:'DM Sans',Helvetica,Arial,sans-serif;font-size:15px;line-height:1.5;"><tbody><tr><td style="padding:7px 0;color:#16282D;"><strong>The book</strong> &nbsp;·&nbsp; <a href="https://amzn.eu/d/0cAuR2K1?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=the-six-week-decay" style="color:#17695C;text-decoration:underline;">Get Metrics & Mayhem →</a></td></tr><tr><td style="padding:7px 0;border-top:1px solid #EAF6F3;color:#16282D;"><strong>Podcast</strong> &nbsp;·&nbsp; <a href="https://www.masteringobservability.com/metrics-and-mayhem/podcast?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=the-six-week-decay" style="color:#17695C;text-decoration:underline;">Listen on Spotify & Apple →</a></td></tr><tr><td style="padding:7px 0;border-top:1px solid #EAF6F3;color:#16282D;"><strong>Connect</strong> &nbsp;·&nbsp; <a href="https://www.linkedin.com/in/allanmann1/?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=the-six-week-decay" style="color:#17695C;text-decoration:underline;">Allan on LinkedIn →</a></td></tr></tbody></table></td></tr></tbody></table></div><hr class="content_break"><h3 class="heading" style="text-align:left;" id="stop-doomscrolling-for-tech-news">Stop doomscrolling for tech news</h3><div class="image"><a class="image__link" href="https://tldr.tech/?utm_campaign={{publication_alphanumeric_id}}&utm_source=beehiiv&utm_medium=newsletter&_bhiiv=opp_900c87de-db1b-4019-9144-c36f6fcd6b88_8a911f41&bhcl_id=ee4ddf75-2483-4974-ab02-f3a0a3204a21_{{subscriber_id}}_{{email_address_id}}" rel="noopener" target="_blank"><img class="image__image" style="" src="https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/c7fb0443-ae7c-438a-a74e-b9beca35b2b7/subscriber_acquisition_primary_ad_v2.png?t=1780590176"/></a></div><p class="paragraph" style="text-align:left;"><a class="link" href="https://tldr.tech/?utm_campaign={{publication_alphanumeric_id}}&utm_source=beehiiv&utm_medium=newsletter&_bhiiv=opp_900c87de-db1b-4019-9144-c36f6fcd6b88_8a911f41&bhcl_id=ee4ddf75-2483-4974-ab02-f3a0a3204a21_{{subscriber_id}}_{{email_address_id}}" target="_blank" rel="noopener noreferrer nofollow">TLDR</a> is the free daily email that does the reading for you. Each issue summarizes the most interesting stories in startups, tech, and programming, curated by ex-Google and Anthropic engineers, so you get the signal without the noise.</p><p class="paragraph" style="text-align:left;">It&#39;s one email instead of 12 open tabs. A 5-minute read with your morning coffee, and you walk in already caught up.</p><p class="paragraph" style="text-align:left;">Tech is just the start. We also cover AI, marketing, dev, and more, so you can follow the topics that matter to your work.</p><p class="paragraph" style="text-align:left;">Free, daily, and read by 7M+ subscribers. <a class="link" href="https://tldr.tech/?utm_campaign={{publication_alphanumeric_id}}&utm_source=beehiiv&utm_medium=newsletter&_bhiiv=opp_900c87de-db1b-4019-9144-c36f6fcd6b88_8a911f41&bhcl_id=ee4ddf75-2483-4974-ab02-f3a0a3204a21_{{subscriber_id}}_{{email_address_id}}" target="_blank" rel="noopener noreferrer nofollow">Subscribe</a> and get your mornings back.</p><p class="paragraph" style="text-align:left;"><a class="link" href="https://tldr.tech/?utm_campaign={{publication_alphanumeric_id}}&utm_source=beehiiv&utm_medium=newsletter&_bhiiv=opp_900c87de-db1b-4019-9144-c36f6fcd6b88_8a911f41&bhcl_id=ee4ddf75-2483-4974-ab02-f3a0a3204a21_{{subscriber_id}}_{{email_address_id}}" target="_blank" rel="noopener noreferrer nofollow">Subscribe for Free</a></p></div><div class='beehiiv__footer'><br class='beehiiv__footer__break'><hr class='beehiiv__footer__line'><a target="_blank" class="beehiiv__footer_link" style="text-align: center;" href="https://www.beehiiv.com/?utm_campaign=72f1d8c5-8f92-4365-8d13-f45d688b5af5&utm_medium=post_rss&utm_source=mastering_observability">Powered by beehiiv</a></div></div>
  ]]></content:encoded>
</item>

      <item>
  <title>When The Agent Acts, Who Owns The Decision?</title>
  <description>The moment an AI agent acts, someone owns the outcome. Most teams have not said who.</description>
      <enclosure url="https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/1b36d866-1e2f-4373-9531-16987469054d/who-owns-the-agent_og_1200x630.png" length="87195" type="image/png"/>
  <link>https://www.masteringobservability.com/p/ai-agent-accountability-who-owns-the-decision</link>
  <guid isPermaLink="true">https://www.masteringobservability.com/p/ai-agent-accountability-who-owns-the-decision</guid>
  <pubDate>Sun, 05 Jul 2026 06:56:00 +0000</pubDate>
  <atom:published>2026-07-05T06:56:00Z</atom:published>
    <dc:creator>Allan Mann</dc:creator>
    <category><![CDATA[Aiops]]></category>
    <category><![CDATA[Governance]]></category>
    <category><![CDATA[Ai Agents]]></category>
    <category><![CDATA[Automation]]></category>
    <category><![CDATA[Leadership]]></category>
  <content:encoded><![CDATA[
    <div class='beehiiv'><style>
  .bh__table, .bh__table_header, .bh__table_cell { border: 1px solid #C9DCDC; }
  .bh__table_cell { padding: 5px; background-color: #FFFFFF; }
  .bh__table_cell p { color: #16282D; font-family: 'DM Sans',Lato,Montserrat,sans-serif !important; overflow-wrap: break-word; }
  .bh__table_header { padding: 5px; background-color:#EAF6F3; }
  .bh__table_header p { color: #16282D; font-family:'DM Sans',Lato,Montserrat,sans-serif !important; overflow-wrap: break-word; }
</style><div class='beehiiv__body'><p class="paragraph" style="text-align:left;">For the last couple of years, the only question we really asked about these systems was whether the answer was any good. You typed something in, it came back with text, and you decided what to do with it. The accountability question barely came up, because a human was still the one who acted. The model advised. A person pressed the button, and that person owned what happened next.</p><p class="paragraph" style="text-align:left;">That arrangement is quietly ending. The systems are moving from answering questions to taking actions on the business&#39;s behalf. The agent does not tell you the order is ready to be placed. It places it. It does not suggest the service might need a restart. It restarts it. The interesting question is no longer how good the answer is. It is the one who owns the action.</p><p class="paragraph" style="text-align:left;">I started thinking about this again, watching the agentic-future framing land at Meta&#39;s Conversations event, where the pitch was agents that do not just reply to a customer but carry out the transaction for them. The platform numbers in a keynote like that are a marketing slide, not evidence, and consumer messaging is not my field. But the shape underneath it is one I keep running into in operations, and the shape is what matters.</p><h2 class="heading" style="text-align:left;" id="the-shift-is-in-the-risk-not-the-co">The shift is in the risk, not the convenience</h2><p class="paragraph" style="text-align:left;">It is easy to read &quot;agents that act&quot; as a convenience story. The work that used to take three clicks now takes none. That framing misses where the change actually lands.</p><p class="paragraph" style="text-align:left;">When a system only advises, the risk sits with the human who chooses to act on the advice. The decision, and the ownership of the decision, never leaves the room. When the system acts, the decision leaves the room. Something now happens in the world (an order placed, a refund issued, a production service bounced) without a person in the loop at the moment it happens. The convenience is real. So is the transfer. You have moved the action out of human hands, and unless you were deliberate about it, you have moved the ownership of the outcome to nobody in particular.</p><p class="paragraph" style="text-align:left;">That is the part most rollouts skip. We get excited about what the agent can now do, and we forget to say who is responsible when it does the wrong thing at the wrong time.</p><h2 class="heading" style="text-align:left;" id="the-question-everyone-skips">The question everyone skips</h2><p class="paragraph" style="text-align:left;">Here is the question. The moment an agent takes an action, someone owns the outcome. Who is it.</p><p class="paragraph" style="text-align:left;">Most organisations cannot answer cleanly. They can tell you who built the agent, who pays for it, and who was in the demo. Ask who owns the decision the agent makes at two in the morning, and you get the answer that should worry you most: the team. &quot;The team&quot; owns it. Which is to say no single named person does, because an outcome owned by everyone is owned by no one, and that becomes obvious at exactly the worst time.</p><p class="paragraph" style="text-align:left;">This is not a new failure. It is the oldest failure in operations wearing new clothes. We have always had automations that act: the script that scales the cluster, the rule that reroutes traffic, the job that clears the queue. The good ones had an owner, a person who understood what the automation did, who got the call when it misfired, and who could be asked, afterwards, why it did what it did. The bad ones were inherited, undocumented, and owned by whoever happened to be holding the rota when they went wrong. Agents make this question louder because they act across more of the business and with more apparent independence. They do not change the question.</p><h2 class="heading" style="text-align:left;" id="the-same-question-two-very-differen">The same question, two very different rooms</h2><p class="paragraph" style="text-align:left;">Picture two rooms. In one, an agent is booking a customer&#39;s order over a messaging app. In the other, an agent is restarting a production service in the middle of a degraded night.</p><p class="paragraph" style="text-align:left;">These feel like completely different problems. One is commerce; one is ops. One has a customer on the other end; one has an on-call rota. But the accountability question is word-for-word identical. When the booking is wrong, when the restart takes down the thing it was meant to protect, who owns that outcome? And in both rooms, the failure mode is identical too: if the honest answer is &quot;the team&quot;, you have an accountability gap, and the gap does not announce itself until the action has already gone wrong.</p><p class="paragraph" style="text-align:left;">The reason this matters is that the commerce version and the ops version are converging. The same pattern (an agent empowered to act on the business&#39;s behalf) is arriving in the customer channel and in the control plane at the same time. If you only ever reasoned about it as a customer-experience feature, you will import the same unowned-automation problem straight into production, where the blast radius is your own estate.</p><h2 class="heading" style="text-align:left;" id="automate-the-action-never-automate-">Automate the action. Never automate the accountability.</h2><p class="paragraph" style="text-align:left;">So what does owning it actually look like. The model I trust is approval by blast radius, and it is not complicated.</p><p class="paragraph" style="text-align:left;">You sort the actions an agent can take by what happens if they are wrong. Low blast radius (the action is cheap to reverse and contained if it misfires) and the gate can be open: let the agent act, log it, review the pattern later. High blast radius (the action is expensive, customer-facing, or hard to undo) and the gate stays human: the agent prepares the action, a named person approves it, and the act and the approval are both recorded. The dial is not &quot;automated or not&quot;. It is &quot;how much can this break, and has someone agreed to own it at that level&quot;.</p><p class="paragraph" style="text-align:left;">The rule that holds the whole thing together is this. You can automate the action. You must never automate the accountability. Every autonomous action an agent is allowed to take needs a named human owner attached to it before it is switched on. Not a team. A person, who understands what the agent does in their area, who gets the call when it goes wrong, and who can answer for it afterwards. The agent does the work. The named owner keeps the responsibility, because responsibility is the one thing you are not allowed to hand to a process.</p><p class="paragraph" style="text-align:left;">This is also what makes the system improvable. When an action has an owner, a bad outcome has a person who notices, investigates, and tightens the gate. When it does not, the same bad outcome just happens again, a little more expensively each time, until it is big enough to become everyone&#39;s problem and therefore still no one&#39;s.</p><h2 class="heading" style="text-align:left;" id="buy-the-agent-first-buy-the-answer">Buy the agent. First buy the answer.</h2><p class="paragraph" style="text-align:left;">None of this is an argument against agents that act. The capability is genuine and a lot of it is worth having. Buy the agent if it earns its place.</p><p class="paragraph" style="text-align:left;">But buy the other thing first. Before you let a system act on the business&#39;s behalf, decide who owns each thing it is allowed to do, and write the name down. The technology question (can it do this) is mostly solved and getting cheaper by the month. The question that decides whether this goes well for you is the one nobody is selling: when the agent is wrong, and at some point it will be, who answers for it. Assign that owner before the agent ever takes its first action, or you will be assigning it mid-incident, which is the most expensive moment to discover you never did.</p><hr class="content_break"><div class="custom_html"><table width="100%" align="center" cellpadding="0" cellspacing="0" border="0" style="max-width:600px;margin:0 auto;"><tbody><tr><td style="background-color:#2F9E8D;height:4px;line-height:4px;font-size:4px;">&nbsp;</td></tr><tr><td style="background-color:#0D2127;padding:24px 24px 18px 24px;"><p style="margin:0 0 8px 0;font-family:'Space Mono','Courier New',monospace;font-size:12px;letter-spacing:2px;color:#74DDCD;text-transform:uppercase;">Work with me</p><p style="margin:0;font-family:'DM Sans',Helvetica,Arial,sans-serif;font-size:22px;line-height:1.3;color:#ffffff;font-weight:700;">An honest read on what your observability is actually doing.</p></td></tr><tr><td style="background-color:#ffffff;padding:20px 24px 24px 24px;font-family:'DM Sans',Helvetica,Arial,sans-serif;"><p style="margin:0 0 18px 0;font-size:16px;line-height:1.5;color:#1a1a1a;">If you lead observability in a regulated enterprise, I run a fixed-scope Observability Assessment for senior IT and engineering leaders. It ends in a written roadmap and a readout, not a sales deck.</p><table cellpadding="0" cellspacing="0" border="0"><tbody><tr><td style="background-color:#2F9E8D;border-radius:4px;"><a href="https://www.masteringobservability.com/advisory?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=when-the-agent-acts-who-owns-the-decision" style="display:inline-block;padding:12px 22px;font-family:'DM Sans',Helvetica,Arial,sans-serif;font-size:16px;font-weight:700;color:#0D2127;text-decoration:none;">See how it works →</a></td></tr></tbody></table><p style="margin:16px 0 0 0;font-size:14px;line-height:1.5;color:#44595b;">Not ready to talk? Start with a <a href="https://www.masteringobservability.com/metrics-and-mayhem/free-chapter?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=when-the-agent-acts-who-owns-the-decision" style="color:#0D2127;text-decoration:underline;">free chapter of Metrics & Mayhem</a>.</p></td></tr></tbody></table></div></div><div class='beehiiv__footer'><br class='beehiiv__footer__break'><hr class='beehiiv__footer__line'><a target="_blank" class="beehiiv__footer_link" style="text-align: center;" href="https://www.beehiiv.com/?utm_campaign=0048bb2f-3940-4424-a33a-16d3cf5cabe8&utm_medium=post_rss&utm_source=mastering_observability">Powered by beehiiv</a></div></div>
  ]]></content:encoded>
</item>

      <item>
  <title>What the customers said at Dynatrace Innovate London</title>
  <description>Strip away the vendor gloss and the message from the floor was steady, honest, and unglamorous.</description>
      <enclosure url="https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/37eac466-feb1-45bc-9cd4-fa8850941f75/Staduim.JPG" length="1406052" type="image/jpeg"/>
  <link>https://www.masteringobservability.com/p/what-customers-said-dynatrace-innovate-london</link>
  <guid isPermaLink="true">https://www.masteringobservability.com/p/what-customers-said-dynatrace-innovate-london</guid>
  <pubDate>Fri, 03 Jul 2026 11:21:43 +0000</pubDate>
  <atom:published>2026-07-03T11:21:43Z</atom:published>
    <dc:creator>Allan Mann</dc:creator>
    <category><![CDATA[Aiops]]></category>
    <category><![CDATA[Observability]]></category>
    <category><![CDATA[Blog]]></category>
    <category><![CDATA[Event]]></category>
    <category><![CDATA[Dynatrace Innovate]]></category>
    <category><![CDATA[Leadership]]></category>
  <content:encoded><![CDATA[
    <div class='beehiiv'><style>
  .bh__table, .bh__table_header, .bh__table_cell { border: 1px solid #C0C0C0; }
  .bh__table_cell { padding: 4px; background-color: #f9fafb; }
  .bh__table_cell p { color: #2D2D2D; font-family: 'Space Grotesk',Helvetica,Arial,sans-serif !important; overflow-wrap: break-word; }
  .bh__table_header { padding: 6px; background-color:#e0f7f3; }
  .bh__table_header p { color: #222222; font-family:'Space Grotesk',Helvetica,Arial,sans-serif !important; overflow-wrap: break-word; }
</style><div class='beehiiv__body'><p class="paragraph" style="text-align:left;">I spent Thursday at the Emirates Stadium for Dynatrace Innovate London. I went as an independent voice, not a buyer and not a seller, with a bag of books and one question I ask at every event like this: what is your team actually wrestling with right now?</p><p class="paragraph" style="text-align:left;">The vendor keynotes were polished, and the platform demos did what platform demos do. And while Dynatrace is definitely leading the pack, that’s not what I want to write about. The part worth your time was the customer stories, so cool, and from the coalface. When peers and practitioners stand up in front of three hundred people and describe what worked, what did not, and what they are still nervous about, you learn more in forty minutes than a year of analyst decks will teach you. Three patterns ran through every one of them.</p><div class="image"><img alt="" class="image__image" style="" src="https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/c83aff17-a8c7-453b-9245-c5d4ffe805eb/Me_Stadium.JPG?t=1783075473"/></div><h2 class="heading" style="text-align:left;" id="pattern-one-the-foundation-came-fir">Pattern one: the foundation came first, always</h2><p class="paragraph" style="text-align:left;">Not a single customer who reported real gains from AI had started with AI. They started with the boring work. A regulated bank described six months of ingesting logs, wiring up queries and building dashboards before any of the clever automation arrived. A large photo-products group described merging five legacy platforms and roughly 200 microservices, and <a class="link" href="https://www.masteringobservability.com/p/what-is-opentelemetry?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=what-the-customers-said-at-dynatrace-innovate-london" target="_blank" rel="noopener noreferrer nofollow">getting their telemetry in order</a> before they let a model near any of it. Their line was the one I keep repeating to clients: work out what needs to be true for you before AI can take part.</p><p class="paragraph" style="text-align:left;">This matters because a broken foundation does not stay neutral when you add AI. It gets amplified. Point a model at noisy, disconnected, half-instrumented telemetry and it will give you confident answers that are wrong. The firms seeing value had earned it by doing the groundwork first. There is no shortcut, and anyone selling you one is selling you a future incident.</p><div class="image"><img alt="" class="image__image" style="" src="https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/95dc0c8e-c484-4904-a885-d6d58c4c80a3/new.JPG?t=1783080270"/></div><h2 class="heading" style="text-align:left;" id="pattern-two-the-unit-of-measure-mov">Pattern two: the unit of measure moved from health to outcome</h2><p class="paragraph" style="text-align:left;">The most striking talk of the day came from a national hospitality brand. Their observability leads did not talk about uptime or error rates. They talked about a booking screen that took around 2.7 seconds to load, the work they did to bring it closer to 1.4, and the revenue and customers that the shift was worth to the business. On stage, they put numbers to it. I will not repeat those figures as fact, because a slide at a vendor event is a claim and not an audited result, but the framing is the point: they measured a technical change in customers retained and pounds earned, not milliseconds saved.</p><p class="paragraph" style="text-align:left;">That is the shift I have been arguing for in the book and the newsletter. For years, observability answered the question &quot;Is it up?&quot; The teams pulling ahead are answering a different question: &quot;What is it worth?&quot; When you can connect a load-time improvement to conversion, you stop begging for engineering time and start being asked for more of it. The tool matters far less than the decision to make business impact the unit of measure.</p><h2 class="heading" style="text-align:left;" id="pattern-three-the-human-is-coming-o">Pattern three: the human is coming out of the loop slowly, and on purpose</h2><p class="paragraph" style="text-align:left;">Every customer was moving toward more automation, and every serious one was doing it with the brakes on. The bank had cut its incident root-cause time dramatically by putting natural language over its query engine, so an ordinary engineer could ask a question instead of waiting a day or two for an expert to write it. The honest detail was the best moment of the session. Their own words, roughly: it is not one hundred percent correct, and that is the point. It gets the team to the right area fast. It is not yet reliable enough to trust blind.</p><p class="paragraph" style="text-align:left;">That is exactly the right posture, and it is rarer than it should be. They are drafting fixes automatically and still routing them through a human, in a regulated shop, with guardrails, and only planning to remove that human once confidence and controls justify it. Nobody in the room who runs something that matters was automating the accountability. They were automating the toil and keeping the judgement. If you have read anything I have written on trust-gated operations, you will know why that made me happy.</p><div class="image"><img alt="" class="image__image" style="" src="https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/4545bda5-5f57-4c85-b270-1c1e2da200bb/20260702_094041.jpg?t=1783075743"/></div><h2 class="heading" style="text-align:left;" id="the-line-i-wrote-down">The line I wrote down</h2><p class="paragraph" style="text-align:left;">The afternoon had a communication session that had nothing to do with observability and everything to do with why good observability work still fails to land. The speaker&#39;s favourite line was this: the winner of the communication race is the person who speaks last. He talked about the curse of knowledge, the trap where the more you understand, the more you assume everyone else does too. He described the job of a technical leader as director of translation, turning MCP, DQL and <a class="link" href="https://www.masteringobservability.com/p/what-is-distributed-tracing?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=what-the-customers-said-at-dynatrace-innovate-london" target="_blank" rel="noopener noreferrer nofollow">distributed tracing</a> into something a board can act on.</p><p class="paragraph" style="text-align:left;">I thought about every brilliant observability programme I have seen stall, not because the engineering was wrong, but because nobody translated it upward. If you run one of these teams, that is your real second job. You are not only building the platform. You are the chief storyteller for what it makes possible, and if you cannot tell that story to the people who hold the budget, the platform does not get its second year.</p><h2 class="heading" style="text-align:left;" id="what-i-took-away">What I took away</h2><p class="paragraph" style="text-align:left;">Innovate London was a good day, and the credit goes to the customers who stood up and were honest, not just triumphant. Strip away the vendor gloss and the message from the floor was steady and unglamorous. Build the foundation before you reach for AI. Measure what the business measures. Automate the work, keep the human on the accountability, and remove that human only when your guardrails have earned it. Then go and learn to explain all of it in plain language, because the best observability in the world is worth nothing if the person paying for it cannot see what it bought them.</p><p class="paragraph" style="text-align:left;">None of that is new. It is just easier to believe when three hundred of your peers are nodding along in a football stadium.</p><p class="paragraph" style="text-align:left;"><i>Allan Mann is the author of Metrics & Mayhem: A CTO&#39;s Guide to Observability That Actually Works and writes the Observability Digest.</i></p><div class="custom_html"><table width="100%" align="center" cellpadding="0" cellspacing="0" border="0" style="max-width:600px;margin:0 auto;"><tbody><tr><td style="background-color:#2F9E8D;height:4px;line-height:4px;font-size:4px;">&nbsp;</td></tr><tr><td style="background-color:#0D2127;padding:24px 24px 18px 24px;"><p style="margin:0 0 8px 0;font-family:'Space Mono','Courier New',monospace;font-size:12px;letter-spacing:2px;color:#74DDCD;text-transform:uppercase;">Get the next one</p><p style="margin:0;font-family:'DM Sans',Helvetica,Arial,sans-serif;font-size:22px;line-height:1.3;color:#ffffff;font-weight:700;">One signal a week. No noise.</p></td></tr><tr><td style="background-color:#ffffff;padding:20px 24px 24px 24px;font-family:'DM Sans',Helvetica,Arial,sans-serif;"><p style="margin:0 0 18px 0;font-size:16px;line-height:1.5;color:#1a1a1a;">If this was useful, Metrics & Mayhem sends one short, practical piece like it to IT operations leaders most weeks. No fluff, no vendor noise.</p><table cellpadding="0" cellspacing="0" border="0"><tbody><tr><td style="background-color:#2F9E8D;border-radius:4px;"><a href="https://masteringobservability.com?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=what-the-customers-said-at-dynatrace-innovate-london" style="display:inline-block;padding:12px 22px;font-family:'DM Sans',Helvetica,Arial,sans-serif;font-size:16px;font-weight:700;color:#0D2127;text-decoration:none;">Join free →</a></td></tr></tbody></table><p style="margin:16px 0 0 0;font-size:14px;line-height:1.5;color:#44595b;">Prefer to start with the book? Read a <a href="https://www.masteringobservability.com/metrics-and-mayhem/free-chapter?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=what-the-customers-said-at-dynatrace-innovate-london" style="color:#0D2127;text-decoration:underline;">free chapter</a>.</p></td></tr></tbody></table></div></div><div class='beehiiv__footer'><br class='beehiiv__footer__break'><hr class='beehiiv__footer__line'><a target="_blank" class="beehiiv__footer_link" style="text-align: center;" href="https://www.beehiiv.com/?utm_campaign=686c490f-c830-4134-bedb-b7b338c06292&utm_medium=post_rss&utm_source=mastering_observability">Powered by beehiiv</a></div></div>
  ]]></content:encoded>
</item>

      <item>
  <title>Your Real Resilience Number Is How Fast You Can Leave</title>
  <description>Exit time belongs on the board beside RTO and RPO. Most teams have never measured it.</description>
      <enclosure url="https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/1f9b9c8c-d5da-4136-9cfa-d823acb44ecf/exit-time-the-observability-resilience-metric_og_1200x630.png" length="79920" type="image/png"/>
  <link>https://www.masteringobservability.com/p/exit-time-the-observability-resilience-metric</link>
  <guid isPermaLink="true">https://www.masteringobservability.com/p/exit-time-the-observability-resilience-metric</guid>
  <pubDate>Sun, 28 Jun 2026 07:00:00 +0000</pubDate>
  <atom:published>2026-06-28T07:00:00Z</atom:published>
    <dc:creator>Allan Mann</dc:creator>
    <category><![CDATA[Vendor Lock In]]></category>
    <category><![CDATA[Observability]]></category>
    <category><![CDATA[Cost]]></category>
    <category><![CDATA[Resilience]]></category>
    <category><![CDATA[Blog]]></category>
  <content:encoded><![CDATA[
    <div class='beehiiv'><style>
  .bh__table, .bh__table_header, .bh__table_cell { border: 1px solid #C9DCDC; }
  .bh__table_cell { padding: 5px; background-color: #FFFFFF; }
  .bh__table_cell p { color: #16282D; font-family: 'DM Sans',Lato,Montserrat,sans-serif !important; overflow-wrap: break-word; }
  .bh__table_header { padding: 5px; background-color:#EAF6F3; }
  .bh__table_header p { color: #16282D; font-family:'DM Sans',Lato,Montserrat,sans-serif !important; overflow-wrap: break-word; }
</style><div class='beehiiv__body'><p class="paragraph" style="text-align:left;">We can all recite our resilience numbers. Four nines of uptime. A recovery time objective measured in minutes. A recovery point objective that the business signed off on in a meeting nobody enjoyed. These numbers live on dashboards, in board packs, in the slide that gets dusted off every audit. They are the language we use to say &quot;we are in control&quot;.</p><p class="paragraph" style="text-align:left;">There is one number we never put on the board. How long would it take to leave the platform that everything else is built on? Not the cost of leaving, although that is bad enough. The time. The elapsed wall-clock months between deciding to go and actually being gone.</p><p class="paragraph" style="text-align:left;">I started thinking about this properly after listening to Barry Pilling pull apart the Broadcom, VMware and Tesco licensing dispute on the Tech Leaders Podcast. The legal detail is not my lane, the case is unresolved, and I have no view to offer on who is right. What stayed with me was the shape of the problem underneath it. A customer builds an estate around a platform over many years, the terms of that platform change in a way the customer did not choose, and at that exact moment the most important question in the room is one nobody had measured: how fast can we get out.</p><h2 class="heading" style="text-align:left;" id="the-number-nobody-boards">The number nobody boards</h2><p class="paragraph" style="text-align:left;">We are precise about almost everything else. We will argue for an afternoon about whether a service needs three nines or four. We will model failover regions, run game days, and write recovery runbooks for outages that may never happen. All of that is good work, and I am not knocking it.</p><p class="paragraph" style="text-align:left;">But ask a leadership team how long it would take to migrate off their core observability platform, or their hypervisor, or their primary cloud, and you get a pause. Then an estimate that is really a hope. Then, if you push, an admission that nobody has actually tried to cost it, because trying to cost it would mean admitting how deep the roots have gone.</p><p class="paragraph" style="text-align:left;">Exit time is a resilience number. It belongs on the same board as your RTO and your RPO, because it measures the same thing they do: your exposure when something outside your control goes wrong. The difference is that an outage is an act of weather and a change of terms is an act of will. Someone on the other side of a contract can decide to create your worst day. If you cannot state your exit time, they already know it and you do not.</p><h2 class="heading" style="text-align:left;" id="the-day-the-terms-change">The day the terms change</h2><p class="paragraph" style="text-align:left;">Resilience planning has a quiet assumption baked into it. The threat is failure: a region falling over, a disk filling, a dependency timing out. So we design for failure, and we are good at it.</p><p class="paragraph" style="text-align:left;">The threat that exit time measures is different. It is not that the platform breaks. It is that the platform works exactly as designed, and the commercial reality around it shifts. An acquisition. A new pricing model. A licensing change that lands the week your renewal does. The platform is still up. Your problem is that you cannot afford to stay and cannot quickly leave.</p><p class="paragraph" style="text-align:left;">This is where the embedded customer discovers something uncomfortable. Every integration that made the platform more useful also made it harder to leave. The deeper the value, the longer the exit. And the lever that decides what happens next, the speed at which you could credibly walk, is held by the other party in the negotiation, not by you. You optimised for capability and quietly sold off your optionality to pay for it.</p><h2 class="heading" style="text-align:left;" id="observability-lockin-is-the-exit-ti">Observability lock-in is the exit time you cannot see</h2><p class="paragraph" style="text-align:left;">I will bring this home to the discipline I actually work in, because observability is a near-perfect case study in exit time growing while nobody watches.</p><p class="paragraph" style="text-align:left;">Think about how a mature observability estate gets built. You start with one vendor&#39;s agents because they are easy. Then your dashboards encode a year of hard-won operational knowledge in that vendor&#39;s query language. Then your alerting logic, the rules that decide who gets the call at three in the morning, lives in their system and nowhere else. Then your data sits in a proprietary format in their storage, priced by <a class="link" href="https://www.masteringobservability.com/p/the-observability-cost-conundrum?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=your-real-resilience-number-is-how-fast-you-can-leave" target="_blank" rel="noopener noreferrer nofollow">ingest and retention</a> in ways that are hard to predict and harder to escape. None of these decisions was wrong on the day you made it. Each one was the sensible, shippable choice. Together they are a set of roots that take years to grow and, if you ever need to, years to pull up.</p><p class="paragraph" style="text-align:left;">That is the part that should worry an operator. Exit time in observability is not a line item you can see. It is the sum of a hundred reasonable integrations, each of which added a little capability and a little lock-in, and you only find out the total on the day you try to leave. By then, it is a discovery, not a decision.</p><p class="paragraph" style="text-align:left;">The honest version of this is not &quot;never commit to a vendor&quot;. Commitment buys you real things, and a heterogeneous estate held together with string has its own failure modes. The honest version is: know the number. Treat the depth of your commitment as a measured quantity, not a vibe.</p><h2 class="heading" style="text-align:left;" id="designing-for-exit">Designing for exit</h2><p class="paragraph" style="text-align:left;">If exit time is a resilience number, then you manage it the way you manage the others. You measure it, you review it, and you do the engineering that keeps it inside a range you can live with. A few things make the difference, and none of them is exotic.</p><p class="paragraph" style="text-align:left;"><b>Own your data in an open format.</b> The single biggest driver of exit time is data you cannot take with you in a shape anyone else can read. <a class="link" href="https://www.masteringobservability.com/p/opentelemetry-collector-implementation-guide?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=your-real-resilience-number-is-how-fast-you-can-leave" target="_blank" rel="noopener noreferrer nofollow">Open formats and standards</a> exist precisely so that your telemetry is portable by default rather than by heroic project. If your signals are stored in a format only one vendor can read, your exit time is whatever that vendor decides it is.</p><p class="paragraph" style="text-align:left;"><b>Own your storage, or at least own the decision about where it lives.</b> The further your data sits from a proprietary, all-in-one bundle, the cheaper and faster it is to point a different tool at it. <a class="link" href="https://www.masteringobservability.com/p/building-a-comprehensive-cost-effective-observability-strategy?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=your-real-resilience-number-is-how-fast-you-can-leave" target="_blank" rel="noopener noreferrer nofollow">Separating where the data lives from what reads the data</a> is one of the highest-leverage architectural choices you can make for portability.</p><p class="paragraph" style="text-align:left;"><b>Test portability before you need it.</b> Resilience that has never been exercised is a guess. We do not trust a backup we have not restored. We should not trust an exit we have not rehearsed. A small, honest extraction test, can we get a month of one signal type out and into something else, tells you more about your real exit time than any architecture diagram.</p><p class="paragraph" style="text-align:left;"><b>Keep a live migration estimate.</b> Not a perfect one. A current, deliberately rough number, owned by someone, reviewed when your estate changes materially, and reported alongside your other resilience figures. The value is not precision. The value is that the number exists at all, and that someone is watching it move.</p><p class="paragraph" style="text-align:left;">The point of all this is not paranoia about vendors. It is the same instinct that makes us write a runbook for an outage we hope never comes. We are not predicting the bad day. We are making sure that if it arrives, our options are wider than the other party assumes.</p><h2 class="heading" style="text-align:left;" id="close">Close</h2><p class="paragraph" style="text-align:left;">You can probably state your uptime to four nines without checking. You can quote your RTO from memory. Those numbers are real, and they matter.</p><p class="paragraph" style="text-align:left;">Now try the other one. If a platform you depend on changed its terms tomorrow, how long would it take you to be gone? If you can answer that with a number you have actually tested, you are in a stronger position than most organisations I have seen. If you cannot, the silence is the answer, and it is not a comfortable one.</p><p class="paragraph" style="text-align:left;">Resilience was never only about surviving failure. It is about keeping your choices open when someone else would prefer them closed. Exit time is how you measure that. Put it on the board.</p><p class="paragraph" style="text-align:left;"><i>I write about this kind of thing, the numbers we should be tracking and the ones we actually do, every week in the Observability Digest. If the idea of exit time as a resilience number is new to you, the </i><a class="link" href="https://masteringobservability.com?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=your-real-resilience-number-is-how-fast-you-can-leave" target="_blank" rel="noopener noreferrer nofollow"><i>newsletter</i></a><i> is where I work these arguments out in the open. If you would rather start with the book, the </i><a class="link" href="https://www.masteringobservability.com/metrics-and-mayhem/free-chapter?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=your-real-resilience-number-is-how-fast-you-can-leave" target="_blank" rel="noopener noreferrer nofollow"><i>fourth chapter of Metrics and Mayhem is free</i></a><i>.</i></p><hr class="content_break"><div class="custom_html"><table width="100%" align="center" cellpadding="0" cellspacing="0" border="0" style="max-width:600px;margin:0 auto;"><tbody><tr><td style="background-color:#2F9E8D;height:4px;line-height:4px;font-size:4px;">&nbsp;</td></tr><tr><td style="background-color:#0D2127;padding:24px 24px 18px 24px;"><p style="margin:0 0 8px 0;font-family:'Space Mono','Courier New',monospace;font-size:12px;letter-spacing:2px;color:#74DDCD;text-transform:uppercase;">Work with me</p><p style="margin:0;font-family:'DM Sans',Helvetica,Arial,sans-serif;font-size:22px;line-height:1.3;color:#ffffff;font-weight:700;">An honest read on what your observability is actually doing.</p></td></tr><tr><td style="background-color:#ffffff;padding:20px 24px 24px 24px;font-family:'DM Sans',Helvetica,Arial,sans-serif;"><p style="margin:0 0 18px 0;font-size:16px;line-height:1.5;color:#1a1a1a;">If you lead observability in a regulated enterprise, I run a fixed-scope Observability Assessment for senior IT and engineering leaders. It ends in a written roadmap and a readout, not a sales deck.</p><table cellpadding="0" cellspacing="0" border="0"><tbody><tr><td style="background-color:#2F9E8D;border-radius:4px;"><a href="https://www.masteringobservability.com/advisory?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=your-real-resilience-number-is-how-fast-you-can-leave" style="display:inline-block;padding:12px 22px;font-family:'DM Sans',Helvetica,Arial,sans-serif;font-size:16px;font-weight:700;color:#0D2127;text-decoration:none;">See how it works →</a></td></tr></tbody></table><p style="margin:16px 0 0 0;font-size:14px;line-height:1.5;color:#44595b;">Not ready to talk? Start with a <a href="https://www.masteringobservability.com/metrics-and-mayhem/free-chapter?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=your-real-resilience-number-is-how-fast-you-can-leave" style="color:#0D2127;text-decoration:underline;">free chapter of Metrics & Mayhem</a>.</p></td></tr></tbody></table></div></div><div class='beehiiv__footer'><br class='beehiiv__footer__break'><hr class='beehiiv__footer__line'><a target="_blank" class="beehiiv__footer_link" style="text-align: center;" href="https://www.beehiiv.com/?utm_campaign=6b68ba54-ab97-4afc-9e54-eb16040e837a&utm_medium=post_rss&utm_source=mastering_observability">Powered by beehiiv</a></div></div>
  ]]></content:encoded>
</item>

      <item>
  <title>The Open-Source Observability Stack: A Field Guide</title>
  <description>A single map of the tools, and the honest catch on every piece.</description>
      <enclosure url="https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/89db2ef8-3fdf-47c3-8b5e-2fd78dfeeead/diagram_observability_stack_1200x680.png" length="129076" type="image/png"/>
  <link>https://www.masteringobservability.com/p/observability-stack</link>
  <guid isPermaLink="true">https://www.masteringobservability.com/p/observability-stack</guid>
  <pubDate>Wed, 01 Jul 2026 14:49:17 +0000</pubDate>
  <atom:published>2026-07-01T14:49:17Z</atom:published>
    <dc:creator>Allan Mann</dc:creator>
    <category><![CDATA[Newsletter]]></category>
    <category><![CDATA[Observability]]></category>
    <category><![CDATA[Byte Size]]></category>
  <content:encoded><![CDATA[
    <div class='beehiiv'><style>
  .bh__table, .bh__table_header, .bh__table_cell { border: 1px solid #C9DCDC; }
  .bh__table_cell { padding: 4px; background-color: #FFFFFF; }
  .bh__table_cell p { color: #2D2D2D; font-family: 'Space Grotesk',Helvetica,Arial,sans-serif !important; overflow-wrap: break-word; }
  .bh__table_header { padding: 6px; background-color:#EAF6F3; }
  .bh__table_header p { color: #222222; font-family:'Space Grotesk',Helvetica,Arial,sans-serif !important; overflow-wrap: break-word; }
</style><div class='beehiiv__body'><p class="paragraph" style="text-align:left;"><i>A single map of the open-source observability stack, and the honest catch on every piece.</i></p><p class="paragraph" style="text-align:left;">The open-source observability world looks crowded until you see the shape of it. Almost everything sorts into four jobs: collect the telemetry, ship it, store it by signal, and look at it. Here is the whole stack in one place, with a link to the plain-English, honest read on each tool.</p><div class="image"><a class="image__link" href="https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/b2cc70e5-8f83-4a2b-a2ca-5a92516bc79f/diagram_observability_stack_1200x680.png?t=1782916035&utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=the-open-source-observability-stack-a-field-guide" rel="noopener" target="_blank"><img alt="Open-source observability stack: a collector ships telemetry over OTLP to Prometheus for metrics, Loki for logs and Tempo or Jaeger for traces, all visualised in Grafana." class="image__image" style="" src="https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/b2cc70e5-8f83-4a2b-a2ca-5a92516bc79f/diagram_observability_stack_1200x680.png?t=1782916035"/></a><div class="image__source"><span class="image__source_text"><p>Every tool is mapped to the signal it owns and how they connect.</p></span></div></div><h2 class="heading" style="text-align:left;" id="the-pieces-and-the-catch-on-each">The pieces, and the catch on each</h2><ul><li><p class="paragraph" style="text-align:left;"><b>Collect and ship.</b> <a class="link" href="https://www.masteringobservability.com/p/what-is-opentelemetry?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=the-open-source-observability-stack-a-field-guide" target="_blank" rel="noopener noreferrer nofollow">OpenTelemetry</a> is the vendor-neutral standard that produces the data, though it is a framework, not a backend. <a class="link" href="https://www.masteringobservability.com/p/what-is-grafana-alloy?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=the-open-source-observability-stack-a-field-guide" target="_blank" rel="noopener noreferrer nofollow">Grafana Alloy</a> is a distribution of the Collector. <a class="link" href="https://www.masteringobservability.com/p/what-is-ebpf?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=the-open-source-observability-stack-a-field-guide" target="_blank" rel="noopener noreferrer nofollow">eBPF</a> is the kernel technology behind zero-code collection, and <a class="link" href="https://www.masteringobservability.com/p/what-is-odigos?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=the-open-source-observability-stack-a-field-guide" target="_blank" rel="noopener noreferrer nofollow">Odigos</a> puts it to work with no code changes.</p></li><li><p class="paragraph" style="text-align:left;"><b>Metrics:</b> <a class="link" href="https://www.masteringobservability.com/p/what-is-prometheus?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=the-open-source-observability-stack-a-field-guide" target="_blank" rel="noopener noreferrer nofollow">Prometheus</a>, the default, now OTLP-native. The catch: cardinality and the memory wall.</p></li><li><p class="paragraph" style="text-align:left;"><b>Logs:</b> <a class="link" href="https://www.masteringobservability.com/p/what-is-loki?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=the-open-source-observability-stack-a-field-guide" target="_blank" rel="noopener noreferrer nofollow">Loki</a>, cheap by design. The catch: weak at open-ended full-text search.</p></li><li><p class="paragraph" style="text-align:left;"><b>Traces:</b> <a class="link" href="https://www.masteringobservability.com/p/what-is-jaeger?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=the-open-source-observability-stack-a-field-guide" target="_blank" rel="noopener noreferrer nofollow">Jaeger</a>, the open, standard trace backend, and the concept behind it, <a class="link" href="https://www.masteringobservability.com/p/what-is-distributed-tracing?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=the-open-source-observability-stack-a-field-guide" target="_blank" rel="noopener noreferrer nofollow">distributed tracing</a>. The catch: the storage behind it is the real work.</p></li><li><p class="paragraph" style="text-align:left;"><b>See it:</b> <a class="link" href="https://www.masteringobservability.com/p/what-is-grafana?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=the-open-source-observability-stack-a-field-guide" target="_blank" rel="noopener noreferrer nofollow">Grafana</a>, one pane over all three. The catch: a green dashboard is not a healthy system.</p></li><li><p class="paragraph" style="text-align:left;"><b>Go deep:</b> the <a class="link" href="https://www.masteringobservability.com/p/opentelemetry-collector-implementation-guide?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=the-open-source-observability-stack-a-field-guide" target="_blank" rel="noopener noreferrer nofollow">OpenTelemetry Collector implementation guide</a> for the full how-to.</p></li></ul><h2 class="heading" style="text-align:left;" id="how-it-connects">How it connects</h2><p class="paragraph" style="text-align:left;">OpenTelemetry is the shared wire format, the language the pieces agreed to speak. A collector like Grafana Alloy gathers everything and routes each signal to its home. Prometheus 3.0 and Jaeger v2 now speak OTLP natively, so the seams keep shrinking. Pick the pieces you need, and hold on to one thing: the tools are the easy part. The honest work is in cardinality, sampling, storage cost, and whether anyone acts on what the dashboard shows.</p><hr class="content_break"><div class="custom_html"><table width="100%" align="center" cellpadding="0" cellspacing="0" border="0" style="max-width:600px;margin:0 auto;"><tbody><tr><td style="background-color:#2F9E8D;height:4px;line-height:4px;font-size:4px;">&nbsp;</td></tr><tr><td style="background-color:#0D2127;padding:24px 24px 18px 24px;"><p style="margin:0 0 8px 0;font-family:'Space Mono','Courier New',monospace;font-size:12px;letter-spacing:2px;color:#74DDCD;text-transform:uppercase;">Get the next one</p><p style="margin:0;font-family:'DM Sans',Helvetica,Arial,sans-serif;font-size:22px;line-height:1.3;color:#ffffff;font-weight:700;">One signal a week. No noise.</p></td></tr><tr><td style="background-color:#ffffff;padding:20px 24px 24px 24px;font-family:'DM Sans',Helvetica,Arial,sans-serif;"><p style="margin:0 0 18px 0;font-size:16px;line-height:1.5;color:#1a1a1a;">If this was useful, Metrics & Mayhem sends one short, practical piece like it to IT operations leaders most weeks. No fluff, no vendor noise.</p><table cellpadding="0" cellspacing="0" border="0"><tbody><tr><td style="background-color:#2F9E8D;border-radius:4px;"><a href="https://masteringobservability.com?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=the-open-source-observability-stack-a-field-guide" style="display:inline-block;padding:12px 22px;font-family:'DM Sans',Helvetica,Arial,sans-serif;font-size:16px;font-weight:700;color:#0D2127;text-decoration:none;">Join free →</a></td></tr></tbody></table><p style="margin:16px 0 0 0;font-size:14px;line-height:1.5;color:#44595b;">Prefer to start with the book? Read a <a href="https://www.masteringobservability.com/metrics-and-mayhem/free-chapter?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=the-open-source-observability-stack-a-field-guide" style="color:#0D2127;text-decoration:underline;">free chapter</a>.</p></td></tr></tbody></table></div></div><div class='beehiiv__footer'><br class='beehiiv__footer__break'><hr class='beehiiv__footer__line'><a target="_blank" class="beehiiv__footer_link" style="text-align: center;" href="https://www.beehiiv.com/?utm_campaign=6616b499-eee8-48e0-9e29-e14831d484da&utm_medium=post_rss&utm_source=mastering_observability">Powered by beehiiv</a></div></div>
  ]]></content:encoded>
</item>

      <item>
  <title>byte-size: What Is Prometheus?</title>
  <description>The open-source standard for metrics, in plain English, and where it stops.</description>
  <link>https://www.masteringobservability.com/p/what-is-prometheus</link>
  <guid isPermaLink="true">https://www.masteringobservability.com/p/what-is-prometheus</guid>
  <pubDate>Wed, 01 Jul 2026 14:48:24 +0000</pubDate>
  <atom:published>2026-07-01T14:48:24Z</atom:published>
    <dc:creator>Allan Mann</dc:creator>
    <category><![CDATA[Newsletter]]></category>
    <category><![CDATA[Observability]]></category>
    <category><![CDATA[Byte Size]]></category>
    <category><![CDATA[Prometheus]]></category>
  <content:encoded><![CDATA[
    <div class='beehiiv'><style>
  .bh__table, .bh__table_header, .bh__table_cell { border: 1px solid #C9DCDC; }
  .bh__table_cell { padding: 4px; background-color: #FFFFFF; }
  .bh__table_cell p { color: #2D2D2D; font-family: 'Space Grotesk',Helvetica,Arial,sans-serif !important; overflow-wrap: break-word; }
  .bh__table_header { padding: 6px; background-color:#EAF6F3; }
  .bh__table_header p { color: #222222; font-family:'Space Grotesk',Helvetica,Arial,sans-serif !important; overflow-wrap: break-word; }
</style><div class='beehiiv__body'><p class="paragraph" style="text-align:left;"><i>The open-source standard for metrics, in plain English, and where it stops.</i></p><p class="paragraph" style="text-align:left;">If you have ever looked at a graph of your system&#39;s CPU or request rate, there is a good chance Prometheus drew it. It is the open-source standard for metrics, and since version 3.0 it speaks <a class="link" href="https://www.masteringobservability.com/p/what-is-opentelemetry?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-prometheus" target="_blank" rel="noopener noreferrer nofollow">OpenTelemetry</a> natively. Here is what it is, and the honest limits.</p><div class="image"><a class="image__link" href="https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/b2cc70e5-8f83-4a2b-a2ca-5a92516bc79f/diagram_observability_stack_1200x680.png?t=1782916035&utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-prometheus" rel="noopener" target="_blank"><img alt="Open-source observability stack: a collector ships telemetry over OTLP to Prometheus for metrics, Loki for logs and Tempo or Jaeger for traces, all visualised in Grafana." class="image__image" style="" src="https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/b2cc70e5-8f83-4a2b-a2ca-5a92516bc79f/diagram_observability_stack_1200x680.png?t=1782916035"/></a><div class="image__source"><span class="image__source_text"><p>How the open-source observability stack fits together. Prometheus owns metrics here.</p></span></div></div><h2 class="heading" style="text-align:left;" id="what-is-prometheus">What is Prometheus?</h2><p class="paragraph" style="text-align:left;">Prometheus is an open-source monitoring system and time-series database built for metrics: numbers sampled over time, like request rate, error count or memory use. It scrapes those numbers from your services on a schedule, stores them efficiently, and lets you query them with its own language, PromQL. It came out of SoundCloud in 2012, and in 2018 became the second project after Kubernetes to graduate from the CNCF. That lineage is why it is effectively the default for cloud-native metrics.</p><h2 class="heading" style="text-align:left;" id="what-changed-in-30">What changed in 3.0</h2><p class="paragraph" style="text-align:left;">Worth knowing if your last look was a while ago. Prometheus 3.0, released in November 2024, added native OpenTelemetry ingestion, so applications instrumented with any OTel SDK can push metrics straight to Prometheus with no collector in between, plus UTF-8 support so metric names keep their dots, a faster remote-write protocol, and a new UI. Prometheus and OpenTelemetry now work together rather than around each other.</p><h2 class="heading" style="text-align:left;" id="what-it-does-well">What it does well</h2><ul><li><p class="paragraph" style="text-align:left;"><b>Purpose-built for metrics.</b> Efficient storage, fast queries, and PromQL, which is genuinely powerful once it clicks.</p></li><li><p class="paragraph" style="text-align:left;"><b>The ecosystem default.</b> Nearly every cloud-native tool exposes Prometheus-format metrics, and Grafana treats it as a first-class source.</p></li><li><p class="paragraph" style="text-align:left;"><b>Simple to run, hard to outgrow quickly.</b> A single binary gets you a long way before you need anything else.</p></li></ul><h2 class="heading" style="text-align:left;" id="from-the-trenches">From the trenches</h2><p class="paragraph" style="text-align:left;">Ask anyone running Prometheus at scale what actually bit them and you get one answer before all others: cardinality, and the memory wall it drives you into. Each active series costs roughly one to three kilobytes of memory, so a sixteen-gigabyte instance is comfortable at a few million series and struggling well beyond that. The labels that blow the budget are the tempting ones: user IDs, session tokens, raw URL paths, container IDs, commit SHAs. Every unique value is a brand new series.</p><p class="paragraph" style="text-align:left;">The quieter trap is churn. Pods that recreate on every deploy, autoscaling, and short-lived jobs all retire old label sets and mint new ones, so your series count climbs even when traffic is flat. And a sharp edge worth knowing: if you ship data to a long-term store with remote-write and that store goes unavailable, each endpoint can consume hundreds of megabytes of memory on its own. None of this is a reason to avoid Prometheus. It is a reason to treat labels as a budget, not a free-text field.</p><h2 class="heading" style="text-align:left;" id="the-honest-ledger">The honest ledger</h2><ul><li><p class="paragraph" style="text-align:left;"><b>Metrics only.</b> Prometheus does not do logs or traces. It is one of three pillars, not the whole picture.</p></li><li><p class="paragraph" style="text-align:left;"><b>Long-term storage and scale need help.</b> A single Prometheus is not built to hold years of data or span many clusters. That is what Thanos, Cortex and Mimir exist for.</p></li><li><p class="paragraph" style="text-align:left;"><b>Cardinality will bite you.</b> Every unique label combination is a new series, and high-cardinality labels like user IDs can blow up memory fast. Discipline here is the difference between a calm Prometheus and a struggling one.</p></li></ul><h2 class="heading" style="text-align:left;" id="so-is-prometheus-for-you">So, is Prometheus for you?</h2><p class="paragraph" style="text-align:left;">If you run modern infrastructure and you need metrics, the answer is almost certainly yes, and you are probably already using it through something else. Pair it with Grafana for dashboards, mind your cardinality, and reach for Thanos or Mimir when one instance is not enough.</p><hr class="content_break"><p class="paragraph" style="text-align:left;"><b>What is the highest-cardinality mistake you have made in Prometheus?</b> I collect these. Reply, or <a class="link" href="https://calendly.com/managing_observability/30min?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-prometheus" target="_blank" rel="noopener noreferrer nofollow">book a slot</a> and tell me what you found.</p><div class="custom_html"><table width="100%" align="center" cellpadding="0" cellspacing="0" border="0" style="max-width:600px;margin:0 auto;"><tbody><tr><td style="background-color:#2F9E8D;height:4px;line-height:4px;font-size:4px;">&nbsp;</td></tr><tr><td style="background-color:#0D2127;padding:24px 24px 18px 24px;"><p style="margin:0 0 8px 0;font-family:'Space Mono','Courier New',monospace;font-size:12px;letter-spacing:2px;color:#74DDCD;text-transform:uppercase;">Get the next one</p><p style="margin:0;font-family:'DM Sans',Helvetica,Arial,sans-serif;font-size:22px;line-height:1.3;color:#ffffff;font-weight:700;">One signal a week. No noise.</p></td></tr><tr><td style="background-color:#ffffff;padding:20px 24px 24px 24px;font-family:'DM Sans',Helvetica,Arial,sans-serif;"><p style="margin:0 0 18px 0;font-size:16px;line-height:1.5;color:#1a1a1a;">If this was useful, Metrics & Mayhem sends one short, practical piece like it to IT operations leaders most weeks. No fluff, no vendor noise.</p><table cellpadding="0" cellspacing="0" border="0"><tbody><tr><td style="background-color:#2F9E8D;border-radius:4px;"><a href="https://masteringobservability.com?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-prometheus" style="display:inline-block;padding:12px 22px;font-family:'DM Sans',Helvetica,Arial,sans-serif;font-size:16px;font-weight:700;color:#0D2127;text-decoration:none;">Join free →</a></td></tr></tbody></table><p style="margin:16px 0 0 0;font-size:14px;line-height:1.5;color:#44595b;">Prefer to start with the book? Read a <a href="https://www.masteringobservability.com/metrics-and-mayhem/free-chapter?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-prometheus" style="color:#0D2127;text-decoration:underline;">free chapter</a>.</p></td></tr></tbody></table></div><h3 class="heading" style="text-align:left;" id="related-from-the-bytesize-series">Related from the byte-size series</h3><ul><li><p class="paragraph" style="text-align:left;"><a class="link" href="https://www.masteringobservability.com/p/what-is-grafana?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-prometheus" target="_blank" rel="noopener noreferrer nofollow">byte-size: What Is Grafana?</a></p></li><li><p class="paragraph" style="text-align:left;"><a class="link" href="https://www.masteringobservability.com/p/what-is-opentelemetry?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-prometheus" target="_blank" rel="noopener noreferrer nofollow">byte-size: What Is OpenTelemetry?</a></p></li><li><p class="paragraph" style="text-align:left;"><a class="link" href="https://www.masteringobservability.com/p/what-is-grafana-alloy?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-prometheus" target="_blank" rel="noopener noreferrer nofollow">byte-size: What Is Grafana Alloy</a></p></li></ul><h3 class="heading" style="text-align:left;" id="sources-further-reading">Sources / further reading</h3><ul><li><p class="paragraph" style="text-align:left;">Prometheus project: <a class="link" href="https://prometheus.io/?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-prometheus" target="_blank" rel="noopener noreferrer nofollow">prometheus.io</a></p></li><li><p class="paragraph" style="text-align:left;">Announcing Prometheus 3.0: <a class="link" href="https://prometheus.io/blog/2024/11/14/prometheus-3-0/?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-prometheus" target="_blank" rel="noopener noreferrer nofollow">prometheus.io/blog</a></p></li><li><p class="paragraph" style="text-align:left;">High cardinality in Prometheus, and how to fix it (Last9): <a class="link" href="https://last9.io/blog/how-to-manage-high-cardinality-metrics-in-prometheus/?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-prometheus" target="_blank" rel="noopener noreferrer nofollow">last9.io</a></p></li><li><p class="paragraph" style="text-align:left;">Optimising Prometheus resource consumption (Palark): <a class="link" href="https://palark.com/blog/prometheus-resource-consumption-optimization/?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-prometheus" target="_blank" rel="noopener noreferrer nofollow">palark.com</a></p></li></ul></div><div class='beehiiv__footer'><br class='beehiiv__footer__break'><hr class='beehiiv__footer__line'><a target="_blank" class="beehiiv__footer_link" style="text-align: center;" href="https://www.beehiiv.com/?utm_campaign=02a75ad8-0d14-4575-9f60-c907fb6ac1b9&utm_medium=post_rss&utm_source=mastering_observability">Powered by beehiiv</a></div></div>
  ]]></content:encoded>
</item>

      <item>
  <title>byte-size: What Is Grafana?</title>
  <description>The dashboard layer that ties your observability together, and what it is not.</description>
  <link>https://www.masteringobservability.com/p/what-is-grafana</link>
  <guid isPermaLink="true">https://www.masteringobservability.com/p/what-is-grafana</guid>
  <pubDate>Wed, 01 Jul 2026 14:47:45 +0000</pubDate>
  <atom:published>2026-07-01T14:47:45Z</atom:published>
    <dc:creator>Allan Mann</dc:creator>
    <category><![CDATA[Newsletter]]></category>
    <category><![CDATA[Observability]]></category>
    <category><![CDATA[Grafana]]></category>
    <category><![CDATA[Byte Size]]></category>
  <content:encoded><![CDATA[
    <div class='beehiiv'><style>
  .bh__table, .bh__table_header, .bh__table_cell { border: 1px solid #C9DCDC; }
  .bh__table_cell { padding: 4px; background-color: #FFFFFF; }
  .bh__table_cell p { color: #2D2D2D; font-family: 'Space Grotesk',Helvetica,Arial,sans-serif !important; overflow-wrap: break-word; }
  .bh__table_header { padding: 6px; background-color:#EAF6F3; }
  .bh__table_header p { color: #222222; font-family:'Space Grotesk',Helvetica,Arial,sans-serif !important; overflow-wrap: break-word; }
</style><div class='beehiiv__body'><p class="paragraph" style="text-align:left;"><i>The dashboard layer that ties your observability together, and what it is not.</i></p><p class="paragraph" style="text-align:left;">Grafana is probably the most recognisable face in observability: the dashboards. But it is worth being precise about what Grafana is, and just as importantly, what it is not.</p><div class="image"><a class="image__link" href="https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/b2cc70e5-8f83-4a2b-a2ca-5a92516bc79f/diagram_observability_stack_1200x680.png?t=1782916035&utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-grafana" rel="noopener" target="_blank"><img alt="Open-source observability stack: a collector ships telemetry over OTLP to Prometheus for metrics, Loki for logs and Tempo or Jaeger for traces, all visualised in Grafana." class="image__image" style="" src="https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/b2cc70e5-8f83-4a2b-a2ca-5a92516bc79f/diagram_observability_stack_1200x680.png?t=1782916035"/></a><div class="image__source"><span class="image__source_text"><p>How the open-source observability stack fits together. Grafana is the pane of glass on the right.</p></span></div></div><h2 class="heading" style="text-align:left;" id="what-is-grafana">What is Grafana?</h2><p class="paragraph" style="text-align:left;">Grafana is an open-source platform for visualising and exploring data. It does not store your telemetry itself. It connects to the systems that do, <a class="link" href="https://www.masteringobservability.com/p/what-is-prometheus?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-grafana" target="_blank" rel="noopener noreferrer nofollow">Prometheus</a> for metrics, Loki for logs, Tempo or Jaeger for traces, and dozens of others, and turns them into dashboards, graphs and alerts in one place. It is made by Grafana Labs, who also build much of the surrounding stack, including Loki, Tempo, Mimir and the Alloy collector. Grafana 13 landed at GrafanaCON 2026.</p><h2 class="heading" style="text-align:left;" id="the-stack-it-anchors">The stack it anchors</h2><p class="paragraph" style="text-align:left;">Grafana is the front of what is often called the LGTM stack: Loki for logs, Grafana for visualisation, Tempo for traces, Mimir for metrics at scale. You do not have to use all of it, Grafana happily sits on top of Prometheus and other backends, but the pieces are designed to fit.</p><h2 class="heading" style="text-align:left;" id="what-it-does-well">What it does well</h2><ul><li><p class="paragraph" style="text-align:left;"><b>One pane over many sources.</b> Metrics, logs and traces from different systems, correlated in a single dashboard.</p></li><li><p class="paragraph" style="text-align:left;"><b>Data-source agnostic.</b> It does not lock you into one backend; you point it wherever your data lives.</p></li><li><p class="paragraph" style="text-align:left;"><b>Huge ecosystem.</b> Community dashboards, plugins and alerting mean you rarely start from scratch.</p></li></ul><h2 class="heading" style="text-align:left;" id="from-the-trenches">From the trenches</h2><p class="paragraph" style="text-align:left;">Ask a platform team about Grafana in anger and the word that comes back is sprawl. The easiest change anyone makes is to duplicate a dashboard, tweak one panel, and keep the original tags, and a year of that leaves you with hundreds of near-identical boards nobody quite trusts. The cost is not storage, it is time to the right answer during an incident, which is the worst moment to be hunting.</p><p class="paragraph" style="text-align:left;">The teams who stay sane treat dashboards as code, lean on template variables so one parameterised board replaces fifty copies, and alert on symptoms rather than causes. This is the practical face of a point worth repeating: a green dashboard is not a healthy system, it is a picture of what someone once chose to measure. Grafana will render whatever you point it at, faithfully. Whether it renders the thing that matters is on you.</p><h2 class="heading" style="text-align:left;" id="the-honest-ledger">The honest ledger</h2><ul><li><p class="paragraph" style="text-align:left;"><b>It is a window, not a warehouse.</b> Grafana shows your data; it does not store or collect it. You still need the backends underneath.</p></li><li><p class="paragraph" style="text-align:left;"><b>A green dashboard is not a healthy system.</b> Grafana will faithfully show you what you chose to measure, which is not always what matters. Visibility and understanding are different things.</p></li><li><p class="paragraph" style="text-align:left;"><b>Dashboard sprawl is real.</b> Without discipline you end up with hundreds of dashboards nobody trusts. Curate ruthlessly.</p></li></ul><h2 class="heading" style="text-align:left;" id="so-is-grafana-for-you">So, is Grafana for you?</h2><p class="paragraph" style="text-align:left;">If you have observability data in more than one place, Grafana is close to essential as the layer that brings it together. Just remember it is the viewing glass. The value is in the signals you choose to put behind it, and in whether anyone acts on them.</p><hr class="content_break"><p class="paragraph" style="text-align:left;"><b>How many dashboards does your team actually trust?</b> The honest number is usually small. Reply, or <a class="link" href="https://calendly.com/managing_observability/30min?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-grafana" target="_blank" rel="noopener noreferrer nofollow">book a slot</a> and tell me what you found.</p><div class="custom_html"><table width="100%" align="center" cellpadding="0" cellspacing="0" border="0" style="max-width:600px;margin:0 auto;"><tbody><tr><td style="background-color:#2F9E8D;height:4px;line-height:4px;font-size:4px;">&nbsp;</td></tr><tr><td style="background-color:#0D2127;padding:24px 24px 18px 24px;"><p style="margin:0 0 8px 0;font-family:'Space Mono','Courier New',monospace;font-size:12px;letter-spacing:2px;color:#74DDCD;text-transform:uppercase;">Get the next one</p><p style="margin:0;font-family:'DM Sans',Helvetica,Arial,sans-serif;font-size:22px;line-height:1.3;color:#ffffff;font-weight:700;">One signal a week. No noise.</p></td></tr><tr><td style="background-color:#ffffff;padding:20px 24px 24px 24px;font-family:'DM Sans',Helvetica,Arial,sans-serif;"><p style="margin:0 0 18px 0;font-size:16px;line-height:1.5;color:#1a1a1a;">If this was useful, Metrics & Mayhem sends one short, practical piece like it to IT operations leaders most weeks. No fluff, no vendor noise.</p><table cellpadding="0" cellspacing="0" border="0"><tbody><tr><td style="background-color:#2F9E8D;border-radius:4px;"><a href="https://masteringobservability.com?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-grafana" style="display:inline-block;padding:12px 22px;font-family:'DM Sans',Helvetica,Arial,sans-serif;font-size:16px;font-weight:700;color:#0D2127;text-decoration:none;">Join free →</a></td></tr></tbody></table><p style="margin:16px 0 0 0;font-size:14px;line-height:1.5;color:#44595b;">Prefer to start with the book? Read a <a href="https://www.masteringobservability.com/metrics-and-mayhem/free-chapter?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-grafana" style="color:#0D2127;text-decoration:underline;">free chapter</a>.</p></td></tr></tbody></table></div><h3 class="heading" style="text-align:left;" id="related-from-the-bytesize-series">Related from the byte-size series</h3><ul><li><p class="paragraph" style="text-align:left;"><a class="link" href="https://www.masteringobservability.com/p/what-is-prometheus?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-grafana" target="_blank" rel="noopener noreferrer nofollow">byte-size: What Is Prometheus?</a></p></li><li><p class="paragraph" style="text-align:left;"><a class="link" href="https://www.masteringobservability.com/p/what-is-loki?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-grafana" target="_blank" rel="noopener noreferrer nofollow">byte-size: What Is Loki?</a></p></li><li><p class="paragraph" style="text-align:left;"><a class="link" href="https://www.masteringobservability.com/p/what-is-grafana-alloy?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-grafana" target="_blank" rel="noopener noreferrer nofollow">byte-size: What Is Grafana Alloy</a></p></li></ul><h3 class="heading" style="text-align:left;" id="sources-further-reading">Sources / further reading</h3><ul><li><p class="paragraph" style="text-align:left;">Grafana project: <a class="link" href="https://grafana.com/oss/grafana/?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-grafana" target="_blank" rel="noopener noreferrer nofollow">grafana.com</a></p></li><li><p class="paragraph" style="text-align:left;">Grafana 13 at GrafanaCON 2026: <a class="link" href="https://grafana.com/?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-grafana" target="_blank" rel="noopener noreferrer nofollow">grafana.com</a></p></li><li><p class="paragraph" style="text-align:left;">Grafana dashboard best practices, on avoiding sprawl: <a class="link" href="https://grafana.com/docs/grafana/latest/visualizations/dashboards/build-dashboards/best-practices/?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-grafana" target="_blank" rel="noopener noreferrer nofollow">grafana.com/docs</a></p></li></ul></div><div class='beehiiv__footer'><br class='beehiiv__footer__break'><hr class='beehiiv__footer__line'><a target="_blank" class="beehiiv__footer_link" style="text-align: center;" href="https://www.beehiiv.com/?utm_campaign=3b3175e9-7833-4322-b230-982fcffa792c&utm_medium=post_rss&utm_source=mastering_observability">Powered by beehiiv</a></div></div>
  ]]></content:encoded>
</item>

      <item>
  <title>byte-size: What Is Loki?</title>
  <description>Grafana&#39;s logs backend, in plain English, and the trade-off at its heart.</description>
  <link>https://www.masteringobservability.com/p/what-is-loki</link>
  <guid isPermaLink="true">https://www.masteringobservability.com/p/what-is-loki</guid>
  <pubDate>Wed, 01 Jul 2026 14:47:10 +0000</pubDate>
  <atom:published>2026-07-01T14:47:10Z</atom:published>
    <dc:creator>Allan Mann</dc:creator>
    <category><![CDATA[Newsletter]]></category>
    <category><![CDATA[Observability]]></category>
    <category><![CDATA[Byte Size]]></category>
    <category><![CDATA[Loki]]></category>
  <content:encoded><![CDATA[
    <div class='beehiiv'><style>
  .bh__table, .bh__table_header, .bh__table_cell { border: 1px solid #C9DCDC; }
  .bh__table_cell { padding: 4px; background-color: #FFFFFF; }
  .bh__table_cell p { color: #2D2D2D; font-family: 'Space Grotesk',Helvetica,Arial,sans-serif !important; overflow-wrap: break-word; }
  .bh__table_header { padding: 6px; background-color:#EAF6F3; }
  .bh__table_header p { color: #222222; font-family:'Space Grotesk',Helvetica,Arial,sans-serif !important; overflow-wrap: break-word; }
</style><div class='beehiiv__body'><p class="paragraph" style="text-align:left;"><i>Grafana&#39;s logs backend, in plain English, and the trade-off at its heart.</i></p><p class="paragraph" style="text-align:left;">Loki is Grafana Labs&#39; answer to log storage, and its pitch is a clever one: be like Prometheus, but for logs. That design choice is both its biggest strength and the thing to understand before you adopt it.</p><div class="image"><a class="image__link" href="https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/b2cc70e5-8f83-4a2b-a2ca-5a92516bc79f/diagram_observability_stack_1200x680.png?t=1782916035&utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-loki" rel="noopener" target="_blank"><img alt="Open-source observability stack: a collector ships telemetry over OTLP to Prometheus for metrics, Loki for logs and Tempo or Jaeger for traces, all visualised in Grafana." class="image__image" style="" src="https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/b2cc70e5-8f83-4a2b-a2ca-5a92516bc79f/diagram_observability_stack_1200x680.png?t=1782916035"/></a><div class="image__source"><span class="image__source_text"><p>How the open-source observability stack fits together. Loki owns logs here.</p></span></div></div><h2 class="heading" style="text-align:left;" id="what-is-loki">What is Loki?</h2><p class="paragraph" style="text-align:left;">Loki is an open-source log aggregation system built to be cost-efficient at scale. Instead of indexing the full text of every log line, the traditional and expensive approach, it indexes only a small set of labels, like the service or environment, and stores the log content itself cheaply. You query it with LogQL, a language deliberately close to <a class="link" href="https://www.masteringobservability.com/p/what-is-prometheus?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-loki" target="_blank" rel="noopener noreferrer nofollow">Prometheus</a>&#39;s PromQL, and you read the results in <a class="link" href="https://www.masteringobservability.com/p/what-is-grafana?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-loki" target="_blank" rel="noopener noreferrer nofollow">Grafana</a>. The current line is Loki 3.x.</p><h2 class="heading" style="text-align:left;" id="the-tradeoff-at-its-heart">The trade-off at its heart</h2><p class="paragraph" style="text-align:left;">The label-only index is why Loki is cheap to run, and also why broad, ad-hoc text searches across huge volumes can be slow: there is no full-text index to lean on. Grafana Labs has been closing that gap, rearchitecting Loki around Kafka for durability and acquiring Logline to speed up needle-in-a-haystack searches. Worth knowing which version you are on.</p><h2 class="heading" style="text-align:left;" id="what-it-does-well">What it does well</h2><ul><li><p class="paragraph" style="text-align:left;"><b>Cheap at scale.</b> The index-light design keeps storage and cost down where full-text systems balloon.</p></li><li><p class="paragraph" style="text-align:left;"><b>Native Grafana and Prometheus fit.</b> Same labels, same mental model, one dashboard for logs and metrics side by side.</p></li><li><p class="paragraph" style="text-align:left;"><b>Simple operational story.</b> Fewer moving parts than a full search cluster for the common case.</p></li></ul><h2 class="heading" style="text-align:left;" id="from-the-trenches">From the trenches</h2><p class="paragraph" style="text-align:left;">The thing that makes Loki cheap is the same thing people trip over. Because it indexes labels rather than the full text of every line, a broad question like &quot;find this error message across every service&quot; cannot lean on an inverted index; it scans the raw log chunks instead. Scoped by a label and a tight time range, that is fast. Unscoped across a large estate, practitioners report it running in minutes or hours where Elasticsearch answers in seconds.</p><p class="paragraph" style="text-align:left;">The cost surprise follows from the same design. If your chunks live in cheap object storage with paid retrieval, a wide search can pull a lot of data back and the bill notices. And the operational story is not as small as the pitch suggests: a real Loki is several interconnected services with non-trivial configuration. None of this makes Loki the wrong choice, it makes it a choice with a shape. It is brilliant for label-scoped operational debugging and weak for open-ended forensic search. Teams that need the latter often keep a search-engine backend alongside, or look at lighter alternatives like VictoriaLogs.</p><h2 class="heading" style="text-align:left;" id="the-honest-ledger">The honest ledger</h2><ul><li><p class="paragraph" style="text-align:left;"><b>It is not a search engine.</b> If your primary need is fast, arbitrary full-text search across everything, a system built for that may serve you better, though Loki is catching up.</p></li><li><p class="paragraph" style="text-align:left;"><b>Labels are a discipline, not a free lunch.</b> High-cardinality labels hurt Loki just as they hurt Prometheus. Design them with care.</p></li><li><p class="paragraph" style="text-align:left;"><b>Best inside the Grafana world.</b> You can use it standalone, but the value shows up next to Grafana and Prometheus.</p></li></ul><h2 class="heading" style="text-align:left;" id="so-is-loki-for-you">So, is Loki for you?</h2><p class="paragraph" style="text-align:left;">If you want affordable logs that sit naturally alongside your metrics in Grafana, and your search patterns lean on labels rather than raw full-text, Loki is a strong, economical choice. If deep full-text search is your daily driver, weigh it against a purpose-built search system, and check where Loki&#39;s newer search features have landed.</p><hr class="content_break"><p class="paragraph" style="text-align:left;"><b>What does your log bill look like, and is the spend buying you answers?</b> Reply, or <a class="link" href="https://calendly.com/managing_observability/30min?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-loki" target="_blank" rel="noopener noreferrer nofollow">book a slot</a> and tell me what you found.</p><div class="custom_html"><table width="100%" align="center" cellpadding="0" cellspacing="0" border="0" style="max-width:600px;margin:0 auto;"><tbody><tr><td style="background-color:#2F9E8D;height:4px;line-height:4px;font-size:4px;">&nbsp;</td></tr><tr><td style="background-color:#0D2127;padding:24px 24px 18px 24px;"><p style="margin:0 0 8px 0;font-family:'Space Mono','Courier New',monospace;font-size:12px;letter-spacing:2px;color:#74DDCD;text-transform:uppercase;">Get the next one</p><p style="margin:0;font-family:'DM Sans',Helvetica,Arial,sans-serif;font-size:22px;line-height:1.3;color:#ffffff;font-weight:700;">One signal a week. No noise.</p></td></tr><tr><td style="background-color:#ffffff;padding:20px 24px 24px 24px;font-family:'DM Sans',Helvetica,Arial,sans-serif;"><p style="margin:0 0 18px 0;font-size:16px;line-height:1.5;color:#1a1a1a;">If this was useful, Metrics & Mayhem sends one short, practical piece like it to IT operations leaders most weeks. No fluff, no vendor noise.</p><table cellpadding="0" cellspacing="0" border="0"><tbody><tr><td style="background-color:#2F9E8D;border-radius:4px;"><a href="https://masteringobservability.com?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-loki" style="display:inline-block;padding:12px 22px;font-family:'DM Sans',Helvetica,Arial,sans-serif;font-size:16px;font-weight:700;color:#0D2127;text-decoration:none;">Join free →</a></td></tr></tbody></table><p style="margin:16px 0 0 0;font-size:14px;line-height:1.5;color:#44595b;">Prefer to start with the book? Read a <a href="https://www.masteringobservability.com/metrics-and-mayhem/free-chapter?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-loki" style="color:#0D2127;text-decoration:underline;">free chapter</a>.</p></td></tr></tbody></table></div><h3 class="heading" style="text-align:left;" id="related-from-the-bytesize-series">Related from the byte-size series</h3><ul><li><p class="paragraph" style="text-align:left;"><a class="link" href="https://www.masteringobservability.com/p/what-is-grafana?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-loki" target="_blank" rel="noopener noreferrer nofollow">byte-size: What Is Grafana?</a></p></li><li><p class="paragraph" style="text-align:left;"><a class="link" href="https://www.masteringobservability.com/p/what-is-prometheus?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-loki" target="_blank" rel="noopener noreferrer nofollow">byte-size: What Is Prometheus?</a></p></li><li><p class="paragraph" style="text-align:left;"><a class="link" href="https://www.masteringobservability.com/p/what-is-opentelemetry?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-loki" target="_blank" rel="noopener noreferrer nofollow">byte-size: What Is OpenTelemetry?</a></p></li></ul><h3 class="heading" style="text-align:left;" id="sources-further-reading">Sources / further reading</h3><ul><li><p class="paragraph" style="text-align:left;">Grafana Loki project: <a class="link" href="https://grafana.com/oss/loki/?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-loki" target="_blank" rel="noopener noreferrer nofollow">grafana.com</a></p></li><li><p class="paragraph" style="text-align:left;">Loki documentation: <a class="link" href="https://grafana.com/docs/loki/latest/?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-loki" target="_blank" rel="noopener noreferrer nofollow">grafana.com/docs/loki</a></p></li><li><p class="paragraph" style="text-align:left;">Loki vs Elasticsearch, the trade-offs (SigNoz): <a class="link" href="https://signoz.io/blog/loki-vs-elasticsearch/?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-loki" target="_blank" rel="noopener noreferrer nofollow">signoz.io</a></p></li><li><p class="paragraph" style="text-align:left;">Why VictoriaLogs is pitched as a lighter alternative (ITNEXT): <a class="link" href="https://itnext.io/why-victorialogs-is-a-better-alternative-to-grafana-loki-7e941567c4d5?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-loki" target="_blank" rel="noopener noreferrer nofollow">itnext.io</a></p></li></ul></div><div class='beehiiv__footer'><br class='beehiiv__footer__break'><hr class='beehiiv__footer__line'><a target="_blank" class="beehiiv__footer_link" style="text-align: center;" href="https://www.beehiiv.com/?utm_campaign=096e2cf3-187d-480f-813a-2d0d33319f6c&utm_medium=post_rss&utm_source=mastering_observability">Powered by beehiiv</a></div></div>
  ]]></content:encoded>
</item>

      <item>
  <title>byte-size: What Is Jaeger?</title>
  <description>The tracing backend behind your traces, in plain English, and where OpenTelemetry fits.</description>
  <link>https://www.masteringobservability.com/p/what-is-jaeger</link>
  <guid isPermaLink="true">https://www.masteringobservability.com/p/what-is-jaeger</guid>
  <pubDate>Wed, 01 Jul 2026 14:46:27 +0000</pubDate>
  <atom:published>2026-07-01T14:46:27Z</atom:published>
    <dc:creator>Allan Mann</dc:creator>
    <category><![CDATA[Newsletter]]></category>
    <category><![CDATA[Observability]]></category>
    <category><![CDATA[Byte Size]]></category>
    <category><![CDATA[Distributed Tracing]]></category>
    <category><![CDATA[Jaeger]]></category>
  <content:encoded><![CDATA[
    <div class='beehiiv'><style>
  .bh__table, .bh__table_header, .bh__table_cell { border: 1px solid #C9DCDC; }
  .bh__table_cell { padding: 4px; background-color: #FFFFFF; }
  .bh__table_cell p { color: #2D2D2D; font-family: 'Space Grotesk',Helvetica,Arial,sans-serif !important; overflow-wrap: break-word; }
  .bh__table_header { padding: 6px; background-color:#EAF6F3; }
  .bh__table_header p { color: #222222; font-family:'Space Grotesk',Helvetica,Arial,sans-serif !important; overflow-wrap: break-word; }
</style><div class='beehiiv__body'><p class="paragraph" style="text-align:left;"><i>The tracing backend behind your traces, in plain English, and where OpenTelemetry fits.</i></p><p class="paragraph" style="text-align:left;">If <a class="link" href="https://www.masteringobservability.com/p/what-is-distributed-tracing?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-jaeger" target="_blank" rel="noopener noreferrer nofollow">distributed tracing</a> tells you where a request spent its time, Jaeger is one of the places those traces go to live. It is a tracing backend, and since v2 it is built directly on OpenTelemetry.</p><div class="image"><a class="image__link" href="https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/b2cc70e5-8f83-4a2b-a2ca-5a92516bc79f/diagram_observability_stack_1200x680.png?t=1782916035&utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-jaeger" rel="noopener" target="_blank"><img alt="Open-source observability stack: a collector ships telemetry over OTLP to Prometheus for metrics, Loki for logs and Tempo or Jaeger for traces, all visualised in Grafana." class="image__image" style="" src="https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/b2cc70e5-8f83-4a2b-a2ca-5a92516bc79f/diagram_observability_stack_1200x680.png?t=1782916035"/></a><div class="image__source"><span class="image__source_text"><p>How the open-source observability stack fits together. Jaeger owns traces here.</p></span></div></div><h2 class="heading" style="text-align:left;" id="what-is-jaeger">What is Jaeger?</h2><p class="paragraph" style="text-align:left;">Jaeger is an open-source, end-to-end distributed tracing platform: it receives traces, stores them, and gives you a UI to search and visualise a request&#39;s journey across services. It came out of Uber, and it was one of the very first projects to graduate from the CNCF. It is a backend, not instrumentation: you produce traces with <a class="link" href="https://www.masteringobservability.com/p/what-is-opentelemetry?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-jaeger" target="_blank" rel="noopener noreferrer nofollow">OpenTelemetry</a>, and Jaeger is where you send them to query and view.</p><h2 class="heading" style="text-align:left;" id="jaeger-v-2-and-open-telemetry">Jaeger v2 and OpenTelemetry</h2><p class="paragraph" style="text-align:left;">This is the big recent change. Jaeger v2, released in November 2024, is built on the OpenTelemetry Collector itself: one streamlined binary that speaks the OpenTelemetry Protocol end to end, with no translation layer. Jaeger v1 is being retired, with the last v1 release in December 2025 and full deprecation in January 2026. If you are starting today, you are starting on OTel.</p><h2 class="heading" style="text-align:left;" id="what-it-does-well">What it does well</h2><ul><li><p class="paragraph" style="text-align:left;"><b>Purpose-built trace views.</b> Timeline and dependency views that make a slow, multi-service request legible.</p></li><li><p class="paragraph" style="text-align:left;"><b>Open and standard.</b> OTLP-native and CNCF-graduated, so it fits the wider ecosystem rather than fencing you in.</p></li><li><p class="paragraph" style="text-align:left;"><b>Focused.</b> It does one job, tracing, and does it well, which keeps it simple to stand up.</p></li></ul><h2 class="heading" style="text-align:left;" id="from-the-trenches">From the trenches</h2><p class="paragraph" style="text-align:left;">Standing up Jaeger is the easy part. Running the storage behind it is where the real work lives, because Jaeger itself does not keep your traces; it hands them to Cassandra or Elasticsearch, and each comes with a bill. Cassandra is quick for pulling a trace by its ID but weak at search, and Jaeger writes extra index entries for service names, operations and every tag, so write amplification is real and storage climbs at roughly a gigabyte or more per million spans. Elasticsearch searches far better but is resource-hungry, and teams have hit hard ceilings: one reported ingesting only about a tenth of the ten terabytes of traces a day their applications produced, for the money they were spending.</p><p class="paragraph" style="text-align:left;">Underneath all of it sits sampling. Keep every trace and the storage problem above gets worse fast; sample too hard and you risk dropping the rare failure you most wanted to see. The honest framing is that Jaeger is a fine, standard place to view traces, and the engineering effort is in the backend and the sampling policy, not the tool.</p><h2 class="heading" style="text-align:left;" id="the-honest-ledger">The honest ledger</h2><ul><li><p class="paragraph" style="text-align:left;"><b>It is a backend, not the whole story.</b> Jaeger stores and shows traces. You still instrument with OpenTelemetry, and you still need a storage backend such as Cassandra or Elasticsearch behind it.</p></li><li><p class="paragraph" style="text-align:left;"><b>Traces alone are not observability.</b> You will correlate with metrics and logs to find root cause. Jaeger is the trace piece, not the full picture.</p></li><li><p class="paragraph" style="text-align:left;"><b>Sampling decisions still apply.</b> As with all tracing, you trade completeness against cost. Decide deliberately.</p></li></ul><h2 class="heading" style="text-align:left;" id="so-is-jaeger-for-you">So, is Jaeger for you?</h2><p class="paragraph" style="text-align:left;">If you have adopted OpenTelemetry for tracing and you want an open, standard place to store and explore those traces, Jaeger is a natural, well-supported choice, and v2 makes the OTel fit seamless. If you would rather not run a backend at all, a managed tracing service takes that on for you.</p><hr class="content_break"><p class="paragraph" style="text-align:left;"><b>Where does your tracing chain break today?</b> Usually it is one un-instrumented service. Reply, or <a class="link" href="https://calendly.com/managing_observability/30min?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-jaeger" target="_blank" rel="noopener noreferrer nofollow">book a slot</a> and tell me what you found.</p><div class="custom_html"><table width="100%" align="center" cellpadding="0" cellspacing="0" border="0" style="max-width:600px;margin:0 auto;"><tbody><tr><td style="background-color:#2F9E8D;height:4px;line-height:4px;font-size:4px;">&nbsp;</td></tr><tr><td style="background-color:#0D2127;padding:24px 24px 18px 24px;"><p style="margin:0 0 8px 0;font-family:'Space Mono','Courier New',monospace;font-size:12px;letter-spacing:2px;color:#74DDCD;text-transform:uppercase;">Get the next one</p><p style="margin:0;font-family:'DM Sans',Helvetica,Arial,sans-serif;font-size:22px;line-height:1.3;color:#ffffff;font-weight:700;">One signal a week. No noise.</p></td></tr><tr><td style="background-color:#ffffff;padding:20px 24px 24px 24px;font-family:'DM Sans',Helvetica,Arial,sans-serif;"><p style="margin:0 0 18px 0;font-size:16px;line-height:1.5;color:#1a1a1a;">If this was useful, Metrics & Mayhem sends one short, practical piece like it to IT operations leaders most weeks. No fluff, no vendor noise.</p><table cellpadding="0" cellspacing="0" border="0"><tbody><tr><td style="background-color:#2F9E8D;border-radius:4px;"><a href="https://masteringobservability.com?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-jaeger" style="display:inline-block;padding:12px 22px;font-family:'DM Sans',Helvetica,Arial,sans-serif;font-size:16px;font-weight:700;color:#0D2127;text-decoration:none;">Join free →</a></td></tr></tbody></table><p style="margin:16px 0 0 0;font-size:14px;line-height:1.5;color:#44595b;">Prefer to start with the book? Read a <a href="https://www.masteringobservability.com/metrics-and-mayhem/free-chapter?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-jaeger" style="color:#0D2127;text-decoration:underline;">free chapter</a>.</p></td></tr></tbody></table></div><h3 class="heading" style="text-align:left;" id="related-from-the-bytesize-series">Related from the byte-size series</h3><ul><li><p class="paragraph" style="text-align:left;"><a class="link" href="https://www.masteringobservability.com/p/what-is-distributed-tracing?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-jaeger" target="_blank" rel="noopener noreferrer nofollow">byte-size: What Is Distributed Tracing?</a></p></li><li><p class="paragraph" style="text-align:left;"><a class="link" href="https://www.masteringobservability.com/p/what-is-opentelemetry?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-jaeger" target="_blank" rel="noopener noreferrer nofollow">byte-size: What Is OpenTelemetry?</a></p></li><li><p class="paragraph" style="text-align:left;"><a class="link" href="https://www.masteringobservability.com/p/opentelemetry-collector-implementation-guide?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-jaeger" target="_blank" rel="noopener noreferrer nofollow">OpenTelemetry Collector Implementation Guide</a></p></li></ul><h3 class="heading" style="text-align:left;" id="sources-further-reading">Sources / further reading</h3><ul><li><p class="paragraph" style="text-align:left;">Jaeger project: <a class="link" href="https://www.jaegertracing.io/?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-jaeger" target="_blank" rel="noopener noreferrer nofollow">jaegertracing.io</a></p></li><li><p class="paragraph" style="text-align:left;">Jaeger v2 released, OpenTelemetry in the core (CNCF): <a class="link" href="https://www.cncf.io/blog/2024/11/12/jaeger-v2-released-opentelemetry-in-the-core/?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-jaeger" target="_blank" rel="noopener noreferrer nofollow">cncf.io</a></p></li><li><p class="paragraph" style="text-align:left;">What database does Jaeger use, Cassandra vs Elasticsearch (SigNoz): <a class="link" href="https://signoz.io/guides/what-database-does-jaeger-use/?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-jaeger" target="_blank" rel="noopener noreferrer nofollow">signoz.io</a></p></li><li><p class="paragraph" style="text-align:left;">Jaeger persistent storage with Elasticsearch, Cassandra, Kafka (CNCF): <a class="link" href="https://www.cncf.io/blog/2021/03/12/jaeger-persistent-storage-with-elasticsearch-cassandra-kafka/?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-jaeger" target="_blank" rel="noopener noreferrer nofollow">cncf.io</a></p></li></ul></div><div class='beehiiv__footer'><br class='beehiiv__footer__break'><hr class='beehiiv__footer__line'><a target="_blank" class="beehiiv__footer_link" style="text-align: center;" href="https://www.beehiiv.com/?utm_campaign=8be26104-263f-40f2-a992-66c1d1f2bea4&utm_medium=post_rss&utm_source=mastering_observability">Powered by beehiiv</a></div></div>
  ]]></content:encoded>
</item>

      <item>
  <title>Observability Digest, June 2026</title>
  <description>Operational judgement you can act on, whether you build it or lead it, plus the few signals worth your time. Starting with the deal that merged security and observability.</description>
      <enclosure url="https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/0c2ae919-9408-4518-8b69-1edf3abc1937/thumbnail_secops_observability.png" length="308137" type="image/png"/>
  <link>https://www.masteringobservability.com/p/observability-digest-june-2026</link>
  <guid isPermaLink="true">https://www.masteringobservability.com/p/observability-digest-june-2026</guid>
  <pubDate>Wed, 01 Jul 2026 06:00:00 +0000</pubDate>
  <atom:published>2026-07-01T06:00:00Z</atom:published>
    <dc:creator>Allan Mann</dc:creator>
    <category><![CDATA[Newsletter]]></category>
    <category><![CDATA[Observability]]></category>
    <category><![CDATA[Signal Drop]]></category>
    <category><![CDATA[Security]]></category>
    <category><![CDATA[Monthly Digest]]></category>
  <content:encoded><![CDATA[
    <div class='beehiiv'><style>
  .bh__table, .bh__table_header, .bh__table_cell { border: 1px solid #C9DCDC; }
  .bh__table_cell { padding: 5px; background-color: #FFFFFF; }
  .bh__table_cell p { color: #2D2D2D; font-family: 'Helvetica',Arial,sans-serif !important; overflow-wrap: break-word; }
  .bh__table_header { padding: 5px; background-color:#EAF6F3; }
  .bh__table_header p { color: #2A2A2A; font-family:'Trebuchet MS','Lucida Grande',Tahoma,sans-serif !important; overflow-wrap: break-word; }
</style><div class='beehiiv__body'><p class="paragraph" style="text-align:left;">You signed up for the Observability Digest, and it went quiet for a stretch. It is back now, on a clearer footing.</p><p class="paragraph" style="text-align:left;">The Signal Drops you have been getting in between, on the people-and-leadership side of running observability under pressure, are deliberate, and they stay. What was missing was the monthly round-up you actually subscribed to, and a clear promise for it. So here it is: <b>observability you can act on, whether you build it or lead it, plus the few signals worth your time.</b> My read on what actually mattered is the spine, technical where it needs to be; a short list of outside reads is the supporting act. Monthly for now, fortnightly once the rhythm holds.</p><p class="paragraph" style="text-align:left;">One ask before you read on. Tell me which cut you want and I will sort the rest:</p><ul><li><p class="paragraph" style="text-align:left;">Reply <b>TECH</b> for the hands-on, practitioner cut.</p></li><li><p class="paragraph" style="text-align:left;">Reply <b>LEADERSHIP</b> for the strategy cut.</p></li><li><p class="paragraph" style="text-align:left;">Do nothing to keep the full issue.</p></li></ul><p class="paragraph" style="text-align:left;">One word. Change it whenever.</p><h2 class="heading" style="text-align:left;" id="this-months-deep-dive">This month&#39;s deep dive</h2><p class="paragraph" style="text-align:left;"><b>Security just bought observability.</b> A $3.35bn deal closed, two more followed, and the vendors have now decided security and observability are one market. The teams have not: roughly 80% already share the tools, only about 45% agree on how to use them. You can buy the converged platform. You cannot buy the converged team.</p><p class="paragraph" style="text-align:left;">🎧 Listen: <a class="link" href="https://www.masteringobservability.com/metrics-and-mayhem/podcast?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=observability-digest-june-2026" target="_blank" rel="noopener noreferrer nofollow">the podcast</a>  ·  📖 Read the companion piece: <a class="link" href="https://www.masteringobservability.com/p/deep-dive-security-bought-observability?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=observability-digest-june-2026" target="_blank" rel="noopener noreferrer nofollow">on the blog</a></p><h3 class="heading" style="text-align:left;" id="analytics-on-live-data-no-pipeline-">Analytics on Live Data. No Pipeline. Just Postgres.</h3><div class="image"><a class="image__link" href="https://www.tigerdata.com/go/trial?utm_source=content-syndication&utm_medium=referral&utm_campaign=beehiv-1k-trial&utm_content={{publication_alphanumeric_id}}&_bhiiv=opp_273c8afd-1600-489f-aa7e-bc57c8b710eb_e48efd09&bhcl_id=2cbecb94-034e-44a5-b49a-8574bdf95ff0_{{subscriber_id}}_{{email_address_id}}" rel="noopener" target="_blank"><img class="image__image" style="" src="https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/4b0a8e83-a573-4f35-8330-d0e841c1d611/td-onedb-ad2.png?t=1780677498"/></a></div><p class="paragraph" style="text-align:left;">Most teams treat analytics as a separate problem. As data grows, they add a warehouse, a pipeline, a sync job. By the time data reaches their dashboard, it&#39;s already stale.</p><p class="paragraph" style="text-align:left;"><a class="link" href="https://www.tigerdata.com/go/trial?utm_source=content-syndication&utm_medium=referral&utm_campaign=beehiv-1k-trial&utm_content={{publication_alphanumeric_id}}&_bhiiv=opp_273c8afd-1600-489f-aa7e-bc57c8b710eb_e48efd09&bhcl_id=2cbecb94-034e-44a5-b49a-8574bdf95ff0_{{subscriber_id}}_{{email_address_id}}" target="_blank" rel="noopener noreferrer nofollow">TimescaleDB</a> takes a different approach: extend Postgres instead of splitting away from it. </p><p class="paragraph" style="text-align:left;">Your transactions and your analytics run on the same database, on live data, with no pipeline in between.</p><p class="paragraph" style="text-align:left;">Hypertables partition time-series data automatically as volume grows. Hypercore compression cuts storage by up to 95%. Continuous aggregates pre-compute rollups so dashboards stay fast without re-querying everything.</p><p class="paragraph" style="text-align:left;">CERN runs it on Postgres to handle sensor data from the Large Hadron Collider. </p><p class="paragraph" style="text-align:left;">No second database. No migration. Same Postgres you already know.</p><p class="paragraph" style="text-align:left;"><a class="link" href="https://www.tigerdata.com/go/trial?utm_source=content-syndication&utm_medium=referral&utm_campaign=beehiv-1k-trial&utm_content={{publication_alphanumeric_id}}&_bhiiv=opp_273c8afd-1600-489f-aa7e-bc57c8b710eb_e48efd09&bhcl_id=2cbecb94-034e-44a5-b49a-8574bdf95ff0_{{subscriber_id}}_{{email_address_id}}" target="_blank" rel="noopener noreferrer nofollow">Get $1000 Credit To Start</a></p><h2 class="heading" style="text-align:left;" id="on-the-blog-this-month">On the blog this month</h2><p class="paragraph" style="text-align:left;">I publish these to the site, not your inbox, so here is the month in one place (round up EVERY web-only blog):</p><ul><li><p class="paragraph" style="text-align:left;"><b>The Regulator Is Not the Brake on AIOps. It Is the Design Spec.</b> The honest state of AIOps in regulated banking: only about 2% of AI in financial services runs fully autonomously, and the gates that earn the rest. <a class="link" href="https://www.masteringobservability.com/p/aiops-regulator-design-spec?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=observability-digest-june-2026" target="_blank" rel="noopener noreferrer nofollow">Read it</a>.</p></li><li><p class="paragraph" style="text-align:left;"><b>SecOps is buying observability;</b> the deep dive above is on the blog for the long read. <a class="link" href="https://www.masteringobservability.com/p/deep-dive-security-bought-observability?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=observability-digest-june-2026" target="_blank" rel="noopener noreferrer nofollow">Read it</a>.</p></li></ul><h2 class="heading" style="text-align:left;" id="from-my-feed">From my feed</h2><p class="paragraph" style="text-align:left;">A few things I posted this month that landed:</p><ul><li><p class="paragraph" style="text-align:left;">Most observability spend goes to detection, the 10%. The dashboard shows what you measured, not what matters. The other 90%, coordination and ownership, is where the time actually goes.</p></li><li><p class="paragraph" style="text-align:left;">The next tool is not the answer. The arrival fallacy in ops: the thing we chase was never on the other side of the purchase; it was on the other side of the work.</p></li><li><p class="paragraph" style="text-align:left;">The loudest person in the incident channel is a signal, not a problem. Anger is usually fear wearing a hi-vis vest, pointing straight at the risk.</p></li></ul><p class="paragraph" style="text-align:left;">I post these as they land. <a class="link" href="https://www.linkedin.com/in/allanmann1/?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=observability-digest-june-2026" target="_blank" rel="noopener noreferrer nofollow">Follow on LinkedIn</a>.</p><h2 class="heading" style="text-align:left;" id="worth-your-attention">Worth your attention</h2><p class="paragraph" style="text-align:left;">Outside reads that earned it this month:</p><ul><li><p class="paragraph" style="text-align:left;">Palo Alto completes its $3.35bn Chronosphere acquisition, the deal that started this conversation. <a class="link" href="https://www.paloaltonetworks.com/company/press/2026/palo-alto-networks-completes-chronosphere-acquisition--unifying-observability-and-security-for-the-ai-era?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=observability-digest-june-2026" target="_blank" rel="noopener noreferrer nofollow">Palo Alto Networks</a></p></li><li><p class="paragraph" style="text-align:left;">Forrester on why this is a dual-motive, AI-driven move, and who gets subordinated. <a class="link" href="https://www.forrester.com/blogs/paying-to-observe-it-all-palo-alto-networks-acquisition-of-chronosphere/?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=observability-digest-june-2026" target="_blank" rel="noopener noreferrer nofollow">Forrester</a></p></li><li><p class="paragraph" style="text-align:left;">Snowflake buys Observe; Cisco moves for Galileo. The convergence did not stop in 2025. <a class="link" href="https://techcrunch.com/2026/01/08/snowflake-announces-its-intent-to-buy-observability-platform-observe/?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=observability-digest-june-2026" target="_blank" rel="noopener noreferrer nofollow">TechCrunch</a></p></li></ul><h2 class="heading" style="text-align:left;" id="worth-a-listen">Worth a listen</h2><p class="paragraph" style="text-align:left;">Four I actually listened to this month, two technical, two on leadership:</p><ul><li><p class="paragraph" style="text-align:left;">Charity Majors on observability 1.0 vs 2.0 (podcast interview): why the &quot;three pillars&quot; were a vendor convenience, and what wide events and high cardinality actually change.</p></li><li><p class="paragraph" style="text-align:left;">Open Observability Talks (Dotan Horovits): OpenTelemetry&#39;s CNCF graduation and where eBPF continuous profiling is heading. Why OTel is now a safe long-term bet to build on.</p></li><li><p class="paragraph" style="text-align:left;">Sir Gareth Southgate on the High Performance podcast: leading under relentless scrutiny, and how rehearsing the hard moment turned England&#39;s penalty shootouts from a lottery into a practised process.</p></li><li><p class="paragraph" style="text-align:left;">No Bullsh*t Leadership (Martin Moore), &quot;12 Hard Leadership Truths&quot;: single-point accountability, speed over consensus, and grace under pressure. Hard truths that map onto running an incident.</p></li></ul><hr class="content_break"><p class="paragraph" style="text-align:left;">That is the reset. Monthly, short, judgement first. If a month is light, it stays short. Next month I take the convergence question one layer down: who actually gets the call when the merged platform goes dark.</p><p class="paragraph" style="text-align:left;">Allan</p><hr class="content_break"><p class="paragraph" style="text-align:left;">📕 The book is out. <i>Metrics & Mayhem: A CTO&#39;s Guide to Observability That Actually Works</i>. Kindle, paperback and hardback.</p><ul><li><p class="paragraph" style="text-align:left;"><a class="link" href="https://amzn.eu/d/0cAuR2K1?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=observability-digest-june-2026" target="_blank" rel="noopener noreferrer nofollow">Get the book on Amazon</a></p></li><li><p class="paragraph" style="text-align:left;"><a class="link" href="https://www.masteringobservability.com/metrics-and-mayhem/free-chapter?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=observability-digest-june-2026" target="_blank" rel="noopener noreferrer nofollow">Read a free chapter</a></p></li><li><p class="paragraph" style="text-align:left;"><a class="link" href="https://www.masteringobservability.com/metrics-and-mayhem/podcast?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=observability-digest-june-2026" target="_blank" rel="noopener noreferrer nofollow">Listen on Spotify and Apple Podcasts</a></p></li><li><p class="paragraph" style="text-align:left;"><a class="link" href="https://www.linkedin.com/in/allanmann1/?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=observability-digest-june-2026" target="_blank" rel="noopener noreferrer nofollow">Follow Allan on LinkedIn</a></p></li></ul></div><div class='beehiiv__footer'><br class='beehiiv__footer__break'><hr class='beehiiv__footer__line'><a target="_blank" class="beehiiv__footer_link" style="text-align: center;" href="https://www.beehiiv.com/?utm_campaign=b75f259d-b18a-4af6-b2a9-b04bef87c4c0&utm_medium=post_rss&utm_source=mastering_observability">Powered by beehiiv</a></div></div>
  ]]></content:encoded>
</item>

      <item>
  <title>Metrics &amp; Mayhem | Signal Drop: Your Role Changes Every Hour</title>
  <description>One style won&#39;t survive pressure. Your job is choosing the right mode at the right moment.</description>
      <enclosure url="https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/7aa93147-ae5c-406d-a961-828703c29f6e/thumbnail_signal_drop_your_role_changes.png" length="3029597" type="image/png"/>
  <link>https://www.masteringobservability.com/p/metrics-mayhem-signal-drop-your-role-changes-every-hour-5590f0de2fac2d2c</link>
  <guid isPermaLink="true">https://www.masteringobservability.com/p/metrics-mayhem-signal-drop-your-role-changes-every-hour-5590f0de2fac2d2c</guid>
  <pubDate>Tue, 30 Jun 2026 14:13:32 +0000</pubDate>
  <atom:published>2026-06-30T14:13:32Z</atom:published>
    <dc:creator>Allan Mann</dc:creator>
    <category><![CDATA[Newsletter]]></category>
    <category><![CDATA[Signal Drop]]></category>
    <category><![CDATA[Observability]]></category>
    <category><![CDATA[Sre]]></category>
    <category><![CDATA[Metric &amp; Mayhem]]></category>
    <category><![CDATA[Incident Response]]></category>
    <category><![CDATA[It Operations]]></category>
    <category><![CDATA[Leadership]]></category>
  <content:encoded><![CDATA[
    <div class='beehiiv'><style>
  .bh__table, .bh__table_header, .bh__table_cell { border: 1px solid #C0C0C0; }
  .bh__table_cell { padding: 5px; background-color: #FFFFFF; }
  .bh__table_cell p { color: #2C2C2C; font-family: 'PT Serif',Georgia,'Times New Roman',serif !important; overflow-wrap: break-word; }
  .bh__table_header { padding: 5px; background-color:#F1F1F1; }
  .bh__table_header p { color: #2C2C2C; font-family:'Trebuchet MS','Lucida Grande',Tahoma,sans-serif !important; overflow-wrap: break-word; }
</style><div class='beehiiv__body'><h3 class="heading" style="text-align:left;" id="oneline-promise-a-twominute-read-on"><b>One-line promise:</b> A two-minute read on switching leadership modes under pressure without turning the room into noise.</h3><h3 class="heading" style="text-align:left;" id="in-this-drop">In this drop</h3><ul><li><p class="paragraph" style="text-align:left;"><b>The point:</b> Your role changes by moment. Sometimes every five minutes.</p></li><li><p class="paragraph" style="text-align:left;"><b>Why it matters</b>: Staying in one mode too long turns manageable incidents into chaos.</p></li><li><p class="paragraph" style="text-align:left;"><b>Try next week</b>: Ten-second check before any call. Pick the mode. Say it out loud.</p></li></ul><h3 class="heading" style="text-align:left;" id="the-point"><b>The point</b></h3><p class="paragraph" style="text-align:left;">In IT Ops, leadership isn&#39;t a personality trait. It&#39;s a response to the moment.</p><p class="paragraph" style="text-align:left;">Some moments need a decision. One owner, one next step, one time check. Other moments require you to step back entirely and let the people closest to the problem breathe. Some need you to coach without stealing the wheel. And some need you to step back and clear blockers while someone else drives.</p><p class="paragraph" style="text-align:left;">The mistake most leaders make isn&#39;t picking the wrong mode. It&#39;s not picking at all. They default to whatever feels comfortable. Usually, the mode that makes them feel useful, not the one the room actually needs.</p><p class="paragraph" style="text-align:left;">Under pressure, that gap between what feels right and what is right gets expensive fast. Stakeholders pile in. Updates start looping. People talk over each other. Someone wants a timeline. Someone wants root cause. Someone wants reassurance. Ten voices, no control. And the leader is still in coaching mode when the room needed a decision three minutes ago.</p><p class="paragraph" style="text-align:left;">I keep four modes simple on purpose. Direct: we need a decision, now. Shield: the team needs space, you manage the noise. Coach: the team is capable but stuck, ask the right question instead of giving the answer. Delegate: someone else should run this, not because you&#39;re weak but because they&#39;re closest to the problem.</p><p class="paragraph" style="text-align:left;">The power isn&#39;t in the modes themselves. It&#39;s in choosing deliberately instead of defaulting.</p><h3 class="heading" style="text-align:left;" id="reality-check"><b>Reality check</b></h3><p class="paragraph" style="text-align:left;">If you lead in one mode all the time, you&#39;re not leading. You&#39;re performing.</p><h3 class="heading" style="text-align:left;" id="one-proof"><b>One proof</b></h3><p class="paragraph" style="text-align:left;">External anchor: Situational leadership is a known pattern for a reason. Hersey and Blanchard formalised it decades ago, and the principle hasn&#39;t changed: different people and different situations require different leadership behaviour. What changes in ops is the speed. You might switch modes three times in a single incident call.</p><p class="paragraph" style="text-align:left;"><b>Field note</b>: Ten seconds. That&#39;s all it takes to pick a mode before you speak. One pause, one deliberate choice, and the next 30 minutes of the call run differently. The room stops guessing. People focus.</p><h3 class="heading" style="text-align:left;" id="where-this-breaks"><b>Where this breaks</b></h3><p class="paragraph" style="text-align:left;">This fails when leaders confuse delegation with disappearing. Stepping back without removing blockers and protecting the team from noise isn&#39;t delegation. It&#39;s abandonment. Delegate mode only works when you stay close enough to clear the path.</p><h3 class="heading" style="text-align:left;" id="try-this-next-week"><b>Try this next week</b></h3><p class="paragraph" style="text-align:left;">Before your next incident call, leadership update, or tense conversation, do a ten-second check.</p><p class="paragraph" style="text-align:left;"><b>Ask</b>: What does this moment need? A decision, space, learning, or ownership?</p><p class="paragraph" style="text-align:left;"><b>Pick the mode</b>. Say it out loud at the start of the call: &quot;I&#39;m going to run this in direct mode for ten minutes. One owner, one next step, then we regroup.&quot;</p><p class="paragraph" style="text-align:left;"><b>Re-check after 15 minutes</b>. If the call is drifting, deliberately switch modes.</p><h3 class="heading" style="text-align:left;" id="three-links-im-watching"><b>Three links I&#39;m watching</b></h3><ol start="1"><li><p class="paragraph" style="text-align:left;">Hersey and Blanchard&#39;s Situational Leadership model. The original framework. Still holds, just faster in ops.</p></li><li><p class="paragraph" style="text-align:left;">Google&#39;s incident command system documentation. How structured role-switching works at scale during incidents.</p></li><li><p class="paragraph" style="text-align:left;">Will Larson&#39;s writing on engineering management modes. Practical thinking on when to direct, when to coach, and when to get out of the way.</p></li></ol><h3 class="heading" style="text-align:left;" id="one-question-for-you"><b>One question for you</b></h3><p class="paragraph" style="text-align:left;">Which mode do you default to under pressure, and what does your team pay for it?</p><hr class="content_break"><p class="paragraph" style="text-align:left;">Allan</p><p class="paragraph" style="text-align:left;"><b>PS</b>: Want the audio version? Listen to the Signal Drop here: <a class="link" href="https://open.spotify.com/show/1mC5tQRYnbzdhnqgWJg9cq?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=metrics-mayhem-signal-drop-your-role-changes-every-hour" target="_blank" rel="noopener noreferrer nofollow">[Spotify link</a>]</p></div><div class='beehiiv__footer'><br class='beehiiv__footer__break'><hr class='beehiiv__footer__line'><a target="_blank" class="beehiiv__footer_link" style="text-align: center;" href="https://www.beehiiv.com/?utm_campaign=0cbfa4e6-81b0-4464-a63c-f469b998cb6a&utm_medium=post_rss&utm_source=mastering_observability">Powered by beehiiv</a></div></div>
  ]]></content:encoded>
</item>

      <item>
  <title>You&#39;re Going To Have To Suffer Today</title>
  <description>You can&#39;t promise a quiet launch. You can give the team the bad night in advance. The ten-minute pre-mortem.</description>
      <enclosure url="https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/38d20550-3ae8-408f-9d1e-c0925d34ea4c/21_thumbnail_og.png" length="76608" type="image/png"/>
  <link>https://www.masteringobservability.com/p/signal-drop-youre-going-to-have-to-suffer-today</link>
  <guid isPermaLink="true">https://www.masteringobservability.com/p/signal-drop-youre-going-to-have-to-suffer-today</guid>
  <pubDate>Mon, 29 Jun 2026 05:44:00 +0000</pubDate>
  <atom:published>2026-06-29T05:44:00Z</atom:published>
    <category><![CDATA[Signal Drop]]></category>
    <category><![CDATA[Sre]]></category>
    <category><![CDATA[Incident Management]]></category>
    <category><![CDATA[On Call]]></category>
    <category><![CDATA[Leadership]]></category>
  <content:encoded><![CDATA[
    <div class='beehiiv'><style>
  .bh__table, .bh__table_header, .bh__table_cell { border: 1px solid #C0C0C0; }
  .bh__table_cell { padding: 4px; background-color: #f9fafb; }
  .bh__table_cell p { color: #2D2D2D; font-family: 'Space Grotesk',Helvetica,Arial,sans-serif !important; overflow-wrap: break-word; }
  .bh__table_header { padding: 6px; background-color:#e0f7f3; }
  .bh__table_header p { color: #222222; font-family:'Space Grotesk',Helvetica,Arial,sans-serif !important; overflow-wrap: break-word; }
</style><div class='beehiiv__body'><hr class="content_break"><p class="paragraph" style="text-align:left;"><i><b>You can&#39;t promise your team a quiet launch. You can give them the bad night in advance. That&#39;s what a pre-mortem is.</b></i></p><h2 class="heading" style="text-align:left;" id="in-this-drop">In this drop</h2><ul><li><p class="paragraph" style="text-align:left;"><b>The point:</b> A pre-mortem isn&#39;t pessimism; it&#39;s naming the hard moment in daylight so it&#39;s recognised, not feared, at 3am.</p></li><li><p class="paragraph" style="text-align:left;"><b>Why it matters:</b> Teams that rehearsed the bad moment treat the first wobble as information; teams that assumed calm treat it as catastrophe.</p></li><li><p class="paragraph" style="text-align:left;"><b>Try this next week:</b> Run a ten-minute pre-mortem before your next launch: it&#39;s 3am and this broke, tell me the story.</p></li></ul><h2 class="heading" style="text-align:left;" id="the-point">The point</h2><p class="paragraph" style="text-align:left;">A launch I&#39;ll never forget, for the wrong reasons. Tested, load-tested, dashboards green. Forty minutes in it wobbled, and the team fell apart. Not because the wobble was huge. Because nobody had told them it was coming, so the first trouble read as &#39;it&#39;s all going wrong&#39; instead of &#39;here&#39;s the hard moment&#39;.</p><p class="paragraph" style="text-align:left;">I heard a story this week about a dad and his ten-year-old before a cup final. Instead of &#39;relax, have fun&#39;, he said: you&#39;re going to have to suffer today, there&#39;ll be a moment you feel you can&#39;t win. It came. The boy played through it, because he&#39;d been warned. The other team, winning then losing, fell apart, because nobody had told them.</p><p class="paragraph" style="text-align:left;">You cannot promise a quiet launch. What you can do is tell the team where it&#39;s going to hurt before it hurts. That&#39;s a pre-mortem: it&#39;s 3am, two days from now, this has gone wrong, what broke, who&#39;s awake, what do they reach for.</p><h2 class="heading" style="text-align:left;" id="reality-check">Reality check</h2><p class="paragraph" style="text-align:left;">What changes everything: a team that rehearsed the bad night meets the first wobble with recognition, not panic.</p><h2 class="heading" style="text-align:left;" id="one-proof">One proof</h2><p class="paragraph" style="text-align:left;">Gary Klein formalised the pre-mortem in Harvard Business Review in 2007: imagine the project has failed, then work backwards to why. Field note: on one rota, after we started running a ten-minute pre-mortem before significant changes, time-to-rollback-decision on the incidents that followed dropped from roughly twenty-five minutes to under ten across the next quarter. The team wasn&#39;t braver. They&#39;d just met the night before.</p><h2 class="heading" style="text-align:left;" id="where-this-breaks">Where this breaks</h2><p class="paragraph" style="text-align:left;">This breaks if the pre-mortem becomes theatre, a document nobody reads on the night. The output isn&#39;t the doc; it&#39;s a team that recognises the failure shape. Keep it to ten minutes and to the people who&#39;ll actually be on call.</p><h2 class="heading" style="text-align:left;" id="try-this-next-week">Try this next week</h2><p class="paragraph" style="text-align:left;">Before your next launch, get the on-call people together for ten minutes. No longer.</p><p class="paragraph" style="text-align:left;">Ask one question and write the answers: it&#39;s 3am, this change broke, tell me the story (symptom, dashboard, who gets the call, first action, rollback, who calls it).</p><p class="paragraph" style="text-align:left;">Don&#39;t aim to predict the exact failure. Aim to make the shape of a bad night familiar.</p><h2 class="heading" style="text-align:left;" id="three-links-im-watching">Three links I&#39;m watching</h2><ul><li><p class="paragraph" style="text-align:left;">Gary Klein, <a class="link" href="https://hbr.org/2007/09/performing-a-project-premortem?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=you-re-going-to-have-to-suffer-today" target="_blank" rel="noopener noreferrer nofollow">&#39;Performing a Project Premortem&#39;</a> (HBR, 2007): the original, two pages, still the best version.</p></li><li><p class="paragraph" style="text-align:left;"><a class="link" href="https://sre.google/workbook/incident-response/?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=you-re-going-to-have-to-suffer-today" target="_blank" rel="noopener noreferrer nofollow">Google SRE Workbook on incident response</a>: the rollback-decision and command structure this habit feeds.</p></li><li><p class="paragraph" style="text-align:left;">Signal Drop, <a class="link" href="https://www.masteringobservability.com/p/metrics-mayhem-signal-drop-progress-isnt-linear?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=you-re-going-to-have-to-suffer-today" target="_blank" rel="noopener noreferrer nofollow">&#39;Progress Isn&#39;t Linear&#39;</a>: the companion on expecting the rough patch rather than fearing it.</p></li></ul><h2 class="heading" style="text-align:left;" id="one-question-for-you">One question for you</h2><p class="paragraph" style="text-align:left;">When did a launch last go sideways on you, and would a ten-minute pre-mortem have changed how the night felt?</p><hr class="content_break"><p class="paragraph" style="text-align:left;">Allan</p><p class="paragraph" style="text-align:left;">📕 The book is out. Metrics & Mayhem: A CTO&#39;s Guide to Observability That Actually Works. Kindle is live now; paperback and hardback launched on 1 June.</p><div class="custom_html"><table width="100%" align="center" cellpadding="0" cellspacing="0" border="0" style="max-width:600px;margin:0 auto;"><tbody><tr><td style="background-color:#2F9E8D;height:4px;line-height:4px;font-size:4px;">&nbsp;</td></tr><tr><td style="background-color:#0D2127;padding:24px 24px 20px 24px;"><p style="margin:0 0 8px 0;font-family:'Space Mono','Courier New',monospace;font-size:12px;letter-spacing:2px;color:#74DDCD;text-transform:uppercase;">Everything in one place</p><p style="margin:0;font-family:'DM Sans',Helvetica,Arial,sans-serif;font-size:22px;line-height:1.3;color:#ffffff;font-weight:700;">New here? Start with the free chapter.</p></td></tr><tr><td style="background-color:#ffffff;padding:22px 24px 24px 24px;font-family:'DM Sans',Helvetica,Arial,sans-serif;"><table cellpadding="0" cellspacing="0" border="0"><tbody><tr><td style="background-color:#2F9E8D;border-radius:4px;"><a href="https://www.masteringobservability.com/metrics-and-mayhem/free-chapter?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=you-re-going-to-have-to-suffer-today" style="display:inline-block;padding:13px 24px;font-family:'DM Sans',Helvetica,Arial,sans-serif;font-size:16px;font-weight:700;color:#0D2127;text-decoration:none;">Read the free chapter →</a></td></tr></tbody></table><p style="margin:20px 0 10px 0;font-family:'Space Mono','Courier New',monospace;font-size:12px;letter-spacing:2px;color:#2F9E8D;text-transform:uppercase;">More from Metrics & Mayhem</p><table width="100%" cellpadding="0" cellspacing="0" border="0" style="font-family:'DM Sans',Helvetica,Arial,sans-serif;font-size:15px;line-height:1.5;"><tbody><tr><td style="padding:7px 0;color:#16282D;"><strong>The book</strong> &nbsp;·&nbsp; <a href="https://amzn.eu/d/0cAuR2K1?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=you-re-going-to-have-to-suffer-today" style="color:#17695C;text-decoration:underline;">Get Metrics & Mayhem →</a></td></tr><tr><td style="padding:7px 0;border-top:1px solid #EAF6F3;color:#16282D;"><strong>Podcast</strong> &nbsp;·&nbsp; <a href="https://www.masteringobservability.com/metrics-and-mayhem/podcast?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=you-re-going-to-have-to-suffer-today" style="color:#17695C;text-decoration:underline;">Listen on Spotify & Apple →</a></td></tr><tr><td style="padding:7px 0;border-top:1px solid #EAF6F3;color:#16282D;"><strong>Connect</strong> &nbsp;·&nbsp; <a href="https://www.linkedin.com/in/allanmann1/?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=you-re-going-to-have-to-suffer-today" style="color:#17695C;text-decoration:underline;">Allan on LinkedIn →</a></td></tr></tbody></table></td></tr></tbody></table></div><hr class="content_break"><h3 class="heading" style="text-align:left;" id="analytics-on-live-data-without-leav">Analytics on Live Data Without Leaving Postgres</h3><div class="image"><a class="image__link" href="https://www.tigerdata.com/go/trial?utm_source=content-syndication&utm_medium=referral&utm_campaign=beehiv-newsletter-trial&utm_term={{publication_alphanumeric_id}}&_bhiiv=opp_eaae6c54-1f42-4ac0-b368-ba2f0fd17415_e48efd09&bhcl_id=c4d7f162-ac86-426c-b919-e8e56ecb02fe_{{subscriber_id}}_{{email_address_id}}" rel="noopener" target="_blank"><img class="image__image" style="" src="https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/f5fb74e3-83ed-4203-a0a2-03f448c6ccdc/_1_Time_Series_1200x600.png?t=1776817593"/></a></div><p class="paragraph" style="text-align:left;">When analytics on Postgres slows down, most teams add a second database. <a class="link" href="https://www.tigerdata.com/go/trial?utm_source=content-syndication&utm_medium=referral&utm_campaign=beehiv-newsletter-trial&utm_term={{publication_alphanumeric_id}}&_bhiiv=opp_eaae6c54-1f42-4ac0-b368-ba2f0fd17415_e48efd09&bhcl_id=c4d7f162-ac86-426c-b919-e8e56ecb02fe_{{subscriber_id}}_{{email_address_id}}" target="_blank" rel="noopener noreferrer nofollow">TimescaleDB by Tiger Data</a> takes a different approach: extend Postgres with columnar storage and time-series primitives to run analytics on live data, no split architecture, no pipeline lag, no new query language to learn. Start building for free. No credit card required.</p><p class="paragraph" style="text-align:left;"><a class="link" href="https://www.tigerdata.com/go/trial?utm_source=content-syndication&utm_medium=referral&utm_campaign=beehiv-newsletter-trial&utm_term={{publication_alphanumeric_id}}&_bhiiv=opp_eaae6c54-1f42-4ac0-b368-ba2f0fd17415_e48efd09&bhcl_id=c4d7f162-ac86-426c-b919-e8e56ecb02fe_{{subscriber_id}}_{{email_address_id}}" target="_blank" rel="noopener noreferrer nofollow">Start building for free</a></p><p class="paragraph" style="text-align:left;"></p></div><div class='beehiiv__footer'><br class='beehiiv__footer__break'><hr class='beehiiv__footer__line'><a target="_blank" class="beehiiv__footer_link" style="text-align: center;" href="https://www.beehiiv.com/?utm_campaign=655bf4a0-183b-4cf9-a731-4f78433a02f6&utm_medium=post_rss&utm_source=mastering_observability">Powered by beehiiv</a></div></div>
  ]]></content:encoded>
</item>

      <item>
  <title>byte-size: What Is eBPF?</title>
  <description>The kernel technology behind zero-code observability, in plain English, and where it stops short.</description>
      <enclosure url="https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/bfebe610-b397-485c-b8da-f3466bfc420f/ebpf_bytesize_og_1200x630.png" length="32808" type="image/png"/>
  <link>https://www.masteringobservability.com/p/what-is-ebpf</link>
  <guid isPermaLink="true">https://www.masteringobservability.com/p/what-is-ebpf</guid>
  <pubDate>Fri, 26 Jun 2026 19:12:55 +0000</pubDate>
  <atom:published>2026-06-26T19:12:55Z</atom:published>
    <category><![CDATA[Newsletter]]></category>
    <category><![CDATA[Observability]]></category>
    <category><![CDATA[Byte Size]]></category>
    <category><![CDATA[Ebpf]]></category>
  <content:encoded><![CDATA[
    <div class='beehiiv'><style>
  .bh__table, .bh__table_header, .bh__table_cell { border: 1px solid #C9DCDC; }
  .bh__table_cell { padding: 4px; background-color: #FFFFFF; }
  .bh__table_cell p { color: #2D2D2D; font-family: 'Space Grotesk',Helvetica,Arial,sans-serif !important; overflow-wrap: break-word; }
  .bh__table_header { padding: 6px; background-color:#EAF6F3; }
  .bh__table_header p { color: #222222; font-family:'Space Grotesk',Helvetica,Arial,sans-serif !important; overflow-wrap: break-word; }
</style><div class='beehiiv__body'><p class="paragraph" style="text-align:left;"><i>The kernel technology behind zero-code observability, in plain English, and where it stops short.</i></p><p class="paragraph" style="text-align:left;">If you have read anything about modern observability lately, you have run into the same four letters over and over: eBPF. It is the engine under a wave of zero-code tools, including <a class="link" href="https://www.masteringobservability.com/p/what-is-odigos?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-ebpf" target="_blank" rel="noopener noreferrer nofollow">Odigos</a>, that promise traces and metrics with no SDK and no redeploy. It is genuinely clever, and it is not magic. Here is what eBPF actually is, in plain English, and the honest limits before you lean a whole strategy on it.</p><h2 class="heading" style="text-align:left;" id="what-is-e-bpf">What is eBPF?</h2><p class="paragraph" style="text-align:left;">eBPF stands for extended Berkeley Packet Filter, though the name now undersells it. It is a feature of the Linux kernel, the layer every application runs on top of, that lets small, sandboxed programs run deep inside the kernel with no change to the kernel&#39;s source code, no module to load, and no reboot. A built-in verifier checks every program before it runs and rejects anything that could crash or hang the system, so the power comes with guardrails. In plain terms, eBPF lets a tool watch what your software and your network are really doing, from the outside, with very little overhead.</p><h2 class="heading" style="text-align:left;" id="a-thirtysecond-history">A thirty-second history</h2><p class="paragraph" style="text-align:left;">The original BPF arrived in 1992 as a way to filter network packets for tools like tcpdump. The extended version landed in the Linux kernel in 2014 and turned that narrow trick into a general-purpose way to observe and shape almost anything the kernel touches: system calls, network traffic, function entry and exit. It is now stewarded by the <a class="link" href="https://ebpf.foundation/?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-ebpf" target="_blank" rel="noopener noreferrer nofollow">eBPF Foundation</a> under the Linux Foundation, formed in 2021, whose members include Meta, Google, Microsoft, Netflix and Isovalent. That breadth matters: eBPF is becoming a cross-industry standard, and it is even spreading beyond Linux through the eBPF for Windows project.</p><h2 class="heading" style="text-align:left;" id="what-it-does-well">What it does well</h2><p class="paragraph" style="text-align:left;">For observability, three things make eBPF compelling:</p><ul><li><p class="paragraph" style="text-align:left;"><b>No code changes.</b> Because it hooks kernel events rather than your application, you get traces, metrics and profiles without importing an SDK or redeploying. That is what zero-code instrumentation actually means.</p></li><li><p class="paragraph" style="text-align:left;"><b>Low overhead, broad reach.</b> Programs are compiled and run inside the kernel, so they capture system-wide signals, every process on the node, network flows and system calls, at a cost most workloads will not feel.</p></li><li><p class="paragraph" style="text-align:left;"><b>One mechanism, many jobs.</b> The same technology powers networking, continuous profiling, security monitoring and tracing. Learn it once and it pays back across the stack.</p></li></ul><h2 class="heading" style="text-align:left;" id="the-honest-ledger">The honest ledger</h2><p class="paragraph" style="text-align:left;">eBPF is powerful, not free. The gaps are real.</p><ul><li><p class="paragraph" style="text-align:left;"><b>It needs a modern Linux kernel.</b> Most useful features want kernel 4.x or newer, and the richest capabilities, like BTF and CO-RE that let one binary run across kernel versions, want 5.8 or newer. Older estates such as RHEL or CentOS 7 are simply not supported. The legacy systems that hold the actual money are often the ones eBPF cannot reach.</p></li><li><p class="paragraph" style="text-align:left;"><b>It is privileged by nature.</b> eBPF programs run in the kernel and need elevated capabilities. That is a security conversation, not a tick box, and some managed Kubernetes platforms restrict the node-level access these agents require.</p></li><li><p class="paragraph" style="text-align:left;"><b>It shows the plumbing, not the meaning.</b> eBPF captures generic runtime and protocol telemetry brilliantly. It does not know your customer journey or your business logic. You still have to define your SLOs and add instrumentation for the spans that matter to your domain.</p></li><li><p class="paragraph" style="text-align:left;"><b>It is not trivial to build with.</b> Writing and debugging eBPF is kernel-level work, and when something fails to load the feedback can be thin. Most teams consume eBPF through a tool rather than writing it themselves, which is fine, as long as you know that is the trade.</p></li></ul><h2 class="heading" style="text-align:left;" id="so-is-e-bpf-for-you">So, is eBPF for you?</h2><p class="paragraph" style="text-align:left;">If you run modern, Kubernetes-native Linux workloads, eBPF is one of the best things to happen to observability in years, and you are almost certainly already benefiting from it through the tools you use. If your weight sits in older infrastructure or locked-down managed platforms, treat it as one valuable layer rather than the whole answer, and check kernel versions and node access before you commit. Either way, hold on to the last point: eBPF gets you the data faster, it does not decide what the data means. That part is still your job.</p><hr class="content_break"><p class="paragraph" style="text-align:left;"><b>Where have you seen eBPF earn its place, or fall short?</b> I would like to hear it, especially from anyone running it in anger. Reply, or <a class="link" href="https://calendly.com/managing_observability/30min?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-ebpf" target="_blank" rel="noopener noreferrer nofollow">book a slot</a> and tell me what you found.</p><div class="custom_html"><table width="100%" align="center" cellpadding="0" cellspacing="0" border="0" style="max-width:600px;margin:0 auto;"><tbody><tr><td style="background-color:#2F9E8D;height:4px;line-height:4px;font-size:4px;">&nbsp;</td></tr><tr><td style="background-color:#0D2127;padding:24px 24px 18px 24px;"><p style="margin:0 0 8px 0;font-family:'Space Mono','Courier New',monospace;font-size:12px;letter-spacing:2px;color:#74DDCD;text-transform:uppercase;">Get the next one</p><p style="margin:0;font-family:'DM Sans',Helvetica,Arial,sans-serif;font-size:22px;line-height:1.3;color:#ffffff;font-weight:700;">One signal a week. No noise.</p></td></tr><tr><td style="background-color:#ffffff;padding:20px 24px 24px 24px;font-family:'DM Sans',Helvetica,Arial,sans-serif;"><p style="margin:0 0 18px 0;font-size:16px;line-height:1.5;color:#1a1a1a;">If this was useful, Metrics & Mayhem sends one short, practical piece like it to IT operations leaders most weeks. No fluff, no vendor noise.</p><table cellpadding="0" cellspacing="0" border="0"><tbody><tr><td style="background-color:#2F9E8D;border-radius:4px;"><a href="https://masteringobservability.com?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-ebpf" style="display:inline-block;padding:12px 22px;font-family:'DM Sans',Helvetica,Arial,sans-serif;font-size:16px;font-weight:700;color:#0D2127;text-decoration:none;">Join free →</a></td></tr></tbody></table><p style="margin:16px 0 0 0;font-size:14px;line-height:1.5;color:#44595b;">Prefer to start with the book? Read a <a href="https://www.masteringobservability.com/metrics-and-mayhem/free-chapter?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-ebpf" style="color:#0D2127;text-decoration:underline;">free chapter</a>.</p></td></tr></tbody></table></div><h3 class="heading" style="text-align:left;" id="related-from-the-bytesize-series">Related from the byte-size series</h3><ul><li><p class="paragraph" style="text-align:left;"><a class="link" href="https://www.masteringobservability.com/p/what-is-odigos?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-ebpf" target="_blank" rel="noopener noreferrer nofollow">byte-size: What Is Odigos?</a></p></li><li><p class="paragraph" style="text-align:left;"><a class="link" href="https://www.masteringobservability.com/p/what-is-grafana-alloy?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-ebpf" target="_blank" rel="noopener noreferrer nofollow">byte-size: What Is Grafana Alloy</a></p></li><li><p class="paragraph" style="text-align:left;"><a class="link" href="https://www.masteringobservability.com/p/opentelemetry-collector-implementation-guide?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-ebpf" target="_blank" rel="noopener noreferrer nofollow">OpenTelemetry Collector Implementation Guide</a></p></li></ul><h3 class="heading" style="text-align:left;" id="sources-further-reading">Sources / further reading</h3><ul><li><p class="paragraph" style="text-align:left;">What is eBPF (project explainer): <a class="link" href="https://ebpf.io/what-is-ebpf/?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-ebpf" target="_blank" rel="noopener noreferrer nofollow">ebpf.io</a></p></li><li><p class="paragraph" style="text-align:left;">eBPF Foundation (Linux Foundation): <a class="link" href="https://ebpf.foundation/?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-ebpf" target="_blank" rel="noopener noreferrer nofollow">ebpf.foundation</a></p></li><li><p class="paragraph" style="text-align:left;">eBPF overview: <a class="link" href="https://en.wikipedia.org/wiki/EBPF?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-ebpf" target="_blank" rel="noopener noreferrer nofollow">en.wikipedia.org/wiki/eBPF</a></p></li><li><p class="paragraph" style="text-align:left;">eBPF for Windows: <a class="link" href="https://github.com/microsoft/ebpf-for-windows?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-ebpf" target="_blank" rel="noopener noreferrer nofollow">github.com/microsoft/ebpf-for-windows</a></p></li></ul></div><div class='beehiiv__footer'><br class='beehiiv__footer__break'><hr class='beehiiv__footer__line'><a target="_blank" class="beehiiv__footer_link" style="text-align: center;" href="https://www.beehiiv.com/?utm_campaign=30ae7490-d437-4daa-8c4a-53a81726eb45&utm_medium=post_rss&utm_source=mastering_observability">Powered by beehiiv</a></div></div>
  ]]></content:encoded>
</item>

      <item>
  <title>byte-size: What Is OpenTelemetry?</title>
  <description>The vendor-neutral standard for telemetry, in plain English, and what it does not do for you.</description>
      <enclosure url="https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/a754ab3e-c2fd-4029-92ec-8b8ef4d355fe/opentelemetry_bytesize_og_1200x630.png" length="30585" type="image/png"/>
  <link>https://www.masteringobservability.com/p/what-is-opentelemetry</link>
  <guid isPermaLink="true">https://www.masteringobservability.com/p/what-is-opentelemetry</guid>
  <pubDate>Fri, 26 Jun 2026 19:08:18 +0000</pubDate>
  <atom:published>2026-06-26T19:08:18Z</atom:published>
    <category><![CDATA[Newsletter]]></category>
    <category><![CDATA[Observability]]></category>
    <category><![CDATA[Byte Size]]></category>
    <category><![CDATA[Opentelemetry]]></category>
  <content:encoded><![CDATA[
    <div class='beehiiv'><style>
  .bh__table, .bh__table_header, .bh__table_cell { border: 1px solid #C9DCDC; }
  .bh__table_cell { padding: 4px; background-color: #FFFFFF; }
  .bh__table_cell p { color: #2D2D2D; font-family: 'Space Grotesk',Helvetica,Arial,sans-serif !important; overflow-wrap: break-word; }
  .bh__table_header { padding: 6px; background-color:#EAF6F3; }
  .bh__table_header p { color: #222222; font-family:'Space Grotesk',Helvetica,Arial,sans-serif !important; overflow-wrap: break-word; }
</style><div class='beehiiv__body'><p class="paragraph" style="text-align:left;"><i>The vendor-neutral standard for telemetry, in plain English, and what it does not do for you.</i></p><p class="paragraph" style="text-align:left;">If <a class="link" href="https://www.masteringobservability.com/p/what-is-ebpf?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-opentelemetry" target="_blank" rel="noopener noreferrer nofollow">eBPF</a> is the engine under a lot of modern observability, OpenTelemetry is the language everyone agreed to speak. It is now almost everywhere, and in May 2026 it became a CNCF graduated project, the formal seal on what was already the de facto standard. Here is what OpenTelemetry actually is, why it matters, and the honest limits before you rip out what you have.</p><h2 class="heading" style="text-align:left;" id="what-is-open-telemetry">What is OpenTelemetry?</h2><p class="paragraph" style="text-align:left;">OpenTelemetry, usually shortened to OTel, is an open-source, vendor-neutral framework for collecting telemetry: the traces, metrics and logs, and now profiles, that your systems produce. Instead of every vendor shipping its own agent and format, OTel gives you one set of APIs, SDKs and a collector that instrument your applications once and send the data anywhere you choose. It is a Cloud Native Computing Foundation project, second only to Kubernetes in activity, with more than 12,000 contributors from over 2,800 companies, and it reached the CNCF&#39;s top maturity tier in May 2026.</p><h2 class="heading" style="text-align:left;" id="the-pieces-that-matter">The pieces that matter</h2><p class="paragraph" style="text-align:left;">You do not need the whole glossary, just four parts: the instrumentation, automatic or manual, that produces the data; the OTLP protocol that carries it; the Collector that receives, processes and exports it; and the semantic conventions that keep names consistent, so an HTTP request means the same thing everywhere. Tools like Grafana Alloy are distributions of that Collector, and tools like Odigos produce OTel data with no code changes.</p><h2 class="heading" style="text-align:left;" id="why-it-matters">Why it matters</h2><ul><li><p class="paragraph" style="text-align:left;"><b>Instrument once, send anywhere.</b> Because the data is standard, you point it at whatever backend you like and switch later without re-instrumenting. That is the end of agent lock-in.</p></li><li><p class="paragraph" style="text-align:left;"><b>It is the safe long-term bet.</b> With graduation and adoption at this scale, building on OTel is building on the industry default, not a single vendor&#39;s roadmap.</p></li><li><p class="paragraph" style="text-align:left;"><b>It unifies the signals.</b> Traces, metrics and logs in one framework, correlated, rather than three disconnected tools.</p></li></ul><h2 class="heading" style="text-align:left;" id="the-honest-ledger">The honest ledger</h2><ul><li><p class="paragraph" style="text-align:left;"><b>It is a framework, not a backend.</b> OTel collects and ships your data. It does not store, query or visualise it. You still need somewhere for the data to land, whether that is Prometheus, Grafana or a commercial platform.</p></li><li><p class="paragraph" style="text-align:left;"><b>Maturity varies.</b> Traces and metrics are stable, logs are maturing, and profiles are still early. Some language SDKs are further ahead than others, so check yours.</p></li><li><p class="paragraph" style="text-align:left;"><b>It adds operational surface.</b> The Collector and its pipelines are powerful, and that power has a cost: more to run, and real discipline needed on data volume and cardinality before the bill surprises you.</p></li><li><p class="paragraph" style="text-align:left;"><b>Migration is real work.</b> Moving from proprietary agents to OTel pays off, but it is a project, not a switch you flip.</p></li></ul><h2 class="heading" style="text-align:left;" id="so-is-open-telemetry-for-you">So, is OpenTelemetry for you?</h2><p class="paragraph" style="text-align:left;">For almost anyone building or running modern systems, yes, and increasingly it is not really a choice: it is the standard your tools already speak. The honest question is not whether to adopt OTel, but how fast, and how much of the pipeline you run yourself versus buy. If you want vendor neutrality and a future-proof foundation, start now, on your most important services. If you need turnkey today, a managed OTel-based offering gets you there without running the plumbing yourself.</p><hr class="content_break"><p class="paragraph" style="text-align:left;"><b>Where has OpenTelemetry paid off for you, or tripped you up?</b> I would like to hear it, especially the migration stories. Reply, or <a class="link" href="https://calendly.com/managing_observability/30min?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-opentelemetry" target="_blank" rel="noopener noreferrer nofollow">book a slot</a> and tell me what you found.</p><div class="custom_html"><table width="100%" align="center" cellpadding="0" cellspacing="0" border="0" style="max-width:600px;margin:0 auto;"><tbody><tr><td style="background-color:#2F9E8D;height:4px;line-height:4px;font-size:4px;">&nbsp;</td></tr><tr><td style="background-color:#0D2127;padding:24px 24px 18px 24px;"><p style="margin:0 0 8px 0;font-family:'Space Mono','Courier New',monospace;font-size:12px;letter-spacing:2px;color:#74DDCD;text-transform:uppercase;">Get the next one</p><p style="margin:0;font-family:'DM Sans',Helvetica,Arial,sans-serif;font-size:22px;line-height:1.3;color:#ffffff;font-weight:700;">One signal a week. No noise.</p></td></tr><tr><td style="background-color:#ffffff;padding:20px 24px 24px 24px;font-family:'DM Sans',Helvetica,Arial,sans-serif;"><p style="margin:0 0 18px 0;font-size:16px;line-height:1.5;color:#1a1a1a;">If this was useful, Metrics & Mayhem sends one short, practical piece like it to IT operations leaders most weeks. No fluff, no vendor noise.</p><table cellpadding="0" cellspacing="0" border="0"><tbody><tr><td style="background-color:#2F9E8D;border-radius:4px;"><a href="https://masteringobservability.com?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-opentelemetry" style="display:inline-block;padding:12px 22px;font-family:'DM Sans',Helvetica,Arial,sans-serif;font-size:16px;font-weight:700;color:#0D2127;text-decoration:none;">Join free →</a></td></tr></tbody></table><p style="margin:16px 0 0 0;font-size:14px;line-height:1.5;color:#44595b;">Prefer to start with the book? Read a <a href="https://www.masteringobservability.com/metrics-and-mayhem/free-chapter?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-opentelemetry" style="color:#0D2127;text-decoration:underline;">free chapter</a>.</p></td></tr></tbody></table></div><h3 class="heading" style="text-align:left;" id="related-from-the-bytesize-series">Related from the byte-size series</h3><ul><li><p class="paragraph" style="text-align:left;"><a class="link" href="https://www.masteringobservability.com/p/opentelemetry-collector-implementation-guide?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-opentelemetry" target="_blank" rel="noopener noreferrer nofollow">OpenTelemetry Collector Implementation Guide</a></p></li><li><p class="paragraph" style="text-align:left;"><a class="link" href="https://www.masteringobservability.com/p/what-is-grafana-alloy?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-opentelemetry" target="_blank" rel="noopener noreferrer nofollow">byte-size: What Is Grafana Alloy</a></p></li><li><p class="paragraph" style="text-align:left;"><a class="link" href="https://www.masteringobservability.com/p/what-is-ebpf?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-opentelemetry" target="_blank" rel="noopener noreferrer nofollow">byte-size: What Is eBPF?</a></p></li><li><p class="paragraph" style="text-align:left;"><a class="link" href="https://www.masteringobservability.com/p/what-is-odigos?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-opentelemetry" target="_blank" rel="noopener noreferrer nofollow">byte-size: What Is Odigos?</a></p></li></ul><h3 class="heading" style="text-align:left;" id="sources-further-reading">Sources / further reading</h3><ul><li><p class="paragraph" style="text-align:left;">OpenTelemetry project: <a class="link" href="https://opentelemetry.io/?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-opentelemetry" target="_blank" rel="noopener noreferrer nofollow">opentelemetry.io</a></p></li><li><p class="paragraph" style="text-align:left;">OpenTelemetry graduates (CNCF announcement): <a class="link" href="https://www.cncf.io/announcements/2026/05/21/cloud-native-computing-foundation-announces-opentelemetrys-graduation-solidifying-status-as-the-de-facto-observability-standard/?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-opentelemetry" target="_blank" rel="noopener noreferrer nofollow">cncf.io</a></p></li><li><p class="paragraph" style="text-align:left;">OpenTelemetry is a CNCF graduated project: <a class="link" href="https://opentelemetry.io/blog/2026/otel-graduates/?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-opentelemetry" target="_blank" rel="noopener noreferrer nofollow">opentelemetry.io/blog</a></p></li></ul></div><div class='beehiiv__footer'><br class='beehiiv__footer__break'><hr class='beehiiv__footer__line'><a target="_blank" class="beehiiv__footer_link" style="text-align: center;" href="https://www.beehiiv.com/?utm_campaign=0418fb06-1807-4267-8657-c482133b0c4e&utm_medium=post_rss&utm_source=mastering_observability">Powered by beehiiv</a></div></div>
  ]]></content:encoded>
</item>

      <item>
  <title>byte-size: What Is Distributed Tracing?</title>
  <description>How you follow one request across many services, in plain English, and where it falls short.</description>
      <enclosure url="https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/e0d894ba-a713-4060-b42e-9a446d45f71b/distributed_tracing_bytesize_og_1200x630.png" length="30283" type="image/png"/>
  <link>https://www.masteringobservability.com/p/what-is-distributed-tracing</link>
  <guid isPermaLink="true">https://www.masteringobservability.com/p/what-is-distributed-tracing</guid>
  <pubDate>Fri, 26 Jun 2026 19:07:13 +0000</pubDate>
  <atom:published>2026-06-26T19:07:13Z</atom:published>
    <category><![CDATA[Newsletter]]></category>
    <category><![CDATA[Observability]]></category>
    <category><![CDATA[Byte Size]]></category>
    <category><![CDATA[Distributed Tracing]]></category>
  <content:encoded><![CDATA[
    <div class='beehiiv'><style>
  .bh__table, .bh__table_header, .bh__table_cell { border: 1px solid #C9DCDC; }
  .bh__table_cell { padding: 4px; background-color: #FFFFFF; }
  .bh__table_cell p { color: #2D2D2D; font-family: 'Space Grotesk',Helvetica,Arial,sans-serif !important; overflow-wrap: break-word; }
  .bh__table_header { padding: 6px; background-color:#EAF6F3; }
  .bh__table_header p { color: #222222; font-family:'Space Grotesk',Helvetica,Arial,sans-serif !important; overflow-wrap: break-word; }
</style><div class='beehiiv__body'><p class="paragraph" style="text-align:left;"><i>How you follow one request across many services, in plain English, and where it falls short.</i></p><p class="paragraph" style="text-align:left;">In a monolith, when something is slow you read one log. In microservices, a single click can touch a dozen services, and the question &quot;where did the time go?&quot; gets genuinely hard. Distributed tracing is how you answer it. Here is what it is, in plain English, and the honest limits.</p><h2 class="heading" style="text-align:left;" id="what-is-distributed-tracing">What is distributed tracing?</h2><p class="paragraph" style="text-align:left;">Distributed tracing follows a single request as it travels through all the services that handle it, and stitches the whole journey into one picture called a trace. A trace is made of spans, and each span is one unit of work: an API call, a database query, a function. Every span records when it started, how long it took, and some context, and they nest, so you can see that a slow checkout spent most of its time waiting on one database call three services deep. It is one of the three pillars of observability, alongside metrics and logs.</p><h2 class="heading" style="text-align:left;" id="how-it-actually-works">How it actually works</h2><p class="paragraph" style="text-align:left;">The real trick is context propagation. When a request enters your system it is given a trace identifier, and every service passes that identifier along as it calls the next one, usually in an HTTP header that follows the W3C Trace Context standard. Each service adds its own spans to the same trace. Break that chain, with a service that does not forward the header, and the trace fragments. This is why a shared standard like <a class="link" href="https://www.masteringobservability.com/p/what-is-opentelemetry?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-distributed-tracing" target="_blank" rel="noopener noreferrer nofollow">OpenTelemetry</a> matters: it makes propagation work across services written by different teams in different languages.</p><h2 class="heading" style="text-align:left;" id="why-it-matters">Why it matters</h2><ul><li><p class="paragraph" style="text-align:left;"><b>It shows the path, not just the symptom.</b> Instead of &quot;the app is slow&quot;, you see exactly which hop is slow, and by how much.</p></li><li><p class="paragraph" style="text-align:left;"><b>It survives complexity.</b> The more services you have, the more you need it. Tracing is what keeps a microservices estate debuggable.</p></li><li><p class="paragraph" style="text-align:left;"><b>It turns finger-pointing into evidence.</b> The trace shows which service owns the delay, so the conversation is about the data, not whose fault it is.</p></li></ul><h2 class="heading" style="text-align:left;" id="the-honest-ledger">The honest ledger</h2><ul><li><p class="paragraph" style="text-align:left;"><b>It is only as good as your instrumentation.</b> Miss a service, or break context propagation, and you get a partial trace that can mislead more than it helps.</p></li><li><p class="paragraph" style="text-align:left;"><b>Sampling is a trade-off.</b> Recording every request is expensive, so most teams sample. Sample too hard and you miss the rare failure you most wanted to see.</p></li><li><p class="paragraph" style="text-align:left;"><b>It shows where, not always why.</b> A slow span points at the location. You still often need the logs and metrics around it to find the root cause.</p></li><li><p class="paragraph" style="text-align:left;"><b>It has a cost.</b> High-volume tracing adds overhead and data, and rich span attributes multiply cardinality, which is where bills grow.</p></li></ul><h2 class="heading" style="text-align:left;" id="so-is-distributed-tracing-for-you">So, is distributed tracing for you?</h2><p class="paragraph" style="text-align:left;">If you run microservices, serverless, or any request that crosses more than one service, distributed tracing is close to essential, and the modern way to get it is OpenTelemetry, often with no code changes through eBPF-based tools. If you run a single, simple application it matters less, though it still helps. Either way, start with your most important user journeys rather than trying to trace everything at once.</p><hr class="content_break"><p class="paragraph" style="text-align:left;"><b>Where has tracing saved you, or let you down?</b> I would like to hear it, especially the sampling war stories. Reply, or <a class="link" href="https://calendly.com/managing_observability/30min?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-distributed-tracing" target="_blank" rel="noopener noreferrer nofollow">book a slot</a> and tell me what you found.</p><div class="custom_html"><table width="100%" align="center" cellpadding="0" cellspacing="0" border="0" style="max-width:600px;margin:0 auto;"><tbody><tr><td style="background-color:#2F9E8D;height:4px;line-height:4px;font-size:4px;">&nbsp;</td></tr><tr><td style="background-color:#0D2127;padding:24px 24px 18px 24px;"><p style="margin:0 0 8px 0;font-family:'Space Mono','Courier New',monospace;font-size:12px;letter-spacing:2px;color:#74DDCD;text-transform:uppercase;">Get the next one</p><p style="margin:0;font-family:'DM Sans',Helvetica,Arial,sans-serif;font-size:22px;line-height:1.3;color:#ffffff;font-weight:700;">One signal a week. No noise.</p></td></tr><tr><td style="background-color:#ffffff;padding:20px 24px 24px 24px;font-family:'DM Sans',Helvetica,Arial,sans-serif;"><p style="margin:0 0 18px 0;font-size:16px;line-height:1.5;color:#1a1a1a;">If this was useful, Metrics & Mayhem sends one short, practical piece like it to IT operations leaders most weeks. No fluff, no vendor noise.</p><table cellpadding="0" cellspacing="0" border="0"><tbody><tr><td style="background-color:#2F9E8D;border-radius:4px;"><a href="https://masteringobservability.com?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-distributed-tracing" style="display:inline-block;padding:12px 22px;font-family:'DM Sans',Helvetica,Arial,sans-serif;font-size:16px;font-weight:700;color:#0D2127;text-decoration:none;">Join free →</a></td></tr></tbody></table><p style="margin:16px 0 0 0;font-size:14px;line-height:1.5;color:#44595b;">Prefer to start with the book? Read a <a href="https://www.masteringobservability.com/metrics-and-mayhem/free-chapter?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-distributed-tracing" style="color:#0D2127;text-decoration:underline;">free chapter</a>.</p></td></tr></tbody></table></div><h3 class="heading" style="text-align:left;" id="related-from-the-bytesize-series">Related from the byte-size series</h3><ul><li><p class="paragraph" style="text-align:left;"><a class="link" href="https://www.masteringobservability.com/p/what-is-opentelemetry?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-distributed-tracing" target="_blank" rel="noopener noreferrer nofollow">byte-size: What Is OpenTelemetry?</a></p></li><li><p class="paragraph" style="text-align:left;"><a class="link" href="https://www.masteringobservability.com/p/opentelemetry-collector-implementation-guide?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-distributed-tracing" target="_blank" rel="noopener noreferrer nofollow">OpenTelemetry Collector Implementation Guide</a></p></li><li><p class="paragraph" style="text-align:left;"><a class="link" href="https://www.masteringobservability.com/p/what-is-ebpf?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-distributed-tracing" target="_blank" rel="noopener noreferrer nofollow">byte-size: What Is eBPF?</a></p></li><li><p class="paragraph" style="text-align:left;"><a class="link" href="https://www.masteringobservability.com/p/what-is-odigos?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-distributed-tracing" target="_blank" rel="noopener noreferrer nofollow">byte-size: What Is Odigos?</a></p></li></ul><h3 class="heading" style="text-align:left;" id="sources-further-reading">Sources / further reading</h3><ul><li><p class="paragraph" style="text-align:left;">OpenTelemetry observability primer (traces): <a class="link" href="https://opentelemetry.io/docs/concepts/observability-primer/?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-distributed-tracing" target="_blank" rel="noopener noreferrer nofollow">opentelemetry.io</a></p></li><li><p class="paragraph" style="text-align:left;">W3C Trace Context standard: <a class="link" href="https://www.w3.org/TR/trace-context/?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-distributed-tracing" target="_blank" rel="noopener noreferrer nofollow">w3.org/TR/trace-context</a></p></li></ul></div><div class='beehiiv__footer'><br class='beehiiv__footer__break'><hr class='beehiiv__footer__line'><a target="_blank" class="beehiiv__footer_link" style="text-align: center;" href="https://www.beehiiv.com/?utm_campaign=563ed1a0-ed59-41d5-9cbc-745acfe22b43&utm_medium=post_rss&utm_source=mastering_observability">Powered by beehiiv</a></div></div>
  ]]></content:encoded>
</item>

      <item>
  <title>byte-size: What Is Odigos?</title>
  <description>Zero-code observability with eBPF and OpenTelemetry, and the one question to ask before you bet your estate on it.</description>
      <enclosure url="https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/357712e9-5800-4002-bd84-73fe59419786/odigos_bytesize_og_1200x630.png" length="49006" type="image/png"/>
  <link>https://www.masteringobservability.com/p/what-is-odigos</link>
  <guid isPermaLink="true">https://www.masteringobservability.com/p/what-is-odigos</guid>
  <pubDate>Fri, 26 Jun 2026 19:06:05 +0000</pubDate>
  <atom:published>2026-06-26T19:06:05Z</atom:published>
    <category><![CDATA[Newsletter]]></category>
    <category><![CDATA[Observability]]></category>
    <category><![CDATA[Byte Size]]></category>
    <category><![CDATA[Ebpf]]></category>
    <category><![CDATA[Opentelemetry]]></category>
  <content:encoded><![CDATA[
    <div class='beehiiv'><style>
  .bh__table, .bh__table_header, .bh__table_cell { border: 1px solid #C9DCDC; }
  .bh__table_cell { padding: 4px; background-color: #FFFFFF; }
  .bh__table_cell p { color: #2D2D2D; font-family: 'Space Grotesk',Helvetica,Arial,sans-serif !important; overflow-wrap: break-word; }
  .bh__table_header { padding: 6px; background-color:#EAF6F3; }
  .bh__table_header p { color: #222222; font-family:'Space Grotesk',Helvetica,Arial,sans-serif !important; overflow-wrap: break-word; }
</style><div class='beehiiv__body'><p class="paragraph" style="text-align:left;"><i>Zero-code observability with eBPF, and the one question to ask before you bet your estate on it.</i></p><p class="paragraph" style="text-align:left;">Every so often a demo actually stops me. Odigos did. I sat down with their team on 7 June 2026, and I will be honest up front: I was impressed. That is exactly why I went and did the homework afterwards, because the tools that impress you in a demo are the ones you most need to pressure-test. So I asked them the questions a careful buyer asks, then checked their answers against the wider world. Here is the balanced read, the good and the awkward, so you can decide for yourself.</p><h2 class="heading" style="text-align:left;" id="what-is-odigos">What is Odigos?</h2><p class="paragraph" style="text-align:left;">Odigos is an open-source observability control plane that auto-instruments your applications using <a class="link" href="https://ebpf.io/?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-odigos" target="_blank" rel="noopener noreferrer nofollow">eBPF</a> and <a class="link" href="https://opentelemetry.io/?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-odigos" target="_blank" rel="noopener noreferrer nofollow">OpenTelemetry</a>, with no code changes, no SDK to import, and no redeploy. You point it at your Kubernetes workloads and it starts producing distributed traces from running services. It is a <a class="link" href="https://www.ycombinator.com/companies/odigos-technologies-inc?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-odigos" target="_blank" rel="noopener noreferrer nofollow">Y Combinator company</a> (W23), the <a class="link" href="https://github.com/odigos-io/odigos?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-odigos" target="_blank" rel="noopener noreferrer nofollow">open-source project</a> has well over 3,000 GitHub stars, and it was built by people who maintain OpenTelemetry itself. That last part matters more than the star count.</p><h2 class="heading" style="text-align:left;" id="first-what-is-e-bpf">First, what is eBPF?</h2><p class="paragraph" style="text-align:left;">The whole approach rests on eBPF, so it is worth thirty seconds. eBPF is a feature built into the Linux operating system that lets small, safe programs run deep inside the kernel, the layer every application sits on top of. In plain terms, it lets a tool watch what your software is actually doing from the outside, without you changing a line of code and without meaningfully slowing anything down. It began life in networking and security, and observability tools worked out they could use the same trick to capture traces and metrics almost for free. That is the engine under Odigos.</p><h2 class="heading" style="text-align:left;" id="what-it-does-well">What it does well</h2><p class="paragraph" style="text-align:left;">The pitch is simple enough to explain to someone who has never written a line of code. Pick your namespaces, and it instruments them on the fly. In their demo I watched a slow checkout get traced to a database lock wait, and a stalled AI call traced to a token limit, with nobody touching the application code.</p><p class="paragraph" style="text-align:left;">Three things stand out once you look past the polish:</p><ul><li><p class="paragraph" style="text-align:left;"><b>It cracks the hard languages.</b> Compiled languages like Go have always been painful to instrument without code changes. Odigos uses eBPF and the official <a class="link" href="https://github.com/odigos-io/opentelemetry-go-instrumentation?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-odigos" target="_blank" rel="noopener noreferrer nofollow">OpenTelemetry Go instrumentation</a> to get traces out of them anyway.</p></li><li><p class="paragraph" style="text-align:left;"><b>It is vendor-neutral by design.</b> Because it emits OpenTelemetry and runs inside your own cluster, you point the data at whatever backend you like and switch later without re-instrumenting. Instrument once, route anywhere. Nothing has to leave your environment, which makes the security conversation shorter.</p></li><li><p class="paragraph" style="text-align:left;"><b>It is helping write the standard, not just sell against it.</b> Odigos is a founding contributor to the <a class="link" href="https://opentelemetry.io/blog/2025/obi-announcing-first-release/?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-odigos" target="_blank" rel="noopener noreferrer nofollow">OpenTelemetry eBPF Instrumentation</a> (OBI) project, alongside Grafana Labs, Splunk and Coralogix. A vendor shaping the open standard is a different proposition from one fencing you in.</p></li></ul><h2 class="heading" style="text-align:left;" id="the-honest-ledger">The honest ledger</h2><p class="paragraph" style="text-align:left;">No tool is a free lunch, and the gaps here are real.</p><ul><li><p class="paragraph" style="text-align:left;"><b>It is Kubernetes and Linux native.</b> eBPF lives in the node kernel and needs a reasonably modern one. When I pushed them on older estates, they were refreshingly straight about it: you need a Linux kernel of 4.18 or newer and Kubernetes 1.21 or newer, and while bare metal is supported, they told me it was not yet running in production at a customer. The shiny cluster is the easy part. The legacy back end nobody can safely re-instrument, the VMs and the older systems that hold the actual money, are where the hard questions live. Coverage beyond Kubernetes is the question, not the headline.</p></li><li><p class="paragraph" style="text-align:left;"><b>Depth has limits today.</b> Independent testers have found database calls surfacing as generic operations rather than full statements, and the richest Go auto-instrumentation reported as part of the paid tier rather than the free one. Worth confirming against your own workloads.</p></li><li><p class="paragraph" style="text-align:left;"><b>The space is consolidating fast.</b> With Beyla folded into OBI and several vendors converging on one standard, some capabilities are still maturing and the competitive map is moving under everyone&#39;s feet.</p></li><li><p class="paragraph" style="text-align:left;"><b>Speed is not the same as meaning.</b> eBPF shows you the plumbing in minutes. It does not tell you whether the customer journey is actually working. You still have to define your SLOs and your business logic. That work does not disappear, it just starts sooner.</p></li></ul><h2 class="heading" style="text-align:left;" id="is-odigos-the-right-tool-for-you">Is Odigos the right tool for you?</h2><p class="paragraph" style="text-align:left;">If you run a modern, Kubernetes-native estate and you are tired of waiting on an engineering backlog to get distributed tracing, Odigos is genuinely worth a serious look, and the vendor-neutral, in-cluster design makes it an easy one to trial. If your weight sits in legacy infrastructure, treat it as one piece of the picture and ask hard questions about coverage and data depth before you bet the estate on it.</p><p class="paragraph" style="text-align:left;">That is the question I would want answered before I put my name next to anything. Not because the demo was weak. Because it was strong, and strong demos are exactly when you should ask.</p><hr class="content_break"><p class="paragraph" style="text-align:left;"><b>Do you have an Odigos story, good or bad?</b> I would like to hear it, especially from anyone running it past a proof of concept. Reply, or <a class="link" href="https://calendly.com/managing_observability/30min?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-odigos" target="_blank" rel="noopener noreferrer nofollow">book a slot</a> and tell me what you found.</p><p class="paragraph" style="text-align:left;"><i>Disclosure: I have spoken with the Odigos team and may collaborate with them on future content. The opinion here is my own and stays honest, including the parts they would rather I left out.</i></p><h3 class="heading" style="text-align:left;" id="related-from-the-bytesize-series">Related from the byte-size series</h3><ul><li><p class="paragraph" style="text-align:left;"><a class="link" href="https://www.masteringobservability.com/p/what-is-ebpf?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-odigos" target="_blank" rel="noopener noreferrer nofollow">byte-size: What Is eBPF?</a></p></li><li><p class="paragraph" style="text-align:left;"><a class="link" href="https://www.masteringobservability.com/p/what-is-grafana-alloy?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-odigos" target="_blank" rel="noopener noreferrer nofollow">byte-size: What Is Grafana Alloy</a></p></li><li><p class="paragraph" style="text-align:left;"><a class="link" href="https://www.masteringobservability.com/p/opentelemetry-collector-implementation-guide?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-odigos" target="_blank" rel="noopener noreferrer nofollow">OpenTelemetry Collector Implementation Guide</a></p></li></ul><h3 class="heading" style="text-align:left;" id="sources-further-reading">Sources / further reading</h3><ul><li><p class="paragraph" style="text-align:left;">Odigos open-source project and docs: <a class="link" href="https://github.com/odigos-io/odigos?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-odigos" target="_blank" rel="noopener noreferrer nofollow">github.com/odigos-io/odigos</a></p></li><li><p class="paragraph" style="text-align:left;">Odigos on Y Combinator (W23): <a class="link" href="https://www.ycombinator.com/companies/odigos-technologies-inc?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-odigos" target="_blank" rel="noopener noreferrer nofollow">ycombinator.com</a></p></li><li><p class="paragraph" style="text-align:left;">OpenTelemetry eBPF Instrumentation (OBI), first release: <a class="link" href="https://opentelemetry.io/blog/2025/obi-announcing-first-release/?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-odigos" target="_blank" rel="noopener noreferrer nofollow">opentelemetry.io</a></p></li><li><p class="paragraph" style="text-align:left;">OTel Go auto-instrumentation (eBPF): <a class="link" href="https://github.com/odigos-io/opentelemetry-go-instrumentation?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-odigos" target="_blank" rel="noopener noreferrer nofollow">github.com/odigos-io</a></p></li><li><p class="paragraph" style="text-align:left;">Grafana Beyla / OBI background: <a class="link" href="https://grafana.com/oss/beyla-ebpf/?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-odigos" target="_blank" rel="noopener noreferrer nofollow">grafana.com</a></p></li></ul><div class="custom_html"><table width="100%" align="center" cellpadding="0" cellspacing="0" border="0" style="max-width:600px;margin:0 auto;"><tbody><tr><td style="background-color:#2F9E8D;height:4px;line-height:4px;font-size:4px;">&nbsp;</td></tr><tr><td style="background-color:#0D2127;padding:24px 24px 18px 24px;"><p style="margin:0 0 8px 0;font-family:'Space Mono','Courier New',monospace;font-size:12px;letter-spacing:2px;color:#74DDCD;text-transform:uppercase;">Get the next one</p><p style="margin:0;font-family:'DM Sans',Helvetica,Arial,sans-serif;font-size:22px;line-height:1.3;color:#ffffff;font-weight:700;">One signal a week. No noise.</p></td></tr><tr><td style="background-color:#ffffff;padding:20px 24px 24px 24px;font-family:'DM Sans',Helvetica,Arial,sans-serif;"><p style="margin:0 0 18px 0;font-size:16px;line-height:1.5;color:#1a1a1a;">If this was useful, Metrics & Mayhem sends one short, practical piece like it to IT operations leaders most weeks. No fluff, no vendor noise.</p><table cellpadding="0" cellspacing="0" border="0"><tbody><tr><td style="background-color:#2F9E8D;border-radius:4px;"><a href="https://masteringobservability.com?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-odigos" style="display:inline-block;padding:12px 22px;font-family:'DM Sans',Helvetica,Arial,sans-serif;font-size:16px;font-weight:700;color:#0D2127;text-decoration:none;">Join free →</a></td></tr></tbody></table><p style="margin:16px 0 0 0;font-size:14px;line-height:1.5;color:#44595b;">Prefer to start with the book? Read a <a href="https://www.masteringobservability.com/metrics-and-mayhem/free-chapter?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=byte-size-what-is-odigos" style="color:#0D2127;text-decoration:underline;">free chapter</a>.</p></td></tr></tbody></table></div></div><div class='beehiiv__footer'><br class='beehiiv__footer__break'><hr class='beehiiv__footer__line'><a target="_blank" class="beehiiv__footer_link" style="text-align: center;" href="https://www.beehiiv.com/?utm_campaign=490b0351-a50a-42c2-987c-d75ecad999a2&utm_medium=post_rss&utm_source=mastering_observability">Powered by beehiiv</a></div></div>
  ]]></content:encoded>
</item>

      <item>
  <title>Metrics &amp; Mayhem | Signal Drop: The Burnout You Don&#39;t Notice</title>
  <description>Not the dramatic kind. The quiet one, when work and life drift out of alignment.</description>
      <enclosure url="https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/7fd98b2f-70b7-4f45-be46-5a32c2802453/a-minimalist-book-cover-design-featuring_KxY5If-ETL6wh7NYHHN4Bw_W42soZtaTBGrublqjRO3iA.png" length="56661" type="image/jpeg"/>
  <link>https://www.masteringobservability.com/p/metrics-mayhem-signal-drop-the-burnout-you-dont-notice</link>
  <guid isPermaLink="true">https://www.masteringobservability.com/p/metrics-mayhem-signal-drop-the-burnout-you-dont-notice</guid>
  <pubDate>Sat, 29 Nov 2025 15:06:09 +0000</pubDate>
  <atom:published>2025-11-29T15:06:09Z</atom:published>
    <category><![CDATA[Newsletter]]></category>
    <category><![CDATA[Signal Drop]]></category>
    <category><![CDATA[Leadership]]></category>
  <content:encoded><![CDATA[
    <div class='beehiiv'><style>
  .bh__table, .bh__table_header, .bh__table_cell { border: 1px solid #C9DCDC; }
  .bh__table_cell { padding: 4px; background-color: #FFFFFF; }
  .bh__table_cell p { color: #2D2D2D; font-family: 'Space Grotesk',Helvetica,Arial,sans-serif !important; overflow-wrap: break-word; }
  .bh__table_header { padding: 6px; background-color:#EAF6F3; }
  .bh__table_header p { color: #222222; font-family:'Space Grotesk',Helvetica,Arial,sans-serif !important; overflow-wrap: break-word; }
</style><div class='beehiiv__body'><p class="paragraph" style="text-align:left;">This one was recorded after a 3am incident call, close to twenty-four hours awake, with no energy left for pretending.</p><p class="paragraph" style="text-align:left;">The burnout worth watching is not the dramatic kind. It is the quiet one. It shows up when your work and your life drift out of alignment and you are too busy to admit it. Pressure on its own does not break people. Misalignment does, slowly, until a night like that forces the clarity you have been avoiding.</p><p class="paragraph" style="text-align:left;">Notice one place this week where the drift is showing, and say it out loud to someone you trust. Naming it is how it stops being invisible, and how you start to close the gap.</p><p class="paragraph" style="text-align:left;"><b>Listen:</b> <a class="link" href="https://podcasters.spotify.com/pod/show/allan-mann/episodes/Signal-Drop-The-Burnout-You-Dont-Notice-e3bkslr?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=metrics-mayhem-signal-drop-the-burnout-you-don-t-notice" target="_blank" rel="noopener noreferrer nofollow">the Signal Drop, about 5 minutes</a></p><div class="custom_html"><table width="100%" align="center" cellpadding="0" cellspacing="0" border="0" style="max-width:600px;margin:0 auto;"><tbody><tr><td style="background-color:#2F9E8D;height:4px;line-height:4px;font-size:4px;">&nbsp;</td></tr><tr><td style="background-color:#0D2127;padding:24px 24px 18px 24px;"><p style="margin:0 0 8px 0;font-family:'Space Mono','Courier New',monospace;font-size:12px;letter-spacing:2px;color:#74DDCD;text-transform:uppercase;">Get the next one</p><p style="margin:0;font-family:'DM Sans',Helvetica,Arial,sans-serif;font-size:22px;line-height:1.3;color:#ffffff;font-weight:700;">One signal a week. No noise.</p></td></tr><tr><td style="background-color:#ffffff;padding:20px 24px 24px 24px;font-family:'DM Sans',Helvetica,Arial,sans-serif;"><p style="margin:0 0 18px 0;font-size:16px;line-height:1.5;color:#1a1a1a;">If this was useful, Metrics & Mayhem sends one short, practical piece like it to IT operations leaders most weeks. No fluff, no vendor noise.</p><table cellpadding="0" cellspacing="0" border="0"><tbody><tr><td style="background-color:#2F9E8D;border-radius:4px;"><a href="https://masteringobservability.com?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=metrics-mayhem-signal-drop-the-burnout-you-don-t-notice" style="display:inline-block;padding:12px 22px;font-family:'DM Sans',Helvetica,Arial,sans-serif;font-size:16px;font-weight:700;color:#0D2127;text-decoration:none;">Join free →</a></td></tr></tbody></table><p style="margin:16px 0 0 0;font-size:14px;line-height:1.5;color:#44595b;">Prefer to start with the book? Read a <a href="https://www.masteringobservability.com/metrics-and-mayhem/free-chapter?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=metrics-mayhem-signal-drop-the-burnout-you-don-t-notice" style="color:#0D2127;text-decoration:underline;">free chapter</a>.</p></td></tr></tbody></table></div></div><div class='beehiiv__footer'><br class='beehiiv__footer__break'><hr class='beehiiv__footer__line'><a target="_blank" class="beehiiv__footer_link" style="text-align: center;" href="https://www.beehiiv.com/?utm_campaign=af465a4f-a4bb-4e0a-9911-9df05d73ebcb&utm_medium=post_rss&utm_source=mastering_observability">Powered by beehiiv</a></div></div>
  ]]></content:encoded>
</item>

      <item>
  <title>Metrics &amp; Mayhem | Signal Drop: Skip the Theatre</title>
  <description>Real leadership is the quiet, consistent stuff. If it does not improve clarity, stability or reliability, it is theatre.</description>
      <enclosure url="https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/5eca94e2-30f0-4ab9-8d7e-92af3d0edbf5/a-minimalist-book-cover-design-featuring_KxY5If-ETL6wh7NYHHN4Bw_W42soZtaTBGrublqjRO3iA.png" length="56661" type="image/jpeg"/>
  <link>https://www.masteringobservability.com/p/metrics-mayhem-signal-drop-skip-the-theatre</link>
  <guid isPermaLink="true">https://www.masteringobservability.com/p/metrics-mayhem-signal-drop-skip-the-theatre</guid>
  <pubDate>Fri, 28 Nov 2025 08:08:31 +0000</pubDate>
  <atom:published>2025-11-28T08:08:31Z</atom:published>
    <category><![CDATA[Newsletter]]></category>
    <category><![CDATA[Signal Drop]]></category>
    <category><![CDATA[Leadership]]></category>
  <content:encoded><![CDATA[
    <div class='beehiiv'><style>
  .bh__table, .bh__table_header, .bh__table_cell { border: 1px solid #C9DCDC; }
  .bh__table_cell { padding: 4px; background-color: #FFFFFF; }
  .bh__table_cell p { color: #2D2D2D; font-family: 'Space Grotesk',Helvetica,Arial,sans-serif !important; overflow-wrap: break-word; }
  .bh__table_header { padding: 6px; background-color:#EAF6F3; }
  .bh__table_header p { color: #222222; font-family:'Space Grotesk',Helvetica,Arial,sans-serif !important; overflow-wrap: break-word; }
</style><div class='beehiiv__body'><p class="paragraph" style="text-align:left;">A lot of what passes for leadership is performance. The update that exists to look busy. The ceremony that moves nothing.</p><p class="paragraph" style="text-align:left;">The real work is quieter: hygiene, ownership, honest signals. Here is the test I use. If a thing does not improve clarity, stability, or reliability, it is probably theatre. Cut it, and let the team shine instead of watching you perform.</p><p class="paragraph" style="text-align:left;">Find one recurring ritual this week that fails that test. Drop it, and see if anyone misses it.</p><p class="paragraph" style="text-align:left;"><b>Listen:</b> <a class="link" href="https://podcasters.spotify.com/pod/show/allan-mann/episodes/Signal-Drop-Skip-the-Theatre-e3bjbql?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=metrics-mayhem-signal-drop-skip-the-theatre" target="_blank" rel="noopener noreferrer nofollow">the Signal Drop, about 5 minutes</a></p><div class="custom_html"><table width="100%" align="center" cellpadding="0" cellspacing="0" border="0" style="max-width:600px;margin:0 auto;"><tbody><tr><td style="background-color:#2F9E8D;height:4px;line-height:4px;font-size:4px;">&nbsp;</td></tr><tr><td style="background-color:#0D2127;padding:24px 24px 18px 24px;"><p style="margin:0 0 8px 0;font-family:'Space Mono','Courier New',monospace;font-size:12px;letter-spacing:2px;color:#74DDCD;text-transform:uppercase;">Get the next one</p><p style="margin:0;font-family:'DM Sans',Helvetica,Arial,sans-serif;font-size:22px;line-height:1.3;color:#ffffff;font-weight:700;">One signal a week. No noise.</p></td></tr><tr><td style="background-color:#ffffff;padding:20px 24px 24px 24px;font-family:'DM Sans',Helvetica,Arial,sans-serif;"><p style="margin:0 0 18px 0;font-size:16px;line-height:1.5;color:#1a1a1a;">If this was useful, Metrics & Mayhem sends one short, practical piece like it to IT operations leaders most weeks. No fluff, no vendor noise.</p><table cellpadding="0" cellspacing="0" border="0"><tbody><tr><td style="background-color:#2F9E8D;border-radius:4px;"><a href="https://masteringobservability.com?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=metrics-mayhem-signal-drop-skip-the-theatre" style="display:inline-block;padding:12px 22px;font-family:'DM Sans',Helvetica,Arial,sans-serif;font-size:16px;font-weight:700;color:#0D2127;text-decoration:none;">Join free →</a></td></tr></tbody></table><p style="margin:16px 0 0 0;font-size:14px;line-height:1.5;color:#44595b;">Prefer to start with the book? Read a <a href="https://www.masteringobservability.com/metrics-and-mayhem/free-chapter?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=metrics-mayhem-signal-drop-skip-the-theatre" style="color:#0D2127;text-decoration:underline;">free chapter</a>.</p></td></tr></tbody></table></div></div><div class='beehiiv__footer'><br class='beehiiv__footer__break'><hr class='beehiiv__footer__line'><a target="_blank" class="beehiiv__footer_link" style="text-align: center;" href="https://www.beehiiv.com/?utm_campaign=abb5357e-0f52-4b32-b215-440374671161&utm_medium=post_rss&utm_source=mastering_observability">Powered by beehiiv</a></div></div>
  ]]></content:encoded>
</item>

      <item>
  <title>Metrics &amp; Mayhem | Signal Drop: The Basics Are Not Optional</title>
  <description>Most outages are not surprises. They are a skipped basic, done while chasing something shiny.</description>
      <enclosure url="https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/7e02a6e5-4be4-46b0-880d-6b40a3ee3cb6/a-minimalist-book-cover-design-featuring_KxY5If-ETL6wh7NYHHN4Bw_W42soZtaTBGrublqjRO3iA.png" length="56661" type="image/jpeg"/>
  <link>https://www.masteringobservability.com/p/metrics-mayhem-signal-drop-the-basics-are-not-optional</link>
  <guid isPermaLink="true">https://www.masteringobservability.com/p/metrics-mayhem-signal-drop-the-basics-are-not-optional</guid>
  <pubDate>Wed, 03 Dec 2025 02:40:16 +0000</pubDate>
  <atom:published>2025-12-03T02:40:16Z</atom:published>
    <category><![CDATA[Newsletter]]></category>
    <category><![CDATA[Signal Drop]]></category>
    <category><![CDATA[Reliability]]></category>
  <content:encoded><![CDATA[
    <div class='beehiiv'><style>
  .bh__table, .bh__table_header, .bh__table_cell { border: 1px solid #C9DCDC; }
  .bh__table_cell { padding: 4px; background-color: #FFFFFF; }
  .bh__table_cell p { color: #2D2D2D; font-family: 'Space Grotesk',Helvetica,Arial,sans-serif !important; overflow-wrap: break-word; }
  .bh__table_header { padding: 6px; background-color:#EAF6F3; }
  .bh__table_header p { color: #222222; font-family:'Space Grotesk',Helvetica,Arial,sans-serif !important; overflow-wrap: break-word; }
</style><div class='beehiiv__body'><p class="paragraph" style="text-align:left;">Most outages I have ever seen were not surprises. They were someone skipping a simple check because they were chasing something shiny.</p><p class="paragraph" style="text-align:left;">The basics are boring on purpose. Patience, consistency, the same checks done the same way. Reliability is not built in the heroic moment. It is built in the dull one, when you are tired, bored, annoyed, and ready to throw the laptop into the sea, and you do the check anyway.</p><p class="paragraph" style="text-align:left;">Pick the one basic your team keeps deferring until things calm down. Do it this week, while things are not calm. That is the whole point.</p><p class="paragraph" style="text-align:left;"><b>Listen:</b> <a class="link" href="https://podcasters.spotify.com/pod/show/allan-mann/episodes/Signal-Drop-The-Basics-Are-Not-Optional-e3bq5pb?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=metrics-mayhem-signal-drop-the-basics-are-not-optional" target="_blank" rel="noopener noreferrer nofollow">the Signal Drop, about 5 minutes</a></p><div class="custom_html"><table width="100%" align="center" cellpadding="0" cellspacing="0" border="0" style="max-width:600px;margin:0 auto;"><tbody><tr><td style="background-color:#2F9E8D;height:4px;line-height:4px;font-size:4px;">&nbsp;</td></tr><tr><td style="background-color:#0D2127;padding:24px 24px 18px 24px;"><p style="margin:0 0 8px 0;font-family:'Space Mono','Courier New',monospace;font-size:12px;letter-spacing:2px;color:#74DDCD;text-transform:uppercase;">Get the next one</p><p style="margin:0;font-family:'DM Sans',Helvetica,Arial,sans-serif;font-size:22px;line-height:1.3;color:#ffffff;font-weight:700;">One signal a week. No noise.</p></td></tr><tr><td style="background-color:#ffffff;padding:20px 24px 24px 24px;font-family:'DM Sans',Helvetica,Arial,sans-serif;"><p style="margin:0 0 18px 0;font-size:16px;line-height:1.5;color:#1a1a1a;">If this was useful, Metrics & Mayhem sends one short, practical piece like it to IT operations leaders most weeks. No fluff, no vendor noise.</p><table cellpadding="0" cellspacing="0" border="0"><tbody><tr><td style="background-color:#2F9E8D;border-radius:4px;"><a href="https://masteringobservability.com?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=metrics-mayhem-signal-drop-the-basics-are-not-optional" style="display:inline-block;padding:12px 22px;font-family:'DM Sans',Helvetica,Arial,sans-serif;font-size:16px;font-weight:700;color:#0D2127;text-decoration:none;">Join free →</a></td></tr></tbody></table><p style="margin:16px 0 0 0;font-size:14px;line-height:1.5;color:#44595b;">Prefer to start with the book? Read a <a href="https://www.masteringobservability.com/metrics-and-mayhem/free-chapter?utm_source=www.masteringobservability.com&utm_medium=newsletter&utm_campaign=metrics-mayhem-signal-drop-the-basics-are-not-optional" style="color:#0D2127;text-decoration:underline;">free chapter</a>.</p></td></tr></tbody></table></div></div><div class='beehiiv__footer'><br class='beehiiv__footer__break'><hr class='beehiiv__footer__line'><a target="_blank" class="beehiiv__footer_link" style="text-align: center;" href="https://www.beehiiv.com/?utm_campaign=e34c85d0-b9ad-4064-bb63-ae320a62ed6d&utm_medium=post_rss&utm_source=mastering_observability">Powered by beehiiv</a></div></div>
  ]]></content:encoded>
</item>

  </channel>
</rss>
