Hey Everyone!

Today we’ll be talking about

• The Architecture of Grab's Data Lake

  • Introduction to Data Storage Formats

  • Design Choices when picking a Data Storage Format

  • High Throughput vs. Low Throughput Data at Grab

  • Using Avro and Merge on Read for High Throughput Data

  • Using Parquet and Copy on Write for Low Throughput Data

• Tech Snippets

  • A Detailed Guide to Software Architecture Documentation

  • The really important job interview questions engineers should ask (but don’t)

  • How Figma overhauled their Performance Testing Framework

  • Is this the simplest sorting algorithm ever?

  • How SQLite got 10x faster for Analytical Queries

Our API Could Probably Run a Small Country.

Your API can barely handle 10 requests without throwing a tantrum, and you wanna scale? 

CarsXE is so powerful, it could run a startup, a Fortune 500, or a small dictatorship—and still have bandwidth left for your grandma’s car blog. 

Instant license plate decoding, real-time vehicle specs, and enterprise-level security. Your move.

Or keep using that weak API that breaks when someone sneezes.

sponsored

The Architecture of Grab's Data Lake

Grab is one of the largest technology companies in Southeast Asia, with over 35 million monthly users. They run a "super-app" that offers ride-sharing, food delivery, banking, and communication all within a single app.

As you might guess, operating all these services generates a lot of data. Grab's data analysts need to comb through this data for insights that can help the company improve its operations.

The data is primarily stored in a data lake. Some of the data is high-throughput and gets frequent updates (multiple updates every second/minute). Other data is low-throughput and is rarely updated (updated daily/weekly).

Grab needs to store this data efficiently and also allow data analysts to run ad-hoc queries on it without it being too costly.

One crucial design choice is picking the right storage format for the data on the data lake. Choosing the wrong format can make data storage significantly more expensive and make gaining insights from the data more difficult.

We'll first explore data storage formats, discussing the tradeoffs involved and some commonly used technologies.

Then, we'll talk about what Grab chose and why. You can read the full article by Grab here.

Introduction to Data Storage Formats

Let's say you have a bunch of sensor data from a weather satellite. If you need to store it on S3, how would you do it?

Do you just upload the CSV? What if the file is 50 gigabytes and has many repeat values? Would you pick a format that compresses the data?

The format you choose for encoding your data is crucial.

Here are some of the tradeoffs you might consider:

• Human Readability - It's pretty easy for you to read a CSV or a JSON file however these formats aren't very efficient. Compressed formats like Protobuf are much smaller but not human-readable.

• Row vs. Column-oriented - In a row-oriented format, you put data in the same row next to each other on disk. In a column-oriented format, you put data in the same column next to each other on disk. This choice has a ton of effects on read/write performance, compression efficiency and more. We did a deep dive on Row vs. Column-oriented databases that you can read here.

• Schema Evolution - how easy is changing the data schema over time without breaking existing data? Some formats support adding/removing fields.

• Compression - How efficiently can you compress the data? Compression can save storage costs and reduce latencies.

• Split-ability - How easy is it to divide the file format into smaller chunks? Split-ability is important if you need to do parallel processing on the data.

• Ecosystem Compatibility - Is the format commonly used? Does it have good support with tools like Spark, Redshift, Snowflake , etc.

Some common formats that you'll frequently see are

• JSON - this is a text-based format so it's human-readable. Almost every programming language will have support for JSON. I'm assuming you've already used JSON before but you can read more here if you haven't.

• CSV - a text format for storing data in a table-like way. Each line corresponds to a row and every value in the line is separated by a comma. CSV is also human-readable but it's not an efficient way to store data.

• Avro - a binary data serialization framework developed within the Hadoop ecosystem. Data on disk is row-oriented and compressed. Avro supports robust schema evolution. Learn more here.

• Parquet - developed in 2013 as a joint effort between Twitter and Cloudera. Parquet is column-oriented and provides efficient compression on disk. It supports schema evolution so you can add/remove columns without breaking compatibility. Read more here.

• ORC - created in 2013 at Facebook. It's optimized for large streaming reads and efficient data compression. Read more.

Data Storage Formats at Grab

For Grab, they looked at the characteristics of their data sources and used that to determine their storage formats.

One characteristic they used was throughput.

• High Throughput data is updated/changed frequently (several times a second). An example is the stream of booking events from customers who book a ride share.

• Low Throughput data could be transaction events generated from a nightly batch process.

High Throughput Data

For High Throughput data, Grab uses Apache Avro with a strategy called Merge on Read (MOR).

Here's the main operations with Merge on Read:

• Write Operations - When data is written, it's appended to the end of a log file. This is much more efficient than merging it in the current data and reduces the latency of writes.

• Read Operations - When you need to read data, the base file is combined with the updates in the log file to provide the latest view. This can make reads more costly as you have to merge the updated data.

• Periodic Compaction - To prevent reads from becoming too costly, updates in the log files are periodically merged with the base files. This limits the number of past updates you must merge during a read.

Low Throughput Data

For low throughput data, Grab uses Parquet with Copy on Write (CoW).

Here's the main operations for Copy on Write:

• Write Operations - Whenever there's a write, you create a new version of the file that includes the latest change. You can also keep the previous version for consistency and rollback purposes. This helps prevent data corruption, inconsistent reads, and more.

• Read Operations - You read the latest versioned data file. Reads are faster than Merge on Read since there is no merge process.

Our API Could Probably Run a Small Country.

Your API can barely handle 10 requests without throwing a tantrum, and you wanna scale? 

CarsXE is so powerful, it could run a startup, a Fortune 500, or a small dictatorship—and still have bandwidth left for your grandma’s car blog. 

Instant license plate decoding, real-time vehicle specs, and enterprise-level security. Your move.

Or keep using that weak API that breaks when someone sneezes.

sponsored

Tech Snippets

How Figma overhauled their Performance Testing Framework

Initially, Figma relied on a single macbook for their in-house performance testing system. As you might imagine, they eventually had to find a more scalable solution.

This is an interesting read from the Figma engineering blog on how the company built a new system to spot performance regressions in the app.

They ultimately shipped two systems: a cloud based system that covered the majority of tests and a hardware system for highly targeted tests.

www.figma.com/blog/keeping-figma-fast

Hey Everyone!

Today we’ll be talking about

• How Pinterest Optimized Video Playback

  • Introduction to Adaptive Bitrate Streaming, HLS and DASH

  • Why Pinterest was experiencing high startup latency for videos

  • Embedding the video manifest files in their metadata API and improving performance with caching

• Tech Snippets

  • A Dive into Database Fundamentals

  • Digital Signatures and how to avoid them

  • The Architecture of Open Source Applications

The Hidden Costs of Building Product Integrations In-House

Building integrations in-house might seem like the obvious choice—full control, endlessly customizable, and no external dependencies. But what about the costs you don’t see upfront?

From ongoing maintenance to unexpected edge cases, hidden costs of building every third party integration in-house can quickly add up and drain engineering resources.

This article from Nango spells out the gotchas you should watch out for before deciding to go 100% in-house, including:

• Development costs - Understand how engineering hours and project scope can balloon over time

• Maintenance overhead - See all the nitty gritty your team needs to take care of in order to have stable and scalable integrations in prod

• Opportunity cost - Consider how building integrations in-house can pull focus away from core product innovation

sponsored

How Pinterest Optimized Video Playback

Pinterest is a social media platform that helps you discover ideas and inspiration related to whatever you’re interested in (cooking recipes, home decor, clothing, etc)

The platform was launched in 2010 and it’s grown to over 500 million monthly active users. Pinterest is now publicly traded and valued at more than $20 billion.

Like every other social platform, video content is one of the most popular mediums on Pinterest. When you’re serving videos to your users, one of your highest priorities should be to minimize any buffering and startup delay. With the modern day attention span, even having your video buffer for a couple of seconds can result in a huge number of users leaving your app.

Pinterest engineering published a great blog post on how they optimized video playback and reduced startup latency by 36%.

We’ll give some context on how videos are streamed, what protocols are involved and what Pinterest did to optimize playback.

Introduction to Adaptive Bitrate Streaming

When you’re delivering video to users, one technique that’s used universally nowadays is Adaptive Bitrate Streaming.

This is where you take the video and encode it at multiple bitrates and resolutions and store them all on your server. When a user wants to play the video, their phone will select the optimal rendition based on factors like network bandwidth and device characteristics to minimize any buffering.

With Adaptive Bitrate Streaming, the player can also switch dynamically between different bitrates. If the internet connection weakens while they’re watching a video on their phone, ABR allows the player to automatically switch to a lower bitrate stream so playback can be smooth without any buffering interruptions.

When the network improves, the player will automatically switch back to the higher bitrate stream to provide better video quality.

Basics of Adaptive Bitrate Streaming

There are different protocols you can use for Adaptive Bitrate Streaming, but they share some common fundamentals.

• Chunking - the video file is broken up into small chunks. Each chunk ranges from 2-10 seconds in length.

• Multiple Renditions - Each chunk is encoded at multiple bitrates and resolutions. 

• Manifest File - a manifest file contains metadata about the available renditions for every chunk, including their bitrates and resolutions.

• Dynamic Selection - the user’s video player will use the manifest file to determine which chunk to download based on the current network conditions and device capabilities.

The most widely adopted Adaptive Bitrate protocols are HTTP Live Streaming (HLS) and Dynamic Adaptive Streaming over HTTP (DASH).

You’ve probably realized this by the names but HLS and DASH are both based on HTTP.

HTTP Live Streaming (HLS)

HLS was developed by Apple in 2009 and it’s one of the earliest and most widely adopted ABR protocols. The video stream is broken into small, HTTP-based downloads. It supports both live and on-demand streaming.

It’s developed and maintained by Apple so it’s natively supported on iOS, macOS and Safari.

HLS uses .m3u8 manifest files to guide the player in selecting the most appropriate video chunks based on real-time network conditions. 

Dynamic Adaptive Streaming over HTTP (DASH)

DASH was created by a consortium of companies led by MPEG (Moving Picture Experts Group). The protocol was first published in 2012 and it currently powers platforms like YouTube and Netflix.

DASH uses .mpd manifest files to provide metadata about the available renditions and chunk URLs.

Video Streaming at Pinterest

At Pinterest, both HLS and DASH are used for delivering videos across iOS and Android platforms, respectively. 

• HLS: Utilized for video streaming on iOS devices through Apple’s AVPlayer, accounting for approximately 70% of video playback sessions on iOS apps.

• DASH: Employed for video streaming on Android devices using ExoPlayer, representing around 55% of video playback sessions on Android.

One of the key metrics Pinterest measures for video performance is startup latency - the time it takes for a video to begin playing after a user initiates playback.

As we stated above, both HLS and and DASH require a manifest file before you can initiate video playback. With HLS, you might have to download additional manifest files (for the specific rendition) after downloading the main one.

 Only after you download the manifest file can the video player start downloading the first few chunks of the video. This is the primary contributor to users’ perceived latency.

The Pinterest team decided to eliminate the latency from the round trips by embedding all the relevant manifest files in the original API response. When a user first requests metadata for a video (thumbnail, title, etc.), the API response to that request will also contain the manifest files of the video.

During playback, the player can swiftly access the manifest information locally and immediately start downloading video chunks.

Reducing API Response Time

When Pinterest started including manifest files in the API responses, the primary issue they faced was increased latency for the API endpoint. The backend now had to retrieve manifest files before it could respond with video metadata.

They were able to solve this issue with caching. They added a MemCache layer into the manifest serving process to cache the most popular video manifest files.

Here’s the new process for retrieving manifest files.

1. API Request - a client requests Pins metadata

2. Manifest Embeddings - the Backend retrieves manifest files from S3, serializes them and embeds the bytes within the API response

3. MemCache - Subsequent requests for popular video manifest files are served immediately from the MemCache caching layer.

4. Response Delivery - the API delivers the payload with the manifest data embedded

Results

With this new setup, Pinterest was able to see a 36.7% reduction in p90 startup latency on iOS. They also saw a 12.3% reduction in the number of users who had to wait longer than 1 second for a video to start.

Why is OAuth still hard in 2025?

OAuth in 2025 is like JavaScript browser APIs in 2008. It’s a complete mess. But why is OAuth still hard?

This article breaks down the key reasons and all the gotchas to think about when implementing OAuth.

Read this article to see what to watch out for with OAuth.

Tech Snippets

A Dive into Database Fundamentals

This article is a terrific deep dive into database fundamentals. It talks about ACID properties (and how they can be implemented in a toy database), storage engines, indexes, LSM trees, the CAP theorem and much more.

It gives a fantastic overview of many of the most interesting topics in databases and also provides additional resources if you want to go further.

tontinton.com/posts/database-fundementals

Digital signatures and how to avoid them

Neil Madden is the author of API Security in Action and has worked as a Security Architect and software engineer.

He wrote a really interesting blog post on digital signatures, how they work and when they should be used (and when they should be avoided). He talks about the fragility of current signature schemes and how they can lose important contextual details. For many use-cases, Madden suggests using simpler methods like HMAC for authentication instead of digital signatures.

neilmadden.blog/2024/09/18/digital-signatures-and-how-to-avoid-them

Hey Everyone!

Today we’ll be talking about

• How GitHub Rebuilt their Push Processing System

  • GitHub rebuilt their system for handling code pushes to make it more decoupled

  • We’ll give a brief overview of decoupled architectures and their pros/cons

  • After, we’ll talk about why GitHub split their push processing system from a single job to a group of smaller, independent jobs.

• Tech Snippets

  • Pair Programming Antipatterns

  • Resources for CTOs

  • Nine ways to shoot yourself in the foot with Postgres

  • How CloudFlare debugged an issue with dropped packets

How GitHub Rebuilt their Push Processing System

GitHub has over 420 million repos and 100 million registered users. Every month, the platform handled over 500 million code pushes from 8.5 million developers.

Whenever someone pushes code to a GitHub repository, this kicks off a chain of tasks.

GitHub has to do things like:

• Update the repo with the latest commits

• Dispatch any Push webhooks

• Trigger relevant GitHub workflows

And much more. In fact, Github has 20 different services that run in response to a developer pushing code.

Previously, push requests were handled by a single, enormous job (called RepositoryPushJob). Whenever you pushed code, GitHub’s Ruby on Rails monolith would enqueue RepositoryPushJob and handle all the underlying sub-jobs in a sequential manner.

However, the company faced issues with this approach and decided to switch to a more decoupled architecture with Apache Kafka. GitHub published a great blog post delving into the details.

In this article, we’ll first give an overview of decoupled architectures and the pros/cons. Then, we’ll talk about the changes Github made.

Overview of Decoupled Architectures

If a user does some significant action on your app, you might have to perform a series of different jobs. If you have a video sharing website, you’ll have a bunch of different things that need to be done when someone uploads a video (encoding, generating transcripts, checking for piracy, etc.)

A key question is how coupled you want these jobs to be.

On one side of the spectrum, you can combine these sub-jobs (encoding, generating transcripts, etc.) into a single larger job (ProcessVideo) and then execute them in a sequential manner.

On the other side, you can have different services for each of the jobs and have them execute in parallel. Whenever a user uploads a video, you’ll add an event with the video’s details to an event streaming platform (like Kafka). Then, the different sub-jobs will consume the event and run independently.

Some of the pros of a decoupled approach are

• Scalability - Each of the components can be scaled up/down independently based on their specific load and demand.

• Fault Isolation - Components are independent so a failure in one component can be contained (hopefully).

• Easier Development - Each component can be deployed independently. This makes things much easier if you have a large number of developers working together. 

Cons with the decoupled approach include

• Increased Complexity - Managing coordination between the independent components can be much more complex. You may need additional tooling for observability and monitoring.

• System Overhead - Communication between components can become slower, especially if it requires a network request. If there are network requests involved, then you’ll have significantly more latency and failures that you’ll have to deal with.

• Data Consistency - You’ll need to think about making sure data is consistent across the components. 

 GitHub’s Old Tightly Coupled Architecture

Previously, GitHub used a single massive job called RepositoryPushJob for handling pushes. This job managed all the sub-jobs and triggered them one after another in a sequential series of steps.

However, the GitHub team was facing quite a few issues with this approach

• Difficulty with Retries - If RepositoryPushJob failed then it would have to be retried. However, this caused issues with some of the sub-jobs that were not idempotent (you couldn’t run them multiple times). For example, sending multiple push webhooks could cause issues with clients that were receiving the webhooks. 

• Huge Blast Radius - The fact that jobs were set in a sequential series of steps meant that later sub-jobs had an implicit dependency on initial sub-jobs. As you increase the number of sub-jobs in RepositoryPushJob, the probability of failure increases.

• Too Slow - Having a super long sequential process is bad for latency. The sub-jobs at the end of RepositoryPushJob had to wait for the sub-jobs in the beginning. This structure led to unnecessary latency for many user-facing push tasks (over a second in some cases).

GitHub New Architecture

To decrease the coupling in the push system, GitHub decided to break up RepositoryPushJob into smaller, independent jobs.

They looked at each of the sub-jobs in RepositoryPushJob and grouped them based on dependencies, retry-ability, owning service, etc.  Each group of sub-jobs was placed into an independent job with a clear owner and appropriate retry configuration.

Whenever a developer pushes to a repo, GitHub will add a new event to Kafka. A Kafka consumer service will monitor the Kafka topic and consume the events.

If there’s a new event, the service will enqueue all the independent background jobs onto a job queue for processing. A dedicated pool of worker nodes will then handle the jobs in the queue.

In order to catch any issues, GitHub built extensive observability to monitor the flow of events through the pipeline.

Results

GitHub has seen great results with the new system. Some of the improvements include

• Reliability Improvements - The old RepositoryPushJob was able to fully process 99.987% of pushes with no failures. The new pipeline is able to fully process 99.999% of pushes.

• Lower Latency - GitHub saw a notable decrease in the pull request sync time with a drop of nearly 33% (in the P50 time).

• Smaller Blast Radius - previously, an issue with a single step in RepositoryPushJob could impact all subsequent sub-jobs. Now, failures are much more isolated and there’s a smaller blast radius for when things go wrong.

Tech Snippets

Debugging Dropped Packets at Cloudflare

This is an interesting article that delves into a problem Cloudflare was facing where they had drops in bandwidth and failing API requests after making a change to their load balancers. They eventually traced the problem to a bug in the linux kernel.

Terain Stock is a software engineer at Cloudflare and he wrote a post delving into packet handling and using tools like pwru to debug network issues and kprobe for kernel issues.blog.cloudflare.com/lost-in-transit-debugging-dropped-packets-from-negative-header-lengths

https://blog.cloudflare.com/lost-in-transit-debugging-dropped-packets-from-negative-header-lengths/

Resources for CTOs

Resources for CTOs This is a great github repo with resources for CTOs (or aspiring CTOs).

It contains resources on software development processes, architecture, product management, hiring and much more!

github.com/kuchin/awesome-cto

Hey Everyone!

Today we’ll be talking about

• How GitHub Rebuilt their Push Processing System

  • GitHub rebuilt their system for handling code pushes to make it more decoupled

  • We’ll give a brief overview of decoupled architectures and their pros/cons

  • After, we’ll talk about why GitHub split their push processing system from a single job to a group of smaller, independent jobs.

• Tech Snippets

  • Pair Programming Antipatterns

  • Resources for CTOs

  • Nine ways to shoot yourself in the foot with Postgres

  • How CloudFlare debugged an issue with dropped packets

Protect your App with WorkOS Radar

Does your app get fake signups, throwaway emails, or users abusing your free tier?

Or worse, bot attacks and brute force attempts?

WorkOS Radar can block all this and more. A simple API gives you advanced device fingerprinting that can detect bad actors, bots, and suspicious behaviors.

Your users trust you. Let’s keep it that way.

sponsored

How GitHub Rebuilt their Push Processing System

GitHub has over 420 million repos and 100 million registered users. Every month, the platform handled over 500 million code pushes from 8.5 million developers.

Whenever someone pushes code to a GitHub repository, this kicks off a chain of tasks.

GitHub has to do things like:

• Update the repo with the latest commits

• Dispatch any Push webhooks

• Trigger relevant GitHub workflows

And much more. In fact, Github has 20 different services that run in response to a developer pushing code.

Previously, push requests were handled by a single, enormous job (called RepositoryPushJob). Whenever you pushed code, GitHub’s Ruby on Rails monolith would enqueue RepositoryPushJob and handle all the underlying sub-jobs in a sequential manner.

However, the company faced issues with this approach and decided to switch to a more decoupled architecture with Apache Kafka. GitHub published a great blog post delving into the details.

In this article, we’ll first give an overview of decoupled architectures and the pros/cons. Then, we’ll talk about the changes Github made.

Overview of Decoupled Architectures

If a user does some significant action on your app, you might have to perform a series of different jobs. If you have a video sharing website, you’ll have a bunch of different things that need to be done when someone uploads a video (encoding, generating transcripts, checking for piracy, etc.)

A key question is how coupled you want these jobs to be.

On one side of the spectrum, you can combine these sub-jobs (encoding, generating transcripts, etc.) into a single larger job (ProcessVideo) and then execute them in a sequential manner.

On the other side, you can have different services for each of the jobs and have them execute in parallel. Whenever a user uploads a video, you’ll add an event with the video’s details to an event streaming platform (like Kafka). Then, the different sub-jobs will consume the event and run independently.

Some of the pros of a decoupled approach are

• Scalability - Each of the components can be scaled up/down independently based on their specific load and demand.

• Fault Isolation - Components are independent so a failure in one component can be contained (hopefully).

• Easier Development - Each component can be deployed independently. This makes things much easier if you have a large number of developers working together. 

Cons with the decoupled approach include

• Increased Complexity - Managing coordination between the independent components can be much more complex. You may need additional tooling for observability and monitoring.

• System Overhead - Communication between components can become slower, especially if it requires a network request. If there are network requests involved, then you’ll have significantly more latency and failures that you’ll have to deal with.

• Data Consistency - You’ll need to think about making sure data is consistent across the components. 

 GitHub’s Old Tightly Coupled Architecture

Previously, GitHub used a single massive job called RepositoryPushJob for handling pushes. This job managed all the sub-jobs and triggered them one after another in a sequential series of steps.

However, the GitHub team was facing quite a few issues with this approach

• Difficulty with Retries - If RepositoryPushJob failed then it would have to be retried. However, this caused issues with some of the sub-jobs that were not idempotent (you couldn’t run them multiple times). For example, sending multiple push webhooks could cause issues with clients that were receiving the webhooks. 

• Huge Blast Radius - The fact that jobs were set in a sequential series of steps meant that later sub-jobs had an implicit dependency on initial sub-jobs. As you increase the number of sub-jobs in RepositoryPushJob, the probability of failure increases.

• Too Slow - Having a super long sequential process is bad for latency. The sub-jobs at the end of RepositoryPushJob had to wait for the sub-jobs in the beginning. This structure led to unnecessary latency for many user-facing push tasks (over a second in some cases).

GitHub New Architecture

To decrease the coupling in the push system, GitHub decided to break up RepositoryPushJob into smaller, independent jobs.

They looked at each of the sub-jobs in RepositoryPushJob and grouped them based on dependencies, retry-ability, owning service, etc.  Each group of sub-jobs was placed into an independent job with a clear owner and appropriate retry configuration.

Whenever a developer pushes to a repo, GitHub will add a new event to Kafka. A Kafka consumer service will monitor the Kafka topic and consume the events.

If there’s a new event, the service will enqueue all the independent background jobs onto a job queue for processing. A dedicated pool of worker nodes will then handle the jobs in the queue.

In order to catch any issues, GitHub built extensive observability to monitor the flow of events through the pipeline.

Results

GitHub has seen great results with the new system. Some of the improvements include

• Reliability Improvements - The old RepositoryPushJob was able to fully process 99.987% of pushes with no failures. The new pipeline is able to fully process 99.999% of pushes.

• Lower Latency - GitHub saw a notable decrease in the pull request sync time with a drop of nearly 33% (in the P50 time).

• Smaller Blast Radius - previously, an issue with a single step in RepositoryPushJob could impact all subsequent sub-jobs. Now, failures are much more isolated and there’s a smaller blast radius for when things go wrong.

Protect your App with WorkOS Radar

Does your app get fake signups, throwaway emails, or users abusing your free tier?

Or worse, bot attacks and brute force attempts?

WorkOS Radar can block all this and more. A simple API gives you advanced device fingerprinting that can detect bad actors, bots, and suspicious behaviors.

Your users trust you. Let’s keep it that way.

sponsored

Tech Snippets

Debugging Dropped Packets at Cloudflare

This is an interesting article that delves into a problem Cloudflare was facing where they had drops in bandwidth and failing API requests after making a change to their load balancers. They eventually traced the problem to a bug in the linux kernel.

Terain Stock is a software engineer at Cloudflare and he wrote a post delving into packet handling and using tools like pwru to debug network issues and kprobe for kernel issues.blog.cloudflare.com/lost-in-transit-debugging-dropped-packets-from-negative-header-lengths

https://blog.cloudflare.com/lost-in-transit-debugging-dropped-packets-from-negative-header-lengths/

Resources for CTOs

Resources for CTOs This is a great github repo with resources for CTOs (or aspiring CTOs).

It contains resources on software development processes, architecture, product management, hiring and much more!

github.com/kuchin/awesome-cto

Hey Everyone!

Today we’ll be talking about

• The Architecture of Uber’s ETL Platform

  • Introduction to ETL

  • Tools used for ETL

  • Architecture of Sparkle, Uber’s ETL framework built on Apache Spark

• Tech Snippets

  • Software Testing Anti-Patterns

  • Free University Courses for Learning CS

  • How to Get Better at Finishing Your Side Projects

  • 15 Life and Work Principles from Jensen Huang (CEO of Nvidia)

The Architecture of Uber’s ETL Platform

Uber is the largest ride sharing company in the world with over 150 million monthly active users and approximately 25 million daily trips.

With this scale comes a huge amount of data (and a cloud-bill that’s larger than the GDP of a small island nation). Uber generates petabytes of data daily from ride history, logs, payment transactions, etc.

This data needs to be extracted from the various data sources (payment processor, OLTP database, logs, etc.) and then loaded into data warehouses, data lakes, machine learning platforms and more.

To do this, Uber relies on ETL (Extract, Transform, Load) processes. The Uber engineering team published a terrific blog post talking about exactly how they handle ETL at their scale. They have 20,000+ critical data pipelines and 3,000+ engineers who use this system.

Introduction to ETL

Extract, Transform, Load (ETL) is the process where you

1. Extract - you extract data from the various data sources (places where data is created/temporarily stored). This can be a transactional database, payment processor, a CRM (Salesforce or HubSpot), message queue, etc.

2. Transform - you clean, validate and standardize the data. You might need to check for duplicates, handle missing values, join the data with another dataset and more.

3. Load - you load the data into various data sinks. This can be a data warehouse like Google BigQuery, a data lake like HDFS, archival storage like Amazon Glacier or something else.

Building data pipelines for ETL can be quite painful. You’ll need to consider several things:

• Data Integrity - ensure the accuracy and consistency in your data. You’ll need to check for duplicate records, missing values, inconsistent formatting, outlier values and more.

• Schema Evolution - as business requirements change, the data you’ll be processing will change. You’ll need to account for new fields, data type changes, deprecated fields, etc.

• Monitoring/Debugging - you’ll need logging for the different stages of your pipeline and real-time alerts for failures/performance issues so you minimize downtime (and don’t lose data)

• Scalability - the pipeline shouldn’t require a complete re-architecture as your data volume grows. You may also have to deal with bursts in incoming data depending on the usage patterns.

• Reliability and Failure Recovery - For some systems, you might need to guarantee at least once processing. You’ll have to make sure that the system rarely goes down and that you have a process in place to minimize data loss in case of crashes.

• Compliance - you might have to consider internal data governance/privacy policies when doing transformations.

Some common tools used for ETL include

• Apache Spark - an open-source engine for large-scale data processing. It’s a great choice for complex ETL jobs at scale.

• dbt (data build tool) - a toolkit for building data pipelines that encourages software engineering best practices like version control,testing, code review and more.

• Apache Airflow - a popular open-source platform for orchestrating workflows. You can schedule, monitor and manage your ETL pipelines with Python.

• AWS Glue - fully managed, serverless ETL service from Amazon. 

• Google Cloud Dataflow - fully managed service for ETL from Google Cloud

In 2023, Uber migrated all their batch workloads to Apache Spark. Recently, they built Sparkle, a framework on top of Apache Spark with the goal of simplifying data pipeline development and testing.

ETL at Uber with Sparkle

As your ETL jobs get more and more complex, it becomes crucial to use software engineering best practices when writing/maintaining them. Observability, version control, testing, documentation, etc. are a couple of best practices that have become increasingly adopted in the data community.

Leading this charge is dbt, a data engineering platform that helps you apply these best practices to your data transformations.

However, at Uber, switching from Spark to an entirely new ETL tool wasn’t possible. The scale of Uber’s platform, the developer learning curve and investment required just wasn’t worth it. (try telling your boss you need to rewrite 20,000 mission-critical data pipelines)

Instead, the Uber team decided to build Sparkle, a framework on top of Apache Spark that lets engineers write configuration-based modular ETL jobs. Sparkle added features for observability, testing, data lineage tracking and more.

The core idea behind Sparkle is modularity. Rather than writing complex, monolithic Spark jobs, engineers break their ETL logic down into a series of smaller, reusable modules. Each module can be in SQL, Java/Scala or Python and they’re defined with YAML. Check the blog post for samples of what Sparkle jobs look like.

Developers can just focus on the business logic around their data pipeline. Sparkle will handle infrastructure and boilerplate with pre-built components like

• Connectors - handles all the connection details to pull data from all the various data sources at Uber

• Readers/Writers - handles translating data into different formats like Parquet, JSON, Avro, etc.

• Observability - provides logging, metrics and data lineage tracking

• Testing - you can write unit tests for your modules using mock data and SQL assertions to make sure your transformations are doing what you expect.

Tech Snippets

The Art of Finishing

If you’re a serial project starter, this article has some useful strategies to help you finally cross the finish line.

It talks about the main reasons why we avoid finishing things (fear of imperfection, illusion of productivity, etc.) and how you can overcome these blockers.

www.bytedrum.com/posts/art-of-finishing

15 Life and Work Principles from Jensen Huang (CEO of Nvidia)

Jensen Huang has a unique leadership style. He has 60+ direct reports but does no 1:1 meetings. Instead, he tries to emphasize transparency and open discourse (where everyone in the company is involved in decision making).

He also looks to chase “zero-billion dollar markets“, where Nvidia is doing something completely new where they have no competitors.

Read the full article for the rest of his work/leadership principles.

creatoreconomy.so/p/15-life-and-work-principles-from-jensen

Hey Everyone!

Today we’ll be talking about

• The Architecture of Uber’s ETL Platform

  • Introduction to ETL

  • Tools used for ETL

  • Architecture of Sparkle, Uber’s ETL framework built on Apache Spark

• Tech Snippets

  • Software Testing Anti-Patterns

  • Free University Courses for Learning CS

  • How to Get Better at Finishing Your Side Projects

  • 15 Life and Work Principles from Jensen Huang (CEO of Nvidia)

The Developer’s Guide to Product Integrations: Build vs. Buy

Building product integrations is no small feat—balancing timelines, resources, and technical complexity can feel overwhelming.

Should you build integrations in-house, or is it better to leverage third-party solutions? 

Nango wrote a fantastic, in-depth guide that walks you through everything you need to know to make an informed choice. They talk about the trade-offs involved and offer practical tips to make your decision easier.

In the guide, you’ll learn:

• Key Considerations - Understand the costs, risks, and benefits of building vs. buying integrations.

• When to Build - Discover scenarios where in-house development gives you the most control and flexibility.

• When to Buy - Learn when leveraging pre-built solutions can accelerate timelines and reduce maintenance overhead.

sponsored

The Architecture of Uber’s ETL Platform

Uber is the largest ride sharing company in the world with over 150 million monthly active users and approximately 25 million daily trips.

With this scale comes a huge amount of data (and a cloud-bill that’s larger than the GDP of a small island nation). Uber generates petabytes of data daily from ride history, logs, payment transactions, etc.

This data needs to be extracted from the various data sources (payment processor, OLTP database, logs, etc.) and then loaded into data warehouses, data lakes, machine learning platforms and more.

To do this, Uber relies on ETL (Extract, Transform, Load) processes. The Uber engineering team published a terrific blog post talking about exactly how they handle ETL at their scale. They have 20,000+ critical data pipelines and 3,000+ engineers who use this system.

Introduction to ETL

Extract, Transform, Load (ETL) is the process where you

1. Extract - you extract data from the various data sources (places where data is created/temporarily stored). This can be a transactional database, payment processor, a CRM (Salesforce or HubSpot), message queue, etc.

2. Transform - you clean, validate and standardize the data. You might need to check for duplicates, handle missing values, join the data with another dataset and more.

3. Load - you load the data into various data sinks. This can be a data warehouse like Google BigQuery, a data lake like HDFS, archival storage like Amazon Glacier or something else.

Building data pipelines for ETL can be quite painful. You’ll need to consider several things:

• Data Integrity - ensure the accuracy and consistency in your data. You’ll need to check for duplicate records, missing values, inconsistent formatting, outlier values and more.

• Schema Evolution - as business requirements change, the data you’ll be processing will change. You’ll need to account for new fields, data type changes, deprecated fields, etc.

• Monitoring/Debugging - you’ll need logging for the different stages of your pipeline and real-time alerts for failures/performance issues so you minimize downtime (and don’t lose data)

• Scalability - the pipeline shouldn’t require a complete re-architecture as your data volume grows. You may also have to deal with bursts in incoming data depending on the usage patterns.

• Reliability and Failure Recovery - For some systems, you might need to guarantee at least once processing. You’ll have to make sure that the system rarely goes down and that you have a process in place to minimize data loss in case of crashes.

• Compliance - you might have to consider internal data governance/privacy policies when doing transformations.

Some common tools used for ETL include

• Apache Spark - an open-source engine for large-scale data processing. It’s a great choice for complex ETL jobs at scale.

• dbt (data build tool) - a toolkit for building data pipelines that encourages software engineering best practices like version control,testing, code review and more.

• Apache Airflow - a popular open-source platform for orchestrating workflows. You can schedule, monitor and manage your ETL pipelines with Python.

• AWS Glue - fully managed, serverless ETL service from Amazon. 

• Google Cloud Dataflow - fully managed service for ETL from Google Cloud

In 2023, Uber migrated all their batch workloads to Apache Spark. Recently, they built Sparkle, a framework on top of Apache Spark with the goal of simplifying data pipeline development and testing.

ETL at Uber with Sparkle

As your ETL jobs get more and more complex, it becomes crucial to use software engineering best practices when writing/maintaining them. Observability, version control, testing, documentation, etc. are a couple of best practices that have become increasingly adopted in the data community.

Leading this charge is dbt, a data engineering platform that helps you apply these best practices to your data transformations.

However, at Uber, switching from Spark to an entirely new ETL tool wasn’t possible. The scale of Uber’s platform, the developer learning curve and investment required just wasn’t worth it. (try telling your boss you need to rewrite 20,000 mission-critical data pipelines)

Instead, the Uber team decided to build Sparkle, a framework on top of Apache Spark that lets engineers write configuration-based modular ETL jobs. Sparkle added features for observability, testing, data lineage tracking and more.

The core idea behind Sparkle is modularity. Rather than writing complex, monolithic Spark jobs, engineers break their ETL logic down into a series of smaller, reusable modules. Each module can be in SQL, Java/Scala or Python and they’re defined with YAML. Check the blog post for samples of what Sparkle jobs look like.

Developers can just focus on the business logic around their data pipeline. Sparkle will handle infrastructure and boilerplate with pre-built components like

• Connectors - handles all the connection details to pull data from all the various data sources at Uber

• Readers/Writers - handles translating data into different formats like Parquet, JSON, Avro, etc.

• Observability - provides logging, metrics and data lineage tracking

• Testing - you can write unit tests for your modules using mock data and SQL assertions to make sure your transformations are doing what you expect.

Connect your SaaS product to Salesforce in hours, not weeks

Stop wasting weeks wrestling with Salesforce's API.

Nango wrote a terrific guide showing how you can build a robust Salesforce integration in just 3 hours.

This guide breaks down the entire process with actionable tips and insights that’ll save your team dozens of hours.

sponsored

Tech Snippets

The Art of Finishing

If you’re a serial project starter, this article has some useful strategies to help you finally cross the finish line.

It talks about the main reasons why we avoid finishing things (fear of imperfection, illusion of productivity, etc.) and how you can overcome these blockers.

www.bytedrum.com/posts/art-of-finishing

15 Life and Work Principles from Jensen Huang (CEO of Nvidia)

Jensen Huang has a unique leadership style. He has 60+ direct reports but does no 1:1 meetings. Instead, he tries to emphasize transparency and open discourse (where everyone in the company is involved in decision making).

He also looks to chase “zero-billion dollar markets“, where Nvidia is doing something completely new where they have no competitors.

Read the full article for the rest of his work/leadership principles.

creatoreconomy.so/p/15-life-and-work-principles-from-jensen

Hey Everyone!

Today we’ll be talking about

• How Pinterest Stores and Transfers Hundreds of Terabytes of Data Daily

  • Introduction to Change Data Capture

  • The Design Goals for CDC at Pinterest

  • Architecture of Pinterest’s CDC System

• Tech Snippets

  • The Programmer’s Brain

  • A Curated List of Cryptography Resources and Links

  • Memory Allocation Visualized

  • Getting Things Done in a Chaotic Environment

How to Pick Technologies for your Tech Stack

One of the hardest decisions you’ll have to make is around what technologies your team adopts. A wrong decision can be extremely costly and take years to reverse. On the other hand, not making a decision can be just as costly (lost revenue, poor developer productivity, etc.)

Product for Engineers wrote a fantastic blog post on their advice for choosing technologies to adopt.

Some of their tips include

1. Prioritize based on set Criteria - There will always be some shiny new toy that your team can adopt. Instead, prioritize based on problems your team is facing. This can be excessive costs, scaling challenges, or a customer need.

2. Mimic the Real World when Evaluating - The engineers who will be using the technology should have significant sway in the decision. They should be able to test the technology in production (safely) and build proof of concepts before deciding.

3. Ensure you consider technical AND business factors - You should talk to all stakeholders and clarify what the set of evaluation criteria are. Some potential criteria include performance, cost, reliability, support, flexibility and more.

Subscribe to Product for Engineers for the rest of their tips on picking technologies. It’s free!

sponsored

How Pinterest built a Change Data Capture System

Pinterest is a social media platform that lets users share images/links for things they’re interested in. You can use the site to get low-carb pasta recipes, find destinations for a wedding or get inspiration for a DIY project (and learn that DIY projects are way harder than they look).

Pinterest first launched in 2010 and they’ve scaled to hundreds of millions of users with billions of monthly visits to the platform.

As you might imagine, Pinterest handles a massive amount of data. Real-time data processing is crucial for delivering personalized recommendations, detecting fraudulent accounts, reporting results to advertisers and more.

Change Data Capture (CDC) is a critical tool that Pinterest uses to power their data infrastructure. The engineering team published a fantastic blog post talking about how they built a generic CDC system for all the online databases at Pinterest. This system handles millions of database queries per second and transfers hundreds of terabytes per day.

Introduction to Change Data Capture (CDC)

Change Data Capture is a set of software design patterns that lets you track database changes in real-time. It captures data modifications like inserts, updates and deletes as they occur. Typically, you’ll set up CDC on your OLTP database (Postgres, MySQL, MongoDB, etc.) and transfer the modifications over to your analytics platform, audit/compliance system, data warehouse, etc.

This ensures that all your systems are kept up-to-date with the most recent data (compared to running nightly batch jobs for syncing changes). It can also reduce the load on your OLTP database since you’re only transferring the changes instead of doing full data loads.

All the commonly used database systems offer a mechanism for tracking database changes. 

Examples are

• Postgres - provides a write-ahead log (WAL) where every change to the database is written to the WAL first. Plugins (like pgoutput) can decode the WAL entries into JSON for a CDC tool to consume. 

• MySQL - provides a binary log (binlog) that records all data modifications. CDC tools can tap into the binlog to capture changes in real-time. 

• MongoDB - uses an operations log (oplog) to store a rolling record of all write operations. CDC systems can tail the oplog to capture changes. MongoDB also provides change streams that you can subscribe to for real-time data changes (change streams are built on top of the oplog).

• DynamoDB - offers DynamoDB streams for a time-ordered sequence of records that captures all the data modifications in your tables. 

• Microsoft SQL Server - provides a built-in feature called Change Data capture (CDC)  that captures insert, update and delete activity and makes the details available in an easily consumed relational format.

• Couchbase - uses the Database Change Protocol (DCP) to stream any mutations that happen within the database. Applications can connect to DCP and get a real-time feed of changes.

• Cassandra - provides a feature called CDC on Apache Cassandra that lets you capture changes on a per-table basis and write them to the local filesystem. 

Debezium is the most popular open source platform for CDC and it supports databases like Postgres, MySQL, MongoDB and more.

Architecture of Pinterest’s Generic CDC System

Before Pinterest’s Generic CDC System, individual teams at the company were building their own CDC solutions. This led to issues around wasted engineering-time, unclear ownership, reliability issues and more.

The Pinterest team decided to solve this by building a Generic CDC solution based on Debezium.

Their goals were

• Distributed - Pinterest has many distributed databases with some having 10,000+ shards. The CDC system should connect to all these shards and transfer the changes.

• Reliability - data should be transferred reliably with a guarantee of at least once processing.

• Scalability - the CDC system should scale to hundreds of terabytes of throughput. Databases at Pinterest receive millions of queries per second.

• Configurability - the CDC system should allow configurability around connectors, failure recovery, load balancing and more.

Here’s the architecture of the system they built

Here’s the components

• Control Plane - manages the configuration and the coordination of the CDC system. It handles things like creating new connectors for new database shards, fixing failed connectors, updating configuration of existing connectors, etc. It runs on a single host and executes its logic on a scheduled basis (typically every minute). 

• Data Plane - the machines in the data plane are responsible for capturing changes from the Pinterest database shards and sending them to Apache Kafka. Each host runs Kafka Connect and runs multiple Debezium connectors (with each connector responsible for a single database shard).

• Kafka - Kafka is used as the message broker to store and transport the change events. The actual CDC data is stored in preconfigured topics. Consumers can subscribe to these topics for updates.

How to run A/B tests for Engineers

Product for Engineers is a fantastic newsletter by PostHog that helps developers learn how to find product-market fit and build apps that users love.

A/B testing and experimentation are crucial for building a feature roadmap, improving conversion rates and accelerating growth. However, many engineers don’t understand the ins-and-outs of how to run these tests effectively (they just leave it to the data scientists).

This edition of Product for Engineers delves into A/B tests and discusses

• The 5 traits of good A/B tests

• How to think about statistical significance and p-values

• Avoiding false positives

And more.

To hone your product skills and read more articles like this, check out Product for Engineers below.

sponsored

Tech Snippets

The Programmer's Brain

This is a practical guide on improving your coding skills by understanding the cognitive processes involved.

The book introduces the Cognitive Dimensions of Notations (CDN) framework, a tool for accessing the usability of codebases. It also provides great insights for onboarding new devs, like limiting tasks to one programming activity and supporting their memory through extensive diagrams.

It’s a really useful read if you want to apply cognitive science principles to be more effective and efficient as a developer.

yoan-thirion.gitbook.io/knowledge-base/software-craftsmanship/the-programmers-brain

A Curated list of Cryptography Resources and Links

If you’re looking to learn more about cryptography, this curated list is fantastic. It covers everything from basic cryptographic theory and algorithms to practical tools/libraries.

github.com/sobolevn/awesome-cryptography#readme

Memory Allocation Visualized

This is a great article that dives into the basics of memory allocation. It explains how programs use malloc and free to dynamically manage memory and the challenges that arise from this.

Check it out if you’re looking to understand the inner workings of memory management.

samwho.dev/memory-allocation

Hey Everyone!

Today we’ll be talking about

• How Cloudflare Defines, Measures and Stops DDoS Attacks

  • DDoS Attacks Explained (Volumetric, Application layer and Protocol layer attacks)

  • How to Measure DDoS Attacks

  • Steps for Protecting from DDoS Attacks

• Tech Snippets

  • Resources on Public Speaking for Software Engineers

  • Things we Learned about LLMs in 2024

  • How to Track Engineering Time

  • Static Search Trees and how they can be 40x Faster than Binary Search

How Cloudflare Defines, Measures and Stops DDoS Attacks

Cloudflare is one of the largest internet infrastructure companies in the world, providing CDN services, DDoS protection, DNS management and more to millions of websites. They operate a massive global network with data centers in over 330 cities worldwide, handling trillions of requests per day.

One of Cloudflare's core offerings is protection against Distributed Denial of Service (DDoS) attacks. Unfortunately, these malicious attacks are a constant threat to websites. In 2024 alone, CloudFlare mitigated over 14.5 million DDoS attacks (an average of 2,200 DDoS attacks per hour).

Recently, the Cloudflare engineering team published an awesome blog post delving into the evolution of DDoS attacks over the past decade. We’ll be summarizing the post and adding some extra context on DDoS attacks.

DDoS Attacks Explained

With a Distributed Denial of Service Attack, a hacker will use many geographically distributed machines to send traffic to a website. These machines usually belong to unsuspecting users and have been infected with malware to make them part of the attacker’s botnet.

The goal is to overwhelm the target’s backend with traffic, so the site can no longer serve legitimate users. The attacker might then request a ransom from the company, promising to stop the DDoS attack if the company pays up.

DDoS Attacks can roughly be split into 3 main types: Volumetric, Application layer and Protocol attacks.

Volumetric Attacks

These attacks are based on brute force techniques where the target server is flooded with data packets to consume bandwidth and server resources.

Volumetric attacks will frequently rely on amplification and reflection.

Amplification is where a request in a certain protocol will result in a much larger response (in terms of the number of bytes); the ratio between the request size and response size is called the Amplification Factor.

Reflection is where the attacker will spoof the source of request packets to be the target victim’s IP address. Servers will be unable to distinguish legitimate requests from spoofed ones so they’ll send the (much larger) response payload to the targeted victim’s servers and unintentionally flood them.

Network Time Protocol (NTP) DDoS attacks are an example of volumetric attacks where you can send a 234-byte spoofed request to an NTP server, which will then send a 48,000 byte response to the target victim. Attackers will repeat this on many different open NTP servers simultaneously to DDoS the victim with all the NTP responses.

Application Layer Attacks

These DDoS attacks target the “top” layer in the OSI model - the application layer. Attackers might flood the backend with HTTP requests, exploit expensive API endpoints, create many SSL/TLS handshakes, etc.

Database DDoS attacks are quite common, where a hacker will look for requests that are particularly database-intensive and then spam those in an attempt to exhaust the database resources. Scaling your database through read replicas takes time, so this attack can be pretty successful.

HTTP Floods are some of the most widely seen layer 7 DDoS attacks, where hackers will spam a web server with HTTP GET/POST requests. Sophisticated hackers will specifically design these to request resources with low usage in order to maximize the number of cache misses the web server has.

Protocol Layer Attacks

Protocol attacks will rely on weaknesses in how particular protocols are designed. Examples of these kinds of exploits are SYN floods, BGP hijacking, Smurf attacks and more.

A SYN flood attack exploits how TCP is designed, specifically the handshake process. The three-way handshake consists of SYN -> SYN-ACK -> ACK,  where the client sends a synchronize (SYN) message to initiate, the server responds with a synchronize-acknowledge (SYN-ACK) message and the client then responds back with an acknowledgement (ACK) message.

In a SYN flood attack, a malicious client will send large volumes of SYN messages to the server, who will then respond back with SYN-ACK. The client will ignore these and never respond back with an ACK message. The server will waste resources (open ports) waiting for the ACK responses from the malicious client. If repeated on a large scale, this can bring the web server down since the server won’t know which requests are legitimate.

How to Measure DDoS Attacks

First of all, defining an individual DDoS attack can be surprisingly difficult. It is not just a one-time spike in requests. A DDoS attack can last for several hours/days and consist of many smaller incidents (also known as pulses).

Cloudflare analyzes a combination of factors to create a “fingerprint” that helps identify different attacks as part of the same DDoS targeting. 

Some factors Cloudflare looks at are:

• Attack Vectors: Are the same attack vectors being used across the set of events?

• Targets: Are all the attacks focused on the same target website/entity?

• Payload Signatures: Do the payloads share anything that could mark them to a certain botnet?

Once they have gotten a rough sense of which pulses are part of the attack, Cloudflare can measure how large the DDoS was.

The main metrics they use are:

• Bits per Second (BPS): Measures the total data transferred per second. This is useful for evaluating network-layer attacks that aim to saturate bandwidth like UDP floods.

• Requests per Second (RPS): Measures the number of protocol requests made each second. This is useful for application-layer attacks (Layer 7).

• Packets per Second (PPS): Represents the number of individual packets sent to the target per second, regardless of size. This is critical for network-layer attacks (Layers 3 and 4) like SYN floods.

How to Protect from DDoS Attacks

Unfortunately, there is no single solution to protecting your service from a DDoS attack. Large companies use many different approaches. Some of them include

• Rate Limiting: This is the first line of defense. You set thresholds on the number of requests your server will accept from a single IP address within a given time frame.

• Caching and CDNs: You can significantly reduce the load on your web server by caching content and using a Content Delivery Network (CDN). CDNs will distribute your files across multiple servers globally, so that the impact of a DDoS attack is spread out.

• Reducing the Attack Surface: Minimize the number of services exposed to the public internet. If you have an endpoint with expensive operations, then protect it through authentication (by requiring an API key).

• Monitoring: You should continuously monitor network traffic to detect anomalies and potential attacks. First, establish a baseline for normal traffic patterns. That way, you can quickly identify unusual spikes or patterns that indicate an attack.

• Machine Learning - services like Cloudflare use machine learning algorithms to identify suspicious traffic patterns in real time. They wrote an interesting blog post on the ML models they use here.

Tech Snippets

Things we learned about LLMs in 2024

Over the course of 2024, a ton of rapid advancements came in LLMs. Multiple models were released that surpassed GPT-4’s capabilities and the competition drove LLM prices down dramatically. Multimodal models also became prevalent with capabilities extending to text, image, audio and video.

However, there’s still a ton of challenges around creating reliable evals, the environmental impact, agents and more.

This is a great post that summarizes the year for LLMs.

simonwillison.net/2024/Dec/31/llms-in-2024

How to Track Engineering Time

This article presents a useful playbook for tracking engineering time. It splits time spent spent into features, bugs/debt and toil.

Analyzing ratios like the time spent on features vs. bugs/debt can be very helpful for making informed decisions about resource allocation and project prioritization.

jacobian.org/2024/feb/7/tracking-engineering-time

Static Search Trees and how they can be 40x faster than Binary Search

Binary Search can be surprisingly inefficient for large datasets due to poor cache utilization. Ragnar Groot Koerkamp tackled this problem by optimizing S+ trees for high-throughput searching.

He wrote a terrific article talking about his optimizations and how he was able to achieve a 40x speedup over binary search.

curiouscoding.nl/posts/static-search-tree

Hey Everyone!

Today we’ll be talking about

• How Cloudflare Defines, Measures and Stops DDoS Attacks

  • DDoS Attacks Explained (Volumetric, Application layer and Protocol layer attacks)

  • How to Measure DDoS Attacks

  • Steps for Protecting from DDoS Attacks

• Tech Snippets

  • Resources on Public Speaking for Software Engineers

  • Things we Learned about LLMs in 2024

  • How to Track Engineering Time

  • Static Search Trees and how they can be 40x Faster than Binary Search

The Complete Guide to OAuth 2.0

OAuth 2.0 is the industry standard for authorization. It lets you grant a third-party application access to data on your Google/Meta/Dropbox account without sharing your account’s password.

If you’re building an app and want to add a “sign in with google” button then you’ll need to understand how OAuth works.

Recently, WorkOS published a fantastic guide on OAuth, covering everything you need to know to implement it.

The guide covers

• Roles and Terminology: Explains the OAuth jargon like Resource Owner, Authorization Server, Refresh Token, etc.

• Tokens and Credentials: Understand the different types of tokens like Access Tokens, Refresh Tokens, and Authorization Codes and more

• OAuth Flows: Dive into the various OAuth flows, including Authorization Code Grant, PKCE, and Client Credentials, and learn when to use each one based on your application needs

Read the full guide to learn more about OAuth 2.0 and how to implement it with WorkOS.

sponsored

How Cloudflare Defines, Measures and Stops DDoS Attacks

Cloudflare is one of the largest internet infrastructure companies in the world, providing CDN services, DDoS protection, DNS management and more to millions of websites. They operate a massive global network with data centers in over 330 cities worldwide, handling trillions of requests per day.

One of Cloudflare's core offerings is protection against Distributed Denial of Service (DDoS) attacks. Unfortunately, these malicious attacks are a constant threat to websites. In 2024 alone, CloudFlare mitigated over 14.5 million DDoS attacks (an average of 2,200 DDoS attacks per hour).

Recently, the Cloudflare engineering team published an awesome blog post delving into the evolution of DDoS attacks over the past decade. We’ll be summarizing the post and adding some extra context on DDoS attacks.

DDoS Attacks Explained

With a Distributed Denial of Service Attack, a hacker will use many geographically distributed machines to send traffic to a website. These machines usually belong to unsuspecting users and have been infected with malware to make them part of the attacker’s botnet.

The goal is to overwhelm the target’s backend with traffic, so the site can no longer serve legitimate users. The attacker might then request a ransom from the company, promising to stop the DDoS attack if the company pays up.

DDoS Attacks can roughly be split into 3 main types: Volumetric, Application layer and Protocol attacks.

Volumetric Attacks

These attacks are based on brute force techniques where the target server is flooded with data packets to consume bandwidth and server resources.

Volumetric attacks will frequently rely on amplification and reflection.

Amplification is where a request in a certain protocol will result in a much larger response (in terms of the number of bytes); the ratio between the request size and response size is called the Amplification Factor.

Reflection is where the attacker will spoof the source of request packets to be the target victim’s IP address. Servers will be unable to distinguish legitimate requests from spoofed ones so they’ll send the (much larger) response payload to the targeted victim’s servers and unintentionally flood them.

Network Time Protocol (NTP) DDoS attacks are an example of volumetric attacks where you can send a 234-byte spoofed request to an NTP server, which will then send a 48,000 byte response to the target victim. Attackers will repeat this on many different open NTP servers simultaneously to DDoS the victim with all the NTP responses.

Application Layer Attacks

These DDoS attacks target the “top” layer in the OSI model - the application layer. Attackers might flood the backend with HTTP requests, exploit expensive API endpoints, create many SSL/TLS handshakes, etc.

Database DDoS attacks are quite common, where a hacker will look for requests that are particularly database-intensive and then spam those in an attempt to exhaust the database resources. Scaling your database through read replicas takes time, so this attack can be pretty successful.

HTTP Floods are some of the most widely seen layer 7 DDoS attacks, where hackers will spam a web server with HTTP GET/POST requests. Sophisticated hackers will specifically design these to request resources with low usage in order to maximize the number of cache misses the web server has.

Protocol Layer Attacks

Protocol attacks will rely on weaknesses in how particular protocols are designed. Examples of these kinds of exploits are SYN floods, BGP hijacking, Smurf attacks and more.

A SYN flood attack exploits how TCP is designed, specifically the handshake process. The three-way handshake consists of SYN -> SYN-ACK -> ACK,  where the client sends a synchronize (SYN) message to initiate, the server responds with a synchronize-acknowledge (SYN-ACK) message and the client then responds back with an acknowledgement (ACK) message.

In a SYN flood attack, a malicious client will send large volumes of SYN messages to the server, who will then respond back with SYN-ACK. The client will ignore these and never respond back with an ACK message. The server will waste resources (open ports) waiting for the ACK responses from the malicious client. If repeated on a large scale, this can bring the web server down since the server won’t know which requests are legitimate.

How to Measure DDoS Attacks

First of all, defining an individual DDoS attack can be surprisingly difficult. It is not just a one-time spike in requests. A DDoS attack can last for several hours/days and consist of many smaller incidents (also known as pulses).

Cloudflare analyzes a combination of factors to create a “fingerprint” that helps identify different attacks as part of the same DDoS targeting. 

Some factors Cloudflare looks at are:

• Attack Vectors: Are the same attack vectors being used across the set of events?

• Targets: Are all the attacks focused on the same target website/entity?

• Payload Signatures: Do the payloads share anything that could mark them to a certain botnet?

Once they have gotten a rough sense of which pulses are part of the attack, Cloudflare can measure how large the DDoS was.

The main metrics they use are:

• Bits per Second (BPS): Measures the total data transferred per second. This is useful for evaluating network-layer attacks that aim to saturate bandwidth like UDP floods.

• Requests per Second (RPS): Measures the number of protocol requests made each second. This is useful for application-layer attacks (Layer 7).

• Packets per Second (PPS): Represents the number of individual packets sent to the target per second, regardless of size. This is critical for network-layer attacks (Layers 3 and 4) like SYN floods.

How to Protect from DDoS Attacks

Unfortunately, there is no single solution to protecting your service from a DDoS attack. Large companies use many different approaches. Some of them include

• Rate Limiting: This is the first line of defense. You set thresholds on the number of requests your server will accept from a single IP address within a given time frame.

• Caching and CDNs: You can significantly reduce the load on your web server by caching content and using a Content Delivery Network (CDN). CDNs will distribute your files across multiple servers globally, so that the impact of a DDoS attack is spread out.

• Reducing the Attack Surface: Minimize the number of services exposed to the public internet. If you have an endpoint with expensive operations, then protect it through authentication (by requiring an API key).

• Monitoring: You should continuously monitor network traffic to detect anomalies and potential attacks. First, establish a baseline for normal traffic patterns. That way, you can quickly identify unusual spikes or patterns that indicate an attack.

• Machine Learning - services like Cloudflare use machine learning algorithms to identify suspicious traffic patterns in real time. They wrote an interesting blog post on the ML models they use here.

The Complete Guide to OAuth 2.0

OAuth 2.0 is the industry standard for authorization. It lets you grant a third-party application access to data on your Google/Meta/Dropbox account without sharing your account’s password.

If you’re building an app and want to add a “sign in with google” button then you’ll need to understand how OAuth works.

Recently, WorkOS published a fantastic guide on OAuth, covering everything you need to know to implement it.

The guide covers

• Roles and Terminology: Explains the OAuth jargon like Resource Owner, Authorization Server, Refresh Token, etc.

• Tokens and Credentials: Understand the different types of tokens like Access Tokens, Refresh Tokens, and Authorization Codes and more

• OAuth Flows: Dive into the various OAuth flows, including Authorization Code Grant, PKCE, and Client Credentials, and learn when to use each one based on your application needs

Read the full guide to learn more about OAuth 2.0 and how to implement it with WorkOS.

sponsored

Tech Snippets

Things we learned about LLMs in 2024

Over the course of 2024, a ton of rapid advancements came in LLMs. Multiple models were released that surpassed GPT-4’s capabilities and the competition drove LLM prices down dramatically. Multimodal models also became prevalent with capabilities extending to text, image, audio and video.

However, there’s still a ton of challenges around creating reliable evals, the environmental impact, agents and more.

This is a great post that summarizes the year for LLMs.

simonwillison.net/2024/Dec/31/llms-in-2024

How to Track Engineering Time

This article presents a useful playbook for tracking engineering time. It splits time spent spent into features, bugs/debt and toil.

Analyzing ratios like the time spent on features vs. bugs/debt can be very helpful for making informed decisions about resource allocation and project prioritization.

jacobian.org/2024/feb/7/tracking-engineering-time

Static Search Trees and how they can be 40x faster than Binary Search

Binary Search can be surprisingly inefficient for large datasets due to poor cache utilization. Ragnar Groot Koerkamp tackled this problem by optimizing S+ trees for high-throughput searching.

He wrote a terrific article talking about his optimizations and how he was able to achieve a 40x speedup over binary search.

curiouscoding.nl/posts/static-search-tree

Hey Everyone!

Today we’ll be talking about

• The Architecture of Canva’s Data Platform

  • Introduction to Snowflake

  • Architecture of Canva’s Data Platform

  • How Canva Monitors their Snowflake Usage to avoid Expensive Surprises

• Tech Snippets

  • The Four Kinds of Optimization

  • Unicode is Harder Than You Think

  • How I Automated My Job Application Process

  • Three Bucket Framework for Engineering Metrics

The Architecture of Canva's Data Platform

Canva is an online graphic design platform that lets you create social media posts, posters, videos, logos and more. They have a ton of pre-built templates that help you create professional looking designs even if you have the visual-design skills of a 4 year old.

The company was founded in 2013 and has exploded in popularity with hundreds of millions of users globally. This hyper growth has caused a ton of interesting engineering challenges, particularly with their data platform.

Canva uses Snowflake for the core of their data platform. They store over 25 petabytes of data and execute over 90 million queries a month. Over two-thirds of employees at Canva use Snowflake in some way (writing SQL queries, relying on business-intelligence dashboards, etc.)

However, Snowflake costs can get out-of-hand very quickly if you don’t effectively monitor and optimize your usage. Canva wrote a fantastic engineering blog on the tooling they built to do this.

Introduction to Snowflake

Snowflake is a cloud data warehouse platform that has grown incredibly quickly since its launch in 2014.

At the time of their founding, companies were mainly using on-prem data warehouses like Teradata or Oracle. However, these were super expensive to scale and maintain. You’d have to purchase a ton of hardware upfront and managing all the infrastructure was a huge pain.

In response, Snowflake built a cloud-native data warehouse that runs on top of AWS, Azure and Google Cloud. You pick the cloud provider that fits with the rest of your infrastructure.

Data in Snowflake is stored in a columnar format in “micro-partitions”. These are contiguous storage units that contain 50-500 mb of data and help massively with speeding up analytical queries.

You can access data in Snowflake with SQL, the API (with connectors for Python, Java, NodeJS, etc.), the web interface and other third party tools.

Some of the core selling points of Snowflake are

• Separation of Storage and Compute - Snowflake allows you to scale your compute resources independently of your storage resources. If you have a small dataset but need extensive processing, then you can scale up your compute resources without having to pay for storage you don’t need. AWS Redshift added this capability with RA3 instances in late 2019.

• Low Management and Security - Snowflake is cloud-native so it has far lower maintenance than an on-prem data warehouse. It also provides a ton of security features like Time Travel that lets you access historical data versions for data recovery and auditing.

• Data Integrations - Snowflake supports both structured and semi-structured data formats like CSV, JSON, Avro, Parquet and more. It also has integrations to a ton of data tools like Kafka, Spark, Tableau, etc.

The main cons of Snowflake are vendor lock-in and (potentially) the costs.

Snowflake’s pricing can be notoriously complex and difficult to predict. Not monitoring/optimizing your usage can quickly result in a very expensive surprise.

Snowflake at Canva

Here are the steps in Canva’s Data Platform

1. Data Ingestion - First party data (generated by services at Canva) is ingested through AWS S3. Third party data (Facebook ad spend results, Google organic search data, etc.) is ingested with Fivetran for ETL

2. Data Transformation - Canva uses dbt for transforming and cleaning raw data into a format that supports analysis and reporting. dbt is an open source tool where you can write your data transformation logic in SQL or Python and organize it into maintainable components. 

3. Data Views - Canva uses Census to synchronize enriched datasets to third-party systems. They also use Looker and Mode for data visualization and exploration.

Canva’s Snowflake Monitoring System

In order to avoid overspending, the Canva team built out an extensive monitoring system around Snowflake and their usage.

They started by using Snowflake’s account_usage.query_history view to collect usage data and they stored this in a dedicated dbt project. They also captured detailed metadata on all their other dbt runs (for their main data transformations) and stored this data in S3.

In order to link specific Snowflake queries with the corresponding dbt models, Canva developed a custom dbt query tagging macro that appended JSON metadata to each query. They could use this to track the usage at a per-query level and then assign costs to individual queries and transformations.

With this foundation, Canva created dashboards that provided Data Platform engineers with real-time metrics on how Snowflake was being used, which teams were incurring costs and how usage could be optimized.

Tech Snippets

Three-Bucket Framework for Engineering Metrics

Abi Noda wrote a terrific article on how engineering leaders can effectively report metrics to stakeholders.

His three-bucket framework is
1. Demonstrate Business Impact with ROI
2. Show System Performance with Uptime, Incidents, Scale
3. Present Developer Effectiveness with SPACE and DORA

newsletter.getdx.com/p/choosing-engineering-metrics

Hey Everyone!

Today we’ll be talking about

• The Architecture of Canva’s Data Platform

  • Introduction to Snowflake

  • Architecture of Canva’s Data Platform

  • How Canva Monitors their Snowflake Usage to avoid Expensive Surprises

• Tech Snippets

  • The Four Kinds of Optimization

  • Unicode is Harder Than You Think

  • How I Automated My Job Application Process

  • Three Bucket Framework for Engineering Metrics

The Architecture of Canva's Data Platform

Canva is an online graphic design platform that lets you create social media posts, posters, videos, logos and more. They have a ton of pre-built templates that help you create professional looking designs even if you have the visual-design skills of a 4 year old.

The company was founded in 2013 and has exploded in popularity with hundreds of millions of users globally. This hyper growth has caused a ton of interesting engineering challenges, particularly with their data platform.

Canva uses Snowflake for the core of their data platform. They store over 25 petabytes of data and execute over 90 million queries a month. Over two-thirds of employees at Canva use Snowflake in some way (writing SQL queries, relying on business-intelligence dashboards, etc.)

However, Snowflake costs can get out-of-hand very quickly if you don’t effectively monitor and optimize your usage. Canva wrote a fantastic engineering blog on the tooling they built to do this.

Introduction to Snowflake

Snowflake is a cloud data warehouse platform that has grown incredibly quickly since its launch in 2014.

At the time of their founding, companies were mainly using on-prem data warehouses like Teradata or Oracle. However, these were super expensive to scale and maintain. You’d have to purchase a ton of hardware upfront and managing all the infrastructure was a huge pain.

In response, Snowflake built a cloud-native data warehouse that runs on top of AWS, Azure and Google Cloud. You pick the cloud provider that fits with the rest of your infrastructure.

Data in Snowflake is stored in a columnar format in “micro-partitions”. These are contiguous storage units that contain 50-500 mb of data and help massively with speeding up analytical queries.

You can access data in Snowflake with SQL, the API (with connectors for Python, Java, NodeJS, etc.), the web interface and other third party tools.

Some of the core selling points of Snowflake are

• Separation of Storage and Compute - Snowflake allows you to scale your compute resources independently of your storage resources. If you have a small dataset but need extensive processing, then you can scale up your compute resources without having to pay for storage you don’t need. AWS Redshift added this capability with RA3 instances in late 2019.

• Low Management and Security - Snowflake is cloud-native so it has far lower maintenance than an on-prem data warehouse. It also provides a ton of security features like Time Travel that lets you access historical data versions for data recovery and auditing.

• Data Integrations - Snowflake supports both structured and semi-structured data formats like CSV, JSON, Avro, Parquet and more. It also has integrations to a ton of data tools like Kafka, Spark, Tableau, etc.

The main cons of Snowflake are vendor lock-in and (potentially) the costs.

Snowflake’s pricing can be notoriously complex and difficult to predict. Not monitoring/optimizing your usage can quickly result in a very expensive surprise.

Snowflake at Canva

Here are the steps in Canva’s Data Platform

1. Data Ingestion - First party data (generated by services at Canva) is ingested through AWS S3. Third party data (Facebook ad spend results, Google organic search data, etc.) is ingested with Fivetran for ETL

2. Data Transformation - Canva uses dbt for transforming and cleaning raw data into a format that supports analysis and reporting. dbt is an open source tool where you can write your data transformation logic in SQL or Python and organize it into maintainable components. 

3. Data Views - Canva uses Census to synchronize enriched datasets to third-party systems. They also use Looker and Mode for data visualization and exploration.

Canva’s Snowflake Monitoring System

In order to avoid overspending, the Canva team built out an extensive monitoring system around Snowflake and their usage.

They started by using Snowflake’s account_usage.query_history view to collect usage data and they stored this in a dedicated dbt project. They also captured detailed metadata on all their other dbt runs (for their main data transformations) and stored this data in S3.

In order to link specific Snowflake queries with the corresponding dbt models, Canva developed a custom dbt query tagging macro that appended JSON metadata to each query. They could use this to track the usage at a per-query level and then assign costs to individual queries and transformations.

With this foundation, Canva created dashboards that provided Data Platform engineers with real-time metrics on how Snowflake was being used, which teams were incurring costs and how usage could be optimized.

Tech Snippets

Three-Bucket Framework for Engineering Metrics

Abi Noda wrote a terrific article on how engineering leaders can effectively report metrics to stakeholders.

His three-bucket framework is
1. Demonstrate Business Impact with ROI
2. Show System Performance with Uptime, Incidents, Scale
3. Present Developer Effectiveness with SPACE and DORA

newsletter.getdx.com/p/choosing-engineering-metrics

Hey Everyone!

Today we’ll be talking about

• The Architecture of Stripe’s Document Database - Stripe wrote a great blog post describing DocDB, their internal database as a service. DocDB is built on MongoDB and stores petabytes of data.

  • Brief Intro to MongoDB and its Benefits

  • Why Stripe built DocDB

  • Architecture of DocDB and how it works

  • Rebalancing Data Shards on DocDB for Efficiency

• Tech Snippets

  • Going from New Grad to Staff Engineer in 3 Years at Meta

  • How Nginx Scales

  • An Engineering Philosophy of “Let It Fail“

  • How to Give Constructive Feedback to Coworkers

The Developer’s Guide to Fine Grained Authorization

As your application grows in complexity, you’ll need to implement a more granular and scalable authorization system.

Fine-grained authorization (FGA) systems are where you define permissions at the resource level and give individual users permissions to read/modify specific resources (a document in google drive, bank account in JP Morgan, etc.). To build an FGA system, you’ll need to provide precision while also handling thousands of authentication requests per second.

WorkOS published an in-depth guide that walks you through everything you need to know about FGA-from designing your data model, leveraging third-party solutions and optimizing your UI.

In the guide, you’ll understand

• FPA Basics - Grasp the the relationship between users, resources and roles

• How to Build Resolver Logic - Implement custom logic to handle complex authorization scenarios

• Centralize vs. Decentralize FGA - how to decide the best architectural approach for your authentication system

Read the full guide to learn more.

sponsored

The Architecture of Stripe’s Document Database

Stripe is one of the largest payment processors in the world. In 2023, they processed over $1 trillion USD of payment volume, and they did this with a 99.999% uptime.

A crucial system that helped the company achieve this is DocDB, Stripe's internal Database as a Service.

Developers at Stripe can use the API for reading/writing data (OLTP reads/writes) and not have to worry about scaling compute, increasing storage, schema changes, etc. They can just focus on the product they're building.

Stripe's Database Infrastructure team published a fantastic blog post delving into the internals of DocDB and how it works.

When Stripe was founded in 2011, the company adopted MongoDB as their online database. They found it to be easier to use than a traditional relational database.

As the company grew to hundreds of terabytes of data, Stripe built DocDB on top of MongoDB to make scaling easier.

DocDB handles dynamic rebalancing between shards, gives fine-grained control over data distribution, ensures data consistency during migrations and more.

In this article, we'll first give a brief overview of MongoDB and then talk about how Stripe designed DocDB.

MongoDB Overview

MongoDB is a document-oriented database. It stores your data in semi-structured documents using BSON ( a binary format that extends JSON).

It was first developed in 2007 and released as an open-source database in 2009 (note - in 2018, they changed their license to be more restrictive).

The database was created by the founders of DoubleClick, the startup that would later get acquired by Google and become Google Ads.

The founders faced scalability and usability issues with traditional relational databases so they built MongoDB with certain design goals:

• Developer-Friendly - Relational databases will store data in structured tables with relationships defined by primary and foreign keys. This doesn't naturally map with the traditional data structures you use in an object-oriented programming language (this issue is called Object-Relational Impedance Mismatch). MongoDB stores data as flexible, JSON-like documents so it's much more natural to map to objects in your code.

• Scalability - Horizontal scalability is built into the core of MongoDB and it supports range, hash and zone-based sharding. Document-oriented databases encourage denormalization (where all related data is embedded into a single document) to minimize joins. Avoiding cross-shard joins is crucial for scalability.

• Schema Flexibility - With a Relational Database, you need to define a schema and ensure that any data you insert follows the schema. Changing the schema means doing a database migration. On the other hand, MongoDB is schemaless. Each document can have different fields and the data types of those fields can vary from document to document.

Why Stripe built DocDB

Stripe originally started with MongoDB. As they grew (at an insanely fast rate), the engineering team desired additional features.

In order to utilize their database infrastructure most efficiently, they needed to transfer data between different shards in their fleet. Stripe has thousands of shards, so managing this can be very complex.

They wanted a solution that gave them complete operational control, so they could move individual data chunks between shards. Additionally, this needs to be done with minimal downtime and strong data consistency (Stripe is dealing with financial data).

To solve this, they built DocDB on top of MongoDB.

Architecture of DocDB

As we mentioned earlier, DocDB is a Database as a Service that Stripe engineers can use through an API.

Developers can send a read/write request to DocDB and it'll first go to a Database Proxy server.

The proxy server will first check for things like access controls, potential bugs, scalability, etc.

Then, it'll figure out which specific data chunks are being read/modified and it'll talk to a central Chunk Metadata Service to get the locations of the specific database shards.

Finally, the proxy server will send the read/write requests to the specified database shards. Each of the database shards has multiple replicas so they use a CDC (Change Data Capture) service to replicate changes between the replicas.

Data Movement Platform

When you're sharding horizontally, it's important to remember that your data won't be static. You'll need to move data between shards for expanding capacity, hardware upgrades, rebalancing hot/cold shards and more.

In Stripe's case, this is especially complicated because they're handling financial data.

Some of their requirements were

• Data Consistency - data being migrated needs to be consistent between both the source and target shards.

• Zero Downtime - any prolonged downtime is unacceptable as businesses need to process payments 24/7. Downtime should be under a few seconds so the product application can just retry the read/write request and there is minimal impact to the customer.

• Granularity - they should be able to migrate an arbitrary number of data chunks between shards without any restrictions on the number of in-flight transfers or the number of migrations a given shard can perform at once.

Here's the steps they follow for zero-downtime migrations across database shards:

1. Confirm Migration and build Indexes - the system first registers the start of the migration in the Chunk Metadata Service. It also builds indexes on the target shards for the data chunks that are being migrated.

2. Bulk Data Import - take a snapshot of the data chunk on the original shard and copy it onto one or more target database shards.

3. Asynchronous Replication - After you copy over the original shard to the target shard, the original shard will still be getting writes. In this step, you asynchronously replicate any writes that happen on the original shard over to the target shards.

4. Correctness Check - Take point-in-time snapshots of the source and target shards and compare them to ensure data completeness and correctness.

5. Traffic Switch - Once the data is imported to the target shard and mutations are being properly replicated, DocDB's coordinator will switch traffic over to the target shard. Stripe does this by first blocking any new writes on the source shard. Then, they wait for the replication service to replicate any outstanding writes to the target shard. Finally, the update the route for the data chunk to point to the target shard in the Chunk Metadata service.
   This step takes a few seconds, so any database requests that get blocked during this process can just retry after a small timeout and get served by the new shards.

6. Finalize Migration - The last step is to mark the migration as complete in the chunk metadata service. They can also delete the chunk data off the source shard.

Results

DocDB's ability to migrate data between shards in a consistent, granular and reliable way has made it significantly easier for Stripe to scale.

In 2023, they migrated petabytes of data between shards and it helped them achieve much better utilization of their database infrastructure.

Get smarter about Software and AI in 5 minutes

Save 50+ hours/week with deep dives, trends, and tools hand-picked from 100+ sources.

It’s read by engineers at Meta, Google, Uber, Amazon, and big startups.

Tech Snippets

How Nginx Was Able to Support 1 Million Concurrent Connections on a Single Server

Neo Kim wrote a fantastic article breaking down how Nginx achieves its impressive scalability through parallelism and concurrency. The web server uses a master-worker model where each worker runs as a single-threaded process with an event loop, allowing it to handle multiple client requests efficiently.

To prevent blocking operations from impacting performance, workers delegate CPU-intensive tasks to a shared thread pool. Nginx also improves scalability through shared memory that lets workers share cached data, session information, and rate limiting data.

newsletter.systemdesign.one/p/how-does-nginx-work

Hey Everyone!

Today we’ll be talking about

• How Airbnb Processes a Million User Events Every Second

  • How Airbnb built a User Events Platform to track, process and store billions of user interactions

  • Introduction to the Lambda Architecture

  • Overview of Apache Flink

  • The Architecture of Airbnb’s User Events Platform

• Tech Snippets

  • Breaking down the Browser Rendering Process

  • How to Maintain Code Quality in the age of AI

  • How Duolingo cut their Cloud Spend by 20%

  • Explaining the Modular Monolith Architecture

  • Google’s State of the Art Quantum Chip

The Architecture of Airbnb’s User Signals Platform

Airbnb is one of the largest travel platforms in the world with over 200 million active users. When travelers browse through the app, there are millions of properties/destinations that Airbnb can recommend. Small improvements in their recommendation system can result in a huge increase in bookings (and hundreds of millions of dollars in revenue)

To provide the best recommendations, Airbnb needs to keep track of past user actions like viewing listings, favoriting experiences, starting a booking process, etc. This data needs to be processed, cleaned and stored in a database.

The Airbnb team built the User Signals Platform to handle this. It ingests and processes over 1 million user events per second and stores them in a key-value database. The platform serves over 70k+ queries per second to other internal teams at Airbnb that need access to this data.

Last week, the Airbnb engineering team published a terrific blog post delving into how they built this platform and the design choices they made.

User Signal Platform Goals

The Airbnb team had quite a few objectives for the User Signals Platform. Some of the goals were: 

• Ingest Real-time and Historical User Data - The platform should store real-time user engagement data as it occurs, but it should also allow for batch jobs that write historical user engagement data.

• Low Latency - Other services at Airbnb will be relying on the User Signals Platform for real-time user engagement data, so the platform should ingest and process new user events in under 1 second.

• Asynchronous Computation - Engineers at Airbnb should be able to run asynchronous computation jobs on the data in the User Signals platform to generate deeper insights.

In this article, we’ll talk about the architecture of the User Signals Platform and also delve into the design patterns and technologies Airbnb used.

Introduction to Lambda Architecture

The core design pattern Airbnb used for their platform is the Lambda Architecture. The Lambda architecture is composed of two layers:

• Speed/Streaming Layer (Real-Time): Processes streaming data as it arrives, delivering low-latency, up-to-date results. Airbnb implements this with Apache Flink and achieves latencies less than one second.

• Batch Layer (Offline): Periodically processes large volumes of historical data to generate more accurate or corrected views. The batch layer ensures long-term accuracy and handles late-arriving data or retrospective fixes. The batch layer will typically operate on a longer timescale, updating views every few hours.

Credits to GeeksforGeeks

By combining these two layers, the Lambda architecture provides the best of both worlds. The speed layer ensures fresh, low-latency data for online queries and personalization. The batch layer ensures correctness, allowing retrospective updates and improvements to data quality.

Introduction to Apache Flink

The core technology Airbnb used for their User Signals Platform is Apache Flink, an open source engine built for processing real-time data with very low latency.

Prior to Flink, data processing systems would rely on “micro-batching” to process data in “real-time”. They would collect data over a small fixed period (every few seconds/minutes) and then process that data as a batch job.

On the other hand, Flink takes an event-driven approach. Instead of waiting for a batch window to fill up, Flink processes each event as soon as it arrives. This results in much lower latencies.

Another benefit of Flink is that it is stateful. Traditional data processing systems might require an external database to maintain state across events. Flink integrates state management directly into the engine, allowing it to remember, accumulate, and update contextual information as events stream in. This is great if you want to do operations like aggregations or joins across your messages.

Other benefits of Apache Flink are:

• Fault Tolerance - Flink provides checkpointing mechanisms to ensure that, if a job or node fails, the system can recover to a previously consistent state. This guarantees exactly-once processing, so each event is reflected in the application’s state exactly once, even in the face of failures.

• Understanding Event-Time - Flink understands the concept of “event time“ (the time when an event actually occurred) instead of just processing time (when the event is processed by the system). This makes it much easier to handle out-of-order events or late-arriving data accurately.

• Integration with the Ecosystem - Flink is widely used and comes with connectors to all the other data tools you might be using (Kafka, Postgres, S3, etc.)

Architecture of Airbnb’s User Signals Platform

Here’s the architecture of Airbnb’s User Signals Platform

As mentioned earlier, it’s based on the Lambda Architecture, so it consists of a real-time ingestion layer and a batch layer.

Here are the steps:

1. User Events: Guests interacting with Airbnb’s apps generate raw events when they view properties, add an experience to their wishlist, search for “rooms in London”, etc.

2. Real-Time Transformation (Speed Layer): Events flow into Kafka, where Flink jobs consume and transform them into “User Signals.”  Some transformations are just simple mappings from raw events, while others may require joining multiple events based on user ID to create richer signals.

3. KV Storage and Serving: The transformed User Signals are stored in a Key-Value store with append-only writes. Using append-only writes helps ensure idempotency and makes data operations much simpler.

4. Batch Processing (Batch Layer): Periodic batch jobs will reprocess the historical data sets and identify any discrepancies or missed events from the speed layer. They’ll backfill the missing/incorrect data to ensure long-term data accuracy and consistency.

5. Asynchronous Computations: In addition to the immediate user signals that are stored in the KV store, Airbnb has Flink jobs that consume the new user signals to generate more insights. These jobs do things like categorize users into segments or group a single user’s actions into “sessions” to get a better understanding of the user’s intent. These jobs are run asynchronously.

6. Online Queries and Services: The USP service provides a way for downstream services at Airbnb to use the user signals data for their own insights.

Results

With this setup, the User Signals Platform processes over 1 million events per second across 100+ Flink jobs. The USP service serves over 70k queries per second to various teams/services at Airbnb.

Tech Snippets

Breaking down the Browser Rendering Process

Abhishek Saha published a fantastic blog post that talks about exactly what happens between going to “www.google.com“ and seeing the page load on your computer.

He delves into the DNS lookup, TCP/TLS handshake, Browser Rendering process and much more. The article is filled with interactive graphics to help you understand the process.

abhisaha.com/blog/exploring-browser-rendering-process

Willow - Google’s state-of-the-art quantum chip

Google’s Quantum AI team just announced Willow, their latest quantum processor.

Google used the Random Circuit Sampling (RCS) benchmark to measure its performance and it was able to complete an extremely computation in under 5 minutes. That same computation would take 10 septillion years for the world’s fastest supercomputer.

At scale, Quantum computers would break many current encryption methods so there’s a big push for quantum-resistant cryptography.

blog.google/technology/research/google-willow-quantum-chip

Explaining the Modular Monolith Architecture

Modular Monoliths are becoming increasingly popular, where you have a balance between the efficiency of traditional Monoliths and the separation of Microservices.

This is a great article that delves into this pattern, it’s defining characteristics, pros and cons.

blog.levelupcoding.com/p/luc-66-breaking-down-modular-monolithic-architecture-blending-tradition-with-innovation

How Duolingo cut their Cloud Spend by 20%

Duolingo published a fantastic blog post delving into the exact strategies they used to cut their AWS cloud spend.

Some of the key optimizations included
- Extending cache TTLs for rarely-changing resources
- Reducing unnecessarily verbose logging in production
- Changing databases to more optimal configurations

and more.

blog.duolingo.com/reducing-cloud-spending

How to Maintain Code Quality in the age of AI

While AI can help write code faster, it can also create a tradeoff with control vs. quality. In this article, Refactoring.fm provides a six-step process for “Lifecycle of Quality“ to help ensure that your codebase doesn’t suffer as you use tools like Claude or gpt-4o.

refactoring.fm/p/code-quality-in-the-age-of-ai

Hey Everyone!

Today we’ll be talking about

• How Airbnb Processes a Million User Events Every Second

  • How Airbnb built a User Events Platform to track, process and store billions of user interactions

  • Introduction to the Lambda Architecture

  • Overview of Apache Flink

  • The Architecture of Airbnb’s User Events Platform

• Tech Snippets

  • Breaking down the Browser Rendering Process

  • How to Maintain Code Quality in the age of AI

  • How Duolingo cut their Cloud Spend by 20%

  • Explaining the Modular Monolith Architecture

  • Google’s State of the Art Quantum Chip

How to Pick Technologies for your Tech Stack

One of the hardest decisions you’ll have to make is around what technologies your team adopts. A wrong decision can be extremely costly and take years to reverse. On the other hand, not making a decision can be just as costly (lost revenue, poor developer productivity, etc.)

Product for Engineers wrote a fantastic blog post on their advice for choosing technologies to adopt.

Some of their tips include

1. Prioritize based on set Criteria - There will always be some shiny new toy that your team can adopt. Instead, prioritize based on problems your team is facing. This can be excessive costs, scaling challenges, or a customer need.

2. Mimic the Real World when Evaluating - The engineers who will be using the technology should have significant sway in the decision. They should be able to test the technology in production (safely) and build proof of concepts before deciding.

3. Ensure you consider technical AND business factors - You should talk to all stakeholders and clarify what the set of evaluation criteria are. Some potential criteria include performance, cost, reliability, support, flexibility and more.

Subscribe to Product for Engineers for the rest of their tips on picking technologies. It’s free!

sponsored

The Architecture of Airbnb’s User Signals Platform

Airbnb is one of the largest travel platforms in the world with over 200 million active users. When travelers browse through the app, there are millions of properties/destinations that Airbnb can recommend. Small improvements in their recommendation system can result in a huge increase in bookings (and hundreds of millions of dollars in revenue)

To provide the best recommendations, Airbnb needs to keep track of past user actions like viewing listings, favoriting experiences, starting a booking process, etc. This data needs to be processed, cleaned and stored in a database.

The Airbnb team built the User Signals Platform to handle this. It ingests and processes over 1 million user events per second and stores them in a key-value database. The platform serves over 70k+ queries per second to other internal teams at Airbnb that need access to this data.

Last week, the Airbnb engineering team published a terrific blog post delving into how they built this platform and the design choices they made.

User Signal Platform Goals

The Airbnb team had quite a few objectives for the User Signals Platform. Some of the goals were: 

• Ingest Real-time and Historical User Data - The platform should store real-time user engagement data as it occurs, but it should also allow for batch jobs that write historical user engagement data.

• Low Latency - Other services at Airbnb will be relying on the User Signals Platform for real-time user engagement data, so the platform should ingest and process new user events in under 1 second.

• Asynchronous Computation - Engineers at Airbnb should be able to run asynchronous computation jobs on the data in the User Signals platform to generate deeper insights.

In this article, we’ll talk about the architecture of the User Signals Platform and also delve into the design patterns and technologies Airbnb used.

Introduction to Lambda Architecture

The core design pattern Airbnb used for their platform is the Lambda Architecture. The Lambda architecture is composed of two layers:

• Speed/Streaming Layer (Real-Time): Processes streaming data as it arrives, delivering low-latency, up-to-date results. Airbnb implements this with Apache Flink and achieves latencies less than one second.

• Batch Layer (Offline): Periodically processes large volumes of historical data to generate more accurate or corrected views. The batch layer ensures long-term accuracy and handles late-arriving data or retrospective fixes. The batch layer will typically operate on a longer timescale, updating views every few hours.

Credits to GeeksforGeeks

By combining these two layers, the Lambda architecture provides the best of both worlds. The speed layer ensures fresh, low-latency data for online queries and personalization. The batch layer ensures correctness, allowing retrospective updates and improvements to data quality.

Introduction to Apache Flink

The core technology Airbnb used for their User Signals Platform is Apache Flink, an open source engine built for processing real-time data with very low latency.

Prior to Flink, data processing systems would rely on “micro-batching” to process data in “real-time”. They would collect data over a small fixed period (every few seconds/minutes) and then process that data as a batch job.

On the other hand, Flink takes an event-driven approach. Instead of waiting for a batch window to fill up, Flink processes each event as soon as it arrives. This results in much lower latencies.

Another benefit of Flink is that it is stateful. Traditional data processing systems might require an external database to maintain state across events. Flink integrates state management directly into the engine, allowing it to remember, accumulate, and update contextual information as events stream in. This is great if you want to do operations like aggregations or joins across your messages.

Other benefits of Apache Flink are:

• Fault Tolerance - Flink provides checkpointing mechanisms to ensure that, if a job or node fails, the system can recover to a previously consistent state. This guarantees exactly-once processing, so each event is reflected in the application’s state exactly once, even in the face of failures.

• Understanding Event-Time - Flink understands the concept of “event time“ (the time when an event actually occurred) instead of just processing time (when the event is processed by the system). This makes it much easier to handle out-of-order events or late-arriving data accurately.

• Integration with the Ecosystem - Flink is widely used and comes with connectors to all the other data tools you might be using (Kafka, Postgres, S3, etc.)

Architecture of Airbnb’s User Signals Platform

Here’s the architecture of Airbnb’s User Signals Platform

As mentioned earlier, it’s based on the Lambda Architecture, so it consists of a real-time ingestion layer and a batch layer.

Here are the steps:

1. User Events: Guests interacting with Airbnb’s apps generate raw events when they view properties, add an experience to their wishlist, search for “rooms in London”, etc.

2. Real-Time Transformation (Speed Layer): Events flow into Kafka, where Flink jobs consume and transform them into “User Signals.”  Some transformations are just simple mappings from raw events, while others may require joining multiple events based on user ID to create richer signals.

3. KV Storage and Serving: The transformed User Signals are stored in a Key-Value store with append-only writes. Using append-only writes helps ensure idempotency and makes data operations much simpler.

4. Batch Processing (Batch Layer): Periodic batch jobs will reprocess the historical data sets and identify any discrepancies or missed events from the speed layer. They’ll backfill the missing/incorrect data to ensure long-term data accuracy and consistency.

5. Asynchronous Computations: In addition to the immediate user signals that are stored in the KV store, Airbnb has Flink jobs that consume the new user signals to generate more insights. These jobs do things like categorize users into segments or group a single user’s actions into “sessions” to get a better understanding of the user’s intent. These jobs are run asynchronously.

6. Online Queries and Services: The USP service provides a way for downstream services at Airbnb to use the user signals data for their own insights.

Results

With this setup, the User Signals Platform processes over 1 million events per second across 100+ Flink jobs. The USP service serves over 70k queries per second to various teams/services at Airbnb.

6 Common Traits of the Most Impactful Developers

You’ll often hear about the mythical “10x engineer” - the go-to person on the team whenever you need a feature shipped fast. However, 10x engineers aren’t just super-technical, they also have a great sense of what to build.

If you’re working on the wrong feature, then it doesn’t matter how fast you work. The company won’t see a big impact from your work.

Product for Engineers wrote a great article delving into the most impactful engineers and identified six common traits that they share.

Here’s a couple of the traits.

1. Always Prototyping and Experimenting - they ship MVPs early and often, iterate quickly based on feedback and aren’t afraid to pivot or kill features that aren’t working.

2. Are Comfortable Writing - Clear writing skills are a must for documenting features, providing PR feedback, and making big technical decisions with RFCs.

3. Understand the Broader Context - they understand the organization’s goals and align their decisions/work with the company’s strategy.

For the rest of the traits, check out the Product for Engineers newsletter.

They send out fantastic articles every month to help you develop the skills you need to deliver the most impact (and get promoted faster).

sponsored

Tech Snippets

Breaking down the Browser Rendering Process

Abhishek Saha published a fantastic blog post that talks about exactly what happens between going to “www.google.com“ and seeing the page load on your computer.

He delves into the DNS lookup, TCP/TLS handshake, Browser Rendering process and much more. The article is filled with interactive graphics to help you understand the process.

abhisaha.com/blog/exploring-browser-rendering-process

Willow - Google’s state-of-the-art quantum chip

Google’s Quantum AI team just announced Willow, their latest quantum processor.

Google used the Random Circuit Sampling (RCS) benchmark to measure its performance and it was able to complete an extremely computation in under 5 minutes. That same computation would take 10 septillion years for the world’s fastest supercomputer.

At scale, Quantum computers would break many current encryption methods so there’s a big push for quantum-resistant cryptography.

blog.google/technology/research/google-willow-quantum-chip

Explaining the Modular Monolith Architecture

Modular Monoliths are becoming increasingly popular, where you have a balance between the efficiency of traditional Monoliths and the separation of Microservices.

This is a great article that delves into this pattern, it’s defining characteristics, pros and cons.

blog.levelupcoding.com/p/luc-66-breaking-down-modular-monolithic-architecture-blending-tradition-with-innovation

How Duolingo cut their Cloud Spend by 20%

Duolingo published a fantastic blog post delving into the exact strategies they used to cut their AWS cloud spend.

Some of the key optimizations included
- Extending cache TTLs for rarely-changing resources
- Reducing unnecessarily verbose logging in production
- Changing databases to more optimal configurations

and more.

blog.duolingo.com/reducing-cloud-spending

How to Maintain Code Quality in the age of AI

While AI can help write code faster, it can also create a tradeoff with control vs. quality. In this article, Refactoring.fm provides a six-step process for “Lifecycle of Quality“ to help ensure that your codebase doesn’t suffer as you use tools like Claude or gpt-4o.

refactoring.fm/p/code-quality-in-the-age-of-ai

Hey Everyone!

Today we’ll be talking about

• How LinkedIn scaled their Restrictions and Enforcement System to 5 million queries per second

  • Introduction to BitSets and their use at LinkedIn

  • How Bloom Filters work and their use-cases

  • Full Refresh-ahead Caching and the pros/cons

  • The Architecture of LinkedIn’s System

• Tech Snippets

  • Latency Numbers Every Programmer Should Know (Visualized)

  • Serving a Billion Web Requests with Boring Code

  • Lies we tell ourselves to keep using Golang

  • 7 questions I get asked frequently as an EM

  • How to Scale a Large Codebase

The Architecture of LinkedIn’s Restriction Enforcement System

LinkedIn is the largest professional social network in the world, with over 1 billion users. Over 100 million messages are sent daily on the platform.

With this scale, you’ll inevitably have some bad actors causing issues on the site. It might be users sending harassing/toxic messages, spammers posting about some cryptocurrency or LinkedIn influencers sharing how their camping trip made them better at B2B sales.

To combat this malicious behavior, LinkedIn provides a number of safeguards like reporting inappropriate content and blocking problematic/annoying users.

However, implementing these safeguards at LinkedIn’s scale introduces a ton of technical challenges.

Some of the requirements LinkedIn needs for their Restrictions Enforcement system include:

• High QPS - The system needs to support 4-5 million queries per second. Many user actions (viewing the feed, sending messages, etc.) require checking the restricted/blocked accounts list. 

• Low Latency - Latency needs to be under 5 milliseconds. Otherwise, basic actions like refreshing your feed or sending a message would take too long.

• High Availability - This system needs to operate with 99.999% availability (5 9s of availability), so less than 30 seconds of downtime per month. 

• Low Ingestion Delay - When a user blocks/reports an account, that should be reflected in the restrictions enforcement system immediately. If they refresh their feed right after, the posts from the blocked user should be immediately hidden.

Earlier this year, LinkedIn’s engineering team published a fantastic blog post detailing how they built their restrictions enforcement system. They talked about the different generations of the system’s architecture and problems they faced along the way.

In our Quastor article, we’ll focus on the specific data structures and strategies LinkedIn used. We’ll explain them and delve into the pros and cons LinkedIn saw.

BitSets

One of the key data structures LinkedIn uses in their restrictions system is BitSets.

A BitSet is an array of boolean values where each value only takes up 1 bit of space. A bit that’s set represents a true value whereas a bit that’s not set represents false.

BitSets are extremely memory efficient. If you need to store boolean values for 1 billion users (whether a user is restricted/not restricted), you would only need 1 billion bits (approximately 125 megabytes).

To give a more concrete example of how LinkedIn uses BitSets, let’s say LinkedIn needs to store restricted/unrestricted account status for 1 billion users. Each user has a memberID from 1 to 1 billion. 

To store this, they could use an array of 64-bit integers. Each integer can store the restriction status for 64 different users (we need 1 bit per user) so the array would hold ~15 million integers (around 125 megabytes of space). 

If LinkedIn needs to check whether user with memberID 525234320 is restricted, the steps would be:

1. Divide 523,234,320 by 64 to get the index in the integer array (which would be 8,175,536)

2. Take 523,234,320 modulo 64 to get which bit to check in that integer (which would be 32)

3. Use bitwise operations to check if that specific bit is set to 1 (restricted) or 0 (not restricted)

The time and space requirements with BitSets are very efficient. Checking whether users are restricted takes constant time (since the membership lookup operations are all O(1)) and the storage necessary is only a couple hundred megabytes.

Bloom Filters

In addition to BitSets, the other data structure LinkedIn found useful was Bloom Filters.

A Bloom filter is a probabilistic data structure that lets you quickly test whether an item might be in a set. Bloom Filters are probabilistic so they will tell you if an item is in a set but will occasionally give false positives (it will mistakenly say an item is in the set when it’s not).

Under the hood, Bloom Filters use hashing to map items to a bit array. The issue is that collisions in the hashing function can cause false positives. Here’s a fantastic article that delves into how Bloom Filters work with visuals and a basic Python implementation. 

For LinkedIn, they also used Bloom Filters to quickly check whether a user’s account was restricted.  

The pros were that the Bloom Filters were extremely space efficient compared to traditional caching techniques (using a set or hash table).

The downside was the false positives. However, the Bloom Filter can be tuned to make false positives extremely rare and LinkedIn didn’t find it to be a big issue. 

Full Refresh-ahead Caching

LinkedIn explored various caching strategies to achieve their QPS and latency requirements. One approach was their full refresh-ahead cache.

The dataset of account restrictions was quite small (thanks to using the BitSet and Bloom Filter data structures) so LinkedIn had each client application host store all restriction data in their in-memory cache.

In order to maintain cache freshness, they implemented a polling mechanism where clients would periodically check for any new changes to member restrictions.

This system resulted in a huge decrease in latencies but came with some downsides.

The client-side memory footprint ended up being substantial and strained infrastructure. Additionally, the caches were stored in-memory so they weren’t persistent. Clients had to frequently build and rebuild this cache which put strain on LinkedIn’s underlying database.

System Architecture

Here’s the current architecture for LinkedIn’s restriction enforcement system.

The key components are

1. Client Layer - Clients that use this system include the LinkedIn newsfeed, their recruiting/talent management tools and other products/services at the company. These clients use a REST API to query the system.

2. LinkedIn Restrictions Enforcement System - This component consists of the BitSet data structures and the main restriction enforcement system. The BitSet data structures are stored client-side and maintain a cache of all the restriction records.

3. Venice Database - this is the central storage (source of truth) for all user restrictions. Venice is an open-source, horizontally scalable, eventually-consistent storage system that LinkedIn built on RocksDB. You can read more about how Venice works here.

4. Kafka Restriction Records - When a user gets reported/blocked, the change in their account status is sent through Kafka. This allows near real-time propagation of changes.

5. Restriction Management System - LinkedIn has a legacy system (check the blog post for a full explanation on the previous generations) of the Restriction Enforcement system that connects to Espresso to store and update blocked/restricted users. Espresso is LinkedIn’s distributed, document data store that’s built on MySQL. You can read more about Espresso here. 

Tech Snippets

Latency Numbers Every Programmer Should Know (Visualized)

You’ve probably seen the blog posts with latency numbers that every programmer should know.

These are approximate numbers for how it takes to search RAM vs. disk, transfer a byte from the US to Europe, compress 1 kilobyte of data and more.

This is an awesome tool that helps you visualize those numbers and understand the orders of magnitude difference that exists between some of them.

samwho.dev/numbers

Hey Everyone!

Today we’ll be talking about

• How LinkedIn scaled their Restrictions and Enforcement System to 5 million queries per second

  • Introduction to BitSets and their use at LinkedIn

  • How Bloom Filters work and their use-cases

  • Full Refresh-ahead Caching and the pros/cons

  • The Architecture of LinkedIn’s System

• Tech Snippets

  • Latency Numbers Every Programmer Should Know (Visualized)

  • Serving a Billion Web Requests with Boring Code

  • Lies we tell ourselves to keep using Golang

  • 7 questions I get asked frequently as an EM

  • How to Scale a Large Codebase

The Architecture of LinkedIn’s Restriction Enforcement System

LinkedIn is the largest professional social network in the world, with over 1 billion users. Over 100 million messages are sent daily on the platform.

With this scale, you’ll inevitably have some bad actors causing issues on the site. It might be users sending harassing/toxic messages, spammers posting about some cryptocurrency or LinkedIn influencers sharing how their camping trip made them better at B2B sales.

To combat this malicious behavior, LinkedIn provides a number of safeguards like reporting inappropriate content and blocking problematic/annoying users.

However, implementing these safeguards at LinkedIn’s scale introduces a ton of technical challenges.

Some of the requirements LinkedIn needs for their Restrictions Enforcement system include:

• High QPS - The system needs to support 4-5 million queries per second. Many user actions (viewing the feed, sending messages, etc.) require checking the restricted/blocked accounts list. 

• Low Latency - Latency needs to be under 5 milliseconds. Otherwise, basic actions like refreshing your feed or sending a message would take too long.

• High Availability - This system needs to operate with 99.999% availability (5 9s of availability), so less than 30 seconds of downtime per month. 

• Low Ingestion Delay - When a user blocks/reports an account, that should be reflected in the restrictions enforcement system immediately. If they refresh their feed right after, the posts from the blocked user should be immediately hidden.

Earlier this year, LinkedIn’s engineering team published a fantastic blog post detailing how they built their restrictions enforcement system. They talked about the different generations of the system’s architecture and problems they faced along the way.

In our Quastor article, we’ll focus on the specific data structures and strategies LinkedIn used. We’ll explain them and delve into the pros and cons LinkedIn saw.

BitSets

One of the key data structures LinkedIn uses in their restrictions system is BitSets.

A BitSet is an array of boolean values where each value only takes up 1 bit of space. A bit that’s set represents a true value whereas a bit that’s not set represents false.

BitSets are extremely memory efficient. If you need to store boolean values for 1 billion users (whether a user is restricted/not restricted), you would only need 1 billion bits (approximately 125 megabytes).

To give a more concrete example of how LinkedIn uses BitSets, let’s say LinkedIn needs to store restricted/unrestricted account status for 1 billion users. Each user has a memberID from 1 to 1 billion. 

To store this, they could use an array of 64-bit integers. Each integer can store the restriction status for 64 different users (we need 1 bit per user) so the array would hold ~15 million integers (around 125 megabytes of space). 

If LinkedIn needs to check whether user with memberID 525234320 is restricted, the steps would be:

1. Divide 523,234,320 by 64 to get the index in the integer array (which would be 8,175,536)

2. Take 523,234,320 modulo 64 to get which bit to check in that integer (which would be 32)

3. Use bitwise operations to check if that specific bit is set to 1 (restricted) or 0 (not restricted)

The time and space requirements with BitSets are very efficient. Checking whether users are restricted takes constant time (since the membership lookup operations are all O(1)) and the storage necessary is only a couple hundred megabytes.

Bloom Filters

In addition to BitSets, the other data structure LinkedIn found useful was Bloom Filters.

A Bloom filter is a probabilistic data structure that lets you quickly test whether an item might be in a set. Bloom Filters are probabilistic so they will tell you if an item is in a set but will occasionally give false positives (it will mistakenly say an item is in the set when it’s not).

Under the hood, Bloom Filters use hashing to map items to a bit array. The issue is that collisions in the hashing function can cause false positives. Here’s a fantastic article that delves into how Bloom Filters work with visuals and a basic Python implementation. 

For LinkedIn, they also used Bloom Filters to quickly check whether a user’s account was restricted.  

The pros were that the Bloom Filters were extremely space efficient compared to traditional caching techniques (using a set or hash table).

The downside was the false positives. However, the Bloom Filter can be tuned to make false positives extremely rare and LinkedIn didn’t find it to be a big issue. 

Full Refresh-ahead Caching

LinkedIn explored various caching strategies to achieve their QPS and latency requirements. One approach was their full refresh-ahead cache.

The dataset of account restrictions was quite small (thanks to using the BitSet and Bloom Filter data structures) so LinkedIn had each client application host store all restriction data in their in-memory cache.

In order to maintain cache freshness, they implemented a polling mechanism where clients would periodically check for any new changes to member restrictions.

This system resulted in a huge decrease in latencies but came with some downsides.

The client-side memory footprint ended up being substantial and strained infrastructure. Additionally, the caches were stored in-memory so they weren’t persistent. Clients had to frequently build and rebuild this cache which put strain on LinkedIn’s underlying database.

System Architecture

Here’s the current architecture for LinkedIn’s restriction enforcement system.

The key components are

1. Client Layer - Clients that use this system include the LinkedIn newsfeed, their recruiting/talent management tools and other products/services at the company. These clients use a REST API to query the system.

2. LinkedIn Restrictions Enforcement System - This component consists of the BitSet data structures and the main restriction enforcement system. The BitSet data structures are stored client-side and maintain a cache of all the restriction records.

3. Venice Database - this is the central storage (source of truth) for all user restrictions. Venice is an open-source, horizontally scalable, eventually-consistent storage system that LinkedIn built on RocksDB. You can read more about how Venice works here.

4. Kafka Restriction Records - When a user gets reported/blocked, the change in their account status is sent through Kafka. This allows near real-time propagation of changes.

5. Restriction Management System - LinkedIn has a legacy system (check the blog post for a full explanation on the previous generations) of the Restriction Enforcement system that connects to Espresso to store and update blocked/restricted users. Espresso is LinkedIn’s distributed, document data store that’s built on MySQL. You can read more about Espresso here. 

Tech Snippets

Latency Numbers Every Programmer Should Know (Visualized)

You’ve probably seen the blog posts with latency numbers that every programmer should know.

These are approximate numbers for how it takes to search RAM vs. disk, transfer a byte from the US to Europe, compress 1 kilobyte of data and more.

This is an awesome tool that helps you visualize those numbers and understand the orders of magnitude difference that exists between some of them.

samwho.dev/numbers

Hey Everyone!

Today we’ll be talking about

• How LinkedIn uses Event Driven Architectures to Scale their Infrastructure

  • An Introduction to the Actor Model and how it works

  • How LinkedIn collects and processes server metrics from their fleet with an Event Driven Architecture

  • LinkedIn’s monitoring system for server consoles

• Tech Snippets

  • How I Ship Projects at Big Tech Companies

  • How Binary Vector Embeddings work and why they’re so useful

  • How Coinbase uses ML to Predict Traffic and Auto-scale Databases

How LinkedIn uses Design Patterns to Scale their Infrastructure

LinkedIn is the largest professional social networking platform in the world with over 950 million users in 200+ countries.

To serve this user base, they maintain dozens of data centers around the world with hundreds of thousands of servers globally. 

In order to manage these servers, LinkedIn makes use of many tried-and-tested design patterns.

One pattern is the Producer-Consumer pattern, commonly used in event driven architectures (EDAs).

This pattern consists of three main components:

• Producer - generates events/messages (server metrics, status updates, data from queries, etc.)

• Queue - acts as a buffer to store messages until they’re ready to be processed. LinkedIn uses Redis, Kafka or built-in queues for this.

• Consumer - reads and processes messages from the queue

Saira Khanum is a Staff Software Engineer at LinkedIn and she wrote a fantastic blog post delving into how the engineering team uses this pattern in three different systems:

1. To Collect and Maintain Data from Servers for Real-time and Analytical Queries

2. To Check Servers for Availability and Accessibility

3. To Detect and Fix any Access Policy Violations on the Servers

We’ll explore these and talk about how LinkedIn implemented them.

Actor Pattern

When building event driven architectures, LinkedIn frequently uses the Actor Pattern. Event Driven Architectures are loosely defined so the Actor Pattern (or Actor Model) is a specific implementation of an EDA.

With this model, everything is represented as an actor.

An actor is an independent entity that can

• Send messages to other actors

• Process messages/requests

• Create new actors and designate their behavior

• Have independent state

To give you a better sense of how this might work, here’s a hypothetical example of an Actor model at Uber for handling ride requests.

1. When a user first requests a ride, a RequestActor is created specifically for their request. This actor maintains the state of the request (whether it’s active or canceled) and coordinates the entire matching process.

2. The RequestActor might first create a child PricingActor to figure out a reasonable price for the request based on the trip distance and time of day. The PricingActor will run internal logic based on the RequestActor’s message and return the ride price.

3. Once it has the pricing figured out, the RequestActor will communicate with nearby DriverActors (one actor per active driver on Uber) by sending them ride offer messages.

4. The DriverActor will then handle sending a notification to the Uber driver that there's someone looking for a ride. If the driver accepts the ride then the DriverActor might create a new TripActor to handle the ongoing ride (tracking location updates, route changes, payment processing, etc.)

If you’re looking for more details, here’s a fantastic article that delves deeper on the Actor model.

Back to LinkedIn…

Event Driven Architectures at LinkedIn

LinkedIn talks about a few systems where they’ve found EDAs useful for managing infrastructure.

Distributed Server Queries at LinkedIn

The first system is LinkedIn’s distributed server query system. This is responsible for collecting system facts (CPU/memory usage, network connections, disk space usage, etc.) from across the server fleet and storing them so they can be queried and analyzed.

Some of the requirements are

• Scale - the system needs to process terabytes of data from hundreds of thousands of servers in near real-time

• Data Refresh - the data needs to be collected several times every hour

• Data Maintenance - the last known good snapshot of system facts needs to be maintained for a defined retention period. (after the retention period is over, the system facts need to be marked as stale)

Here’s the high level architecture of the system

1. Agents (producers) are deployed across the server fleet to collect system facts

2. These facts are sent to worker processes (using the Actor Pattern) and stored on Redis

3. Different worker processes consume the data from Redis, process it and store it in different datastores

Some of the choices LinkedIn made were

• Redis - LinkedIn picked Redis as the queue since they were looking for low latency. The messages are short-lived and introducing a tool like Kafka would introduce too much overhead.

• Actor Pattern - Workers that collect and process server metrics use the Actor pattern. They’re implemented with Gunicorn. 

Server Console Monitoring

The second system is LinkedIn’s distributed system to monitor the server console for their infrastructure. Server consoles (often called service processors) allow administrators to manage and monitor physical servers remotely (even when the server is powered off or unresponsive). They’re essential for troubleshooting, rebooting and maintaining servers.

LinkedIn’s monitoring system checks that these server management consoles are available and accessible.

Here’s the architecture for how they do that.

1. Satellite servers run checks across the servers in the data center. Each check is handled by a separate actor.

2. Messages from each check are passed through RabbitMQ. The result of each check determines if the next check should be run (if the next actor should be created)

3. Final results are sent to Kafka. Consumer applications can read results for storage/analysis from the various Kafka streams.

Some of the tech choices LinkedIn made were

• Actor Pattern - Each check that LinkedIn has to do is an actor. The checks are done sequentially so they pass messages to each other to send results and status updates.

• Kafka and RabbitMQ - RabbitMQ is used for communication between the actors whereas Kafka is used for forwarding the final results down to the consumer applications for further processing and storage

Tech Snippets

How Coinbase uses Machine Learning to Predict Traffic and Autoscale Infrastructure

With crypto markets, sudden price movements can cause massive traffic surges on Coinbase.

Instead of scaling reactively based on CPU usage (which is often too late), Coinbase built an ML model that predicts traffic spikes in advance by analyzing:
- price volatility in major cryptocurrencies
- current traffic patterns and growth rates
- historical seasonal trends
- load testing data

This has helped them prevent system outages and reduce costs from over-provisioning resources.

www.coinbase.com/blog/how-coinbase-is-using-machine-learning-to-predict

How Binary Vector Embeddings work and why they’re so useful

Vector Embeddings allow you to convert text into numbers that represent meaning. This is super useful for semantic search and similarity matching.

Traditional embeddings use 32-bit floating point numbers but binary quantization converts each number to a single bit.

With this approach, you can compress embeddings to just 3% of their original size while retaining 95%+ of the original accuracy.

Evan Schwartz wrote a fantastic article on this technique.

emschwartz.me/binary-vector-embeddings-are-so-cool

Essential Reading For Engineering Leaders

If you find Quastor useful, you should check out Pointer.

It’s essential reading for engineering leaders to hone in on improving their soft skills. They send out super high quality engineering-related content twice a week.

Sign Up for Free!

(cross promo)

www.pointer.io/?utm_source=quastor&utm_medium=crosspromo

Hey Everyone!

Today we’ll be talking about

• How LinkedIn uses Event Driven Architectures to Scale their Infrastructure

  • An Introduction to the Actor Model and how it works

  • How LinkedIn collects and processes server metrics from their fleet with an Event Driven Architecture

  • LinkedIn’s monitoring system for server consoles

• Tech Snippets

  • How I Ship Projects at Big Tech Companies

  • How Binary Vector Embeddings work and why they’re so useful

  • How Coinbase uses ML to Predict Traffic and Auto-scale Databases

How to build AI Agents that interact with Slack and Salesforce in your Product

Interested in building AI agents into your product?

Your AI agent may need to automate tasks that take place outside your application, in your user’s third-party apps.

This could be reading data from the third party (checking inventory in Shopify, a ticket’s status in Jira, etc.) and then writing data in those third-party apps (Slack, Salesforce, Jira, etc.)

That’s where AI actions and function tools come into play.

In this tutorial and demo, you’ll learn how to equip your AI agent with AI actions such as:

• Sending a Slack message

• Creating/Updating a Record in Salesforce

• Chain actions together

Read/watch the tutorial and access the GitHub Repo below.

sponsored

How LinkedIn uses Design Patterns to Scale their Infrastructure

LinkedIn is the largest professional social networking platform in the world with over 950 million users in 200+ countries.

To serve this user base, they maintain dozens of data centers around the world with hundreds of thousands of servers globally. 

In order to manage these servers, LinkedIn makes use of many tried-and-tested design patterns.

One pattern is the Producer-Consumer pattern, commonly used in event driven architectures (EDAs).

This pattern consists of three main components:

• Producer - generates events/messages (server metrics, status updates, data from queries, etc.)

• Queue - acts as a buffer to store messages until they’re ready to be processed. LinkedIn uses Redis, Kafka or built-in queues for this.

• Consumer - reads and processes messages from the queue

Saira Khanum is a Staff Software Engineer at LinkedIn and she wrote a fantastic blog post delving into how the engineering team uses this pattern in three different systems:

1. To Collect and Maintain Data from Servers for Real-time and Analytical Queries

2. To Check Servers for Availability and Accessibility

3. To Detect and Fix any Access Policy Violations on the Servers

We’ll explore these and talk about how LinkedIn implemented them.

Actor Pattern

When building event driven architectures, LinkedIn frequently uses the Actor Pattern. Event Driven Architectures are loosely defined so the Actor Pattern (or Actor Model) is a specific implementation of an EDA.

With this model, everything is represented as an actor.

An actor is an independent entity that can

• Send messages to other actors

• Process messages/requests

• Create new actors and designate their behavior

• Have independent state

To give you a better sense of how this might work, here’s a hypothetical example of an Actor model at Uber for handling ride requests.

1. When a user first requests a ride, a RequestActor is created specifically for their request. This actor maintains the state of the request (whether it’s active or canceled) and coordinates the entire matching process.

2. The RequestActor might first create a child PricingActor to figure out a reasonable price for the request based on the trip distance and time of day. The PricingActor will run internal logic based on the RequestActor’s message and return the ride price.

3. Once it has the pricing figured out, the RequestActor will communicate with nearby DriverActors (one actor per active driver on Uber) by sending them ride offer messages.

4. The DriverActor will then handle sending a notification to the Uber driver that there's someone looking for a ride. If the driver accepts the ride then the DriverActor might create a new TripActor to handle the ongoing ride (tracking location updates, route changes, payment processing, etc.)

If you’re looking for more details, here’s a fantastic article that delves deeper on the Actor model.

Back to LinkedIn…

Event Driven Architectures at LinkedIn

LinkedIn talks about a few systems where they’ve found EDAs useful for managing infrastructure.

Distributed Server Queries at LinkedIn

The first system is LinkedIn’s distributed server query system. This is responsible for collecting system facts (CPU/memory usage, network connections, disk space usage, etc.) from across the server fleet and storing them so they can be queried and analyzed.

Some of the requirements are

• Scale - the system needs to process terabytes of data from hundreds of thousands of servers in near real-time

• Data Refresh - the data needs to be collected several times every hour

• Data Maintenance - the last known good snapshot of system facts needs to be maintained for a defined retention period. (after the retention period is over, the system facts need to be marked as stale)

Here’s the high level architecture of the system

1. Agents (producers) are deployed across the server fleet to collect system facts

2. These facts are sent to worker processes (using the Actor Pattern) and stored on Redis

3. Different worker processes consume the data from Redis, process it and store it in different datastores

Some of the choices LinkedIn made were

• Redis - LinkedIn picked Redis as the queue since they were looking for low latency. The messages are short-lived and introducing a tool like Kafka would introduce too much overhead.

• Actor Pattern - Workers that collect and process server metrics use the Actor pattern. They’re implemented with Gunicorn. 

Server Console Monitoring

The second system is LinkedIn’s distributed system to monitor the server console for their infrastructure. Server consoles (often called service processors) allow administrators to manage and monitor physical servers remotely (even when the server is powered off or unresponsive). They’re essential for troubleshooting, rebooting and maintaining servers.

LinkedIn’s monitoring system checks that these server management consoles are available and accessible.

Here’s the architecture for how they do that.

1. Satellite servers run checks across the servers in the data center. Each check is handled by a separate actor.

2. Messages from each check are passed through RabbitMQ. The result of each check determines if the next check should be run (if the next actor should be created)

3. Final results are sent to Kafka. Consumer applications can read results for storage/analysis from the various Kafka streams.

Some of the tech choices LinkedIn made were

• Actor Pattern - Each check that LinkedIn has to do is an actor. The checks are done sequentially so they pass messages to each other to send results and status updates.

• Kafka and RabbitMQ - RabbitMQ is used for communication between the actors whereas Kafka is used for forwarding the final results down to the consumer applications for further processing and storage

How to build AI Agents that interact with Slack and Salesforce in your Product

Interested in building AI agents into your product?

Your AI agent may need to automate tasks that take place outside your application, in your user’s third-party apps.

This could be reading data from the third party (checking inventory in Shopify, a ticket’s status in Jira, etc.) and then writing data in those third-party apps (Slack, Salesforce, Jira, etc.)

That’s where AI actions and function tools come into play.

In this tutorial and demo, you’ll learn how to equip your AI agent with AI actions such as:

• Sending a Slack message

• Creating/Updating a Record in Salesforce

• Chain actions together

Read/watch the tutorial and access the GitHub Repo below.

sponsored

Tech Snippets

How Coinbase uses Machine Learning to Predict Traffic and Autoscale Infrastructure

With crypto markets, sudden price movements can cause massive traffic surges on Coinbase.

Instead of scaling reactively based on CPU usage (which is often too late), Coinbase built an ML model that predicts traffic spikes in advance by analyzing:
- price volatility in major cryptocurrencies
- current traffic patterns and growth rates
- historical seasonal trends
- load testing data

This has helped them prevent system outages and reduce costs from over-provisioning resources.

www.coinbase.com/blog/how-coinbase-is-using-machine-learning-to-predict

How Binary Vector Embeddings work and why they’re so useful

Vector Embeddings allow you to convert text into numbers that represent meaning. This is super useful for semantic search and similarity matching.

Traditional embeddings use 32-bit floating point numbers but binary quantization converts each number to a single bit.

With this approach, you can compress embeddings to just 3% of their original size while retaining 95%+ of the original accuracy.

Evan Schwartz wrote a fantastic article on this technique.

emschwartz.me/binary-vector-embeddings-are-so-cool

Essential Reading For Engineering Leaders

If you find Quastor useful, you should check out Pointer.

It’s essential reading for engineering leaders to hone in on improving their soft skills. They send out super high quality engineering-related content twice a week.

Sign Up for Free!

(cross promo)

www.pointer.io/?utm_source=quastor&utm_medium=crosspromo

Hey Everyone!

Today we’ll be talking about

• How Reddit built a Metadata store that handles 100k reads per second

  • High level goals of Reddit’s metadata store

  • Picking Sharded Postgres vs. Cassandra

  • Data Migration Process

  • Scaling with sharding and denormalization

• Tech Snippets

  • 5 Non-LLM Software Trends To Be Excited About

  • How Python Asyncio Works

  • Writing a File System from Scratch in Rust

  • How GitLab automates engineering management

How Reddit built a Metadata store that Handles 100k Reads per Second

Over the past few years, Reddit has seen their user-base triple in size. They went from 430 million monthly active users in 2019 to 1.2 billion in 2024.

The good news with all this growth is that they finally IPO’d earlier this year and let employees cash in on their stock options. The bad news is that the engineering team had to deal with a bunch of headaches.

One issue that Reddit faced was with their media metadata store. Reddit is built on AWS and GCP, so they store any media uploaded to the site (images, videos, gifs, etc.) on AWS S3.

Every piece of media uploaded also comes with metadata. Each media file will have metadata like video thumbnails, playback URLs, S3 file locations, image resolution, etc.

Previously, Reddit’s media metadata was distributed across different storage systems. To make this easier to manage, the engineering team wanted to create a unified system for managing all this data.

The high-level design goals were

• Single System - they needed a single system that could store all of Reddit’s media metadata. Reddit’s growing quickly, so this system needs to be highly scalable. At the current rate of growth, Reddit expects the size of their media metadata to be 50 terabytes by 2030.

• Read Heavy Workload - This data store will have a very read-heavy workload. It needs to handle over 100k reads per second with latency less than 50 ms.

• Support Writes - The data store should also support data creation/updates. However these requests have significantly lower traffic than reads and Reddit can tolerate higher latency for this.

Reddit wrote a fantastic article delving into their process for creating a unified media metadata store.

We’ll be summarizing the article and adding some extra context.

Picking the Database

To build this media metadata store, Reddit considered two choices: Sharded Postgres vs. Cassandra.

Postgres

Postgres is one of the most popular relational databases in the world and is consistently voted most loved database in Stack Overflow’s developer surveys.

Some of the pros for Postgres are

• Battle Tested - Tens of thousands of companies use Postgres and there’s been countless tests on benchmarks, scalability and more. Postgres is used (or has been used) at companies like Uber, Skype, Spotify, etc.
  
  With this, there’s a massive wealth of knowledge around potential issues, common bugs, pitfalls, etc. on forums like Stack Overflow, Slack/IRC, mailing threads and more.

• Open Source & Community - Postgres has been open source since 1995 with a liberal license that’s similar to the BSD and MIT license. There’s a vibrant community of developers who help teach the database and provide support for people with issues. Postgres also has outstanding documentation.

• Extensibility & Interoperability - One of the initial design goals of Postgres was extensibility. Over it’s 30 year history, there’s been a countless number of extensions that have been developed to make Postgres more powerful. We’ll talk about a couple Postgres extensions that Reddit uses for sharding.

Cassandra

Cassandra is a NoSQL, distributed database created at Facebook in 2007. The initial project was heavily inspired by Google Bigtable and also took many ideas from Amazon’s Dynamo.

Here’s some characteristics of Cassandra:

• Large Scale - Cassandra is completely distributed and can scale to a massive size. It has out of the box support for things like distributing/replicating data in different locations.

  
  Additionally, Cassandra is also designed with a decentralized architecture to minimize any central points of failure.

• Highly Tunable - Cassandra is highly customizable so you can configure it based on your exact workload. You can change how the communications between nodes happens (gossip protocol), how data is read from disk (LSM Tree), the consensus between nodes for writes (consistency level) and much more.

• Wide Column - Cassandra uses a wide column storage model, which allows for flexible storage. Data is organized into column families, where each can have multiple rows with varying numbers of columns. You can read/write large amounts of data quickly and also add new columns without having to do a schema migration.

We did a much more detailed dive on Cassandra that you can read here.

Picking Postgres

After evaluating both choices extensively, Reddit decided to go with Postgres.

The reasons why included:

• Challenges with Managing Cassandra - They found some challenges with managing Cassandra. Ad-hoc queries for debugging and visibility were far more difficult compared to Postgres.

• Data Denormalization Issues with Cassandra - In Cassandra, data is typically denormalized and stored in a way to optimize specific queries (this is based on your specific application). However, this can lead to challenges when creating new queries that your data hasn’t been specifically modeled for.

Reddit uses AWS, so they went with AWS Aurora Postgres. For more on AWS RDS, you can check out a detailed tech dive we did here.

Data Migration

Migrating to Postgres was a big challenge for the team. They had to transfer terabytes of data from the different systems to Postgres while ensuring that the legacy systems could continue serving over 100k reads per second.

Here’s the steps they went through for the migration

1. Dual Writes - Any new media metadata would be written to both the old systems and to Postgres.

2. Backfill Data - Data from the older systems would be backfilled into Postgres

3. Dual Reads - After Postgres has enough data, enable dual reads so that read requests are served by both Postgres and the old system

4. Monitoring and Ramp Up - Compare the results from the dual reads and fix any data gaps. Slowly ramp up traffic to Postgres until they could fully cutover.

Scaling Strategies

With that strategy, Reddit was able to successfully migrate over to Postgres.

Currently, they’re seeing peak loads of ~100k reads per second. At that load, the latency numbers they’re seeing with Postgres are

• 2.6 ms P50 - 50% of requests have a latency lower than 2.6 milliseconds

• 4.7 ms P90 - 90% of requests have a latency lower than 4.7 milliseconds

• 17 ms P99 - 99% of requests have a latency lower than 17 milliseconds

They’re able to achieve this without needing a read-through cache.

We’ll talk about some of the strategies they’re using to scale.

Table Partitioning

At the current pace of media content creation, Reddit expects their media metadata to be roughly 50 terabytes. This means they need to implement sharding and partition their tables across multiple Postgres instances.

Reddit shards their tables based on post_id where they use range-based partitioning. All posts with a post_id in a certain range will be put on the same database shard.

post_id increases monotonically, so this means that their table will be partitioned by time periods.

Many of their read requests involve batch queries on multiple IDs from the same time period, so this design helps minimize cross-shard joins.

Reddit uses the pg_partman Postgres extension to manage the table partitioning.

Denormalization

Another way Reddit minimizes joins is by using denormalization.

They took all the metadata fields required for displaying an image post and put them together into a single JSONB field. Instead of fetching different fields and combining them, they can just fetch that single JSONB field.

This made it much more efficient to fetch all the data needed to render a post.

All the metadata needed to render an image post

It also simplified the querying logic, especially across different media types. Instead of worrying about exactly which data fields you needed, you just fetch the single JSONB value.

Tech Snippets

5 Non-LLM Software Trends To Be Excited About

With all the AI hype that’s been going around, almost everyone has been completely focused on LLMs. However, there’s been a ton of research and advancements in other areas of software engineering.

Engineer’s Codex published a fantastic blog post talking about some of the other exciting tech in software.

Some of the advancements discussed are CRDTs, WebAssembly improvements, strides in Cross-Platform development and more.

read.engineerscodex.com/p/5-non-llm-software-trends-to-be-excited

How Python Asyncio Works

This is an excellent deep dive into Python’s asyncio library. It explains how it works under the hood by recreating a simplified version from scratch.

The post goes through creating a basic event loop and how to build a sleep generator that pauses a task’s execution for a certain duration.

jacobpadilla.com/articles/recreating-asyncio

How GitLab automates engineering management

Engineering managers at GitLab use a ton of automation scripts to help them manage projects and keep track of issues.

This is a post by GitLab Engineering with some examples of automation scripts they use to help keep projects on track.

about.gitlab.com/blog/2021/11/16/engineering-managers-automate-their-jobs