Stephen here.

I spend my weekends researching, learning, and creating content for this newsletter.

It would mean the world to me, if you took a few seconds of your time to check out The Rundown AI.

They’re the fastest growing AI newsletter for a reason (I’m a subscriber too).

Now let’s get on with the show.

Learn how to make AI work for you.

How do you stay up-to-date with the insane pace of AI? Join The Rundown – the world’s fastest-growing AI newsletter with over 500,000+ readers learning how to become more productive using AI every morning.

1. Our team spends all day researching and talking with industry experts.

2. We send you updates on the latest AI news and how to apply it in 5 minutes a day.

3. You learn how to become 2x more productive by leveraging AI.

Subscribe with one click.

Production is always hard.

There’s databases, configurations, infrastructure, performance, security, observability…the list goes on.

On top of all of these things, the stakes are high. Defects can lead to downtime, and the company could lose money.

For Node.js applications, there’s several things we need to keep in mind when moving to production.

Let’s go over them.

Configure Node.js Properly

It’s common for a company to have multiple servers and environments. Each environment will typically have different configurations and environment variables.

For example, a common setup is the following: Development → Staging → Production.

For environment variables, Node.js contains a built-in core module named process. The process object has a property named env with key-value pairs of all environment variables on startup.

One particular environment variable to pay attention to is NODE_ENV. Not only is it part of Node.js, it’s also widely used in NPM packages.

When targeting a production environment, NODE_ENV needs to be set to production. Doing so provides several benefits such as reduced logging (essential) and performance optimizations.

Otherwise, sensitive information used during development and stack traces may be logged unintentionally.

Audit NPM Packages

In the Node.js ecosystem, there are over 2.1 million NPM packages (in 2022). Not all packages are properly maintained, leading to security vulnerabilities.

The latest incident was a commit in the popular colors package. The commit added an infinite loop, creating a Domino Effect. It broke many widely-used NPM packages such as Jest, Rollup, and Faker.

Other examples include the event-stream incident in 2018 and the left-pad debacle in 2016.

These cases highlight the importance of vetting packages in an application. NPM provides commands to address these issues:

# Scan for vulnerabilities
npm audit

# Fix issues automatically
npm audit fix

# Update and fix
npm update && npm audit fix

Scans can also by automated. At my company, we use Snyk.

We’ve integrated Snyk into our CI/CD pipelines and trigger when pull requests are opened.

Use a Non-Root User in Docker

Nowadays, most Node.js applications are containerized with Docker.

Docker packages up the code and dependencies needed to run the application into a container image.

High-Level View Of How Docker Containers Are Used

The Docker Engine then creates the container from the image at runtime.

Pipelines and orchestration tools like Docker Compose and Kubernetes can be used to streamline the deployment process.

By default, Docker runs commands inside containers as root. Following the Principle of Least Privilege (PoLP), a user should only have access to the necessary data and resources needed to perform the task.

The official Docker image for Node provides a non-root user:

FROM node:6.10.3
...
# At the end, set the user to use when running this image
USER node

The name, uid, and gid of the user can be changed as needed.

Implement Strong Authentication Policies

Node.js has a core module called crypto. It’s a general purpose library for doing things like password hashing, encryption, and decryption (and much more).

For password hashing, crypto uses the scrypt algorithm to hash a password using a salt.

Other packages like bcrypt (blowfish algorithm) and node-argon2 (argon2 algorithm) can be used to address more specific cases like brute force attacks.

Multi-Factor Authentication (MFA) is also becoming a standard security practice for web applications and should be added as an additional layer of security.

Protect Against Vulnerabilities

Cross-site scripting (XSS), clickjacking, and injections are different types of attacks you may run into.

Adding proper headers mitigates many different vulnerabilities. If you’re application uses Express, you can use an NPM package called Helmet.

Helmet sets the following headers and can be customized accordingly:

• Content-Security-Policy: A powerful allow-list of what can happen on your page which mitigates many attacks

• Cross-Origin-Opener-Policy: Helps process-isolate your page

• Cross-Origin-Resource-Policy: Blocks others from loading your resources cross-origin

• Origin-Agent-Cluster: Changes process isolation to be origin-based

• Referrer-Policy: Controls the Referer header

• Strict-Transport-Security: Tells browsers to prefer HTTPS

• X-Content-Type-Options: Avoids MIME sniffing

• X-DNS-Prefetch-Control: Controls DNS prefetching

• X-Download-Options: Forces downloads to be saved (Internet Explorer only)

• X-Frame-Options: Legacy header that mitigates clickjacking attacks

• X-Permitted-Cross-Domain-Policies: Controls cross-domain behavior for Adobe products, like Acrobat

• X-Powered-By: Info about the web server. Removed because it could be used in simple attacks

• X-XSS-Protection: Legacy header that tries to mitigate XSS attacks, but makes things worse, so Helmet disables it

For DDoS and brute force attacks, rate limiting policies can be added to the load balancer (like NGINX).

During development, security rules can be enforced using a linter. ESLint provides a security plugin for identifying potential hotspots.

Integrate Monitoring Tools

There are many different Application Performance Monitoring (APM) tools on the market.

Datadog, New Relic, and Dynatrace are some of the more popular ones, all with different price points.

If you’re using a cloud platform, there’s native tools. For AWS, CloudWatch can be used for monitoring and observability (logs and metrics).

X-Ray can be used for request tracing. This is useful for debugging issues spanning multiple services (end-to-end).

Open source is also an option. A combination of Prometheus, Loki, and Grafana can be used to create dashboards and track metrics.

Practice Debugging

As long as there’s code, there’s going to be bugs. Bugs need to be fixed, and code needs to be maintained. It’s inevitable.

Debugging is a skill and takes practice.

Learn how to use the debugger (IDE) and how to debug memory leaks and performance issues.

For memory leaks, a combination of a heap profiler, heap snapshots, and garbage collection traces can be used.

For performance issues, Node.js has a built-in profiler (from V8) to sample the stack at certain intervals. A generated tick file shows which functions take the most CPU time.

Linux Perf is another tool that can be used for low level CPU profiling. The output can then be used to generate a flame graph.

Flame Graph For Visualizing Stack Traces

Netflix used flame graphs to debug a performance issue related to Express.

If you made it this far, thank you for reading! I hope you enjoyed it.

If I made a mistake, please let me know.

P.S. If you’re enjoying the content of this newsletter, please share it with your network and subscribe: https://www.fullstackexpress.io/subscribe

Resources

[1] “Node.js Documentation,” nodejs.org.
https://nodejs.org/docs/latest/api/.

[2] “User Journey,” nodejs.org.
https://nodejs.org/en/learn/diagnostics/user-journey.

[2] “Docker Documentation,” docs.docker.com.
https://docs.docker.com/.

Stephen here.

I spend my weekends researching, learning, and creating content for this newsletter.

It would mean the world to me, if you took a few seconds of your time to check out The Rundown AI.

They’re the fastest growing AI newsletter for a reason (I’m a subscriber too).

Now let’s get on with the show.

Stay up-to-date with AI.

The Rundown is the world’s fastest-growing AI newsletter, with over 500,000+ readers staying up-to-date with the latest AI news and learning how to apply it.

Our research team spends all day learning what’s new in AI, then distills the most important developments into one free email every morning.

Sign up with one click.

Jersey Mike’s has been my latest fast food addiction.

My go-to order is a #9 Club Supreme, Mike’s Way, with cherry pepper relish and jalapeños.

So good.

Don’t believe me? Give it a try some time. You won’t regret it.

But what separates Jersey Mike’s from other sandwich chains is their commitment to giving back to the community.

During the COVID pandemic, Jersey Mike’s ran an advertising campaign offering 50% off all sandwiches.

To make the deal even sweeter, the company offered free deliveries.

As you can guess, the majority of these orders came from their mobile app.

The ad went viral, jumping from an average of 800 orders a day to 80,000+ orders within the first four hours.

Jersey Mike’s legacy infrastructure couldn’t fulfill the orders, and customers were unhappy.

Changes needed to be made.

A Sub Above

Jersey Mike’s used AWS for their existing services and pipelines. The logical choice was to continue building off the current environment.

But that’s not the case.

The team wanted a new mobile app. Like an eCommerce platform that provided customers with a complete ordering experience.

An app with a rewards system, exclusive offers and multiple ordering options (delivery, pickup, catering).

To reach these goals, the team needed to build from the ground up.

An environment capable of supporting and maintaining a business-critical application.

A system built around security, reliability, and performance.

Freshly Made, Right in Front of You

Following the Cloud Foundations Framework, Jersey Mike’s built their deployment around AWS Control Tower.

Jersey Mike’s Cloud Foundation Around AWS Control Tower

The first step was to setup the following key services:

• AWS Organizations: Centrally manage multiple AWS accounts.

• AWS Single Sign-On: Manage user access to AWS applications and services (now known as AWS IAM Identity Center).

• AWS CloudTrail: Audit, monitor, and troubleshoot user activity and API usage.

• AWS Config: Manage resource configuration changes.

• Amazon SNS: Publish messages to subscribers.

By creating a proper landing zone and leveraging Account Factory, Jersey Mike’s built an automated provisioning strategy for multiple accounts.

Automated Account Provisioning Strategy With Account Factory

This strategy provisioned accounts in a standardized, secure and repeatable way.

It’s Not Just a Sandwich, It’s a Jersey Mike’s

With the core AWS accounts completed, the team used AWS CloudFormation to automate infrastructure within each account.

The team created CloudFormation templates to do the following:

• Deploy infrastructure components: EKS resources, Fargate, RDS, and more.

• Deploy VPC for Flow Logs: Logs aggregated to centralized S3 bucket.

• Attach VPC to AWS Transit Gateway: Gain visibility of network traffic patterns.

How about scalability?

Jersey Mike’s new mobile app needed to scale with demand in a cost efficient way.

By pairing EKS with Fargate, the system would scale accordingly, and the company would only pay for the resources needed to run the app.

In addition, both services can easily be integrated into existing application pipelines.

Scaling Jersey Mike’s Mobile App with EKS on Fargate

As an influx of orders came in, Kubernetes would deploy more pods to handle the increased load through autoscaling. This is also known as horizontal scaling.

With these new infrastructure changes, Jersey Mike’s is able to meet high user demand, optimize cloud spend, and reduce the operation workload for their team.

If you made it this far, thank you for reading! I hope you enjoyed it.

If I made a mistake, please let me know.

P.S. If you’re enjoying the content of this newsletter, please share it with your network and subscribe: https://www.fullstackexpress.io/subscribe

Resources

[1] “Jersey Mike’s is a Brand that Believes,” jerseymikes.com.
https://www.jerseymikes.com/culture.

[2] “Jersey Mike’s: Securely Scaling Cloud Capabilities to Meet Demand,” wwt.com.
https://www.wwt.com/case-study/jersey-mikes-securely-scaling-to-meet-demand.

[3] “AWS Well-Architected,” aws.amazon.com.
https://aws.amazon.com/architecture/well-architected.

[4] “Cloud Foundations Framework Overview,” aws.amazon.com.
https://aws.amazon.com/architecture/cloud-foundations/cloud-foundations-framework-overview/.

This is the final part of the SoundCloud series. If you haven’t read the previous parts, I encourage you to do so. You can read Part 1 here and Part 2 here.

Hi, I’m Stephen!

I spend my weekends researching, learning, and creating content for this newsletter.

It would mean the world to me, if you took a few seconds of your time to check out Bright Data.

At least see why they’re the world’s #1 web data platform. They even have their own dog walkers.

Go ahead and check them out. Now let’s get on with the show.

Web Intelligence, Unlocked

With Bright Data's cutting-edge proxy solutions, harness the full potential of web data for your business. Tap into our global proxy network to scale your data collection activities. Ecommerce platforms, travel agencies, financial institutions, and market researchers are all leveraging web data to gain a competitive edge.

Bright Data offers the scalability and flexibility necessary for gathering and analyzing web data. Take the first step towards data-driven excellence.

Previously, we learned why SoundCloud pivoted towards a microservices architecture.

The catalyst for this decision started with the engineering team’s inefficient development process.

In order to deliver more features, the team as a whole needed to be more productive.

As a result, a collaborative approach was implemented across the engineering, design, and product teams.

All working together in unison to deliver new features.

SoundCloud’s progression towards a microservice architecture proved to be challenging.

Even though new services were stood up, they still had to work in conjunction with the Mothership.

This is the evolution of SoundCloud’s architecture over the last 10 years.

Backends for Frontends

The combination of autonomous teams, new microservices, and an internal API led to a new concept: Backends for Frontends (BFFs).

Phil Calçado first introduced this architectural pattern at SoundCloud in 2013.

Backends for Frontends Pattern

The pattern focuses on creating multiple, dedicated API gateways for each client platform (web, mobile). BFFs are also responsible for:

• Rate limiting

• Authentication

• Header sanitization

• Cache control

One advantage is autonomy. Separate APIs allow each specific client type to be optimized.

Another is resilience. If a BFF goes down in an availability zone, the rest of the platform would still be available.

Finally, sometimes ugly workarounds and mitigations are necessary. It’s much easier to implement these in the BFF versus a more complex application like the Mothership.

Today, SoundCloud handles dozens of BFFs, each one powering a dedicated API (Mobile, Web, Public, Partner).

These BFFs handle hundreds of millions of requests per hour.

With Great Power Comes Great Responsibility

Like any solution, there’s always tradeoffs.

For example, multiple BFFs can oftentimes lead to duplicate code. Even worse, inconsistent implementations.

Authorization rules that can only be applied at integration time can also be problematic. Imagine if private or unreleased tracks were leaked.

Two areas needing improvement were the Track and Playlist entities (more on these later).

As they grew, these entities were decomposed into multiple microservices.

How does one decide when to create a new BFF or when to reuse an existing one?

And how many BFFs is too many?

Both valid questions. Clearly, there’s a tradeoff between autonomy and operational overhead.

BFFs require extensive collaboration between frontend and backend engineers.

Value-Added Services

With the amount of business logic and authorization code living in the BFFs, they became difficult to maintain.

Keeping code synchronized and consistent was a big challenge.

To address these issues, the team needed to make some architecture improvements.

Enter Value-Added Services (VAS).

VAS are business services solely responsible for returning entities and associated value objects (aggregate).

Consolidating the business logic into its own service keeps the BFFs clean. In doing so, centralized point is created where metadata and authorization rules can be defined.

This is known as the separation of concerns principle.

SoundCloud’s Architecture Containing Multiple Service Layers

The new architecture is divided into three core service layers:

1. Edge: BFFs live in this layer and maintain dedicated APIs. Each API is tailored to a specific client.

2. Value Added: Consumes and processes data from other services (VAS or Foundation) for user experience.

3. Foundation: Serves as the building blocks around different domains and contains low-level services.

Value-Added Service’s Building Blocks

Each VAS also consists of the following building blocks:

1. Domain: Represents a user or business concern. Used to draw boundaries around service integrations.

2. Entity: Represents an object with an independent identifier and lifecycle.

3. Value Objects: Contains an entity’s metadata. Also tied to the lifecycle of the entity.

4. Aggregate: Contains one or more relate entities.

Tracks and Playlists

In 2019, the team began applying the VAS pattern, starting with the Tracks VAS.

A track is an entity with associated value objects like metadata, transcodings, and authorization policies to determine who can see that track.

Since each track has an owning user (user is its own entity), the track object holds a user id as a reference.

Simultaneously, if another consuming service has a track id that needs to be resolved, that service can call the Tracks VAS.

The Tracks VAS will then determine whether the request is authorized and return the track aggregate accordingly.

This effort was successful and in 2020, the team started a major refactoring of the Public API.

Up first were the track-related endpoints.

With the new Tracks VAS, all track-related endpoints just had to be re-routed to the new service (Public API was calling the Mothership directly).

But the playlist-related endpoints were far more complex. Business logic and authorization rules were spread across multiple BFFs and services.

Original Architecture Without VAS

The logical solution was to stand up a new Playlists VAS to centralize all playlist-related logic.

By doing so, cross-platform feature development, refactoring and optimizations would be easier and faster.

New Architecture With VAS

This also solved the authorization inconsistencies across multiple services and reduced the risk of potential vulnerabilities.

SoundCloud’s Architecture Today

As SoundCloud’s platform grew, so did the scope of each VAS.

For example, SoundCloud provides music tracks to consumer applications.

But the platform also provides tools for creators to upload and distribute their music.

As both use cases grew, so did the complexity of the Tracks VAS. The engineering team determined that the VAS pattern needed to evolve.

A more scalable approach was needed.

What was originally one domain has now grown into two similar but different business domains.

As a result, using the same implementation for both was no longer the best solution.

It was time to create different implementations specific for each use case (same line of thinking as BFFs).

Different Domain Gateways For Tracks

This led to the adoption of Domain Gateways and a Domain-Oriented Microservice Architecture (DOMA), an architecture also used by Uber.

Each business domain would have its own implementation, but still rely on the same foundational layer of services.

Overall, this façade provides both stability and an anti-corruption layer for each of the domains.

This pattern is still being implemented at SoundCloud today.

If you made it this far, thank you for reading! I hope you enjoyed it.

If I made a mistake, please let me know.

P.S. If you’re enjoying the content of this newsletter, please share it with your network and subscribe: https://www.fullstackexpress.io/subscribe

Resources

[1] “Evolution of SoundCloud’s Architecture,” developers.soundcloud.com.
https://developers.soundcloud.com/blog/evolution-of-soundclouds-architecture.

[2] “Building Products at SoundCloud,” developers.soundcloud.com.
https://developers.soundcloud.com/blog/building-products-at-soundcloud-part-1-dealing-with-the-monolith.

[3] “Service Architecture at SoundCloud,” developers.soundcloud.com.
https://developers.soundcloud.com/blog/service-architecture-1.

[4] “How SoundCloud Uses HAProxy with Kubernetes for User-Facing Traffic,” developers.soundcloud.com.
https://developers.soundcloud.com/blog/how-soundcloud-uses-haproxy-with-kubernetes-for-user-facing-traffic.

[5] “The End of the Public API Strangler,” developers.soundcloud.com.
https://developers.soundcloud.com/blog/end-of-the-strangler.

[6] “How We Develop New Features Using Offsites and Clean Architecture,” developers.soundcloud.com.
https://developers.soundcloud.com/blog/how-we-develop-new-features-using-offsites-and-clean-architecture.

[7] “How We Ended Up With Microservices,” philcalcado.com.
https://philcalcado.com/2015/09/08/how_we_ended_up_with_microservices.html.

[8] “BFF @ SoundCloud,” thoughtworks.com.
https://www.thoughtworks.com/insights/blog/bff-soundcloud.

[9] “Pattern: Backends For Frontends,” samnewman.io.
https://samnewman.io/patterns/architectural/bff/.

[10] “Pattern: Backends For Frontends,” developers.soundcloud.com.
https://developers.soundcloud.com/blog/announcing-twinagle.

[11] “Microservices and the Monolith,” developers.soundcloud.com.
https://developers.soundcloud.com/blog/microservices-and-the-monolith.

This is Part 2 of the SoundCloud series. If you haven’t read Part 1, I encourage you to do so. You can read it here.

Previously, we learned SoundCloud wasn’t initially built for scale. The platform was built for opportunity.

The team was adaptive, product-driven, and developed a simple architecture.

To SoundCloud, scaling was a luxury problem. A problem that could be addressed over time.

SoundCloud at this point was a combination of a large social network and music distribution platform.

Every minute, there was about 12 hours of music uploaded, with hundreds of millions of daily users.

The Mothership was battle-tested against peak performance, and great tooling was created to facilitate the next level of growth.

But as it turns out, technical challenges were only a small piece of the puzzle.

The core issue holding SoundCloud back was in their development process.

Visualizing the Development Process

The engineering team at the time was really made up of two core teams:

1. App team: Responsible for the Mothership and old (current) user interface.

2. Web team: Responsible for v2, branded as The Next SoundCloud, a single-page JavaScript web application.

There was a big disconnect as both teams worked in isolation (separate buildings) and communicated mainly through issue trackers and IRC.

In order to identify problems in the development process, the teams used a method called value-stream mapping.

This tool can be used to visualize critical steps in a process, and quantify the time taken in each stage.

In this case, it would be the different stages the teams need to deliver a feature.

Value-Stream Mapping: Original Process

With the current process, it would take over two months to go live with a feature.

More than half the time was waiting for the feature to be picked up by the next person responsible.

The flow looked something like this:

1. There’s an idea for a new feature. Spec and screen mockups are written and stored in Google Drive.

2. Waiting

3. Design team gets the spec and designs user experience. Development card is added to the Web team’s Trello board.

4. Waiting

5. A front-end engineer converts the design into client-side code using mock data. A story is created in Pivotal Tracker with the necessary API (Rails) changes.

6. Waiting

7. A back-end engineer writes the code, integration tests, and any changes needed to get the API live. Card in Trello is updated.

8. Waiting

9. A front-end engineer implements the new backend changes and gives the green light for deployment.

10. Deployments were risky and painful, so the App team waits for several features before deploying to production. Rollbacks happened frequently due to unrelated code issues.

Out of a grand total of 66 days, only 11 days was actually used for development.

From 66 Days to 24 Days

In order to improve the development process, fundamental changes were needed.

Through value-stream mapping, letting features accumulate in the main branch before deployment was not ideal.

Instead, the engineering team adopted an Agile Release Train (ART) approach for the Mothership. In fact, regardless of how many features were in the main branch, deployments would happen every day after standup.

As a result, features were deployed to production when they were ready, not several days later.

Another problem was the disconnect between the Web and App teams. Not only were the teams working in isolation, but the back-end developers felt they didn’t have a say in the product.

The process was extremely frontend-driven.

To address this issue head on, the process was modified to pair the front-end and back-end developers together.

Value-Stream Mapping: Pairing Front-End and Back-End Engineers

This change ensured constant communication between the two and would hold the pair responsible for the completion of a feature.

Last but not least, the team added a mandatory code review to the process. Every pull request had to at least be approved by a second engineer before the changes could be merged to the main branch.

The collaboration-driven approach worked well for the engineering team as a whole. So much so, it was also implemented across the design and product teams as well.

Value-Stream Mapping: Final Process

The end result was 24 days.

A 42 day improvement for a feature to go from idea to go live!

Building Alongside the Mothership

Even though the process improved significantly, the previous diagram still showed features sitting in GitHub for 7 days before being reviewed.

Why would a new feature sit in queue for so long?

The Mothership codebase was massive and complex.

Not only did large changes require time to be reviewed, big changes were extremely risky due to all the tightly-coupled code over the years.

The first thought is to create smaller pull requests.

Breaking large code changes into smaller pull requests meant that things would be more manageable and reviews would be easier, but could also lead to unforeseen architecture mistakes.

Well, why does a single codebase need to implement so many features and components?

Why not break the monolith into multiple, smaller systems?

Now that the development process has been addressed, it’s time to tackle the technical challenges.

Even though the Mothership had a battle-tested architecture, in order for the team and the platform to continue to scale, a new strategy had to be implemented.

A strategy that could be implemented incrementally and could deliver value right away.

Enter microservices.

Adding Microservices Into the Mix

Instead of splitting up the monolith immediately, the engineering team made the decision to first build new features as microservices.

Nothing new would be added to the Mothership, and any feature that needed refactoring in the monolith, would be extracted out.

But as more microservices were created, a new problem arose.

Since the Mothership still had the majority of the logic, the majority of the microservices still had to communicate with it in some shape or form.

A common approach would be to have the microservices access the Mothership’s database directly. The issue with this approach is the public versus published interfaces problem.

Having both the Mothership and microservices share the same tables would make it difficult to make table structure changes down the road.

Instead, the engineering team decided to have the microservices consume the Public API, a published interface.

This meant that SoundCloud’s own internal microservices would behave exactly like the third-party applications integrating with the platform.

This model had a major flaw: The microservices needed to be aware of user activity updates.

For example, if a track received a new comment, the push notifications system needed to know about it.

With the current architecture, there wasn’t a way for these microservices to be notified. Polling would also be a bad solution at this scale.

The engineering team decided on two major architecture changes:

1. A new model called Semantic Events was created.

2. A new Internal API was created using Rails’ engines.

SoundCloud’s Transition to Microservices Architecture

Whenever there was a change in the domain objects, a corresponding message would be dispatched to the broker.

This message could then be consumed by any of the microservices, allowing them to react appropriately to the event.

These new changes also enabled Event Sourcing, allowing the microservices to deal with shared data.

Since the Public API was used by external applications, there were limitations on certain data that could be accessed. By creating an Internal API, microservices could access private data and notify users of private information.

The main idea around the new architecture changes was to break the coupling between the Mothership and the new microservices, by adding push and pull interfaces.

Contrary to what most people think, the catalyst for SoundCloud’s move towards a microservice architecture was driven mostly to improve team productivity.

Without streamlining the development process first, the team wouldn’t be able to produce and scale fast enough to build out the features the platform needed.

These new changes led to shorter feedback cycles, and allowed the team to push out production-ready features at a faster pace.

In Part 3 of this series, we’ll learn about SoundCloud’s new challenges and design patterns with the microservices architecture.

If you made it this far, thank you for reading! I hope you enjoyed it.

If I made a mistake, please let me know.

P.S. If you’re enjoying the content of this newsletter, please share it with your network and subscribe: https://www.fullstackexpress.io/subscribe

Resources

[1] “Evolution of SoundCloud’s Architecture,” developers.soundcloud.com.
https://developers.soundcloud.com/blog/evolution-of-soundclouds-architecture.

[2] “Building Products at SoundCloud,” developers.soundcloud.com.
https://developers.soundcloud.com/blog/building-products-at-soundcloud-part-1-dealing-with-the-monolith.

[3] “Service Architecture at SoundCloud,” developers.soundcloud.com.
https://developers.soundcloud.com/blog/service-architecture-1.

[4] “How SoundCloud Uses HAProxy with Kubernetes for User-Facing Traffic,” developers.soundcloud.com.
https://developers.soundcloud.com/blog/how-soundcloud-uses-haproxy-with-kubernetes-for-user-facing-traffic.

[5] “The End of the Public API Strangler,” developers.soundcloud.com.
https://developers.soundcloud.com/blog/end-of-the-strangler.

[6] “How We Develop New Features Using Offsites and Clean Architecture,” developers.soundcloud.com.
https://developers.soundcloud.com/blog/how-we-develop-new-features-using-offsites-and-clean-architecture.

[7] “How We Ended Up With Microservices,” philcalcado.com.
https://philcalcado.com/2015/09/08/how_we_ended_up_with_microservices.html.

[8] “BFF @ SoundCloud,” thoughtworks.com.
https://www.thoughtworks.com/insights/blog/bff-soundcloud.

[9] “Pattern: Backends For Frontends,” samnewman.io.
https://samnewman.io/patterns/architectural/bff/.

[10] “Pattern: Backends For Frontends,” developers.soundcloud.com.
https://developers.soundcloud.com/blog/announcing-twinagle.

[11] “Microservices and the Monolith,” developers.soundcloud.com.
https://developers.soundcloud.com/blog/microservices-and-the-monolith.

[12] “How SoundCloud’s Broken Business Model Drove Artists Away,” theverge.com.
https://www.theverge.com/2017/7/21/15999172/soundcloud-business-model-future-spotify-streaming.

Stephen here.

I spend my weekends researching, learning, and creating content for this newsletter.

It would mean the world to me, if you took a few seconds of your time to check out Clumio.

At least see why they were able to raise $75 million. Even Atlassian is using it for Jira.

Doing so supports my work. Now let’s get on with the show.

Get smarter about protecting your S3 data

With rising costs for Amazon S3 storage and potentially devastating business consequences from data loss, you need a holistic approach to cutting unnecessary spending and guarding against risks. Lawrence Miller, a consultant to multinational corporations who holds numerous networking certifications, has authored a concise volume that lays out the path to success in managing backup and compliance for S3 data lakes.

Remember how difficult it used to be to find music?

People used to use record players. And then cassette players. And then MP3 players.

People pirated music through Napster and LimeWire.

Separate devices had to be carried around in order to listen to music on the go.

And then everything changed.

The Apple iPhone was released in 2007, transforming the way we access and listen to music. iPod, phone, and Internet all-in-one.

This revolutionary product paved the way for companies like SoundCloud today.

As engineers, we can learn many valuable lessons analyzing the evolution of SoundCloud’s architecture over the last two decades.

Scaling is a Luxury Problem

From the start, the engineering team optimized for opportunity.

Instead of designing an architecture that could support millions of users, they started with a simple setup: Ruby on Rails application (called Mothership), Apache web server, and MySQL database.

SoundCloud’s Initial Architecture

Simple right?

SoundCloud launched in 2008. There was no high availability. In fact, the architecture wasn’t even asynchronous.

If a new comment was posted on a track, communication was blocked until all followers were notified.

What would be the reason for this?

The answer goes back to optimizing for opportunity. They leveraged a simple tech stack the team knew well, and focused on delivering value to their users.

As a result, SoundCloud was able to move fast and build a “sticky” platform with strong product-market fit.

One way of showing this was by eating their own dog food.

From the beginning, SoundCloud’s public API was developed alongside their website.

Third-party applications integrating with SoundCloud used the same exact APIs used by the engineering team’s internal application.

Shortly after, Apache was switched out for Nginx (incremental changes).

SoundCloud Changes Web Servers

Nginx provided better connection pooling and simplified routing configurations between different environments.

The Grocery Store and the Post Office

As SoundCloud grew, traffic grew.

And as traffic grew, there was a growing problem: some workloads took much longer than others (hundreds of milliseconds).

This was a problem with Nginx at the time. More specifically HTTP/1.

In computer networking, there’s something called Head-of-Line (HoL) blocking. This happens when slower requests clog up connections, and other requests are stuck waiting to be processed.

Imagine waiting in line at the grocery store to checkout. Because the first customer has a cart full of items, the rest of the line is delayed.

Grocery Store HoL Blocking Example

Today, HoL blocking is partially solved with HTTP/2 through multiplexing requests over a connection. This only solves blocking at the HTTP level but blocking can still happen at the TCP level.

This begs the question, how can the current architecture process requests concurrently?

When SoundCloud’s architecture was initially developed (2008), concurrent request processing in Rails was still considered immature.

Instead of investing more time auditing dependencies, the engineering team decided to stay with the existing model:

1. Single concurrency per application server process.

2. Multiple processes per host.

Even though there are multiple processes per host, SoundCloud was already experiencing high traffic. Several long-running requests could easily recreate the HoL blocking problem.

For example, with 5 processes instead of 1, the system would theoretically be able to handle an average of 5 times as many slow requests.

Again, picture yourself at the post office. You’re waiting in line for an available worker to help you with your package. With more people, there’s a higher chance of delays (people with multiple packages).

Post Office Example

How could this HoL problem be solved?

The engineering team came to a realization. What they wanted was a system that never queued. Or at least a queue with minimal wait time.

To accomplish this, they had to make sure each Rails application server never received more than one request at a time.

They made the following changes:

1. Ensured servers were stateless.

2. Added HAProxy to infrastructure.

3. Configured backend with a maximum connection count of 1.

With a multi-server queueing model (M/M/c) and HAProxy as the queueing load balancer, any temporary back-pressure would be buffered.

Again, simple design choices.

Synchronous to Asynchronous

These new changes may have solved the HoL issue, but long-running requests is still a problem in and of itself.

One example is user notifications.

When a user uploads a new track to SoundCloud, the user’s followers are notified. This may be fine for a users with less followers, but for more popular users, there were huge delays.

In fact, the fan-out of notifications would frequently exceed tens of seconds. These long-running requests needed to be jobs (queue) instead.

Remember how everything was still synchronous in this architecture?

Enter RabbitMQ.

Since storage was also growing rapidly for sounds and images, the team decided to offload assets to Amazon S3. Storage scaled nicely, while transcoding compute stayed in Amazon EC2.

To connect everything, RabbitMQ’s Advanced Messaging Queuing Protocol (AMQP) was used as a middleware to manage the lifecycle of these jobs.

“One broker to queue them all.”

Jobs fell into two main categories based on work time:

1. Interactive - less than 250ms work time

2. Batch - everything else

Identifying Points of Scale

At this point, SoundCloud was at the hundreds of thousands user mark.

In order for the architecture to continue to evolve, the engineering team knew they needed to decouple the read and write paths.

The read and write paths could then be individually optimized.

This technique resembles the Command Query Responsibility Segregation (CQRS) pattern today.

One area of focus was the widget.

As it turns out, SoundCloud’s highest volume request was a single endpoint delivering data for the widget.

Memcached and Varnish were added to cache the following:

1. Full pages

2. DOM fragments

3. Partially rendered templates

4. Read-only API responses

These performance improvements solved CPU issues (rendering engine and runtime) in the application tier.

Another area of focus was the Dashboard, a user’s personalized view of activities.

When the Dashboard receives an update, the appropriate users are notified across all devices and third party applications.

The read path needed to be optimized for sequential access per user over a time range.

The write path needed to be optimized for random access where one event can affect the indexing of millions of users.

To account for these optimizations, Cassandra was chosen as the storage system and Elasticsearch was chosen to enhance search. These solutions provided persistence and scaling.

The final monolithic architecture looked like this:

SoundCloud’s Monolithic Architecture

The Story Continues

SoundCloud was known as the “YouTube for audio”, with unique features such as the waveform player, track comments, and collaborations.

The engineering team’s architecture decisions in its early years allowed the company to be adaptive and product-driven.

The key takeaways from SoundCloud’s early success can be summarized in three points:

1. Optimize for opportunity. Focus on the product.

2. Scaling is a luxury problem. Architect for growth over time.

3. Identify points of scale. Define integration points well for organic growth.

In the next part of this series, we’ll learn about SoundCloud’s transition to a microservices architecture, and their challenges of scaling to hundreds of millions of users.

If you made it this far, thank you for reading! I hope you enjoyed it.

If I made a mistake, please let me know.

P.S. If you’re enjoying the content of this newsletter, please share it with your network and subscribe: https://www.fullstackexpress.io/subscribe

Resources

[1] “Evolution of SoundCloud’s Architecture,” developers.soundcloud.com.
https://developers.soundcloud.com/blog/evolution-of-soundclouds-architecture.

[2] “Building Products at SoundCloud,” developers.soundcloud.com.
https://developers.soundcloud.com/blog/building-products-at-soundcloud-part-1-dealing-with-the-monolith.

[3] “Service Architecture at SoundCloud,” developers.soundcloud.com.
https://developers.soundcloud.com/blog/service-architecture-1.

[4] “How SoundCloud Uses HAProxy with Kubernetes for User-Facing Traffic,” developers.soundcloud.com.
https://developers.soundcloud.com/blog/how-soundcloud-uses-haproxy-with-kubernetes-for-user-facing-traffic.

[5] “The End of the Public API Strangler,” developers.soundcloud.com.
https://developers.soundcloud.com/blog/end-of-the-strangler.

[6] “How We Develop New Features Using Offsites and Clean Architecture,” developers.soundcloud.com.
https://developers.soundcloud.com/blog/how-we-develop-new-features-using-offsites-and-clean-architecture.

[7] “How We Ended Up With Microservices,” philcalcado.com.
https://philcalcado.com/2015/09/08/how_we_ended_up_with_microservices.html.

[8] “BFF @ SoundCloud,” thoughtworks.com.
https://www.thoughtworks.com/insights/blog/bff-soundcloud.

[9] “Pattern: Backends For Frontends,” samnewman.io.
https://samnewman.io/patterns/architectural/bff/.

[10] “Pattern: Backends For Frontends,” developers.soundcloud.com.
https://developers.soundcloud.com/blog/announcing-twinagle.

Rocket Turtle, Node’s New Mascot

How It Started

$60,000, Google Chrome, and six months in a Starbucks. This was all that was needed to create the first prototype for Node.js.

Node.js was created in 2009 by a brilliant engineer named Ryan Dahl. If you don’t know his story, I would encourage you to read about it. It’s a good one.

I had just started college at the time, on track to becoming a petroleum engineer. I hated it, I changed majors halfway through, and ended up learning ARM and C as my first programming languages (for microcontrollers).

I remember those classes. They were hard. But they taught me the fundamentals and how to do things like memory management. And what pointers are. So when I transitioned to the web, I found JavaScript and Node.js to be different, but in a pleasant way. It was my gateway drug into web development.

I didn’t have to worry about allocating and freeing up memory because there’s garbage collection. Node.js was simple and easy to work with. There was only one rule: Don’t block the event loop.

Event Loop

I can write code in JavaScript (or TypeScript), enjoy fast startup times, and prototype quickly. What’s not to like about all of that?

Yes, there are things in JavaScript that are hard to explain. In fact, there’s an entire GitHub repository highlighting weird scenarios.

There’s node_modules, which gets bloated from long dependency chains and long install times. Getting ES Modules, TypeScript, and Jest to work correctly together is a pain.

The list goes on, but like all technologies, there’s always pros and cons. Like this ad.

Take control of your AWS spend and cut backup bills by 50%

As cloud adoption increases to run modernized applications, costs can quickly rise out of control. How do best-in-class companies manage their storage spend while continuing to grow the business? Clumio, a cloud-native backup solution, depends on cloud storage to run their entire business. They took a FinOps approach to optimizing their costs, and reduced their AWS dev costs by over 50%.

The Growth of Node.js

Many developers complain about the standard library. They say it’s not as complete as other languages (like Python or C#). They say it’s missing features like data manipulation out of the box.

But that’s okay. That’s the beauty of Node.js.

Node.js Runtime

It’s a JavaScript runtime. It allows us to run JavaScript anywhere and everywhere. We can build web apps, mobile apps, desktop apps, etc. SpaceX is even using it in outer space!

By keeping the core standard library small, it’s much easier to maintain. Anything else that’s needed, we can use external packages from npm or build it ourselves. I know it’s not perfect but over the last decade, the most used npm packages have matured. Many are battle tested and used in production for many years.

For example, Express, one of the most popular packages, was published over 13 years ago. The package is in millions of projects. Another is Axios, an HTTP client still widely used today.

Node.js has been a catalyst for the open source community’s growth. Many great developers sharing their work and helping others. New projects and tooling around the JavaScript ecosystem.

A great example of this is core-js. The creator, Denis Pushkarev, worked extremely hard to standardize JavaScript across the web. His project provided universal polyfills for the latest ECMAScript standards. His story is amazing.

Remember Lodash? John-David Dalton created this library. It’s used in more than 28 million repositories on GitHub. He also created esm, the popular ECMAScript module loader. Now, he’s working on Bun.

Matteo Collina, member of the Node.js Technical Steering Committee, created many projects for the community. You’ve probably used Fastify or Pino at some point. Yeah, he created them.

Without Node.js, there wouldn’t have been Deno or Bun. And there’s many more to come.

The New Wave of Developers

Is Node.js the answer for everything? Of course not.

Node.js is great at I/O operations and sending data over the network. But that doesn’t mean developers shouldn’t use tools and technologies they enjoy.

There’s countless examples of people complaining about Node.js. Here’s one. And of course, we can’t forget about this video.

Not only does this spread negativity (which we have enough of already in the world), but also confuses aspiring developers. Let’s discuss openly and share, but be kind towards each other in the community. After all, we’re all just trying to improve and get better each day.

Instead of gatekeeping, lets help the next wave of developers break into the industry. Regardless if someone has a degree, was a bootcamp graduate, or came from a different field, let’s welcome them with open arms.

Programming is not just a valuable skill in the modern age, it’s a superpower. Anyone can build anything at anytime from the comfort of their home.

How cool is that?

It’s important as developers, we don’t lose sight of the goal. There’s so much back and forth these days about how one language is better than the other. Or how one technology is superior. At the end of the day, it doesn’t matter. Focus on delivering value. John Carmack said it best:

To the newcomers, it doesn’t matter how you start. What matters is that you continue to learn. After Node.js, I learned many other technologies like Kafka and Kubernetes. Several other programming languages like Python, C#, and Go. Looking back at how my career developed, I do consider myself lucky. But if you work hard enough and long enough, you will get lucky eventually. I believe that.

If you’re new, pick a language you’re curious about. Become a student of the craft. Don’t let AI distract you. Don’t use autocomplete. Go deep on topics. Learn how things actually work.

And most importantly, don’t give up, because the world needs more engineers like you.

P.S. If you’re enjoying the content of this newsletter, please consider subscribing or share it with your network: https://www.fullstackexpress.io/subscribe

Young And Old Enjoying Pokémon GO

My mother-in-law is absolutely obsessed with this game.

As a developer, I had a burning desire to understand how Pokémon GO started, its architecture, and the individual components driving this popular game.

This is Pokémon GO as I know it.

Origin Story

In the last 30 days, there’s been more than 81 million active players. Astronomical numbers, but it didn’t start off this way. In fact, Pokémon GO is the result of many iterations.

Niantic, the company behind Pokémon GO, was founded by John Hanke and Phil Keslin. Not only were they Google veterans, but they were originally the co-founders of a company called Keyhole.

Keyhole was the company Google acquired to start Google Earth and later Google Maps. The core technology behind Keyhole was the ability to stream large databases of mapping data across the internet.

As experts in 3D mapping, John and Phil will go on to leverage this technology to create Niantic’s first augmented-reality multiplayer game, Ingress. Ingress played a crucial role in the development of Pokemon GO by forming the data pool of locations that later became the PokéStops and gyms in the game.

A combination of historical markers, geo-tagged photos from Google, and popular player-submitted places were chosen as the starting point for Pokémon GO.

High-Level Architecture

Today, Pokémon GO handles millions of player requests a second through every day gameplay.

With many founders and early engineers originally from Google, choosing Google Cloud Platform (GCP) as their cloud infrastructure was the logical choice.

Pokémon GO: High-Level Architecture

At a high level, Pokémon GO’s architecture consists of the following key components and services:

• Google Kubernetes Engine (GKE): Manages and scales workloads and services needed for the game.

• Cloud Load Balancing: Distributes traffic across K8 clusters and services. Entry point for all requests.

• Cloud CDN: Caches and serves content to users.

• Spanner: Stores player, map and Pokémon data.

• Pub/Sub: Handles data use for analytics.

• Bigtable: Used for logging and tracking user actions.

• Dataflow: Processes player logs.

• BigQuery: Stores game data for analyzing player behavior and verifying features.

• Frontend Service: Handles player interactions in the app.

• Spatial Query Backend: Caches map data based on location.

In order to handle such high traffic, the engineering team leveraged two key GCP services: Google Kubernetes Engine (GKE) and Spanner.

GKE is a Kubernetes cluster manager and orchestration system for running Docker containers. Not only did Google create Kubernetes, but GKE implements the full Kubernetes API with robust autoscaling and support…up to 15,000 nodes!

With cluster autoscaling, the cluster’s node pool can automatically resize based on workload demands. Through GKE, pods can scale out horizontally based on CPU utilization or custom metrics (e.g. high network traffic). Additionally, pods can scale up as well based on CPU and memory usage.

From Datastore to Spanner

Originally, Niantic made the decision to store Pokémon GO data using Datastore (Firestore is the new version), a fully-managed NoSQL database service. With a managed service, the engineering team was able to meet the initial demands of Pokémon GO.

Though Datastore is considered highly scalable and can theoretically handle up to 740k operations per second, this is based on a ramp up schedule using the 500/50/5 rule. This rule states a maximum of 500 operations per second and increasing traffic by 50% every 5 minutes.

Based on these limitations as well as the natural growth of the game, the engineering team decided to migrate to another managed service called Spanner.

What’s particularly interesting is the transition from a NoSQL database to a relational database, which is most likely due to a need for global consistency and high transactionality. Spanner also boasts unlimited write and read scalability by decoupling compute resources from data storage.

At any given time, there’s more than 5,000 Spanner nodes and thousands of Kubernetes nodes running for Pokémon GO. This is not including the GKE nodes needed for other microservices that augment the gaming experience.

Navigating the Pokémon World

When a user first opens the Pokémon GO app, all static media such as Pokémon images and music files are downloaded to the user’s phone.

Photo by David Grandmougin

These files are stored in Cloud Storage and also cached with Cloud CDN at the Cloud Load Balancing level. By caching these files, content will load faster, traffic to the web servers are decreased, and overall user experience is improved.

As the user walks around in the physical world, the app uses the phone’s GPS to determine your location. This is important because your location determines the Pokémon you see on the map, and also the PokéStops, gyms, and landmarks you see around you.

All of the mapping data is fetched from Spanner and handled by a service called the Spatial Query Backend. This service caches all the mapping data which is sharded based on location.

As the user interacts with the app, requests are reverse proxied through NGINX to the frontend service, where the appropriate data is served back to the user, either from Spanner or from cache.

In a nutshell, the frontend handles player interactions within the game, while the Spatial Query Backend handles the geospatial data needed to render the map, what the user sees (e.g. Pokémon, Gyms, etc), and any other features that are location-based.

So what exactly happens behind the scenes when a user tries to catch a Pokémon?

Catching a Pokémon

Pokémon can be seen on the phone through Augmented Reality (AR). Once the user interacts with the Pokémon and the Pokémon is caught, an event is sent from the frontend service to Spanner through the Spanner API.

Catching Pokémon in AR+ Mode

This is where the player entity and player data is normally stored. The only difference is that the data pertaining to catching a Pokémon is not cached in the Spatial Query Backend, unlike the data for gym battles and PokéStops.

These events are handled as write requests, and since Spanner is strongly consistent, all nodes see the same data at the same time, regardless of which node is accessed. As a result, all subsequent read operations from any node will return the most recent write value.

For gym battles and PokéStops, once an update is received, the spatial data is updated in memory for caching, and then used to serve future requests from the frontend service. It’s important to note that these updates don’t apply to just a single user, but all the players in the same location.

All user actions are also stored in Bigtable for logging and tracking (e.g. business telemetry), and messages are published to GCP’s Pub/Sub messaging service for analytics.

Live Events

In addition to normal gameplay, there are regular live events. Events like Raid Battles, can see transactions increase from 400k per second to close to a million in minutes.

Players Teaming Up For Raid Battles

Niantic found that their GCP servers would get overloaded during the initial phase of these events when players were forming teams. This is due to many players trying to join the Raid lobby in preparation for the event, causing a large influx of traffic.

Even though Pokémon GO operates in a multi-server environment and players are typically evenly distributed across all the servers, Raids require all players to be on the same server in order for everyone to access the game data. As a result, a large amount of data needs to be shared across the team of players participating in the Raid, leading to increased server traffic and latency.

Not only does this cause lag and delays for players joining the Raid, but also other players in the vicinity who may not be participating. This is due to being on the same server.

So what can be done to handle live events that require high volumes of data? Enter Redis.

Niantic initially designed Pokémon GO around a stateful architecture, making scaling and server restarts challenging.

Since Pokémon GO runs on GKE, pods need to be cordoned and existing sessions to expire before servers can be restarted to add more players. Cordon is a Kubernetes operation that marks a node as unschedulable. This is usually done in a Kubernetes cluster when a node needs to be taken offline for maintenance or upgrading.

Not only is this process time consuming, sometimes as long as 30 minutes, but costly as well. The architecture team realized that in order to improve the player experience, the Pokémon GO servers needed to be stateless. This would allow them to scale up and down quickly to match the increased loads.

The team chose Redis on Google Cloud as their solution, providing zero-downtime scaling.

With the new Raid architecture, it’s no longer needed for players to be on the same server. The data is stored in a centralized Redis cache where all servers have access to the same information.

As players join a Raid lobby, data such as player information and time for each group is stored as JSON keys in Redis. Statistics such as player locations, tendencies, and performance are also stored here.

Because Raid events typically only last 15 minutes, there isn’t a need for persistent data storage and all data can be cached in-memory.

Max Latency Decreased from 1 Second to ~250 Milliseconds

Not only has server hot spots reduced significantly, max latency has dropped from over 1 second to 250 milliseconds. This is a 75 percent reduction!

Niantic not only greatly improved the player experience but also did so by maintaining high availability and performance while reducing operational costs.

Real-Time Experience

When Raids were introduced into the game, an influx of players wanted to participate. Since these events happened in real time, it’s unacceptable for notifications to be delayed as players could miss the event. With an average notification latency of 9 seconds, a better real-time notification system was needed.

Notification System: High-Level Architecture

Originally, the Pokémon GO client would poll the player’s message inbox every 15 seconds to retrieve new notifications. This led to redundant polling, and more importantly, the possibility of high latency if the notification was sent right after the client polled.

The team decided to pivot to a new pub-sub service called PushGateway. This service sits between the backend servers and game clients, and pushes messages through the WebSocket to the clients in real time.

In-App Notification Latency With PushGateway

These architecture changes led to huge improvements. Pokémon Go’s in-app notification latency dropped from 9 seconds to 1 second and the load from querying player inboxes for notifications was reduced by 85%.

If you made it this far, thank you for reading! I hope you enjoyed it.

If I made a mistake, please let me know.

Resources

[1] “Optimizing Pokémon GO,” redis.com.
https://redis.com/customers/niantic/.

[2] “How Pokémon GO Scales to Millions of Requests,” cloud.google.com.
https://cloud.google.com/blog/topics/developers-practitioners/how-pokémon-go-scales-millions-requests.

[3] “Improving the Pokémon GO Player Experience with PushGateway,” nianticlabs.com.
https://nianticlabs.com/news/improving-pokemon-go-player-experience?hl=en.

[4] “Using Spanner for Non-Relational Workloads,” cloud.google.com.
https://cloud.google.com/blog/products/databases/using-spanner-for-non-relational-workloads.

[5] “How the Gurus Behind Google Earth Created Pokémon GO,” mashable.com.
https://mashable.com/article/john-hanke-pokemon-go.

[6] “Pokémon GO’s Wild First Year: A Timeline,” theverge.com.
https://www.theverge.com/2017/7/6/15888210/pokemon-go-one-year-anniversary-timeline.

[7] “Pokémon GO Catch $6 Billion in Lifetime Player Spending,” sensortower.com.
https://sensortower.com/blog/pokemon-go-6-billion-revenue.

P.S. If you’re enjoying the content of this newsletter, please share it with your network: https://www.fullstackexpress.io/subscribe

🌙 Hello world ☀️ 

As we approach the end of 2023, a wealth of survey data is being released. Here are my favorites so far:

• Web development trends of 2023

• The most popular Node.js frameworks of 2023

• CSS Wrapped 2023

In this week’s email:

• Architecture: Pinterest’s architecture transition from PinLater to Pacer.

• Node.js: Improving Node.js loader performance.

• JavaScript: The await event horizon in JavaScript.

• CSS: A full-width slider case study.

• Productivity: Four simple programming habits to transform productivity.

ARCHITECTURE
Pinterest’s New Asynchronous Computing Platform

Created with Midjourney

Background

Pinterest, described as a visual search engine, has seen their Global Monthly Active Users (MAUs) increase to 465 million and revenue growth to $708 million this year.

Some of the most notable features on the platform include Pins, Boards, image thumbnail generation, follows, and platform-wide notifications.

How does Pinterest serve so many users at such a large scale?

These core features were handled by an asynchronous job execution system called PinLater, and was one of Pinterest’s most mission critical systems.

PinLater has been open-sourced (and archived) and the codebase can be found here.

Old Architecture: PinLater

Looking at the architecture, we can see PinLater is composed of three core main components:

1. PinLater Thrift service for scheduling and managing job submissions.

2. Backend data store to save the jobs, with payloads and metadata.

3. Workers in worker pools to process the stored jobs and to track the execution status of these jobs.

Pinterest’s PinLater Architecture

As traffic increased and more features were added to the platform, the Pinterest team found flaws with the PinLater system.

The first problem is lock contention in the data store. Since queues have one table in each data store shard, and each dequeue request scans all the shards for available jobs, multiple threads are attempting to retrieve data from the same table.

How does this impact the system on a larger scale?

As traffic increases, the Thrift service naturally scales up to account for the increased load.

Scaling horizontally would ideally distribute the load across multiple instances, but in this case, lock contention actually degrades the performance of PinLater.

Not only does this negate the scalability, but also the throughput of the platform.

Some other issues include mixing jobs from multiple job queues, and sharing the same Thrift services for different team functions.

Job queues may require different instance types, which makes performance tuning extremely challenging.

Also, different functions could have different reliability requirements, where utilizing a shared service is not ideal.

These issues became the driving need for a new architecture, Pacer.

New Architecture: Pacer

With a focus on performance, the Pinterest team introduced new components and mechanisms for storing, accessing, and isolating job data and queues.

These components include:

1. Broker service to pull jobs from the backend data store.

2. Helix and Zookeeper to manage job queues for the broker service.

3. Dedicated worker pools for each queue.

The combination of Helix and Zookeeper ensures that each partition of a job queue is assigned to a dedicated broker host. As a result, there is no competition over the same job data.

Furthermore, since the dequeue broker service is in charge of pulling job queues from the data store, the service is able to prefetch and cache jobs in local memory buffers.

This reduces latency when reading from memory and also decouples the dequeue and enqueue process completely.

Key Improvements

With these new changes to the architecture, the Pinterest team has seen the following improvements:

1. Lock contention is completely resolved in the data store.

2. Significant improvement in hardware utilization.

3. Improved performance and scalability due to independent job execution (own environment) and custom configurations.

In a side-by-side comparison with PinLater, it’s clear that Pacer is built for scalability and resilience.

Enqueue and Dequeue Decoupled From Thrift Service

In PinLater, lots of resources were wasted due to redundant partitions that were created in the data store shards. This led to extra calls where more than 50% of the calls were empty results.

The original system was also limited, as some functionalities simply couldn’t be supported. For example, job execution order can only be guaranteed locally. There was no global order.

In order to manage such a large number of partitions optimally, a cluster management framework was needed to properly partition, replicate, and distribute resources hosted on a cluster of nodes.

Helix: Cluster Management Framework

Helix monitors for various events, such as configuration changes and state changes for broker hosts.

In doing so, the Helix controller is able to calculate and manage resources, while also communicating with the broker cluster to bring it to the ideal state.

Each broker host reports its liveness to Zookeeper and are notified when assigned tasks are changed.

P.S. If you’re enjoying the content of this newsletter, please share it with your network: https://www.fullstackexpress.io/subscribe

NODE.JS
Improving Node.js Loader Performance

Created with DALL·E 3

Yagiz Nizipli

Big picture: The author discusses improvements in Node.js loader performance, focusing on how Node.js differentiates between EcmaScript (ES) and CommonJS modules for loading. The choice depends on factors like file extensions and package.json file configurations.

What you’ll learn: You will learn about the inner workings of the Node.js loader, specifically how it decides whether to use the ES or CommonJS module loader based on file extensions and package.json contents.

Why this matters: The optimizations discussed are particularly relevant for developers looking to enhance the performance of their Node.js applications, offering practical knowledge on improving load times and reducing operational overhead.

JAVASCRIPT
The Await Event Horizon in JavaScript

Created with DALL·E 3

Charles Lowell

Big picture: The author discusses the practical implications and challenges of async functions in JavaScript, where once execution crosses the await event horizon, it becomes impossible to forcibly escape, potentially leading to issues like resource leaks.

What you’ll learn: You'll learn about the inherent risks in async functions, specifically how they can lead to resource leaks if a Promise never settles or takes too long to do so. This is demonstrated through the example of a lock in an async function that may never get released.

Why this matters: This concept is significant for developing robust and efficient JavaScript applications, as it highlights the need for careful structuring and management of asynchronous code to prevent potential issues like resource leaks and unresponsive behaviors.

CSS
A Full-Width Slider Case Study

Created with DALL·E 3

Brecht De Ruyte

Big picture: The case study focuses on implementing a full-width slider using CSS scroll snapping, integrated seamlessly with a global page layout. It addresses the challenge of aligning the slider's content with other page elements while maintaining consistent padding and alignment.

What you’ll learn: You will learn practical techniques for using CSS variables and scroll snapping properties to align a full-width slider with the rest of a webpage's layout. The case study provides step-by-step guidance on achieving a responsive and visually coherent design.

Why this matters: This tutorial demonstrates advanced CSS techniques for creating responsive, aligned, and visually appealing web components. Understanding these methods is crucial for designing modern, user-friendly websites.

PRODUCTIVITY
Four Simple Programming Habits that Transformed my Productivity

Created with Midjourney

Leonardo Creed

Big picture: The article presents four simple yet effective habits that can significantly enhance the productivity of software engineers. These habits focus on optimizing everyday workflow and decision-making processes in software development.

What you’ll learn: You will learn about practical strategies such as strategically leaving work slightly unfinished, utilizing keyboard shortcuts, maintaining an accessible list of commands and links, and the importance of saying "no" to manage workload effectively.

Why this matters: For software engineers, adopting these habits is crucial for improving efficiency and managing complex tasks in a demanding environment. The article provides actionable tips that can be easily integrated into daily routines for better productivity and work-life balance.

DATA STRUCTURES & ALGORITHMS
Balanced Binary Tree

Missed the solution to the latest coding challenge?

This question is asked by Adobe and Microsoft. Learn how to solve this problem here.

JAVASCRIPT ECOSYSTEM
JS Weekly Pulse

📢 OpenAI SDK for Deno: Improvements to module loading via HTTPS, import TypeScript, and availability of GPT-4 Turbo, Assistants API, DALL-E 3 and more.

📢 Vue 2: Reaches End of Life on December 31, 2023, as the Vue team focuses on Vue 3, meaning Vue 2 will no longer receive updates or fixes but will still be available on existing distribution channels.

📜 React Summit: The talk focuses on reducing architectural complexity by structuring code into self-contained modules.

📜 shadcn/ui: A deep dive into the architecture of this unique JavaScript library.

🚀 SvelteKit 2.0: Incremental release supporting Vite 5 with various improvements and the much-requested shallow routing feature, is now available and sets the stage for the anticipated Svelte 5 release in 2024.

🚀 Remix 2.4.0: Introduces the Client Data RFC with new APIs, enhances support for Vite, and includes various improvements like shallow routing, strict route exports, and optimizations for both server and client builds.

🚀 date-fns 3.0.0: Includes 100% TypeScript support, smaller build size, string arguments support, Date class extensions support, ESM support for Node.js, named exports for all functions, and no more IE support.

RECOMMENDATIONS
Around The Web

✅ Interesting: JavaScript continued to dominate web development, including in WebAssembly, while generative AI began integrating into web frameworks.

✅ Learn: The 2023 edition of a Node.js best practices repository includes updated content, new libraries, and a demonstration application, Practica.js, to showcase these practices.

✅ Algorithms: A deep dive into Canva’s Shape Assist, a machine learning tool in their Draw tool, which transforms shaky scribbles into sleek vector graphics in the browser.

✅ LLMs: 2023 marked a surge in public interest and debate around Large Language Models (LLMs), with a focus on open-source models for their benefits in research reproducibility, community involvement, and environmental impact.

✅ Advice: The overuse of abstraction layers in technology and software development.

COMMUNITY SPOTLIGHT
Tools and Packages

📦️ Oxc: A collection of high-performance JavaScript tools written in Rust.

📦️ Byte Base: Database DevOps and CI/CD solution designed to streamline the database development lifecycle and facilitate collaboration between developers and DBAs.

📦️ Page Spy Web: Remote debugging tool for web projects, designed to function like Chrome DevTools.

SOCIAL MEDIA
RIP AI: 1941-2023…it was a good run 🤣 

Twitter tweet

🌙 Hello world ☀️ 

Check out the GitHub Innovation Graph if you haven't already. It offers insights based on eight metrics, showcasing JavaScript's continued dominance as the top programming language, followed by Python.

Notably, TypeScript has impressively climbed from 11th to 5th place, while Ruby has seen a significant drop from 5th to the last position at 12th.

My prediction is that TypeScript will be among the top three within the next two years.

In this week’s email:

• Microservices: How Uber created Cinnamon, an advanced load shedder library.

• JavaScript: A new optimizing compiler that enhances JavaScript execution efficiency in V8.

• Database: Insights into the comprehensive process, challenges, and strategies involved in upgrading GitHub’s large-scale MySQL infrastructure.

• Kubernetes: Learn how taints and tolerations work.

• Best Practices: Learn about the purpose, structure, and benefits of RFCs.

MICROSERVICES
How Uber Created Cinnamon

Created with Midjourney

Jakob Holdgaard Thomsen

Introduction

In the fast-paced world of Uber's microservices, catering to approximately 130 million monthly customers involves managing an immense volume of online and offline traffic.

This complexity inevitably leads to scenarios where services become overloaded due to factors like traffic spikes or database slowdowns.

To ensure seamless user experience despite these challenges, Uber developed Cinnamon, an advanced load shedder library, building upon the legacy of its predecessor, QALM.

The Necessity of Graceful Degradation

The key to managing service overload lies in gracefully degrading system performance by shedding lower-priority requests, such as batch jobs, to preserve the quality of high-priority services like customer requests.

Determining Request Priority Between Different Sources

Uber's vast array of services demands an automatic, configuration-free solution for graceful degradation, given the prohibitive cost of configuring each service individually.

Introducing Cinnamon

Cinnamon is the culmination of Uber’s efforts to enhance service stability.

It dynamically adjusts service capacity using a modified TCP-Vegas algorithm and determines load shedding levels using a PID controller concept, dating back to the 17th century.

Compared to previous methods, Cinnamon effectively manages larger overloads while maintaining reasonable latencies, adding minimal overhead per request.

Crucially, it requires no manual configuration, facilitating rapid adoption across Uber's services.

Understanding QALM

Before delving into Cinnamon, it's essential to grasp the workings of QALM, its forerunner.

QALM employed the Controlled Delay (CoDel) algorithm to manage overload by dropping lower-priority requests.

It monitored request queue times and, upon detecting delays, would reject low-priority requests.

However, its reliance on service configuration and priority classification posed challenges to widespread adoption.

Cinnamon's Architecture

Cinnamon functions as an RPC middleware in Uber’s Go services.

It tags each request with a priority at the edge, which is then propagated through the service stack.

This priority tagging helps services understand the importance of each request, enabling more informed load-shedding decisions.

Priorities in Cinnamon

Cinnamon’s prioritization mechanism consists of two elements: tier levels and cohorts.

The tier level indicates the request's importance, with six tiers used at Uber.

Cohorts further segment requests within a tier, allowing consistent load shedding across services.

This system, inspired by WeChat, involves 768 different priority levels.

Library Architecture

Lifecycle Of A Request Traversing through Cinnamon

Cinnamon operates in stages as a request passes through its middleware:

1. Priority Assignment: Assigns or verifies the priority of a request.

2. Rejection Decision: Determines if a request should be rejected based on system load.

3. Priority Queue Management: Prioritizes requests in the queue.

4. Concurrent Request Management: Limits the number of parallel requests.

5. Timeout Management: Handles request timeouts in the queue.

Additionally, Cinnamon's background threads, the PID controller, and the auto-tuner, adjust rejection thresholds and concurrent request limits to optimize system performance.

Experimental Results

Testing Cinnamon against QALM revealed its superior capacity in managing service load and protecting core flows.

Cinnamon vs QALM Goodput Without Load Shedder

It demonstrated higher goodput (successful request rate) and maintained better latencies under heavy loads compared to QALM.

P99 Latencies

Interaction between PID and Auto-Tuner

Cinnamon’s PID controller and auto-tuner work in tandem to balance rejection rates and service throughput.

Heatmaps For 1,500 and 2,000 RPS Cases

This interaction ensures that the system dynamically adapts to varying loads, maintaining optimal service levels.

Conclusion

Cinnamon represents a significant leap in load shedding technology at Uber.

By leveraging time-tested algorithms and incorporating modern enhancements, it offers a robust, efficient, and configuration-free solution to manage service overloads.

Cinnamon's development, inspired by diverse technologies and ideas, stands as a testament to Uber's commitment to innovative and effective software solutions in a complex, high-demand environment.

P.S. If you’re enjoying the content of this newsletter, please share it with your network: https://www.fullstackexpress.io/subscribe

JAVASCRIPT
V8’s Fastest Optimizing JIT

Created with Midjourney

Toon Verwaest

Big picture: Maglev, a new optimizing compiler introduced in Chrome M117, enhances JavaScript execution efficiency in V8.

What you’ll learn: Discover how Maglev bridges the gap between Sparkplug's instant compilation and TurboFan's peak performance optimization, using sophisticated techniques like SSA-based JIT compilation for improved web performance.

Why this matters: Maglev's introduction is pivotal for web developers and users alike, offering faster, more efficient JavaScript execution in browsers, reducing resource consumption, and improving user experience on web applications.

DATABASE
Upgrading GitHub.com To MySQL 8.0

Created with Midjourney

Jiaqi Liu

Big picture: GitHub's transition to MySQL 8.0 for their production fleet is a detailed journey of upgrading over 1200 MySQL hosts, enhancing data storage, and ensuring high availability for the vast relational data of GitHub.com.

What you’ll learn: Insights into the comprehensive process, challenges, and strategies involved in upgrading a large-scale MySQL infrastructure, emphasizing the importance of maintaining service level objectives and handling complex data workloads.

Why this matters: This upgrade demonstrates the critical role of database management in large tech platforms, offering valuable lessons in scalability, reliability, and the necessity of continuous software maintenance and improvement for operational efficiency and future growth.

KUBERNETES
Importance Of Kubernetes And The Need For Tainting Nodes

Created with Midjourney

Abhisman Sarkar

Big picture: The article details the importance of using taints and tolerations in Kubernetes for efficient cluster management and optimal workload distribution.

What you’ll learn: Understand how taints and tolerations work in Kubernetes, enabling them to control pod placements on nodes, which is essential for specific use cases like multi-tenancy, backup strategies, and environment-specific deployments.

Why this matters: Mastering the concept of tainting and tolerating nodes in Kubernetes is crucial for developers and system administrators to ensure that containerized applications are deployed effectively, meeting the diverse requirements of modern software environments.

BEST PRACTICES
On The Importance of RFCs In Programming

Created with Midjourney

Matija Sosic

Big picture: The article emphasizes the critical role of Request For Comments (RFC) documents in programming, particularly in collaborative environments and complex projects.

What you’ll learn: You'll learn about the purpose, structure, and benefits of RFCs, and how they facilitate informed decision-making and efficient implementation in software development by preemptively addressing potential issues and gathering team insights.

Why this matters: Understanding and utilizing RFCs is essential for developers, as they streamline the coding process, ensure clarity of purpose, and foster a collaborative approach to problem-solving, ultimately leading to more effective and successful project outcomes.

DATA STRUCTURES & ALGORITHMS
Middle Of the Linked List

Missed the solution to the latest coding challenge?

This question is asked by Amazon and Apple. Learn how to solve this problem here.

JAVASCRIPT ECOSYSTEM
JS Weekly Pulse

📢 Supabase: Supabase GraphQL (pg_graphql) version 1.4+ now supports User Defined Functions (UDFs) in Postgres, enhancing GraphQL's flexibility and allowing custom SQL logic within GraphQL queries.

📜 Chain Reaction: Dan Abramov demonstrates the significance of naming conventions in HTML and CSS, and how they simplify the browser’s process of displaying web elements.

📜 Postman Collections: A new format for storing Postman collections using a directory structure, addressing challenges like difficult pull request reviews and limited editor options, and enhancing the ease of maintaining and updating API tests in CI pipelines.

📜 Notifications: How to set up automated notifications for CI builds in Slack and Twilio, using NestJS and CircleCI

🚀 Node 21.4.0: Fixed regression affecting fs.writeFileSync method, introduces dirent.parentPath, and default w flag for writeFileSync with utf8 encoding.

🚀 Astro 4.0: Dev Toolbar, internationalization routing, new view transition APIs, and incremental content caching.

🚀 Electron 28.0.0: Support for ECMAScript modules (ESM), additional properties to the display object, and support for ELECTRON_OZONE_PLATFORM_HINT on Linux.

🚀 React Native 0.73: Debugging improvements, stable symlink support in Metro, Kotlin template on Android, Android 14 support, and more.

🚀 Redux Toolkit 2.0.0: Enhanced store configuration and middleware options, improved package compatibility, and TypeScript improvements.

🚀 Million 3.0: 34% faster initial compile, ~2% for code changes on medum-sized Next.js projects, and 100% improvement in hydration times.

🚀 GitHub Enterprise Server 3.11: Generally available, enhanced code security with tools for application security testing, new Activity view for repository history, advanced secret leak prevention, and GitHub CLI extension.

RECOMMENDATIONS
Around the Web

✅ Interesting: Visualization and walkthrough of ChatGPT’s LLM algorithm.

✅ Software Engineering: A review of the original 1970 paper on the waterfall model in software engineering, discussing its key components, the shift in management's role, optimization in development, and iterative approaches.

✅ Learn: Programming challenges of text editors, focusing on reworking fundamental data structures, particularly for handling text.

✅ Career Growth: Ryan Peterman reflects on his transition from mid-level to senior engineer at Meta.

✅ Internet Trends: Cloudflare shares their 2023 Radar Year in Review, a comprehensive analysis of annual global and regional Internet trends based on various metrics.

COMMUNITY SPOTLIGHT
Tools and Packages

📦️ Terminal Trove: Curated collection of terminal tools, including CLIs, TUIs, and developer tools, for all platforms and mediums.

📦️ Ncube: Visualize rotating hypercubes in any dimension, using real-time rotation and perspective projections down to three dimensions.

📦️ Build Your Own X: Enhance your programming skills by building your own versions of popular technologies from the ground up.

📦️ SuperDuperDB: A framework for directly integrating AI with databases, featuring streaming inference, scalable training, and vector search.

📦️ Typia: Fast and easy runtime validations and serializations through transformation.

SOCIAL
🤣🤣🤣🤣🤣

Twitter tweet

🌙 Hello world ☀️ 

The 2023 State of JS survey is still open for another week (ends December 12) and helps the JavaScript community measure popularity of current features and new trends. Take the survey here.

In this week’s email:

• Machine Learning: Deep dive into Griffin 2.0, the core of Instacart’s ML platform.

• JavaScript: Learn hands-on how modern JavaScript frameworks work.

• React: A router-driven approach to building React applications.

• HTML/CSS: Five new modern web development features.

• Internet Trends: How Black Friday and Cyber Monday affect internet traffic.

MACHINE LEARNING
Deep Dive into Instacart’s Griffin 2.0

Created with Midjourney

Han Li

Background

Machine learning is essential to Instacart’s platform, the core of nearly every product and operational innovation at the company. The technology is used to:

1. Enhance customer experience by matching customer preferences with suitable choices from a catalog containing over 1 billion products.

2. Optimize the efficiency of over 600,000 shoppers, enabling swift delivery to millions of customers across the US and Canada.

3. Integrate artificial intelligence into the Instacart platform, enhancing support for over 800 retailers across 70,000 stores in more than 5,500 cities throughout the US and Canada.

4. Facilitates connections between over 5,000 brand partners and potential customers.

In order to create a seamless shopping experience that also handles the day-to-day scalability challenges, the machine learning infrastructure team created Griffin, Instacart’s MLOps Platform.

Shortcomings of Griffin 1.0

Griffin 1.0 at Instacart, while comprehensive for end-to-end ML, revealed several limitations that led to the development of Griffin 2.0.

These included a steep learning curve with complex in-house command-line tools, a complicated deployment process involving AWS ECS, and the need for specialized system tuning.

It also lacked standardization, relying heavily on GitHub PRs for multiple tasks, and had unoptimized scalability, with limited horizontal scalability and an insufficient model registry for high query volumes.

Furthermore, Griffin 1.0 offered a fragmented user experience due to multiple third-party integrations and had inadequate metadata management for training and deployment.

Development of Griffin 2.0

In developing Griffin 2.0's Machine Learning Training Platform (MLTP), the goal was to create a unified, centralized platform for machine learning engineers (MLEs) to easily create, track, and manage training workloads.

The platform was designed to support distributed machine learning, including distributed training, batch inference, and fine-tuning of Large Language Models (LLMs), while addressing the limitations of Griffin 1.0.

Key strategic design decisions for Griffin 2.0 included:

1. A Singular Interface: Unlike Griffin 1.0, which required navigating multiple systems, Griffin 2.0 integrates all tools into one unified web interface, simplifying user experience and streamlining model training development.

2. Centralized Unity: Griffin 2.0 consolidates various training backends into a single Kubernetes platform, reducing maintenance overhead compared to the fragmented approach in Griffin 1.0.

3. Standard ML Runtime: Addressing the lack of standardized modeling frameworks in Griffin 1.0, Griffin 2.0 introduces standard runtimes across different ML frameworks, ensuring consistency in building blocks and package versions.

4. Horizontal Scalability: Griffin 2.0 uses Ray to enable horizontal scalability for distributed workloads, overcoming the vertical scaling limitations of Griffin 1.0.

5. Metadata Store for All: Griffin 2.0 implements a centralized metadata store, enhancing model lineage management and lifecycle oversight, an area where Griffin 1.0 was deficient.

System Architecture

ML Training Platform Architecture

The Machine Learning Training Platform (MLTP) at Instacart is built with several key components to provide a centralized service with distributed computation capabilities:

1. Metadata Store: Includes Model Store for untrained models, Offline Feature Store for training metadata, Workflow Run for managing training jobs, and Model Registry for post-training model information.

2. API Endpoints: Provides RESTful APIs for interacting with the Metadata Store, managing model architectures, registries, features, datasets, and training jobs.

3. Workflow Orchestrator: Comprises the MLTP API service for customizing training jobs and the ISC worker integrated with Kubernetes and Ray for orchestrating and managing training workloads.

Workflow Orchestrator

MLTP Training Process

The process of creating a training job in the Machine Learning Training Platform (MLTP) involves several steps:

1. Customization: Users start by customizing their inputs, including organizing features and experimenting with model designs. They select a model from the Model Store, choose data from the Training Dataset, and configure training settings.

2. Initiation Options: To start a training workload, users can either use the Griffin UI or Python SDKs to send requests to the workflow services.

3. Resource Creation: The workflow services then generate Kubernetes resources based on the user's inputs, which can range from a simple single-container job to a complex multi-node Ray cluster.

4. Post-Training: Upon completion, training results like MLFlow metrics and Datadog logs are displayed, and the model weights and other relevant items are stored in the Model Registry for future use in evaluation and inference.

Screenshot of Griffin Model

The design of MLTP emphasizes a streamlined and standardized approach:

• A centralized service and APIs provide a consistent interface for managing the entire model development lifecycle.

• Users can prototype using Ray clusters from their laptops or Jupyter servers, leveraging distributed computation.

• For production-ready models, the Griffin UI is used to create a production workflow definition.

• In cases with an existing Airflow production pipeline, Griffin's task and sensor operators are used for integrating with workflow APIs.

• Throughout the lifecycle, users interact with MLTP through the same API interface.

Lessons Learned

During the development of Instacart’s next-generation ML training infrastructure, the team learned about:

1. Unified Solutions: By unifying ML training solutions, the team achieved a more consistent training job process and user experience, especially after transitioning to Kubernetes as the sole orchestration platform. This unification not only simplified management but also brought benefits like distributed computation and better metadata management.

2. Balancing Flexibility and Standardization: The platform was designed to be highly flexible to support a broad spectrum of ML applications, while also incorporating standardization to cater to the majority of use cases and enhance development speed.

3. Considering the Bigger Picture: The redesign of the ML Training Platform (MLTP) went beyond just training; it included model serving and feature engineering. This holistic approach involved collaboration with various Griffin 2.0 stakeholders, leading to co-designed data models for training jobs that integrate seamlessly into the end-to-end ML process, simplifying deployment and improving metadata management.

P.S. If you’re enjoying the content of this newsletter, please share it with your network: https://www.fullstackexpress.io/subscribe

JAVASCRIPT
Let’s Learn How Modern JavaScript Frameworks Work By Building One

Created with Midjourney

Nolan Lawson

Big picture: This article provides an in-depth exploration of modern JavaScript frameworks by guiding the reader through the process of building a simple framework.

What you’ll learn: It emphasizes key concepts like reactivity, DOM rendering, and using modern JavaScript APIs, illustrating the underlying principles and techniques that power frameworks like React, Vue, and Svelte.

Why this matters: Understanding these concepts is crucial for JavaScript developers, as it deepens their knowledge of client-side technologies, improves their ability to choose or build suitable frameworks for their projects, and enhances their skills in solving complex front-end challenges.

REACT
TanStack Router - Modern React for the Rest of Us

Created with Midjourney

Swizec Teller

Big picture: The TanStack Router is a modern React routing solution that integrates advanced features like type-safe routing, coordinated data loading, and suspense-first design, representing a significant improvement in how web applications are structured and managed.

What you’ll learn: React developers gain perspective on a router-driven approach that leads to more efficient UI state management, enabling smoother and more responsive user experiences with less coding effort.

Why this matters: It signifies a shift towards more intuitive and powerful web development practices, aligning with higher demands for dynamic, user-centric web applications.

HTML, CSS
Unravelling Five Exciting Modern Web Development Features

Created with Midjourney

Siddhant Varma

Big picture: This article introduces five modern web development features that are transforming the field: Native HTML Dialog, Native HTML Popovers, Container Queries for Responsiveness, CSS Color Mix, and CSS Nesting.

What you’ll learn: It teaches how these features simplify coding, enhance design capabilities, and improve the user experience, without the need for extensive custom coding or external libraries.

Why this matters: The significance lies in their potential to revolutionize web projects, making applications more efficient, visually appealing, and responsive, thus empowering developers to stay ahead in the rapidly evolving field of web development.

INTERNET TRENDS
Cyber Week: Analyzing Internet Traffic and E-Commerce Trends

Created with Midjourney

João Tomé

Big picture: The article presents a comprehensive analysis of internet traffic and e-commerce trends during key shopping events like Black Friday and Cyber Monday, offering insights into global and regional online behavior patterns.

What you’ll learn: Developers will learn about the significant shifts in traffic on major e-commerce dates, the varying device usage trends (mobile vs. desktop), and the cybersecurity landscape, particularly regarding DDoS attacks around these peak times.

Why this matters: This information is crucial for developers in optimizing website performance, enhancing user experience, and bolstering cybersecurity measures during high-traffic periods, which are pivotal for the success and resilience of e-commerce platforms.

DATA STRUCTURES & ALGORITHMS
Maximum Subarray

Missed the solution to the latest coding challenge?

This question is asked by Amazon and Microsoft. Learn the algorithm used to solve this problem here.

JAVASCRIPT ECOSYSTEM
JS Weekly Pulse

• 📢 State of JS Survey: The JavaScript user survey aims to track JavaScript feature and library trends, and will openly publish results post-December 12 for community and company use.

• 📢 2023 State of the API Report: APIs significantly generate revenue, API pricing is a growing concern, positive outlook on API investments, widespread use of AI in API development, and more.

• 📢 Announcing Deno Cron: Deno Cron simplifies web development by offering a streamlined way to create and manage scheduled jobs within Deno's serverless platform, Deno Deploy, ensuring efficient and non-overlapping task execution.

• 📜 Using Infer to Unpack Nested Types: How to use TypeScript's infer keyword for efficient type extraction in nested structures, simplifying the development of type-safe REST APIs.

• 📜 Debugging Cloudflare Workers: Cloudflare Workers now offers enhanced debugging with breakpoints, integrating console logs, Chrome DevTools, and IDE support for easier development and troubleshooting.

• 📜 How Marketing Changed OOP: How JavaScript's prototypal inheritance, influenced by the Self language and marketed as "Java for the web," has shaped current JavaScript development practices, particularly in the use and understanding of prototypes.

• 🚀 Fresh 1.6: Tailwind CSS plugin, partials with forms and error pages, improved islands bundling strategy, support for pre-generated assets, and more.

• 🚀 Bun 1.0.15: 2x faster start for tsc, 40% speed increase for Prettier, stable WebSocket client, syntax-highlighted errors, and more.

• 🚀 XState 5: Actor as the main unit of abstraction, inspect API, deep persistence, stronger type inference, dynamic parameters, and enqueue actions.

• 🚀 ESBuild 0.19.7: Standardization of import attributes and deprecation of import assertions.

RECOMMENDATIONS
To-Do List

✅ Interesting: Global business and finance systems heavily rely on the 60-year-old COBOL programming language, now understood by few programmers, with IBM considering Watson as a potential aid.

✅ Big Tech: Google's code review tool Critique, featuring AI enhancements and efficient guidelines, secures 97% developer satisfaction by simplifying and improving the code review process.

✅ Learn: Matteo dives into Node.js's event loop, exploring its role in efficient asynchronous operation handling and offering best practices for optimizing its performance.

✅ Regex: The use of Regular Expressions in code searches, highlighting their usefulness in simplifying pattern matching and data cleaning in everyday engineering tasks.

✅ Side Projects: Developing a Dropbox-like application, creating MySQL solutions, writing about Regular Expressions, and designing themes for apps like Obsidian are examples of side projects that have actually helped individuals secure jobs.

COMMUNITY SPOTLIGHT
Tools and Packages

📦️ WatermelonDB: Reactive & asynchronous database for React and React Native apps. Built for performance and scale.

📦️ Package Majors: Enhanced tool for analyzing and comparing major version downloads of a package within the last week. Here’s an example with Node.

📦️ Win11React: Recreation of the Windows 11 desktop experience in React. Pretty awesome…for Windows.

📦️ Zx: Write more complex scripts with JavaScript.

MEME
🤣 🤣🤣🤣🤣

🌙 Hello world ☀️ 

Microsoft Ignite 2023 kicks off today at 8 AM PT, featuring a lineup of industry experts on the latest innovations in AI, data, security, and development tools. It’s free.

In this week’s email:

• Security: How Apple is advancing iMessage security.

• Big Data: Netflix’s transition to streaming SQL in data mesh.

• WebAssembly: A new way to bring garbage collected programming languages efficiently to WebAssembly.

• CSS: Speeding up the JavaScript ecosystem using Tailwind CSS.

• DevOps: Highlights from the State of DevOps 2023 report.

• Accessibility: Why accessibility is a requirement, not a feature.

SECURITY
How Apple is Advancing iMessage Security

Created with Midjourney

Apple Security Engineering and Architecture (SEAR)

Background
Apple's iMessage, known for its secure end-to-end encryption since 2011, has constantly been updated to combat evolving security threats. With hardware-aided key management, advanced sandboxing through BlastDoor, and Lockdown Mode, iMessage has established robust security features.

iMessage Contact Key Verification

Challenge
The central key directory service (Identity Directory Service or IDS), which maps user identifiers to public keys, posed a single point of failure. A compromised service could return keys controlled by adversaries, enabling them to intercept messages.

Traditional Solutions
Peer-to-peer verification methods (e.g., QR codes or safety numbers) were used in other messaging systems to confirm key authenticity, but these methods were cumbersome for users, especially when devices changed or new ones were added.

Apple's Approach: iMessage Contact Key Verification With Key Transparency (KT)
Apple introduced Contact Key Verification to tackle the risks associated with the key directory service. Using Key Transparency, which leverages verifiable logs and maps, Apple ensures that users' devices can cryptographically verify the integrity and authenticity of encryption keys.

Key Aspects:

1. Key Transparency and Verification:

   • Devices automatically verify keys using the KT system, which provides cryptographic proofs and can detect inconsistencies.

   • The KT system employs a log-backed map data structure for scalability and user privacy, using techniques like SHA-256 hash and Verifiable Random Functions.

2. Account-Level Security:

   • An ECDSA signing key is generated on the user's device and stored securely in iCloud Keychain.

   • This account key is used to sign iMessage public keys, which are then verified automatically by users’ devices.

3. Automatic and Manual Verification:

   • Devices verify keys automatically through the KT system, with alerts for any inconsistencies.

   • Users can manually compare short codes using the Vaudenay SAS protocol to confirm that they see the same account key for their contacts.

4. Scalability and Notifications:

   • The system is designed to scale to billions of users and to provide notifications only when there's an unexpected security condition.

5. Third-Party Auditing and Gossip Protocols:

   • While initially using an internal auditing service, plans for external auditing are in the pipeline for 2024.

   • The Messages app uses gossip protocols to ensure consistency of the KT logs among iMessage clients.

6. Future Device Sign-ins:

   • The verification extends to new devices automatically, linking the process to the user’s contact card for continuous assurance.

7. Public Persona Verification:

   • Public figures can use public verification codes to verify their identity.

Conclusion
iMessage Contact Key Verification represents a significant step in messaging security. It ensures that even if the key directory service is compromised, users can verify who they are messaging with and trust the integrity of their communication.

P.S. If you’re enjoying the content of this newsletter, please share it with your network: https://www.fullstackexpress.io/subscribe

BIG DATA
Netflix’s Transition to Streaming SQL in Data Mesh

Created with Midjourney

Guil Pires, Mark Cho, MingLiang Liu, Sujay Jain

Introduction
Netflix, known for its massive content delivery and personalized experience, relies heavily on data. The Data Platform team is responsible for creating scalable data processing infrastructure.

The Challenge
As data demands grew, Netflix introduced "Data Mesh" - a data movement and processing platform utilizing Kafka and Flink.

Data Mesh Pipeline Example

Users could build data pipelines with reusable "Processors" for tasks like filtering or renaming fields. However, this posed two major challenges:

1. Limited Expression: Pre-built Processors didn't always fit complex needs, leading to custom, difficult-to-learn solutions.

2. High Overhead: Pipelines with multiple Processors incurred resource and runtime costs.

Solution: Data Mesh SQL Processor
To increase flexibility and reduce complexity, Netflix developed the Data Mesh SQL Processor.

SQL Process Workflow

This innovation leveraged Flink SQL to allow complex data transformations using SQL queries instead of multiple Processors, thus reducing latency and resource usage.

Key Features:

• Managed Flink Jobs: Simplified the transformation process with parameterized jobs.

• Streaming SQL Support: Enabled direct use of Flink SQL within Data Mesh without architectural changes.

• Interactive Query Mode: Provided real-time feedback and query validation for faster iteration.

• Schema Inference: Automated output schema creation from SQL queries, streamlining schema management.

User Workflow
Software engineers using the SQL Processor follow these steps:

1. Sample Data: Use Interactive Query Mode to sample upstream data.

2. Iterate on Queries: Modify and validate SQL queries in real-time.

3. Deploy: Save the query into the Data Mesh Pipeline for execution.

Results
Adoption of the SQL Processor has been met with excitement. It allowed a more intuitive expression of business logic and unlocked new use cases without the need for writing custom Processors.

Lessons Learned
Netflix realized the benefits of Flink SQL early on could have saved significant engineering resources. Additionally, setting up guardrails for Flink SQL was crucial for managing expectations and operational workload.

Future Plans
The Data Platform team continues to expand SQL Processor features, such as supporting stream enrichment and advanced joins, to further harness the power of stream processing.

Conclusion
Netflix's transition to a Streaming SQL approach within Data Mesh signifies a strategic move towards more efficient, scalable, and manageable data processing methodologies that cater to the complex needs of a leading streaming service.

DATA STRUCTURES & ALGORITHMS
Contains Duplicate

Missed the solutions to this week’s coding workout?

Learn the solution to one of the most common interview questions here.

WEBASSEMBLY
A New Way To Bring Garbage Collected Programming Languages Efficiently to WebAssembly

Created with Midjourney

Alon Zakai

WebAssembly's latest optimization includes built-in support for garbage collection, allowing languages that traditionally rely on it, such as Java and Python, to run more efficiently on the Web.

By leveraging WebAssembly's garbage collection, these languages can produce smaller and faster binaries.

This advancement holds promise for more performant web applications utilizing these languages in the future.

CSS
Speeding Up The JavaScript Ecosystem - Tailwind CSS

Created with Midjourney

Marvin Hagemeister

Marvin’s article discusses how Tailwind CSS is used to style web projects and explores ways to optimize its performance.

He analyzes the Tailwind architecture on its own website but encounters issues with noisy traces, so he runs the Tailwind CLI to identify key areas for improvement.

Marvin suggests that significant speed improvements in Tailwind CSS can be achieved not by following best practices but by understanding and optimizing the code based on its purpose and the problems it solves.

DEVOPS
State of DevOps 2023 Highlights

Created with Midjourney

Tomas Fernandez

Tomas highlights key findings from the State of DevOps 2023 report including the influence of AI on productivity, factors affecting job satisfaction and burnout, and the importance of cloud choices.

It emphasizes user-focused development, key technical capabilities like CI/CD, the role of flexible cloud infrastructure, and the impact of quality documentation on job satisfaction and performance.

The report also advocates for a generative culture to boost innovation and performance.

ACCESSIBILITY
Accessibility is a Requirement, Not a Feature

Created with Midjourney

Nicole Tietz

In her article, Nicole argues that accessibility in software is a crucial requirement, not an optional feature.

She likens it to ongoing needs like security, needing constant attention in development.

Her personal experience with temporary disability underscores the importance of accessible design for all users, not just those with permanent impairments.

JAVASCRIPT ECOSYSTEM
JS Weekly Pulse

• 📖 The science behind the JavaScript engine.

• 📖 How the Chrome Profiler helps you understand the JavaScript event loop.

• 📖 React Server Components without a framework.

• 🚀 TypeScript 5.3 RC - Import attributes, stable support for resolution-mode in import types, improved type narrowing, optimizations in JSDoc parsing and type intersections.

• 🚀 Angular v17 - Deferrable views, 90% faster runtime, 87% faster builds for hybrid rendering, 67% faster client-side rendering, and new interactive learning journey.

• 🚀 Deno 1.38 - Deno doc improvements, hot module replacement, Node.js compatibility improvements, faster JSX transforms, and WebSockets improvements.

• 🚀 Bun 1.0.11 - Addition of Bun.semver, and bug fixes for bun install, astro, and @google-cloud/storage.

• 🚀 TSX 4.0.0 - Variadic tuple types, labeled tuple elements, class property inference from constructors, custom JSX factories, and short-circuiting assignment operators.

RECOMMENDATIONS
To-Do List

✅ Interesting: How AT&T lost over $60 million due to one line of code.

✅ TypeScript: A complex technique in TypeScript to specialize generics.

✅ Learn: Have the technical skills but communication needs improvement? Check out this guide to public speaking for software engineers.

✅ Watch: An in-depth video on the growth of JavaScript runtimes over the years from James Snell himself.

✅ Listen: The inside story of Spotify’s engineering growth (10 to 1000s) on the CoRecursive podcast.

COMMUNITY SPOTLIGHT
Hot Picks in the Dev Community

Svelte Flow - A customizable Svelte component for building node-based editors and interactive diagrams.

Lenis - A better smooth scroll library created for performance and accessibility. The demo.

Fkill CLI - A CLI for killing processes cross-platform.

Imba - A full-stack programming language for the web that compiles to performant JavaScript.

Pannellum - A lightweight panorama viewer for the web.

MEME
Pair Programming Be Like…

🌙 Hello world ☀️ 

I'm traveling this week, so this issue will be shorter. We’ll be back in full swing next week 😎 

In this week’s email:

• Data Infrastructure: Learn how Meta’s Systematic Code and Asset Removal Framework (SCARF) automatically identifies and removes dead code.

• Next.js/Remix: Kent’s thoughts on why he prefers Remix for its simplicity and flexibility, cautioning against Next.js’s complexity and urging clearer distinctions in the React community.

• Node.js: Matteo’s guide on boosting Node.js API performance with Platformatic DB, emphasizing automated CRUD creation, multi-database support, and efficient caching, tested using Autocannon.

• Career Development: Why aspiring senior software engineers should focus on data modeling, asynchronous systems, and cloud technologies.

DATA INFRASTRUCTURE
How Meta Automates Dead Code Cleanup

Created with Midjourney

Will Shackleton, Andy Pincombe, Katriel Cohn-Gordon

Meta's Systematic Code and Asset Removal Framework (SCARF) plays a critical role in automating dead code cleanup by utilizing a mix of static and dynamic program analysis.

The framework is designed to identify redundant code from both a business and programming language standpoint, subsequently generating change requests to delete the identified dead code, thus reducing the burden on developers.

Dead Code Removal Process
SCARF encompasses a subsystem specifically tailored for automatic dead code identification and removal, employing static, runtime, and application analysis to achieve this.

SCARF’s Code Analysis Subsystem

The process involves gathering comprehensive data from various sources to construct an augmented dependency graph, which serves as the foundation for identifying unreachable code segments that can be safely removed.

The framework is versatile, supporting multiple programming languages and operating at a symbol level for more precise analysis and cleanup. This approach not only enhances the quality of the systems but also facilitates the removal of unused data by resolving dead code references to data assets.

SCARF’s Automated Code Change Requests

Results

• SCARF has significantly improved system quality and efficiency, automating the removal of over 100 million lines of code through more than 370,000 change requests.

• The framework supports a wide range of programming languages, ensuring comprehensive dead code removal across different parts of the system.

• Despite its effectiveness, SCARF does not provide a complete solution for shutting down deprecated products, highlighting an ongoing need for improvements in integration across various languages, systems, and frameworks.

• The team emphasizes a cautious approach, implementing additional safety mechanisms like textual references search through the BigGrep system to prevent accidental deletions.

• Engineers play a vital role in the deprecation process, often opting to manually delete code for faster turnaround times, showcasing a balance between automated and manual efforts in dead code removal.

P.S. If you’re enjoying the content of this newsletter, please share it with your network: https://www.fullstackexpress.io/subscribe

NEXT.JS, REMIX
Why I Won’t Use Next.js

Created with Midjourney

Kent C. Dodds

Kent C. Dodds advocates for using Remix over Next.js due to its direct use of web platform APIs, enhancing transferable skills, its flexibility to deploy anywhere, and its commitment to stability and simplicity. He expresses concerns about Next.js’s increasing complexity, potential “magic” that could surprise developers, and the marketing of experimental features as stable. He calls for a clearer distinction between Next.js and React, and a more collaborative approach in the React community.

DATA STRUCTURES & ALGORITHMS
Binary Search

Missed the solutions to this week’s coding workout?

Learn the algorithm used to solve this problem here.

NODE.JS
How To Make Your Node.js API 5x Faster with Platformatic

Created with Midjourney

Matteo Collina

In this guide, backend developers are introduced to Platformatic DB, a tool that enhances Node.js API performance by automating CRUD API creation and supporting multiple databases. The importance of caching for API optimization is emphasized, with async-cache-dedupe highlighted as an effective solution for data storage and rapid retrieval, reducing latency and saving bandwidth. The article also introduces Autocannon, an HTTP/1.1 benchmarking tool, for testing the efficiency of the optimizations made.

CAREER DEVELOPMENT
6 Skills Required To Be A Senior Software Engineer

Created with Midjourney

Caleb Mellas

Caleb shares his knowledge on how to work towards senior software engineering positions. To thrive as a senior software engineer, mastering skills in data modeling, asynchronous systems, autoscaling, cloud technologies, caching, and ensuring concurrency and idempotency in distributed systems is crucial, as these areas are often scrutinized in technical interviews and essential for handling complex technical challenges.

JAVASCRIPT ECOSYSTEM
JS Weekly Pulse

• 🚀 Next.js 14 - Speed improvements with Turbopack, Server Actions (stable), Partial Prerendering (preview), and Next.js Learn.

• 🚀 Astro 3.4 - Page partials, image optimization performance, and dev overlay (experimental).

RECOMMENDATIONS
To-Do List

✅ Advice: Positioning yourself near opportunities.

✅ Join: Game Off 2023 will take place from November 1 to December 1. Join GitHub’s annual game jam challenge!

✅ Learn: Pushing Discord’s limits with more than a million online users in a single server.

✅ Watch: Stephen Cooper of Ag-Grid demonstrates different techniques to optimize performance in React.

✅ Listen: On the Software Engineering Unlocked podcast, Abi Noda explains how to measure developer experience and why a good developer experience matters.

COMMUNITY SPOTLIGHT
Hot Picks in the Dev Community

Release Please Action - Google’s Github action for automating releases based on conventional commits.

Fnm - Fast and simple Node.js version manager, built in Rust.

Ungit - The easiest way to use git, anywhere, on any platform.

JSX Email - React components and helpers for building responsive email templates.

Metallic - A powerful web proxy built for speed and customization.

🌙 Hello world ☀️ 

Martin Goetz, regarded by many as the "father of third-party software," has died at the age of 93. Goetz was notable for receiving the first software patent, awarded in 1968, for his sorting system program.

In this week’s email:

• Architecture: How Uber optimizes advertising through ads simulation.

• Performance: How DoorDash standardized and improved microservices caching.

• Solid.js: The importance of locality of thinking and a pass-by-value approach.

• Node.js: Why it’s time to transition from Buffer to Uint8Array.

• Software Development: Critique of the software industry’s neglect of efficiency and optimization, urging a shift in development practices.

ARCHITECTURE
How Uber Optimizes Advertising Through Ads Simulation

Created with Midjourney

Yue Yin

Introduction to Ads Simulation
Uber Eats employs advertising tools to help merchants boost their sales, striving to display the most relevant ads to customers.

Traditional methods to measure changes in ad placements, ranking algorithms, and bidding strategies include costly and time-consuming experiments, or quicker but biased offline analyses.

To overcome these challenges, Uber developed the Ads Simulator, a tool that replicates various ad system scenarios, providing reliable results swiftly and affordably.

Summary of Ads Auction and Workflow

Uber Eats: Ads Auction and Bidding Flow

1. The system initiates and gathers user context upon receiving an ad request.

2. Potential ad campaigns matching the user's context are identified and selected.

3. Campaigns bid in real-time, with performance optimized for pacing and budget.

4. Winners are determined, and ad placements are assigned by the auction module.

5. Interactions and effectiveness of the ads are tracked and analyzed.

6. Campaigns continuously adapt their internal states based on market conditions.

Architecture and Configuration

Ads Simulator Architecture

The Ads Simulator replicates the ads auction and bidding flow, providing outcomes from potential ad strategies that reflect the real-world dynamics of the ads marketplace. It captures the complexities of budget pacing and utilizes a full spectrum of advertisers, offering a more holistic and accurate simulation.

Core Ads Flow Simulator

• Central to the architecture, managing auctions and bids for all market advertisers.

• Generates comprehensive simulation data logs, with inputs from Simulation Configuration and Auction Context.

Configuration

• Enables tailoring of the simulator for a variety of ad strategies.

• Simple adjustments of parameters allow for straightforward customization.

Auction Context

• Consists of auctions, bidders, and user details for simulating results and adjusting future auctions.

• Utilizes real auction logs to overcome selection bias and synthetic data for testing different market conditions.

Core Ads Flow

• Accurately replicates entire marketplace interactions.

• Includes Ad Campaigns with a three-agent system, Auctioneer for replicating actual auction mechanisms, and User Response Simulator for simulating customer interactions.

Logging and Analysis

• Captures extensive simulation data, offering built-in functions for advanced analysis and visualization.

• Highly extensible, allowing for customized integrations and valuable insights into ad strategies and marketplace dynamics.

Practical Use Cases at Uber

The Ads Simulator at Uber provides critical insights and accurately mirrors the ads marketplace dynamics, proving its reliability through real-world experimental results.

Engineers and scientists at Uber can confidently use the simulator to make well-informed decisions, enhance campaign performance, and uncover new innovative opportunities.

Use Case 1: Reserve Price Design in Auction Mechanism

• Demonstrates the simulator's ability to efficiently find the optimal reserve price design, saving time and resources compared to traditional trial-and-error methods.

• Experimentation results closely align with simulated results, showcasing the simulator's accuracy and its capability to significantly impact ads revenue and advertiser outcomes without risking real-world assets.

Use Case 2: Budget Allocating for Ad Campaigns

• Assists the platform in identifying optimal geographical regions for campaign budget allocation, enhancing advertiser impact and ensuring personalized customer experiences.

• Overcomes limitations of traditional experimentation and observational data, providing reliable data for evaluating budget scaling strategies and validating its results through online experimentation.

The presented use cases underscore the simulator's precision, efficiency, and its role as a dependable tool for driving business success at Uber.

P.S. If you’re enjoying the content of this newsletter, please share it with your network: https://www.fullstackexpress.io/subscribe

PERFORMANCE
How DoorDash Standardized and Improved Microservices Caching

Created with Midjourney

Lev Neiman, Jason Fan

DoorDash's growing microservices architecture has led to a surge in interservice traffic, resulting in the need for a streamlined caching solution.

This issue was addressed by introducing a Kotlin library, providing a uniform, efficient, and secure method for backend developers to implement new caches, improving performance and reducing costs.

Multi-Layer Cache Request Flow

Problems and Challenges
Given DoorDash's prioritization of implementing business logic over performance optimization, there was a significant challenge in enhancing system efficiency without modifying the existing codebase.

Several issues came up as a result, including:

• Cache Staleness: Ensuring cache remains updated with the original data source is complex, and resolving issues from outdated cache entries is time-consuming.

• Heavy Dependency on Redis: Services often face a high rate of failure during Redis downtimes or other issues.

• No Runtime Control: Introducing a new cache is risky without real-time adjustments; changes require new deployments or rollbacks, consuming time and resources.

• Inconsistent Key Schema: The lack of standardized cache keys complicates debugging and tracing between Redis cache and Kotlin code.

• Inadequate Metrics and Observability: The absence of uniform metrics across teams leads to a lack of essential data for performance analysis.

• Difficulty in Implementing Multilayered Caching: The previous setup did not support the use of multiple caching layers for the same method easily.

Streamlined Caching and and Performance Enhancement

A simplified, streamlined approach was adopted to address these issues and ensure seamless system integration.

• Pilot Program on DashPass: Addressed caching issues and scaling challenges by developing a simple caching library, enhancing performance with minimal disruption.

• Standardization of Caching Clients: Introduced a simplified interface for application developers, standardizing function signatures and APIs across various caching clients for easier integration and maintenance.

• Dependency Injection and Polymorphism: Employed to maintain consistent cache calls and ensure uniform caching processes, enhancing backend efficiency.

• Multi-layered Caching System: Implemented a layered caching approach, comprising request local cache, local cache, and Redis cache, to optimize performance and reduce dependency on slower network layers.

• Optimized Retrieval and Storage: Ensured quicker access to cached data in subsequent requests by storing keys in all earlier layers once found in a later layer, enhancing system efficiency.

• Runtime Feature Flag Control: Enabled individual cache management, allowing for quick adjustments, rollouts, and tuning without new deployments, and providing flexibility in caching strategy.

• Performance Metrics and Cache Shadowing: Collected key performance metrics and implemented a cache shadowing mechanism for validating cache entries, ensuring optimal performance and data accuracy.

• Empirical Validation of Cache Invalidation Strategy: Utilized extensive metrics and observability into cache staleness for empirical validation, ensuring the system’s optimal functionality.
  

DATA STRUCTURES & ALGORITHMS
Valid Anagram

Missed the solutions to this week’s coding workout?

Learn how to use a hash table to solve this problem here.

SOLID.JS
Thinking Locally with Signals

Created with Midjourney

Ryan Carniato

SolidJS, influenced by React, emphasizes design principles like unidirectional flow and explicit mutation for building user interfaces, rather than technologies like Virtual DOM. Ryan highlights the importance of "locality of thinking," enabling developers to understand and work with components in isolation, boosting productivity and maintainability. To achieve this, the article recommends a pass-by-value approach, treating all props as reactive, and avoiding implicit mutations, guiding developers to create more predictable and manageable React and SolidJS applications.

NODE.JS
Goodbye, Node.js Buffer

Created with Midjourney

Sindre Sorhus

Buffer has long been the go-to solution for handling binary data in Node.js, but it’s becoming outdated due to the cross-platform, native JavaScript type Uint8Array. The extensive methods that Buffer adds to Uint8Array lead to compatibility issues with other JavaScript environments and potential security risks. To modernize practices and avoid unpredictable behavior, it’s recommended to transition from Buffer to Uint8Array.

SOFTWARE DEVELOPMENT
Software Disenchantment

Created with Midjourney

Nikita Prokopov

The article expresses a deep-seated frustration with the current state of software development, highlighting the industry's disregard for efficiency and optimization in favor of faster hardware and bloated software. It draws parallels between software and other industries like automotive and construction, emphasizing how software lags behind in terms of reaching optimal performance and efficiency. The author criticizes the unnecessary complexity and size of modern applications, calling for a reevaluation of development practices to prioritize simplicity, excellence, and thoughtful resource utilization.

JAVASCRIPT ECOSYSTEM
JS Weekly Pulse

• 📢 The React team announced that Server Actions are ready for adoption by libraries and frameworks via React Canary.

• 🚀 Bun 1.0.7 - Optional peer dependencies in bun install, improvements to Node.js compatibility, and lots of bug fixes.

• 🚀 Platformatic 1.0.0 - Breaking change detector for microservices and stackables.

• 🚀 Yarn 4.0 - Hardened mode to protect against attacks, improved user interface, TypeScript integration, interactive tools, and better performance.

• 🚀 Storybook 7.5 - Next.js improvements, enhanced Angular support, and 2.2x faster startup for React TypeScript projects.

RECOMMENDATIONS
To-Do List

✅ Deep Dive: Node vs Deno vs Bun.

✅ Register: AWS re:Invent is taking place on Nov. 27 - Dec. 1, 2023 in Las Vegas, NV. Learn about the latest cloud industry innovations, meet with AWS experts, and build connections.

✅ Learn: What every developer should know about GPU computing.

✅ Watch: Matteo Collina demonstrates how to improve Node.js API performance, increasing requests per second by 5x and reducing latency to 1/3.

✅ Listen: Brad Frost discusses design systems, best practices, and keeping codebases in sync on the Syntax podcast.

✅ Read: The Startup CTO’s Handbook, covering leadership, management and technical topics for leaders of software engineering teams

COMMUNITY SPOTLIGHT
Hot Picks in the Dev Community

OpenSheetMusicDisplay - A MusicXML renderer for the browser.

Backroad - Create web apps with no frontend-experience, all with Node.js.

Atrament - JS library for beautiful drawing and handwriting on HTML Canvas.

Nexe - command-line utility that compiles your Node.js application into a single executable file.

MEME
It Does Run 😄 

🌙 Hello world ☀️ 

Exciting News! TanStack Query v5.0.0 has officially launched after an intense year of development, improving from v4 with a streamlined API and size reduction of around 20%. New features include simplified optimistic updates, 1st class suspense support, sharable mutation state, and enhanced Devtools.

In this week’s email:

• Design System: Learn how Tinder created Obsidian.

• Machine Learning: Understand how Expedia uses price alerts to increase engagement with their users.

• React: Million.js compiles React components, reducing virtual DOM use and cutting memory usage by 55%.

• Node.js: Improve Node.js application performance by leveraging the cluster module for multi-instance concurrency and scaling.

• Next.js: Optimized barrel file imports resulted in 70% faster development, 28% quicker builds, and 40% improved cold starts.

• Productivity: Managing capacity instead of time to do our best work.

DESIGN SYSTEM
The Creation of Tinder’s Design System Obsidian

Created with Midjourney

Darragh Burke

Introduction
In 2012, Tinder introduced an innovative approach to online dating with its Swipe Right® and Swipe Left™ features. This intuitive design quickly gained traction, leading to the app's immense popularity.

As Tinder grew over the decade, the design complexities expanded, and a design overhaul became essential for coherence and consistency.

The Challenge
Tinder's initial success led to swift feature introductions. This rapid expansion, while essential for keeping user engagement high, resulted in a design language that began to diverge in different parts of the app.

Not only did this create challenges in maintaining a unified user experience, but it also led to increased costs in design adaptations and a more intricate codebase.

Difficulty Maintaining Styles

The Obsidian Solution
In response to the growing design challenges, Tinder conceptualized 'Obsidian'. This new system was intended to be a design ecosystem composed of standardized design tokens, reusable UI components, documentation, and design tools.

• Design Tokens: These became the building blocks of Obsidian. Design Tokens were standardized references for every design aspect ranging from colors to fonts.

  

  Tokens Studio, A Figma Plugin

  • Base Tokens: These were direct design references, pointing to specifics such as exact color shades or font measurements.

  • Context Tokens: To provide adaptability, especially in scenarios like shifting between dark and light modes, context tokens were introduced.

• Compatibility with Prevailing Tools: The Obsidian system was not built in isolation. It effectively integrated existing tools like Tokens Studio on Figma for defining design tokens. Moreover, to ensure consistent representation of design aspects across various platforms like iOS, Android, and the Web, Style Dictionary was employed.

To support designers and developers in adapting to Obsidian, Tinder went the extra mile:

• A detailed guide was crafted using Zeroheight to help teams navigate the nuances of the new system.

  

  Design Documentation Created Using Zeroheight

• A token lookup tool was introduced. This handy tool streamlined the process of selecting and employing design elements.

  

  Tinder’s Token Lookup Tool

Impact & Results

The benefits of Obsidian were immediately noticeable:

• Uniform Design: Features such as the sexual orientation selection screen, once fragmented in design, now showcased greater uniformity and accessibility.

• Feedback from the Trenches: Senior iOS engineer at Tinder, Siddharthan Asokan, was particularly impressed. He emphasized the reduced time spent in design deliberations and how Obsidian made design implementation remarkably straightforward.

P.S. If you’re enjoying the content of this newsletter, please share it with your network: https://www.fullstackexpress.io/subscribe

MACHINE LEARNING
How Expedia Uses Price Alerts to Increase Engagement

Created with Midjourney

Oarinde Henok Yemam

Expedia Group (EG) offers travelers optimal deals using technology like flight price alerts. These alerts notify users of price changes, allowing bookings at desired prices.

Expedia Price Alerts

Expedia’s transition from business rule-driven alerts to machine learning (ML) has helped improve traveler engagement by delivering more precise notifications and avoiding over-communication.

Message Relevancy (MR) Model
Price alerts, originally based on rigid business rules, lacked flexibility. In contrast, the ML-driven approach tailors notifications for a more enhanced user experience.

By analyzing data on subscriptions, flight routes, and user behavior, the system anticipates user engagement, emphasizing metrics like recall and alert open rates.

Key aspects incorporated:

• Outcome Objective: Ensure alerts meet traveler's financial goals while also optimizing business costs.

• Data Usage: Data about subscriptions, interactions, routes, and flight price changes are used to predict if a traveler will open a notification.

• Success Metrics: Focused on recall metric and business metrics like subscriber retention and notification open rates.

• Techniques and Optimizations: Used Random Forest Classifier for its simplicity and later introduced improvements like automated model tuning to increase efficiency.

Model Architecture and Improvements
The model was built on the Random Forest Classifier and later expanded to include different types of "opens" as targets.

General Random Forest Classifier Architecture

• Enhancements:

  • Automated Model Tuning (AMT) was used for better hyperparameter tuning, leading to a 20% improvement in the F1 score.

  • Moved from scikit-learn's RandomizedSearchCV to AMT for better optimization and efficient model training.

• Safeguard Rules:

  • Implemented to ensure a wider subscriber base receives notifications, such as sending alerts to long-time subscribers or when price changes cross a certain threshold.

The improved model provides timely notifications from the second day of a traveler's subscription.

Results
The introduction of the MR model brought notable improvements:

• Increase in the opt-in rate for price alerts.

• More subscribers are receiving and opening notifications.

• Demonstrates the model's positive impact on the price alerts campaign, ensuring better engagement and satisfaction among travelers.

DATA STRUCTURES & ALGORITHMS
Best Time to Buy and Sell Stock

Missed the solutions to this week’s coding workout?

Learn how to solve this common interview question here.

REACT
Making React Apps Memory Efficient | Million.js Beyond Speed

Created with Midjourney

Ricardo Nunez

Million.js is a tool designed to optimize React applications, making them more memory-efficient.

Traditional React apps use the virtual DOM to represent UI components, which can lead to large nested JavaScript objects that consume significant memory, especially in complex component trees.This becomes a challenge with continuous object diffing, garbage collection, and potential memory leaks.

Million.js addresses these issues by compiling React components into optimized Higher Order Components, reducing the reliance on the virtual DOM, and leading to a reduction in memory usage by approximately 55% compared to standard React, offering performance closer to vanilla JavaScript.

NODE.JS
How To Scale Node.js Applications With Clustering

Created with Midjourney

Stanley Ulili

In Node.js, by default, applications run on a single CPU thread, which can lead to performance issues for CPU-intensive tasks.

To address this, Node.js offers the cluster module, allowing for the creation of multiple application instances running concurrently on a machine. This module also features a built-in load balancer using the round-robin algorithm to distribute incoming requests among the instances.

This tutorial teaches how to implement clustering in a Node.js app, enhance scalability with the pm2 module, and evaluate performance differences using load testing tools.

NEXT.JS
How We Optimized Package Imports in Next.js

Created with Midjourney

Shu Ding

In the recent version of Next.js, significant optimizations were made to package imports, leading to quicker local development and improved production cold starts.

The optimization specifically addresses the inefficiencies of barrel files in JavaScript, which are used for grouping and exporting multiple modules. Traditional methods, like tree-shaking, have limitations, so Next.js introduced the "optimizePackageImports" option to auto-analyze and map imports.

This change has resulted in up to 70% faster development times, 28% faster production builds, and 40% quicker cold starts.

PRODUCTIVITY
Manage Your Capacity, Not Your Time

Created with Midjourney

James Stanier

Managing your capacity, which is influenced by energy levels, is more crucial than just managing time. Being mindful of tasks that drain or boost your energy can optimize productivity and prevent burnout. To enhance efficiency, periodically reflect on your work patterns and balance deep work with necessary breaks.

JAVASCRIPT ECOSYSTEM
JS Weekly Pulse

• 📢 Node.js 21 is now available. The release features an updated V8 engine, introduces stable WebStreams, offers a flag to switch module defaults, and enhances the test runner for functional tests.

• 📢 API improvements for working with files in the browser.

• 🚀 SolidJS 1.8.0 - Enhanced streaming serialization, deep data de-duping, improved hydration, reduced template sizes, and improved global script management.

• 🚀 Electron 27.0.0 - Upgrades to Chromium 118.0.5993.32, Node 18.4.0, and V8 11.8.

• 🚀 Fresh 1.5 - Partials, improved client-side navigation, UI enhancements.

• 🚀 Bun 1.0.6 - Addresses regression impacting Docker usage with Bun and implements support for the overrides & resolutions fields in package.json.

• 🚀 TanStack 5.0.0 - Simplified optimistic updates, sharable mutation state, suspense support, and more.

• 🚀 Parcel 2.10 - Big performance improvements, an improved --lazy mode, reduced runtime bundle sizes, and web extension improvements.

RECOMMENDATIONS
To-Do List

✅ Explore: Tesla releases FleetAPI is a RESTful data and command service providing access to Tesla vehicles and energy devices

✅ Learn: Growing as an engineer at Stripe.

✅ Watch: How Spotify manages software at scale. Kelsey Hightower talks with Niklas Gustavsson about Fleet.

✅ Listen: The future of HTTP with Nick Shadrin and Roman Arutyunyan, software architect and principal software engineer at Nginx.

✅ Network: PayPal Developer meetup at Money 20/20 in Las Vegas, October 23rd.

COMMUNITY SPOTLIGHT
Hot Picks in the Dev Community

Tumblr.js - Official JavaScript client for the Tumblr API. Detailed docs here.

Viselect - Lightweight library delivering high performance, offering a visual method for element selection similar to desktop interfaces.

OXC - A high-performance toolkit for JavaScript and TypeScript, written in Rust.

Nostalgist - A JavaScript library used for running emulators of retro consoles inside browsers.

Cockpit - Web-based graphical interface for servers.

MEME
If ChatGPT Was A Person…

🌙 Hello world ☀️ 

If you’re lucky enough to be in London today, don’t miss out on Google Cloud Next London. This is a free, in-person event featuring the latest innovations in AI, Data, and Security with keynotes from Google, Google Cloud and Google DeepMind executives.

In this week’s email:

• Architecture: Learn how Gusto revamped their payment system, achieving an 81x speed improvement.

• Infrastructure: Learn how Slack rebuilt their cron execution system for scalability and efficiency.

• JavaScript: Speeding up the JavaScript ecosystem, the barrel file debacle.

• React: Building accessible and typesafe modern web forms.

• Next.js/Remix: An in-depth case study comparing Next.js 13 and Remix.

• Developer Tools: Why successful tools must align with developers' practical needs rather than idealistic aspirations.

ARCHITECTURE
Gusto’s 81x Performance Improvement in Payments Packaging

Created with Midjourney

Peter Harris, Simul Kadakia

In 2022, Gusto's Payments Engineering team faced an impending challenge: a predicted spike in payment volumes by year's end threatened to overwhelm their existing payment packaging system.

How It Started
The original process, which bundled money movements into ACH files for bank transfers, had a few challenges:

1. Ran in 5-minute cycles, gathering a maximum of 5,000 entries each time.

2. This single-threaded system presented multiple inefficiencies, such as often producing ACH files with minimal entries and mishandling cancellations.

How Gusto Originally Handled Payments

Revamping For Efficiency
Addressing these challenges required a shift in architecture. Gusto adopted a map-reduce framework, enabling parallel processing of ACH entries using tools like Sidekiq and Redis.

Gusto’s New Architecture

This architectural change significantly leaned on tools like Sidekiq and Redis to manage parallel tasks without compromising data integrity.

The team also introduced the ACH Engine, a solution which:

• Enabled packaging closer to submission time.

• Optimized file sizes.

• Cut out redundant operations.

Rethinking Redlock and Smarter Cancellations
The Redlock system, initially implemented to prevent race conditions, became a notable bottleneck. However, by refining their interaction with Redis, the team markedly boosted efficiency.

Furthermore, Gusto implemented a better cancellation solution that prioritized such requests, eliminating potential process conflicts. Through targeted optimizations, including eliminating redundant checks and refining datastore queries, Gusto significantly amplified their processing capability.

Gusto’s New ACH Entry Cancellation Process

Results
After nearly a year of effort, Gusto's Payment Engineering team achieved remarkable improvements in performance:

• Entry Count

  • Before: 149,418

  • After: 692,920

• Processing Time

  • Before: 55 minutes

  • After: 3 minutes 9 seconds

• Throughput

  • Before: 2,716 entries per minute

  • After: 219,974 entries per minute

• Overall Improvement:

  • 81x faster

P.S. If you’re enjoying the content of this newsletter, please share it with your network: https://www.fullstackexpress.io/subscribe

INFRASTRUCTURE
Rebuilding Slack’s Cron Execution System for Scalability and Efficiency

Created with Midjourney

Claire Adams

Cron scripts are pivotal for Slack's operations such as reminders, email notifications, and database cleanups. As Slack grew, the number of scripts and the data they process expanded, leading to reliability concerns and scaling challenges.

This led to the need for a new, scalable, and more reliable system for executing cron scripts.

Slack’s New Cron Execution System

Original System and Shortcomings
In its infancy, Slack's cron infrastructure was fairly straightforward. It comprised a single node that housed all the scripts and one crontab file that spelled out the schedules for each script.

While this served its purpose initially, as the data and scripts grew, merely migrating to larger nodes became an untenable solution. The main problem lay in the fact that the entire system hinged on this one node, making any issues—be it provisioning, rotation, or configuration—a potential system-wide disruption.

New Cron Execution System
The new system consists of three core components:

1. Scheduled Job Conductor

   

   Scheduled Job Conductor

   • What: A Golang service for mimicking cron functionality, leveraging a Golang cron library.

   • Why: To preserve the original cron string format for easier migration.

   • How it Operates:

     • Utilizes Bedrock (Slack’s Kubernetes wrapper) for scalability.

     • One pod does the scheduling, while others wait to take over if needed to avoid downtime.

2. Slack's Job Queue

   

   Slack’s Job Queue

   • What: An asynchronous compute platform running around 9 billion jobs daily.

   • Composition: Jobs flow through Kafka, then to Redis, and finally to a job worker for execution.

   • Advantage: Offloads compute and memory needs to an existing robust system, reducing build and maintenance time.

3. Vitess Database Table

   

   Vitess Table

   • Purpose: Deduplication, job tracking, and reporting.

   • Functionality:

     • Ensures only one copy of a script runs at once.

     • Tracks the job state in the system.

     • Supports a user-interface for script execution information, aiding in error detection and state verification.

By utilizing existing solutions, the effort for both development and future maintenance is reduced. This approach ensures longevity while optimizing resource allocation.

DATA STRUCTURES & ALGORITHMS
Linked List Cycle

Missed the solutions to this week’s coding workout?

Learn the algorithm used to solve this problem here.

JAVASCRIPT
Speeding Up the JavaScript Ecosystem - The Barrel File Debacle

Created with Midjourney

Marvin Hagemeister

Numerous projects are cluttered with files that simply re-export other files. These "barrel files" significantly contribute to the sluggish performance of JavaScript tools in larger projects. Marvin analyzes the effects of barrel files on performance.

REACT
Accessible, Typesafe, Progressively Enhanced Modern Web Forms

Created with Midjourney

Kent C. Dodds

Building web forms is a complex task often requiring developers to override browser defaults. Modern frameworks like Remix simplify form handling by emulating default browser behaviors while adding enhancements. Tools like Zod and Conform aid in creating typesafe and accessible forms with schema-based validation and seamless integration between forms and underlying data structures.

NEXT.JS, REMIX
Next.js 13 vs Remix: An In-depth Case Study

Created with Midjourney

Prateek Surana

Next.js and Remix are both popular frameworks for building React web applications. Prateek Surana provides a comparative analysis of their new routing and data-fetching techniques, using a Twitter clone project as an example. While Remix uses loaders for efficient parallel data fetching, Next.js 13 leverages React Server Components for more direct and streamlined data integration within React components.

DEVELOPER TOOLS
Why Aren’t There More Programming Languages Startups?

Created with Midjourney

Jean Yang

Jean Yang, former professor and founder of Akita Software, examines why cutting-edge programming languages and tools from academia don't often translate to mainstream adoption in the tech industry. She argues that developers prioritize tools that enhance their workflows and address immediate needs rather than those offering perfect guarantees. For "deep tech" tools to gain traction, they must align more with developers' current workflows, be interoperable, and focus on incremental improvements that mesh with existing technologies.

JAVASCRIPT ECOSYSTEM
JS Weekly Pulse

• 📢 Security efforts to implement control-flow integrity (CFI) in V8 by preventing memory-corrupted exploits from executing arbitrary code.

• 📢 Japa offers a streamlined testing framework for Node.js without the transpilation overhead associated with Jest.

• 📢 NodeConf EU is set for November 6-8, 2023 in Ireland.

• 📢 GitHub Actions has restricted the use of GITHUB_ENV for setting the NODE_OPTIONS variable in workflows.

• 🚀 Bun 1.0.4 - Virtual modules support, server.requestIP, memory consumption reduction in Bun.serve(), and bug fixes.

• 🚀 TypeScript 5.3 Beta - Import attributes, enhanced narrowing, and various optimizations.

• 🚀 Redux Toolkit 1.9.7 - Enhancements to RTKQ hook TypeScript types for notably better performance.

• 🚀 pnpm 8.9.0 - Performance improvements for macOS and Windows Dev Drives

RECOMMENDATIONS
To-Do List

✅ Interesting: Firefox fixes a 22 year old bug.

✅ Join: GitHub is launching a certification program in 2024 to validate developers' skills and expertise, with an early exam opportunity at GitHub Universe on November 8-9. If you can’t make it, join the waitlist.

✅ Learn: Pinterest's rapid growth, highlighting the platform's ability to accommodate 11 million users while being supported by a lean team of just six engineers.

✅ Watch: Replay ViteConf 2023, including 44 talks about the Vite ecosystem.

✅ Listen: OWL is the world’s fastest JavaScript framework, inspired by Vue and React, designed specifically for Odoo's unique modular requirements and built with modern features like asynchronous rendering and a virtual DOM.

COMMUNITY SPOTLIGHT
Hot Picks in the Dev Community

Crank.js - A web framework that allows components to be defined using sync, async, and generator functions.

Gitify - GitHub notifications on your menu bar.

Kysely - Type-safe and autocompletion-friendly typescript SQL query builder. Inspired by knex.

Npkill - Easily find and remove old and heavy node_modules folders.

Wretch - Small wrapper around fetch that simplifies network requests and responses.

Docx - Easily generate and modify .docx files with JS/TS. Works for Node and on the Browser.

MEME
Friday Deployments Be Like…

🌙 Hello world ☀️ 

Two years ago today, Meta (formerly Facebook) had its infamous worldwide outage that lasted 7 hours and 11 minutes, affecting every one of its platforms. As we reflect on that day, it serves as a reminder of the importance of continuous learning and improvement. After all, it's our dedication to the craft, that truly puts us on the map.

In this week’s email:

• Testing: Netflix simulates expected traffic to identify scalability issues and validate their Ads algorithms.

• Security: PayPal implements graph database to combat fraud.

• TypeScript: The saga of the Closure Compiler, and why TypeScript won.

• Node.js: Connect, a universal Socket API for JavaScript runtimes.

• Career Advice: How to influence and think like a Staff Engineer.

TESTING
How Netflix Ensured a Smooth Launch for its Ad-Integrated Tier

Jose Fernandez, Ed Barker, Hank Jacobs

In November 2022, Netflix introduced a new tier called "Basic with ads". This tier added new backend components and a remote call to their ads partner during playback.

To ensure a smooth launch, Netflix simulated the expected traffic weeks in advance, using a traffic migration framework. This simulation helped identify scalability issues and validate their Ads algorithms.

Realistic Test Traffic
Netflix's traffic generally follows a predictable pattern, with occasional spikes due to new content releases or regional AWS data center shifts. To test their new ad system, Netflix decided to replay real traffic, making it appear as if it was "Basic with ads" traffic. This approach allowed for realistic testing of the new systems and algorithms.

Netflix’s Traffic Patterns

The Setup
Netflix aimed to ensure that the simulation did not impact actual customers. They used real member viewing habits for the simulation, but no ads were shown to these members.

The data science team projected the subscriber count for the new tier a month post-launch. This data was used to simulate a subscriber population. Traffic matching this simulated population was stored in a Mantis stream.

This stream was then replayed in a separate production environment designed for this purpose. The responses from this environment, which included ad metadata, were stored in a Keystone stream. This data was then used to simulate device playback and ad impression tracking.

Netflix’s Traffic Replay Setup

The Rollout
Netflix started by replaying a small percentage of traffic, gradually increasing it while monitoring metrics. They reached 100% replay and ran it continuously.

They also tested their system's response to traffic spikes caused by regional AWS evacuations. During this period, they set up and tested their ad monitoring and alerting system. They also conducted chaos experiments to test their systems under failure scenarios, ensuring resilience.

Key Takeaways

• Simulating real traffic provides confidence in new systems and algorithms.

• Large-scale testing with representative traffic can reveal unexpected issues.

• Replay traffic can be used for more than just load testing, offering potential for new product and feature development at Netflix.

P.S. If you’re enjoying the content of this newsletter, please share it with your network: https://www.fullstackexpress.io/subscribe

SECURITY
PayPal’s Usage of Graphs in Combating Fraud Risk

Xinyu Zhang

Account takeover (ATO) poses a significant fraud risk in online payments. To address this, PayPal has implemented a real-time graph database, which uses edges and vertices to represent relationships between objects.

Asset Sharing
Assets such as IP addresses, physical addresses, and device IDs are connected to PayPal accounts. By representing these connections in a graph structure, it becomes evident how assets are shared among different accounts.

Typically, regular accounts maintain consistent asset usage over time. However, accounts that have been taken over by fraudsters often exhibit frequent and abrupt changes in their associated assets.

By analyzing the Asset-Account Graph, PayPal can detect unusual asset-sharing patterns and identify potentially fraudulent accounts.

Asset to Account Graph Highlighting Abnormal Linking Behaviors

Transaction Patterns
Users often engage in multiple transactions in a short time frame. While traditional relational databases struggle to efficiently represent complex transaction patterns, graph databases excel in this area.

For instance, a pattern named "ABABA" indicates two accounts sending money back and forth in quick succession.

Such patterns, which are prevalent in ATO incidents, can be swiftly detected using graph databases, thereby preventing potential fraudulent activities.

Example of Transaction Patterns

Graph Features
Graph features involve calculating specific metrics based on the graph's topology. One notable feature is the "connected community", which describes interconnected subgraphs within the primary graph.

These features are invaluable for identifying groups of closely connected accounts and their transaction behaviors, which can be instrumental in detecting anomalies.

By examining these graph features, PayPal can identify key nodes in the graph, offering insights that can help prevent large-scale fraudulent activities.

Account-Transaction Graph

Key Takeaways

1. Graph databases are essential tools in combating ATO fraud, offering a more efficient way to represent and analyze complex relationships.

2. Asset sharing patterns in the graph can highlight abnormal behaviors, signaling potential account takeovers.

3. Transaction patterns, like the "ABABA" pattern, can be more easily detected and analyzed using graph databases compared to traditional relational databases.

4. Graph features, such as the "connected community", provide a deeper understanding of transaction behaviors and can be crucial in anomaly detection.

DATA STRUCTURES & ALGORITHMS
Valid Parenthesis

Missed the solutions to this week’s coding workout?

Learn how to use the stack data structure to solve this problem here.

TYPESCRIPT
The Saga of the Closure Compiler, and Why TypeScript Won

Dan Vanderkam

The article discusses the saga of Google's Closure Compiler and how TypeScript eventually became more dominant in the JavaScript ecosystem. The Closure Compiler focused on optimal JavaScript minification, but faced challenges adapting to the modern JavaScript landscape. On the other hand, TypeScript prioritized developer experience, type safety, and adaptability, leading to its widespread adoption among software engineers. Dive into this chapter of JavaScript history.

REACT
Introduction to the New React UseEffectEvent Hook

Gabriel Chinedu, Tomas Fernandez

The useEffectEvent is an experimental React hook that streamlines the process of adding and removing event listeners in React components. This hook offers developers a cleaner and more efficient way to handle events in functional components. See it in action.

NODE.JS
Best Practices for Securing Node.js Applications in Production

Antonello Zanini, Tomas Fernandez

Node.js is a widely-used backend technology that's increasingly targeted by online attacks. This guide highlights 15 best practices, including not running Node.js with root privileges, regularly updating NPM libraries, and adopting an HTTPS-only policy. Implementing these measures ensures robust security for your Node.js applications in production. Integrate these techniques and tips.

CAREER ADVICE
Lessons from a Staff Engineer and Ex-Twitter Tech Lead

Jordan Cutler, Ron DeVera

This article features insights from Ron DeVera, a seasoned Staff Engineer and former Twitter Tech Lead, on influence, managing relationships, and differences between staff and senior engineers. Key takeaways include understanding situations from others' perspectives, avoiding pushing suggestions too forcefully, using the Situation-Behavior-Impact framework for feedback, and recognizing the workload of managers when discussing topics. Learn how to think like a staff engineer.

JAVASCRIPT ECOSYSTEM
JS Weekly Pulse

• 📢 NPM now supports publishing packages with provenance from GitHub Actions and GitLab CI/CD, linking them verifiably to their public source repositories.

• 📢 A new socket API that works across JavaScript runtimes is in the works.

• 📢 Updates from the 98th TC39 meeting include Resizable ArrayBuffers, ShadowReam, and RegExp escaping.

• 📢 Next.js Conf 2023 is set for October 26 at 10am Pacific.

• 🚀 Astro 3.2 - Enhanced view transitions and more seamless integrations.

• 🚀 Node.js 20.8.0 - Performance improvements to streams.

• 🚀 Babel 7.23.0 - Decorator metadata and new import features.

• 🚀 ESLint 9.0.0 - Breaking changes for rule authors.

• 🚀 pnpm 8.8.0 - New options for pnpm commands, improvements to error handling and node-linker.

RECOMMENDATIONS
To-Do List

✅ Explore: GitHub’s Innovation Graph compares metrics by economy as well as specific categories such as Git pushes, repositories, and programming languages.

✅ Learn: Understand the significance of securing container images, and how key tools like Sigstore Cosign, Notary v2, and Docker Content Trust are integrated.

✅ Watch: Discover key principles for scaling distributed systems, drawing from Amazon and eBay’s architecture, and explore the importance of simplicity, communication methods, and consistent feedback.

✅ Listen: Andrew Hoying from Google discusses the evolution of system hardening, challenges at Google's scale, the balance between security and productivity, adapting to regulations, and how lessons learned have influenced product improvements.

✅ Try it out: ElectricSQL offers a reactive real-time sync layer for Postgres, enabling instant local-first data synchronization seamlessly within your existing Postgres-driven apps.

 ✅ Register: P99 CONF is a premier tech conference featuring talks from engineers at leading companies like Netflix, Google, and TikTok, covering high-performance applications, distributed systems, and many more.

COMMUNITY SPOTLIGHT
Hot Picks in the Dev Community

N64js: An n64 emulator written in (mostly) pure ES6 JavaScript. It runs many ROMs at full frame rate.

Dropzone: An easy to use drag & drop JavaScript library. It supports image previews and shows nice progress bars.

Chatgpt.js: A powerful, open source client-side JavaScript library for ChatGPT.

Xterm.js: A terminal for the web.

LangChain.js: A framework for building apps with language models. Written in TypeScript.

MEME
First Day on the Job

Hello world.

In the ever-evolving world of software, where Bun vs Node.js debates seem to be the new coffee-break chatter, there's a fresh update that might just overshadow our favorite arguments.

ChatGPT, once confined to the realm of text, has now expanded its horizons to see, hear, and speak. As we navigate these rapid advancements, let's ensure we're not just debating, but also adapting.

In this week’s email:

• Architecture: Expedia introduces the Juggler model to optimize hotel rankings.

• Infrastructure: Canva saves $3.6M annually by optimizing AWS S3 storage.

• Node.js: v20 showcases enhanced performance, as analyzed by Rafael Gonzaga.

• React: Josh Comeau explores the nuances of React's Server Components as React celebrates its 10th anniversary.

• JavaScript: Dan Abramov discusses the evolving complexities of the JavaScript ecosystem.

• Development Practices: Emphasizing the importance of addressing technical debt for software efficiency.

ARCHITECTURE
Expedia’s Solution to Optimizing Lodging Rankings

Tiago Cunha

Expedia has developed a system called the Juggler model, an algorithm designed to optimize the balance between travelers, hotel properties, and their platform's objectives.

How Expedia Ranks Hotels
When a user initiates a hotel search on Expedia, properties are ranked based on a set of algorithms.

Expedia’s Lodging Ranking Search

This system not only considers raw data but also user interactions, such as clicks and bookings, which are then fed back into the system for continuous learning.

System Architecture and Stakeholders
The platform's architecture is designed to cater to three primary entities:

1. Hotel properties, which are keen on maximizing their visibility.

2. Travelers, who demand optimal choices based on their preferences and budget.

3. Expedia's platform, which aims to maximize revenue while ensuring user retention.

Algorithmic Ranking
Rather than relying solely on direct relevance (like matching a user's desired location or price range), the Juggler model integrates multiple factors to determine the ranking.

This is achieved through a composite scoring system that combines direct relevance with other business-driven adjustments.

The weights for these adjustments are determined dynamically, ensuring a balance between immediate user satisfaction and long-term platform health.

The Juggler Model's Core
At its heart, the Juggler model is a meta-learning system.

Expedia’s Juggler Framework

It analyzes patterns from historical searches to predict optimal weight parameterizations for future queries.

This is similar to recommendation systems but tailored for the complexities of hotel bookings.

Future Tech Stack Enhancements
The success of the Juggler model has spurred further R&D. Expedia is exploring:

• Enhanced personalization techniques, potentially integrating neural networks for deeper user profiling.

• Advanced deep learning architectures to refine the search context extraction.

• Reinforcement learning to allow the system to adapt in real-time based on user interactions.

P.S. If you’re enjoying the content of this newsletter, please share it with your network: https://www.fullstackexpress.io/subscribe

INFRASTRUCTURE
How Canva Saves $3.6 Million Annually in S3 Costs

Josh Smith

Canva, a globally recognized online design platform, relies heavily on AWS for its production workloads.

• Services used: Amazon S3, ECS, RDS, DynamoDB.

• User growth: Scaled to over 100 million monthly active users since 2013.

• Challenge: Efficiently storing a vast library and user-generated content.

Amazon S3: The Backbone
Amazon S3 offers a robust, cost-effective storage solution. Canva's data footprint on S3 is immense, with over 230 Petabytes of data.

The challenge?

Understanding the various S3 storage classes and optimizing them for cost and performance.

Amazon S3 Storage Classes

Storage Classes and Canva's Usage
For frequently accessed content like Canva's template libraries, the S3 Standard storage class is ideal.

However, user-generated content, which is often accessed briefly and then seldom revisited, was stored in the S3 Standard-Infrequent Access (S3 Standard-IA) class.

This class is cost-effective but still offers quick retrieval times. For specific cases like logs and backups, Canva uses S3 Glacier Flexible Retrieval.

The introduction of S3 Glacier Instant Retrieval in 2021, which combines low-cost storage with millisecond retrieval times, prompted Canva to reevaluate its storage strategy.

Deep Dive into Data Patterns
To optimize storage costs, Canva needed a granular understanding of its data access patterns.

• Canva's primary content: Templates, stock photos, graphics.

• User activity: Majority access content shortly after creation.

• Previous strategy: Transition data from S3 Standard to S3 Standard-IA after 30 days.

• New tool: S3 Storage Class Analysis for better data understanding.

• Observation: Retrieval rate drops significantly after the first 15 days.

Canva’s S3 Access Patterns Over Time

Cost Analysis and Transitioning
Transitioning between S3 storage classes incurs costs based on the number of requests.

With over 300 billion objects in Canva's inventory, a blind transition would be costly.

However, the potential savings from using S3 Glacier Instant Retrieval, given its lower storage cost, are recurring.

By analyzing the average object size in each bucket, Canva could calculate the breakeven point for transitioning data and prioritize buckets for migration.

Canva’s S3 Breakeven Comparison

Results and Takeaways
Not only was Canva’s migration successful, the results are clear:

• Migration simplicity: Applied a lifecycle policy to buckets.

• Speed: Migrated nearly 80 billion objects in about two days.

• Current data: 130 petabytes of Canva’s 230 petabytes in S3 now in S3 Glacier Instant Retrieval.

• Cost savings: Approximately $300,000 per month ($3.6 million annually).

• ROI: Positive return seen just a few months after the transition.

• Initial cost: Over $1.6 million to transition roughly 80 billion objects.

• AWS partnership: Continuous support and tailored storage solutions.

• Key lesson: Understand data access patterns before making transitions.

NODE.JS
The State of Node.js Performance in 2023

Rafael Gonzaga

Rafael Gonzaga's analysis of Node.js v20 reveals significant performance enhancements compared to earlier versions, using three distinct benchmark suites. Strategic initiatives, such as the 'Performance' and 'Startup Snapshot', alongside impactful pull requests, are driving these improvements. Despite some regressions, the Node.js performance team's ongoing efforts suggest a bright future for the platform's efficiency.

REACT
Making Sense of React Server Components

Josh Comeau

Josh Comeau highlights React's 10th anniversary and its latest innovation, React Server Components, which allows components to run exclusively on the server. Amidst the online confusion about this new feature, Comeau dives deep into its workings, emphasizing its potential and aiming to clarify its benefits and integration with Server Side Rendering (SSR). For context, he provides a primer on SSR, explaining how it offers users a fully-formed HTML document, improving initial load times, and how client-side React then enhances this with interactivity.

JAVASCRIPT
The Melting Pot of JavaScript

Dan Abramov

In this article by Dan Abramov, he discusses the complexities and evolution of the JavaScript ecosystem. Abramov highlights the inventive and ubiquitous nature of JavaScript, emphasizing its reflection of human culture. He advocates for making JavaScript tools more user-friendly, emphasizing the importance of sensible defaults, minimal configuration, and clear output to ensure a more approachable experience for newcomers. An older article, but very much still relevant today.

DEVELOPMENT PRACTICES
Measuring Technical Debt to Avoid Boiling Frog Syndrome

Egor Savochkin

Technical debt represents the gap between the current and ideal state of software, impacting the ease of future modifications. It can accumulate from quick solutions, intentional decisions, or changing requirements. Addressing this debt early, through visibility and proactive management, is crucial to maintain software agility and efficiency.

AROUND THE WEB
What Else is Trending

✅ Learn: Sir Tim Berners-Lee launched the first website in 1991 at CERN on a NeXT computer. While there are now over 1.8 billion websites, the original can still be visited at it’s original address.

✅ Watch: Dive into TypeScript's captivating documentary, exploring its origins, purpose, and insights from its creators and Microsoft insiders.

✅ Listen: Explore solutions for solving glitchy animations in the latest episode of The CSS Podcast.

✅ Contribute: Contributing to open source is a rewarding way to collaborate and innovate. If you're wondering where to start, look no further.

✅ Join: Project IDX is Google's new cloud-based initiative to streamline full-stack, multiplatform app development through a familiar yet innovative web-based workspace. Join the waitlist.

COMMUNITY SPOTLIGHT
Hot Picks: Cool Tools in the Dev Community

Node SerialPort

This GitHub repository provides a cross-platform JavaScript library for interfacing with serial ports

github.com/serialport/node-serialport

Vest

Vest is a framework-agnostic validation library for web forms, using a syntax inspired by testing tools like Mocha and Jest.

github.com/ealush/vest

Semgrep

Semgrep is an open-source static analysis tool that detects bugs, vulnerabilities, and enforces code standards without uploading your code.

github.com/returntocorp/semgrep

Loaders.gl

Loaders.gl is a framework-independent tool within the vis.gl suite, offering loaders tailored for geospatial, 3D, and big data visualizations.

https://github.com/visgl/loaders.gl

Fast XML Parser

Validate and convert between XML and JS objects directly in JavaScript without relying on C/C++ libraries or callbacks.

github.com/NaturalIntelligence/fast-xml-parser

Want to showcase something that you’re working on? Let us know!

SOCIAL MEDIA
Hall of Fame or Wall of Shame?

The MGM Grand is on a quest for a Red Hat Linux System Admin superhero, ready to forsake sunlight and weekends, all to resurrect the almighty slot machines from their digital slumber. Who needs a social life when you can rebuild IT infrastructure?

Twitter tweet

Over to you: What would you do?

MEMES
Battle of the Runtimes

What men are thinking about: Roman Empire
What JavaScript developers are thinking about: Bun

Good morning! Welcome back to this week's edition of Full Stack Express, your go-to newsletter for web development, software architecture, and system design.

Headlines

• Microsoft's AI Team Exposes 38TB of Private Data on GitHub

• Google Unveils Major Bard Update: Now with Extensions for Maps, YouTube, Flights, and Hotels

• Vercel Releases v0.dev: A Natural Language-Powered Web UI Generator for React

• Remix Rolls Out Version 2 of Its Full Stack Web Framework

• Introducing Nue: The Newest, Fastest Thing in Frontend Development

Featured Deep Dives

• How Instacart Scaled Through the Pandemic Managing 80,000+ Retail Locations and Predicting Real-Time Item Availability

• Figma’s Performance Testing Journey from a Single Macbook to a Dual-System Infrastructure

Quick Bytes

• How Instagram Scaled to 14 Million Users with Just 3 Engineers

• Understanding Backpressure in Software Systems

• Bard’s latest AI Updates and Improvements

• Runtime Comparison Between Node, Deno, and Bun

• AWS's IPv4 Estate Now Worth $4.5 Billion

Community Spotlight

Kiesel, Theatre.js, nanoGPT, and More

Tip of the Week

Boost Your React App's Performance with Lazy Loading and Suspense

Meme of the Week

JavaScript Frameworks: The Never-Ending Story

HOW INSTACART SCALED THROUGH THE PANDEMIC MANAGING 80k+ RETAIL LOCATIONS

The Challenge

Instacart, an online grocery delivery service, faced a complex challenge in predicting real-time item availability across 80,000+ retail locations, especially during the pandemic.

The company needed to ensure that its machine learning (ML) models could scale to predict the availability of millions of items while maintaining low latency and high consistency.

The Turning Point

The pandemic led to a surge in customer demand and fluctuating in-store inventories.

Instacart needed to evolve its Real-Time Availability (RTA) infrastructure to keep pace with these changes and maintain customer trust.

Objectives and Requirements

1. Low Latency: Fast and bulk fetching of availability scores at the retrieval stage.

2. High Consistency: Consistent availability information across all user interfaces.

3. Scalability: Ability to handle predictions for hundreds of millions of items.

4. Experimentation: A framework to test multiple ML models efficiently.

The Solution

Instacart implemented two methods for ingesting ML-generated scores into their database (DB) storage:

1. Full Sync: ML Availability Scoring Service updates a Snowflake table multiple times a day. DB ingestion workers read this table and update the availability scores.

2. Lazy Score Refresh: Scores are updated on-demand when an item appears in search results and exceeds the allowable age.

Full Sync and Lazy Score Refresh Architecture

To foster faster experimentation, Instacart developed a Multi-Model Experimentation Framework with three key components:

1. DB Column per Model: Each model's score is stored in a dedicated DB column.

2. Model-Column Mapping: A service-level configuration system maps the model version to its corresponding unique column.

3. Experiment-Column Mapping: A/B experiments are easily conducted with a unique feature flag associated with each column.

Multi-Model Experimentation Framework

To manage the complexity of different thresholds for various segments, Instacart introduced the Deltas Framework, which allows for the application of fixed deltas to base thresholds, computed at runtime.

Deltas Framework

Real-world Impact

1. Scalability: The lazy score refresh reduced the ingestion load by 2/3rds.

2. Experimentation: A 6X increase in experiments run using the new framework.

3. Customer Trust: Improved "good found rate," crucial for customer retention.

Key Takeaways

1. Scalability vs. Consistency: Balancing these two can be challenging but is crucial for maintaining customer trust.

2. Data Ingestion Strategies: Combining full sync and lazy refresh can optimize both latency and consistency.

3. Modular Experimentation: A well-designed experimentation framework can significantly reduce engineering work and speed up ML testing.

4. Threshold Management: Dynamic thresholding can be a powerful tool for handling complex, multi-segment optimization problems.

Instacart's innovative approach to RTA infrastructure demonstrates how engineering, machine learning, and product design can come together to solve complex, real-world challenges at scale.

Source

FIGMA’S PERFORMANCE TESTING JOURNEY FROM A SINGLE MACBOOK TO A DUAL-SYSTEM INFRASTRUCTURE

The Challenge

In 2018, Figma's entire in-house performance testing system ran on a single MacBook, looping through a series of key test scenarios.

Stress Test with 5,000 Comment Pins

Fast forward to 2023, and the landscape has changed dramatically. Figma's codebase has grown in complexity, with new features, products, and a team distributed globally.

The single MacBook approach was no longer sustainable, especially with the team expanding to over 400 engineers and managers.

The Turning Point

The MacBook that had been running tests overheated in October 2020.

Attempts to replicate the tests on another laptop failed, signaling the need for a more scalable, sophisticated approach to performance testing.

Early Stress Tests Created in FigJam

Objectives and Requirements

Figma aimed to build a system that could:

1. Test every proposed code change in the main monorepo.

2. Complete performance guardrail checks in under 10 minutes.

3. Provide detailed performance metrics and comparisons.

The Solution

Figma deployed two systems connected by the same Continuous Integration (CI) system:

1. Cloud-based System: Runs in GPU-enabled virtual machines (VMs) for every pull request. Due to the noise levels in VMs, a 20% pass margin was set to catch only the most significant regressions.

2. Hardware System: Runs on an array of test laptops, including older machines, allowing for custom test scenarios and more precise performance metrics.

Both systems shared features like detailed HTML reports and CPU profiles for each run.

Real-world Impact

The new system went live in October 2022 and has been instrumental in identifying performance bottlenecks and regressions early in the development cycle.

It has also empowered engineers to collaborate on performance-sensitive code across teams and time zones.

Key Takeaways

1. Start Lean, Plan for Scale: While starting with a minimal setup is good, always have a plan for scaling your testing infrastructure.

2. Proactive vs. Reactive: Shifting from a reactive to a proactive approach in performance testing can save time and improve product quality.

3. Parallelization and CI: Utilizing parallel runs in CI can significantly speed up the testing process.

4. Hardware Matters: For graphics-heavy applications, real hardware testing can provide insights that VMs cannot.

5. Continuous Monitoring: Automated tests and detailed reporting can catch regressions before they impact users, maintaining development velocity.

By embracing a scalable, dual-system approach, Figma has set itself up for sustainable growth, ensuring that performance doesn't become a bottleneck as the product evolves.

Source

BYTE-SIZED TOPICS

How Instagram scaled to 14 million users with only 3 engineers

Instagram's guiding principles and tech stack explained simply

engineercodex.substack.com/p/how-instagram-scaled-to-14-million

Backpressure explained — the flow of data through software

Backpressure is something nearly every software engineer will have to deal with at some point, and for some it’s a frequent problem. But…

medium.com/@jayphelps/backpressure-explained-the-flow-of-data-through-software-2350b3e77ce7

‎Bard's Latest AI Capability Updates & Improvements - Bard

Explore the latest Google Bard updates which include improvements in generative AI capabilities, expanded access, and more.

bard.google.com/updates

Node.js vs. Deno vs. Bun: JavaScript runtime comparison | Snyk

Node.js, Bun, and Deno... oh my! In this post, we compare these three popular JavaScript runtimes so you can determine which is the right one for your project.

snyk.io/blog/javascript-runtime-compare-node-deno-bun

AWS IPv4 Estate Now Worth $4.5 Billion

AWS grew its IPv4 estate with an additional 27 million IP addresses to now owning 128 Million IPv4 addresses. At a value of $35 per IPv4 address, the total value of AWS’ IPv4 estate is ~4.5 Billion dollars. An increase of $2 billion

toonk.io/aws-ipv4-estate-now-worth-4-5-billion/index.html

INTERESTING PRODUCTS, TOOLS & PACKAGES

OrbStack · Fast, light, simple Docker & Linux on macOS

Say goodbye to slow, clunky containers and VMs. The fast, light, and easy way to run containers and Linux. Develop at lightspeed with our Docker Desktop alternative.

orbstack.dev

kiesel

⚡ A JavaScript engine written in Zig

codeberg.org/kiesel-js/kiesel

Theatre.js - animation toolbox for the web

Theatre.js is an animation editor with a visual interface.

www.theatrejs.com

GitHub - karpathy/nanoGPT

The simplest, fastest repository for training/finetuning medium-sized GPTs.

github.com/karpathy/nanoGPT

Download RustRover - JetBrains Rust IDE

Download the latest version of RustRover for Windows, macOS or Linux.

www.jetbrains.com/rust/download

TIP OF THE WEEK

React’s React.lazy() and Suspense allow you to perform code splitting, which can significantly improve the performance of your app by reducing the initial bundle size.

import React, { Suspense, lazy } from 'react';

const HeavyComponent = lazy(() => import('./HeavyComponent'));

function App() {
  return (
    <div>
      <main>
        <Suspense fallback={<div>Loading...</div>}>
          <HeavyComponent />
        </Suspense>
      </main>
    </div>
  );
}

export default App;

In this example, HeavyComponent will only be loaded when it's about to be rendered, reducing the initial load time for your application.

The fallback prop in Suspense allows you to display a loading indicator or some other placeholder while the component is being loaded.

This approach is especially useful for components that are not immediately visible, like modals or tabs that the user might never click, or for splitting routes in a single-page application.

MEME OF THE WEEK

Together with Quick Byte

Good morning and welcome back to another edition of Full Stack Express, your weekly newsletter on web development, software architecture, and system design.

Here’s the notable announcements for the week:

• Bun 1.0 released on September 8, the all-in-one JS/TS toolkit.

• React Aria components is now in Beta.

• OpenAI’s first developer conference in San Francisco.

• Turbo 8 is dropping TypeScript.

• Visual Studio Code adds port forwarding.

We’ll also do deep dives into:

• DoorDash’s Holiday Forecasting Strategy

• How PayPal Scales Kafka to Trillions of Messages

And finish off with:

• The latest tech news and articles

• Interesting tools and packages created by the community

• A useful but lesser-known Linux tip

• A spicy JavaScript meme

ELEVATE YOUR TECH JOURNEY WITH QUICK BYTE (SPONSOR)

Stay ahead of the curve with Quick Byte! Dive into the latest tech trends 📈, discover sizzling startups 📱, and master AI tips & tricks 🤖.

Complement your Full Stack Express insights with a daily dose of tech treats from Quick Byte. Don't miss out!

Subscribe to Quick Byte Today!

DOORDASH’S HOLIDAY FORECASTING STRATEGY

One of the many challenges DoorDash faces is forecasting supply and demand during holidays.

For a company that services thousands of markets, maintaining high-quality experiences at scale for both customers and Dashers (delivery drivers) is extremely challenging.

Operations must be proactively planned in order to ensure that enough Dashers are available and adding additional compensation when less drivers are anticipated.

DoorDash’s Mobilization System

DoorDash’s mobilization system is a collaborative effort between operations, finance, engineering, and machine learning teams.

DoorDash Mobilization System

To run flawlessly, reliable forecasts and predictions are needed to accommodate supply and demand for all days of the year, including holidays.

Limitations to DoorDash’s Current Model

Traditional tree-based models such as random forest and gradient boosting machine (GPM) are popular in the industry due to their ease of use and high accuracy.

While typically excelling at time-series forecasting, they struggle with high variations and anomalies, a common pattern during holiday demand.

Tree-Based Model Generating Inaccurate Forecasts For Holidays

From this example, the model gave a prediction of -35%, the average of all the holidays.

In other words, the Fourth of July (-10%) and Christmas (-5%) were over-forecasted, while Thanksgiving was under-forecasted (+15%), a pretty significant margin of error.

Cascade Modeling Approach

To account for these discrepancies, DoorDash adopted a cascade modeling approach through a series of steps:

1. Calculating holiday multipliers

2. Preprocessing holiday multipliers and model training

3. Generating forecasts and post-processing

Cascade Model in Multiple Steps

These changes led to a significant improvement in the GBM model’s accuracy, measured in terms of weight mean absolute percentage error (wMAPE).

Model Performance

The results speak for itself.

Cascade Approach vs Traditional Feature Engineering

The wMAPE saw a dramatic reduction, dropping from a range of 60-70% to just 10-20% during the Christmas time. Thanksgiving showed a comparable improvement.

When averaged over an entire year of holidays, the wMAPE improved by an absolute 10%.

In conclusion, DoorDash's cascade modeling approach offers a promising solution to the challenges of holiday forecasting, balancing the need for accuracy with the complexities of real-world operations.

Key Takeaways

• Adaptability is Key: Traditional models may not be suitable for all scenarios. Being flexible and open to adopting new techniques can yield better results.

• Complexity vs. Gain: While adding complexity can improve model performance, it's crucial to weigh the benefits against the added complexity and computational resources needed.

• Multi-Disciplinary Approach: Collaboration between operations, finance, engineering, and machine learning teams is essential for implementing a robust forecasting system.

• Validation Challenges: In cases where traditional experimental designs are not feasible, alternative validation methods like backtests and limited A/B tests can be useful.

• Stakeholder Communication: Transparently communicating the benefits and limitations of a new approach can help in gaining stakeholder buy-in, which is crucial for successful implementation.

Source

HOW PAYPAL SCALES KAFKA TO TRILLIONS OF MESSAGES

PayPal, a giant in the fintech space with over 350 million customers, processes trillions of messages per day on their payments platform.

How exactly does PayPal handle all of these messages at such a high scale while also ensuring high availability, fault tolerance, and optimal performance?

PayPal at Scale

Since its introduction at PayPal in 2015, Kafka has scaled dramatically to meet the company's growing data needs.

Initially starting with a few isolated clusters, Kafka now powers a wide range of applications at PayPal, from first-party tracking to analytics, each handling over 100 billion messages per day.

The platform boasts over 1,500 brokers, 20,000 topics, and nearly 2,000 Mirror Maker nodes (mirror data among clusters), achieving a remarkable 99.99% availability.

During peak times like Black Friday in 2022, Kafka at PayPal handled a staggering 1.3 trillion messages in a single day.

Incoming Messages Per Second Over Holiday Season

The system continues to scale seamlessly, especially during high-traffic holiday seasons, without impacting business operations.

Total Messages Per Day Over Holiday Season

How Kafka is Used at PayPal

At a high-level, PayPal’s infrastructure consists of multiple geographically distributed data centers and security zones.

Kafka clusters are deployed across these zones depending on their data classification and business requirements.

Kafka Cluster Deployments in Security Zones Within a Data Center

Data is then mirrored across the data centers, helping with disaster recovery and zone communication.

But what libraries and components do these Kafka clusters interact with?

At PayPal, a wide array of clients are supported:

• Java

• Python

• Spark

• Node

• Golang

• Internal frameworks

Kafka Libraries and Components

With multiple frameworks, tech stacks, and applications that need to be supported, reducing and managing operational overhead is always a challenge.

Operational Challenges and Solutions

Over the years, PayPal has identified four key areas of improvement in order to maintain security, consistency, and platform stability:

• Cluster management

• Monitoring and alerting

• Configuration management

• Enhancements and automation

To improve cluster management, PayPal introduced a Kafka Config service to manage broker IPs and configurations, which cut down on maintenance from upgrades, patching, etc.

Furthermore, Access Control Lists (ACLs) were added to secure the platform for business-critical workflows.

To enhance security and streamline integration across various frameworks, PayPal has developed its own set of Kafka libraries.

The Resilient Client Library simplifies connectivity and configuration, improving developer efficiency and business resilience.

Meanwhile, the Monitoring Library enables real-time health checks and alerts, and the Security Library automates SSL authentication, significantly reducing operational overhead for managing certificates and keys.

Supported Tech Stack for Kafka Libraries

Wrapping Up

Kafka has been instrumental in supporting PayPal's massive data streaming needs, and the journey has been filled with challenges and learnings.

Through strategic investments in tooling, monitoring, and automation, PayPal has managed to scale Kafka efficiently while maintaining high availability and performance.

As the platform continues to evolve, PayPal remains committed to enhancing its Kafka infrastructure to provide a seamless and robust experience for its end-users.

Key Takeaways

• Tooling for Scalable Kafka Management: Custom tools and automation are indispensable for efficiently managing large-scale Kafka deployments. These tools help in reducing manual overhead and errors, making it easier to scale the system as the data traffic grows.

• Precision Monitoring for Availability: Fine-tuning the monitoring metrics and setting up a responsive alerting system are crucial for maintaining high availability. This ensures that any issues can be quickly identified and resolved, minimizing downtime and impact on mission-critical applications.

• Securing Access with ACLs: Implementing ACLs has proven to be effective in enhancing the security of Kafka clusters. ACLs also provide better control over which applications can access specific Kafka resources, thereby making application management more structured and secure.

• Benchmarking for Insight and Efficiency: Conducting performance benchmarks across different operational environments, such as on-premises and cloud, provides valuable insights. These insights help in making informed decisions about optimizing performance and cost-efficiency.

Source

BYTE-SIZED TOPICS

Node.js (Fastify) vs Go: Performance comparison for JWT verify and MySQL query

Find out which tech performs better for the use case of verifying JWT and reading record from MySQL: Node.js (Fastify) or Go?

medium.com/deno-the-complete-reference/node-js-fastify-vs-go-performance-comparison-for-jwt-verify-and-mysql-query-73c056f19dcc

Parallel Processing: How Modern Cloud Servers Leverage Different System Architectures To Optimize Parallel Compute - Scaleyourapp

Modern cloud servers leverage several system architectures to process data parallelly, which increases throughput, minimizes latency and optimizes resource consumption.

scaleyourapp.com/parallel-processing

Nvidia Says New Software Will Double LLM Inference Speed On H100 GPU | CRN

Nvidia said it plans to release open-source software that will significantly speed up inference performance for large language models powered by its GPUs, including the H100.

www.crn.com/news/components-peripherals/nvidia-says-new-software-will-double-llm-inference-speed-on-h100-gpu

INTERESTING PRODUCTS, TOOLS & PACKAGES

GitHub - modularml/mojo: The Mojo Programming Language

The Mojo Programming Language. Contribute to modularml/mojo development by creating an account on GitHub.

github.com/modularml/mojo

GitHub - oven-sh/awesome-bun: ⚡️ A curated list of awesome things related to Bun

⚡️ A curated list of awesome things related to Bun - GitHub - oven-sh/awesome-bun: ⚡️ A curated list of awesome things related to Bun

github.com/oven-sh/awesome-bun

GitHub - microsoft/promptflow: Build high-quality LLM apps - from prototyping, testing to production deployment and monitoring.

Build high-quality LLM apps - from prototyping, testing to production deployment and monitoring.

github.com/microsoft/promptflow

TIP OF THE WEEK

A lesser-known but useful Linux tip is the use of the !! (double exclamation mark) to repeat the last command you executed.

This is particularly handy when you forget to run a command with sudo.

For example, let's say you try to update the package list with apt:

apt update

You'll likely get a permission denied error because you didn't use sudo. Instead of retyping the whole command, you can simply type:

sudo !!

This will execute the last command (apt update) but with sudo prepended, effectively running sudo apt update.

MEME OF THE WEEK

Good morning and welcome back to another edition of Full Stack Express, your weekly newsletter packed with both technical and non-technical tips, the latest tech news, insightful articles, and real-world case studies.

Here’s the notable announcements for the week:

• The Node.js team releases version 20.6.0, adding built-in support for .env files.

• The Svelte team releases SvelteKit v1.24, adding support for view transitions (view transitions are in right now).

• The open source Git project releases Git 2.42, containing interesting features such as excluding references by pattern.

• Astro is launching Astro Studio in 2024, a globally-distributed edge data platform. Join the waitlist here.

We’ll also do deep dives into:

• Apple’s AI Breakthrough in Smarter Recommendations

• The Intricacies of how Slack Manages Billions of Requests Daily

• Why Developers Love Rust: The Rise of Tech’s Most Desired Language

• (Bonus) How LinkedIn Automates Application Security for a Safer User Experience

And finish off with:

• Byte-sized topics of trending tech news

• Interesting tools and packages created by the community

• Tip of the week (Kubernetes)

• Meme of the week

APPLE’S AI BREAKTHROUGH IN SMARTER RECOMMENDATIONS

Ever wondered how Netflix knows you'll love that new sci-fi series or how Amazon predicts you need a new coffee maker?

The secret sauce is often "Collaborative Filtering," a technique that analyzes your behavior to recommend items you'll likely enjoy.

But Apple is taking this to a whole new level with their groundbreaking model—Sliced Anti-symmetric Decomposition (SAD).

SAD: Sliced Anti-symmetric Decomposition for Collaborative Filtering

Traditional methods use something called "latent vectors" to understand both you and the items you interact with.

But SAD adds a twist—an extra latent vector for each item, making the algorithm smarter and more nuanced.

What makes SAD so special?

Apple's genius lies in a three-dimensional approach to user-item interactions.

Imagine it as a 3D chessboard where every move you make helps the algorithm understand not just what you like, but how you compare multiple items.

This results in more accurate and personalized recommendations.

So why exactly does Apple’s model stand out?

When the additional vector in SAD is set to 1, it's like any other state-of-the-art (SOTA) collaborative filtering model. But here's the kicker: Apple lets this value be driven by actual data.

This opens up a world of possibilities, suggesting that our decision-making might be more complex than previously thought.

SAD vs Traditional Collaborative Filterings

Ever find yourself stuck in a loop, comparing two or three items endlessly? SAD gets that, and it's built to understand these cycles.

The proof is in the numbers.

Apple didn't just make claims; they put SAD to the test.

Test Results for Different Parameters (e.g. Learning Rate)

With over a million user-item interactions from both simulated and real-world datasets, SAD didn't just meet the bar—it raised it.

Outperforming seven SOTA models, SAD proved to be a juggernaut in delivering consistent, highly accurate recommendations.

SAD outperforming other models in consistency and accuracy

With a whopping $22.6 billion invested in AI and machine learning, Apple may be quiet, but they're far from inactive.

Between their cutting-edge research and massive R&D investments, it's clear: Apple is gearing up to redefine the future of personalization.

So, the next time your device suggests something eerily perfect, you'll know that Apple's SAD algorithm is the wizard behind the curtain.

Find the full publication here.

Find the model and inference algorithms here.

Find the full in-depth analysis here.

THE INTRICACIES OF HOW SLACK MANAGES BILLIONS OF REQUESTS DAILY

How exactly does Slack juggle billions of network requests daily?

The answer starts with Slack's Edge Points of Presence (PoPs)—a global network of data centers designed to minimize latency and maximize performance.

These PoPs connect to Slack's main AWS region, where core services and storage reside.

The architecture is robust and built for resilience, ensuring a seamless Slack experience even if a data center goes down.

How Requests Flow Through Slack’s Infrastructure

When you interact with Slack, your requests fall into two main categories:

1. WebSocket Requests: The engine behind real-time messaging

2. Non-WebSocket Requests: All other functionalities, like file uploads and API calls

Slack uses Amazon Route53 and NS1 to manage domain name resolution, making sure your requests find their way through the network maze efficiently.

WebSocket Traffic for Slack Messages

WebSocket connections are the backbone of Slack's real-time messaging.

Managed by a system called "envoy-wss," these connections ensure that your messages are delivered almost instantly via Slack’s DNS domains.

How Slack Handles WebSocket Traffic

These DNS records are managed by NS1 through a series of filters called Filter Chains.

This sequence directs your messages to the closest Network Load Balancers (NLBs), optimizing for speed and reliability.

Filter Chains

The request is finally directed to either Gatewayserver or Applink based on its parameters.

Gatewayserver is an internal service that maintains WebSocket connections at Slack.

Applink is an internal service designed to lighten the burden of security for customers when building apps that need to receive data.

Slack’s API and Integration Ecosystem

Slack isn't just about messaging. It's also a hub for:

• Slack API

• Slackbot

• Webhooks

• Third-party applications

These types of traffic are managed by a dedicated set of servers known as "envoy-edge," fine-tuned for handling heavy, API traffic.

How Slack Handles API Traffic

Internally, a host of specialized internal services work together to manage different aspects of Slack's traffic:

• Flannel: An edge cache that holds essential user and channel information.

• Imgproxy: Manages image content within Slack.

• Supra and Miata: Handle your file uploads and downloads.

After navigating this intricate network, your requests finally land on Slack's WebApp, where you see your messages, files, and integrations.

So, the next time you send a message or share a file on Slack, you'll have a newfound appreciation for the complex but highly efficient system running in the background.

Find the full in-depth analysis here.

WHY DEVELOPERS LOVE RUST: THE RISE OF TECH’S MOST DESIRED LANGUAGE

For an impressive eight consecutive years, Rust has claimed the title of "Most Desired Programming Language" in Stack Overflow's yearly survey of developers.

With over 80% of developers eager to continue using it next year, it's remarkable how this language, founded less than two decades ago, has captivated the global developer community.

In this feature, we'll explore Rust's intriguing history, its typical applications, and the reasons behind its skyrocketing popularity.

The Rust Origin Story

Born out of a need for a safer alternative to C and C++, Rust has rapidly ascended to become a cornerstone in systems programming.

Initially developed by Graydon Hoare in 2006, and later backed by Mozilla in 2009, Rust was a game-changer with its stable 1.0 release in 2015.

Did you know?

The spark for Rust actually came from a malfunctioning elevator experience, inspiring Hoare to create a language that could eliminate common memory issues.

The language prioritizes safety, performance, and productivity, employing static typing to ensure memory safety and reduce errors.

Fast forward to 2023, Rust is the go-to language for about 2.8 million developers, known as "Rustaceans," and has even been embraced by Big Tech companies like Microsoft for its memory-safe capabilities.

The language even has an unofficial mascot, Ferris, adding a whimsical touch to its growing popularity.

Ferris, the Unofficial Mascot for Rust

What Makes Rust Unique

Rust stands out for its unique blend of speed and reliability, addressing developers' memory management challenges often seen in C and C++.

Its ownership system and borrowing model offer built-in support for concurrent programming, eliminating data races and memory issues.

Furthermore, unlike languages that use garbage collection, Rust relies on its ownership rules for efficient memory management, giving developers precise control over resource allocation.

Its package manager, Cargo, not only simplifies project management but also boasts a rich ecosystem of performance-critical libraries.

Rust was the first systems language with a standard package manager, enhancing its robustness.

Other key features include:

• Zero-cost abstractions, allowing high-level code without performance loss

• Pattern matching for clean data structure handling

• Advanced type inference, which automates expression detection based on context.

These combined features make Rust a uniquely powerful and efficient programming language.

Where Rust Excels

Rust's adaptability makes it a top choice for a myriad of applications:

• Backend Systems: Its performance and thread safety are invaluable for building performance-critical backend systems, as seen in GitHub's code search feature.

• Operating Systems: Rust is a preferred language for OS development, powering systems like Redox OS and Google's Fuchsia.

• Web Development: With async programming and high performance, Rust is making waves in server-side applications, supported by frameworks like Rocket.

• Emerging Technologies: Rust is carving a niche in cryptocurrency and blockchain technologies, and its minimal runtime makes it ideal for embedded systems and IoT development.

In summary, Rust has captivated developers with its focus on safety, performance, and versatility, earning it a place of honor in the programming world.

Let’s end with something for the Rust aficionados:

fn main() {
    break rust;
}

Find the full in-depth analysis here.

HOW LINKEDIN AUTOMATES APPLICATION SECURITY FOR A SAFER USER EXPERIENCE

LinkedIn's Information Security team is committed to creating a secure environment for its users.

At its core is the Application Security team, who manage key security headers like Content Security Policy (CSP) to fortify web applications.

Their mission is to automate security protocols, boost developer productivity, and reduce manual oversight.

They’ve experimented with two architectural models to achieve this:

1. Centralized, Rule-Heavy Approach: This model scrutinizes outgoing HTTP responses against a set of stringent rules, ensuring they meet the gold standard of security.

2. Decentralized, Developer-Empowered Model: This approach offers developers the latitude to innovate while maintaining a robust security framework.

Both models deploy rules to LinkedIn's proxy hosts, monitored by specialized plugins. This dual strategy not only fortifies security but also streamlines the development process.

Analyzing the Centralized Approach

Initially, LinkedIn employed a centralized approach, using the Traffic Headers Plugin to write Content Security Policy (CSP) headers into responses.

While this offered benefits like easier maintenance and uniform secure policies, it posed significant challenges as LinkedIn scaled.

The centralized system became increasingly unwieldy, placing the Application Security team in a bottleneck position for approving changes, thereby increasing the risk of human error and site issues.

LinkedIn’s Centralized Architecture Model

Recognizing these limitations, LinkedIn pivoted to a decentralized approach, aligned with the latest CSP standards. This new architecture aimed to achieve five key objectives:

• Minimize the impact of changes

• Empower developers

• Provide a consistent testing experience

• Enhance security posture

• Build a generalized solution beyond just CSP headers

While the old centralized architecture remains for legacy applications, the focus has shifted towards migrating to a more agile and scalable decentralized system.

Migrating to a New Decentralized System

In this new architecture, a CSP Filter became an integral part of the frontend frameworks.

This filter decorates the outgoing response with CSP headers defined by developers, maintaining a safety net with a fallback mechanism at the traffic layer.

LinkedIn’s Decentralized Architecture Model

This approach reduced the scope and impact of CSP changes, empowering developers to implement and test changes efficiently.

However, the decentralized model posed challenges in security governance, including reduced visibility for the Application Security team and the complexity of CSP modifications.

Implementing Risk-Based Validation Rules

To address these challenges, LinkedIn adopted a "shift-left" approach, implementing security validators that analyze code at the time of commit using risk-based rules.

Validation checks on GitHub guide developers on appropriate actions, such as blocking pull requests that violate key risk rules.

Example of LinkedIn’s Critical Rules

LinkedIn's balanced approach aims to combine the rigor of a centralized system with the flexibility of a decentralized model, making it a safer and more efficient platform for both users and developers.

In summary, LinkedIn's dual approach to application security serves as a compelling case study in balancing security rigor with developmental agility.

Find the full in-depth analysis here.

BYTE-SIZED TOPICS

The Engineer’s Guide to Career Growth — Advice from My Time at Stripe and Facebook

Raylene Yung has spent a decade scaling eng and product teams at Facebook and Stripe. Here's her advice for engineers at every stage of their careers, from IC to org leader.

review.firstround.com/the-engineers-guide-to-career-growth-advice-from-my-time-at-stripe-and-facebook

OpenAI Plugs ChatGPT Into Canva to Sharpen Its Competitive Edge in AI - Decrypt

Now ChatGPT Plus users can "talk" to Canva directly from OpenAI's bot, making their graphic design workflow easier.

decrypt.co/154923/chatgpt-canva-plugin-openai-design-tool

Robinhood Wallet Adds Support for Bitcoin and Dogecoin, and Enables Ethereum Swaps - Robinhood Newsroom

We have launched custody, send, and receive support for Bitcoin and Dogecoin to all users, and are enabling in-app swaps on the Ethereum network.

newsroom.aboutrobinhood.com/robinhood-wallet-adds-support-for-bitcoin-and-dogecoin-and-enables-ethereum-swaps

Okra raises $12M as it brings solar power to grids in developing areas | TechCrunch

Today Okra Solar is announcing a $12 million Series A aimed at serving wide swaths of the world that presently lack access to electricity.

techcrunch.com/2023/09/05/okra-raises-12m-as-it-brings-solar-power-to-grids-in-developing-areas

INTERESTING PRODUCTS, TOOLS & PACKAGES

Marketsy.ai, Created by Igor Boky

Looking to start an online store or marketplace? Marketsy.ai can whip up a complete e-commerce solution in just a few seconds via a single prompt. Say goodbye to the hassle of building your own website - let AI do the heavy lifting for you.

marketsy.ai

GitHub - lgrammel/modelfusion: Build multimodal AI applications, chatbots, and agents with JavaScript and TypeScript.

Build multimodal AI applications, chatbots, and agents with JavaScript and TypeScript. - GitHub - lgrammel/modelfusion: Build multimodal AI applications, chatbots, and agents with JavaScript and Ty...

github.com/lgrammel/modelfusion

GitHub - measuredco/puck: The self-hosted drag and drop editor for React.

The self-hosted drag and drop editor for React. Contribute to measuredco/puck development by creating an account on GitHub.

github.com/measuredco/puck

TIP OF THE WEEK

When it comes to deploying workloads on Kubernetes, defining resource requests and limits for CPU and memory is not just good practice—it's a necessity.

Take a look at this sample pod configuration:

apiVersion: v1
kind: Pod
metadata:
  name: my-application
spec:
  containers:
  - name: my-container
    image: my-image
    resources:
      requests:
        memory: "64Mi"
        cpu: "250m"
      limits:
        memory: "128Mi"
        cpu: "500m"

Here's four reasons why:

• Predictable Resource Allocation: Think of resource requests as RSVPing a table at a restaurant for your pod. It ensures your pod gets the resources it needs to run smoothly.

• Avoid Resource Starvation: Setting limits is your node's bouncer, keeping your pod from hogging all the resources and starving others. It's your defense against the "noisy neighbor."

• Improve Stability: Without limits, a rogue pod could go on a resource spree, crashing other pods or even the node. It's your speed limit for safe pod driving.

• Efficiency: Fine-tuning these values is like optimizing your car for fuel efficiency. You get peak performance without waste.

MEME OF THE WEEK