Prerequisites

• Our lab on deploying a commercial CSPM (FireMon Cloud Defense) and your FireMon credentials.

The Lesson

If you’ve been following along with these labs, you might remember we covered the different categories of cloud threat detectors. We then followed with labs on threat detection using logs and events, but I only threatened (haha) to cover detection using configuration.

<batman voice> I don’t make threats; I make promises. </batman voice>

Yeah, alright, maybe I’m overdoing the drama. Anyway…

Although this lab is focused on threat detection, I actually want to cover two topics.

Managing CSPM Findings

CSPMs and Cloud Native Application Protection Platforms (CNAPP, which adds workload and other tools to CSPM) aren’t really traditional vulnerability scanners because they look for misconfigurations, but they do share one trait. They tend to find… a lot… of things. Like, a LOT. But like a vulnerability scanner, not all of them are of equal value, and not all are relevant in the context of your given deployment.

For example most tools look for default NACLs, which we didn’t cover because I never use them. This finding will appear once for every subnet you have. It isn’t a misconfiguration, it’s a reasonable choice, but all the tools look for it since someone slapped it into some industry benchmark.

Looking at my current CloudSLAW org scan with Cloud Defense, I have 138 critical and high misconfigurations.

That number is right, but in a non-useful way. I probably only care about a handful of those. Having helped build a CSPM, the problem as the vendor/maintainer is that you need to be comprehensive, but you will never have enough context. We saw this in our Prowler scan as well.

Okay, so how do I prioritize? I always start by looking for the Big Gaping Security Holes, then correlating findings with my high-value accounts:

• Static credentials (IAM users)

• Users/roles with full admin credentials

• Anything public

I work my way down the list eventually, but at the start if you keep a laser focus on the main ways you get exploited (public and IAM stuff) it really helps you filter out the noise. I also override severities to meet my own preferences, with tools which support this.

I suggest you poke around your own findings. We don’t really have time for depth in this lab, but here’s the challenge which I’ll walk through it quickly in a sec when we log in:

• Identify and remediate any security group with port 22 exposed to the Internet. That’s a BGSH, so see how fast you can find and fix it.

CSPM for Threat Detection

I think we all agree that big holes are bad, but what does that have to do with threat detection? Well, the evidence from multiple cloud providers and incident response teams consistently shows that the main vector for attacking cloud is to steal credentials and use them to create misconfigurations which expose data or allow them to hijack resources.

For example, if an attacker steals an access key/secret key which allows them to change a security group, they can open up port 22 and start attacking an instance directly. Or they’ll make something public, or share it with an account under their control. There are… a lot of options. Thus…

A misconfiguration may be your first and best indicator of attack!

An attacker in the management plane will change things in ways that probably don’t look right. The faster you can detect and respond, the better. They aren’t just brute forcing virtual machines, they are stealing credentials using infostealer malware and then using those credentials to change configurations to steal or damage you.

Imagine you detect a public S3 bucket full of customer data. Is that a mistake? Someone violating policy? An attacker using an employee’s credentials to steal data?

Yeah, GuardDuty will probably detect my super obvious example, but the list is much longer (actually infinite). Like taking a snapshot of an instance with customer data and sharing it with an account under the attacker’s control.

Any CSPM can help augment threat detection, but I prefer ones with two capabilities: real-time detection, and capture of management plane API calls. Depending on your tool of choice, it might support real-time but that may not be the default deployment option, so just be aware it isn’t supported by all tools and might require manual enablement.

Also, not every CSPM’s findings are good for threat detection. We don’t have time to get into all the options today, especially since they also change based on your particular environment.

Key Lesson Points

• When getting started with CSPM, focus on the Big Gaping Security Holes to reduce overload.

• CSPM is an excellent source for threat detection, since it identifies the outcomes (misconfigurations) of activities attackers use to compromise cloud.

The Lab

First I’ll walk through finding a BGSH (Big Gaping Security Hole) and then we’ll create an exposure and investigate it in real time.

Video Walkthrough

Step-by-Step

Open up 2 tabs, and then log into your Sign-in portal and https://defense.prod2.firemon.cloud.

We’ll start in Cloud Defense and find one of those BGSHs. This time, a security group with port 22 exposed to the Internet. Start in Posture Monitoring > Checks > Security Group Allows Unrestricted Access to Ports with High Risk:

I like the Checks view for identifying systemic issues, since it quickly shows you which posture checks have the most findings. In this case that sliver of red is for failing security groups. See a lot of red in the bar? That’s one where you consistently fail.

Scroll down and drill down to see the exact exposed security group. I picked the one in Oregon. With the account ID and region we can quickly access the offending account for remediation:

I’m jumping through quickly, but you should be able to follow along.

From there it’s easy to go into that account and region. Here’s a hint: it will be your TestAccount1 account and Oregon. Delete that rule from the security group in the AWS console.

Within a few seconds it will show is Pass in Cloud Defense, but the failure will still show up in the history.

Yeah, real-time is pretty cool.

Okay, on to threat detection! 

First we need to set up email alerts. Click the user icon in the upper right corner > User Settings:

You’ll need to set a day and time for scheduled alerts, but don’t worry, we won’t enable them. Instead just check Send CSPM alert emails > Save:

Now go to Posture Monitoring > Alerts > + Alert:

Let’s build an alert for anytime a snapshot becomes public.

We only need to change 2 settings to alert on a specific finding. Set results to “Fail”, and Checks to “EBS Snapshot is Publicly Accessible”:

Now let’s trigger it. Sign-in portal > TestAccount1 > Oregon region > EC2 > Snapshots > check the one labeled CSI-incident-1 if you have it. If not? Any snapshot will work. Don’t have one? Go to Volumes, pick one, and create a snapshot. Don’t have a volume? Launch an instance (any instance) and then snapshot. Then Modify permissions: 

Then check Public > Modify permissions:

WHY DID YOU HIT YOURSELF IN THE FACE???

Okay, with that exposure in place, you probably received an alert (maybe 2, depending on Security Hub). Let’s go back into Cloud Defense > Posture Monitoring >Findings to start investigation. (Or click the link in the email notification).

You’ll see the EBS snapshot finding right at the top. Click it:

A lot of the information you want is right on this first page. When it happened, who caused it, their IP address, and the API call that caused the misconfiguration.

With the tabs on this screen you can investigate most of the incident using my RECIPE PICKS mnemonic for incident response. In my case I hit a wall when I looked at the related resources — it identified the volume the snapshot came from, but I deleted that volume a long time ago.

Go ahead and click around — you can see all the consolidated API calls referring to that resource and all the calls from the IAM role which triggered the event, and review the configurations of all the items in inventory.

This example was very simple. A single API call created a clear misconfiguration. But there are many other attacks which are more complex and involve multiple API calls. The advantage of using your CSPM for threat detection is it can alert on the misconfigurations which correlate best with attacker activity. It’s more an Indicator of Compromise than an Indicator of Attack. What’s the difference? An IoC is a sign the attacker is already in and did something bad. An IoA will fire off during an attack, including (sometimes) attacks which were stopped.

To clean out Cloud Defense

You can keep it for at least 60 days. Even if it goes past 60 days, you won’t be charged. To remove it from your organization just go into your organization management account and delete the stack (not the StackSet — deleting the stack will take care of that).

-Rich

Prerequisites

• An AWS Organization. This one works even if you don’t have our standard lab structure.

The Lesson

At the start of our series on Cloud Security Posture Management I mentioned that we would explore the three main options, each of which you will encounter at some point if your job touches cloud security:

• Open Source. We used Prowler, which also offers a commercial version.

• Cloud Service Provider (CSP) native. We used AWS Security Hub (which is in transition as I write these labs).

• Commercial. Which we will cover today, using the Cloud Defense product from FireMon.

Note that this barely scratches the surface — there are many open source and commercial tools on the market (heck, there are currently 2 Security Hubs!). But by the time we are finished you will have a very good sense of different deployment options, capabilities, and user experiences.

There are two main learning objectives for this lab:

• See how to deploy a third-party, SaaS-based security tool connected to your AWS Organization.

• Understand a real-time detection architecture. (Similar to what we deployed for our S3 lab series, but at larger scale).

• Get a big-picture understanding of CSPM capabilities.

We will follow up with one more lab which goes into more detail on using a CSPM to detect and investigate misconfiguration.

Disclaimers and Disclosures

I debated a fair bit on using commercial tools in any of these labs, or even an Open Source tool (with commercial options) like Prowler. But you’ll encounter these tools in the wild, and I can’t teach certain skills using only what AWS provides. Besides, that would also be problematically biased because, in many cases, a third-party tool will be the better choice. With that out of the way…

• Today we will use FireMon Cloud Defense, a commercial CSPM.

• I founded the startup that was acquired by FireMon and became Cloud Defense. I worked there full-time for 3 years and still have an ownership stake, so if anyone ever buys FireMon I can stop paying for beer with stock certificates (VC/tech stuff is weird).

• FireMon already had a free tier, but they stood up a special, isolated version for these labs. It’s a complex architecture but all Infrastructure as Code, so they literally just copied and pasted into a new AWS account(s).

• FireMon created a special CloudSLAW deployment process which automates nearly everything in a single CloudFormation stack run from the management account. It even auto-registers using your management account email address.

• Everything is free for 60 days, then it disables itself. You will never be billed. They wouldn’t know how to bill you even if they wanted to.

Having our own little sandbox for these labs is pretty great, and I want to thank Brandy Peterson (one of my co-founders) for making it all possible.

Why Every CSPMs Has an Inventory

Let’s think about everything we’ve learned about assessing configurations, in the context of preventing attacks.

You’ve already seen multiple ways of scanning for misconfigurations. From the timed serverless scans we built ourselves, to running code in an instance like Open Source Prowler, to manually checking things. The problems with all of them are time, scale, and the overhead of actually looking for misconfigurations. And that’s before we get into the problem of API service limits (the number of API calls you can make in a period of time).

This is why every single major CSPM platform uses its own inventory. Instead of making a lot of costly API calls to run different assessments on the same (or different) resources, the tools:

• Sweep through accounts and run the API calls to gather all the configuration information on what’s running.

• Store it in a database.

• Run assessments against that database, which is more flexible and efficient.

• Keep the database updated.

Think about the problem: if I want to run 20 different kinds of assessments on an Instance, that would be a lot of repeated API calls. And sometimes I want to know things like “is this instance in this security group?” which is also a pain to run on its own. But if I have all that data in a database, those are queries rather than API calls, and a lot more efficient. Having an inventory gives you a better picture without having to poke the cloud provider every time.

The problem is that you still need to collect that information, and keep it up to date. Different tools handle that on their own schedules — ranging from every few minutes, to every hour, to once a day. They still need to make those DESCRIBE API calls to get the updated into.

A few platforms, including Cloud Defense, update their inventory in real time. It’s kind of a cool approach (I didn’t code it, but I did work on the initial design). It is similar to what we previously did in our real-time autoremediation lab.

Here’s how it works:

• Capture all the CloudTrail calls via EventBridge.

• Filter those calls for any “change” events (create, update, delete).

• For each event, extract the resource identifier (e.g., the instance ID).

• This triggers a lambda which reads the configuration of only that resource.

• This updated configuration is stored in the database.

• That change event triggers all the assessments associated with that resource type.

This is weirdly more efficient that running timed scans, since it only requests updated information. However, since things can slip through the cracks, any platform which uses this model should still run a daily scan (and with Cloud Defense, you’ll see it also run an initial scan when it first onboards a new account).

Go reread our old labs, because we already implemented simplified versions of this. I mean, I originally came up with the idea, so I need to milk it for all it’s worth.

Automating Deployment

There’s kind of a cool thing going on with the deployment that you’ll see in a minute, and it’s worth explaining. Actually, two cool things:

• We only deploy a single CloudFormation Stack in a single account, but somehow it deploys the tool into all our accounts. What happens is the Stack creates the StackSet, which then deploys the required resources (IAM role and EventBridge stuff) into the rest of the organization. It’s more complicated to build, but done well can be easier and more reliable than messing with StackSets directly.

• The stack auto-registers with the Cloud Defense platform and creates your account there. This uses Lambda functions, and you can read all the code in the template. For our labs FireMon created a special version which pulls our management account email and auto-registers using that as your username (I requested this to save us steps).

These are some common techniques used in more-advanced infrastructure as code, and if you read through the template you might notice some of the other things we’ve learned in these labs (like unique external IDs).

The Lab

This lab is mostly setup and exploration — we will follow up with the next lab, where we’ll run through the core capabilities of CSPM.

The setup is kind of cool, especially that they been it tuned for CloudSLAW, even though it’s a commercial tool. And as a reminder this is running in a separate version of Cloud Defense that FireMon stood up just for these labs, isolated from their current production environment.

Video Walkthrough

Step-by-Step

Start in your Sign-in portal > CloudSLAW > AdministratorAccess > (double check you are in the Oregon region) > Organizations > copy out your root OU ID (r-xxxx):

CloudFormation > Create stack > With new resources:

Use the following:

• Name: CloudDefense

• S3 URL: https://cloudslaw.s3-us-west-2.amazonaws.com/lab68.template

The ONLY parameter you should change is to paste in your org root OU ID (which starts with ‘r-’, not ‘o-’):

Don’t change anything else, run through the rest of the screens, and Submit.

This is a complicated stack which:

• Creates some resources in your management account for onboarding that account with Cloud Defense. Remember, normally a StackSet won’t deploy into the management account! Using a normal stack gets us around that.

• It creates a Lambda function to collect the email of the management account, and then use that to create a new customer account with Cloud Defense. This was the magic little bit I suggested to FireMon to make the lab easier. This feature is not part of the process as a paying customer, where you might want things configured differently.

• It creates StackSets in Virginia and Oregon and pushes them into every account. Technically we could have deployed to just certain OUs (this was the parameter where we used the root OU instead).

That’s the magic bit special for this CloudSLAW lab — auto-registration. Outside this lab you need to create an account with your SaaS provider first.

I recommend taking a look at this stack, and then the StackSets it deploys. You’ll see an external ID, as we discussed in our Prowler lab, and some other more-advanced techniques like integrating Lambda registration functions (which we previously used) and nested stacks (common in larger deployments).

As this stack is running, go check your management account email! You’ll have an account registration message from FireMon (check your spamtrap if you don’t see it within 5 minutes). Click Activate My Account:

I’m planning at least one more lab on this platform, so set a password you won’t lose. As a reminder, this account will be shut down in 60 days, and I’ll show you how to remove the stack in the next lab:

IF YOU RUN INTO ANY ISSUES CONTACT ME DIRECTLY AT RMOGULL@SECUROSIS.COM. Since this is free I’ll be handling support — not FireMon.

Once you’re logged in, click the little side arrow to expand the left menu and start exploring:

Your CloudFormation stack is probably still running and provisioning all your accounts (FireMon set a limit of 10 accounts, which is plenty for us… their commercial version supports thousands). It takes time for Cloud Defense to do its first sweep of all your accounts to fill in the Inventory. But you can click it to watch it work.

Assessments are all performed instantly as each resource lands in the Inventory, so you’ll see those results filling in as well — the entire platform is serverless and event-driven:

This is a fully functional version of the platform. The only limit is the 10 accounts, and it has some internal governors running to keep costs down, so it may be a little slower if hundreds of you sign up all at once (or maybe not — nobody’s ever tested this lab version).

Feel free to poke around and explore the capabilities. In the next lab we’ll learn how to filter results and create exemptions, then investigate a misconfiguration. These are core skills you’ll need no matter which platform you end up using.

-Rich

Prerequisites

• The target environment configured in our second Prowler lab.

• Our first Security Hub lab, which delegated administration to our SecurityAudit account.

The Lesson

Okay, this is the weirdest lab I’ve built yet. Why? Because I have absolutely no idea how long it will last or what things will look like when you log in. Sure, I try to keep these things updated, but I’m JUST ONE PERSON, and some things change darn quickly in the cloud.

Here’s the TL;DR: Security Hub is Amazon’s built-in security center, and it included CSPM capabilities. But AWS recently split out the non-CSPM parts, named them all Security Hub, renamed the old Security Hub “Security Hub CSPM”, and has said more changes are coming since the new version is only in public preview, so… YOLO!

The Curious Case of Two Security Hubs

Security Hub CSPM is what you're probably thinking of when someone says "Security Hub," if you laid hands on AWS before this change. It runs all those security standards checks, compliance frameworks, and best practice assessments. This is the part that competes with tools like Prowler, which we covered in the last lab. Security Hub CSPM generates findings about misconfigurations, compliance violations, and security gaps.

The "new" Security Hub (which is still in public preview as I write this, but will probably reach general availability soon) is actually a meta-layer that sits on top. Think of it as a security data aggregator and correlation engine. It takes findings from Security Hub CSPM, but also pulls in alerts and findings from GuardDuty, Inspector, Macie, and other AWS security services. Then it correlates all that data, reduces noise, and helps you prioritize what actually matters.

I’ll let AWS marketing do the writing for me: “Security Hub is a unified cloud security solution that prioritizes your critical security issues and helps you respond at scale. Security Hub detects security issues by automatically correlating and enriching security signals from multiple sources, such as posture management, vulnerability management (Amazon Inspector), sensitive data (Amazon Macie), and threat detection (Amazon GuardDuty). … Through intuitive visualizations, Security Hub transforms complex security signals into actionable insights…”

Here's the flow: Security Hub CSPM checks your environment and generates findings → those findings feed into Security Hub → Security Hub correlates them with other security data (GuardDuty, Access Analyzer, Inspector, etc.) → you get a prioritized, deduplicated view of what needs attention.

Now I know you are asking yourself, “didn’t Security Hub always aggregate findings from other services?” Yep. But … it didn’t really do any fancy correlation, so Security Hub wasn’t very competitive with other Cloud Service Provider (CSP) native security tools — like Defender for Cloud and the Google Security Command Center.

Now, about those costs I mentioned. Security Hub CSPM relies heavily on AWS Config to do its job. Config is what actually tracks the state of your resources and detects changes. Some checks happen in real time when resources change, while others run on a schedule. The problem? Config ain’t cheap, so we can’t (yet) just turn on all of Security Hub + Security Hub CSPM without keeping an eye on our bill.

New CSPM, Who Dis?

Although I expect Security Hub to change significantly as it reaches general release and continues to evolve, for educational purposes this new version offers more baseline CSPM capabilities we can use to expand our knowledge:

• Inventory: While Security Hub doesn’t offer a full and complete cross-account inventory, it does aggregate its inventory for covered resources — the big ones like EC2 instances and Lambda functions.

• Findings: Similar to our Prowler findings, these cover CSPM results (misconfigurations), but also any findings from other tools, such as a vulnerability detected by Inspector (a vulnerability scanner which we haven’t used yet). This starts sounding more like a CNAPP (Cloud Native Application Protection Platform): it can combine management plane misconfigurations with vulnerability info.

• Correlation: Combine different sources so I mostly only need to look in one place to figure out what’s going on. This ties tightly into the Inventory (resource) view so I can see misconfigurations, GuardDuty alerts, and more without bouncing into 5 tools.

• Exposure and Attack Graph: Exposure findings are higher-priority issues along the lines of, “Oh crap, that is public.” Is something misconfigured, reachable, and/or vulnerable? The attack graph then shows you visually how it might be exploited. It’s eye candy, but who doesn’t love candy?

• Automation: Mostly this is “create a ticket somewhere for someone to fix this thing” or “write stupidly complex EventBridge-Lambda-based autoremediations like that insane dude who runs the coleslaw newsletter.”

All of these capabilities (and more) are available in major third-party CSPM/CNAPP tools. They tend to have much more robust capabilities in terms of what they look for, compliance and security standards they support, report generation, filtering and organization of findings, etc. But it’s nice to see AWS building more of this natively, and it works well for us to take our learning to the next level.

Remember, my advice is to turn on your CSPM on day one. We only waited this long to cover the tools in labs because I wanted to give you a solid baseline of security knowledge first.

Key Lesson Points

• Modern CSPM tools (many of which are full CNAPP tools) do more than just find basic misconfigurations. They correlate a wide range of security telemetry to help identify and remediate issues.

• Security Hub is Amazon’s latest security center, with core CNAPP capabilities to aggregate findings from multiple sources, including workload security scans.

• Security Hub CSPM is the “old” Security Hub, and is now the feature that only handles management plane misconfigurations and compliance. It feeds its results into the new Security Hub.

The Lab

As I warned, this lab is a bit… rough around the edges. Aside from the AWS issues, I also managed to make a change in my account that I failed to record, and can’t go back and reset. We will:

• Update delegated administration for Security Hub

• Enable Security Hub

• Enable a Security Standard in Security Hub CSPM

• Enable Config

• Regret our life decisions

Video Walkthrough

Step by Step

First we need to update our delegated administration policy. THIS MAY BE DIFFERENT depending on when you finished our first Security Hub lab (since you might already be on the new policy). Start in your Sign-in portal > CloudSLAW > AdministratorAccess > Security Hub (remember, NOT Security Hub CSPM — and don’t blame me!). Then click Get Started:

This is where things may get weird. You may need to update your delegated administrator policy, or you might be fine and just need to Enable Security Hub for my account. This is due to the transition. You are a pro now, so figure it out. Then Configure. Here was mine:

Now we need to get Config up and running, since that what Security Hub CSPM currently relies on to get its… configuration information. Since we are already in our management account we can do this with CloudFormation > StackSets > Create stack:

You’ve done this a lot, so here are the settings to change as you walk through the screens:

• Create stack set

• Use the template: https://cloudslaw.s3-us-west-2.amazonaws.com/lab67.template

• Name: “Config”

• Deploy new stacks

• Deploy to organization

• Specify regions us-west-2 and us-east-1

• Max concurrent accounts: 10

• Failure tolerance: 9

• Submit and wait until it’s Succeeded

• Exit the management account (close the tab)

Okay, now let’s go to Sign in portal > SecurityAudit > Administrator Access > Security Hub > Getting started (yes, again) > Enable Security Hub:

You can see the four “feeder” services that fuel Security Hub here. Including… Security Hub (CSPM). We already have GuardDuty running, and ‘sorta’ Security Hub CSPM. Depending on the state of this lab, you likely enabled it but without any Security Standards enabled (so it isn’t looking for much) since we didn’t want to run Config. Skip past this for now and Go to Security Hub.

Next we want to set Security Hub to aggregate all findings to our current region. You’ll need to go to Settings > General > Cross-Region aggregation > US West (Oregon) and then also select US East Virginia, then Save:

Right now this only has Security Hub running in this one account. We need to enable it for our entire organization so go to Configurations > Policies > Create policy:

• Name: Default

• Enable all regions

• Scroll to the bottom and click Next, then Create:

Even without Security Hub CSPM fully enabled, findings will start showing up. Most of them are warnings related to not having all the AWS services turned on.

You can already see the problem with CSPM. As important and powerful as it is, basically every tool generates a ton of findings that may or may not matter. In fact, this is why the new Security Hub exists! As you’ll see in a second; while it takes all these feeds it tries to aggregate, correlate, and prioritize.

How well does it work? I don’t know for sure yet, since I’m very early in my testing.

Let’s finish turning on posture checks so we get more information and can compare it more directly to other tools, like Prowler. Go to Detection engines > Security Hub CSPM, which will pop open another window.

To enable CSPM for our entire organization we need to use a Policy. Go to Configuration > Policies > check the default one already there > Edit (If you don’t see one, go with Create and use the same settings):

Enable AWS Foundational Security Best Practices:

Then Enable all controls and un-check Customize control parameters:

Then Next > Save policy and apply:

This will take a bit to roll out through your accounts, but it is updating everything to start collecting EVEN MORE CSPM FINDINGS BWAHAHAHAHA!!!!

It takes hours for findings to show up. I left mine running overnight and it found… a lot. Too much, since most of them I don’t care about.

Yes, I do need to know that for compliance, but not always for security. Or maybe it’s only a security issue sometimes. This is the challenge of CSPM, and why we waited.

The new Security Hub attempts to address this by offering some more-focused finding types such as Threats and Exposures. None of these triggered in my account even after running it overnight, but our organization should be in pretty good shape, even though we opened a few things up.

At this point, feel free to click around, look at Resources and Findings, and maybe even see if you can create an exposure finding and see the attack graph. I… ran out of time to write this lab up. :)

It may take a bit of time for these to show up.

I’d leave it running for a day or overnight, then delete:

Deletion instructions

• In SecurityAudit

  • Go to Security Hub CSPM

    • Configuration > Policies > check your policy > Edit > delete (click the X) for the Security Best Practices

    • Save and apply

  • Go to Security Hub

    • Configurations > Policies > Default > Disable all regions

      • This actually prevents Security Hub from being turned on manually anywhere — remember that for the future.

• Go to Sign-in portal > CloudSLAW > AdministratorAccess > Organizations and copy your root OU ID (it starts with “r-xxxx”)

  • CloudFormation > StackSets > Config > Delete stacks from stack set

    • Enter the root OU

    • Add all regions

    • Max concurrent accounts: 10

    • Failure tolerance: 9

    • Submit

  • Then Actions > Delete stack set

That’s it! All Security Hub versions and Config are now disabled.

Lab Key Points

• Security Hub is the new security center for AWS and it aggregates, correlates, and prioritizes security information.

• Security Hub CSPM is the old version of Security Hub, which now “feeds” posture findings into the new Security Hub.

• Our labs all used delegated administration, which we previously configured and had to update.

• These services are not free.

-Rich

Prerequisites

• Our first Prowler lab.

The Lesson

In our last lab we set up our environment to run the Open Source version of Prowler from our SecurityAudit account. We configured it to use an External ID to prevent the confused deputy attack when it assumes our cross account role to run scans.

Now it’s time to learn a little more about CSPM, it’s big brother CNAPP, and why these tools are so important that they are one of the first things I install when I set up a new (non-lab) AWS organization.

CSPM Capabilities

I’ll be honest, I kind of hate the name “Cloud Security Posture Management”. Like, who cares if my cloud is slouching in its chair? But I also have to admit it makes sense. We can’t call these cloud vulnerability scanners because they aren’t necessarily vulnerabilities, they are just things configured in a way that maybe in the right context could be useful to an attacker. These aren’t really vulnerabilities in the traditional sense because they are allowed configurations, just ones that could be problematic. We need public S3 buckets, but a public bucket might also be bad if we didn’t actually want that particular bucket public.

When I first started in cloud we had to figure all this out ourselves by hand. I quickly moved from doing everything in the console to building my own tools. Then I found Prowler (which we installed last week) and swapped over. Tony kept Prowler up to date with great coverage. It was originally a really complicated bash script that used the AWS CLI!

Obviously everyone had this problem, and thus vendors were born (including mine, although we had a bit of a different focus). Over time these tools consolidated to a core feature set and eventually Gartner caught up and branded the category “CSPM”. At their simplest, modern CSPM tools:

• Connect to a cloud account and read the configuration of the services

• Identify misconfigurations (based on their own list and public standards)

• Rate the misconfiguration severity

• Generate reports

Practically speaking, modern tools all have additional features:

• They run centrally or as a SaaS platform and can assess large multi-account and multi-cloud deployments

• Store all the configuration information in their own internal database, providing customers a full inventory of what is running

• Include an extensive security and compliance database to assess against multiple standards

• Offer extensive complex analysis, such as identifying IAM misconfigurations that could lead to privilege escalation

• Support visualizations of relationships, such as showing the security groups and IAM roles associated with an instance (often called a security graph)

• Offer alerting and complex reporting

It’s really the scale and deep multi-cloud support that sets these tools apart from our earlier ones. But there’s more you might find:

• Real time inventory and assessments (most tools are time based, and some only update daily)

• Correlation with CloudTrail and other logs (e.g. identify who caused a misconfiguration)

• Advanced security graphs that identify potential attack sequences

• And more!

From CSPM to CNAPP

CSPM is the foundation, helping you visualize and manage your cloud “posture” (configuration). They’ve become essential for managing security and compliance, and every major CSP has one built in (for a fee).

Yeah, you’d think they would include a critical security capability in your subscription, but nope. On the upside, third-party tools historically have done a much better job.

While CSPM focuses on core security posture, another tool started developing in parallel. Cloud Workload Protection Platforms (CWPP… blame Gartner not me) actually ARE vulnerability scanners- for instances, containers, and even serverless functions. “Wait” you ask, “didn’t we already have vulnerability scanners for operating systems and stuff?”.

Why yes we did! And they royally sucked for cloud! Those vendors were too fat and happy and asked us to do painful things like open up our security groups for scans, or install agents in containers that… really don’t run well in containers. CWPP products made up cool tricks like scanning snapshots and images so they didn’t have to muck with running things, or build better agents.

Anyway, as you might imagine CWPP started showing up in CSPM tools, or they were bought, or they bought the other guys, or all of the above. And then vendors started adding or buying up features to scan Infrastructure as Code, assess APIs, perform Cloud Detection and Response, focus on identities (a category called Cloud Identity and Entitlement Management (CIEM) because WE JUST CAN’T HELP OURSELVES WITH THESE NAMES AND OH PLEASE MAKE IT STOP!!!!

Anyway, that’s Cloud Native Application Protection Platform (CNAPP). It started as CSPM+CWPP and now includes whatever the vendor can buy or copy from someone else. This isn’t actually a criticism, just the reality that when you see CNAPP it most definitely will include CSPM and workload protection, and might include a LOT of other things you may or may not need depending on who you are and what you are doing.

Why I subjected you to this painful history lesson

My goal with CloudSLAW is to provide you deep, practical knowledge to work as a cloud security professional. Although this lab is over a year into the program, CSPM/CNAPP will be one of the first tools you likely touch if you get a cloud security job. It’s important to know what they are and how they work, which is why we are running through this series and you will get hands-on with multiple different types of tools.

Today is a very basic deployment of Prowler, which was the first Open Source CSPM available, long before anyone put those four words together. Prowler also has a commercial version and can do a lot more than how we are going to use it.

This week we will keep it simple. Last week you scanned your local account, and this week we will use our cross-account role to scan our Workload1 account after we create some misconfigurations. Then you’ll have the chance to change settings and see some different results, or even scan ALL your accounts (which will probably break since our instance size is too small).

This lab is just level 1- a simple cross account, non-parallelized scan. It’s something I might still do if I’m performing a hand assessment, but mostly I would use a SaaS platform our automate Prowler to run at larger scale with parallel operations.

Today’s lesson key points are:

• Cloud Security Posture Management is an essential security tool to find and fix security and compliance misconfigurations.

• Cloud Native Application Protection Platforms are the “big brother/sister” of CSPM and add in other capabilities, like vulnerability scanning the operating systems and applications in instances.

The Lab

This is an easy one- we’ll deploy a CloudFormation template in a target account, start up our Prowler instance, log in, make a quick config change, run a scan, and then review the results.

One important point: when I set up this lab I used an instance size for the free tier. This is really underpowered, and in my testing the instance would hang on a full Prowler scan. We’ll use a command line to narrow the scope of the scan which speeds things up and works every time. If you want to run a bigger scan, you can always pull the AMI for the instance, launch a new one that’s sized larger, and then run the scan.

Video Walkthrough

Step-by-Step

Start in your Sign-in portal > TestAccount1 > AdministratorAccess > CloudFormation > Create Stack (with new resources). You’ll use these settings:

• Template URL: https://cloudslaw.s3-us-west-2.amazonaws.com/lab66.template

• Name: misconfigurations

The rest is just the defaults, including clicking the checkbox in the blue box to enable creation of IAM resources.

Wait until it’s complete in case there are any errors.

Okay, now close that tab > sign in portal > SecurityAudit > AdministratorAccess > us-west-2 (Oregon) region > EC2 > Instances. Note… if you don’t see any instances make sure you don’t have the screen filtered to “instance state = running” since we stopped the instance at the end of the lab last week.

Then:

• Click the checkbox next to the instance

• Instance state

• Start instance

• Wait about 3 minutes

We need to provide the instance enough time to start and for the SSM agent to run and connect to the Session Manager part of the service. Then Connect > Session Manager > Connect:

Once you’re connected, we will run three command lines to get things set up:

• sudo su - ec2-user (log in as the default ec2-user)

Then copy and paste this line:

sed -i '/prowler aws \\/a\  --services iam s3 ec2 \\' scan-account.sh

What’s this doing? This is our fix for that performance issue I ran into. This line gets added to the command line we use to call Prowler, and tells it to only scan 3 services: aim, s3, and ec2. The full script looks like this:

The script:

• Sets some variables, including the target account that we will send in from the command line, and the external ID which is set is our configuration file (review last week to see how we did this).

• It also sets our target S3 directory for the report.

• It runs Prowler with those settings, and limits the scan to only those 3 services.

• It outputs links to get to the reports for the scan. This just makes your life a lot easier (as you’ll see in a few minutes).

If you want to see the scripts in the directory just type ls to list the contents and cat <filename> to see the file.

All your prep is done, so hop back over to your sign in portal and copy out the account ID of TestAccount1:

Now start your scan with bash scan-account.sh <paste in the account ID>

5

There’s a little cloud magic going on now:

• Your instance has a role with permissions to assume the scanning role in the target account.

• We provide Prowler with the role and the external ID in our script.

• Prowler assumes the role in TestAccount1 using the external ID that was passed in originally in the user-data field (last lab).

• This setup will work to scan any account in your organization except your management account.

It will take 5 minutes or so to run, maybe less. Once it’s finished click the link (or copy/paste) to go to the directory in S3 with your results:

You can download any or all of them, but for this lab start with the HTML version. Click on it, then download, then open it up in your browser (just drag and drop the html file into your browser):

Woah, that’s a LOT of results! And even if you filter out to only FAIL you still… have a LOT of failed checks. This is the fundamental problem of CSPM! Every single tool on the market is VERY noisy. It’s up to you to understand and prioritize the results. This is why I really worry when I advise an organization and they are completely tool driven and the security team doesn’t really understand the fundamentals of cloud. That’s one of the main reasons I waited a year for a lab on a tool I normally use day 1; I needed to make sure you had solid fundamentals so you could interpret the results.

AND WE ONLY SCANNED 3 SERVICES!!!

For example, look at the Network ACL results. There are a lot of failed findings there for things like leaving default NACLs. Guess what? I always leave default NACLs! This isn’t Prowler’s fault, since some compliance standards say you should lock down your NACLs all the time. This standards were probably written by someone who hasn’t ever seen a cloud in the sky, never mind AWS/Azure/GCP/TheBadPlace.

Okay, the next bit of this lab is a little choose your own adventure. I recommend you do the following:

• Filter the results and try to identify which ones you would consider important.

• See if you can identify the 3 misconfigurations (well, maybe 4) that I have in the CloudFormation template. Then go review the template to see if you found them all.

• There’s a stopped instance in TestAccount1. Start it, run another scan, and see if your results change and how.

• Scan different accounts.

• Try the scan-all-org.sh script. If you do this you MUST increase your instance size or add the services line to the script so it doesn’t max out your instance. (use the same sed line but change the target file name).

  • This scripts calls the AWS Organizations API, lists all your accounts, and then runs the scan on each account serially.

• Download the CSV file and play around with sorting and other spreadsheet stuff I can’t do because I hate spreadsheets.

There’s a LOT of information in this deceptively simple lab. Take your time, play with Prowler, and get a sense of things. As a reminder this is not the best way to run Prowler!!!! I have it running in a way that works very well for labs, but in a real situation I would use the web interface instead of saving files to S3.

Phew. Cya in the next lab where we’ll learn about yet another way to use CSPMs!

-Rich

Hey everyone,

Just a quick note and invitation. As part of my new role as Chief Analyst for the Cloud Security Alliance I’m expanding the program with exclusive briefings for our corporate members.

Tomorrow (Friday) will be the first one, and I have permission to invite all 5000+ of you to check it out!

We probably won’t do this often, but as I build out new CSA programs don’t be surprised if I try some things out. I’m also working on a little extra topping for Pro members.

Here are the details.

Member Briefing: Lessons from the AWS Outage

CSA is hosting a member-only emergency briefing this Friday, October 24 at 10:00 AM PT to unpack what this week’s AWS outage tells us about cloud resiliency and the realities of multi-cloud. Register below and we’ll add you to the Zoom invite. Seats are limited to CSA Corporate Members.

Sign up here: https://airtable.com/appZnEzwBda493De4/pag9p5q9rr9jwliHq/form?prefill_Source=CloudSLAW

Prerequisites

• An AWS Organization with a SecurityAudit account and delegated administration for running StackSets.

The Lesson

You know how I said we’d build all this security stuff in pretty much the same order as I would do it for real?

Yeah, I kinda blew it.

This is a lab we should have done quite a while back, but there are a lot of little logistical things that kept putting it off. But today is our day! (How’s that for motivational speaking?)

Today we finally get around to starting our setup to deploy a Cloud Security Posture Management (CSPM) tool. In real life this is something I try to do on day 1 when I walk into a new org, but these are the sort of tools that make a lot more sense after you have a strong cloud security knowledge foundation.

So congratulations, young padawans. You are ready for the next big step on your journey to cloud security Jedi status.

(We may have had a little coffee issue today and perhaps my brain is… being weird).

Cloud Security Posture Management

I’ve mentioned CSPM a few times in previous posts. You saw hints of it when we enabled Security Hub (but not the actual CSPM part), and when I talked about it as an essential element of cloud incident response.

The TL;DR is that CSPM tools connect to your cloud accounts via API, run a crap ton of read API calls (e.g. Describe/List/Get) and try to identify misconfigurations which could lead to security and compliance issues. Modern tools can do a lot more, but they still all rely on this core capability.

FireMon Cloud Defense, a CSPM tool

There was a time when I was one of a very small group of people performing cloud security assessments. Many of us had to write our own tools since there were no features in the cloud platforms (heck, Azure didn’t even exist) and trying everything manually via the CLI or console wasn’t scalable. Netflix was the first to talk about building their own internal tool (Security Monkey) and, long story short, eventually commercial tools flooded the market (including mine — we’ll get there).

What’s a misconfiguration you ask? It can be as simple as a public S3 bucket or port 22 exposed in a security group, or as complex as a misconfigured role trust policy that could lead to account takeover.

These aren’t traditional vulnerabilities per se, because we aren’t talking about an unpatched flaw an attacker can exploit. They are more like "unforced errors” that can potentially be exploited by an adversary in an attack. Why am I wording this so weirdly? It isn’t my coffee issue (broken grinder, fixed now), but rather that every misconfiguration is something that technically has a valid purpose or the capability wouldn’t be in the platform in the first place.

We’ll talk a lot more about CSPM over the next few labs — including CNAPP, CSPM’s younger, bigger cousin.

Stop Confused Deputies with External IDs

To explore CSPM we will use some external tools (Prowler Open Source and FireMon Cloud Defense). Prowler will run inside one of our accounts, but FireMon is a SaaS platform out there… on the Internet someplace.

This creates an interesting situation. We will allow an outside account (via a role) access inside our Organization… and that account will have a lot of power.

But that outside account? It’s a multi-tenant application connecting to a lot of different customers. And the normal pattern for these is for the SaaS platform to use the same IAM role to connect to all their customers. This creates an interesting dilemma: what if one of those customers figures out how to trick the platform into doing something in our account instead of their account? They… confuse… the application (the “deputy”) into accessing the wrong account.

This is known as the confused deputy problem, and it isn’t unique to AWS. Heck, it’s well known that compromising a security tool is one of the best ways to get broad access to a target.

Okay, let’s break down how this works:

• Security Tool A in its own AWS account has access to Bob’s customer account via the SecToolVendor role, which has permission to assume the SecToolCustomer role in Bob’s account.

• Alice uses the same security tool, which uses the SecToolVendor role to assume access to her accounts.

• Alice (why is it always Alice?) tells Security Tool A that they own the account with ID of Bob’s account.

• Since Bob already trusts the SecToolVendor role, if the tool is tricked by Alice and assumes into Bob’s account, the tool is now doing whatever Alice tells it to, to Bob’s account.

AWS came up with a good way to reduce this risk. The External ID is a shared secret we build into our role trust policy. The External ID has to be provided during the AssumeRole API call, and must match. It’s a value in the SecToolCustomer role trust policy. Thus…

• Alice tries to add Bob’s account, since she knows they both use Security Tool A.

• When Alice tries to AssumeRole into Bob’s account, Security Tool A uses her External ID in the AssumeRole call (the vendors keep these in a database or secure configuration someplace on their side).

• That External ID doesn’t match Bob’s external ID in the SecToolCustomer role, so it fails.

But if Alice can trick Security Tool A into using Bob’s External ID, this falls apart. Or if Security Tool A shares its external ID across customers, well, no one would be that stupid. (Narrator: “they were that stupid”). (Thanks to Kesten Broughton for that excellent research and fwd:cloudsec presentation).

Here’s our old AssumeRole diagram, updated with the External ID:

The External ID isn’t stored with the SecurityToolVendor role, it’s kept someplace else and only used during the AssumeRole API call

Okay, enough background for today. This is the first lab set where we will deploy third-party tools into our, organization so it’s a good time to play with External IDs. Then, CSPM is so central to cloud security that we will explore multiple options over a series of labs:

• Preparing our organization (today’s lab)

• Open source (using Prowler)

• Commercial (using FireMon Cloud Defense’s free trial)

• Security Hub

You’ll see different deployment options, user interfaces, and capabilities. We’ll also see how to use CSPM for incident response, not just as a “vulnerability scanner”. As always, the labs are configured to be as inexpensive as possible.

Key Lesson Points

• Cloud Security Posture Management is a foundational cloud security tool.

• CSPM identifies cloud configurations that could lead to security issues.

• A common pattern to access our accounts is for a multi-tenant, external third-party tool to assume a role inside our accounts.

• This could enable another user of the tool to trick their way into our accounts. This is called the confused deputy problem.

• An External ID — a separate, shared secret — reduces the risk of a confused deputy.

The Lab

Today is the boring setup lab. The main learning objective is how to configure a cross-account role with an External ID. Although we don’t technically need the External ID, since everything is running inside our organization, this is an important skill to learn for connecting external tools.

Our tool of the day is Prowler — an Open Source (and also commercial) CSPM. We will go into more details on CSPM and Prowler in the next lab!

But! We also need to configure a hodgepodge of other settings to make our CSPM labs run smoothly:

• Change our session timeout so we can stay logged in longer.

• Deploy a CloudFormation template which creates:

  • A role for Prowler to use

  • A VPC for Prowler to run in

  • An instance with Prowler running

• Use a StackSet to deploy the role our tool will assume when it assesses accounts.

Video Walkthrough

Step-by-Step

As I was building this lab I got super frustrated with session timeouts. The lab steps themselves are pretty quick, but running Prowler can take some time, and definitely may blast past our default of an hour. Let’s expand that to 4 hours. This is some pretty simple ClickOps so I’ll just blast you through it with a couple screenshots.

Start in your Sign-in portal > CloudSLAW > AdministratorAccess > IAM Identity Center (like, is there a different Identity Center?) > Permission Sets > Administrator Access > Edit > Session duration > set 4 hours > Save changes > Close the tab.

It won’t kill you if you skip that, but… you’ll regret it.

Let’s go deploy the Prowler template. Head over to SecurityAudit > AdministratorAccess > CloudFormation > Create stack (with new resources): 

Use the template https://cloudslaw.s3-us-west-2.amazonaws.com/lab65-1.template

Name it “prowler” and set your External ID. This needs to be 8-64 characters, and you need to write it down or remember it. The External ID isn’t stored in IAM or with the role — instead it is passed in as User Data with a bash script that saves it to a file in the ec2-user home directory. This image includes some scripts which launch Prowler and pass it the External ID from the saved configuration file.

Click through the next couple screens and Submit. Then wait for it to finish, since we need the IAM Role that Prowler will use to be created before we try to deploy our cross-account role which trusts it. (You can’t create the trust relationship until the role is created — AWS will error out on a role ARN in a trust policy if the role doesn’t exist.)

Now Close the tab > Sign-in portal > SecurityOperations > AdministratorAccess > CloudFormation > StackSets > Service-managed > Create StackSet. It’s really darn important to swap into SecurityOperations, since that’s the account with delegated administration permission to deploy stacks. SecurityAudit does not have any permissions to make changes in other accounts.

This time use the template https://cloudslaw.s3-us-west-2.amazonaws.com/lab65-2.template

Then name: prowler-role, set the SAME EXTERNAL ID!!!! and grab the SecurityAudit account ID from your Sign-in Portal tab. It will look like this…

This ExternalId IS embedded into the role trust policy! It looks like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::533267272350:role/ProwlerLabInstanceRole"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "sts:ExternalId": "ProwlerLabExternalId456"
                }
            }
        }
    ]
}

You’ve done this more than a few times before, so the only other settings that need to change are:

• Deploy to organization

• Only the us-east-1 (Virginia) region, since this is an IAM role which will end up there anyway (if you try to deploy to multiple regions it will fail, since the role will already exist).

• Under Deployment options I recommend setting the max concurrent accounts to 10, and the failure tolerance at 9.

Then Submit.

No need to wait for this one to deploy. Close the tab > Sign-in portal > SecurityAudit > AdministratorAccess > Systems Manager > Session Manager > Preferences > Edit:

We will extend our idle session timeout for Session Manager. It won’t matter this week, but when we run Prowler scans next week, they won’t finish within the default of 20 minutes, and if the session closes you lose your scan. Max it out to 60 minutes:

Now let’s connect to our Prowler instance. Just so you know what that template did, it:

• Created a role for Prowler with a bunch of Read permissions. These are the default policies for read only and view only (don’t ask), and some extended permissions Prowler recommends to read settings that aren’t included in those two policies.

• Created a VPC with only a public subnet and a security group with no inbound access. Why did I use a public subnet? To save money — this instance will be running for multiple labs so if we used a NAT gateway it would cost a lot, and if we wanted to use service endpoints we would need them for every service Prowler uses to scan. We’ll rely on our locked-down security group.

• An S3 bucket to save Prowler results. Prowler has a pretty great Web UI, but then we would need to deal with opening up our security group/ports. You can do that on your own if you want, but I decided dumping reports to S3 was easiest when I have 5,000+ of you running your own labs in your own accounts.

• An EC2 instance of an image I created with Prowler pre-installed and some bash scripts to run it in different ways, using our External ID if needed.

Okay, go to EC2 > check the Prowler instance > Connect > Session Manager > Connect:

Now run the following command lines to test it:

• sudo su - ec2-user # switch to be ec2-user, in that home directory

• bash quick-scan.sh # run a quick Prowler scan and save the results to S3

That should execute within a couple minutes, and look like this when done:

This scan ran just on the local account and didn’t use our External ID — we’ll get to that next week. If you want a little side quest, you can check out the User Data by going to Instance settings > Edit user data.

Before we look at our results, go back to your instance and Instance state > Stop instance to save a little moolah.

Okay, let’s look at our results! Go to S3 > prowler-reports-<your account ID> and follow the directory tree until you reach the reports. Click the HTML report:

Then click Open and it will pop out the report in a new tab:

And there you go! These are just quick results — in the next lab we will run a deeper scan on a target with… some problems.

Lab Key Points

• External ID is a shared secret to prevent the confused deputy problem.

  • It’s a condition in the trust policy of a role being assumed.

  • The assuming party (e.g., your SaaS vendor) only provides it during AssumeRole, and it should be different for each customer.

• Prowler is a CSPM tool for identifying misconfigurations.

-Rich

I know what you’re thinking…

“Rich is such a slacker! Another week without a new lab? We’re only getting them every 2-3 weeks anymore!”

Yeah, I own that. For the couple thousand of you in the front of the herd the recent new labs have been a little erratic, and for those of you hiding in the middle of the pack I’ve been a bit slow to respond to comments and feedback.

But hey, I have good reasons! And, uh, this is all for free so, like, you know. The tl;dr:

• The latest labs take a LOT more prep work. Some of these are up to 8-12 hours of prep, even though they are still only 30 minutes for you. We are on some advanced topics and the hard part is figuring out how to structure things and keep them as cheap as possible.

• I’ve been transitioning into a new job.

New day, “sorta” new job

I’m excited to announce that as of today I’m the new Chief Analyst at the Cloud Security Alliance! This is a brand-new role they created for me so I… better not screw it up. I’ll spare you the details, but I wrote it all up at this post.

The short version is that my full time job is now research, advising CSA members, and training. Basically to help professionals and organizations as they deal with complex cloud and AI security topics. I’ve been involved with the CSA since the start and built the CCSK training program, wrote multiple versions of the CSA Security Guidance, wrote the Cloud Security maturity Model, and participated in a lot of different working groups and initiatives.

It’s been a 15 year job interview and I finally closed the deal! Time for coffee!

As for the recent lab slowdowns? Closing out my work at FireMon (where I’m still an advisor) so I could transition over to the CSA changed my priorities for the past couple of months and I had less time to devote to CloudSLAW.

The future of CloudSLAW is bright

As of now there are no intended changes to CloudSLAW and it’s now something I can do actively as part of my job, not something that has to be on the side. CloudSLAW will stay where it is and stay free, and I’m keeping Pro since I still handle all the costs through Securosis.

New labs will still be on the every 2-3 week schedule since they really are harder to create.

For those of you interested in continuing education credits and certifications, this is where things will get interesting. While we haven’t figured out exactly how we are going to configure things, we are looking at taking CloudSLAW labs, copying them into the CSA’s training platform, and adding in the tracking and quizzes required to issue credits and certificates.

The main CloudSLAW will remain free to the public and is the first place for new labs! I’m not taking anything away, and we have no plans to lock down old or new labs. The nice thing about what we have going now is that it allows me to build, test, and experiment with labs in a way I can’t do if everything is tied to certificates. \

Remember, the Cloud Security Alliance is a non-profit and its mission is to define and promote cloud and AI security best practices. Membership and certifications are two of the main ways the CSA funds its mission, and keeping CloudSLAW out there for free is completely aligned with the goals of the CSA.

Upcoming Content

As long as I have your attention…

• The next lab will be a week late due (and come out next week). It’s mostly drafted but I ran out of time to test and refine the tech side.

• That lab starts a series on Cloud Security Posture Management (CSPM) and will have about 5-6 labs.

• At the end we will circle back to our Incident Response series (since one facet of incident response is CSPM) and hopefully cover some basics of detection as code.

• I’m unsure about what topic to pick after that and plan on polling the CloudSLAW Pro members. One idea is securing AI apps in AWS. Another is containers. And at some point we need to get to data perimeters.

  • I’m very open to additional ideas, so send them over!

My sincere thanks for your time and support as this crazy experiment continues. People have been coming up to me to talk about CloudSLAW as I’ve been out at conferences and it's really motivational to know that these bits I spew out into the aether sometimes actually help people.

-rich

I realize you're probably asking yourself, "What the heck is a 'stage set'"?!? "Where's my lab?" Or "couldn't he at least have done a stage check with a video?!?!" Or maybe even, "did he get his nested quotes correct in the first sentence, and would those work in Python?"

Well, you see, sometimes when you run a long-term newsletter/blog/training program, you get caught up in the absolute relentless pace, and realize you mighta missed a couple of things or need a small breather.

Today, my friends, is one of those days.

I try to write these labs as if they exist out of time, since some of you see them the day they release but many more won't see them for over a year. But sometimes the real-time version of my life interferes with getting out timely labs, and rather than have a big gap I thought this would be a great time to zoom out and discuss the big picture of what we have been building with recent labs. Unlike a stage check we aren't finished yet, but I realized as I looked back at what we've covered that I wasn't doing a great job of providing very important context. Sure, some of it is in there, but we are pretty deep in the technical weeds so it's easy to lose perspective.

Let's clip in, hang back, and enjoy the mid-wall view for a moment (yeah, that's a rock climbing reference). I'm going to break out the main components of cloud incident response, show you where our labs fit in, and give you a preview of what's next. And I'll try and keep it all to about 7 minutes of reading time. This is absolutely one of my favorite topics, having built multiple Black Hat training classes on it and integrating it into the Cloud Security Alliance CCSK course and a few other places.

Oh — no video for this one, at least not yet. I'm writing it on an airplane as I head out to speak at 2 different events in 2 different countries. Like I said, life's been busy.

Cloud Incident Response 101(ish)

The TL;DR is that across our labs, in some cases going back to the first few months of CloudSLAW, we've been slowly building out a pretty robust incident response program. There's still a lot to do, but we've made some big strides.

Our overall objective is to effectively detect and respond to security incidents.

Simple enough, eh? But to do that we need to:

• Collect the right information.

• Detect security incidents.

• Have the data and skills to analyze and investigate.

• Use it all to contain, eradicate, and recover.

There are a few different incident response frameworks out there, and I semi-normalized them when building my CSA and Black Hat classes. Here's a slide I stole from the official CCSK training (Certificate of Cloud Computing Security Knowledge, from the Cloud Security Alliance... which I wrote):

You have completed multiple labs on preparation, detection, and analysis. Now let's see where that all fits.

Preparation Step 1: Feeds and Speeds

The very first thing we need for incident response is the right security telemetry to detect and analyze/investigate incidents. What's security telemetry? Basically, any logs or other information sources germane to security. This can potentially include any and every log file and event source, but some sources are better than others for security.

It's also critically important to know the timing of these security feeds, especially in cloud. It's hard to overstate the speed and scale of operating in the cloud, and this is just as true for attackers as for defenders. Attackers today are highly automated, work directly with cloud APIs, and can decimate environments in seconds or minutes. In the early days of cloud many organizations fed their cloud data into their on-premises Security Information and Event Management (SIEM) tooling, then had teams work exclusively from that data. The problem is those pipelines can take an hour or more before coughing up the data, so they were always working behind the attackers.

You don't need everything in real time, but you do need to know which sources you're collecting and how fresh the data is. Feeds and speeds.

Our labs set up a good core:

• CloudTrail data for all accounts and regions, saved to S3 with about a 5-15 minute lag. This is the most important log source for our cloud management plane, and our primary tool for detecting AWS-native attacks.

  • In Assume the Role! (Centralized Logging, Part 1)

• CloudTrail data is also available in near-real-time via EventBridge, but only when we push out EventBridge rules into the accounts using CloudFormation StackSets.

• Security Hub for all accounts and regions, which is available in the service itself (we aren't saving it to S3 yet) with all alerts sent to email. Security Hub alerts are real-time, but the services that feed Security Hub have their own timings:

  • GuardDuty, which is currently our main "intrusion detection" for cloud. Some GuardDuty alerts are pretty quick, but most don't appear for 20-40 minutes after the offending activity.

  • Access Analyzer for detecting publicly exposed resources (it can do more, but that's all we have set so far). It's pretty quick in my testing, but I don't have a good enough sense to give you consistent timing.

• We've dabbled with some other logs, but haven't built much to use them yet. These are also of lower security value. Not "no security value" — just lower.

  • CloudWatch Logs, which in our case is mostly storing logs from Lambda functions.

  • Session Manager logs which we learned about in Enabling Logs in Session Manager.

It's a good start. Here's a short sample of sources we haven't touched yet:

• Network logs (DNS and VPC Flow logs).

• Workload/system logs from instances (and containers, which we aren't using yet).

• Load balancer and API gateway logs.

• Logs from S3 "data events" (accessing the data — we do get management events like changing configurations).

I personally categorize sources based on two broad criteria:

• Are they security specific (GuardDuty and most security tools), or operational logs which also have security value (CloudTrail)?

• Is the source logs, events, or from tools? How can I use it for threat detection? Let's talk about that one for a moment...

Preparation Step 2: Threat Detectors

Different sources play different roles in threat detection and analysis. When I first started training traditional incident responders in cloud I found it helpful to break them into these categories. This one comes from my Black Hat training (okay, it's also in the CCSK):

In recent labs we built threat detectors for two of these three source types:

• Logs: That's what we did with Build a Time-Based Threat Detector with Lambda and Athena.

• Events: Most recently we built this with Build a Real Time Threat Detector with IaC.

  • We also receive alerts through Security Hub, per Use EventBridge for Security Hub Alerts. This feeds us everything from GuardDuty and Access Analyzer.

• Configuration: We get a little of this from GuardDuty, and a little from when we built out S3 autoremediation for exposed buckets (Schedule Security Scanning with a Serverless Fanout Pattern) But this is really about CSPM, on which we will soon have labs.

Hopefully this gives you some perspective on how these preparation labs fit together. We aren't done with the topic by any means, and will be integrating more sources and building more threat detectors in future labs, as well as learning about detection as code, but we've largely covered the core concepts for Feeds and Speeds and detection engineering.

Analysis

Okay, our threat detector fires... so what now? Panic? Send Rich an email begging for help? Nah, you got this. Weirdly, I covered analysis before threat detection in Getting Started with CloudTrail Security Queries, since we needed to learn about Athena and queries. These labs are a decent introduction to basic analysis of management plane events, and we even simulated a full attack in our Skills Challenge and Skills Solution. Remember my RECIPE PICKS mnemonic?

And yes, more later. Of course. :)

Where We Are, Where We're Going

Cloud incident response is an entire career, so there's no way we can cover everything, but at this point we have a good foundation. We have:

• Prepared our AWS Organization by enabling fundamental security feeds.

• Built threat detectors based on data in our logs, cloud events, and cloud security tools.

• Learned the basics of analyzing an attack with a live simulation.

The big gap in these labs is our lack of discussion on cloud configuration management and how that fits within incident response. We also haven't discussed managing threat detectors. These are both huge topics, so I'm engineering some nice labs to cover the basics and give you ideas for more to explore on your own.

Well, they just told me it's time to detach my keyboard from the iPad so the plane can land. Hopefully this has helped you orient yourself, and understand how these labs all fit together.

-Rich

Prerequisites

• An AWS organization with accounts named SecurityAudit and TestAccount1, and an OU named Workloads. I mean, are any of you really just dropping into arbitrary labs without doing all the other ones? Really?

The Lesson

In our last lab we built a time-series threat detector using AWS Athena and Lambda, based on CloudTrail logs. When it comes to threat detection, logs are about as foundational as it gets. Heck, back in the old days they were pretty much all we had.

But cloud is awesome, and aside from logs we have real-time events (which I explained using Star Wars LEGO way back in this lab). So far we’ve worked with two types of events: those coming from tools (like Security Hub in that post) and those generated by AWS services, as illustrated in our autoremediation lab, where we triggered based on S3 events sent to CloudTrail. 

Today we will build on those skills and create a new pipeline for CloudTrail event-based threat detectors. But to add a twist, I’ll walk you through the entire process of creating and deploying your own CloudFormation template. Sure, you can skip to the end I suppose, but that’s cheating and cheaters never win! (I mean, outside politics, business, sports, and… well, every other facet of life, I guess. But hey! Sometimes they get caught! I mean… nevermind).

The setup

We will be building the simplest kind of event-based threat detector: one that triggers off a single API call. As powerful as they are, these are also the most prone to being pains in the ass. Because we are triggering on an activity that might be authorized, without any nuance or checking to understand the context.

In other words, these can be very noisy. They aren’t false positives, because they alert on exactly what we asked for, but they can and do alert on legitimate authorized activity.

Today I picked an event I always want to know about: the creation of a new IAM user. Yeah, this could be super noisy most places, but I do know real large enterprises which don’t allow IAM user creation without a documented approval.

To make the lab more interesting, we also want to deploy our threat detector to our Workloads OU, but send out all alarms from our SecurityAudit account using our existing SNS topic. Here’s our design criteria:

As a security administrator I want a real-time alert every time someone creates an IAM user in any of my accounts. I want these alerts to come from my existing SNS topic in my SecurityAudit account.

Okay, translating that user story, we need to do the following:

• Capture the IAM:CreateUser event in every account.

• Send that event to our SecurityAudit account.

• Trigger our SNS topic for the alert.

This is different than how we set up Security Hub, since that service already centralized events for our entire organization. For this one we need to build that centralization ourselves.

I want to be very clear: the objective of this lab is to only teach the foundational technique for capturing these events. In a future lab we will build in more intelligence by routing to Lambda, as in the last lab, instead of directly forwarding all events!

The Lab

To pull this off we will build two CloudFormation templates: one to set up our SecurityAudit account with a new event bus to receive events and forward them to SNS, and another we deploy with StackSets to forward all the events we want centrally.

Instead of hosting a template for you, this time you will build it yourself and host it in your own S3 bucket.

Video Walkthrough

Step-by-Step

We’ll start by building our two templates and storing them locally. Once they’re done, we will build an S3 bucket and upload them. Then you’ll go into each account and deploy the appropriate template.

First, open up a text editor. I recommend one meant for coding/scripting like Sublime Text.

The SecurityAudit template

We need this one first, since we need the event bus set up before we can send it events. As a reminder, CloudFormation can use JSON or YAML; we will use YAML because it’s easier to work with. YAML requires proper indentation. If you run into issues and don't see any obvious errors, check your tabs and spaces.

First start with our usual CloudFormation header and a description so we can remember what this thing is for:

AWSTemplateFormatVersion: "2010-09-09"
Description: >
  EventBridge bus to collect CloudTrail and other events from accounts within the same organization and forward them to SNS.

The recommended practice is to list the template version (despite it being old).

The next section collects our Parameters. They are the variables we enter when we run the template. We could hardcode both of them, since we won’t use this template anywhere else, but this reduces copy/paste errors among students. No, not you — that person sitting next to you, of course.

Parameters:
  OrganizationId:
    Type: String
    Description: AWS Organizations ID (e.g., o-abc123def4)
    AllowedPattern: "^o-[a-z0-9]{10,32}$"
  SnsTopicArn:
    Type: String
    Description: ARN of the SNS topic (e.g., arn:aws:sns:us-west-2:123456789012:SecurityHubAlerts)

Now we start our Resources section, where we tell CloudFormation what to build. I pair our new event bus with the resource policy which allows any account in our Organization to send it events. Notice how we use our parameter for the Organization ID to fill in the blank?

Resources:
  SecurityMonitoringBus:
    Type: AWS::Events::EventBus
    Properties:
      Name: SecurityMonitoring

  AllowOrgPutEvents:
    Type: AWS::Events::EventBusPolicy
    Properties:
      EventBusName: !Ref SecurityMonitoringBus
      StatementId: AllowPutEventsFromOrg
      Action: events:PutEvents
      Principal: "*"
      Condition:
        Type: StringEquals
        Key: aws:PrincipalOrgID
        Value: !Ref OrganizationId

That’s the magic !Ref. Notice we use that substitution twice? The first time is at EventBusName in the resource policy, and we refer to whatever name was assigned when we made the event bus. The second time we refer to our OrganizationId parameter.

Referring to other values in a template is a critical capability for infrastructure as code. In this case we hardcoded the name of our event bus, but you may recall we don’t always know the unique identifier for a resource until AWS creates it and provides the identifier, like an instance ID. Referring to other resources in the same template tells CloudFormation to grab the value it needs. CloudFormation will then figure out the order to build things and wait until the first thing is ready before it creates the second that relies on it (usually — it ain’t like tech is totally reliable, or anything).

Okay, we have our event bus, and other accounts can send it events. Now let’s create a new EventBridge Rule to send those events to SNS:

AllEventsToSnsRule:
    Type: AWS::Events::Rule
    Properties:
      Name: AllEventsToSns
      Description: "Forward all events on the SecurityMonitoring bus to SNS"
      EventBusName: !Ref SecurityMonitoringBus
      State: ENABLED
      EventPattern: |
        {
          "source": [{
            "exists": true
          }]
        }
      Targets:
        - Id: SecurityMonitoringSnsTarget
          Arn: !Ref SnsTopicArn

Putting it all together (because let’s be real, this is the part you’ll copy and paste), we get our full template to create the bus, allow cross-account access, and forward all events to our existing SNS topic:

AWSTemplateFormatVersion: "2010-09-09"
Description: >
  EventBridge bus to collect CloudTrail and other events from accounts within the same organization and forward them to SNS.

Parameters:
  OrganizationId:
    Type: String
    Description: AWS Organizations ID (e.g., o-abc123def4)
    AllowedPattern: "^o-[a-z0-9]{10,32}$"
  SnsTopicArn:
    Type: String
    Description: ARN of the SNS topic (e.g., arn:aws:sns:us-west-2:123456789012:SecurityHubAlerts)

Resources:
  SecurityMonitoringBus:
    Type: AWS::Events::EventBus
    Properties:
      Name: SecurityMonitoring

  AllowOrgPutEvents:
    Type: AWS::Events::EventBusPolicy
    Properties:
      EventBusName: !Ref SecurityMonitoringBus
      StatementId: AllowPutEventsFromOrg
      Action: events:PutEvents
      Principal: "*"
      Condition:
        Type: StringEquals
        Key: aws:PrincipalOrgID
        Value: !Ref OrganizationId

  AllEventsToSnsRule:
    Type: AWS::Events::Rule
    Properties:
      Name: AllEventsToSns
      Description: "Forward all events on the SecurityMonitoring bus to SNS"
      EventBusName: !Ref SecurityMonitoringBus
      State: ENABLED
      EventPattern: |
        {
          "source": [{
            "exists": true
          }]
        }
      Targets:
        - Id: SecurityMonitoringSnsTarget
          Arn: !Ref SnsTopicArn

Save that file locally with the name securityauditeventcollector.template.

The Workloads OU Stackset Template

This one is even shorter. All we need is an EventBridge Rule which matches the CloudTrail Action we want, and forwards that to the central event bus we just created. But there’s a little twist — can you spot it?

AWSTemplateFormatVersion: '2010-09-09'
Description: 'Forward IAM CreateUser events to EventBus in another account (us-west-2)'

Parameters:
  TargetEventBusArn:
    Type: String
    Description: ARN of the target EventBus in another account (us-west-2)
    AllowedPattern: '^arn:aws:events:us-west-2:\d{12}:event-bus/[\w\-]+$'
    ConstraintDescription: Must be a valid EventBus ARN in us-west-2

Conditions:
  IsUSEast1: !Equals [!Ref 'AWS::Region', 'us-east-1']

Resources:
  # IAM Role for EventBridge to assume when putting events to target bus
  EventBridgeRole:
    Type: AWS::IAM::Role
    Condition: IsUSEast1
    Properties:
      RoleName: ThreatDetectorRole
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: events.amazonaws.com
            Action: 'sts:AssumeRole'
      Policies:
        - PolicyName: PutEventsToTargetBus
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - 'events:PutEvents'
                Resource: !Ref TargetEventBusArn

  # EventBridge Rule to capture IAM CreateUser events
  IAMCreateUserEventRule:
    Type: AWS::Events::Rule
    Properties:
      Description: Captures IAM CreateUser API calls and forwards to target EventBus
      State: ENABLED
      EventPattern:
        source:
          - aws.iam
        detail-type:
          - AWS API Call via CloudTrail
        detail:
          eventSource:
            - iam.amazonaws.com
          eventName:
            - CreateUser
      Targets:
        - Arn: !Ref TargetEventBusArn
          Id: CrossAccountEventBusTarget
          RoleArn: !If
            - IsUSEast1
            - !GetAtt EventBridgeRole.Arn
            - !Sub 'arn:aws:iam::${AWS::AccountId}:role/ThreatDetectorRole'

Since we want to monitor every region of every account, and CloudTrail events are only generated in each region of each account, we’ll need to push this using StackSets. But there’s one small problem: we can’t create the IAM role twice, since roles are always in us-east-1. This creates an error.

To get around this problem we use the ‘code’ part of infrastructure as code. That Condition block creates something called IsUSEast1 which only evaluate to True when the template is running in us-east-1. Then, in the Resource block for the Role, we use “Condition: IsUSEast1”, which says “only create this in us-east-1”.

Pretty cool, right? The template will create our EventBridge Rule in every region where we run the template, but only create the role in one region.

Now I want you to focus on the event pattern. There’s also a small redundancy in there… can you see it?

We specify source as the originating service for the event, but then within that event we also validate the eventSource. This is optional — technically we only need one or the other. Using both slightly reduces the risk of someone spoofing an event (they can spoof eventSource, but not source).

And one last note: we didn’t create an event bus in every region, just in the one region. One nice capability of EventBridge is that we can forward events across regions and accounts pretty easily.

Okay, save that template as threatdetectors.template.

Time to Deploy!

Start in your Sign-in portal > SecurityOperations > AdministratorAccess > Organizations. Then copy your Organization ID and paste it into a new text file.

Then go to S3 > Create bucket:

Name it “cloudslaw-templates-<your org ID>-<some random characters>” and Create bucket:

Then click your bucket > Permissions > Bucket policy > Edit and copy and paste in this policy. Then replace the bucket ARN and organization ID!!! Make sure you leave “/*” at the end of the ARN:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowOrgReadObjects",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::<bucketName>/*",
      "Condition": {
        "StringEquals": {
          "aws:PrincipalOrgID": "<o-YourOrgId>"
        }
      }
    }
  ]
}

Then Save changes. Then Objects > Upload and upload your two template files:

With those hosted in S3, go back to Objects, then click each file and copy its URL:

Paste both into your open text file, since you’ll need them to use CloudFormation: 

Now we just need to deploy the darn things. To start we need to create the central event bus and our forwarding rule in SecurityAudit. So… close the tab > sign in portal > SecurityAudit > AdministratorAccess > CloudFormation > Create stack > With new resources:

Paste in YOUR template URL for securityauditeventcollector.template:

You’ll need the ARN of your existing SNS topic, which should be arn:aws:sns:us-west-2:<your account ID>:SecurityHubAlerts. Name it SecurityEventCollector > paste in your Organization ID and SNS topic ARN. Remember, if you forget your account ID just click in the upper-right to grab it:

Click through everything else and Submit. Once it’s complete go to EventBridge > Event buses > copy the ARN of your new event bus:

Now close the tab > sign in portal > SecurityOperations > AdministratorAccess > CloudFormation > StackSets > Create StackSet: 

This time paste in your threat detectors.template URL:

Name it ThreatDetectors and paste in the ARN of your Event Bus:

For Set deployment options, choose the following:

• Deploy to organization

• Specify regions: us-east-1 and us-west-2

• Maximum current accounts: 10

• Failure tolerance: 9

• Region concurrency: Parallel

Then Submit.

This is now deploying concurrently across all your accounts and regions. It should only take a few minutes. And being an “organization” StackSet, it will automatically deploy into all new accounts. Pretty cool, eh?

Okay, testing this one is optional. The easy way is go into any of your accounts and create an IAM user, but don’t attach any policies. You’ll see an alert within 15 seconds, and I strongly recommend then going back to delete that IAM user.

Lab Key Points

• Real-time event-based threat detectors must deploy to every account and region where you want to track activity.

• When hosting in S3 you need a bucket policy to allow access from other accounts in your organization.

• EventBridge Rules can forward events from one event bus to another, even across regions.

-Rich

Prerequisites

• Complete the Cross-Account Athena lab (which has a few prerequisites itself).

The Lesson

Over the past few labs we learned how to run security-style queries with Athena, which in the realm of SecOps (incident response) is one of our best tools for analyzing incidents. Keep in mind that everything we’ve covered would also translate to a commercial Security Information and Event Management (SIEM) product.

Today we’ll expand on the concept and learn how to use queries as threat detectors. A threat detector is basically an alarm. In SecOps we talk a lot about “indicators of compromise” and “indicators of attack” and other signs that an adversary is trying to break in (IoA) or has broken in (IoC). There’s a ton of art and science to these and we have various ways we can implement them.

There are many techniques and places to implement threat detection, including use of specialized tools like Intrusion Prevention System (IPS) network appliances. GuardDuty? Yep, that’s a threat detection service run by AWS that tracks various activities in your account looking for something suspicious based on what they are doing (e.g. making a bucket public) or who is doing it (like activity from a known-suspicious IP address). (If you are interested in detection engineering I highly recommend the Detection Engineering Weekly Newsletter by Zack Allen.).

We’ve already set up “detections” for public S3 buckets and GuardDuty alerts, but what about activity based on multiple things happening in our logs?

The following, for example, are potential threat activities that could show up in CloudTrail but we can’t capture with a single EventBridge rule or relying on something like GuardDuty:

• Multiple failed/denied API calls over a time period that could indicate someone is trying to see what privileges an identity has.

• A change in the user agent used for making API calls that indicates maybe a credential was compromised and is now used in an attack tool.

• A sequence of actions that aren’t suspicious alone, but could indicate an attack when looked at together. Like the combination of GetCallerIdentity followed by Describe API calls followed by taking a snapshot of an instance and sharing it with a new account (as we showed in our incident analysis skills challenge).

All of these are in the logs, but to make them threat detectors we need to figure out how to trigger based on the activity.

Sliding Query Windows for Threat Detection

In my examples above, you probably noticed that some of the examples look like we could see them in a single query, while others might take a little more analysis. You can probably see how to run the queries yourself to look for the activity, but for threat detection we want something that runs automatically on a schedule.

This does have two problems:

• You can run queries on a schedule, but what if one step of the attack happened before your query window?

• You can’t always detect these patterns with a single query. For example, you can’t just write SQL for “find me every time the user agent for this role changes”.

Let me illustrate the first point. Imagine an attack with four steps, and it’s only a potential threat if we see all four of those things in a row. In an ideal world, we could run queries for any user that does all of those four things over, say, the past 15 minutes.

But what if those steps occur partially in one query’s time window, and partially in another?

Oops, we missed! But hey, you are all smart people (I mean, to a point, you are silly enough to be listening to me which shows terrible personal judgement). All we need to do is have overlapping/sliding query windows like this:

Notice how I extended the query window for EACH query by 5 minutes (to a 20 minute window) and run them every 5 minutes? With this overlap I shouldn’t miss the sequence that indicates the attack.

How do we figure out our time windows? There’s actually a formula that helps (ah crap, math 😭 ). Say we want to find 5 failed API calls within an hour, we just divide the detection window (60 minutes) by the number of actions (5) and thus run our query every 12 minutes.

This is better than how I illustrated it, since if you did the math you can see I should have kept my 15 minute query window and run it every 3.75 minutes. Either approach is valid depending on how important timing is. Usually, for threat detection, we don’t care if X actions occur in Y timeframe so we can play with our windows a bit to reduce the frequency of queries and save a little cash.

What I’ve illustrated here is the most basic way to approach the problem. It probably wouldn’t shock you to know that there are dedicated real-time stream-data analysis tools like Apache Flink for all sorts of advanced fun. Or we can send events we care about into a database like DynamoDB and run code or queries on top of that.

Anyway, the Flink logo is so cute I’m just going to paste it here and call it a day. We’ll cover some of the other scenarios in future labs.

Today’s lesson key points are:

• Threat detectors look for activity that is known to be a potential indicator of attack or indicator of compromise (IoA means someone is possibly attacking, IoC means they maybe succeeded).

• Scheduled queries are one common type of threat detector.

• When using scheduled queries and look for time-series events that could indicate an attack, we use sliding windows so we don’t miss the threat.

• Unless you are already using

The Lab

Today we will use a combination of EventBridge, Lambda, and Athena to run a query-based threat detector. Specifically we will look for 5 failed API calls over the past hour and use that 12 minute sliding time window. Why Lambda? Because there is no other way to schedule Athena queries.

Oh, and then we get all fancy and will transform our results into the Amazon Security Findings Format before we email them off (using our existing Security Hub alert SNS topic). Why ASFF? Because that’s what Security Hub uses natively in case we want to change things up later and push our findings into Security Hub directly.

Maybe I should have used OCSF, if you know what that is, but for our lab it doesn’t matter.

We’ll finish by running a CloudFormation template in our TestAccount1 that creates an IAM role and lambda function that can’t really do anything that then tries to do things and generates failed API calls.

Video Walkthrough

Step-by-Step

Start in your Sign-in portal > SecurityAudi > SecurityFullAdmin > us-west-2 region > CloudFormation.

Quickly snag your Account ID from the upper right corner and paste it someplace safe since you will need it twice.

Then Create stack:

• Deploy https://cloudslaw.s3-us-west-2.amazonaws.com/lab63.template

• Name it Threat Detector

• Paste in your Account ID

• Click through the rest and Submit

I highly encourage you to read this one while it deploys. The tl;dr is it:

• Creates a new role for our lambda function with permissions for Athena, S3, CloudWatch Logs, and SNS.

• Creates a lambda function that:

  • Runs our query, which includes every user that has 5 failed API calls within the last hour (see below).

  • For every user with the 5 failed API calls, it creates an ASFF-formatted finding and sends it to our Security Hub SNS topic (that we made a while ago).

• Creates an EventBridge rule to run our lambda function every 12 minutes.

The python code is pretty long so I encourage you to read that yourself, but here’s the core logic for our threat detector (our Athena query):

SELECT 
     useridentity.arn as user_arn,
     COUNT(*) as failed_count,
     ARRAY_JOIN(ARRAY_AGG(DISTINCT eventname), '|||') as failed_actions,
     ARRAY_JOIN(ARRAY_AGG(DISTINCT useragent), '|||') as user_agents,
      MIN(eventtime) as first_failed_event_time,
      MAX(eventtime) as last_failed_event_time
FROM {DATABASE_NAME}.{TABLE_NAME}
WHERE 
      from_iso8601_timestamp(eventtime) >= CURRENT_TIMESTAMP - INTERVAL '75' MINUTE
       AND errorcode IN ('AccessDenied', 'UnauthorizedAccess', 'UnauthorizedOperation')
       AND useridentity.arn IS NOT NULL
       AND useridentity.arn != ''
       GROUP BY useridentity.arn
       HAVING COUNT(*) > {FAILED_API_THRESHOLD}

It’s pulling the user identity, the event names (what the user tried to do), the user agent, (e.g. python or browser), the time of the first failed event and the time of the last failed event (useful for investigating).

Yes, I used an LLM to write that… and all the python code. And the CloudFormation. DON’T JUDGE ME!!!!

Okay with that deployed we need to make some modifications to the bucket policy in our LogAudit account since, right now, only our SecurityFullAdmin role has the cross-account privileges and this poor little lambda function is erroring out left and right until we give it access.

Close the tab > Sign in portal > LogArchive > AdministratorAccess > S3 > click your cloudtrail bucket > Permissions > scroll to Bucket Policy

The very top of your policy looks like this with a different account ID:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "SecurityFullAdminCloudTrailAccess",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::533267272350:role/aws-reserved/sso.amazonaws.com/us-west-2/AWSReservedSSO_SecurityFullAdmin_c45dc754d51c1875"
            },

You need to edit the policy to add in the IAM role used by the lambda function. This is a little tricky, so copy this and paste it into a text file so you can edit in your own details:

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "SecurityFullAdminCloudTrailAccess",
			"Effect": "Allow",
			"Principal": {
				"AWS": [
				"THE ARN OF YOUR SECURITYFULLADMIN ROLE IN THE OLD POLICY",
				"arn:aws:sts::YOUR ACCOUNT ID:assumed-role/TimedThreatDetectorsLambdaRole/TimedThreatDetectors"
				]
			},

Now paste the update in-place over the “old” version, making sure to check you didn’t mess up any commas or other characters.

The full policy will look like this when you are done.

Scroll down and click Save changes!

Now it’s time to test! We will run a CloudFormation template that creates a lambda function with a role that doesn’t have any privileges, but still tries to do things, and an EventBridge rule to run it on a schedule. This will generate plenty of Denied API calls for our logs.

 Close the tab > Sign in portal > TestAccount1 > CloudFormation > Create stack

• Use https://cloudslaw.s3-us-west-2.amazonaws.com/lab63-test.template

• Name it deleteme

• Click through the rest and Submit

  

  Walk away from your computer for 15 minutes, then Delete the stack!!!!

Now wait about 15-30 minutes (since it takes a little time for CloudTrail to write the logs to S3) and check your email. You should have an alert that looks like this:

Pretty cool, eh? We have a nicely-formatted alert based on an Athena query that’s running on a timed schedule with sliding windows!

-Rich

Prerequisites

• This is the solution for the challenge Skills Challenge: Investigate Your Own AWS Attack with Athena.

The Solution

The prior lab simulated an attack in your account to generate logs we can use to start learning the basics of security queries for Athena and how to investigate incidents. This is all based on my RECIPE PICKS mnemonic, which I wrote up in this blog post.

In this lab I’ll walk you through my process and how I ran through the challenge. Remember, this is not a full incident investigation — we are still in the early learning phase, so there are parts I skipped!

Video Walkthrough

To start here’s the graphical mnemonic, and I’ll run through everything in order:

Maybe I should use an AI to make this look cooler!

Now, how do I even know what I need to investigate? Well, it turns out I got this handy email from Access Analyzer via Security Hub (since we configured email alerts… many labs ago).

Resource

That email tells me I have a public snapshot, and it provides the account ID and the snapshot ID. The first thing I did was copy that snapshot ID and put it into a text file for reference.

Then I logged into that account, which will always be via your sign in portal > TestAccount1 > AdministratorAccess, and went to view any snapshots via EC2 > Snapshots > deleteme. Remember, we did not delete the CloudFormation stack yet, so should still have the resources we need, but if you DID delete it before investigating you can run it again, or just rely on what you see in the logs.

I’ll call this one sick, so I want to stop the bleed. Why? Because it’s a public snapshot and I really shouldn’t have those if I wasn’t expecting them. If you want you can click Modify permissions and make the snapshot private.

In a real investigation I might also start looking at the instance the snapshot is based on, but we’ll skip that for this exercise.

Events

Go to sign-in portal > SecurityAudit > SecurityFullAdmin > Athena to run your queries.

This is the query we ran last week. I like to know the event, when it happened, and who did it. In our sample we don’t pull the requestParameters, but I often also grab those to see the full parameters of the API call.

SELECT
useridentity.arn,
eventname,
sourceipaddress,
eventtime,
resources
FROM cloudtrail_logs.organization_trail
WHERE requestparameters like '%snap-YOUR ID%' OR responseelements like '%snap-YOUR ID%'
ORDER BY eventtime

As a reminder, this shows every API call where the request or response includes that snapshot ID. I can see the original create call, the tag (with the deleteme name), a bunch of describe calls as the code was waiting for the snapshot to finish (you’ll see them a lot in logs), and then that ModifySnapshotAttribute, which I just happen to know is the call to make it public.

Nearly every call came from the slaw-demo IAM user, although you can also see Access Analyzer and Security Hub at work.

Changes

These are the before and after states. For the snapshot in this lab we see the full history (not public, then public). But this is a very simple demo, and in a larger incident we would need full change management tracking from AWS Config or a CSPM tool. That’s more than we have time for today.

Identity

Who made the API calls? We can see that it’s an IAM user (bad on you) with the name slaw-demo-<account ID>. Paste this into your notes, and we will come back later. There’s also an interesting CreateTags, with no associated identity, which we will ignore today (unless you want to look yourself), since it’s something internal to AWS.

At this point we know that IAM user created a snapshot of an instance and made it public, so we have a good sense of what’s going on here.

Permissions

This tells us the IAM blast radius of that IAM user. What permissions does it have? How much damage can it do?

To make life easy, I just logged into TestAccount1 > IAM > Users, clicked that user, and looked at the permissions. Instead of a truncated screenshot, here’s the full permissions policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "iam:GetUser",
                "sts:GetCallerIdentity"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "ec2:DescribeInstances",
                "ec2:DescribeVolumes",
                "ec2:DescribeSnapshots",
                "ec2:DescribeSnapshotAttribute"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "ec2:CreateSnapshot",
                "ec2:CreateTags"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "ec2:ModifySnapshotAttribute"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

I’d call this patient sick. Why? Because that IAM user can describe and snapshot any of my instances, and change the attributes to make them public!

In a real incident, I’d probably add a DenyAll policy to stop the bleed, but not today.

Entitlements

Okay, sometimes you just need to make a mnemonic work! I wrote this before any nifty AI could help out. In this case we are shifting to the permissions of the resource. Does a snapshot ever have permissions? Nope. But the instance it’s based on? Maybe

You can go into EC2 > instances and poke around and see whether your instance has an IAM attached role (it doesn’t). It might still have embedded credentials for something, but save some time and don’t bother — I just gave you the answer. Too many instances to figure out which one is affected? You can pull the ID of the storage volume the snapshot was based on in the RequestParameters, then look at that volume to see which instance is attached.

Like I said, today is just the basics.

Public

Been there, done that, and the answer was in the alert that said “this thing is public.”

IP

For this I did two things. First, I pulled the IP from the query results and ran it through an IP checker (whatismyip.com).

Well, it came from within AWS. And to save you time, AWS uses a lot of instances to run things (including lambda functions, which is what we did today) so that… is not overly helpful. However, this query will tell us all the API calls to our account from that IP, and might give us a better sense of what’s going on:

Open another query window and use the provided JSON

SELECT
awsregion,
eventname,
eventtime,
useragent
FROM cloudtrail_logs.organization_trail
WHERE sourceIpAddress = 'IPADDRESS'
ORDER BY eventtime

Well that’s interesting, the useragent string shows this originated from a lambda function running python (that’s thanks to the Boto part of the string, which is the name of the python SDK for AWS).

I also see GetCallerIdentity, which can be an indicator of attack, because it’s the API call you make when you… don’t know who you are. If you are an attacker and you get an access key and secret key, you often start with GetCallerIdentity so you can learn what account you just got access to — this is a noisy method, there are other quieter ones out there.

Caller

Let’s look at all the API calls from that IAM user. Open another query window and use this JSON:

SELECT eventname, useridentity.username, sourceIPAddress, eventtime, requestparameters
from cloudtrail_logs.organization_trail
where useridentity.username = 'slaw-demo-YOURID'
order by eventtime asc;

I have more results than you, since I ran this multiple times, and we are searching on a name (if I looked for the access key it would only show unique results, since those aren’t reused… mostly). I also added the RequestParameters to the results, just to show how you can add and remove the data you are pulling based on what you are doing at the time.

As I look at this, I don’t see anything I didn’t already know from pulling the other logs. The IAM user runs GetCallerIdentity, then describes instances, takes a snapshot of one, then makes it public.

Track

This is where we look for indications of a pivot. There isn’t really a query for this, so we look for indicators like multiple IAM users or roles being used from the same weird IP address, signs of role chaining and similar. It’s more than we can really cover in this basic scenario, sorry!

But hey, I don’t want to leave you with nothing! Here’s a query to identify any API calls which were denied or had errors. It can be super noisy and not always useful, but combined with the other information we gathered, it might help give you a sense of what security boundaries the attacker was trying to get past:

SELECT count (*) as TotalEvents, useridentity.arn, eventsource, eventname, errorCode, errorMessage
FROM cloudtrail_logs.organization_trail
WHERE (errorcode like '%Denied%' or errorcode like '%Unauthorized%')
GROUP by eventsource, eventname, errorCode, errorMessage, useridentity.arn
ORDER by eventsource, eventname

Nothing really stood out to me when I went through these. Normally I would time-bound around the attack, but I was lazy and pulled them all from the last 3 months.

Forensics

This is… everything else. Do I need to see everything running on the instance or what data was stored on it? Do I have other kinds of logs like database or VPC flow logs to review?

Incident Summary

Our objective today was to gain more experience with Athena security queries through a very basic incident. Based on our queries and looking around, we now know:

• A lambda function from another account:

  • Used GetCallerIdentity to figure out what account it belonged to.

  • Described our instances.

  • Took a snapshot of an instance (technically, of the storage volume).

  • Tagged the snapshot deleteme.

  • Used ModifySnapshotAttribute to make the snapshot public.

• The IAM user had permissions which allowed it to snapshot any instance and make it public.

• The lambda function used the python SDK.

• The instance did not have any permissions (roles) assigned to it.

I’d guess this was an exposed credentials situation, largely due to the GetCallerIdentity API call. Also, because I coded the attack and I sent your credentials to my account in the first place.

I hope this was useful. We’ll come back to this exact scenario later, and go into much greater depth to fill in the gaps I glossed over. This set of queries is nearly always where I start, and if you are using a commercial SIEM you can easily modify them for the appropriate query language for your platform.

-Rich

Prerequisites

• Athena set up so you can query CloudTrail.

The Lesson and Lab

Today we'll do a bit of a skills challenge. Unlike some of our other skills challenges, this one's not designed to test new or existing skills. It's designed to actually help you learn and work your way through a particular problem.

The Setup

In our last lab we learned how to set up Athena to query CloudTrail logs, and tried a few basic queries. For this challenge I’ll point you towards a blog post I use as my reference for initial incident response, and then we will simulate an attack on your account.

Your challenge, should you choose to accept it, is to use those queries to piece together more of what happened. Being early in our incident response learning, I don’t expect you to run completely through the process, but you should work with a few of the queries to get a sense of how things piece together. In the next lab I’ll run through it myself, showing you how I approach the problem.

Our goal is to reinforce how to write Athena queries for security — this isn’t a test, but a chance to learn and explore more on your own! 

Here’s how it will work:

You’ll run a CloudFormation Template in TestAccount1. This will build some things, including an IAM User with static credentials, and share that information with me over EventBridge. On my side it will trigger a simulated attack: a series of API calls (run in a Lambda function) similar to an attacker finding credentials and then using them to exfiltrate data.

Everything is tightly constrained and I’m happy to share the Lambda source code if you have concerns. The TL;DR is that my simulator can only work with resources with a very specific naming pattern which embeds the account ID, and the account ID is cross-referenced with the metadata from EventBridge, so you can’t spoof the system and trick it into touching an account other than the one which originated the message. Nor can you trick it into touching resources which don’t match the names it expects.

You will:

1. Run that CloudFormation.

2. Let it run for a few minutes.

3. That simulates an attack.

4. If you’ve done our other labs, you’ll get an alert in email!

5. Then it shuts everything down.

6. You investigate with the queries.

7. You delete the stack (you can delete it whenever you want after it initially runs — what we need is already in the logs at that point).

It’s a one and done, and all the hard work happens with the logs.

Video Walkthrough

Let's Do This!

First open the blog post at https://securosis.com/blog/aws-cloud-incident-analysis-query-cheatsheet/

Sign into the console, go to TestAccount1 > AdministratorAccess > region us-west-2 (Oregon) > CloudFormation > Create stack:

At this point, you know the drill:

• Use the S3 url https://cloudslaw.s3-us-west-2.amazonaws.com/lab61.template

• Name it simulation.

• Leave everything else at defaults and Submit.

You’ve done this like 87.5 times now, so no more screenshots!

What's Happening Behind the Scenes

In the background this:

• Creates a VPC (because you didn't have any in your account) with a private-only subnet.

• Launches an instance in that subnet.

• Immediately stops that instance so you don’t get billed for it.

• Then creates an IAM user with an access key and secret key.

An access key and a secret? Remember when I said never to allow this? Well, now we're going to learn why, because we will actually simulate what an attack would look like.

That access key and secret key are sent to me over EventBridge. Those feed into a Lambda function. Now, just so you know, I don’t storing them anywhere. They are not logged. They are not stored in S3. They're not stored anywhere.

That Lambda function uses those credentials to create a snapshot of our specially-named instance, and to make the snapshot public.

Finding Your Starting Point

Two things should happen within a few minutes:

1. If you set up Access Analyzer correctly in our previous labs, you'll actually get an email alert coming out of Security Hub from Access Analyzer that something is now public. If you get that email, copy the snapshot ID.

2. Otherwise, you can go into EC2 > Snapshots and look for one named deleteme which is public. Copy the snapshot ID.

Ugly email, but it gets the job done

I love how our controls work together, so a lab from months ago triggers an alert near real-time. With that snapshot ID it’s time to start your queries.

Time to Investigate

Will we query the logs here in this account? Nope, we close the tab > sign in portal > SecurityAudit > SecurityFullAdmin > Athena. Why? Because that’s where Athena is set up from previous labs.

You likely have tabs open from our last lab. Close them out. While I want you to work with the queries over at the Securosis blog post on incident investigations on your own, I’ll show you the first one to get started.

Your First Query

Here’s the first query, taken right from that post, with the table name pre-filled, since we are all using the same one:

SELECT
useridentity.arn,
eventname,
sourceipaddress,
eventtime,
resources
FROM cloudtrail_logs.organization_trail
WHERE requestparameters like '%<your snapshot id>%' OR responseelements like '%<your snapshot id>%'
ORDER BY eventtime

Copy this, paste it into your Athena window, replace <your snapshot id> with your snapshot ID (don’t leave those brackets!) and wait about 45 seconds for results:

You will see all results where the request or response included the snapshot ID, which is a pretty good representation of everything involving that snapshot.

Your Mission

There are a bunch of queries on this list. What I’d like you to do is play with the queries on your own to gather more information. Your objective: piece together the story of what my little attack script did.

You’re starting with that snapshot ID because that’s where you got your alert. From that alert, you want to piece together, well, what actually happened? We have all of our events — that's what we did with Athena. But there are other areas to look at, such as:

• Who or what made the API calls?

• What is that snapshot of?

• What permissions did that user have? (You may need to go back into TestAccount1 to look).

Your objective here isn’t to do everything you would in an incident response. I want you to focus on the basic Athena queries. Try to spend about 15 or 20 minutes working on this, but feel free to do less or more. I’m not your dad. (I can say that because my kids never read anything I write anyway).

If you follow the RECIPE PICKS process, you should be able to figure out the timeline of events, the potential origin (hint, it was leaked credentials), the blast radius (what else could that identity do?), and what was exposed.

Hints and Cleaning Up

I told you exactly what happened, since this is about testing out those queries — not performing a full incident investigation. We’ll do that next lab, as I walk through the solution step by step.

The biggest hint is to have a private window or another browser open that stays logged into TestAccount1. That will enable you to look at the identity and resources (instance, storage volume, and snapshot) so you have context for your investigation. These are much harder to figure out when you are just looking at the logs.

Once you’re done go back into TestAccount1 to clean things up. Delete the snapshot and the CloudFormation stack. Yep, it’s that simple.

That’s it, and see you soon with a full walkthrough of how I approach this!

-Rich

Prerequisites

• Have Athena configured for querying CloudTrail, like we showed you in this previous lab.

The Lesson AND the Lab!

That’s right, this week we are skipping to the fun part — for the first time the Lesson and Lab are combined! Why? Because usually I use the Lesson section to focus on core principles, and then the Lab to put them into action. But for today’s topic it makes more sense to walk you through some basic queries as we explore the structure of CloudTrail, instead of just having you read about it and then… stare at it.

Today we will dig into CloudTrail and focus on searching on and extracting key data, which we tend to use in threat detectors and incident analysis. CloudTrail logs and events have a consistent structure, but since every AWS service is run by a different team, and the team generates its own CloudTrail events, there’s enough variation to… be kind of annoying at times. It isn’t all that bad once you learn some basics, so that’s our focus today.

Remember that a CloudTrail log entry is a record of an API call. Every API call includes an identity, an action, the parameters of that action (e.g., “delete THIS S3 object”), maybe a response from AWS (not all API calls provide a verbose response), and metadata like the time, region, source IP address, etc.

Bah, enough talk — let’s do!

Video Walkthrough

First, let’s make a couple API calls, which will generate log entries we can dig into.

Log into your Sign in Portal > TestAccount1 > AdministratorAccess > EC2:

We’ll keep it simple and run a basic instance, let it finish booting up, and then terminate it. Follow these steps…

Launch Instance > name it ‘delete’ > Launch instance:

Click through to proceed without key pair > Launch instance:

Now wait 3-5 minutes. Then Instances > click your instance > Instance state > Terminate (delete) instance:

Great, we generated what we need, and it should up in our central CloudTrail within 5 minutes. So, go take 5 minutes and do what you want. NO, NOT THAT!!! Oh, never mind, just… see you in a few minutes.

Now close the TestAccount1 tab > Sign In Portal > SecurityAudit > SecurityFullAdmin > Athena:

My Athena queries from the last lab still showed up, so I just closed them out and deleted whatever was in the Query 1 tab.

Let’s set up a basic scenario. Imagine you get a call from a scared intern who noticed that a critical EC2 instance is… not there anymore, and the entire Internet is down. I mean, this is the sort of things interns tend to do, but in this case the intern swears they didn’t touch anything. In fact, they were at the beach when it went down (please don’t tell their manager). Also, they have no idea who even owns the instance — they just know the Internet is down and their little brother can’t watch Bluey and this is bad.

Okay, where to start?

Well, if we think about it, we want to know:

• Who terminated the instance and when.

• Who originally launched the instance, which could indicate who owns it.

• Whether this was a mistake, or maybe a hack? I mean we don’t even have an instance ID to work with at this point.

Look, this is highly contrived and barely makes sense, but I’ve had multiple 5 and 6 am work calls this week, so it’s the best you’re going to get.

Let’s start by running a query to find recently terminated instances and order them with the most recent ones at the top. That should help narrow it down. And notice in the query that I’m searching based only the Action (TerminateInstances).

Querying based on the Action is a common starting point, especially when building a threat detector.

Assuming you have the same table name as me from the prior lab, this query will work without any modification and you can just Copy > Paste > Run. If you used different names you will need to change it for your <databasename>.<tablename>:

SELECT 
    eventtime,
    eventname,
    sourceipaddress,
    useridentity.type as user_type,
    useridentity.principalid,
    useridentity.arn as user_arn,
    useridentity.userName,
    awsregion,
    requestparameters,
    responseelements,
    errorcode,
    errormessage,
    recipientaccountid
FROM cloudtrail_logs.organization_trail
WHERE eventname = 'TerminateInstances'
ORDER BY eventtime DESC;

Most of the fields are very obvious. Notice that userIdentity contains a lot of valuable data, it and we pulled out a few specific fields with the “.” nomenclature. Not every possible field is always present, so be careful how you write queries when searching on identity information.

Scroll around to the right and take a look at the data. Most of the rest, like the eventSource (the service) and eventName (the action) are pretty obvious, but there are two very important fields which can be confusing. requestParameters includes all the parameters in the request (in this case, the instance we are terminating) and the responseElements is everything AWS sent back in response to the API call.

Now here’s a bummer. Although there is sometimes a field for the resource, it’s … almost always missing or empty. You’d think every API call would include its resource (the full ARN, or at least the ID) but it doesn’t work that way. Why? Because sometimes you are making a request with a specific resource ID (e.g., TerminateInstances to terminate this instance) but in other cases you don’t even have an ID yet. For example, RunInstances — it isn’t like we get to pick our instance ID. AWS assigns that and returns it in the response.

Look in the requestParameters column, find the instance ID, and copy that into a text file.

This leads to another common starting point: a specific resource, when you have the ID. When searching all API calls for a resource, if you want all actions on that resource you need to search in both the requestParameters and responseElements!

And both of those fields embed other fields, so here’s how I handle it: the ever powerful “LIKE” search with wildcards! In SQL the ‘%’ sign is the wildcard, not ‘*’!!!

This query below pulls every single action involving that instance, since it’s looking in both requests and responses.

Hit “+” and paste this into Query 2, replacing your instance ID where indicated:

SELECT 
    eventtime,
    eventname,
    eventsource,
    sourceipaddress,
    useridentity.type as user_type,
    useridentity.arn as user_arn,
    useridentity.userName,
    awsregion,
    requestparameters,
    responseelements,
    errorcode,
    errormessage,
    recipientaccountid
FROM cloudtrail_logs.organization_trail
WHERE requestparameters LIKE '%YOUR INSTANCE ID%' 
   OR responseelements LIKE '%YOUR INSTANCE ID%'
ORDER BY eventtime DESC;

This is pretty great. We now see every action on the instance in reverse chronological order. Look at the responseElements field of the RunInstances API call — that’s where AWS first assigned the instance ID. This is why we need to look in both fields. Another example: if I ran a generic DescribeInstances to find one to attack, that would return every instance, even though I never specified an instance ID.

It’s call and response! “Give me an instance!” “Okay, here is the ID for your instance, and everything else I think is important.”

These don’t show very well in screenshots, so explore your results and pay particular attention to the request and the response for both the Run and Terminate actions. Or you can check out the video and get dizzy watching me scroll around.

Now what about all those Describe actions? That’s because we are working in the console, and the console wants to show us stuff, so every time we click into a new screen or view it makes a Describe call to … show us stuff.

To level set, we now have starting points based on an action or based on a resource, but what about the identity?

Looking at our two actions (Run and Terminate) they both use the same identity. We can just query directly on that identity and get a sense of everything that user/role did:

Copy useridentity.arn > Click “+” for Query 3 > Copy and Paste this query > replace with your useridentity.arn:

SELECT 
    eventtime,
    eventname,
    eventsource,
    sourceipaddress,
    awsregion,
    requestparameters,
    responseelements,
    errorcode,
    errormessage,
    recipientaccountid,
    useragent
FROM cloudtrail_logs.organization_trail
WHERE useridentity.arn = 'YOUR ARN'
ORDER BY eventtime DESC;

Very nice. Well, all that looks like normal stuff. Heck, it looks like I’m the one who terminated my very own instance! How about some clues if I think this is nefarious activity?

Well, we can look at all the IP addresses that originated the API calls. We can also look at the useragent and see if it suddenly changes to something different. Changing IP addresses and user agents often indicate use of stolen credentials or sessions.

Keep in mind that although we are querying our Organizations trail, which has all the logs from every account (which is why are queries are slow, so eventually we will need to partition)… this particular user identity query only finds things in one account. Why? Well, we are looking at the activity after assuming a role, and if you review that ARN it has the account ID and a unique role name.

What if we want to look in every account? That’s important when we suspect an attacker is bouncing around accounts with that one base IAM Identity Center user. This makes it tougher to be 100% accurate, but we don’t need 100%. This query goes back to using LIKE and our ‘%’ wildcard, and only specifies the last username.

Copy and Paste into a new query window, and replace “rmogull” with whatever is after the last / in your ARN:

SELECT 
    eventtime,
    eventname,
    eventsource,
    sourceipaddress,
    useridentity.arn as user_arn,
    useridentity.userName,
    awsregion,
    recipientaccountid
FROM cloudtrail_logs.organization_trail
WHERE useridentity.arn LIKE '%/rmogull'
ORDER BY eventtime DESC;

Now scroll to the right and look at the recipientaccountid — you now see the API calls across every account in your org:

And thus we cover our three main security entry points for queries:

• The action

• The resource

• The identity

In future labs we will simulate and trace a full attack, but first we need a better understanding of how to write Athena queries with a focus on the data in CloudTrail logs. Feel free to peek ahead and see some more complex queries in this Securosis blog post.

Key Lesson Points

• Actions, resources, and identities tend to be starting points for security related queries.

• We can use these queries both as threat detectors and for incident analysis (we will cover both in future labs).

• Using the LIKE keyword and a “%resource or other id%” wildcard helps query, even when we don’t know all the exact parameters or response elements.

-Rich

Prerequisites

• Any account with CloudTrail will work, but I recommend completing our Organizations CloudTrail lab first.

The Lesson

Everyone likes to think their job is cool. And if you have “security” in your profession, apparently the more you can use terms out of a James Bond, Jason Bourne, or other special forces movie… the better.

So it’s time to introduce you to SecOps: security operations. I put it in bold so you know how important it is!

The term Security Operations is a bit of a loaded one — it means different things to different people. It can cover anything operational in security, which is a huge scope that spans from running tools and fixing things, through domains including security architecture (design), to security engineering (installing and maintaining security tools). That said, if you go to a conference and see “SecOps” on a banner, they probably mean everything we wrap around incident detection and response.

Look, don’t get too hung up on this stuff. I’ve been in the industry nearly 30 years and try not to get emotionally attached to, I dunno, dictionaries?

But SecOps sounds cool and makes good clickbait for your friendly neighborhood cloud security newsletter/blog, so that’s what I’ll use for our new series of labs exploring the different domains of Security Operations and Cloud Incident Response.

Security Monitoring: Your Feeds and Speeds

When I wrote the Cloud Security Maturity Model, I placed Security Monitoring in the Foundational domain. It’s one of the most important cloud security activities, one I slot just behind IAM. We’ve already set up nearly all our core monitoring, but I never really gave you the full context for how we tie this all together for incident detection, analysis and response.

This is one of those areas I’ve been writing, teaching, and speaking about since before CloudTrail even existed. I call it the Feeds and Speeds talk because effective security monitoring relies on knowing what security telemetry sources to collect, how to collect them, and the time it takes from something happening to someone seeing it. This is a huge deal in cloud since attackers use automated tools which operate lighting fast, and many traditional methods of security monitoring lag 15-60 minutes behind an attack.

Today I’ll limit myself to reviewing the 3 main categories of security feeds:

• Logs are activity streams saved into files stored someplace. So far we’ve set up CloudTrail feeding S3 as our primary log for management plane activity. Another example of logs, which we haven’t used yet, is VPC Flow Logs.

• Events are the real-time stream of activity, alerts, and… uh… events issued by various services. In our labs we have used CloudTrail events (when we need real time, instead of the slower S3 logs), and GuardDuty and Access Analyzer via Security Hub. 

• Configuration(s) are detected misconfigurations. We haven’t set this up yet and will get to it, but there is an entire category of tools called Cloud Security Posture Management (CSPM) to look for these misconfigurations. Many security professionals treat these more like a vulnerability scanner (scan, get report, complain to someone to go fix their sh-t), but savvy incident responders know that most cloud native attackers create misconfigurations to exfiltrate data and do bad things, and we should treat some of these as security alerts.

Each of these provides a security pro with a different piece of the puzzle. Many many services generate logs, and these are our bread and butter for monitoring and detection (even for you gluten-free types). They support everything from fast detection, to deep investigation, to meeting compliance requirements.

Events, as you’ve seen if you’ve competed our previous labs, offer near-real-time security visibility. Logs always have a delay to reach analysis tools because they must be generated, batched, and transferred; while events come in a real-time stream of information (which isn’t saved unless we take action to save it).

Configuration is a bit special, and I don’t want to get distracted by it today. The TL;DR is that posture tells us the outcome of actions, and can identify risky or malicious outcomes such as sharing an S3 bucket to an unknown account.

Log Analysis with Athena

We’ve been collecting CloudTrail logs since our earliest labs, and today we’ll set up a service we can use for threat detection and analysis. Threat detection is the process of identifying potential attacker activity, while analysis is the process of determining what happened and tracing attacks. A threat detector is an automated alert for “oh crap, someone may have broken into the safe,” while analysis helps you figure out whether it was an attacker or just a legit employee who forgot to tell someone they were going into the safe.

In previous labs we showed how to alert directly off a CloudTrail event in EventBridge. That’s great for simple triggers, but analysis and more complex triggers require searching and analyzing logs. That’s where AWS Athena comes in.

Athena is a service which enables you to run SQL-like queries on data in S3. What’s cool is that it works with different types of data — it doesn’t need to be in a structured database. Under the hood Athena uses Presto, which gets quickly into data science above my pay grade. To use it you define a schema. The schema is essentially a logical mapping layer which tells Athena, "when you read these JSON files in S3, interpret field X as a timestamp, field Y as a string, etc." without actually modifying or preprocessing the underlying files.

Athena isn’t free, and there is a lot of nuance to working with it at scale, but we can get into those details later (hello, partitions!). Our costs should be extremely low, especially since we configured S3 so we only keep 3 months of CloudTrail logs.

But there’s one major nuance for our labs: our logs are in LogArchive, but we plan to run security investigations from our SecurityAudit account. The good news is that it’s pretty simple to configure Athena in one account to query a database in a different account, with just a permissions update.

Key Lesson Points

• Different security telemetry feeds have different latency.

• There are three major categories of feeds:

  • Logs are saved to files, and are the most common and detailed.

  • Events are real-time, but aren’t saved unless we capture them when they happen.

  • Configuration “alerts” come from Cloud Security Posture Management tools and tell us when something is misconfigured, which can indicate an attack.

• AWS Athena enables us to directly query logs saved in S3, including CloudTrail, for threat detection and incident analysis.

The Lab

For our lab we will add a permission to our CloudTrail bucket in S3, so we can run Athena in our SecurityAudit account. Then we’ll configure Athena to work with an Organizations CloudTrail, which is a bit different than setting it up in a single account. We’ll finish by running a test query, seeing how crappy the performance is; then I’ll say “partitions later”, since we want to keep things simple for now.

Video Walkthrough

Step-by-Step

First up: for this lab there is a lot of copying and pasting, so open up a blank text file.

Then go to your Sign-in portal > SecurityAudit > IAM > copy and paste the ARN of AWSReservedSSO_SecurityFullAdmin_xxx.

We also need our Organizations ID. Go to Organizations > copy and paste the Org ID:

Really, you want these in a text file for reference later:

NOW CLOSE THAT TAB AND > Sign-in portal > LogArchive > AdministratorAccess > S3 > Click your CloudTrail bucket:

Time for more copy and paste! Properties > Copy and paste the bucket ARN and the bucket name:

My text file now looks like this:

Permissions > Bucket policy > Edit:

We need to add a statement to the policy to allow our SecurityFullAdmin role from SecurityAudit access to this bucket so it can run Athena queries. First up, copy this JSON and paste your role ARN and the bucket ARN in the indicated spots (***replace all this***). You should copy and paste a total of 3 times!

{
    "Sid": "SecurityFullAdminCloudTrailAccess",
    "Effect": "Allow",
    "Principal": {
        "AWS": "***arn of the AWSReservedSSO_SecurityFullAdmin role***"
    },
    "Action": [
        "s3:GetObject",
        "s3:ListBucket",
        "s3:GetBucketLocation"
    ],
    "Resource": [
        "***bucket arn***",
        "***bucket arn***/*"
    ]
},

Then copy your updated JSON and paste it right before the first statement, to look like this:

Now it looks like this, and I highlighted the text I replaced:

Scroll down and click Save Changes.

See? Now I’m pretty!

Normally this is where we would set up an S3 bucket to hold our Athena queries. That's great, but can add to our costs and, as you know, I'm nearly as cheap as all of you. Our costs would have been low, but hey, now we have a free option! AWS just launched (as in, I learned about it yesterday) a new managed query capability which stores queries and results 24 hours for free. This is a great new option for security operations since we tend to either run ad hoc queries for incident investigations, or automated queries for threat detection, and don't need to necessarily store those results long-term.

There's one exception, and it's a big one. When analyzing a real security incident, you absolutely need to save all your query results. But you can export or save them selectively — you don't need to keep every query ever written until the dawn of time (or until someone complains about your storage costs).

NOW CLOSE THAT TAB AND > Sign-in portal > SecurityAudit > SecurityFullAdmin:

Then Go to Athena > Click the "hamburger" (three lines in the upper left corner) > Workgroups > Create workgroup:

Name it security and leave all the defaults, scroll to the bottom, and click Create workgroup. The main defaults we are using are Athena SQL for the query engine, and setting up the brand new very shiny Athena managed query result configuration. This keeps your queries and results for 24 hours free. Why would you want it longer? That’s more for ongoing data analysis stuff we security grunts try to avoid.

Click Query editor:

Okay, we have our data in S3 and our permissions all ready to go, but we need to set Athena up to talk to our bucket ‘o CloudTrail logs, which is hanging out in LogArchive. If you’ve worked with databases before, you are probably used to running SQL queries to do everything — including creating databases and tables. That’s how Athena works; we start by creating our database, then we create our table. When we “create the database”, that doesn't actually create a new database — it actually creates the metadata and mapping to our files in S3. Don't think about it too hard — data stuff is weird, and beyond our scope as security plebes. You’ll paste in each of these commands and then click Run.

First Workgroup > security:

Copy and paste, then Run:

CREATE DATABASE IF NOT EXISTS cloudtrail_logs

Easy, right? Now we need to create our table, which maps our expected JSON fields into table columns. This is the magic of this distributed database stuff. For this one you need to copy this SQL, then swap in your S3 bucket name AND your organization’s ID at the bottom (look for *** on the last line):

CREATE EXTERNAL TABLE IF NOT EXISTS cloudtrail_logs.organization_trail (
    eventversion STRING,
    useridentity STRUCT<
        type: STRING,
        principalid: STRING,
        arn: STRING,
        accountid: STRING,
        invokedby: STRING,
        accesskeyid: STRING,
        userName: STRING,
        sessioncontext: STRUCT<
            attributes: STRUCT<
                mfaauthenticated: STRING,
                creationdate: STRING>,
            sessionissuer: STRUCT<
                type: STRING,
                principalId: STRING,
                arn: STRING,
                accountId: STRING,
                userName: STRING>>>,
    eventtime STRING,
    eventsource STRING,
    eventname STRING,
    awsregion STRING,
    sourceipaddress STRING,
    useragent STRING,
    errorcode STRING,
    errormessage STRING,
    requestparameters STRING,
    responseelements STRING,
    additionaleventdata STRING,
    requestid STRING,
    eventid STRING,
    resources ARRAY<STRUCT<
        ARN: STRING,
        accountId: STRING,
        type: STRING>>,
    eventtype STRING,
    apiversion STRING,
    readonly STRING,
    recipientaccountid STRING,
    serviceeventdetails STRING,
    sharedeventid STRING,
    vpcendpointid STRING
)
ROW FORMAT SERDE 'com.amazon.emr.hive.serde.CloudTrailSerde'
STORED AS INPUTFORMAT 'com.amazon.emr.cloudtrail.CloudTrailInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION 's3://***bucket name***/AWSLogs/***org-id***/'

Take a minute to look at that. You can see that we defined all the potential CloudTrail fields you see in the JSON, then mapped them to strings and stuff. This is the schema.

There's one big flaw in what we just did. As I mentioned, Athena supports partitioning to improve performance and reduce query costs. I went back and forth on implementing partitioning in this first lab, but my wife reminded me that when you are first learning the, queries it's easier not to deal with the added complexity. Partitioning comes with some big complexity demands in our scenario, because once you partition you need to specify more criteria in your queries. Since we only keep 3 months of logs for a relatively small number of accounts, we should be fine on costs and can get by with simpler queries. I'll cover partitioning in a future lab, since it's important to understand.

With all that out of the way, let's run our first query to make sure everything works:

SELECT eventtime, eventname, useridentity.accountid, awsregion
FROM cloudtrail_logs.organization_trail
ORDER BY eventtime DESC
LIMIT 100

Notice how long that took? That's the big downside of not using partitions: Athena needs to scan everything. Costs aren't bad, $5 per terabyte scanned (per query) for our region, but you can see that could add up quickly. Queries also take a very long time to run (about a minute for mine). You tend not to see these issues in a single account setup, but organizations run into them quickly.

Don't worry — once we cover all the query basics we will partition, and then I'll show you how to use lambda functions to automate these queries and use them as threat detectors.

Key Lab Points

• To query S3 in one account using Athena in another account, start by setting up permissions for the role(s) which will access S3.

• Athena now supports managed query storage, which keeps your queries free for 24 hours. You configure it in Workgroup settings.

• Working with Athena is like working with any other database. You use SQL queries for everything.

• Without partitioning Athena can be slow and expensive. We will get to partitioning later.

-Rich

Prerequisites

• All you need are admin credentials in an organization with the OrganizationAccountAccessRole available.

The Lesson

It didn’t fit well into the lesson, but during the lab we’ll be using the command line to explore more like a hacker trying to knock down annoying walls.

I’ve been teaching cloud security for something like 15 years now, and there is nothing I fear more than walking into a classroom, staring at a sea of personal and work laptops, and knowing I need to figure out how to get every one of them to run an SSH terminal so we can use class scripts and commands.

DO YOU POSSIBLY UNDERSTAND THE NIGHTMARE OF FIGURING OUT HOW TO MAKE AN SSH CONNECTION ON A WINDOWS CORPORATE LAPTOP?!?! DO YOU?!?!?!

I… have…scars. And, if you recall our prior use of Session Manager (from, like, a bunch of previous labs) we now have a great way to connect to instances using the web console without ever needing a local terminal program.

While we’ve made those connections to learn skills like managing credentials, we haven’t really used the AWS command line much. Nearly all our work has been using the GUI that is the AWS console — which we fondly call ClickOps because… we click all the things. We have also used a ton of Infrastructure as Code (IaC, via CloudFormation) which is closer to how you want to manage applications and infrastructure stacks.

As your cloud skills mature, you’ll also find yourself using the AWS command line more. Personally I don’t use it a ton — I weirdly tend to swap around between the console, IaC, and Python scripts for work. When I automate I use Python, but there are a lot of you out there who automate primarily with bash or even PowerShell.

Using the command line, and understanding its security implications, is a valuable skill I probably should have included earlier. But hey, better late than never! And it isn’t like most of you are paying for this!

One option is to use Session Manager and connect to an instance, and run commands from there. Although we’ve used this a few times, it still requires a running instance to connect to. Sometimes we just want to use some AWS command lines to manage our account, and don’t necessarily want or need a persistent image. This is where AWS CloudShell comes into play

On CloudShell, the CLI, and Credentials

CloudShell is a terminal which runs right in the AWS console. I don’t know what it is under the hood (okay, poking around, it’s a container), but when you launch it AWS creates a compute environment, connects 1GB of persistent storage, and pre-authenticates using your existing credentials. Just click the little icon, and your shell opens right up:

That environment includes the latest version of the AWS CLI, options for bash or PowerShell, and 1GB of storage, conveniently encrypted with KMS and backed up for you.

What about credentials? Those are loaded up similar to the way an instance gets an IAM role; we will dig in during the lab. Those credentials also use normal AWS credentials precedence. What’s that? Yet another thing I should probably have covered earlier. Whenever you run a tool, like the AWS CLI or an SDK, it pulls credentials in a particular priority order (unless some a-hole programmer decided to handle it some other way, in which case you do not need their piece of crap tool):

1. Direct credentials submitted on the command line or in code as parameters.

2. Environment variables, which we will use today.

3. The AWS credentials file located at ~/.aws/credentials (mature operating systems) or %USERPROFILE\.aws\credentials (Windows. Ugh).

4. The AWS config file, in those same locations.

5. Container or instance profile credentials (IAM roles) or CloudShell credentials (which are basically in a metadata service).

The CLI/SDK just checks each spot until it finds credentials, and uses the first set it finds. If you use a defined profile, it will use the first matching profile it finds, again searching in that order.

CloudShell is great because it never saves your credentials, they aren’t stored where all the nasty malware looks for them, and even if you store (other) credentials yourself they are encrypted. You also can’t SSH or connect to CloudShell from outside the console! It’s a browser-based shell. Unlike your computer, an adversary can’t pop a shell with malware and use your credentials while you’re sleeping.

And for those of you in enterprise environments, you can restrict CloudShell to a VPC. This means it is constrained by all your network security controls (and an attacker can’t just download their attack tools), but it also provides access to your service endpoints (including PrivateLink) in case you need to do network connectivity testing without having to drop an instance into the VPC.

Some pre-lab CLI notes

You’ll see the CLI is pretty straightforward to use (here’s the reference). You type aws <service name> --<request parameters>. You can override the persona/identity (by specifying different credentials or choosing a profile) and the region by setting those as parameters. We’ll do both today so you get to see it.

Here’s an example for running an instance:

aws ec2 run-instances \
  --image-id ami-0123456789example \
  --count 1 \
  --instance-type t2.micro \
  --key-name MyKeyPair \
  --security-group-ids sg-0123456789example \
  --subnet-id subnet-0123456789example \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=MyBasicInstance}]'

Here are my general rules of thumb for how I personally use the options:

• To try new things, or quickly enable/disable/build, I usually start in the console. The visual nature helps me wire the process into my head.

  • Sometimes I use the CLI instead. It’s great for quickly checking the full results of commands, which the console sometimes hides or tucks away in a corner.

  • The CLI is awesome for troubleshooting.

• When I need to create something repeatable, or manage the production version of what I’m building, I use IaC (CloudFormation or Terraform).

• For automation I use Python. Plenty of people use the CLI instead; it’s a personal preference.

• If I want to play with some other command line tool and don’t want to install it locally, I might use CloudShell.

CloudShell is great. Easy, secure, and a wonderful gateway drug to get out of the console and onto the command line.

Key Lesson Points

• CloudShell is a terminal which runs in your browser.

• Behind the scenes it’s a container with persistent storage.

• It’s important to understand credential preference order when using the command line and SDKs.

The Lab

I knew I wanted to do a CloudShell and CLI lab but it took me a while to figure out something interesting with a security message. Don’t worry, I cracked it!

Today we’ll launch a CloudShell in our management account. We’ll pull our credentials from the customized metadata service, then AssumeRole into one of our other accounts, and play with the CLI a bit. We’ll also download a file and peek into CloudTrail to see what CloudShell events look like, in case you ever want to build some threat detectors around it.

Video Walkthrough

Step-by-Step

Start in your Sign-in portal > Copy the Account ID for TestAccount1 > Then CloudSLAW > AdministratorAccess.

Then make sure you are in the Oregon region > launch CloudShell, which is located up on the AWS Console menu bar:

Double-check we are logged in using our role:

aws sts get-caller-identity

Okay, let’s try to grab credentials from the environment variables:

echo $AWS_ACCESS_KEY_ID

Huh. Well that didn’t work. Let’s see all the environment variables:

env

Oh, interesting! There’s a lot of cool information in there (like, this is how I know CloudShell is a container… see if you can spot it).

What’s that one that says “AWS_CONTAINER_CREDENTIALS_FULL_URI=http://127.0.0.1:1338/latest/meta-data/container/security-credentials”? That looks a lot like the metadata service, but instead of being at 169.254.129.254, it’s at localhost. (And is it on port 1338 because it’s one better than 1337?). In case you were wondering, this is not documented anywhere. It’s almost as if AWS doesn’t think you need to use these credentials (you don’t).

Well, I see an endpoint, and maybe it works like the metadata service, and maybe I can pull credentials? Hat tip to Christophe Tafani-Dereeper for doing all the hard work a few years ago (and, uh, I was a bit surprised he gave me some credit). Let’s follow the process we use with v2 of the instance metadata service (remember, this is a two-step process):

TOKEN=$(curl -XPUT localhost:1338/latest/api/token -H "X-aws-ec2-metadata-token-ttl-seconds: 60")

curl localhost:1338/latest/meta-data/container/security-credentials -H "X-aws-ec2-metadata-token: $TOKEN"

Whoo hoo, we cracked the Gibson! Well, the… something. 

Okay, let’s shift gears and start using the CLI. To start, let’s assume the OrganizationAccountAccessRole into TestAccount1. This works because we are logged into CloudShell as an administrator in our management account. I’m putting this as one big code block, so you can copy and paste the entire thing — you just need to paste in your own account ID and review the warning AWS may show (it’s a browser thing):

# Replace ACCOUNT_ID with the target member account ID
ACCOUNT_ID=<YOUR ACCOUNT ID>

# Assume the role and store the credentials in a variable
credentials=$(aws sts assume-role \
  --role-arn arn:aws:iam::$ACCOUNT_ID:role/OrganizationAccountAccessRole \
  --role-session-name cloudshell-org-access)

# Extract and export the temporary credentials
export AWS_ACCESS_KEY_ID=$(echo $credentials | jq -r '.Credentials.AccessKeyId')
export AWS_SECRET_ACCESS_KEY=$(echo $credentials | jq -r '.Credentials.SecretAccessKey')
export AWS_SESSION_TOKEN=$(echo $credentials | jq -r '.Credentials.SessionToken')

# Verify you're now operating in the member account
aws sts get-caller-identity

We’re in! (Note, hacker union rules require that I say “We’re in!” Don’t blame me — I just follow the rules). You can see that we are currently using the OrganizationAccountAccessRole, and it plops CloudShell right into the identity, which means it’s logged.

Notice that CloudShell emphasizes your current region. That doesn’t stop you from using the CLI to run a command in another region. Let’s play a bit:

aws cloudformation describe-stacks

aws cloudformation describe-stacks —region us-east-1

Type q to exit those results.

Now for a warning, if you swap regions in the console, that starts a new CloudShell in that region, and you lose your assumed role. This is why knowing how to change the region as a command-line parameter is still important.

Another cool capability is that your browser is still using the credentials you logged in with, but your shell is using the assumed credentials. This can be very helpful when troubleshooting or testing.

You can also open CloudShell into its own tab. Click the little icon in the upper right corner of the shell:

Okay, now let’s play with files. Get Account Authorization Details pulls down a detailed JSON report of all your IAM permissions in an account. We’ll save the results to a file accessible to our shell:

aws iam get-account-authorization-details > auth-details.json

Now go to Actions > Download File. You might hit the error/warning in my screenshot — if so just click and follow the instructions to open the shell up in a dedicated tab (this depends on your browser).

Type in the file name: auth-details.json:

To finish up hop over the CloudTrail and take a look at your activities. This will print out recent activity every minute. Pretty cool, eh?

# Poll for recent events every minute
while true; do
  aws cloudtrail lookup-events \
    --lookup-attributes AttributeKey=ReadOnly,AttributeValue=false \
    --start-time $(date -d '2 minutes ago' -u +"%Y-%m-%dT%H:%M:%SZ") \
    --max-results 10
  sleep 60
done

Lab Key Points

• CloudShell has its own metadata service for handling credentials, but you should never need to access it directly (it’s undocumented).

• You can assume roles into other accounts (or the same account) if you have permissions and don’t need to bounce around the console.

• The CLI is incredibly useful and powerful, and can even run scripts.

• It even includes Python, along with file upload and download capabilities!

-Rich

Prerequisites

• This is part 9 (the final lab!) of the Advanced Cloud Security Problem Solving series, which I’ve been calling Epic Automation. If you haven’t completed parts 1-6 yet, jump back to the first post in the series and complete all prior posts before trying this one.

The Lesson

This is, I swear, the absolute last lab in our Advanced Cloud Security Problem Solving series! (Epic Automation fit the title card better).

By now you’ve probably realized that I, uh, maybe pulled a fast one on you. It seems we’ve spent these 9 labs learning as much about development and cloud architecture patterns as about cloud security. The thing is, these two skills are impossible to separate if you want to work as a cloud security professional. Just as a firewall engineer needs to understand network topologies, technologies, and how to interpret packet captures… on the cloud side we need to understand our own fair share of non-security fundamentals.

Today’s lab is a very cool one, which I was excited to get up and running. If you recall, way back in the first lab of this series, we defined our objective: “if a bucket is tagged sensitive, only allow access from within my organization.” To do this we spent 8 labs building out an event-driven architecture to would assess any buckets as their configuration changed, and then enforce our desired settings. The diagram looked like this:

So far we’ve built out basically everything except that teeny-tiny little blue box on the left side that says “Time: Every Hour”. And, uh, that little blue box contains a big ol’ can of worms.

Scaling with a Lambda Fanout Pattern

In our little environment we only have one S3 bucket to care about, so running a time-based scan isn’t a big deal. But imagine an organization with many thousands of buckets spread across hundreds of accounts (which I personally have — never mind a big enterprise, a factor of 10 larger than that). Running a scan at that scale, across multiple accounts, would be both massively time consuming and incredibly inefficient.

We don’t call AWS a “hyperscale cloud provider” for nothing!

What we have already built is an event-driven architecture pattern that only runs as things change. No changes? Nothing runs. Modify 10 buckets at once? We invoke 10 lambda functions in parallel. This architecture is incredibly scalable and efficient, and nicely shows off the power of on-demand cloud computing. What makes it so efficient is that we know exactly which bucket changed, and our autoremediation lambda only has to look at that one bucket, and only when it changes.

However, this doesn’t work for time-based scans since we don’t have an event that identifies a changed bucket, so they need to look at every single bucket.

The good news is we have other patterns we can use, and the one we will cover today is a lambda fanout. We will use a series of 3 different lambda functions to identify and then scan every bucket, basically in parallel. Each lambda can trigger other lambdas in a fan pattern to run as much in parallel as possible. Here’s how it works:

• We run an EventBridge Rule which triggers our first lambda every 12 hours (I picked this instead of every hour to save costs).

• That lambda, called enumerate-accounts, takes an environment variable specifying which OU to start in. This is just our Production OU because we defined that as the only place where we want to enforce isolation. enumerate-accounts does two things:

  • It identifies any child OUs. For each child OU it invokes another version of itself. This continues to fan out until there are no more child OUs. Each of these is a separate invocation running a new copy of the lambda, so it’s incredibly fast.

  • It identifies every account in the current OU. For each account, it invokes the enumerate-tagged-buckets lambda function. So if it finds 37 accounts, enumerate-tagged-buckets runs 37 copies in parallel.

• Our second lambda, enumerate-tagged-buckets, uses a very cool little feature: Resource Group Tagging. Instead of having to scan every bucket to find tagged buckets, we can find all tagged buckets with a single API call! This dramatically improves efficiency and, I hate to admit, I had never used it before this lab.

  • This creates a list of all buckets tagged with a key of “classification” and value of “sensitive”.

  • We iterate over that list, and invoke our already-created security-auto-s3 lambda function once for each bucket. This is the third phase of our fanout and all of these run in parallel.

    • We directly invoke the function; we don’t need to create a new EventBridge event to trigger it. However, we create a fake event with the account ID and bucket name so we don’t need to update our existing code!

• security-auto-s3 works exactly the same as if we triggered it from EventBridge, thanks to passing in the synthetic event with the fields it needs to find the bucket.

Even in a large environment this should run in under 30 seconds.

Cool, eh? This architecture allows us to run a time-based scan with a bunch of parallel resources which don’t even run until the clock triggers them. Then they fan out and invoke the next layer of the stack, as efficiently as possible. Here’s what our 3-tier fanout architecture looks like:

But does it really scale? What else do I need to know?

While this pattern scales for even massive environments, this is about the simplest possible implementation, and I would modify it if I was operating it in an enterprise:

• There is no error handling outside some basics in the functions. And you won’t see those if you aren’t checking the logs.

  • At a large enough scale you might hit service limits (AWS limits the number of simultaneous API calls on a service), and we don’t handle these errors or attempt retries (well, until 12 hours later).

• There is no state management, which is one of the main ways we manage errors in event-driven architectures.

• We don’t track or log changes, outside of what you find in CloudTrail logs.

All those are manageable problems, using tools we mix and match like EventBridge, DynamoDB (or ElasticSearch), Simple Queue Service, and Simple Notification Service; and we may get to those someday.

Or just go buy a commercial tool. This is very similar to how our FireMon Cloud Defense platform operates, and most of you in multicloud enterprise environments should strongly consider buying vs. building (and you probably already have something).

Key Lesson Points

• Scheduled scans are more difficult than event-driven triggers since you need to manage scale and respect service limits.

• A lambda “fanout” pattern scales nearly instantly by running multiple, parallel invocations, which directly match the resources you have.

• Fanouts work well with different tiers to match scaling requirements.

  • But don’t forget you might need error handling and state management when using this in an important environment.

The Lab

This would be a lot to build out completely by hand, so I packaged it all up into a CloudFormation template. There are still a few steps to get it up and running, and I’m including all the code here in the blog post so you can review each piece. We have a few steps to make this work:

• We need to delegate administration for AWS Organizations to our SecurityOperations account, so it can read the accounts and OUs. As you will see, we use a restrictive resource policy for only the minimal API calls required.

• We must deploy the main template, which loads the 2 new lambda functions from my public S3 bucket; we also create 2 new roles and the EventBridge Rule which runs every 12 hours.

• We finish by updating our SecurityAutoremediation role using an updated template in CloudFormation StackSets. This step must occur last, since it allows our new enumerate-tagged-buckets lambda to use the existing role. If you recall, we locked this down in a previous lab so only allowed lambda functions can use the role.

  • We also add permission to use the tag:GetResources API to find tagged buckets.

Since all this deploys at once, here is the code for the enumerate-accounts lambda function which finds all accounts in the current OU, then runs another version of itself if there are any child OUs. This is the first step of our fanout:

import os
import boto3

org_client = boto3.client('organizations')
lambda_client = boto3.client('lambda')

def lambda_handler(event, context):
	production_ou = os.environ['production_ou']
	account_ids = []

	# Recursively collect all account IDs under the given OU
	def collect_accounts(parent_ou):
		# List accounts directly under this OU
		paginator = org_client.get_paginator('list_accounts_for_parent')
		for page in paginator.paginate(ParentId=parent_ou):
			for acct in page['Accounts']:
				account_ids.append(acct['Id'])
		# List child OUs and recurse
		ou_paginator = org_client.get_paginator('list_organizational_units_for_parent')
		for ou_page in ou_paginator.paginate(ParentId=parent_ou):
			for ou in ou_page['OrganizationalUnits']:
				collect_accounts(ou['Id'])

	collect_accounts(production_ou)

	# Fan-out: invoke 'find-sensitive-buckets' Lambda for each account
	for account_id in account_ids:
		lambda_client.invoke(
			FunctionName='enumerate-tagged-buckets',
			InvocationType='Event',  # async
			Payload=f'{{"account_id": "{account_id}"}}'
		)

	return {
		'statusCode': 200,
		'body': f'Invoked find-sensitive-buckets for {len(account_ids)} accounts.'
	}

And here’s the code for enumerate-tagged-buckets which finds all tagged buckets, and then runs security-auto-s3 on each of them. This is the second stage of the fanout, and the security-auto-s3 is the third stage. Look for the tag:GetResources command — that’s the magic which saves us from having to list all buckets and then look for all the tagged ones:

import boto3
import os
import json

REGIONS = ["us-east-1", "us-west-2"]

def assume_role(account_id, region):
	sts = boto3.client('sts')
	role_arn = f"arn:aws:iam::{account_id}:role/SecurityOperations/SecurityAutoremediation"
	response = sts.assume_role(
		RoleArn=role_arn,
		RoleSessionName="FindSensitiveBucketsSession"
	)
	creds = response['Credentials']
	return boto3.client(
		'resourcegroupstaggingapi',
		region_name=region,
		aws_access_key_id=creds['AccessKeyId'],
		aws_secret_access_key=creds['SecretAccessKey'],
		aws_session_token=creds['SessionToken']
	)

def lambda_handler(event, context):
	account_id = event['account_id']
	lambda_client = boto3.client('lambda')
	found_buckets = set()

	for region in REGIONS:
		tagging_client = assume_role(account_id, region)
		paginator = tagging_client.get_paginator('get_resources')
		for page in paginator.paginate(
			ResourceTypeFilters=['s3'],
			TagFilters=[{'Key': 'classification', 'Values': ['sensitive']}]
		):
			for resource in page['ResourceTagMappingList']:
				arn = resource['ResourceARN']
				# S3 bucket ARN format: arn:aws:s3:::bucket-name
				bucket_name = arn.split(":::")[-1]
				if bucket_name not in found_buckets:
					found_buckets.add(bucket_name)
					# Mimic a CloudTrail CreateBucket event (minimal fields)
					payload = {
						"account": account_id,
						"detail": {
							"resources": [
								{
									"type": "AWS::S3::Bucket",
									"ARN": arn
								}
							],
							"requestParameters": {
								"bucketName": bucket_name
							},
							"eventSource": "s3.amazonaws.com",
							"eventName": "CreateBucket",
							"recipientAccountId": account_id,
							"accountId": account_id
						}
					}
					lambda_client.invoke(
						FunctionName="security-auto-S3",
						InvocationType="Event",  # async
						Payload=json.dumps(payload)
					)

	return {
		"statusCode": 200,
		"body": f"Invoked S3-secure for {len(found_buckets)} sensitive buckets in account {account_id}."
	}

Okay, let’s jump into the lab:

Video Walkthrough

Step-by-Step

Start in your Sign-in portal > CloudSLAW > AdministratorAccess > Organizations. Copy/Paste the Workloads > Prod OU ID and the SecurityOperations Account ID. You will need both.

Now we need to delegate administration for Organizations to our SecurityOperations account. Still with Organizations > Settings > Delegate administrator for AWS services > Delegate:

Then copy this JSON and paste into the window > Replace ACCOUNTID with your SecurityOperations account ID:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Statement",
      "Effect": "Allow",
      "Principal": {"AWS": "arn:aws:iam::ACCOUNTID:root"},
      "Action": [
        "organizations:ListAccountsForParent",
        "organizations:ListOrganizationalUnitsForParent"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}

Notice that we only include 2 permissions. This is all our code needs (for now) to find child accounts and OUs when given a parent. We can walk down the tree, but we can’t climb up. Maybe we’ll change that later, maybe not. I’m kind of whimsical that way.

Close the tab > Sign-in Portal > SecurityOperations > AdministratorAccess > CloudFormation > Create stack:

Specify template https://cloudslaw.s3-us-west-2.amazonaws.com/lab57.template and name it S3Scanner > paste your Prod OU > click through the rest and Create stack:

Let the stack deploy completely!!!!

This creates:

• A role for each lambda function

• The enumerate-accounts lambda function

  • With the Prod OU ID as an environment variable. This defines the “top” of the tree to scan.

• The enumerate-tagged-buckets lambda function

• The EventBridge rule to run every 12 hours

Now we must wait because our next step will add permission for the enumerate-tagged-buckets lambda function to use our role in the target accounts we created for security-auto-s3. Remember we locked that down so only allowed lambda functions could use it? Yep, time to … uh … allow it.

Did I waste enough time? Good, now we need to go into CloudFormation > StackSets > AutoremediationRole > copy the OU ID:

Then go to Actions > Edit StackSet details:

Replace current template > paste this URL https://cloudslaw.s3-us-west-2.amazonaws.com/lab57-role.template > Next

This update adds permission for the tag:GetResources action, and allows our new lambda to use the role.

Click Next on the page with the name, then Paste in the OU ID you just copied > Add all regions > Next > Submit (which is on the next page):

That should roll out quickly; then it’s time to test!

To test we will run a manual test and review the logs. Go to Lambda > Functions > enumerate-accounts:

Then Test > Test (we don’t need a test event because this is triggered by the clock and doesn’t need any input):

This enumerates all the OUs and accounts, then triggers the enumerate-tagged-buckets function once for every account, then that finds all buckets tagged “sensitive” and runs security-auto-s3. So to see if it works we can skip to the end and check the logs.

Go to CloudWatch > Log Groups > /aws/lambda/security-auto-s3. Click the most recent one (which should match the current day) and you should see it ran successfully. This thing is so lightning fast that if you don’t see logs by the time you read all this and click around, odds are something went wrong so, uh, try again? (Or ask me for help):

Lab Key Points

• You can find all tagged resources without enumerating them individually, using tag:GetResources.

• Before updating permissions you need to create the entity that will get the permissions; otherwise AWS will error out.

• We can manually synthesize an event with the same structure as a CloudTrail event to trigger our existing function without modification.

-Rich

Prerequisites

• This is part 8 of the Advanced Cloud Security Problem Solving series, which I’ve been abbreviating to Epic Automation. If you haven’t completed parts 1-6 yet, jump back to the first post in the series and complete all prior posts before trying this one.

The Lesson

This is it folks, the culmination of months of work! (Or days, for you overachievers blasting through the website). We have everything set up, and today we’ll update permissions and deploy the code to enforce our desired outcome.

Oh, you forgot that? Me too! Let’s review…

If an S3 bucket is tagged with a classification of sensitive, only allow access from within our AWS Organization.

We originally hoped to set this using a Resource Control Policy, but there’s a limitation in IAM and RCP conditions: they don’t support S3 bucket tags. But we (okay, I) decided we could achieve the same objective with an autoremediation guardrail which would trigger whenever anyone tags a bucket or makes permissions changes to a bucket. While policies like SCPs and RCPs are great because they prevent something from happening, they can’t really handle all the complex scenarios we might encounter. For those situations we can use a corrective control which fixes something after the fact… often nearly instantly, thanks to the power of cloud.

Here’s the logic we built out. We monitor CloudTrail for certain API calls, use them as triggers, assess the bucket, then apply corrective controls to lock it down.

To this point we’ve set triggers, linked them in to run our Lambda function (code) to get the bucket’s information, and established our cross-account access and role chaining permissions.

All that’s left is the remediation code itself. And really, there isn’t much more to say about it in terms of a general lesson. But here are some points to note:

• The main body is lambda_handler. This gets the event and some header information (context).

• We extract the account ID of the account where the event occurred, and extract the name of the bucket from the event.

• The function then assumes our target role in the target account, and the get_bucket_info function gathers all the information we want to make our security decisions, including the current tags.

• If the bucket is tagged with classification:sensitive, then go “do things”.

  • Turn on Block Public Access (BPA), disable ACLs, and check the bucket policy.

  • Retrieve the bucket policy (if one exists) and add a statement to the bucket policy to restrict access to the current organization and AWS services.

    • For this to work the function needs to know your Organization ID. Since this account (SecurityOperations) does not have permissions for Organizations, it doesn’t have a way to pull the ID. Instead we will set it as an environment variable.

To make the lab easier to understand, I wrote all the code in a single that you can copy and paste into the console. The code is well-commented and should be easy to follow even if you are new to Python. Start in lambda_handler at the bottom, and you can walk through the logic and the various functions.

This should work well in a lot of situations, but it doesn’t have any IP address based permissions. This means it blocks all IP-based access, and only allows API-based access! We could write more complex code which could evaluate IP addresses and allow them if they are private-network-only, or from a specific IP address range, but that was beyond our current requirements. Just keep that very important limitation in mind if you use this technique yourself.

Another quick note on the code: you’ll notice that I don’t rely on any bucket information in the event, other than the bucket name and account ID. Every time I create an autoremediation, I always check the current state of the resource before making any changes. Why? Because life comes at you fast, especially in cloud, and I always want to operate knowing the real-time state of a resource.

Also, after my first pass I needed to go back through and make sure that the code only makes changes if the expected security isn’t in place. Why? The first version (in one spot) just set the secure condition every time, even if it was already there. But this change API call was picked up as one of our triggers, which ran the lambda again, which made the change again, and… infinite loops in the cloud are fun!

When you play battle bots in the cloud, only Amazon’s accounting department wins!

As I mentioned above, it took me about an hour to code up all the remediations with the help of an AI tool. I used our test event to trigger each test, but I specified a target bucket I could safely modify to play with different conditions. You can check out the Pro content in the Patreon for a deeper walkthrough, and to see how I built everything.

Finally, to make this work we need to update permissions and set that environment variable… both of which are in the lab.

Key Lesson Points

• Always test your code on real resources in a test environment whenever you can. Local testing stacks are great, but don’t totally rely on them.

• Consider using multiple browsers (or terminal shells) logged into different accounts so you can test without having to bounce around too much.

  • I show that in the Pro content, but we didn’t really need it for this lab. For deeper testing I write command line scripts instead of using the browser.

• Watch out for infinite loops: make sure your changes don’t continuously trigger your app code.

  • There’s a different architecture that can handle this a little better which we didn’t use: tracking state with SQS and Dynamo Database. That’s much more complex to set up correctly, and the small amount of re-triggering this architecture performs won’t increase costs.

The Lab

We will do 3 things in this lab, and I recommend an optional 4th step:

1. Get our org ID and set it as an environment variable for our Lambda function.

2. Update our code with the new code I wrote that actually sets the security conditions.

3. Update our automation role’s permissions, now that we’ve finished the code and know exactly what it will do.

4. Make changes to our test bucket and watch the magic work!

Video Walkthrough

Step-by-Step

Start in your Sign-in portal > CloudSLAW > AdministratorAccess > Organizations >copy your Organization ID:

Close that tab > Sign-in portal > SecurityOperations > AdministratorAccess > Lambda > security-auto-s3:

Go to Configuration > Environment variables > Edit:

Then Add environment variable (click that once to show the text fields) and Key: org_id; Value: your organization’s ID > Save:

Then go to Code > copy and paste all the code below to replace the existing code > Deploy:

import json
import boto3
import re
import logging
from botocore.exceptions import ClientError
import os

# Set up logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)

def assume_role(account_id):
    """
    Assume the SecurityAutoremediation role in the target account
    """
    role_arn = f"arn:aws:iam::{account_id}:role/SecurityOperations/SecurityAutoremediation"
    logger.info(f"Assuming role: {role_arn}")
    
    sts_client = boto3.client('sts')
    
    try:
        response = sts_client.assume_role(
            RoleArn=role_arn,
            RoleSessionName="S3SecurityAnalysis"
        )
        
        credentials = response['Credentials']
        
        return {
            'aws_access_key_id': credentials['AccessKeyId'],
            'aws_secret_access_key': credentials['SecretAccessKey'],
            'aws_session_token': credentials['SessionToken']
        }
    except Exception as e:
        logger.error(f"Failed to assume role {role_arn}: {str(e)}")
        raise

def extract_bucket_name(event):
    """
    Extract S3 bucket name from event by checking multiple possible locations
    """
    # Check in resources array
    if 'resources' in event['detail'] and event['detail']['resources']:
        for resource in event['detail']['resources']:
            if resource.get('type') == 'AWS::S3::Bucket' and resource.get('ARN'):
                arn = resource.get('ARN')
                bucket_match = re.search(r'arn:aws:s3:::([^/]+)', arn)
                if bucket_match:
                    return bucket_match.group(1)
    
    # Check in requestParameters
    if 'requestParameters' in event['detail']:
        req_params = event['detail']['requestParameters']
        if req_params and 'bucketName' in req_params:
            return req_params['bucketName']
    
    # Check in responseElements
    if 'responseElements' in event['detail'] and event['detail']['responseElements']:
        resp_elements = event['detail']['responseElements']
        # Various response element formats
        if 'x-amz-bucket-region' in resp_elements:
            bucket_match = re.search(r'arn:aws:s3:::([^/]+)', 
                                    str(resp_elements))
            if bucket_match:
                return bucket_match.group(1)
    
    return None

def get_bucket_info(bucket_name, session):
    """
    Get comprehensive information about an S3 bucket using the provided session
    """
    s3_client = session.client('s3')
    sts_client = session.client('sts')
    
    # Get the account ID from the assumed role session
    account_id = sts_client.get_caller_identity().get('Account')
    
    result = {
        'bucket_name': bucket_name,
        'arn': f'arn:aws:s3:::{bucket_name}',
        'account_id': account_id,
        'tags': None,
        'has_bucket_policy': False,
        'bucket_policy': None,
        'acls_enabled': True,  # Default to True, will check if acl is "private"
        'acls': None,
        'block_public_access_bucket': None
    }
    
    # Get bucket tags
    try:
        tags_response = s3_client.get_bucket_tagging(Bucket=bucket_name)
        result['tags'] = tags_response.get('TagSet', [])
    except ClientError as e:
        if e.response['Error']['Code'] == 'NoSuchTagSet':
            result['tags'] = []
        else:
            logger.warning(f"Error getting bucket tags: {str(e)}")
    
    # Get bucket policy
    try:
        policy_response = s3_client.get_bucket_policy(Bucket=bucket_name)
        result['has_bucket_policy'] = True
        result['bucket_policy'] = json.loads(policy_response.get('Policy', '{}'))
    except ClientError as e:
        if e.response['Error']['Code'] == 'NoSuchBucketPolicy':
            result['has_bucket_policy'] = False
        else:
            logger.warning(f"Error getting bucket policy: {str(e)}")
    
    # Get bucket ACLs and determine if ACLs are enabled
    try:
        acl_response = s3_client.get_bucket_acl(Bucket=bucket_name)
        result['acls'] = acl_response.get('Grants', [])
        
        # Check bucket ownership controls to determine if ACLs are enabled
        try:
            ownership = s3_client.get_bucket_ownership_controls(Bucket=bucket_name)
            rules = ownership.get('OwnershipControls', {}).get('Rules', [])
            for rule in rules:
                if rule.get('ObjectOwnership') == 'BucketOwnerEnforced':
                    result['acls_enabled'] = False
                    break
        except ClientError as e:
            if e.response['Error']['Code'] != 'OwnershipControlsNotFoundError':
                logger.warning(f"Error getting bucket ownership: {str(e)}")
    except ClientError as e:
        logger.warning(f"Error getting bucket ACL: {str(e)}")
    
    # Get block public access settings for bucket
    try:
        bpa_response = s3_client.get_public_access_block(Bucket=bucket_name)
        result['block_public_access_bucket'] = bpa_response.get('PublicAccessBlockConfiguration', {})
    except ClientError as e:
        if e.response['Error']['Code'] != 'NoSuchPublicAccessBlockConfiguration':
            logger.warning(f"Error getting bucket public access block: {str(e)}")
    
    return result

def remediate_bpa(session: boto3.Session, bucket_name: str) -> dict:
    """
    Enable S3 Block Public Access (BPA) for a specified S3 bucket.
    
    Args:
        session (boto3.Session): The boto3 session with assumed role credentials
        bucket_name (str): The name of the S3 bucket
        
    Returns:
        dict: Response from the API call
    """
    try:
        s3_client = session.client('s3')
        
        response = s3_client.put_public_access_block(
            Bucket=bucket_name,
            PublicAccessBlockConfiguration={
                'BlockPublicAcls': True,
                'IgnorePublicAcls': True,
                'BlockPublicPolicy': True,
                'RestrictPublicBuckets': True
            }
        )
        
        logger.info(f"Successfully enabled Block Public Access for bucket: {bucket_name}")
        return response
        
    except Exception as e:
        logger.error(f"Failed to enable Block Public Access for bucket {bucket_name}: {str(e)}")
        raise e

def remediate_acls(session: boto3.Session, bucket_name: str) -> dict:
    """
    Disable ACLs for a specified S3 bucket by setting ObjectOwnership to BucketOwnerEnforced.
    Only makes changes if ACLs are currently enabled.
    
    Args:
        session (boto3.Session): The boto3 session with assumed role credentials
        bucket_name (str): The name of the S3 bucket
        
    Returns:
        dict: Response from the API call or None if no change needed
    """
    try:
        s3_client = session.client('s3')
        
        # Check current ownership controls
        try:
            ownership = s3_client.get_bucket_ownership_controls(Bucket=bucket_name)
            rules = ownership.get('OwnershipControls', {}).get('Rules', [])
            for rule in rules:
                if rule.get('ObjectOwnership') == 'BucketOwnerEnforced':
                    logger.info(f"ACLs already disabled for bucket: {bucket_name}")
                    return None
        except ClientError as e:
            if e.response['Error']['Code'] != 'OwnershipControlsNotFoundError':
                raise
        
        # If we get here, either no ownership controls exist or they're not BucketOwnerEnforced
        response = s3_client.put_bucket_ownership_controls(
            Bucket=bucket_name,
            OwnershipControls={
                'Rules': [
                    {
                        'ObjectOwnership': 'BucketOwnerEnforced'
                    }
                ]
            }
        )
        
        logger.info(f"Successfully disabled ACLs for bucket: {bucket_name}")
        return response
        
    except Exception as e:
        logger.error(f"Failed to disable ACLs for bucket {bucket_name}: {str(e)}")
        raise e

def remediate_bucket_policy(session: boto3.Session, bucket_name: str, current_policy: dict) -> dict:
    """
    Check bucket policy size and add organization restriction for sensitive buckets.
    Allows AWS service principals when acting on behalf of the organization.
    Only updates policy if needed.
    
    Args:
        session (boto3.Session): The boto3 session with assumed role credentials
        bucket_name (str): The name of the S3 bucket
        current_policy (dict): The current bucket policy or empty dict
        
    Returns:
        dict: Response from the API call or None if no change needed
    """
    try:
        s3_client = session.client('s3')
        org_id = os.environ.get('org_id')
        
        if not org_id:
            raise ValueError("org_id environment variable is not set")
        
        # Create org deny statement with service principal exception
        org_deny_statement = {
            "Sid": "DenyAccessOutsideOrg",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:*",
            "Resource": [
                f"arn:aws:s3:::{bucket_name}",
                f"arn:aws:s3:::{bucket_name}/*"
            ],
            "Condition": {
                "StringNotEquals": {
                    "aws:PrincipalOrgID": org_id
                },
                "StringNotLike": {
                    "aws:PrincipalArn": [
                        "arn:aws:iam::*:role/aws-service-role/*",
                        "arn:aws:iam::*:role/service-role/*"
                    ]
                },
                "Bool": {
                    "aws:PrincipalIsAWSService": "false"
                }
            }
        }
        
        # Check if org restriction already exists
        org_restriction_exists = any(
            statement.get('Sid') == "DenyAccessOutsideOrg" and
            statement.get('Effect') == "Deny" and
            statement.get('Condition', {}).get('StringNotEquals', {}).get('aws:PrincipalOrgID') == org_id
            for statement in current_policy.get('Statement', [])
        )
        
        if org_restriction_exists:
            logger.info(f"Organization restriction already exists in bucket policy for: {bucket_name}")
            return None
            
        # If we get here, we need to add the org restriction
        # Ensure policy has basic structure
        if not current_policy:
            current_policy = {
                "Version": "2012-10-17",
                "Id": f"SecurityPolicy-{bucket_name}",
                "Statement": [org_deny_statement]
            }
        else:
            # Ensure policy has correct version and ID
            current_policy['Version'] = "2012-10-17"
            if 'Id' not in current_policy:
                current_policy['Id'] = f"SecurityPolicy-{bucket_name}"
            if 'Statement' not in current_policy:
                current_policy['Statement'] = []
            current_policy['Statement'].append(org_deny_statement)
        
        # Check policy size before applying
        policy_size = len(json.dumps(current_policy))
        if policy_size > 19456:  # 19K in bytes
            logger.error(f"Bucket policy for {bucket_name} is too large ({policy_size} bytes). Cannot add org restriction.")
            return None
            
        # Apply updated policy
        response = s3_client.put_bucket_policy(
            Bucket=bucket_name,
            Policy=json.dumps(current_policy)
        )
        
        logger.info(f"Successfully added organization restriction to bucket policy for: {bucket_name}")
        return response
        
    except Exception as e:
        logger.error(f"Failed to update bucket policy for bucket {bucket_name}: {str(e)}")
        raise e

def lambda_handler(event, context):
    """
    Main Lambda handler function
    """
    try:
        logger.info(f"Processing event: {json.dumps(event)}")
        
        # Extract account ID from the event
        if 'account' in event:
            account_id = event['account']
        else:
            logger.error("Could not extract account ID from event")
            return {
                'statusCode': 400,
                'body': json.dumps('Could not extract account ID from event')
            }
        
        logger.info(f"Extracted account ID: {account_id}")
        
        # Assume role in the target account
        try:
            credentials = assume_role(account_id)
            session = boto3.Session(**credentials)
        except Exception as e:
            logger.error(f"Failed to create cross-account session: {str(e)}")
            return {
                'statusCode': 500,
                'body': json.dumps(f'Failed to create cross-account session: {str(e)}')
            }
        
        # For EventBridge events, format is slightly different
        if 'detail' in event:
            # Extract bucket name from CloudTrail event
            bucket_name = extract_bucket_name(event)
        else:
            # Direct S3 event format
            bucket_name = extract_bucket_name({'detail': event})
        
        if not bucket_name:
            logger.error("Could not extract bucket name from event")
            return {
                'statusCode': 400,
                'body': json.dumps('Could not extract bucket name from event')
            }
        
        logger.info(f"Extracted bucket name: {bucket_name}")
        
        # Get information about the bucket using the cross-account session
        bucket_info = get_bucket_info(bucket_name, session)
        logger.info(f"Results: {bucket_info}")
        
        # Check if bucket has sensitive classification tag
        is_sensitive = any(
            tag.get('Key') == 'classification' and tag.get('Value') == 'sensitive'
            for tag in bucket_info.get('tags', [])
        )
        
        if is_sensitive:
            logger.info(f"Bucket {bucket_name} is tagged as sensitive")
            
            # Check if BPA is already fully enabled
            current_bpa = bucket_info.get('block_public_access_bucket', {})
            bpa_fully_enabled = all([
                current_bpa.get('BlockPublicAcls', False),
                current_bpa.get('IgnorePublicAcls', False),
                current_bpa.get('BlockPublicPolicy', False),
                current_bpa.get('RestrictPublicBuckets', False)
            ])
            
            if not bpa_fully_enabled:
                logger.info(f"Enabling Block Public Access for sensitive bucket {bucket_name}")
                remediation_response = remediate_bpa(session, bucket_name)
                bucket_info['remediation_response'] = remediation_response
            else:
                logger.info(f"Block Public Access already fully enabled for bucket {bucket_name}")
                
            # Check if ACLs are enabled and disable them if they are
            if bucket_info.get('acls_enabled', True):
                logger.info(f"Disabling ACLs for sensitive bucket {bucket_name}")
                acl_response = remediate_acls(session, bucket_name)
                bucket_info['acl_remediation_response'] = acl_response
            else:
                logger.info(f"ACLs already disabled for bucket {bucket_name}")
            
            # Add or update bucket policy with org restrictions
            current_policy = bucket_info.get('bucket_policy', {}) if bucket_info.get('has_bucket_policy') else {}
            policy_response = remediate_bucket_policy(session, bucket_name, current_policy)
            if policy_response:
                bucket_info['policy_remediation_response'] = policy_response
        else:
            logger.info(f"Bucket {bucket_name} is not tagged as sensitive")
            
        return {
            'statusCode': 200,
            'body': json.dumps(bucket_info, default=str)
        }
        
    except Exception as e:
        logger.error(f"Error processing event: {str(e)}")
        return {
            'statusCode': 500,
            'body': json.dumps(f'Error: {str(e)}')
        }

The code is pretty well commented. And yes, I used AI for a lot of it (I haz a day job, ya know!). To follow the logic, start in lambda_handler; you can see the flow and how we extract the account ID and the bucket name, then assumerole into the account, then pull the bucket data, then make any remediations if the tag is present.

You can hit Test now if you want, but even if your bucket is tagged sensitive, not all the remediations will work yet, because we need to fix permissions.

Let’s go fix those permissions! We’ll deploy a new StackSet with an updated template.

CloudFormation > StackSets > Service-managed > AutoremediationRole:

The first thing we need is the OU ID where the stack is deployed. Copy the OU ID and paste it into a text editor (we’ll also need to copy the URL to the updated template in a second):

Then Actions > Edit StackSet details:

Select Replace current template, then under Specify template, paste in this URL: https://cloudslaw.s3-us-west-2.amazonaws.com/lab56.template, then click Next:

Click Next on the next 2 screens, then Paste in the OU and select All regions (or just US East (Virginia)). Then Next > Submit:

It took me about 3 minutes for the template to update. As a reminder, this new template is the same as the previous two, but with the new permissions added and some unneeded ones removed.

Now it’s time to test things out. We’ll do this in the Production1 account, so we need to switch over to that.

Start in your Sign-in portal > CloudSLAW > AdministratorAccess. We need to go back to our Production1 account and change a tag on our bucket to a sample event to work with. (If you still have one in your email you can use that and skip this part). If you still have your account in your Role history, then just click it. If not here are the screenshots to switch roles (which is just assumerole from your AdministratorAccess role into the OrganizationAccountAccessRole in that account… yep, a simple role chain!).

That was just a reminder in case you didn’t have the account in your Role history.

We need to make two changes to ensure we can test the remediations, then we’ll change the tag on the instance to trigger the autoremediation and see it in action. I’m going to describe what to do and then show key screenshots — this should be easy to follow:

• Disable block public access

  

• Change Object Ownership to ACLs enabled

  

Time to test! Go to Properties > Tags > Edit > add a tag with the key “classification” and value “sensitive”:

Then go back to Permissions and refresh the page in about 15 seconds. You should now see that BPA is enabled, there’s a restrictive bucket policy, and ACLs are disabled!!!

I’m sorry, this is just really freaking cool! You have just implemented an advanced, reusable pattern for security autoremediation which operates from a central account!

Lab Key Points

• This is cool

• I write too much sometimes

-Rich

Prerequisites

• This is part 7 of the Advanced Cloud Security Problem Solving series, which I’ve been abbreviating to Epic Automation. If you haven’t completed parts 1-6 yet, jump back to the first post in the series and complete all prior posts before trying this one.

The Lesson

This lesson is all about the hands-on. We’ve covered the principles, so now it’s time to start implementing. We will focus on two mechanics:

• How to implement a role chain in code

• How to test Lambda function in the console and work through error messages.

Let’s quickly cover each of these, and then get into the meat of this lab.

Role Chaining in Code

As a reminder, a role chain is when you use one role to assume another role. In our last lab we learned how to define permissions to secure a role chain. With that out of the way, everything is in place to assume a role. This isn’t the first time you role chained: in previous labs when we logged into IAM Identity Center and then used the little Switch Role link in the upper-right corner of the console, that was technically chaining from our Identity Center role into OrganizationAccountAccessRole in the target account. But stuff in the console is a bit magical and abstracted — now we’ll see the guts.

The trick in code is that you are creating a session. You can code in multiple sessions, and then pick which to use when making an API call. Heck, you could set up 8 different sessions, go all Doc Oc, and do different things in different accounts simultaneously.

We always need a base identity. In our case, that’s the default lambda execution role. If we assign a role to a lambda function, the lambda service just assumes it for us when our function runs. Any API call that doesn’t specify another session uses the default lambda execution role.

After that, we need to manually assume roles in our code. Assuming a role retrieves temporary credentials used to make API calls.

Here’s the function in our code to assume a role in another account. Since we use this to assume roles in multiple accounts, but the role name is always the same, this little lambda function just needs the Account ID of the account to jump into; it already knows the role name. Boto3 is the name of the python library for AWS — don’t let that confuse you!

It’s doing the following:

• Get an account ID to use when running,

• Create the full “role_arn”: the ARN of the role we will assume, including the account ID and path/name.

• Start up a connection (boto3.client) to the security token service (which we use to assume roles).

• Assume the target role (role_arn) and sets a session name as S3SecurityAnalysis. You can code in whatever session name you want, which shows up in the CloudTrail logs.

• sts.assumerole returns some stuff; we pull out the temporary credentials (Access Key, Secret Access Key, and Session Token).

• The function returns those credentials.

That’s the key point: assuming a role retrieves temporary credentials, which we use to make API calls; we still have credentials, they are just ephemeral.

Here’s where it gets a bit tricky. This process gives us temporary credentials, but we need to create the session. What’s that? Basically it’s a container (object) in the code which holds the credentials, region, etc. for the actual API calls. This function doesn’t reconnect with AWS — it just says, “Create a session object that stores the credentials which I can use for API calls.” Remember that a “session” on the AWS side is just “a set of credentials which expire”, not some persistent pipe.

The “session” object in boto3 is just the container, so you don’t need to specify the keys and configuration every time you make an actual API call.

# Assume role in the target account
        try:
            credentials = assume_role(account_id)
            session = boto3.Session(**credentials)
        except Exception as e:
            logger.error(f"Failed to create cross-account session: {str(e)}")
            return {
                'statusCode': 500,
                'body': json.dumps(f'Failed to create cross-account session: {str(e)}')
            }

Now we have our credentials in a little object we can use for API calls with lines like s3_client = session.client('s3') — which sets us up to talk to the S3 service. The boto3 Software Development Kit (SDK) converts a command like s3_client.get_bucket_policy(Bucket=bucket_name) into an encrypted request for the bucket policy, packages it up, and sends it to the S3 service.

Phew. This is one of those situations where I don’t want to overcomplicate or oversimplify. It’s a very elegant system — especially once you go deep into the weeds to see how AWS processes all these events. There is crazy stuff like service, region, and even daily keys under the covers. Let’s just say these requests aren’t just decrypted and shared on the back end of AWS… everything is really locked down at every level.

Okay, one last attempt to keep it simple:

• You start with a base identity — otherwise you can’t even talk to the Security Token Service (STS) to try assuming a role.

• You make the assumerole API call, which returns a set of temporary credentials you can use for a session (until they expire).

• The boto3.session API call creates a reusable object in code which holds those credentials and makes it easier to use them. (Among other things, when they expire we only need to update the session object once, instead of in all the clients).

• session.client(service) sets us up to make an API call to a service. The API call itself gets wrapped up as a secure letter, using the keys retrieved with assumerole.

In our code we only have one session, so we call it “session”. I’ve written code where I need to juggle multiple sessions to multiple accounts, and I just name the session so I can track which one I’m using. E.g. “session_logs”, “session_database”, “session_breakthings”.

Testing Lambda Code

Wow, this is already longer than I planned.

Okay… to test a lambda there are the “I’m a real developer” options, and the “I’m a hacker/security person who just slaps stuff together” options.

We’ll go with the latter. Duh.

If you are doing full-time and big project dev work, you probably want to use a local testing framework like LocalStack or AWS SAM CLI. For our purposes we will run tests in the console with a built-in capability.

Last lab we made changes which triggered our function, and then we looked in the logs to see what broke. That is inefficient, and testing in the console with test events is a much better option. So, yeah, we’ll do that.

Key Lesson Points

• When you assume a role, that asks the Security Token Service to give you a set of temporary credentials which work for a set time — we call that a session.

• In code, you need to specify those credentials for those API calls.

• To make this easier, the SDKs for different programming languages allow you to create an object called a “session” which holds those credentials, so you don’t need to specify them every time.

  • The Python library is called Boto3. No, I don’t know why. Yes, I could ask one of my AIs to figure it out. No, I’m not going to.

• You can test lambda functions offline using special tools, but you can also load a test event into the console and use the feature there.

The Lab

We will:

1. Generate an event in our Production1 account, which we can use as a test event for our lambda function.

2. Create the test event in the console.

3. Update our function with the new code Rich wrote, which includes the role chain and some API calls to gather information about the S3 bucket. Rich promises he wrote it all himself and didn’t use AI. (Rich lies).

4. Run the test, and then fix 2 problems with the code.

Video Walkthrough

Step-by-Step

Start in your Sign-in portal > CloudSLAW > AdministratorAccess. We need to go back to our Production1 account and change a tag on our bucket to a sample event to work with. (If you still have one in your email you can use that, and skip this part). If you still have your account in your Role history then just click it. If not, here are the screenshots to switch roles (which is just assumerole from your AdministratorAccess role into the OrganizationAccountAccessRole in that account… yep, a simple role chain!).

That was just a reminder in case you didn’t have the account in your Role history.

Okay, let’s go into S3 and change a tag on our bucket. Since we’ve done this already I’ll just shortcut it, showing the main screenshots in order. I mean, you’ve been with me over a year at this point, so I have a lot of confidence in you. 😀 

After you Save changes, go and close the tab. Then check your email for the event. Remember we left the EventBridge rule in place to email you events? It’s almost like I planned some things for once!!! Take that event, copy it, and paste it into a text editor (we won’t edit it, but we need a copy we can get easily). Also, only copy the JSON text! Don’t leave any spaces or anything:

Now let’s go into our Lambda function and update the code. Sign-in portal > SecurityOperations> AdministratorAccess > Lambda > security-auto-s3

Click Test, then name it S3Test, and paste in your test event. To make sure it’s a proper event click Format JSON — it’s easy to make mistakes like missing a character, and this helps catch those. Then Save.

We need to swap in our new code. Although I focused on role chaining in the Lesson section, here’s what it all does:

1. It pulls the account ID from the event, so it knows which account generated the event.

2. It pulls the bucket ARN from the event, so it knows which bucket changed.

3. In assumes the SecurityAutoremediation role in the identified target account.

4. It gathers some information about the S3 bucket, which we will use later to analyze the security posture.

Click Code. Here’s our new code. Copy this and paste it over the existing code > Deploy:

import json
import boto3
import re
import logging
from botocore.exceptions import ClientError

# Set up logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)

def assume_role(account_id):
    """
    Assume the SecurityAutoremediation role in the target account
    """
    role_arn = f"arn:aws:iam::{account_id}:role/SecurityOperations/SecurityAutoremediation"
    logger.info(f"Assuming role: {role_arn}")
    
    sts_client = boto3.client('sts')
    
    try:
        response = sts_client.assume_role(
            RoleArn=role_arn,
            RoleSessionName="S3SecurityAnalysis"
        )
        
        credentials = response['Credentials']
        
        return {
            'aws_access_key_id': credentials['AccessKeyId'],
            'aws_secret_access_key': credentials['SecretAccessKey'],
            'aws_session_token': credentials['SessionToken']
        }
    except Exception as e:
        logger.error(f"Failed to assume role {role_arn}: {str(e)}")
        raise

def extract_bucket_name(event):
    """
    Extract S3 bucket name from event by checking multiple possible locations
    """
    # Check in resources array
    if 'resources' in event['detail'] and event['detail']['resources']:
        for resource in event['detail']['resources']:
            if resource.get('type') == 'AWS::S3::Bucket' and resource.get('ARN'):
                arn = resource.get('ARN')
                bucket_match = re.search(r'arn:aws:s3:::([^/]+)', arn)
                if bucket_match:
                    return bucket_match.group(1)
    
    # Check in requestParameters
    if 'requestParameters' in event['detail']:
        req_params = event['detail']['requestParameters']
        if req_params and 'bucketName' in req_params:
            return req_params['bucketName']
    
    # Check in responseElements
    if 'responseElements' in event['detail'] and event['detail']['responseElements']:
        resp_elements = event['detail']['responseElements']
        # Various response element formats
        if 'x-amz-bucket-region' in resp_elements:
            bucket_match = re.search(r'arn:aws:s3:::([^/]+)', 
                                    str(resp_elements))
            if bucket_match:
                return bucket_match.group(1)
    
    return None

def get_bucket_info(bucket_name, session):
    """
    Get comprehensive information about an S3 bucket using the provided session
    """
    s3_client = session.client('s3')
    sts_client = session.client('sts')
    
    # Get the account ID from the assumed role session
    account_id = sts_client.get_caller_identity().get('Account')
    
    result = {
        'bucket_name': bucket_name,
        'arn': f'arn:aws:s3:::{bucket_name}',
        'account_id': account_id,
        'tags': None,
        'has_bucket_policy': False,
        'bucket_policy': None,
        'acls_enabled': True,  # Default to True, will check if acl is "private"
        'acls': None,
        'block_public_access_bucket': None
    }
    
    # Get bucket tags
    try:
        tags_response = s3_client.get_bucket_tagging(Bucket=bucket_name)
        result['tags'] = tags_response.get('TagSet', [])
    except ClientError as e:
        if e.response['Error']['Code'] == 'NoSuchTagSet':
            result['tags'] = []
        else:
            logger.warning(f"Error getting bucket tags: {str(e)}")
    
    # Get bucket policy
    try:
        policy_response = s3_client.get_bucket_policy(Bucket=bucket_name)
        result['has_bucket_policy'] = True
        result['bucket_policy'] = json.loads(policy_response.get('Policy', '{}'))
    except ClientError as e:
        if e.response['Error']['Code'] == 'NoSuchBucketPolicy':
            result['has_bucket_policy'] = False
        else:
            logger.warning(f"Error getting bucket policy: {str(e)}")
    
    # Get bucket ACLs and determine if ACLs are enabled
    try:
        acl_response = s3_client.get_bucket_acl(Bucket=bucket_name)
        result['acls'] = acl_response.get('Grants', [])
        
        # Check bucket ownership controls to determine if ACLs are enabled
        try:
            ownership = s3_client.get_bucket_ownership_controls(Bucket=bucket_name)
            rules = ownership.get('OwnershipControls', {}).get('Rules', [])
            for rule in rules:
                if rule.get('ObjectOwnership') == 'BucketOwnerEnforced':
                    result['acls_enabled'] = False
                    break
        except ClientError as e:
            if e.response['Error']['Code'] != 'OwnershipControlsNotFoundError':
                logger.warning(f"Error getting bucket ownership: {str(e)}")
    except ClientError as e:
        logger.warning(f"Error getting bucket ACL: {str(e)}")
    
    # Get block public access settings for bucket
    try:
        bpa_response = s3_client.get_public_access_block(Bucket=bucket_name)
        result['block_public_access_bucket'] = bpa_response.get('PublicAccessBlockConfiguration', {})
    except ClientError as e:
        if e.response['Error']['Code'] != 'NoSuchPublicAccessBlockConfiguration':
            logger.warning(f"Error getting bucket public access block: {str(e)}")
    
    return result

def lambda_handler(event, context):
    """
    Main Lambda handler function
    """
    try:
        logger.info(f"Processing event: {json.dumps(event)}")
        
        # Extract account ID from the event
        if 'account' in event:
            account_id = event['account']
        else:
            logger.error("Could not extract account ID from event")
            return {
                'statusCode': 400,
                'body': json.dumps('Could not extract account ID from event')
            }
        
        logger.info(f"Extracted account ID: {account_id}")
        
        # Assume role in the target account
        try:
            credentials = assume_role(account_id)
            session = boto3.Session(**credentials)
        except Exception as e:
            logger.error(f"Failed to create cross-account session: {str(e)}")
            return {
                'statusCode': 500,
                'body': json.dumps(f'Failed to create cross-account session: {str(e)}')
            }
        
        # For EventBridge events, format is slightly different
        if 'detail' in event:
            # Extract bucket name from CloudTrail event
            bucket_name = extract_bucket_name(event)
        else:
            # Direct S3 event format
            bucket_name = extract_bucket_name({'detail': event})
        
        if not bucket_name:
            logger.error("Could not extract bucket name from event")
            return {
                'statusCode': 400,
                'body': json.dumps('Could not extract bucket name from event')
            }
        
        logger.info(f"Extracted bucket name: {bucket_name}")
        
        # Get information about the bucket using the cross-account session
        bucket_info = get_bucket_info(bucket_name, session)
        logger.info(f"Results: {bucket_info}")
        
        return {
            'statusCode': 200,
            'body': json.dumps(bucket_info, default=str)
        }
        
    except Exception as e:
        logger.error(f"Error processing event: {str(e)}")
        return {
            'statusCode': 500,
            'body': json.dumps(f'Error: {str(e)}')
        }

If you are a Pro subscriber, I’ll be holding a code review session after we finish these labs — and if you are reading this a year later, check out the recorded video in the library.

Alrighty, click Test.

And it should fail. 😦 

Okay, it was a timeout. Easy enough to fix! Jump off the Code tab to Configuration > General Configuration > Edit > set the timeout to 30 seconds:

Here’s where things get a bit tricky. As a heads up, there is a lot of error-handling built into the code. So errors are managed gracefully and the function will run completely, but some errors will only show up in the logs, since they were handled. I deliberately left a common error in the code since — one you will hit a lot over the years. Click Test again, then scroll through the Output:

Here’s what happened. As far as the console is concerned, our function works because the internal error handling caught and handled the error. The problem is… we still have an error. In this case we are missing the permission (in the target account) to check the BucketOwnershipControls, which are important to know whether ACLs are enabled (we talked about that a long time ago).

To fix this, we need to change permissions in the account with that S3 bucket. How? Come on, you got this… StackSets! We just need to update our StackSet to add that permission! In a real enterprise this would be handled using a CI/CD pipeline, but it isn’t hard for us to make the change in the console. And one big difference: since people run through these labs at their own pace, and we need this error for this lab, we will use a new template instead of fixing the old one.

Personally, this is where I like to open up another tab, so I can keep the function open to run the test again. If you right-click the AWS icon in the upper-right corner, then Open link in new tab, then CloudFormation > StackSets > Service-managed > AutoremediationRole:

The first thing we need is the OU ID where the stack is deployed. Copy the OU ID and paste it into a text editor (since we also need to copy the URL to the updated template in a second):

Then Actions > Edit StackSet details:

Select Replace current template, then under Specify template paste in this URL: https://cloudslaw.s3-us-west-2.amazonaws.com/lab55.template, then click Next:

Click Next on the next 2 screens, then Paste in the OU and select All regions (or just US East (Virginia). Then Next > Submit:

It took me about 3 minutes for the template to update. As a reminder, this new template is exactly the same as the previous one, with just the new permission added.

Go back to your lambda function > Test > PROFIT!

No more error messages, and the results have all the information we need to make a security decision!

We are getting close to the end of this series, so the next few labs will be heavy on code. We still need to analyze the state of the buckets and, if needed, lock them down so they can only be accessed from our Organization. We also need to set up our time-based sweep to analyze all buckets, which is very tricky from a performance standpoint.

I appreciate you sticking with me on this extended series. By the end you should have all the tools you need to write your own enterprise-scale cloud security automation. This is absolutely an advanced skill!

-Rich

Prerequisites

• This is part 6 of the Advanced Cloud Security Problem Solving series, which I’ve been abbreviating to Epic Automation. If you haven’t done parts 1-5 yet, jump back to the first post in the series and complete all prior posts before trying this one.

The Lesson

I’m one of those people who, as I get older, definitely suffers the consequences of not sticking to the good habits everyone told us to start when we were younger.

Yes, I’m talking about flossing.

You see, the tap water where I grew up was very good and highly fluoridated. I’d never had a single cavity when I started college. TL;DR, I paid the price later in life. You haven’t lived until you’ve been on a work trip to some remote client location, and needed to get an emergency crown thanks to a protein bar.

And let’s stop pretending that any of you are thinking this analogy is about anything other than IAM.

Way back in Assume the Role! we learned about IAM roles and cross-account access. In this series we have already created a centralized role for our cross-account EventBridge. But I kinda glossed over a couple of details, one of which is both important and often missed when setting up a centralized service, like we are.

Let’s review the diagram from that first role assumption lab:

As a reminder, in that case an IAM user had permission to assume a role in another account. The IAM user needed the sts:AssumeRole permission in their IAM policy, and the remote account needed a Role Trust Policy to allow that IAM user to assume it. You need permission to assume roles, and the specific role needs to allow the identity to assume it. You need permissions on both sides. 

Because of how we’ve done most things in the console, these permissions have often been defaulted for us. They have also been a little over-provisioned in some cases. For example we allow sts:AssumeRole but don’t restrict it to only assuming a single role.

Now it’s time to take things a bit further, especially since we are enabling code (our lambda function) to change things across multiple accounts. There are some interesting things we can do to lock things down tight, and limit the potential for privilege escalation.

Locking down both sides of a role chain

To enhance security we will add conditions on both sides of the relationship:

• Only our designated lambda function will be able to assume the lambda role. Other functions can’t use it (the default when using the console and in most examples).

• Our lambda role will only be allowed to assume target roles which have a specific path we will consistently use in every account. (With a wildcard for the account ID).

• Our lambda role will only be allowed to assume those roles in the specific OU.

• Our target roles in each account will only allow the centralized lambda role to assume them.

Again, we’ve seen hints of how this works, but today is the day to call out the details. It absolutely takes more time and a bit more knowledge to set up. But it means the privileged role in a bunch of accounts, with permissions to change things, can’t be used by anything other than the expected lambda function, and that function can’t be abused to assume any role other than the intended one.

Even I often get my neurons tangled up as I work through where I need to put all the right pieces, so here’s a simple way to keep things straight in your head when having roles assume other roles (we call this role chaining — and yes, this also applies to other principals assuming roles, but I want to keep it simple today):

To do anything, your lambda function first needs its “lambda execution role” which is the role that allows it to do anything. The lambda service performs an AssumeRole for you. Then, for an entity with a role in one account to do things in another account, it needs to assume a role in that other account.

This process of using one role to assume another role is called “role chaining”, and we use it a lot!

In terms of what it can do, remember that all IAM permissions policies live in the account where the actions are happening. You can’t just do something like “PutBucketPolicy” from a central account, because permissions policies are attached to the IAM principals, and that principal doesn’t exist in the target account so there is no way to define what it can and can’t do. Your role chains, but your permissions are always limited to whatever is at the end of the chain. This diagram might help you visualize it:

Or not — I’m no graphic designer.

Remember that AssumeRole is a way of creating a session, and the AssumeRole API call returns temporary credentials: an Access Key, a Secret Access Key, and a Security Token. When you role chain you are gathering these at every step, and your permissions are limited by which permissions are associated with whatever set of credentials you use to make an API call. Obviously those only work in the account where that role lives, and you never inherit permissions from anywhere else on the chain! 

Honestly this is easier to see when you are coding, since in code we specify the session used to make the API calls.

It’s easy to get confused because you need sts:AssumeRole in the permissions policy of the role which wants to assume another role. And we also need sts:AssumeRole in the trust policy of the role being assumed.

It’s not that hard when you think about it: “I need permission to assume roles” and “I give you permission to assume me”.

What’s cool is that we can put restrictions on that source role’s permission policy to assume roles, limiting which roles it can assume, and we can put restrictions on the target role’s trust policy so that only a specific source role can assume it. Both sides are locked down so no one else can shove their way into the relationship. I could have used that in college.

We will also use the role trust policy to restrict the lambda service so it can only assume this role for our security-auto-s3 lambda function. This ensures that no other lambda functions can use this role to muck around with S3 in other accounts.

Let’s see what these policies look like. To start, here’s the permissions policy for our source role.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "logs:CreateLogGroup",
      "Resource": "arn:aws:logs:us-west-2:730335263677:*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": [
        "arn:aws:logs:us-west-2:730335263677:log-group:/aws/lambda/security-auto-S3:*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": "arn:aws:iam::*:role/SecurityOperations/SecurityAutoremediation",
      "Condition": {
        "ForAnyValue:StringLike": {
          "aws:ResourceOrgPaths": "<your org path>/*"
        }
      }
    }
  ]
}

The top part is what we started with last week, and just says the lambda function can write its logs to CloudWatch.

The meat is the sts:AssumeRole part. This says the role can only assume a role named /SecurityOperations/SecurityAutoremediation (as a reminder using paths keeps things cleaner, and we will use that later when we protect this role with an SCP). But it can assume that role in any account where that role exists, thanks to “*” in the Resource path.

Not only can it only assume that role, but notice that extra condition with ForAnyValue:StringLike? That’s to further restrict it so it can only assume that role in our Workloads Organizational Unit! Even if someone created that role in, say, our Security OU, they still couldn’t assume it. We like things nice and locked down.

Here’s what the source role’s trust policy looks like. Notice how it only allows the lambda service, and only our named function (thanks to the aws:SourceArn condition key)?

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "aws:SourceArn": "arn:aws:lambda:us-west-2:<SecOps account ID>:function:security-auto-S3"
        }
      }
    }
  ]
}

Now let’s look at the target role. In this case we use the role trust policy to ensure that it can only be used by that one source role, with the ARN hard-coded in. (This is a snippet from the CloudFormation template we will use to create the role in the target accounts. I’m skipping the rest for brevity.)

Resources:
  SecurityAutoremediationRole:
    Type: 'AWS::IAM::Role'
    Properties:
      RoleName: SecurityAutoremediation
      Path: /SecurityOperations/
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              AWS: !Sub 'arn:aws:iam::${SecurityOperationsAccountID}:role/lambda-slaw-s3'
            Action: 'sts:AssumeRole'

Look at AssumeRolePolicyDocument, which is the trust policy. It has Allow and then requires an ARN: the ARN of the lambda role in our SecurityOperations account (remember that !Sub is you’ll swap in the account ID, since you’ll be running this from my S3 bucket).

And I’ll skip the permissions policy — you’ll see it in a minute, and it’s just the usual S3 API calls.

This image shows how it all works together:

• In the SecurityOperations account:

  • In the lambda-slaw-s3 role we created for our lambda function:

    • Only the specific lambda function can use this role. The role trust policy only allows the lambda service, when working on behalf of the security-auto-s3 lambda function, to sts:AssumeRole.

    • When assumed, this returns session credentials to the lambda function.

    • The permission policy on the role which the function is now using only allows the function to assume a role named SecurityAutoremediation in any account in our organization where it exists. But…

      • We added a condition to restrict it to only assuming this role if it is in our Workloads OU.

• In the Target Workload account:

  • We have a SecurityAutoremediation role.

    • With a role trust policy which only allows our SecurityOperations lambda-slaw-s3 role to assume it.

    • With permissions to make certain S3 API calls.

• For the lambda function to make an API call on a target account:

  • The lambda service assumes the lambda-slaw-s3 role.

  • This returns session credentials to our running function.

  • The function uses those credentials to assume the SecurityAutoremediation role.

  • That returns a different set of session credentials, which enable the function to make those S3 API calls.

  • The chain breaks whenever any of those sessions expires (the default is 1 hour).

This pattern is the flossing of cross-account access. We restrict both sides of the relationship, but in a way which still allows us to work across multiple accounts. This is another reason a good OU structure is so important, and having a good mental model for how to use both role trust policies and permissions policies together.

Key Lesson Points

• Role chaining enables us to use one role to assume another role… and then that role can assume the next role in the chain.

• We use role chaining to go from one account into other accounts. It’s the basic way to build the IAM for security automation.

• Trust policies define who or what can use a role, and we can lock that down pretty specifically to prevent the role from being abused.

• A role’s permissions policies must include sts:AssumeRole if you want to use it to assume other roles. We can also put resource restrictions and conditionals on it, to further lock down both sides of the relationship.

The Lab

If you managed to read this far, you know what’s coming. We will:

• Update the permission policy of our lambda execution role so it can assume roles, but only one role, only in one OU.

• Update the trust policy of the lambda execution role so it is only usable by our named lambda function and not others.

• Use StackSets to push out our target/security automation role into our Workloads OU.

Video Walkthrough

Step-by-Step

First we need to build the path to our Workloads OU so we can use it as a condition later. Start in your Sign-in portal > CloudSLAW > AdministratorAccess > Organizations. Then collect your organization ID, the root OU ID, and the Workloads OU ID:

Just paste that string someplace — we’ll need it later. When you specify an OU using a global condition key, you need to use the entire path like this: o-0sqvc6dvj1/r-na6s/ou-na6s-ag0wxvj3/*

With that out of the way, let’s start changing our various permissions. Go to your lambda execution role (the one the lambda function will always use): Sign-in portal > SecurityOperations > AdministratorAccess > IAM > Roles > lambda-slaw-s3:

Click Permissions policies > lambda-s3 > Edit:

Copy this policy, substitute in your Account ID twice and your org path, and paste over the existing policy. As a reminder, you can get your current account ID by clicking in the upper-right corner dropdown in the AWS console.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "logs:CreateLogGroup",
      "Resource": "arn:aws:logs:us-west-2:<your account ID>:*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": [
        "arn:aws:logs:us-west-2:<your account ID>:log-group:/aws/lambda/security-auto-S3:*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": "arn:aws:iam::*:role/SecurityOperations/SecurityAutoremediation",
      "Condition": {
        "ForAnyValue:StringLike": {
          "aws:ResourceOrgPaths": "<your org path>/*"
        }
      }
    }
  ]
}

It will then look like this, but with your IDs, not mine! Then click Next, and on the next page click Save changes:

New you need to adjust the role trust policy. Click Trust relationships > Edit trust policy:

You’ll need that account ID again, and this time copy this policy, replace the account ID, and then paste the policy into the window > Update policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "aws:SourceArn": "arn:aws:lambda:us-west-2:<SecOps account ID>:function:security-auto-S3"
        }
      }
    }
  ]
}

Now it’s time for the easy part. We’ll use StackSets to push out our SecurityAutomation role, which has permissions to make changes in the accounts. When we do this, we’ve also set it to use the /SecurityOperations/ path, which helps keep things organized.

Go to CloudFormation > StackSets > Service-managed > Create StackSet:

Paste in this template URL and click Next:

 https://cloudslaw.s3-us-west-2.amazonaws.com/lab54.template

Name it AutoremediationRole, provide the description, then Copy & Paste the Account ID, like we’ve been doing. Then click Next:

Click the checkbox > Next:

Then Deploy to organizational units > Paste in your Workloads OU ID, which you previously copied. Previously we deployed our role for central event collection everywhere, but for autoremediation (which will change things, and could break things) we want to be more cautious.

Pick US East as the region, then Next. Don’t worry about concurrency — there are only 2 accounts within that OU.

On the next page click Submit. No screenshot for you — it’s, like, a big button in the corner. You can figure out which corner. I have confidence in you.

It will take a few minutes, but you can click Stack instances to track the deployment and see when it’s finished:

If you click Template: as you can see we have a block of permissions for S3 read actions, and a second block for write actions.

One last point: you’ll notice that we named everything S3 for the lambda and the role over in the SecurityOperations account, but this template creates a generic SecurityAutoremediation role in all target accounts. Why? Because down the road we will add more automated guardrails, but we don’t want to stuff accounts with different roles for each service. We will update this role with more permissions, and then limit what the automation lambda can actually do by inserting an inline policy called a session policy. You’ll see how that works once we get to writing the code.

-Rich

Prerequisites

• This is part 5 of the Advanced Cloud Security Problem Solving series, which I’ve been abbreviating to Epic Automation. If you haven’t done parts 1-4 yet, jump back to the first post in the series and complete all prior posts before trying this one.

The Lesson

Over the past four labs we’ve built out the foundation of our automation platform by wiring together our centralized event bus, using a simple EventBridge rule and SNS to validate that the pipeline works. This week we’ll set up our first Lambda function, which is where all application logic will live.

Oh, never heard of Lambda functions? Or never worked with them before? Yeah, this is where things get pretty awesome.

Serverless 101

Let’s clean up some terminology to start. Everything I’m about to cover is more on the development and operations side, but you can’t secure things you don’t understand.

Serverless is the term we use for cloud services you use where you don’t manage the underlying servers. Huh? Think of it this way: with EC2 we ran an instance and we handled all the configuration of the software and the operating system. We are responsible for scaling, sizing, and overall management. S3? It’s just a… thing… where we put files, and we can configure it to serve those files like a web server, without installing or managing any web server software. We have no idea what’s under the hood and we never touch an operating system. EC2 isn’t serverless, but by my definition (which not everyone agrees with), S3 is. Especially if you configure it for use as a web server.

It’s a little confusing because some people (okay, a lot) tend to only use the term serverless when referring to Function as a Service (FaaS). What’s FaaS? It’s a kind of workload, which is a term we use to describe where code runs. It’s easier to understand what FaaS is if we list the major workload types:

• Physical server: If I need to explain what this is you probably skipped ahead a few posts, and should come back after learning the alphabet.

• Virtual Machine (VM): We run our code/application in a full operating system on a hypervisor. An EC2 instance is a virtual machine.

• Container: Containers encapsulate an application and its dependencies. They usually run on a VM or physical host, on top of an operating system in a little isolated bubble. Some containers are heavy, very like a full virtual machine, while others are very lightweight (which is how they should be!) Containers are isolated, but less strongly than virtual machines (because modern hypervisors use all sorts of special hardware for performance and isolation, but containers share operating system resources).

• Serverless Functions/FaaS: You load and run your code in the cloud provider, but you don’t deal with an underlying operating system or even container. Your code is triggered, runs, and then shuts down. It often runs in a container in a VM, but these aren’t good times for Russian doll references.

That’s a key difference between FaaS and other types of workloads: the code you load runs when triggered, not all the time. When Lambda (Amazon’s FaaS, the first on the market) was released I think it timed out in something like 30 seconds. It was very expensive to run it for more than a few seconds.

Lambda is what we use to build event-driven applications. The function is triggered and passed some data, it does something fast, sends out results, and then resets itself for the next request. Here’s where it gets really cool: you don’t need to worry about scaling or performance (okay, that’s not really true once things get big enough, but stick with me for now). If I send in 1 event on my event bus, it triggers one lambda. If I send in 10,000 events at the same time, 10,000 lambdas run in parallel. I pay for every invocation, but I don’t pay a penny when the lambda is just sitting there without any events.

Lambda functions are ideal for the kind of application we are building. It’s lightweight, I never need to worry about a server or a container, and I don’t need to patch or maintain anything other than my own code. It doesn’t cost anything when there aren’t events, and for many architectures it’s super cost effective. And yes, you can support complex application logic using multiple lambdas, event buses, message queues, and other components. These designs also scale really really well. If you are going to build a new startup thingy, go event-driven and serverless as much as possible.

Lambda Security 101

Let’s start with how lambda functions work:

• You create a function, and provide your application code and configuration options — we’ll talk more about these later in this lab.

• Your function uses an IAM role. Every function needs a role, and you want to use least privilege.

• You trigger the function from the outside. This can be an EventBridge rule, an SNS notification, an SQS message, direct invocation from a URL, from a load balancer, an API gateway, or more.

• When a function is triggered you pass in the event or data for it to process.

• The function runs. It uses that IAM role, and you can add in things like environment variables, or even connect it to a VPC if it needs network access.

  • If you don’t put it on a VPC, it can access the Internet using Amazon’s network. We’ll discuss this in future labs.

• The function sends its output wherever you specify in its code.

• By default the function logs to CloudWatch Logs.

• When it’s done it’s done. That function ends execution. Depending on how things are set up and what’s going on at the time, the runtime environment may also tear itself down.

Sure, there can be a lot of security nuance with lambda functions, but the fundamentals are pleasantly straightforward. Here are Rich’s 6 Fundamentals of Lambda Security^(TM):

• Control who and what can trigger your function. I’ve found some unsecured/unauthenticated ones during assessments.

• Use a least-privilege IAM role for the function. A few times not only have I found unsecured functions, but they had full-admin IAM roles.

• Write secure code. Okay, we’ll try not to screw our code up, but in an enterprise you want to use ALL the code scanning tools. And keep your code up to date if you include other libraries which might become vulnerable.

• Don’t store secrets in your lambda function or the environment variables you configure when you set it up. Pull from a secrets manager instead (specifics will be a future lesson).

• Write your code to log prolifically. There are frameworks available to make this better.

• If you need to access internal resources (e.g., databases) put your lambda on a VPC and don’t use the default AWS network access.

That’s most of it. Our application is relatively simple so we won’t need secrets management or network access, but you’ll see the rest in the lab.

Key Lesson Points

• Serverless functions are a key workload type, especially suited for event-driven applications.

• Serverless functions still run on servers under the hood, but we don’t need to patch or maintain anything other than our own code.

• Security focuses on secure code, triggers, IAM, secrets management, logging, and network security.

  • But, as always, it’s mostly IAM.

The Lab

For this lab we will implement our first Lambda function and trigger it using the EventBridge infrastructure we already built. The function is super simple and just logs the event to CloudWatch Logs, but that validates our application logic and that all the right pieces are connected.

In our next labs we’ll start building out the actual coding logic in Lambda and modify our function and its IAM privileges as we go.

Video Walkthrough

Step-by-Step

Go to your IAM Identity Center sign-in portal > SecurityOperations > AdministratorAccess > IAM > Roles > Create role:

Choose AWS service > Lambda > Lambda > Next. This will create a role for lambda to use. Behind the scenes it’s basically just automation to set up the Role Trust Policy. If we were doing this via API or CLI we would need to write/include that policy ourselves.

Do not add any permissions!!! Just scroll down and click Next. We will add permissions in a minute using an inline policy, since this is a set of permissions we won’t use anywhere else. It’s also an excuse for me to show you inline policies (which live alongside the role and can’t be used anywhere else) vs. managed policies, which we’ve been using previously.

Use the role name lambda-slaw-s3. Then scroll down and Create role. I’m not worried about a description on this one — that name is spiffy on its own.

Now click the lambda-slaw-s3 role:

Then Add permissions > Create inline policy:

Select JSON and then Copy and paste this policy. There are three statements in here, and in a moment we’ll swap in your account ID. This policy is for the Lambda we are about to build. It:

• Allows the Lambda function to create a log group in CloudWatch Logs.

• Allows the function to create a new stream each time a function launches (this is how it works) and to store events in the stream. This is hard-coded to the name we will use for our function when we create it (security-auto-s3, showing my usual lack of creativity).

• Allows various S3 read API calls.

Anyone notice the flaw in what I’ve done here? As written this onlys allow access to S3 in this account. If you remember, IAM permissions apply only where the policy lives. We’ll actually need to change this up later for cross-account access, and we’ll need to push out a role with CloudFormation StackSets in our other accounts for this function to assume. But for today this is fine, since we aren’t making cross-account calls yet, and it will help us understand that process when we get to that lab (like, probably the next lab).

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "logs:CreateLogGroup",
            "Resource": "arn:aws:logs:us-west-2:<accountID>:*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:us-west-2:<accountID>:log-group:/aws/lambda/security-auto-S3:*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketAcl",
                "s3:GetBucketPolicy",
                "s3:GetBucketPolicyStatus",
                "s3:GetBucketTagging",
                "s3:GetBucketLocation",
                "s3:GetBucketPublicAccessBlock"
            ],
            "Resource": "*"
        }
    ]
}

Now you need to get your account ID from the upper right corner and paste it to over <accountID> in two places in the policy.

Then Next > name it lambda-S3 > Create policy:

Phew! We’re done with IAM (for now). To review, what we just did was create a role for our lambda function. That role can log things and access S3… in the same account. Yeah, my bad, we’ll talk more about this, probably in the next post when we set up cross-account access.

So where does that leave us? It’s time to create our function! Go to Lambda > Create a function: 

On this next page we set all the core configuration we need:

• Function name: security-auto-S3 (case matters — this is hardcoded into our IAM policy).

• Runtime: Python (3.13 is the current version as of this writing, but whatever is the default python will work).

• Architecture: amd64. Either architecture works, but AMD in AWS tends to be cheaper with better performance across the board.

• Permissions > Use an existing role > lambda-slaw-s3 to use the role we just created.

• Then Create function.

Before we swap in our code, let’s look at the default python lambda function.

I’m not here to teach you how to code, but … I think I need to teach you how to sorta code. Like, just enough to get by. This is very simple: first it loads up the python library to read json because everything in cloud is json (events, policies, etc.) and the odds are nearly nil that you would ever not need this library.

Then we have the “lambda_handler” function. This is the part of the code which the lambda service will call when triggered. It takes the event and the context, both of which are passed in by EventBridge (or another trigger). Not all your functions will start in this exact way, but it’s exactly what we want for our design.

Then there’s the “return” statement, which is what the function returns to the service when it’s done running. By default this logs to CloudWatch.

Okay, so let’s look at our code:

import json
import logging

# Configure the logger
logger = logging.getLogger()
logger.setLevel(logging.INFO)

def lambda_handler(event, context):
    """
    Lambda function that logs the incoming event to the default CloudWatch log stream.
    
    Parameters:
    event (dict): The event data passed to the Lambda function
    context (LambdaContext): The runtime information of the Lambda function
    
    Returns:
    dict: A response with statusCode 200 and the logged event
    """
    try:
        # Log the entire event as JSON
        logger.info(f"Received event: {json.dumps(event, default=str)}")
        
        # Log some context information
        logger.info(f"Function name: {context.function_name}")
        logger.info(f"Request ID: {context.aws_request_id}")
        logger.info(f"Remaining time: {context.get_remaining_time_in_millis()} ms")
        
        return {
            'statusCode': 200,
            'body': json.dumps({
                'message': 'Event successfully logged',
                'event': event
            }, default=str)
        }
    except Exception as e:
        # Log any exceptions
        logger.error(f"Error processing event: {str(e)}")
        return {
            'statusCode': 500,
            'body': json.dumps({
                'message': 'Error processing event',
                'error': str(e)
            })
        }

In our case we add the logging library for better log messages. If we didn’t use that we could just use print statements. Our function handler is the same, taking the event and the context, but then we change things up. We take the event and dump the event right into our log stream (you’ll see this soon enough). Then our exit message is that we successfully logged the event. The try/except are to catch potential errors “try this, and if it fails, then do that”.

More words, like a bunch more words, but this basically says “log the event, and log if it was a success or failure”. And while this will work for any event, as you’ll see we are only triggering it from our S3 events we built out last week (once we add the trigger).

Now copy and paste the code into the window, then click Deploy to make it active. Remember, there isn’t any concept of saving — a live function is a live function, so “Deploy” is the right word: we are deploying code. (And Lambda can track versions).

If you want to see some other settings which affect security but we aren’t using, check out the video or feel free to look around (just don’t change anything). Your function is all loaded up and ready to run, we just need to set a trigger and test it.

Got to EventBridge > Rules, choose the SecurityAutomation event bus, S3-security-remediations:

One of the cool things about EventBridge rules is that they support multiple targets. Our existing rule sends us an email via SNS, and we can add a second target to trigger our lambda function. For now we’ll keep both because this helps us test — if we get the email but don’t see the log message, we know that the trigger works but the function has an issue.

Go to Targets > Edit > Add another target:

Then use the following settings:

• AWS service

• Lambda function

• security-auto-s3

• Use execution role

• Create a new role for this specific resource

If you decide to look at the IAM role created, it has permissions to invoke the lambda. Since this is one AWS service talking to another, you absolutely need this permission.

Then just click through Next > Next > Update rule.

To save you some pain, if you go into the lambda function there is a section at the top which lists triggers. For whatever reason ours won’t show, and I suspect it’s because we are using a custom event bus. Don’t worry — I tested everything and it works.

Now close the tab and go back to your sign-in portal > CloudSLAW > Administrator access > Switch role to Production1. We will drop into our Production1 account using the OrganizationAccountAccessRole again to create a tag on our S3 bucket. If you are using the same browser as the last lab you can drop into the account with one click. If not, just review how to switch roles into this account from last week’s lab (I’m worried this is getting so long it might be filtered by mail servers):

Go to S3 > whatever you called your bucket > Properties > Tags > Edit and add a tag. You can review the screenshots from the last lab if you need — I’m definitely taunting the email delivery gods with this long lab.

Last step. First of all, you should already have an email triggered by the change. Now we want to go back into our SecurityOperations account to see if our lambda function logged to CloudWatch. So Close the tab > sign in portal > SecurityOperations > AdministratorAccess > CloudWatch > Log groups. You will only see a log group if it worked. Then click your log group:

It worked! Yeah, this was a lot for one lab, but my video clocked in right at 31 minutes so I’m calling that within the margin of error for my 15-30 minute rule. Now feel free to click any of the log streams and read the messages. Every single function creates its own stream when invoked. When operating at scale you definitely want to keep an eye on costs, but logging is critical for security so I wouldn’t cut corners here. The devs should be paying the bill anyway.

Now we have our alerting infrastructure set up, and we know we are passing selected S3 events into our central event bus and lambda. Next up we’ll fix up our permissions for cross-account access, then we’ll have a few labs working through the code to meet our requirements.

Yes, this series is a lot, but I haven’t seen any good tutorials out there to walk you through every piece of the process to build out autoremediation (or even an event-driven application). Could I have just dumped some CloudFormation on you? Well, yeah, but that makes it tough to learn how to architect and implement for yourself in a different context.

-Rich