Working in security is a commitment to a career of cognitive dissonance. Our work requires (more like demands) healthy skepticism. Yet we somehow find ourselves coming to work each day. We know about all the proverbial skeletons in the closet of services, devices and apps and all the myriad ways in which they could go wrong and yet we have committed ourselves to fixing it. A curious skeptical nature mixed with a bizarre desire to do good will get you very far in security.

It might also be your undoing. The same instinct that makes you good at this job, assuming everything will break, can quietly become the reason you refuse to engage when something new shows up.

Epistemic and defensive skepticism

Skepticism is an interesting concept philosophically. It can present as both a defence mechanism and an epistemic (meaning a means of seeking knowledge) virtue and separating between which brand you have requires self examination. A fair distinguishing feature between epistemic and defensive skepticism is the direction in which it faces. An epistemic skeptic is equally skeptical of all things, seeks balance between belief and evidence and withholds or caveats judgement under uncertainty. This framework is applied uniformly to both internal and external ideas or values. Whereas a defensive skeptic typically applies this framework only to that which threatens existing beliefs or could introduce emotional risk. This leaves uncomfortable assumptions unexamined and is arguably just a form of intellectualisation rather than a legitimate desire to be more critically minded.

Are you fairly applying your skepticism to AI?

Defensive skepticism in practice

It’s likely you’ve experienced this before, perhaps it’s something you might upon reflection notice in yourself. Refusing to evaluate or approve AI tooling because “it represents too great a risk”, yet not choosing to understand it. Being critical of those that use AI as part of their workflow, not because their approach is flawed, but because they have an approach at all. Or perhaps you’ve seen the perpetual sandman. Not reading, not testing, not forming an opinion and treating that absence as a position. These all seem like unique and perhaps understandable responses but they all produce the same outcome: you’re not in the room when it matters.

The price of being defensive

Absence is not a defence nor is refusal. AI adoption is something that will happen, LLMs do not need to be correct to be successful. They just need to be close enough. Your developers are going to use AI, your HR staff are going to use AI and your executives are going to push AI. In security we frequently talk about Known Knowns, Known Unknowns and Unknown Unknowns. Usually it’s the unknowns that keep us awake at night. But here’s an unknown worth losing sleep over: In every room you’re absent from you’ve left an empty chair. That chair was your chance to help make decisions about how AI gets adopted, what guardrails should exist, what data should flow where. Those decisions are getting made right now, by those that show up. You’re on the back foot, responding not guiding, flailing not leading.

Philosophy meets practicality

Question. Why do you patch vulnerabilities? There are a lot of answers to this question, and the majority of them are correct. Consider for a moment, if you will, the futility of the practice. Do vulnerabilities ever stop? You don’t refuse to patch a vulnerability simply because there will be more. When we must, we simply do. Yet somehow this logic doesn't survive contact with AI. What you're doing every time you patch is what Camus described. You push the boulder knowing it rolls back. You don't stop because it's futile, you find meaning in the effort. Camus concluded this wasn't despair, it was engagement in spite of the absurdity. "One must imagine Sisyphus happy."

So why is AI the boulder you've decided to sit down in front of?

Being in the room

So where does all this leave us? The answer is clear, you need to be in the room. You need to be an active part of the decision making process. This does not mean you’re a cheerleader. You don’t have to be “pro-ai”. Rather you should let the very skepticism that is excluding you become a virtue. This means your solution to AI in your organisation cannot simply be no. 

I've had the privilege of working on some amazing CTFs (Capture the Flags) in my career. From building internal ones, to delivering them for companies at cons or being part of the team running them at community events. Making challenges and seeing people learn from them is something that brings me great joy. There is a pretty insane amount of effort that goes into a good CTF. Coming up with challenges, spinning up reliable infrastructure for hundreds or thousands of people to use and wrangling a bunch of volunteers takes patience and perseverance and hundreds of hours usually crunched into a few weeks/days/hours prior to the event.

CrikeyCon 11 was no exception. We had big dreams about starting earlier but soon found ourselves working right up to the bell (even pushing challenges during the event). In a lot of ways I probably wouldn't change a thing about that, pressure makes diamonds.

As usual I want to thank the challenge writers (you guys rock) and my fellow organisers and volunteers (you know who you are). Your continued dedication and hard work is so impressive and something you should be proud of.

Without further ado, here is my retrospective for the CrikeyCon 11 CTF.

The ai-lephant in the room

For those that didn't attend CrikeyCon this year or attend the closing ceremony (where were you!?) you missed what could be considered a reasonably large disruption within the CTF space: AI. Now to most CTF attendees or organisers this isn't a surprise: AI is pretty well equipped to aid competitors. This year however, it definitely exceeded just aiding. We had our first competitor to simply authenticate Claude Code with their CTFd cookies and prompt it to complete the CTF. There was a lot more to it and I won't steal that competitor's thunder and will share/link them once they have written it up for themselves. For clarification we (the organisers of the CTF) were aware of this and largely saw it as a reasonable experiment. The surprising (or perhaps unsurprising to me) outcome was that this competitor won. The winning competitor gracefully passed on their first prize to the non-agentic winner. The agent basically flattened our CTF completing 79 out of 80ish challenges including access control challenges but excluding lock-picking challenges.

As with any disruptive technology there are different schools of thought as to what this means and I want to capture some of them for posterity.

CTFs are dead

This was probably the most common reaction from challenge writers. Writing unique challenges is hard and there is an expectation that people will put effort in to solving them. It's a natural reaction, one of the greatest rewards from writing a challenge is when you absolutely nail it. Nailing a challenge has a few different forms:

• You make a challenge that gets solved by 1 team

• You make a challenge that fits its category and difficulty rating perfectly

• You make a challenge that is unique and quirky and absolutely frustrates the hell out of a team/player

All of these are what I personally consider success when writing a challenge. Sometimes your challenges are meant to be solved (especially 101 challenges), sometimes they aren't meant to be solved and sometimes you just really want the winning team to beg you for the answer.

The outcome on Saturday takes away all of these rewards. If your challenge can be solved by AI then every team has access to it, therefore point 1 doesn't apply. Point 2: the category and difficulty doesn't matter. And point 3? Well, Claude doesn't complain. This leaves less incentive for challenge authors to create challenges which then leaves the CTF scene less vibrant and interesting.

CTFs are going to have to change but I am not quite sure how

The other reaction was: hmmm, what do we do? I call this the what's next reaction. Essentially these people acknowledge the coolness of the technology and are also not sure how we can incorporate it. I think this is probably the second most popular viewpoint. They see that there is a future but aren't quite sure about how we get there. In some cases there is a feeling of not wanting to own the future. This is something I saw in some CTF players and challenge writers as well.

Wow this is cool / Wow this is scary

Pretty self-explanatory. Most people fall into this category. They are either excited or afraid. Either one is a perfectly natural response. I have a post I am cooking up about this topic so I won't spend too much time on it.

So what does the future of CTF look like?

These are just my thoughts so take them with a grain of salt.

We probably need two streams of CTF: a classic CTF and an agentic centric CTF

The traditional CTF still has a place and I think it should continue. The reason I say this is that the core benefit for participants (aside from Lego and bragging rights) is learning. We have a responsibility to learners as CTF writers to keep giving them a place to play and learn. That being said we will probably have to begin to take a harder line towards AI within these competitions and rethink our challenges a little bit.

During the CTF we offered prizes to people that would come up and admit/show us that they were using AI. We had at one point a line of 12 people waiting to show us laptops open with ChatGPT and Claude windows on display. AI is inescapable so we need to think about how we can account for that. This is going to take time and creativity.

The second stream is an agentic centric stream. I have some thoughts about what this might look like.

Agent jungle gym

It has long been a desire of myself and other CrikeyCon peeps to bring back a form of the Red vs Blue CTF challenge. These are dynamic attack and defence challenges that more closely imitate the way in which attackers and defenders work in a network. I think the outcome of this weekend just past justifies that. I've sketched out something I like to call the jungle gym. It's a hybrid between red vs blue and a king of the hill challenge with a bit of a twist: agents only, or a combination of agents and humans.

Essentially there are three rounds. At the beginning of each round you get to submit your agent/agents. That agent is presented with a series of boxes they are tasked with protecting and a series of boxes they are tasked with attacking. The boxes they are protecting have services and flags that must remain available; this is a prerequisite to score offensive points. Points are awarded for keeping your services available and taking down other teams' services. Each round is a novel environment and in between rounds competitors get to work on their agentic approaches before the next one starts.

This creates an interesting and dynamic environment for players and observers and also removes the inherent edge of AI. I am also exploring how to showcase what the agents are doing, and as to if it should be run by a form of AI dungeon master that handles injects. It would also be interesting to see if a team for example decided to not use AI and compare their placing to others. This is still early stages and we are a fair way out from it. My soft goal is to run an online version of this sometime in the later half of this year. If you're interested in taking part reach out to me, I will be writing and sharing more about this soon.

I will also be seeking sponsors to help pay for tokens as well so that there isn't a financial barrier to participation. If you've got someone that can help with that please point them in my direction.

AI has benefits for challenge creators

Writing challenges is time consuming. There are a lot of ways that we as challenge creators can use agents to improve this process.

I experimented a fair bit with agents this year and have the following observations:

Agents make much better UI/UX than we would normally, especially in web challenges

UI/UX historically hasn't been part of the challenge space. A lot of people that write challenges are not actively incentivised to provide a good UI for something that will be used for one day. Agents change this a bit. Having an LLM create a functioning user interface that goes over the top of your web challenge can be extremely powerful. This allows for much better visual storytelling and theming as well. We had some genuinely unique and interesting challenges that were greatly enhanced by the interface we could put on top of them.

Agents can add dynamism to challenges

Last year I experimented with agentic challenges by creating an IRC server that was populated with LLMs. Each LLM had different levels of protection from prompt injection and had a flag they needed to protect. This year we initially didn't have any LLM challenges, but during the day I set myself a challenge to use an agent to build an agentic challenge and push it live before the day ended.

The result was a challenge called slop-machine. Essentially, you pulled the lever and you got a random challenge displayed within an iFrame on the page. This challenge was live vibe-coded by an LLM (qwen3.5-35b-a3b, an open-source model) immediately after you pulled the lever and presented to you without a human reviewing it. Topics for challenges were mostlyweb and crypto, obviously limited by the implementation. There was an element of luck to the challenge in that you had a much greater chance of being presented with a challenge you literally could not solve (either because the LLM just borked it or the answer was wrong).

It was interesting to watch. Some people came up and asked why it was so easy, while I watched others pull and pull the lever getting perpetually wrong challenges. It's fun watching the non-deterministic nature of LLMs play out in real time. Dynamic interaction is a really powerful means of making an engaging and interesting challenge and I've got a lot more ideas that I will be bringing to future CTFs.

Agents are generally pretty capable of handling your 101 challenges

101 challenges are a must for any CTF, especially for Crikey. A lot of attendees are students or people new to the industry. These challenges are all very similar each year but still need to be slightly different and fit the yearly theme. These make up a reasonable portion of challenges each year and are still time consuming. Agents are great at making these (with a bit of prodding). So if you're building 101 challenges, agents are your friend. Avoid trying to one-shot multiple at the same time. I tried a mix of both and the one-shot multiple challenges were extremely unreliable. Use planning mode and be clear about what you want.

Agents are better at solving challenges than writing them

In a similar vein, agents are much better at solving challenges currently than writing them. Agentic output is inherently derivative; it's noisy enough to be interesting but not yet truly novel. Challenge writing requires novel thinking and current context that LLMs lack. Humans currently are still the best people to write difficult and interesting challenges. I would recommend avoiding using agents for anything beyond the easiest challenges, agent centric challenges, UI/UX, dynamic testing and writing IaC for infrastructure at the moment.

Alright, enough AI.

What we did right

Created a great vibe within the CTF room

CTFs should be social and communication/community reliance are super important skills. Participants are choosing to give up a lot of the benefits of a con by taking part so we want to try to bring them that experience. This year (thanks to our awesome community and sponsors) we had a bunch of prizes. This allowed us to come up with some interesting means of giving out prizes. Aside from the AI confessional we also had a rap battle. We offered 1000 points to the winner of the battle, giving them 45 minutes to come up with a unique rap. This was entirely off the cuff and honestly we did not expect the uptake; something like 8 teams came up and competed. We ended up having a tie for first and gave each team another 15 minutes to come up with some lines for a rap battle.

The finalist teams were both university cybersecurity societies. It was awesome to see the next generation of cybersecurity being so willing to get up in front of the community and have fun. I think that says a lot about the unique community we have in our industry and it's something I and the other organisers want to bring to future cons.

Improved infrastructure maturity

Last year we had a largely click-ops based approach to deploying our CTF infrastructure and I spent days making sure everything was working. This year was entirely different. Underlying infrastructure was handled by Terraform, scheduling services was handled by Nomad and challenge definition files were subject to linting and CI/CD. This wasn't perfect; we had a few little blips where we had to scale up the instances larger than we expected, but it made the process of getting started so much easier. Taking the learnings from this away, I will have a post soon where I officially open source the infrastructure template I've created. My hope is that this will help everyone run better CTFs themselves.

Some changes I want to make include creating traditional test harnesses as well as agentic ones, so that LLMs can have a crack at challenges. This will give us an idea as to whether our challenges are solvable via LLMs and also help spot issues with availability and performance earlier.

Fresh blood in the challenge creator space

I've said it a million times but I am so thankful for our challenge writers. I have an extra pile of thanks for our new challenge writers. You've become part of a strange dysfunctional family that is so much better for you being there. It takes a lot of courage to write a CTF challenge; you're basically asking people to hack something you made. We saw some awesome challenges from new creators that were fresh, vibrant and different.

What we could do better

Start earlier (we try)

My perpetual dream with every CTF is that I wish I'd started earlier. This is a pretty hard ask for our volunteers and creators so it's probably going to stay a wish. There are definitely project management and organisational style bits and bobs we could implement as well.

Expand our challenge base

Lots of great feedback this year about blue team and forensics challenges. These are things we'd love to have more of. Our challenge make up currently reflects our creators, that is very web/offensive sec heavy. Variety is the spice of life. So if you're interested in becoming a challenge writer please reach out to me. There is so much to gain from it and writing challenges is such a great way to validate understanding. We are open to any challenge from anyone and if you're looking for ideas we are also full of those.

Embrace AI

I've said enough about this above. There is more we can do in this space and we will.

Happy hacking and see you next year.

Subscribe to get the next post delivered directly to your inbox

In the mid-to-late 1800s humanity developed the humble beginnings of evidence based western medicine, doctors treated illness using medication (mostly opium) and performed basic surgery. We began to see reductions in mortality rates and improvements in public health. Medical history of course, didn’t end in the 1800s, beginning in the 1870s new “medical” boom occurred: patent medicine vendors travelling the countryside pitching cures.

These cures weren’t for diseases that medicine was already addressing such as cholera or typhoid rather they were for entirely new ailments. Fun sounding diseases such as americanitis (the disease of living too fast), neurasthenia (exhaustion with technology) and hysteria (a dark stain on our medical history) became the modern plague. They weren’t pitched as evolutions of known diseases, rather they were new conditions that medicine couldn’t understand because it was “outdated”. Opium? How quaint, Surgery? How barbaric.

The new era of disease required a new proprietary fix with its own new secret ingredients. Vendors would publish almanacs, hold lectures and make charts flooded with medical-adjacent terms and classification systems. Suddenly everyone wanted, nay, needed Dr Kilmer’s Swamp Root or Lydia Pinkham’s Vegetable Compound. Doctors were dismayed, and pointed out that these new conditions were in fact just anxiety, poor diet or depression wrapped into a much better looking wrapper. This was dismissed by the public and the patent vendors as just them “not getting it”, they were out of date dinosaurs that hadn’t yet adapted to the reality of the new disease.

Why is this relevant? Well, patent medicines did sometimes contain novel active ingredients (usually boatloads of cocaine and alcohol). Some potentially might have worked. Instead of integrating these discoveries or understandings into medicine, vendors built entire parallel industries around them, replete with new diagnostic frameworks and certifications. This didn’t stop in the US until 1906 when the Pure Food and Drug Act forced vendors to disclose ingredients and stop making fraudulent claims.

Flash forward to the present, in the mid to late 2010s humanity developed the humble beginnings of functional AI. Whilst the concept for a large language model had existed for a while, the emergence of both the internet and affordable hardware enabled scale beyond our wildest dreams. The history of AI didn’t end in the 2010’s, in the early to late 2020’s we started to see these tools explode. The adoption of these tools without consideration or planning led to the development of an entire industry built around the need to secure these tools. This industry created its own taxonomy of threats with pretty charts and lectures designed to legitimise it. Prompt injection, emergent capabilities and model poisoning became all the rage.

This new era of threats required it's own proprietary practices, tools and teams. Suddenly everyone wanted what these vendors were selling. Security practitioners were dismayed and pointed out that these new conditions were in-fact just product and application security wrapped into a different, more appealing wrapper.

Why is this relevant? Well there is some truth to the idea that novel security challenges exist within AI, but it’s being packaged as if there is a need for it to operate separately from ProdSec and AppSec. There is an entire parallel industry being created when its really just an extension of our existing practices with some new learnings necessary.

The patent medicine industry for today’s ailment is inventively named AI security.

We’re in an AI boom/bubble/whatever, I am not going to debate about the validity or reality of it because it isn’t relevant. You, me and everyone else have got a job to do and that job most likely involves securing something that is getting or already has AI, regardless of our feelings or opinions. Technology being technology and hype-cycles being hype-cycles we have quickly found ourselves facing a petri-dish of half-baked immature technology smearing rapidly expanding lifeforms all over the place. We’re also continually being bombarded with the latest and greatest solution to fix the problems that tend to occur when half-baked immature technology gets unleashed upon business and individuals alike. What are these problems? Are they something unique? My argument (if it wasn’t already clear) is no and no.

AI Security is largely just ProdSec and/or AppSec. Note: I say largely, because there is legitimate necessity for researchers to focus on novel attacks that might arise within these models and a need to update our classification methodologies. Furthermore there is space for new approaches to solving problems and ways that we can use these tools to our benefits. But as a defender/blue/purple teamer I struggle to see the need for this to exist as a separate discipline.

AI systems are just software systems (usually APIs) with unusual/quirky characteristics. They take inputs, process them (through parameters as opposed to explicit logic) and produce outputs. The concerns these systems create map neatly to traditional product security domains. To clarify this I have outlined some of them below:

Prompt Injection aka Input Validation Failure -

Regardless as to if you’re sanitising SQL inputs or sanitising prompts to prevent injection, the pattern is identical. We have our methods for addressing this including:

• Input sanitisation

• Structured input formats

• Delimiters between system instructions and user input

• Filtering layers before processing (anyone heard of a WAF before)

• Output validation

• Principle of least privilege

• Input length limiting and rate limiting

Jailbreaking aka AuthN/AuthZ Bypass -

Securing who can query your data and under what circumstances is the same problem whether its a model or an API.

• System level guardrails independent of the model

• Filtering layers before and after processing

• Behavioural analysis for detecting policy violations

• Principle of least privilege

• Monitoring for known patterns (WAF again)

Model Poisoning aka Software Supply Chain attack -

A poisoned npm package and a backdoored model weight from Hugging Face are exactly the same threat category. Both are just third party dependencies

• Model provenance tracking and signing

• Model sourcing decisions

• SBOMs

• Dependency scanning

• Code review

• Integrity of training pipelines

• Controlled training environments

• Version control

This mapping continues all the way up and down the threat landscape and I might dedicate a large post to this alone.

So what's the prescription from this ProdSec lead? Extend your existing ProdSec/AppSec program to cover AI systems. Train your team on the quirks of probabilistic behaviour and adversarial ML, yes. Add some new tools for model scanning and prompt testing, absolutely. But don't create a parallel security organisation. Don't buy a vendor platform that treats AI security as fundamentally separate from the rest of your security posture. The patent medicine era ended when regulators forced transparency about ingredients. The AI security vendor boom will end when organisations realise they already have most of the ingredients in their existing security programs, they just need to apply them to a new (if quirky) type of software system.

The patent medicine vendors were selling sugar water with cocaine. The AI security vendors are selling you AppSec/ProdSec with buzzwords. Some of the ingredients are genuinely useful. Most of what you need, you already have.

Subscribe to get the next post delivered directly to your inbox

If you’ve got an app that even tangentially touches NodeJS/NPM (really if you have eyes and ears) you’ve probably seen these.

"Shai-Hulud" NPM attack runs malicious GitHub Action

An NPM attack compromised dozens of popular packages which then ran malicious GitHub actions in the compromised accounts

getsafety.com/blog-posts/shai-hulud-npm-attack

ctrl/tinycolor and 40+ NPM Packages Compromised - StepSecurity

The popular @ctrl/tinycolor package with over 2 million weekly downloads has been compromised alongside 40+ other NPM packages in a sophisticated supply chain attack dubbed "Shai-Hulud". The malware self-propagates across maintainer packages, harvests AWS/GCP/Azure credentials using TruffleHog, and establishes persistence through GitHub Actions backdoors - representing a major escalation in NPM ecosystem threats.

www.stepsecurity.io/blog/ctrl-tinycolor-and-40-npm-packages-compromised

Nx npm Packages Compromised in Supply Chain Attack Weaponizi...

Malicious Nx npm versions stole secrets and wallet info using AI CLI tools; Socket’s AI scanner detected the supply chain attack and flagged the malwa...

socket.dev/blog/nx-packages-compromised

I am not going to spend a lot of time writing about these particular pieces of malware. The posts above such as that from my mate Paul McCarty will do the job much better than I ever could.

I am also not going to talk much about how to respond to this directly. To those affected I wish you a relatively peaceful IR process.

Rather, I want to spend my words talking about how we as engineers, can move forward from this, providing some steps for short term solutions, plans for medium term solutions and then thought pieces about longer term plays.

Key thoughts:

• Our model of dependency trust is broken and has been broken since the foundation of package managers

• Industry standards for managing this risk are fundamentally faulty.

• Manifest-based composition analysis is at best a great distraction from improving your security posture

• Vulnerability management is less about managing vulnerabilities and more about managing expectations, CVSS/EPSS you name it, they are all largely awful. The number of CVE’s never gets smaller and your system never gets more secure.

• Reachability analysis might give us some relief from this but I’ve yet to see if actually provide benefits to triaging vulnerabilities.

Anyways on to the actual practical bits, this post is going to examine short term solutions for JavaScript and Python in a very compressed manner. I will take a look at other languages in future posts.

Pinning dependencies

Dependencies are typically specified within two key file types, manifest files and lock files. Manifest files package.json, go.mod) typically contain the list of direct dependencies e.g. ones you have explicitly installed via npm install or pip install . Lock files on the other hand contain not only the direct dependencies but they typically also contain indirect or transitive dependencies as well as some form of cryptographic verification information (largely used to reduce downloading things that haven’t changed or preventing corruption).

The dependencies within manifest files are declared typically using a constraint format such as package_name@>1.2.3 this would indicate that we are seeking to install the dependency package_name with a version greater than 1.2.3 from a package repository.

Version tags as represented within most dependency repositories once published are typically available forever. There are circumstances where a package may need to be removed (such as cases of supply chain compromise) and the behaviour of the registries vary. You can see a breakdown of behaviours below.

We can see that some registries actually don’t have facilities for removal whatsoever rather they rely upon the behaviour of the package manager itself.

These behaviours combined with the typical constraint definition within manifest files create the first component of supply chain risk.

Going back to the package_name@>1.2.3 example. When all is working, a maintainer publishes 1.2.4 and you run npm install which ultimately results in you getting v1.2.4 of package_name. What happens if (like recent compromises) a bad actor publishes v.1.2.5 ? Well next time you run npm install you’re getting the bad version and its game over.

How do we fix this?

Provide strict constraints

Define your dependencies using explicit constraints rather than wide constraints. For example you should specify package_name@1.2.4 rather than package_name@>1.2.3

This only protects your direct dependencies, unfortunately most dependencies have dependencies and they may not specify their dependencies in a manner that is as strict as yours.

Use deterministic operating modes

The largest cause of frustration and introducer of risk within dependency managers is when they operate in a non-deterministic manner, NPM is a great example of this behaviour:

Running npm install results in an interesting set of behaviours

• Install performs a full dependency resolution which makes it slow. This process is necessary if you don’t already have a package-lock.json but is entirely superfluous if you do.

• If dependencies (either direct or indirect) have loose constraints install will update them transparently without prompting. This is fine if you intend it, but can be a disaster when security is the aim.

Arguably this violates the entire point of a lock file but hey.

NPM offers an alternative install mode called ci which is designed to install exactly what is within the package-lock.json and will fail if it doesn’t match or can’t be installed.

So for NPM people:

• Developers: Use npm install when adding or upgrading dependencies, then commit package-lock.json. Afterwards use npm ci

• CI/CD & Production: Always use npm ci to guarantee a clean, reproducible install.

I’ve highlighted similar operating modes for other dependency tools within the Javascript/Python ecosystem.

Pythonistas: I recommend the use of Astral’s uv dependency manager. It’s just all around better than any other dependency manager from a speed and safety perspective.

Js/Ts Developers: I recommend the use of bun or pnpm. Again these are just all around better than the other options.

Disabling or limiting lifecycle scripts

A common feature of dependency managers is their ability to run scripts through the lifecycle of the package install process e.g. pre/post install. These are also an extremely common attack path for malware authors as they tend to be overlooked by security tooling. Sources for numbers below will be provided in a research doc.

Javascript

• Total Packages: 2.5M+

• Lifecycle Hooks: 2.2% use install scripts (~55,000)

• Malicious Usage: 0.9-1.3%

The Javascript ecosystem is the clearest case for disabling lifecycle hook scripts, several modern dependency managers such as bun and pnpm actually do this by default. They require you to explicitly enable each lifecycle hook for each dependency that has them. If switching dependency managers isn’t possible you are best served by disabling them via the npm ci --ignore-scripts command. If you’re in the rare situation where you need a script it is best to look at moving to bun or pnpm.

Python

• Total Packages: 530K+

• Lifecycle Hooks: ~33% use setup.py (~175,000)

• Malicious Usage: 0.06-0.2%

The Python ecosystem is a little less clear, unfortunately there isn’t a direct guaranteed means of disabling pre-install scripts nor does the data really support it. There is the ability if you’re operating with a relatively well supported set of dependencies to force the installation of binary wheel formats rather than building anything from source.

Python as an interpreted language relies heavily upon system libraries get a lot of its work done. Some of this code needs to be built for specific systems but in most cases the interfaces they interact with are common enough across a set of systems (or the dependencies license permissive enough) that they can be pre-compiled and distributed as binary that will run just work. In the Python ecosystem these are called wheels a wheel is basically zip file that contains the python source as well as the binary extensions.

Using wheels means you don’t execute the setup.py script which closes down a vector of risk. You can force pip and uv to attempt to only use wheels by passing the --only-binary=all command. This will fail if a dependency you’re using doesn’t have a wheel though. In pip you can also pass a list of dependencies that you want installed only from wheel so this allows you to find the ones that have setup.py scripts and specify them. Luckily uv is just plain better and allows you to pass --no-binary=some-other-package package_name as well, so you can specify exactly the dependencies to install from source and just audit those for risk. If you combine this with the previous suggestion of dependency pinning you are narrowing the window of risk down to when you legitimately update the dependencies that you allow pre-install scripts to run in during a supply chain attack.

That’s enough for now, I will spend a bit of time on Java for the next post.

Welcome to my new series: Pragmatic AppSec. It aims to provide security professionals at organisations that are AppSec curious with the necessary information to kickstart a program.

AppSec, whilst being extremely effective is frequently deprioritised, the success of it at an organisation is frequently dependent upon a single engineers willingness to drive it. It’s important to note that this series is based off my own experience and every business is different. The aim is to make this as agnostic as possible because AppSec is for everyone that has an App. I’ve been at large companies, medium companies and very small companies and have seen and implemented various versions of this. You will see me frequently talk about discovering business needs and appetites, this is an important aspect of any cybersecurity program: your solution needs to be right-sized to risk, resource and scale.

Prerequisites

There is no specific profile of what makes a good Appsec engineer or leader. What you’ve done in the past just determines how much work you’ll need to do to give people the right information and support. That being said there are a few stereotypical archetypes within the Appsec sphere. I’ve outlined these below for posterity.

The former pentester: a lot of people in the field have worked in offensive security. This history shows in the way that they approach solving application security problems. I’d argue this strain of thinking dominates Appsec, sometimes to its detriment.

The compliance person: There is strain of compliance framework driven approaches to AppSec. This attracts a compliance centric archetype to the role. Again this background shows in the way the program runs and problems are solved.

There is nothing inherently right or wrong with these archetypes and their approaches will suit different businesses differently.

Fun fact: I don’t come from either of these backgrounds and a lot of the best Application security engineers I’ve met don’t either. To reflect this I figured I’d do my least favourite thing and talk about myself.

What is my background?

That’s a bit of a weird one, I started by studying tech at university and spent some time writing software. I very quickly discovered that I did not want to do that and ran to Systems Administration. This changed very early onto my career into DevOps and SRE and Security kind of just led from there.

The common thread woven throughout this is that I’ve spent a lot of time being close to engineers. DevOps and SRE was all about the product and engineering interface and Developer efficiency was my aim. My early security roles were all about securing developers either through building a product for them or working as a security engineer within Developer Platforms teams. Engineers productivity and safety was the ultimate deliverable for those roles.

As you will soon see successful Application Security requires working closely and considerately with engineers. Their productivity is the primary currency (aside from dollars) that we spend when we add security to a process.

Part 1 of this series looks at creating a common understanding of not only what AppSec is, but the establishment of an underlying map that helps guide you towards success. It’s not the most practical of the series but is certainly one of the most important. We are establishing the common language a.k.a the lingua franca of AppSec.

What is AppSec?

AppSec, rightly or wrongly, means a lot of different things to different people and that’s great! But for a program to work we need a common definition. Security historically was an afterthought, we built software and then worried about security. This is generally regarded as a bad idea, software is a brittle and fickle entity that will happily work its way around any badly structured controls. As the concept of SaaS became the predominate form of software consumption we quickly discovered through black eyes and breaches that we needed to change how we approached software security. Application Security or AppSec arose to fill this need.

If you asked most people what AppSec is the answer you will get is something along the lines of:

Application security is the practice of protecting software applications throughout their lifecycle by identifying, fixing and preventing security vulnerabilities through secure coding practices, automated testing tools, security reviews and runtime protection.

This is a great description of the kinds of tools and processes that are involved in an AppSec program but I would argue it really misses the crux of AppSec. Culture

I like to define AppSec as:

Application Security (AppSec) is the practice of empowering engineering teams to build secure software through architectural guidance and frictionless, well-integrated tooling that enhance delivery velocity rather than impede it. AppSec prioritises secure architecture and design guidance over tooling alone, with tools serving to support and execute that guidance effectively. Success depends on cultivating a security-conscious culture and communicating meaningful metrics tailored to both engineering teams (actionable technical insights) and management (business risk and program effectiveness).

Yeah, it’s a mouthful. I promise I will explain it better in the future.

A rationale for thinking this way

Now this definition might raise eyebrows, for starters, shift-left happened and we are still dealing with the side effects of that. Why are we focussing on developers? Shift-left and historical approaches like it made a series of mistakes when attempting to involve developers into the security cycle; for starters developers are not security people. The approach I advocate for in the above definition and this series corrects these mistakes by ensuring that responsibility still lies with the AppSec function.

We are engineers as well and therefore need to be at the centre or at the very least operating within the confines of the engineering function at an organisation. Our aim is not to make security developers responsibility, rather we seek to encourage them to build in a manner that is inherently secure. This aim is achieved through guidance, involvement, understanding and consideration.

Whenever your thinking about how to approach this:

• Developers do not exist to fix our vulnerabilities they have a product to deliver. We need to enable them to deliver a product and do it in a way that addresses our vulnerabilities.

• Developers are experts at writing software. We need to mirror this expertise and innately understand what they are doing.

• Developers are overburdened and frequently under the pump. We need to be a pressure relief valve and encourage them to help us find pressure points.

• Developers want to maintain their software and have a functioning safe development environment but frequently lack the ability or capacity to obtain justification for doing so. We need to help them get that justification.

The virtuous cycle of success in AppSec

At the core of any AppSec program you should have three core aims/principles/objectives/goals whatever floats your boat. I’ve created the following acronym as a memorisation method.

Mould, Advise, Publicise - M.A.P

• Mould: Shape a security conscious, blame-free culture centred around collaboration that empowers engineers, whilst enabling you to ask for things when you need them.

• Advise: Provide suitable guidance/advice/support and enable it with tooling that is velocity considerate and meets developers where they are.

• Publicise: Expose and communicate success in a manner suitable to your audience - both engineers and leadership.

Each of these objectives sustain and nurture each other and will enable you to achieve whatever security/risk aims your company has.

What does this look like in practice? A virtuous cycle of success

The M.A.P that reflects the territory

Moulding a culture helps you identify opportunities to advise, these give you success to publicise which then encourages further cultural improvements. The self-sustaining nature of this cycle creates a positive feedback loop that ultimately drives you towards a successful AppSec program.

A practical example

AppSec Engineer Billie has identified that we have a series of vulnerabilities within container images at the organisation that are persisting across multiple sprints. Billie has invested a lot of time with Engineering Team A.

Through conversations and pair coding sessions she has identified that the reason they struggle to patch vulnerabilities in their containers is because they are just too complex. Billie has provided the engineers with a safe space to talk about frustrations and can readily map her needs to theirs.

Billie offers to help them build better containers, focussing on the developer experience in communications. “Let’s make these easier for your team to iterate on these so that you can test our new versions of X language”

Through iteration and work (yes she might be actually helping make a container) Billie has helped produced a container that not only aids the developers in their aims but also reduces the attack surface and vulnerability numbers substantially.

This is communicated to engineering leadership as a win for developer productivity, to security leadership as a win for security and to engineers as a win for making their job easier.

Engineering Team B enjoying the benefits of Engineering Team A’s new containers, reaches out to Billie and asks her to help them with their build pipeline. Billie has identified that vulnerabilities within CI/CD are not being readily maintained.

And so on…

Thus the virtuous cycle sustains

Alright that’s enough for now

In future posts we will examine each of these pillars in detail and look at how we can build them out but for now keep this model in mind as we delve deeper into this.

Thanks for reading

A good friend of mine who runs a great Appsec company, sent me this great article about a month ago. It wasn't new to me, but in recent weeks it's really been playing on my mind.

This hit its peak when I gave a guest lecture at QUT to a class full of people looking to get into tech and cybersecurity. It was a great opportunity and I am extremely grateful for the chance to help guide the next generation of engineers.

The job climate is obviously a little whack at the moment so the usual set of questions came up:

• How do I get into X job?

• What can I do to stand out in an interview?

The kinds of advice I fall back to here beyond what they might be already doing are typically things like:

• Have a portfolio

• Learn in public

• Broaden your skills

• Focus on the market and the needs of the businesses in it

It struck me after this lecture had concluded that a lot of these behaviours are exactly the kinds of extracurriculars mentioned in this article.

These are the things that result in people getting stuck going from mid to senior and above roles (in an individual contributor space). The irony of this thought occurring while I was doing exactly the kind of activity that's the subject of this article is not lost on me. Nor is the fact that I am writing about it in a blog article.

It's a curious industry we work in (cybersecurity), the path to success in our roles is usually circuitous and tangled. The ability to demonstrate your impact when you're slowly cutting your way through the jungle is vital to your long term success, it's also a skill that a lot of people lack.

The most common method (at least from what I have seen) to make up for this shortfall is to focus on doing everything that isn't your job. You're part of the community, you're writing blog posts, maybe you maintain some open source projects. There is nothing wrong with that, we need people to shape communities, maintain software and share their thoughts; but you need to do your damn job be able to communicate the impact it's having.

The idea of both actually having impact and communicating it is something so thoroughly missed in both formal education and informal education. It's also arguably why a lot of big tech companies have such awful approaches to role changes (10k word role change docs read by people that you've never worked is so good right?). Forcing people to write large documents that justify their existence gives people that aren't doing their job, or lack the skills to demonstrate impact, the ability to spend time pretending they are doing their job. The need to create the appearance of impact also creates a whole graveyard of barely functioning PoC's propping up large tech companies. A lot of these same problems still exist in smaller contexts, the difference is that people just don't get promoted/maybe lose your job and/or the size of teams gets larger than it needs.

Here's the rub, I've always found it incredible just how much time people in security engineering roles have to write thought-pieces. Again irony, I am an engineering leader writing a thought piece blog. For full transparency, finding time write a blog is like herding cats for me, my participation in various groups and programs is a carefully negotiated dance and the rare occasion I get to do anything new it requires several rituals to all known gods.

Enterprise security is rough, product security is frequently non-existent and we are always talking about just how threats never stop coming. We speak constantly about how we always need more people. All of this is to say that if you're taking the time to write a blog or run a community group you should consider whether you're doing your job and demonstrating your impact. This is a super vital time to consider this, the world is changing and regardless of your position on AI it will have an impact on how people do their jobs, the types of roles that are available and the quality of work people expect.

So ask yourself an honest question am I:

• doing my job

• demonstrating the impact

If the answer is yes to both, then great, thank you for your time.

If the answer is no to either of these, well here is what can you do?

1. Do your damn job

2. Measure your damn impact (if you don't know how I am going herd some cats and write posts for you)

3. Talk about your damn outcomes

4. If you've still got time left over do the rest