We have all sat through that meeting.

A critical bug hits production. The system goes down, payments fail, or a major client gets locked out. The management team calls a sudden emergency review meeting.

But within ten minutes, the meeting completely derailed. Instead of trying to fix the problem, everyone starts pointing fingers to protect themselves:

• Developers blame vague product requirements.

• Product Managers blame rushed deadlines.

• QA Engineers pull up old Jira tickets to prove they warned everyone weeks ago.

By the end of the meeting, the actual problem is still there. Everyone leaves the room feeling exhausted, defensive, and frustrated.

When reviews turn into blame games, everyone loses.

The root cause gets ignored, the team loses trust, and the same mistakes happen again during the next release.

High-performing engineering teams handle this differently. They use blameless post-mortems. Instead of looking for who made the mistake, they look at the system that allowed the mistake to happen. They protect their culture and deployment speed by focusing on process gaps instead of fixing the person.

Redefining Accountability: What Does "Blameless" Mean

There is a misconception that blamelessness means a lack of accountability. It does not mean that the developers have the right to write sloppy code, skip code reviews, or not follow the core testing workflows.

Blameless postmortem works on a culture with the basic assumption that everyone in the team came to work to do a good job with the available resources, information, tools, and time.

If a senior developer pushes a bad configuration change that accidentally drops a database container, blaming the developer/engineer for that is lazy leadership. The engineer has no intention to crash the system.

When you adopt a blameless mindset, the questions you ask change:

• The Lazy Approach: "Who approved this pull request without checking the configuration script”?

• The Organized Approach: "Why did our deployment pipeline allow an invalid syntax change to bypass our automated validation gates”?

• If your system allows a single human error to trigger a catastrophic production failure, the vulnerability is not the person who does it; it is the architecture.

The Hidden Cost of Fear in QA and Development

When fear enters a team’s culture, software quality is impacted first. If your team members are secretly worried about being out during a retro, their behavior changes, and it might harm the quality of the product:

• Information Hoarding: People stop discussing the flaky tests, minor performance regressions, or architectural technical debt. If they identify an issue, they worry they will be assigned to fix it under an unreasonable timeline or blamed for its existence.

• Analysis Paralysis: Teams avoid taking calculated risks or introducing modern testing methodologies. They feel it is safe to follow the rigid, outdated, old checklist since it has a list of items for self-defense.

• Slower Release Velocity: Code reviews will become bureaucratic bottleneck sessions where developers pass the ball back and forth, and it will make the late deployment times to ensure their names are not attached to an isolated mistake.

A culture of blame actively creates a slower, cumbersome development cycle. When people know they won't be publicly questioned for an unexpected edge case, they share lessons learned immediately, catch vulnerabilities earlier, and work together to build permanent automated barriers.

A Tactical Blueprint for Your Next Incident Review

Transitioning your team to a blameless postmortem review model is how you structure the conversation in the next meeting. It is not needed to have a massive corporate policy change or HR mandate.

1. Separate the Action from the Actor

It is important how you describe the event, the language.  When reviewing the timeline of an incident, restrict the use of personal names in the documentation and the discussion.

• Instead of: "When John deployed the updated package at 2:00 PM..."

• Try: "When the package update was deployed to production at 2:00 PM..."

It looks like a minor language change, but it helps the defensive discussion in the room.  It focuses the entire team’s collaboration on the sequence of events, not the person who has initiated the deployment process.

2. Map the Systemic Layers (The "Three Whys")

Instead of keeping the surface-level action, drill down into the environmental factors.

• Why did the bug hit production? Because the automated integration test didn't catch it.

• Why didn't the test catch it? Because the test environment was running an outdated mock dataset.

• Why was the dataset outdated? Because we don't have an automated process to sync the staging data with production schemas.

Now you have an infrastructure problem to solve, instead of a person to reprimand.

3. Create Process Guardrails, Not Performance Penalties

A great incident review/postmortem meeting should never end with a vague conclusion, “Testers need to pay closer attention next time”, or “Developers need to double-check their work before deployment”.

They are wishes.

Every retrospective must conclude with concrete, trackable engineering tasks. If a severe bug slipped through, the outcome should be like this: "We are writing a native regression test for this specific endpoint and adding it to our main CI/CD pipeline by this Friday”.

Shift Your Focus to Resiliency

Software is built by people, and mistakes are inevitable. The question is whether an organization spends its energy assigning blame or strengthening the systems and processes that help prevent those mistakes from recurring.

World-class product teams focus on continuously improving the way they work instead of on finding someone at fault. Strong systems create better outcomes than blame ever will.

Bio:

Bhavani is the Director of Product Management at QA Touch and a seasoned leader in product management. With certifications as a Scrum Product Owner, Digital Product Manager, and Software Test Manager, Bhavani brings a wealth of expertise to her role. Her passion extends beyond product management to testing, blogging, reading and cooking, making her a well-rounded leader with a keen eye for both technical and creative pursuits.

LinkedIn: https://www.linkedin.com/in/bhavani-ramasubbu-34372222/

Disclaimer: AI tools were used for grammar and spelling review.

Every year, I pay for insurance hoping I never have to use it. That's the point of insurance. It is there, IN CASE we need it. But I would much rather not think about it, honestly. 

Unfortunately, that's exactly how a lot of organizations think about quality.

They know it's important. They know they need it. But it never seems urgent until production is down, a customer escalates an issue, security identifies a vulnerability, or an executive asks the question no one wants to hear:

"How did this happen?"

Suddenly, quality becomes everyone's priority. Budgets appear as if they were cast from a spell. New tools get approved. Processes are reviewed. Meetings fill all the calendars. For a brief moment, everyone is committed to "quality matters."

Then the dust settles.

The incident becomes a story people reference in meetings, usually as a win they were able to steer from an iceberg. Then ...priorities shift, and quality quietly returns to being viewed as “expendable” instead of a business strategy. If I had a dime for each time I've seen this cycle, I could afford a case of eggs at today’s prices.

But  every time, I think the same thing:

Why are we asking quality to fix problems it was never given the opportunity to prevent?

Quality Was Never Meant to Be the Cleanup Crew

One of the biggest misconceptions about Quality Engineering is that our job begins when development ends.

Find the bugs.Log the defects. Approve the release. Repeat.

That approach might have worked when software moved slower. It doesn't hold up in today's world.

Quality isn't something you add at the end of the development lifecycle. By then, you're mostly measuring the outcome of decisions that have already been made.

If those decisions were rushed, unclear, or made without understanding the risks, no amount of testing magically fixes them.

Testing is important. But testing has never been JUST the quality strategy. There is sooooo much more that goes into quality as a discipline, testing as a skill, and building a quality culture.

AI Didn't Create the Problem. It Changed the Timeline

One of the biggest conversations in technology right now is how AI is accelerating software development, which is extremely exciting. There’s so much that can be done. But in that same wide range of change, lies a smaller margin for error.

Imagine your engineering teams are suddenly delivering code several times faster than they were a year ago. Every incremental increase in velocity creates more opportunities for innovation...and risk.

Organizations that see quality as a final checkpoint often discover they've simply moved the bottleneck. Code moves faster, but confidence doesn't. Teams spend more time reacting than improving, as customer trust becomes harder to earn, and they end up the guinea pigs in the organization’s experiment to “token max”.

Organizations navigating this well aren't necessarily the ones with the biggest quality teams or the most tools. They recognized that when engineering changes, quality has to evolve with it.

They didn't wait for velocity to expose the gaps, because they predicted where they would appear.

The Conversation We Should Be Having

 One of the reasons I love working in quality engineering is that it gives you a front-row seat to how organizations make decisions.

 The healthiest organizations I've worked in don't treat quality as someone else's responsibility. Or as that person's responsibility or even that department's responsibility. It's owned and assumed to be owned by everyone. It's the very foundation that we are “in this together” to make an excellent product, and the quality shines through that.

 This doesn't happen in a vacuum. It happens in how people collaborate, how they show up, how they talk about risk, how they balance speed with confidence, if people are comfortable raising concerns before they become a customer problem. Notice how I said nothing about testing because testing is fundamentally a skill embedded within the discipline of quality. Much like surgery is a skill within the medical field. Surgery alone is not indicative of excellent medicine, but removing it where it is needed would leave a dangerous gap in care. Testing is no different. Testing alone does not create quality, but without it, weaknesses form in the foundation and eventually surface somewhere far more expensive. When the organization sees how quality is the investment that protects customers, supports engineers, and strengthens the business over time, it creates the very mindset that cultivates the quality culture.

Maybe the question isn't whether your organization values quality.  Most organizations say that's their highest priority. Maybe the better question is:

“When does quality enter the conversation at your organization?”

The answer reveals whether quality is still being treated like an insurance policy, or as a strategy that helps ensure your systems are working for the business, the people building it, and the people trusting you as they use it. 

Jessica Mosley is a Quality Engineering leader, international speaker, and consultant with more than 20 years of experience helping organizations rethink quality, engineering culture, and AI adoption. She writes and speaks on quality strategy, leadership, and building engineering organizations that deliver with confidence.

LinkedIn: https://linkedin.com/in/jessicamosley
Speaking & Consulting: https://linke.ro/jessicamosley

Traditional bugs crash as a symbol of failure. LLM bugs lie confidently. A null pointer
throws a stack trace you can grep for; a language model hands a customer a confident,
fluent, and completely fabricated refund policy and nothing in your CI pipeline blinks.

In 2024, Air Canada learned this the expensive way when a tribunal held the airline
liable for a policy its chatbot invented. The failure wasn't a crash. It was a plausible
sentence.

If you ship systems where AI is the product - chatbots, RAG pipelines, agents, your
assertion-based test suite is quietly lying to you. This article is about the framework I use to test these systems in regulated financial environments, where a wrong answer carries legal and financial consequences. I call it PROBE.

Why do your existing tests break

The instinct is to reach for the test you already know:

resp = chatbot.ask("What is the refund window?")
assert resp == "You have 30 days to request a refund."

This passes Monday and fails Tuesday, same code, same model, same meaning, different wording. The test is now the problem. The failure isn't a bug in the model; it's a category error in the test. You've encoded a single point estimate as ground truth for a system whose output is a probability distribution over token sequences. Temperature, sampling, a silently updated model checkpoint, or a re-ranked retrieval context will all move that distribution, and your == will fail without anything being wrong.

Four properties make these failures fundamentally harder than traditional ones. The
signal is a plausible wrong answer instead of a loud crash, there's no exception to catch, so the failure is invisible to anything watching for thrown errors. Reproducibility is gone: the same input may not fail twice, which breaks the bisect-and-repro loop
debugging depends on. The failure space is unbounded natural language, you can't
enumerate the inputs, so coverage in the traditional sense is undefined. And the blast
radius is legal and reputational rather than functional, a single bad answer to the wrong
user is a headline, not a Jira ticket.

PROBE is five practices that replace exact-match testing with something that holds up
against non-determinism.

P Probabilistic Baseline

Stop testing for a single correct answer. Run the same prompt 20 to 100 times and
measure the distribution of outputs, then define a pass band instead of a pass value for
example, "95% of responses must stay on-policy."

Treat each run as a Bernoulli trial: on-policy or not. With n samples you're estimating the
true on-policy rate, and sample size sets your precision. At n=20 the 95% confidence
interval around an observed 95% is roughly ±9 points too loose to gate a release on; at
n=100 it narrows to about ±4. So the count isn't arbitrary: pick it from the precision you
need to distinguish "95% compliant" from "88% compliant," because in a regulated flow
that gap is the difference between pass and incident. Define the band against the lower
confidence bound, not the point estimate, so you fail safe.

The part teams forget is drift: today's 95% can quietly erode to 80% next month as
models, prompts, or knowledge bases change. Baseline the distribution, store it, and re-
run on every dependency change, a model version bump is a code change even when
your diff is empty. Monday-morning tool: promptfoo.

R Red Team It

Attack your own system before your users or an adversary do. Probe systematically for
prompt injection, jailbreaks, and policy bypass rather than relying on ad-hoc guesses.
The mechanics matter: direct injection overrides your system prompt in the user turn
("ignore previous instructions"), while indirect injection hides the payload in content the
model retrieves a poisoned document in your RAG corpus, a crafted web page an agent
browses so the attack rides in through data your pipeline trusts. The second class is the
one that bites RAG and agentic systems hardest, because the malicious instruction never appears in anything a human reviewed.

Anchor your attack catalog to established taxonomies - the OWASP LLM Top 10 and
MITRE ATLAS - so coverage is auditable rather than improvised, and so an auditor can
map your tests to a recognized framework. The discipline that pays off: treat every
successful attack as a regression test you keep forever. Your red-team corpus should only grow.

Monday-morning tools: DeepTeam; PromptArmor

O Observe in Production

You cannot pre-test an unbounded input space, so accept that production is your largest test set. Log real prompts, responses, latencies, and tool calls and for RAG, log the retrieved context alongside the answer, because a faithfulness failure is usually a
retrieval failure wearing a generation costume. You can't diagnose a hallucination if you
didn't capture what the model was given to work with.

Sample live traffic and score it continuously with the same evals you run in CI, so
production and pre-release signals are measured on one ruler. Critically, alert on
distribution shifts, not just hard errors, the dangerous failures here don't throw
exceptions, they drift. A faithfulness score sliding from 0.94 to 0.86 over two weeks is the real incident; no log line flags it unless you're watching the distribution.

Monday-morning tools: Langfuse; LogMiner

B Behavioral Contracts

Write the rules in plain English, first, with your stakeholders in the room. Define explicit
MUST and MUST NOT statements: "MUST NOT quote a refund policy that isn't in the
knowledge base." These contracts become the specification your evals are written against — without them, "quality" is whatever the last person to look at the output decided it was.

The practical move is to make each contract measurable: a MUST NOT pairs with a rule-
based or model-graded check, a MUST pairs with a metric and a threshold. A contract
you can't express as an eval is a wish, not a spec. This is the cheapest practice on the list and the one that makes the other four meaningful.

Monday-morning tool: plain-English rules.

E Evals Not Assertions

Replace exact-match with graded quality. Score faithfulness, relevance, and safety rather than string equality, and combine rule-based and model-graded evals: rule based checks (regex, schema, banned-phrase lists) are deterministic and cheap but brittle, while model-graded checks catch semantic failures but carry their own variance so calibrate the grader against a human-labeled set before you trust it. Rules for the bright-line violations, model grading for the judgment calls.

Returning to the refund example, instead of asserting an exact string, check that the
answer is grounded in the knowledge base:

from deepeval.metrics import FaithfulnessMetric
from deepeval.test_case import LLMTestCase

case = LLMTestCase(
  input="What is the refund window?",
  actual_output=resp,
  retrieval_context=kb.search("30 days"),
)
metric = FaithfulnessMetric(threshold=0.9)
metric.measure(case)
assert metric.is_successful() # gate on the score, not the string

Faithfulness here decomposes the answer into atomic claims and checks each against the retrieval context, returning the proportion that are supported. The test no longer cares whether the bot said "30 days" or "you have a month" it cares whether every claim traces back to the source. Then gate releases on eval scores the same way you'd gate on a test pass rate.

Monday-morning tool: DeepEval

Knowing which problem you're testing

Before you run any of this, ask one question: did AI write the code, or is AI the system? If
AI wrote the code, traditional QA still applies SAST (SonarQube, CodeQL), package
verification (Snyk), and mutation testing (PIT, Stryker) though the risk profile has shifted,
since studies in 2025 found roughly 45% of AI-generated code carried security
vulnerabilities, with the rate exceeding 70% for AI-generated Java specifically. Tests still
apply; trust does not.

But if AI is the system, run the full PROBE loop. That's the category where assertions
break and where the framework earns its place.

Start tomorrow

Every tool named here is free and open-source. You don't need all five practices at once, pick one row and ship it this week. Write three behavioral contracts. Wrap one prompt in promptfoo and run it fifty times. Point a faithfulness eval at your highest-risk flow and watch the score for a fortnight.

The mindset shift is the whole game: stop asking "did it return the right string?" and start asking "did it stay within the band of acceptable behavior, and will I know when it drifts out?" Traditional bugs crash. LLM bugs lie. Test for the lie.

Tanvi Mittal is a Senior SDET and Test Automation Lead at U.S. Bank, where she leads
QA for AI/ML systems in regulated financial environments. She is the creator of the open-source LogMiner-QA and PromptArmor tooling, an IEEE Senior Member, and a speaker on testing non-deterministic systems. She writes on AI quality assurance at
medium.com/@tanvimittalQA.

Our current job market is a hard one. There are lots of people on the market and it seems like what employers are looking for changes more often than the weather. The world of Software Quality is also changing rapidly and shaping the job market along the way. Before working in Software Quality I worked in Human resources. In addition to this view from the other side of the desk, I have also applied to literally hundreds of jobs over co-op placements, two lay offs, and several job and career changes. If a close friend asked me about job searching as a Woman in Test in our current market this is what I would tell them:

1. Remember that a lot of Job Searching is not about you and there is so much of the process that you can't control. Postings can get pulled, decisions are made on criteria you don't know about, and some places post both internally and externally. Take nothing personally. You could be the most perfect person for a role and still not get it. This is not a reflection on your worth or value as a person or as a tester.

2. Add a skills summary at the top of your resume and use this as a space to customize your resume to make it apply to job postings you are going for. This saves you time and also makes it easier for people to screen you in - not out. As a bonus you can summarise and personalize this section and use it in your cover letter. This saves you time and energy while creating a cohesive application package.

3. Tell your story with numbers where possible - "5 years of experience automating regression tests" or "reduced manual test load by 40%" is easy for people to understand and does the work to really sell your skills. It also allows someone like a recruiter or HR person to quickly see how you fit what the team is asking for so they can decide who to advance to the interview stage. The easier you make this process the more likely you are to get moved onto the next step.

4. AI screening tools are here to stay. Regardless of how you feel about this fact, making friends with AI can help you stand out and beat those AI gate keepers at their own game. Run your resume, cover letter, and the job posting through an AI agent and ask it to make suggestions for how to make your application package stronger in case the company has an AI screener. This is more likely to get your resume and you in front of the humans that actually make the hiring decisions.

5. Templates can also be a huge help for a busy job seeker. Most of us apply to 1-2 types of jobs in a given job search. Create a basic application for each type of job you are applying for (ex. QE with Manual focus vs. QE with automation focus). Use these as a base and customize your application for each job by focusing on the key things the job application asks for that are unique. This means you aren't starting from scratch every time but you are also not sending out the same generic application over and over.

6. Careful AI usage can also save you time at the writing stage. Go through the posting you are applying for and list all of the skills they are asking for that you have. You can then feed this to an AI agent along with the above mentioned template and use this tool to better customize your application materials to the posting you are applying for. This means you still sound like you but also save time by using AI for some of the heavy lifting. This also avoids the risk of an AI agent hallucinating a resume that is not really you or making statements you can’t back up at the interview stage.

7. Time box your applications - Recognize that even the most perfect app might not get you the interview (see point 1) so decide how much time you think it's reasonable to use on a given application and then stick to that. Every application you send is an investment of your time. Make sure the return on that investment is worth the time related cost.

8. Make your resume outcomes focused - Show what you achieved in each role and not just what you did. This helps paint a picture of what you can get done and why you would be a great fit for a team with an open role. These outcomes can also become great talking points when you get that all important interview.

9. Strive for balance - Job searching is a lot of work. Regardless of if you are doing it while employed and looking to trade up or after a layoff it can become all consuming. Remember that none of us ARE what we do for work and the rest of our lives matter too. Set limits on your job search and take care of yourself in ways that you enjoy so that you can take on the next big challenge that comes your way.

Above all, interviews are always a two way street. You want to sell your skills as best you can but also look for a role that is a great fit for you and your career. We all spend a lot of our lives at work and it is worth the effort finding a role that suits your skills and interests. Application packages are the tool that gets you in a room with some humans. With a little prepwork and a balanced time investment you can be well on your way to your next great adventure. 

Author Bio: Lindsay Weir-Mui is a Senior Software Quality Engineer located in Ontario, Canada. Accessibility in testing and the role of testers as advocates for inclusion and belonging is one of her passions. She also loves code that runs on the first try and exploratory testing a juicy new feature. Outside of work she drinks a lot of bubble tea with her husband and enjoys crime podcasts, baking, and needle felting. You can find her on LinkedIn: https://www.linkedin.com/in/lindsaycweir/ 

Disclaimer: An AI agent (Claude) was used to polish the final draft of this article and give some feedback on clarity and spot typos. The overall voice and ideas shared here are authentically those of the author.

Hi! I’m Judy. I’m peeking out from behind the curtain here at the Women in Testing newsletter to address a topic that is on everyone’s minds and social feeds: AI.

As a community member, I wanted to bring this topic up because I care about you, the reader. Most days, it’s hard to know what is true life or what is generated by some prompts. As our understanding of reality continues to get blurry, I wanted to let you know how we, as a community, feel about and are using AI.

For context, I’m the one who calls for submissions, helps edit, and publishes each article for the newsletter. I get to hang around with these amazing people and read their contributions. It’s a gift. And, I am only one person. I am doing this for free. Many of the members don’t have time to write on a regular basis. That’s why this newsletter exists: to help people share their knowledge, skills, and even to help them be a little bit brave if writing isn’t something they do often. It’s a joy. Such a cool opportunity. 

And, well, there is AI. And, I want you to know how we are using AI in our submissions. 

We’re okay with the robots

Many of us use AI in our day-to-day workflows. Some of us feel very bad about the recent changes in tech. Others are excited. Because we are many, the varied views of AI will show up in the newsletter. I use the term “okay” as a word to capture our many thoughts and feelings about it.

We want to act wisely

I’d say that accounts for almost everyone. In the testing circles I’m part of, we care about the work we do, as well as the tools we are using. No one is acting badly on purpose (to my knowledge). Many of us want to do the right thing and to do it well. This makes using AI complicated at times. 

Where we stand

I had a conversation with the community, and we agreed on a few things. 

• We take a neutral stance on AI. We are not anti or pro. Sometimes AI is useful and helpful. Therefore, if a writer of this newsletter wants to use AI, they are free to do so. 

• We will identify AI use. We want to be transparent about our use of AI. So, as a member of the community, we’ve agreed that if we use AI in our writing, we will make it clear in some form or fashion. It’s to help you, as the reader, decide how you want to think about an article. Again, this may get complicated, but transparency is more important than avoiding complicated conversations.

• It’s also okay not to use AI. Personally, I love the struggle of coming up with the start of a sentence or how to tie separate ideas together. Call me weird. I like being weird. That’s what makes this newsletter so cool. Some of us use AI. Others don’t. Everyone brings ideas that expand our vision of what testing can be, and we can continue to infuse care into the craft. 

Where I stand

Here’s where things get complicated again. I won’t be checking to see who is using AI or not. I trust the members of this community. If they didn’t say they used AI, I’m going to trust that they didn’t. I will remind them to tag it if so, but otherwise, I’m going to trust the adults in the room to be adults. I’m one person. I do this for fun. And, I hope you, dear reader, are having fun too. These are an amazing group of people, and we have so much fun together. It’s my hope that this newsletter will continue doing the good work that it does. I realize that my stance may make things complicated for you. If so, please let me know, either in the comments or on LinkedIn. I want to do my best, and I have limits. Learning what my limits are in an AI world is what I’m exploring. Would love to hear your take on this. 

Here’s to community

We’re the humans trying to figure out who we are, what our roles are, and how to walk through this crazy experience called life. I’d love to start the conversation about how you're being transparent about AI use in your organization and your daily life. The more we can talk about where we are and what is working, the easier it will be to find our way. 

Cheers!

(Scurries back behind the curtain)

Judy Mosley is a QA Engineer, curator of the Women in Testing newsletter, Ministry of Testing Ambassador, and author of the Failure is Feedback newsletter.  As a member of the testing community, she loves breaking software, ensuring confidence across teams,  and amplifying the voices of testers around the world. You can find more about her work on LinkedIn, and Failure is Feedback

Software, and the hardware that supports it, is a constantly changing discipline. It wasn’t that long ago when the majority of software operated in a monolithic system. Around the 80s was the emergence of Modular Architecture, which soon transitioned into Service Oriented Architecture. SOA grew into popularity alongside a popular shift in how we build software–Agile. Then, around the 2010s, we started to see the rise of Microservices, containerization, & DevOps.

These innovations have inherently made software products more complex–especially from a quality perspective–due to their distributed & decentralized nature. Building quality software today means the ecosystem of microservices, frameworks, cloud providers, processes, and so on need to operate as a smooth system to provide the end product to the user.

Why should we care about this stuff?

• Today’s software is inherently complex

• Quality practices should include all parts of the underlying software system

• Observability enables a deeper understanding of the full system supporting the software product–and the user’s experience

Basics of Observability

Observability is the ability to take external outputs, typically telemetry, to assess the state of the whole application or software system. Or, to take the definition from Dora.dev:

“Observability is tooling or a technical solution that allows teams to actively debug their system. Observability is based on exploring properties and patterns not defined in advance.”

Observability typically focuses on three main telemetry types—Logs, Metrics, & Traces—providing the data about your underlying system.

Logs

Logs are very granular & discrete records. They are time-stamped & immutable, so they are valuable when attempting to debug specific issues or errors. The content of logs can differ across different systems & log types. A log might capture a single string defining an error message, a state change, or a user action. A log might be a structured record that includes detailed metadata about an event that was fired.

Because logs are very precise, they are very valuable when debugging issues in production. There are several types of logs, including event logs, transaction logs, & server logs.

Metrics

Metrics are typically a point-in-time measurement of the state of a system or service. They are used to represent an aggregate value & are usually numeric.

When you think of metrics, you want to think about things like:

• How much CPU capacity does your application use in 5 minutes?

• What’s the number of requests per second?

• What’s your application error rate per minute?

• How much memory does your application use in 5 minutes?

You could also create metrics that are specifically focused on Business KPIs, like session duration, page views, & transaction volume.

Metrics are typically used to assess the system's health over time. They can be extremely valuable as an early signal that there is system degradation, trend tracking, or unexpected changes.

Metrics are a great early detection system for quality teams to utilize. A spike in error rates or a gradual increase in response latency can signal systemic issues before they begin impacting the user experience. And when extended to support business KPI’s, they can give your team insight into how users experience your software.

Traces

Traces record the data across a user’s session. Traces are composed of multiple spans–the individual units of work that map the full path of a user’s interaction across microservices, databases, queues, APIs, and anything else that may be a part of the software system. The span is what actually follows a user's request path across the system.

Traces are a great way to understand the flow of data across the different microservices in the application. They can be used to help debug more ‘Why’ questions or pinpoint specific issue areas, like:

• Why does this user flow perform differently than expected?

• Where is the latency coming from?

• Where is the bottleneck?

They are typically used for optimization efforts–they help teams understand where there could be system latency or bottlenecks.

Why Should Quality Teams Care?

The nature of building software is collaborative–we build the best products when we work alongside engineers.

Testers & quality folks are experts in quality–we provide a different perspective & way of approaching problems & solutions than engineers or product managers. It's when we combine these different perspectives & expertise that we build good products for our users.

As quality experts, we are uniquely situated to provide the most value when we are able to understand both the macro- and micro- parts of the system. Observability enables quality teams to shift towards being more proactive.

A quality team that understands telemetry can evaluate real system behavior in production. They can contribute to defining alert thresholds, use traces to triage incidents, identify patterns in metrics for continuous improvement, and so much more.

Observability is more than a single tool or a single setup—it’s a continuous improvement practice requiring thoughtful design and cross-collaboration. But observability is a valuable tool in building a holistic quality practice.

Author Bio: 

Sidney Karoffa is a QA engineer & systems designer focused on engineering enablement & just building good software. Passionate about UX, test innovation, & process optimization. Strong believer in democratizing technology & knowledge.

Learn more about Sidney and her work on LinkedIn.

Over the last year, testing has started to look more active than it has in a long time. Tickets now have generated test cases attached, prompts show up in PRs, and reviews include screenshots of “AI coverage.” There is visible proof that testing happened and that proof creates comfort. For teams that used to struggle to keep testing in sync with delivery, this feels like progress. On paper, it looks like we finally caught up.

Production does not look any safer.

I keep hearing the same two lines across teams. Either “AI will generate the tests” or “we’ll test ourselves.” Both sound reasonable when you say them. Both usually end the same way - more bugs are escaping to production than anyone expected. Not rare edge cases, not complicated failures. Basic correctness issues like double charges, broken access, sessions that never expire, or orders that exist without inventory. The kind of bugs where the only reaction is “how did we miss this?” You might have fired these employees before the vibe coding era.

If all this extra testing was actually reducing risk, these are the first problems that should disappear. The fact that they don’t is the real signal.

The issue is not the number of tests. It is what those tests are actually protecting.

AI doesn’t decide risk. It just expands what you give it.

AI testing sounds smarter than it really is. It does not understand your system, your users, or what failure would hurt the most. It does not discover risk for you. It simply expands whatever you put into the prompt.

A prompt is just your current thinking written down. The model elaborates on that thinking. It does not question it. It does not fill in the gaps.

If your understanding is narrow, the output will be narrow too, just longer and better formatted.

So when AI-generated tests look thorough but still miss obvious problems, nothing went wrong. The model tested exactly what you described and ignored everything you didn’t. It is being literal.

Where this usually breaks

This gap shows up in very normal work.

Take a ticket that says, “Add retry logic to checkout.” Someone writes, “Generate test cases for checkout retry logic.” The output looks perfectly fine. Retries on network failure, retry limits, timeout handling, and eventual success. From a functional point of view, it seems complete.

But checkout is not really a retry problem. It is a money and inventory problem. What actually hurts you is charging twice, reserving inventory twice, or ending up with payment and order out of sync.

Those risks are not mentioned, so they do not appear in the tests.

The model did not miss them. We never named them.

This is the pattern I keep seeing. We prompt around implementation details and forget the system invariants. Then production reminds us what actually mattered.

The quiet trap AI introduces

AI also makes it very easy to look serious about testing. You can generate dozens of scenarios in seconds and attach them neatly to a ticket. The output looks structured and thorough, so it feels like diligence.

Over time, the presence of that list starts to stand in for safety. “AI-generated tests for this” becomes shorthand for “we’re covered.”

But most of those lists are never read carefully. They are too long and too generic. The list becomes paperwork. Something that proves effort happened, not that risk went down.

So activity increases and confidence increases, but production incidents stay roughly the same.

AI did not create this behavior. It just made it cheaper.

Why flows aren’t enough

Most of the bugs that hurt teams are not flow failures. The happy path usually works. The screens load. The steps complete. What breaks are the states underneath.

Payment says success while access says no. A password changes, but old sessions stay active. An order exists without matching inventory. Different parts of the system disagree about reality.

These are states that should never exist.

But most prompts are written around flows. Test the login flow. Test the subscription flow. Test checkout failures. Flows are easy to imagine, so AI happily generates more flows. Meanwhile the real risk lives in conditions nobody explicitly defined.

AI can expand what you specify all day. It cannot guess what must always be true.

If you don’t decide that, nothing is actually being protected.

Where AI actually helps

AI is still useful here, just not in the way people expect.

It is not great at figuring out what to test from scratch. But it is very good at expanding constraints. If you clearly state a risk, it will generate many ways that risk could happen. It can help you see holes and question assumptions you might miss.

But only after you make the risk explicit.

So the leverage point is not “write better prompts.” It is “be clearer about what matters before you prompt.”

Once the intent is sharp, AI becomes useful very quickly.

How to make AI test what matters

The teams that get value from AI testing usually make one small change. Before writing any prompt, they pause and ask a simple question: what state must never exist in production?

Not scenarios. Not coverage. Not edge cases.

Just one unacceptable condition.

For example, payment completed but order missing. Subscription active but access removed. Password changed but old sessions still valid.

Once that is clear, the prompt becomes straightforward. Instead of asking for generic tests, you ask for tests that ensure those states never occur or persist. The output is usually shorter but much sharper, because every test is defending something specific.

Nothing fancy changed. The thinking just got clearer.

Try this next sprint

During refinement, pick the feature being discussed and force the team to agree on one forbidden state. If you cannot agree quickly, that confusion is probably the real risk. Resolve that first.

Then write your AI prompt around protecting that state. Every generated test should exist to prevent or detect that condition.

It takes a few minutes and usually does more for quality than another page of generic scenarios.

The takeaway

AI does not reduce risk just by generating more tests. It reduces risk when you first decide what must not break and then use AI to defend that decision.

Define the constraint first.

Then let AI expand it.

Otherwise, you are mostly producing better-looking paperwork, not safer software.

Our time and availability is limited, and we get to make choices on how we use that limited time. I changed my choice on test automation ten years ago, and framed the change with opportunity cost: Time I used on warning about test automation (and boy, I did, all the work on the snakeoil salesmen warnings) was away from using someone as great as me on doing automation better.

There are plenty of challenges on our way to succeeding with test automation, and looking around on the consulting scale, we haven’t quite cracked the success as an industry yet. Existing automation is more common than useful automation. And automation that pays back the investment needs a better framing than what we’ve been offered. 

With the company I’ve been keeping, I get asked for ROI (Return on Investment) calculations for test automation a lot. 

Tricky experiences

For something that is automated, test automation feels like a lot of (manual) work. We always modeled test execution incorrectly. At least now we have manual programming to align the idea with the introduction of generative AI. Generating code is a small part of software engineering. Exact execution in the same way is a small part of testing. 

It’s easy to come across naive ROI pitches for automation proposals. “Manual test X costs N hours. Automated test X costs M hours once. Therefore, savings!” It completely misses the idea that test automation is a subscription service, not a product purchase, and a major cost factor is adaptive maintenance, which is a key part of the subscription. If there were no change in the target of testing, little testing would be needed. But there is change, and we want to enable change, and the investment becomes continuous. 

Thinking back to a particularly successful test automation effort, in a scale where I would bring in a university researcher to collect Test automation improvement in a DevOps Team: Experience Report and publish it in an academic conference, it was not a massive scale reduction in manual testing efforts as such - the entire style of working for the software development changed with it. Dealing with 160 000 tests run and creating valuable features on an unusual scale with a fairly small team was a definite success. The investment side of it - rewriting product architecture and creating a test lab system where you could have a clean Windows operating system up and running in seconds were quite relevant investments. 

With the whole critical thinking and intellectual honesty framing that we aspire to in testing, my integrity isn’t allowing the answer I got in a recent job interview: “This is what the managers want to hear - I know it’s not that way, but I tell what is needed”

Say no to the labor-replacement calculators

The naive ROI pitch frames test automation as a labor-replacement calculation. I’ve been having one of those around just to show it for critique, adapted from Fewster et al. Test Automation, 1999.

There are wrong assumptions about this:

• Automation is not a one-time cost. Maintenance, particularly adaptive maintenance, is a dominating cost element. 

• Testing done manually is adaptive. We would run different tests and look at what the recent change is likely to impact, not run a blanket regression suite. 

• Automation moves effort elsewhere. While some feedback loops are replaced, new categories of work emerge, and thinking remains.

In search of a better framing: risk management and learning investment

When calculating ROI, a better framing to seek the monetary guesstimates for returns are on framing it as risk management and learning investment.

There’s another naive version on this framing, too, where we don’t discuss the workforce implications, particularly at scale. There are second-order, conditional effort savings with net effort reduction over time we should expect. The time savings are more on rework and coordination than on testing, making the calculations beyond testing activity a necessity. There are also costs related to the pressure for changes in team composition, requiring training and hiring investment in support of the change. 

Test automation doesn’t save money by replacing testers; it pays back by reducing risk, accelerating change, and protecting learning.

The ideas we should discuss: 

• Systems that work are faster to test manually. When basics work, people can repeat things faster without having to start reporting on the most recent surprises automation projects from. 

• Replenished results change the model of the risk surface. We focus on changes and intellectually scoping more, when we have a safety net on the more distant connections we consider relevant and worth always protecting for the business. We make defensive widening conversations part of automation design to change the system we work in. 

• Stop batching tests for releases. Continuous testing means the same tests do not need to be repeated as release testing, making releases faster. The effort is the same size, but continuous. 

• Make repetition cheaper. When automating, you decompose the quest for feedback differently. Same mechanisms of programming are available in the test target and the test system, opening options you did not have before. 

• Institutionalized learning. What you don’t capture in your pipelines leaves with the person leaving.

Tool ROI is not the same as automation ROI

I leave this write-up with one more insight from an open space conference Friends of Good Software (FrOGSConf). The conversations on tool selection ROI are not the same as automation ROI. With tools, we are arguing about our tradeoffs over time and organization. 

Buying a commercial license to a tool is like a gym card. Lovely equipment, takes a bit of learning to safely operate, help available. Stop paying your membership fee, and you no longer have access to whatever routines you built with the assumption of that equipment. It may just be that I would make a poor test automation tool salesperson. 

No matter what framing you come to on your conversation on return on investment, I hope some of these ideas help on the way.

Maaret Pyhäjärvi is Director of Testing Services at CGI Finland. Regardless of fancy titles, she is just a tester, if anyone ever is just anything. To support her testing, she is a polyglot programmer, conference designer, and community facilitator. She has delivered 700 talks and trainings in 28 countries on the side of her hands-on job in changing testing from within, and made it to the top-100 ICT list in Finland for 6 consecutive years, as well as been awarded by two major testing communities: Agile Testing Days Most Influential Testing Professional and EuroSTAR Testing Excellence Award.

by Christine Pinto

This isn't a tutorial. It's a postmortem of how I set up a beta test for 18 people without spending my evening copying the same checklist 18 times.

That was my goal when I launched the beta for Wizzo, your QA companion in Slack that helps turn conversations, specs, and screenshots into test cases. It also supports different kinds of collaborative team sessions, for example, around user scenarios, knowledge gaps, and similar quality discussions. I'm building this at my startup Epic Test Quest.

The product itself was ready. The beta setup wasn't.

Each tester needed their own place to report bugs and track progress. They needed realistic products and user stories to work with. And once the beta ended, I needed a clean way to reset everything without leaving test data scattered across tools and databases.

Doing all of that manually would have turned a quick beta launch into hours of repetitive, error-prone work, exactly the kind of setup I knew I'd regret doing by hand.

So instead of clicking through tools and hoping I didn't miss something, I automated the parts that mattered.

I needed to:

• Give each tester their own tracking thread

• Create five sample products with real user stories

• Make cleanup predictable once testing ends

To do that, I relied on three things: GitHub’s GraphQL API, SQL, and MCP (Model Context Protocol). Here’s how it worked.

The Challenge

Picture this: 18 testers, each needing their own space to report bugs. Each needs a checklist to track progress. And I need to see everyone's status at a glance.

Manually, this would mean creating 18 threads, copying the same checklist over and over, and renaming each one. And one typo means starting over.

Then there's test data. Testers need real products to play with. They need user stories to generate test cases from. Setting up fake data in Jira? Another hour, easy.

And when testing ends? Cleanup. Removing test data. Resetting the database. More manual work.

I knew there had to be a better way.

GitHub Discussions + GraphQL API

I chose GitHub Discussions for tester feedback. Why? It's free. It's public. Testers can work at their own pace. And I can see all threads in one place.

But I didn't want to create 18 threads by hand. GitHub's GraphQL API lets you create discussions programmatically, exactly what I needed.

One script creates all 18 threads:

# Array of tester names
TESTERS=(
  "Sarah Chen"
  "Mike Johnson"
  "Priya Patel"
  # ... 13 more names
)

# Create a discussion for each tester
for tester in "${TESTERS[@]}"; do
  gh api graphql -f query='
    mutation($repositoryId: ID!, $categoryId: ID!, $title: String!, $body: String!) {
      createDiscussion(input: {
        repositoryId: $repositoryId
        categoryId: $categoryId
        title: $title
        body: $body
      }) {
        discussion { url }
      }
    }
  ' -f repositoryId="$REPO_ID" \
    -f categoryId="$CATEGORY_ID" \
    -f title="Beta Testing Report - $tester" \
    -f body="$TEMPLATE"
done

The $TEMPLATE variable holds a progress tracker with checkboxes, resource links, bug report format, and feedback form links.

One command. Eighteen personalized threads. Done in under 2 minutes.

Each tester found their name in the Discussions tab. They checked off items as they tested. I watched progress from one page.

SQL: Setting Up and Cleaning Up Test Data

Beta testers need something to test with. For Wizzo, that meant sample products and user stories.

I created 5 real-world products, Uber, Airbnb, Amazon, Stripe, and Chase Banking, each with features and user stories in Jira. This gave testers real scenarios to generate test cases from.

After testing? Cleanup time. Here's where SQL shines.

-- Remove test products
DELETE FROM products
WHERE team_id = 'beta-test-team-id'
AND created_at > '2024-12-01';

-- Clear expired draft test cases
DELETE FROM draft_test_cases
WHERE expires_at < NOW();

I also wrote queries to export metrics before cleanup:

-- How many test cases did each method generate?
SELECT type, COUNT(*) as count
FROM test_cases
WHERE team_id = 'beta-test-team-id'
GROUP BY type;

-- What was our draft-to-save rate?
SELECT
  COUNT(*) as total,
  COUNT(CASE WHEN status = 'promoted' THEN 1 END) as saved,
  ROUND(100.0 * COUNT(CASE WHEN status = 'promoted' THEN 1 END)
        / COUNT(*), 2) as save_rate
FROM draft_test_cases;

These queries told me what worked. Which features did testers use most? Where did they drop off? Data answered those questions.

MCP: AI-Powered Automation

MCP (Model Context Protocol) lets you connect tools directly to AI assistants like Claude. For Wizzo's beta test, I used it to manage Jira data. Instead of clicking through Jira's web interface, I could just ask Claude:

"Show me all unassigned tasks in the SCRUM project."

Or: "Unassign all 47 tasks so we can reset for the next tester."

Here's how MCP connects to Jira:

{
  {
  "mcpServers": {
    "jira": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "-e",
        "JIRA_URL=https://epictestquest.atlassian.net",
        "-e",
        "JIRA_USERNAME=your-email@domain.com",
        "-e",
        "JIRA_API_TOKEN=your-token",
        "mcp-atlassian"
      ]
    }
  }
}

Once connected, Claude could search, update, and manage Jira issues. No clicking required.

I also used MCP with Supabase (our database). Claude could run SQL queries directly. This made debugging much faster during the beta test.

What I Learned

Time saved: About 3-4 hours of manual work became 30 minutes of scripting.

What worked well: GitHub Discussions gave testers freedom to work async, GraphQL made batch operations simple, SQL made cleanup predictable, and MCP let me manage everything from one place.

Tips for your next beta test:

1. Script everything you'll do twice. If you're creating threads for 5+ people, automate it.

2. Plan your cleanup first. Know how you'll reset before you start.

3. Export metrics before deleting data. You'll want those numbers later.

4. Use MCP if you're doing lots of tool-switching. It saves context-switching time.

Try It Yourself

All the scripts I used are public:

• Beta Testing Setup Documentation

• Quest for Quality repo (our test playground)

Looking back, was this a lot of automation for just 18 testers? Probably.

But the scripts are still there. The setup is repeatable. Cleanup isn't scary. And the next beta won't cost me another evening of copy-paste work.

For me, that's the real win.

Got questions? Find me on LinkedIn or drop a comment below.

Christine Pinto is an ISTQB-certified and award-winning tester turned tech founder with 18 years of showing teams why QA matters. She's been a manual tester, automation engineer, QA lead, and public speaker. She's now co-founder of Epic Test Quest, building Wizzo with testers like you, to make quality management easier, faster, and more collaborative. The future of development will rely on QA more than ever before, and she wants to ensure that every tester is ready to step into that leadership role. As a native Berliner, she brings a direct, no-nonsense style that keeps QA at the center and treats quality as something built in from the start. Outside of QA, she loves gaming, anime, classical music, and adventure travel.

Most of the time when you raise an issue with developers, you get a familiar reply:

“It works for us.”
“We’ve always done it this way.”

And sure - it does work for them.
But that’s not the same as working well. Or working for everyone.
Because when something “just works” for the people who built it, it usually means nobody else can verify, control, or question it.

That’s where testable design comes in.

What Testable Design Actually Means

Testable design isn’t about writing more tests.
It’s about building in a way that makes testing possible, fast, and meaningful.

A design is testable when:

• You can verify behavior without touching production data.
  

• You can isolate components without needing the whole system.
  

• You can observe what’s happening under the hood.
  

• You can change inputs and expect predictable outputs.
  

When none of this is easy, testing becomes expensive - and feedback slows down. The product still “works,” but it stops evolving safely.

The Hidden Cost of “It Works for Us”

When teams say “it works for us,” what they often mean is:

• It works on our setup.
  

• It works with our data.
  

• It works with our assumptions.
  

That mindset builds invisible dependencies - on people, configurations, and habits. The product looks stable, but is actually fragile.
You can’t test it without replicating someone else’s mental model.

An untestable design is a dependency trap. It ties knowledge to individuals instead of systems

How to Design for Testability

Here’s what testable design looks like in practice.
You can use this as a checklist when reviewing your next feature or API.

1. Make behavior observable

Add logs, metrics, or state outputs that help see what’s going on.
If you can’t tell what a system is doing, you can’t test it effectively.

2. Keep boundaries clear

Each module or service should do one thing, expose what it does, and hide what it shouldn’t.
If testers (or automated tests) need to jump through layers to reach a function, it’s not testable - it’s tangled.

3. Inject, don’t hardcode

If your code depends on a specific file, URL, or time, allow that to be injected.
It makes testing easy and production safer.
Design for substitution - fake dependencies should work like real ones.

4. Separate data from behavior

Don’t bake test data or configurations into logic.
Keep them external and swappable. It helps testers validate different scenarios without changing the code.

5. Think about reversibility

A good design lets you undo things easily - reset a state, clear a queue, roll back a transaction.
If testing something leaves behind a mess, people will test less.

A Shared Responsibility

Testability isn’t the tester’s job alone.
It’s a design quality - just like performance or usability.
When teams build with testability in mind, feedback loops shorten.
You spot problems earlier, fix them faster, and gain trust in your system.

Testers don’t slow down development. Untestable design does.

Before You Ship

The next time you’re about to say “it works for us,” ask:

• Can someone else validate it without me?
  

• Can we see what happens when it fails?
  

• Can we isolate and test just this part?

If the answer is no, you’re not done designing - you’re just done coding.

Final Thought

Good design isn’t about adding tests.
It’s about making things testable by design - so the whole team can see, question, and trust how the product behaves.

Because “it works for us” is never enough.
It should work for everyone who has to build, test, or use it.

About the author

Hanisha Arora is a Growth Hacker Lead focused on advocating for products and building better systems. She works where technicalities meet product thinking. Hanisha writes about testing, systems, and the tiny human errors that turn into product lessons.

Learn more about her work on LinkedIn or follow her blog for more.

Growing up to be the tester I have become by now, I had two defining experiences of failing at job interviews. First was having to introduce myself to an audience, and I realized my fear of public speaking was not a limitation I can accept. Second was having to tell a joke, and I am still struggling with the idea of being intentionally funny. 

With a few talks under my belt to practice to overcome my first limitation, I realized that audiences sometimes do laugh when I speak. I like to think they laugh with me, with the space I sometimes manage to create for a bit of lighthearted insight. I still can’t crack a joke for the life of me, so I have learned to replace this with party tricks of testing. 

Party tricks are packaged responses to things we in testing often need to illustrate. For me they come in three forms, and I have a collection to share these days. 

1. Little exercises that create a shared experience

2. Stories with an educational point

3. Testing proverbs (avoid name) and quotes (drop a name)

I have found these to be useful from stages, in educating colleagues about testing perspectives and driving through essential lessons, and leaving something to remember. I will give a few of my go-favorites in the first category. We may come back with another post on the two others later.

Little exercises

Exercises that create a shared experience and drive an essential insight through it are community treasures. Finding them takes a bit of mining, and I have mined mine in conferences, open spaces, and online communities. While the original exercise is created by someone, they may come to me from someone else, and they may be reshaped with what I do with them from their original purpose. 

1. Raster Reveal by James Lyndsay, Workroom Productions
   https://www.workroom-productions.com/raster-reveal/
   
   Raster Reveal is a party trick you take out when your manager is wondering why the test you are doing takes so long - after all, they could get that done in 5 minutes, and you have already used 5 hours!
   
   In Raster Reveal, you move your mouse over an image that takes shape as more movement creates more precision. You can jump to conclusions of a unicorn, you may in fact add more detail before making conclusive judgments. In writing this, I finally concluded: not a unicorn, a white horse. Not a horn, just an ear. And finding the picture original confirms it.
   
   For any tech-minded testers reading the picture from the source, James has included a wealth of layers of FUN while you try to get to the answers.

2. Monkey Business Illusion by Daniel Simons via Cem Kaner
   https://www.youtube.com/watch?v=IGQmdoK_ZfY
   
   Tell people they have a test case: white to white player in the video, the ball is passed 14 times, and they report the status in their favorite testing tool as either pass, fail, or inconclusive. And now they test! For some, it’s a pass, others a fail. Pause the video where it says that the correct amount was 16. Oops, my test case had incorrect expected results. But do you have anything else to report? The gorilla? What gorilla?!? What about the curtains changing color, or the player leaving the field?
   
   When you drive, your speed is something you select. When you want to see all the flowers, maybe pass by more often, and go slow.
   
   I often do a collection of four party tricks I call “testing mathematics”, which is a set of three numbers that has nothing to do with the mathematical formula my old professors wanted me to be able to build for turning specs to test cases. Both this and 20 questions to show why we should not write all test cases when we search for the unknown are my absolute go-to, all the time. And both came to me from meeting the esteemed Cem Kaner, now retired.

3. Gilded Rose by Emily Bache
   https://github.com/emilybache/GildedRose-Refactoring-Kata
   
   It comes with requirements and code. And if it is changed as it needs to, it needs to be tested. It’s a brilliant illustration of specification vs. “works as implemented” as a test oracle helping us decide what matters and its impacts on our time and efforts, but also my favorite party trick to go to when wanting to show manual testing that looks awfully much like automation.
   
   For a longer version of this party trick, I wrote an instructor’s manual.

4. E-Primer by Alan Richardson, Evil Tester
   http://exploratorytestingacademy.com/app/ 
   
   E-Primer is a minuscule implementation of a domain that no one knows (except those who take this exercise) and a target-rich yet surprisingly complicated “how would you test a text field” exercise. You can’t answer in platitudes when you have an actual application. What would you do? Testing is like being given a blank paper with information others may not know of but care, and for this particular one, I can tell I have written 38 things down.
   
   Sometimes my trick is to show all the rabbit holes people get stuck in. Sometimes it is to illustrate how S(tructure) F(unction) D(ata) P(latform) O(perations) T(ime) leads you to better chances at finding things. Sometimes it is to show that we should see the positive basics to recognize what does not work and is reasonable. Starting with special characters may be a common choice, but it is most certainly not a great approach on an application you don’t even understand.

5. No Vehicles in the Park by someone via Elizabeth Zagroba
   https://novehiclesinthepark.com/
   
   Testing is about boundaries, and boundaries are fuzzy. What makes a vehicle? If we can’t define this, why do we think we can have definite specifications and pretend that a fuzzy world can be put in definite terms when examples are a great way of creating a common understanding.
   
   This exercise is at its best when facilitated by Elizabeth, because unlike me, Elizabeth is funny. And brilliant.

Remember, these are party tricks. They help illustrate a perspective. You can run through them fast, or a little slower, but still faster than any of our real applications. They may illustrate what we do in testing, but they are not all we do in testing. While testing is a party, it extends further in delivering information collaborations that matter. Any proposals on how you keep the party going, fun, and focused? Anyone up for pairing on the two other categories, the stories we tell and the bits of wisdom we pass around? Together it’s more fun. Let us know.

Maaret Pyhäjärvi is Director of Testing Services at CGI Finland. Regardless of fancy titles, she is just a tester,  if anyone ever is just anything. To support her testing, she is a polyglot programmer, conference designer, and community facilitator. She has delivered 700 talks and trainings in 28 countries on the side of her hands-on job in changing testing from within, and made it to the top-100 ICT list in Finland for 6 consecutive years, as well as been awarded by two major testing communities: Agile Testing Days Most Influential Testing Professional and EuroSTAR Testing Excellence Award.

The Problem

One of the biggest pain points I’ve heard across the testing community is the difficulty in demonstrating the value that testing/quality roles bring to an organization – which can lead to burnout, frustration, and even layoffs (and that’s just from my own experience!). 

I’ve always known how much value our roles bring; we see that value in the work we do every day. But over my career, I’ve seen that value proposition disappear by the time it hits leadership teams. While visibility is a complex problem that will look different across organizations, you can implement different strategies to improve the visibility of work in the organization. 

What is the “Quality Narrative”? 

The Quality narrative is taking all the pieces of work you do as a part of your role/organization and packaging them in a way that tells a story of the value of your work. As people, storytelling can be a valuable way to show impact that may not be completely quantifiable, like much of the work our role does. We can quantify things like code coverage, bugs reported, errors traced, etc. But there is quite a bit of work we do (or can do) that is not quantifiable. For example, things like bug triage/management, documentation, process improvements, etc., are not trackable by sheer numbers. But we can use those activities, in combination with quantitative data, to tell a story of how we create impact across product quality, processes, & overall engineering enablement. 

Sounds great, but what would this look like in practice? 

Example Scenario

Testing roles can vary pretty drastically, company to company and person to person. It's one of the things I love about the industry — but the nebulous role descriptions often lead to misunderstanding of the value we can bring. 

I’ve been wrestling with this idea for a while, especially after I was laid off. It was a macro version of the micro-problem I had been facing in my job search — how do I tell my story that is compelling & encapsulates the value I bring to the market? 

I landed on a single sentence, highlighting the 3 main areas I can impact with my work — test architecture, process design & optimization, & product quality.

So let’s use an example based on the work I have done previously, and how we can develop a narrative off of those main pillars. 

Product Quality Pillar

Let's start with the easiest pillar, product quality. Obviously, exploratory/functional testing, writing test scripts, facilitating bug bashes, & filing bugs are all good examples. A great data point to track is the number of bugs found before the launch vs. those reported after the launch. But what about beyond those things? 

Do you write documentation on the product and/or features? 

Do you create test data? 

Do you help with the product requirements for new features? 

Do you manage the bug triage? 

All of these are part of product quality, but are often not considered because they are not “measurable numbers”. 

One of the solutions I implemented was building a main Quality Plan document for the large features I supported. In that document, I laid out the known risks & mitigation strategies (if applicable), linked the detailed user scenarios I wrote, high-level release criteria, rollout strategy, test automation/code quality improvement opportunities, bug fix opportunities, & embedded the bug triage list for testing the feature. I used this data to provide high-level summaries for executives as well as sharing the data in our project Slack channels. 

After we launched a feature, I developed what I called “post-launch feature analyses” where I would combine quantitative data (bug fixed vs. released, code coverage, monitors, etc.) with qualitative data (asked team members to give me feedback on how the SDLC went, challenges they faced, etc). I then laid out the top problems found & proposed optimization solutions at the short-term & long-term levels. This could also be converted to be part of a retro ceremony. 

This pillar is where other ad hoc work can be communicated, like: 

• creating & maintaining documentation → standardizing practices across engineering teams, improving team understanding, improving project outcomes

• creating internal educational content/trainings → driving team understanding & adoption of tool/processes, communicating feature changes & enhancing team readiness

• creating test data → enhanced team efficiency & enabled engineers to focus on writing code

Process Design & Optimization Pillar

As I’ve grown in my career, I’ve realized I often do a lot of work in process optimization. These can be big or small changes, but they create an impact regardless. 

It wasn’t until I designed the “post-launch feature analysis” that I realized how much impact I can, and was, making on the processes that made up the SDLC. Some of the issues that came up in the first project I utilized this process in were very simple optimizations. Things like: 

• we need to align on an expectation of a single source of truth that engineers will work from so they are not duplicating their work (design team had a different process than the product/engineering team) 

• we need to track open decisions & decision points visibly for the teams working on the feature

• core functionality should not drastically change once engineering is working on it (easier said than done)

Some of these issues came up in the SDLC, and I resolved them once the problem was identified. I included that work in a basic summary of the quality work done on that feature as part of that analysis. 

There was process optimization work I was doing outside of the SDLC as well. This included:

• establishing & leading monthly observability meetings on the system I supported, sharing monthly trend summaries to highlight potential reliability & performance issues over time

• creating on-call runbooks–documents that show how to debug a system or solve specific problems–to improve incident response efficiency

• creating office hours meeting to improve collaboration & reduce rework

Technical Architecture 

I know testers of various levels of coding experience, or what a lot of people think when they think of technical architecture. While the tech stack is obviously one way to impact this area, it is not the only way. 

I consider myself a “dabbler” when it comes to coding. Almost everything I have learned has been in a work context — typically through solving a problem I stumble across. For example, writing test scripts to propose using a new tool or type of testing or investigating & solutioning testability issues. 

But there is an opportunity for those who don’t necessarily want to write code at any point. You could collaborate with the DevOps team to identify challenges with test environments & data, highlight code quality opportunities like removing “dead” code, and mapping dependencies to user experiences to facilitate data-driven decision making. 

Just remember – you are the expert in quality. That is the expertise you bring to each situation you are in!

Conclusion

The best-case scenario is this can be a top-down approach, where there is a Quality leader who can drive this narrative with the rest of the leadership, and the Quality team members drive the narrative at the individual contributor level. But regardless of the company hierarchy/structure, you can develop a quality narrative within your range of influence. 

First, develop your “pillars” or main areas of impact. Within each pillar, highlight the work you and your team are doing to create a positive impact. Share regular updates during and outside of the SDLC. For example, a monthly Slack update or email, giving a highlight of the value you or the quality organization delivered. 

The more you can highlight the main areas where you and your team make an impact, the better. 

Say it often, say it at every level. 

Document the work you do, even the qualitative work. 

Experiment with different ways to share & visualize the story until you find the way that communicates the impact you are driving.

Sidney Karoffa is a QA engineer & systems designer focused on engineering enablement. Passionate about UX, test innovation, & process optimization. Strong believer in democratizing technology & knowledge.

Learn more about Sidney and her work on LinkedIn.

Testing has evolved massively in the last decade.
We’ve embraced automation, moved toward continuous testing, adopted Agile, DevOps, and CI/CD. Tools are smarter, cycles are faster, and the role of QA is more visible than ever.

But despite all that, one thing hasn’t kept up:
👉 how we think about quality.
👉 How we embed it into the way we work.
👉 How leadership — across all functions — supports it.

This isn’t a tooling problem. It’s a mindset problem.

What QA Leadership Really Means (It’s Not Just Managing People)
Let’s be honest — QA leadership is still too often viewed as a checklist role:

• Manage a small team

• Review reports

• Track test case coverage

• Submit status updates

• Catch defects before releases

But that’s not leadership. That’s maintenance.

Real QA leadership is about enabling transformation.
It’s about helping the business see quality not as a gate to pass, but a culture to live.
It starts by asking simple but powerful questions:

• Is quality considered from the first business discussion, or only once code is written?

• Are testers helping shape product decisions, or just reacting to them?

• Are teams building quality in or still expecting QA to find the issues later?

You’ll Face Resistance — and That’s Normal
Here’s the truth no one tells you early on:
People naturally resist change. Even if it’s good for them. Especially when it feels uncomfortable or uncertain.

As a QA leader trying to shift the culture, you’ll face:

• Pushback from devs who don’t want to “slow down”

• Frustration from managers worried about timelines

• Reluctance from testers who are used to “just executing”

• Concerns from stakeholders about adding resources or effort

Sometimes you’ll even doubt yourself.

Bottlenecks Are Signals, Not Stop Signs
Trying to bring change — especially in how quality is integrated — can feel like swimming upstream. There are deliverables to meet, sprints to close, bugs to chase. Trying to introduce new practices can seem like a disruption.

But here’s what I’ve learned:

👉 Those early hiccups are valuable. They expose gaps, challenge assumptions, and start conversations.
👉 Every bottleneck is a signal — that something is misaligned, or missing. That’s where transformation begins.

Yes, new approaches may slow things temporarily. But they often uncover inefficiencies we’ve been ignoring for months — even years. When you fix those: 

🚀 You improve velocity in the long run
🚀 You reduce production issues
🚀 You earn more trust — from customers and leadership

Investing in Quality Is Investing in Long-Term Impact

Introducing new practices, involving QA earlier, or dedicating time to upskilling your team may seem like added effort — or even added cost — at first. It’s natural for stakeholders to question the return.

But here’s the reality: These aren't just costs — they're investments in building trust, reducing long-term risk, and delivering better user experiences.

The impact?

🚀 More efficient delivery
🚀 Greater customer confidence
🚀 A product that reflects real care — and earns long-term loyalty

Quality done right doesn’t slow you down. It sets you up to scale with stability — and lead with credibility.

Start Small. Start Smart. Build Belief.

You don’t have to launch a company-wide initiative. In fact, you probably shouldn’t.

Start with one team. One feature. One small process shift.

💬 Involve QA earlier in planning.
💬 Help testers ask “why,” not just “how.”
💬 Align with developers on shared goals, not separate phases.
💬 Set the expectation that quality is everyone’s job.

As these small changes succeed, people notice. Confidence builds.
And slowly, what felt like resistance starts to turn into adoption.

Changing Culture Doesn’t Require Permission — Just Consistency

You might not always have the formal authority to mandate change.
But culture is built from the ground up — and it spreads by example.
If your team starts living quality as a mindset:

• Collaborating early

• Asking better questions

• Focusing on prevention, not just detection

...other teams will follow. Curiosity is contagious. So is confidence.

And once results show up — better quality, fewer bugs, faster delivery — leadership will see the value. You won’t need to convince them with slides; the impact will speak for itself.

The Human Side of Change

And finally — never forget the people behind the processes.

Not everyone is reluctant because they’re lazy or stubborn. Sometimes, they’re afraid. They’ve seen change fail. They’re unsure of their role in it. They don’t know what’s expected of them.

👉 Instead of seeing resistance as a threat, see it as a chance to connect.
👉 Ask what’s holding them back.
👉 Show how the change benefits them, not just the business.

When people feel heard, they’re more willing to trust.
And when they trust, they engage. That’s when transformation really begins.

Final Thought: Lead Through Mindset, Not Mandates

Forget shifting left. Forget shifting right.
Let’s shift minds.

The future of testing isn’t about running more tests — it’s about building systems and cultures where quality becomes natural, expected, and shared.

As a QA leader, your role isn’t just to “make sure things don’t break.”
It’s to help your organization build better — together.

Be the one who starts the shift.
From control to collaboration.
From resistance to resilience.
From gatekeeping to growth.

About the Author

Saloni Shrivastava has worked across organizations of all sizes — from startups to large enterprises — and with diverse teams navigating fast-paced, ever-changing environments. She thrives on seeing meaningful transformation, whether it's in the mindset of a team, the direction of a project, or the culture of a company.

A budding writer and aspiring public speaker, she is passionate about sharing her experiences to inspire others and spark conversations around the everyday role of quality in building better products, better teams, and a better world.

You’ve probably heard it in a meeting by now.

“We need to do something with AI.” “AI is the future, how do we capitalize on it?”

It sounds visionary until you realize no one in the room can explain what “something” or “the future” actually means. That’s the problem. Somewhere between flashy conference decks and trend-chasing roadmaps, AI became the macguffin instead of a tool to reach a defined goal, which is like handing someone a hammer and expecting a house to appear.

AI is not a vision. It is not a mission statement. It is not a leadership substitute. It is not a shortcut to product-market fit. It can do wonderful things like automate repetitive tasks, generate content, and find patterns faster than any human ever could. But AI doesn't know your customers, your bottlenecks, or your business objectives. It will not fix dysfunction. It simply accelerates whatever you already have in place, whether it is working or not.

In software teams, especially, AI can be genuinely transformative. It can help generate test cases from user stories before code is written. It can create smart test data, expose edge cases quickly, and assist in exploratory testing by surfacing anomalies in logs or user behavior. It can support testers, QAs, QEs, and automation engineers in elevating their work, especially for those who never thought they would have a voice. It can give people access to insights and support in ways they never had before. Roles that were often reactive now have the opportunity to operate with foresight. But none of that matters without a plan. Without a clear reason for using AI and a structure for how to implement it, the tech becomes noise.

So let’s cut through the noise. Let’s break down what AI is exactly, in its current form.

AI is:

• A pattern finder

• A task automator

• A very confident content generator that sometimes… makes things up

AI is not:

• A business goal alone

• A replacement for leadership or a roadmap

• A magic wand to reduce costs

Last year, I did a keynote talk, “Failing Forward”, at mabl Experience 2024. I spoke about seeing AI as the end-all be-all for the perfect team setup. Implementing AI successfully will be trial and error. During this trial and error, it may not be productive for the team, may cost much more than anticipated, and there will be MANY failures. The failures ENHANCE the progress towards adoption. They become learning lessons, but the real failure comes when an organization refuses to reflect, course-correct, and build with purpose. 

This is backed up by the recent Orgvue survey, which found that more than half of the business leaders admitted later they may have made the wrong decision in laying off employees in lieu of AI adoption. The survey stated that 1 out of 4 executives who have shed workers to implement AI, 55% later regretted this.  IBM, among others, is an organization that aggressively cut staff under the assumption that AI would be this miracle for their HR department. It was soon realized that AI could not fill gaps that required subjectivity or empathy, and eventually rehired many workers to deal with this.

If your AI plan begins and ends with "we should be doing something with AI", with no use case, no problem statement, or a measurable goal, you are not building a strategy. You are reacting to panic about being left behind. I am not here to beat up on AI or its use in technology. My issue is not the technology, but the absence of intent behind it.

Implementing any tool, process, or culture is where real leadership becomes essential. The right question is not what AI tool to use. The right question is, what problem are we solving? If the goal is clarity, speed, or reducing manual tasks, that should drive the decision. Map the need. Understand the pain points. Involve the people doing the work. When you begin with strategy, you end with impact.

I want leaders to understand how to use AI with purpose. Not because it is trendy, but because it supports something meaningful. No one is measuring by how early you adopt technology. You will be measured by whether you use it intentionally.

Three Takeaways

• AI is only as useful as the strategy behind it. Without direction, it becomes noise.

• You cannot fix unclear goals or broken processes by adding more technology.

• AI can be transformative if used in the right way.

About the Author

Jessica Mosley is an award-winning Quality Engineering leader, speaker, and certified breaker of silos. With over two decades of experience across startups, mid-sized companies, and international enterprises, she has built high-performing teams that don't just test software, they elevate it. Jessica is known for championing quality as a strategic asset, blending human-centered leadership with automation, data, and the occasional reality check.

She has been called a “force of nature” in tech, but she’ll settle for “the one who says what everyone else is thinking”.

You can connect with her here

As a Tester, if you were given a stage and a microphone, what would you say?

What are your struggles, solutions, thought processes, practices, and insights?

As Testers, it’s possible that so much is on our minds that we aren’t sure if what we’re thinking about is worthwhile sharing.

Imposter syndrome shakes us.

We let inexperience hold us back.

Going out on our own feels tricky.

And, sometimes, just starting feels too overwhelming.

Welcome to the Women in Testing Newsletter.

Here you will find a variety of voices, with backgrounds of varied experiences and skill levels.

We are not here to amplify a particular practice. Here, we recognize that “Culture eats strategy for breakfast,” so we bring to the table what works for us. Here we will look at tools, strategies, practices, and guidelines that have proven valuable in Software Engineering. The writers represent the unrepresented in the Testing community. This platform exists to give them the stage, hand them the mic, and with a wink and a smile watch them glow.

If you are looking for the latest in testing, this is your next resource.

I welcome you to Women in Testing. Stay a while. Grab a good drink. Welcome to the community.

Women in Testing Newsletter: Springing into action, June 2025