Remember Waterfall? Now we call it a Sprint.
We still think we can plan it all upfront.
What's "correct" changes between iterations.
Also during. And AI doesn't help.
New video:
youtu.be/YInN9soA6aw
#APITesting #SoftwareTesting #AITesting #TestAutomation #QualityEngineering

The Hard Part of AI Evals Isn't the Tooling - Kato Coaching Additional thoughts on session three of the AI Evals and Analytics Playbook1. The first part is here2. Every major shift in how we build and ship software has been followed by a wave of tooling that automates the tractable part and leaves the actual problem to the practitioner. Agile gave us story pointing ceremonies and […]

Agile gave us JIRA. DevOps gave us pipelines. Neither answered the harder question. AI evals is doing the same thing: the tooling is excellent. The question underneath it remains. 

kato-coaching.com/the-hard-part-of-ai-eval...

#AITesting #QualityEngineering #SoftwareTesting

Remember Waterfall? Now we call it a Sprint.
We still think we can plan it all upfront.
What's "correct" changes between iterations.
Also during. And AI doesn't help.
New video:
youtu.be/YInN9soA6aw
#APITesting #SoftwareTesting #AITesting #TestAutomation #QualityEngineering

"Yes, and don't ask again for this session."
Code created. Tests pass. But nobody knows what they do.
That's the Knowledge Void.
Learn how to avoid it:
testingil.com/2026/03/the-...
#APITesting #SoftwareTesting #AITesting #QualityEngineering #TestAutomation

Knowledge Void: a gap between how your system works and how you think it does.
Bad news: you already got one.
AI accelerates delivery, but understanding still at human speed.
In 5 years, who will know how the system really works?
I can help.
testingil.com/contact
#AITesting #QualityEngineering

It's almost time! Hard to believe that #STAREAST 2026 starts in five short weeks. I'm putting finishing touches on my presentation and looking forward! Give me a shout if you're attending. It's always good to meet people in person. #softwaretesting #softwarequality #aitesting

well.tc/qosv

Black box testing sounds pure.
"No bias." "Scientific."
Ignorance doesn't make you a better tester.
It makes you an expensive one.
And if the code does multiple LLM calls, retries, recursive loops.
You're adding to the bill.
New video:
youtu.be/1MP-w9iXyJc
#APITesting #SoftwareTesting #AITesting

Best LLM Eval Tools in 2026: 6 Options Tested A data-driven comparison of DeepEval, Braintrust, Langfuse, LangSmith, Inspect AI, and RAGAS - the top LLM evaluation frameworks for teams building AI in production.

Best LLM Eval Tools in 2026: 6 Options Tested

awesomeagents.ai/tools/best-llm-eval-tool...

#LlmEvaluation #AiTesting #Deepeval

The anatomy of a metric - Kato Coaching Session two closed with a question I couldn’t answer.1 When a product scores 78% on a given metric, what tells you whether that’s good enough to ship? I flagged it as something session three would probably address, and it did, though not in the way I expected. The question can’t be meaningfully answered until you’ve […]

The AI evals field calls it a rubric. If you've done serious testing work, you probably know it by a different name. 
https://kato-coaching.com/the-anatomy-of-a-metric/

#SoftwareTesting #AITesting

Black box testing sounds pure.
"No bias." "Scientific."
Ignorance doesn't make you a better tester.
It makes you an expensive one.
And if the code does multiple LLM calls, retries, recursive loops.
You're adding to the bill.
New video:
youtu.be/1MP-w9iXyJc
#APITesting #SoftwareTesting #AITesting

Generative AI is revolutionizing test creation.

It reads code, generates test cases, improves coverage, and adapts automatically making QA faster and smarter.

Read: https://tinyurl.com/mvt62am7

#GenerativeAI #AITesting #SoftwareQA #Automation #QANinjas

You found the expensive tests. Great.
Then the team swaps the model.
Adds a retry. Changes the prompt.
The tests didn't change. The price tag did.
Test budget is a moving target.
More on April 15th.
us02web.zoom.us/webinar/regi...
#APITesting #SoftwareTesting #AITesting

AI-driven automation testing is now the standard.

It generates tests, predicts defects, adapts to changes, and integrates with CI/CD enabling faster, smarter releases.

Read: https://tinyurl.com/439e3kfb

#AITesting #AutomationTesting #SoftwareQA #DevOps #QANinjas

Are your chatbots ready for real conversations?

Without testing, chatbots can:
• Misunderstand user intent
• Give inaccurate responses
• Break conversation flow

Testing powers reliable AI interactions.

Read more: bitl.to/5m2Z

#AI #Chatbots #ConversationalAI #AITesting

Test runs used to be free.
Fail? Run again. Debug? Run again.
Not anymore. If your tests touch AI features, every run costs tokens. Real money.
Flaky test? You're paying for that too.
Webinar Apr-15.
us02web.zoom.us/webinar/regi...
#APITesting #SoftwareTesting #AITesting

#AITesting #AIQualityAssurance #AIinTesting #FutureOfTesting #GenerativeAI #AIInnovation #SoftwareTesting #AutomationTesting #AIForQA #NextGenTesting

Your tests are green.
But can you trust that green?
The Green Mirage isn't new. AI just makes it cheaper and faster to build more of it.
New post — real code, real example. One question that changes how you write tests.
testingil.com/2026/03/your...
#APITesting #SoftwareTesting #AITesting

API testing was already complex.
AI didn't simplify it.
It multiplied the permutations.
April 15th we'll see it at work.
us02web.zoom.us/webinar/regi...
#APITesting #SoftwareTesting #AITesting #TestAutomation #QualityEngineering

Requirements area communication tool. The anchor everything derives from.
In the AI era if we're not updating your AC as fast as you're learning, everybody loses direction.
That's the Intent Gap. Apr-15 we see it in API testing.
us02web.zoom.us/webinar/regi...
#APITesting #AITesting #TestAutomation

AI amplifies everything. That includes the blind spots. And boy, do we need sensors, because, the monsters are waiting in those blind spots.
I've talked about 9 of them in my webinar.
Watch it:
youtu.be/tXv8JtJhvDs
#AITesting #ResponsibleAI #SoftwareEngineering #GenAI

Running AI tests used to be free. Now, a flaky suite is a line item on R&D's invoice. Do you know how much your tests cost?
AI is cool, but it comes with blind spots.
You need to map them.
Use my AI Quality Assessment.
testingil.com/ai-quality-kit
#AITesting #SoftwareEngineering #Debugging #LLMs

📣 Today’s the day! Join the first QF-Test webinar of the year.

On March 2, 2026, we explore how Artificial Intelligence creates real value in business software testing – reliable, targeted, efficient.

⏰ Last chance to register:
www.qftest.com/en/support/t...

#qftest #AItesting #QA #webinar

AI amplifies everything. That includes the blind spots. And boy, do we need sensors, because, the monsters are waiting in those blind spots.
I've talked about 9 of them in my webinar.
Watch it:
youtu.be/tXv8JtJhvDs
#AITesting #ResponsibleAI #SoftwareEngineering #GenAI

Ran my workshop "Deciding Fast" on Tuesday with a software team in Sweden. Everyone in one room sharing computers, no breakouts. It's built for remote. I adapted. 6/8 Good or Excellent. Best response to "what will you do differently?": "Set clearer success condition."

#SoftwareTesting #AITesting

Running AI tests used to be free. Now, a flaky suite is a line item on R&D's invoice. Do you know how much your tests cost?
AI is cool, but it comes with blind spots.
You need to map them.
Use my AI Quality Assessment.
testingil.com/ai-quality-kit
#AITesting #SoftwareEngineering #Debugging #LLMs

Anthropic Education Report: The AI Fluency Index Anthropic's AI Fluency Index measures 11 observable behaviors across thousands of Claude.ai conversations to understand how people develop AI collaboration skills.

"Tell me what you're uncertain about." "Push back if my assumptions are wrong." Only 30% of people give instructions like these. The model defaults to confident and agreeable. 

https://www.anthropic.com/research/AI-fluency-index

#SoftwareTesting #AITesting #AILiteracy

How to Reduce AI Testing Risk in Dynamics 365 Using Copilot Studio - Microsoft Dynamics 365 CRM Tips and Tricks In the previous blog (Part I), you explored the overview of AI testing evaluation; now, you can dive deeper into the detailed functionality and practical implementation. If you work in IT, especially in a global organization, you understand how critical consistency and accuracy are in daily operations. As a Dynamics 365 CRM Engineer, you likely

How to Reduce AI Testing Risk in Dynamics 365 Using Copilot Studio

www.inogic.com/blog/2026/02...

#CopilotStudio #PowerPlatform #AITesting

With AI, a failed test investigation is harder. Complexity is bigger, and our problems are inside haystacks. Which in turn, are in their own haystacks.
Watch the crime scene investigation here:
youtu.be/r9DvZWHFLUU
#AITesting #SoftwareEngineering #Debugging #LLMs

Anthropic Education Report: The AI Fluency Index Anthropic's AI Fluency Index measures 11 observable behaviors across thousands of Claude.ai conversations to understand how people develop AI collaboration skills.

The strongest predictor of AI fluency, per Anthropic's research: iteration. Treating the first response as a draft, not an answer. 5.6x more likely to question reasoning. Familiar territory if you work in testing. 
https://www.anthropic.com/research/AI-fluency-index

#SoftwareTesting #AITesting

#AIQuality #AITesting #ModelValidation #DataPipelineTesting #BiasDetection #DriftMonitoring #ScalableAI #StressTesting #QAAutomation #ResponsibleAI