But in software testing, just like in every other corner of tech, the problem isn't in using bad tools, rather, it's in using the right tools for the wrong reasons.

The allure of novelty is strong. Most engineers, when asked what they’re looking for in their next role, will say they want to learn something new. It’s a fair desire—tech careers reward the ability to grow. But when “learning something new” turns into chasing every tool that promises faster or smarter testing, it can derail the very quality efforts those tools are supposed to support.

As consultants we’ve seen this play out repeatedly. A team adopts a trendy new testing framework, integrates it into a few services, and by the time the champion of that rollout moves on, the rest of the org is stuck maintaining something they never fully bought into. What’s left is fractured tooling, inconsistent practices, and a test strategy that’s harder to maintain than it was before.

The long-term cost of short-term thinking

Shiny Object Syndrome—the constant urge to chase whatever’s new—affects QA teams in subtle but damaging ways. The testing world has never had more tools available, and the pressure to “keep up” is real. Every week there’s a blog post about the next framework or the next agentic testing layer that promises to change everything.

But most of the work that matters in testing isn’t flashy. Without replacing tools every quarter, we can build confidence in the existing test suite, improving coverage slowly and steadily, and choosing the technologies that will hold up over time—not just the ones that get applause today.

Even the best new tools often come with tradeoffs. They might require retraining. They might integrate poorly with your existing CI/CD system. They might only be supported by a small team. And they often attract the kind of developers who want to be constantly trying something new, rather than sticking around to maintain what’s already working.

Innovation is important. But without a clear reason for adopting a new testing tool—and without someone responsible for long-term ownership—novelty quickly turns into noise.

Amish engineering

The Amish aren’t anti-technology, but they are highly intentional about how they adopt it. Before bringing something new into the community, they ask how it will affect their values, habits, and relationships. They’re not afraid to say no to tools that look useful in isolation but misalign with long-term goals.

Software teams could learn something from that.

There’s nothing wrong with adopting modern testing frameworks or experimenting with AI-based tools. Many of them offer real benefits: more speed, less maintenance, better signal. But only if they’re chosen carefully, integrated thoughtfully, and maintained consistently. Too often, teams are drawn to what looks good on a résumé rather than what serves the product.

Hiring pressures can make this worse. If every candidate seems to be using one specific tool, teams might feel compelled to adopt it just to remain attractive. But that creates a feedback loop where tool choices are driven more by perception than need.

Test tooling should support your strategy—not define it.

Boring is better

The best QA teams I’ve worked with are rarely the most exciting. Their tests run reliably. Their coverage improves quietly. Their developers trust the test suite enough to release confidently. They don’t chase every new library, but they do keep an eye on what’s coming next. And when they make a change, it’s because it supports the work—not because it makes for a shinier tech stack.

That kind of maturity doesn’t show up in a tools list. It shows up in production uptime, bug reports that never get filed, and features that ship without breaking anything else.

Testing is all about being dependable. If that makes your stack a little “boring,” maybe that’s exactly the point.

Using MCP and AI many test engineers have simplified their test code base, to an extent where the testing code is generated from plain English instructions. The approach is getting more and more popular, which is why its vulnerabilities deserve attention.

A major security vulnerability (CVSS 9.6/10) was disclosed in July 2025 in the Model Context Protocol (MCP) ecosystem—specifically within the wildly popular mcp‑remote component. Researchers found that any client reaching out to a malicious or hijacked MCP server can end up executing arbitrary code on its own machine. That’s full remote code execution at the protocol layer, before your application ever sees a byte of data. If you’re gluing your tooling together with MCP, consider this your five‑alarm warning.

In this case, the flaw lies in how mcp‑remote handles the authorization handshake. A compromised server can slip malicious commands straight into the protocol payload, and because that payload is treated as executable instructions, the client dutifully runs whatever it finds. No user prompt, no sandbox prompt—just code, straight from the wire.

How does this work?

During the authorization phase, the client and server exchange a series of JSON messages wrapped in TLS. What the client believes to be innocuous configuration or orchestration data may actually contain shell commands, PowerShell scripts, or binary payloads. On Windows hosts, these commands execute with full OS‑level privileges; on macOS and Linux, they spawn executables that can manipulate files, exfiltrate data, or even load kernel modules if misconfigurations allow.

This isn’t a fanciful “what‑if” scenario. There have already been reports of man‑in‑the‑middle setups—where attackers intercept MCP traffic between CI workers and orchestration servers—and typosquatted endpoints that mimic legitimate MCP servers in internal documentation or Slack channels. In both cases, the exploit requires nothing more exotic than a valid TLS certificate and a willing miscreant.

Perhaps most troubling is that no prompt‑injection wizardry or AI‑specific hack is required. The attack bypasses higher‑level protections by striking at the protocol glue itself. In a world where we trust underlying protocols implicitly, this reminds us how brittle that trust can be.

Why does it matter?

That aphorism has never felt more apt. In today’s AI‑driven testing landscape—where frameworks like Playwright MCP let models write and execute browser tests for you—a single untrusted endpoint can hand over the keys to your CI runners, developer laptops, or even your entire supply chain. Suddenly, your test infrastructure isn’t just a guardian of quality; it’s a potential vector for the very worst kinds of supply‑chain or production‑environment breaches.

Imagine your continuous integration pipeline mysteriously failing tests that never should, or worse, passing tests that should fail. Attackers could falsify results, conceal critical failures, or transparently exfiltrate secrets embedded in environment variables. The fallout isn’t just lost productivity; it’s potential regulatory fines, reputational damage, and a long road to recovery in repair and remediation.

Securing Your MCP Connections

First, treat every MCP endpoint like a stranger at your door. Don’t accept URLs from casual Slack channels or Discord threads; only point your clients at thoroughly vetted, organization‑owned servers. Second, upgrade mcp‑remote to version 0.1.16 or later, where the RCE hole has been patched.

Next, enforce strict TLS‑only connections and apply certificate pinning where possible. Sandbox your test runners with least‑privilege containers or VMs—imagine every MCP‑driven process navigating a sea of hot lava, with only the narrowest bridges in place. Finally, bake automated scans for MCP misconfigurations into your CI pipeline: tools like SecureMCP (open‑source) can flag dangerous setups before they ever reach production.

In the dazzling world of AI‑powered testing, the most dangerous bug is often the one you never thought to test for—the protocol glue binding your clever bots together. Stay curious, stay paranoid, and remember: AI can write your tests, but it still needs a human to lock the doors.

Further reading: read JFrog's original report Critical RCE Vulnerability in mcp-remote: CVE-2025-6514 Threatens LLM Clients

At that point, definitely nobody wants to run it on every commit anymore. The once awesome suite turns into a slow, pre-release gatekeeper instead of a daily safety net — and the whole point of catching issues early starts to fade away.

Michael Feathers, in Working Effectively with Legacy Code, says: “to a large extent, legacy code is simply code without tests.” And that’s entirely true, but notice that when we talk about “testing” we often use differing definitions.

Unit tests run fast because they test small pieces of code directly. End-to-end tests act more like real users: they wait for pages to load, fill out forms, click around on the site. By nature they take longer, and as the suite grows they can also get flakier — sometimes a click doesn’t take, sometimes the network lags just enough to break the test.

When the feedback loop stretches from minutes to hours, developer habits shift. People start batching bigger commits to avoid waiting. They skip running the suite locally because nobody wants to lose half a morning if they don’t absolutely have to. Testing ends up happening mostly in CI, and even then it feels like crossing your fingers and hoping nothing breaks in areas you didn’t touch.

What the big players do

If you're drowning in a two-hour test suite, you're definitely not alone. Good news is that the larger players hit this wall years ago, and invented best practices around it.

Google’s approach is basically to throw hardware at the problem. They built a massive parallel execution grid that can spin up thousands of test runners at once, so the whole suite finishes before lunch. For the rest of us who don’t have Google’s data centers, this idea still scales down: you can parallelize your tests using cloud services like Sauce Labs or BrowserStack, or spin up your own fleet of test runners on Kubernetes. The principle stays the same: if one runner takes two hours, then twenty runners might get it done in six minutes.

Microsoft's Azure DevOps took a different approach with Test Impact Analysis. Instead of running everything faster, they got smarter about what to run in the first place. Their tools look at the code changes in a commit and work out which tests are actually relevant. Change a CSS file? Maybe you can skip the backend API tests. Update a database migration? Better run the integration tests but skip the UI ones.

The immediate wins here are pretty practical. First, parallelize what you can. Second, organize your tests by what they really check because not every test needs to run every time. Your quick smoke tests can run on every push. The full regression suite might run nightly. Performance tests run on weekends or before big releases.

The real low hanging fruit is to stop running tests that don’t add real value. It’s surprisingly common to see thousands of tests, half of which cover the same happy paths in slightly different ways. It’s like having six smoke detectors in a studio flat — more doesn’t always mean safer. Go through your suite, find the duplicates, and focus on the tests that actually catch real bugs, not the ones that just pad your coverage report.

Testing without the Google budget

Now, if you’re reading this from a startup where “massive parallel execution grid” sounds like something out of science fiction — don’t panic. You don’t need Google’s infrastructure to catch bugs before your users see them. What matters more is being smart about where you spend your testing effort.

Martin Fowler’s test pyramid helps here. Lots of unit tests at the bottom: they’re fast, reliable, and catch the obvious stuff. A smaller number of integration tests in the middle: these catch the “my component works, but doesn’t play nicely with others” problems. And just a handful of end-to-end tests at the top: the critical user journeys that simply cannot break. It’s a bit like home security: you don’t need cameras in every room, but you definitely want them watching the front door.

Visual regression testing is where small teams can really punch above their weight. Tools like BackstopJS — or even simple screenshot comparison scripts — can catch unexpected UI changes across your whole app in minutes.

Don’t underestimate a good manual testing checklist. Sure, it’s not automated, and yes, it doesn’t scale forever. But if you’ve got three critical user flows, spending 15 minutes running through them before each release can be more useful than weeks spent building an automated suite that breaks every time you rename a CSS class. The trick is knowing when to switch: usually when you find yourself repeating the same manual checks every few days.

The real lesson here is to match your testing strategy to your product and your team. Catch the bugs that matter — and don’t let “perfect” stop you from shipping.

Automated tests are brilliant, and AI is making them infinitely better—that much we know, TestTheTest being an autonomous testing company and all. Automated tests are great at executing vast numbers of scripted checks: smoke tests, regression sweeps, repetitive workflows. But they can’t pause and ask “What if…?” They miss the odd edge case, the subtle UI quirk, the confusing error message.

Human testers bring unpredictable creativity and deep context. You spot the misaligned tooltip on a rare device, the barely noticeable lag when toggling settings, or the form validation that reads oddly in real-world use. Those “wait, what if…” moments uncover the bugs no script could predict.

The real power comes when you stop treating automation as competition. Let AI‑powered tools handle the heavy lifting—running your Playwright MCP suite at scale—so you can focus on exploratory testing and complex scenarios. Your manual efforts feed insights back into your automated library; your growing automation speeds up your exploratory loops.

Hybrid testers—adept at both coding automated scripts and conducting unscripted investigations—are in highest demand. Data from automated runs informs exploratory testing; your exploratory findings refine your scripts. One plus one equals far more than two.

Testing at machine speed

Automated testing uses scripts and tools to run predefined checks without human intervention, forming the backbone of modern CI/CD pipelines that catch regressions on every commit.

A solid automation suite can execute thousands of tests overnight—work that would take humans weeks—enabling rapid release cycles and confidence in core functionality.

Automation provides a reliable safety net for fast-paced development teams. As Google’s James Whittaker noted in “How Google Tests Software,” automation provides a repeatable, reliable, and efficient way to verify that code changes don’t break existing functionality.

These tests also uncover defects invisible to humans: performance bottlenecks, memory leaks, race conditions—issues that only emerge under precise, repeated conditions. In one case, our load‑testing scripts revealed a database index flaw that would’ve collapsed under real traffic.

The cost-effectiveness becomes increasingly apparent over time. After the initial investment, each run costs virtually nothing, letting teams shift effort from rote verification to high‑value exploratory testing.

But automation has blind spots. It only checks what you’ve scripted, missing undefined behaviors, shifting requirements, and UX or cultural nuances. Maintenance overhead—broken scripts after a UI tweak—and upfront tooling costs can eat into its benefits. As Michael Bolton says, “test automation isn’t testing”—it enables checks, but true testing still demands human insight and adaptability.

Detecting the undetectable

There’s no software without bugs — maybe there’s software without bugs that haven’t yet been discovered.

Or, as Alan J. Perlis, computer scientist and the first recipient of the Turing award says: “There are two ways to write error‑free programs; only the third one works.”

An expert tester brings tacit knowledge to every session—asking not just “Does this work?” but “Does this feel right?” and “Does this actually solve a user’s problem?” Automated checks can’t mimic that instinct.

“The pesticide paradox,” as described by testing expert James Bach, drives the point home: every test method eventually stops finding new bugs. Human testers continuously evolve—crafting fresh approaches whenever old ones go stale—and that adaptability is crucial as products mature.

Exploratory testing—design, execution, and learning in one seamless flow—is manual QA’s superpower. Skilled testers follow hunches, probe unexpected paths, and adapt on the fly. Time and again, I’ve seen exploratory sessions unearth critical flaws long after automation gave up the hunt.

Manual testing also shines at evaluating usability, accessibility, and overall user delight. A tester instantly senses when a UI feels clumsy or an interaction frustrates—insights no script can deliver.

But let’s be real: manual testing alone can’t keep pace with modern release cadences. It’s slower, harder to scale, and prone to fatigue and inconsistency. That’s why the magic happens when you pair human insight with machine speed—letting each cover the gaps the other leaves behind.

The future of QA is hybrid

Hybrid testers—adept at both automated and manual techniques—are the most sought‑after professionals today. By letting scripts handle repetitive checks and giving humans room for exploratory investigation, you create a multiplier effect: automation surfaces patterns and regressions, while exploratory testing uncovers the edge cases no script anticipates.

Industry data backs this up. In the 2024 PractiTest State of Testing™ Report, 92% of organizations follow Agile‑style practices, and 45% have adopted DevOps workflows that blend automated and exploratory tests. After embracing these hybrid models, 68% of teams saw fewer severe bugs in production, and many saved 40+ testing hours per month per user by empowering manual testers with no‑code automation tools.

Market demand mirrors this shift. The global software testing market hit $55.6 billion in 2024, driven in part by a shortage of professionals who can both script automated suites and lead unscripted investigations. Teams that combine these skills release features faster—70% report quicker feature rollouts—and deliver higher overall quality, with 73% noting improved testing coverage.

As applications grow more complex, the need for testers who bridge scripting and exploratory expertise will only intensify. By cultivating both skill sets, you position yourself as an indispensable quality strategist—someone who drives efficiency and uncovers the critical insights that keep software reliable and user‑focused.

What started as purely manual work in the 1970s has evolved through automation in the following decades, and now entered a new phase: autonomous testing. It’s fundamentally changing how teams approach quality assurance.

Manual testing meant checking every feature and function by hand. Automation improved on that by letting us write scripts that computers could run repeatedly. Now, autonomous testing goes a step further—using AI to decide what to test and how to test it, with minimal human involvement.

I like to use the analogy of tending a garden. Manual testing was like planting each seed and watering it by hand—labor-intensive, but precise. Automated testing was like setting up an irrigation system: much faster, but still needing someone to install and manage it. Autonomous testing is like a smart garden—one that waters and fertilizes on its own, monitors weather conditions, adjusts care based on plant needs, and optimizes growth, all without human input.

Testing from the future

Autonomous testing tools stand out for three key capabilities that are reshaping how testing gets done.

First, they use AI to generate test cases automatically, covering more ground than a human tester could reasonably think of. This kind of intelligent test generation increases both speed and breadth.

Second, these systems can create self-healing tests that adapt to changes in code or interfaces. When developers change something in the app, the tests automatically adjust and keep running. This cuts down the maintenance burden that often comes with traditional test automation.

And third, autonomous testing enables predictive analysis—using past test results and code history to identify where problems are likely to occur, even before anything breaks. This shifts the process from reactive bug fixing to proactive quality assurance. Microsoft’s recent release of the Playwright MCP server is a good example—it gives testing agents more contextual understanding and will likely accelerate the rise of “agentic AI” in testing.

This new era promises better test coverage, faster feedback cycles, fewer human errors, and lower maintenance costs. Performance testing also improves, with the ability to simulate thousands of users at once. And maybe most interestingly, these systems learn and improve with every test run—the opposite of what developers experience with traditional test suites, which tend to fall out of sync with the application over time.

Tools of the trade

This field is moving so quickly that it’s probably best if I avoid naming specific tools—many won’t last long anyway, and I’d like this article to stay relevant a bit longer.

Still, it’s a good moment for decision-makers to be smart about avoiding vendor lock-in and picking tools that allow easy migration. Even open source isn’t a sure bet. Right now, most people use Playwright (increasingly with auto-playwright or Microsoft’s Playwright MCP server), but not long ago, automated web testing was mostly dominated by Cypress or Selenium.

In times of rapid change, it’s worth remembering Peter Drucker’s insight:

“The greatest danger in times of turbulence is not the turbulence – it is to act with yesterday’s logic.”

So let’s keep an open mind about how we work with these new tools, what they can do, and where they might fall short. We’re starting to see more focused tools emerge—ones that specialize in things like visual validation or API security testing—instead of trying to solve every problem under one roof.

When choosing a tool, weigh all the costs: not just the sticker price, but the time it takes to learn, integrate, and maintain all the custom code and knowledge. And always remember: the goal isn’t testing. The goal is quality software.

Looping humans back in

Despite the impressive capabilities of autonomous testing, the most interesting conversations happening now aren’t about replacing humans—they’re about finding the right partnership between AI systems and testing professionals.

The testing community has always included skeptics, and for good reason. We’ve been promised testing silver bullets before. The truth is that understanding what makes software valuable to humans still requires human judgment. Testing isn’t just about finding bugs—it’s about understanding whether the software delivers value in ways that matter to people.

What’s emerging is a model where AI handles the repetitive, pattern-based work while humans focus on exploratory testing, user experience evaluation, and ethical considerations. Humans are uniquely equipped to ask questions like “Should this feature exist at all?” or “How might this be misused?” These are questions that existing AI frameworks won’t raise on their own, regardless of how sophisticated its test coverage becomes.

The tools themselves reflect this hybrid approach. They don’t try to eliminate human involvement completely—they try to amplify human judgment by taking over the cognitive drudgery. A tester freed from writing and maintaining basic regression tests can instead think about edge cases, unintended consequences, and creative ways users might interact with the system.

As testing evolves, the most successful teams will likely be those who understand both the power and limitations of AI testing tools. They’ll know when to trust the autonomous system and when human creativity, intuition, and ethical reasoning need to take the lead. The future isn’t robots replacing testers—it’s testers whose capabilities are extended by robots, working together toward something neither could achieve alone.

In 2022 Toyota had to halt all operations at all 14 of its Japanese factories, because of a single software bug in their supplier management system.

The estimated cost: $14 million per day in lost production.

"Could be worse" you could say, and you would be right: A unit conversion error in NASA's Mars Climate Orbiter led to a $327 million mission failure. In a separate incident, Knight Capital Group lost $440 million in 45 minutes due to a trading algorithm bug. Elsewhere, a coding error in Cloudflare's content delivery network briefly took down thousands of websites in 2019, affecting services from Discord to Shopify and impacting millions of users worldwide.

In 2020 alone, poor-quality software reportedly cost businesses $2.08 trillion globally according to the Consortium for Information & Software Quality (CISQ). That's not a typo—trillions of dollars evaporated because someone somewhere wrote a bug that didn't get caught until too late.

And these all happened before we needed a word for "vibe coding".

With more and more software being written by AI — is there anything that development teams can do to prevent bugs, and avoid their financial sinkholes?

For more QA help, visit TestTheTest.com

The 1:10:100 Rule of Software Quality

Taking the "1:10:100 rule" from manufacturing: if fixing a problem during the product’s design phase costs $1, then fixing the same bug during production will be $10, and the costs rise to $100 after the final object was delivered to the customer.

Software engineers have observed similar rules: according to research from IBM's System Science Institute, addressing defects after product release can be up to 100 times more expensive than fixing them during the design phase.

When developers have to fix a bug in production, they're pulled away from whatever they're currently working on. This context switching is incredibly costly in terms of productivity and momentum. Then there's the diagnostic complexity—finding bugs in production requires reconstructing complex user scenarios, often with limited information. Add to that the operational impacts like outages, security breaches, and data integrity issues, and you begin to see why the costs skyrocket. And we haven't even mentioned the reputation damage that can occur when customers encounter bugs—trust, once lost, can be incredibly difficult to regain.

Hard numbers

Back in 1981, the software engineering pioneer Barry Boehm published a seminal study showing that defects become exponentially more expensive to fix as they move through the development cycle. According to his research, a bug costing $100 to fix during coding would cost $1,500 during system testing and up to $10,000 post-release.

While the specific numbers have evolved over time, the fundamental principle—now sometimes called "Boehm's Law"—remains as valid in today's cloud-native world as it was in the era of mainframes. The longer a bug lives in your codebase, the more expensive it becomes to get rid of.

This idea of working quickly to keep costs to a minimum isn't just intuitive—it's supported by extensive research. A study conducted by the National Institute of Standard Technology (NIST) estimated that it takes three times as long for developers to fix a bug in the production stage as it did during the coding stage.

Move fast and fix things

Agile and other modern software development methodologies, and the rise of modern devops tooling & CI/CD helps with catching bugs early, and therefore flattening the cost curve.

Continuous Integration creates shorter feedback loops, allowing teams to catch integration issues within hours instead of weeks. Automated testing increases coverage and consistency, helping teams find more bugs earlier in the process. Feature flags let teams isolate new code in production, limiting the impact of defects. And modern monitoring and observability tools help teams identify abnormal behavior before users even report problems.

These practices don't eliminate the cost gradient—but they do pull bug discovery earlier in the cycle when fixes are less expensive. According to the 2019 State of DevOps Report, teams using best practices experience 24 times fewer failures and recover from incidents 44 times faster than their less mature counterparts. That's a competitive advantage that’s hard to ignore.

The art of triage

The reality is that in any moderately complex software project, you'll probably discover more bugs than you can immediately fix. This is where prioritization becomes crucial. Not all bugs are created equal—some you can live with for months without significant impact, while others require the entire team to drop everything and address them immediately. How do you make these decisions in a systematic way?

Many successful businesses build their prioritization system around two fundamental elements. First, they consider the investment: How much will it cost to fix the bug today compared to fixing it tomorrow, or next week? Sometimes, delaying a fix makes sense from a resource perspective. Other times, the "interest rate" on technical debt is so high that immediate action is the only financially responsible choice.

The second element of prioritization: impact on the business. Does the bug affect the customer experience, or can users still accomplish their goals despite it? Is it affecting core functionality or peripheral features? Does it impact all users or just a small segment? Is it causing data integrity issues that might compound over time?

This prioritization approach can vary based on your company's stage and context. A startup racing toward product-market fit might tolerate non-critical bugs that an established enterprise with millions of users would immediately address. A healthcare application will have different priorities than a mobile game. The key is having a consistent framework that aligns your bug-fixing efforts with your broader business objectives.

Effectively prioritizing bugs isn't just about deciding what to fix first—it's also about explicitly deciding what can wait or perhaps never needs fixing at all. Because in software development, as in life, you can do anything, but you can't do everything.

Visit TestTheTest.com

Further reading

• Boehm, B. W., & Basili, V. R. (2001). Software defect reduction top 10 list. Computer, 34(1), 135-137. https://doi.org/10.1109/2.962984

• Capers Jones. (2008). Applied software measurement: Global analysis of productivity and quality (3rd ed.). McGraw-Hill Education.

• Consortium for Information & Software Quality. (2020). The cost of poor software quality in the US: A 2020 report. https://www.it-cisq.org/pdf/CPSQ-2020-report.pdf

The largest critical bug I personally saw at TestTheTest was when one of our clients broke their user signups. For a B2C startup this is as serious as issues can get: startups spend a huge amount of hard-earned (hard-invested) cash to get signups. So when potential users arrive at the site only to find that it doesn’t work — those people are not likely to ever come back. Money (not) well spent.

Moving fast and breaking things is just the way to go at tech startups, and so we can have an idea about where the balance will fall from the tech product manager’s fast, good and cheap Iron Triangle. In startup land, we’ll definitely see more unicorns than bug-free apps.

Every piece of software has bugs. They happen for many reasons, such as:

• Changes in requirements because features will come and go

• Changes in the ecosystem because Apple and Google will always introduce new devices, new browsers, and new everything

• Rescheduling resources because people in your team will leave at the exact wrong time

• Redoing work because we needed to rush it the last time and nobody understands how it works now

• Too much code because we didn’t have time to find a package or framework – or didn’t care to look

• It’s in someone else’s code because everyone, always, uses 3rd party packages, and we can’t test each and every update, all the time

This also means that regression testing, regular QA and allocating time for fixing bugs is a critical – and ongoing – process. And to make that process as efficient as possible, having a good idea of where bugs are most likely to occur in your app can be hugely beneficial.

For more QA help, visit TestTheTest.com

Bugs: They’re Everywhere!

Bugs can pop up in literally every feature, even the ones that people use often. And they can pop up anytime – Murphy’s law dictates that the bad code will go into production right after the marketing team has blown most of their budget on attracting new customers.

However, there’s one area of an app that testers often think is safe and will skip first when they run out of time – despite it being a hugely popular bug hideout. That place? The oldest, most trusted, established features.

There are two reasons for this:

1. Lack of visibility

Think about the most visible part of your app. It’s the new features, right? During active development, your app’s latest offerings will be seen by your developers, your testers, your project managers, and anyone else keen to check out what the latest features can do.There’s excellent oversight from all directions.

There’s practically zero chance that a bug is hiding out in your new features undetected. It’s much more likely to choose the quiet, cobweb-ridden corner of your old features; that place where no one ever thinks to look. Visibility matters when it comes to bugs.

2. Chain of Effect

In all software and always, functions depend on each other. When a developer works on a new feature, they often rely on functions previously written by themselves or by others. There isn’t always enough time to understand exactly how the old function’s internals work, let alone re-architect the code to be more reusable — so when someone changes the old functions for the new feature well, most of the time nothing bad happens. But sometimes, bugs start showing up.

Everything that developers do – adding a new feature, removing an existing feature, re-designing the UI interface, integrating modules, managing the database – it all has a ripple effect right through the app.

New work can hugely affect old work, either directly, or indirectly by increasing the complexity of the software and the system as a whole. And yet testers rarely take the time to go back and retest everything. They just don’t have the time, or simply don’t think to, because “that feature has always worked and we haven’t changed anything there.

Common bugs to look out for

The secret to keeping on top of things and finding bugs quickly is to get into the habit of not only testing regularly but testing everywhere. Some common things to test include:

• Connection and speed

• Backend functionality

• How the app handles interruptions (e.g. a phone call)

• Permission issues

• Page layouts and screen sizes

However, even the best QA tester can miss bugs in smoke tests. That’s why it’s considered best practice to create testing checklists that QAs can tick off every time they perform their regression tests. Exactly what will be tested will depend on the nature of your app, but don’t skip something, just because something is annoying to test and retest, over and over again.

Visit TestTheTest.com

Experienced painters say that the biggest difference between a beautifully painted wall and a botched job is in the preparation.

It certainly takes time to prep the wall, tape the edges and remove the sockets — but all that time investment pays dividends at the end of the job. Amateurs will spend long hours removing brush marks, mopping up drips and giving a second coat, while professionals have already moved on to the next job.

Sounds familiar? If you’re in software, it very well might.

The Intricacies of Software Development

OK, so building software isn’t quite the same as painting a wall. But the fundamentals are all there. You can’t just grab VSCode, copy-paste some code from StackOverFlow and call it a day as soon as the app starts to resemble the design specifications. Good software is the result of a lot of planning work.

You can spot a good paint job when you see it. High quality speaks for itself, and we can tell when a project was executed well from start to finish. It’s the same with software. It’s easy to see an amateur app: they crash, they do unexpected things, and there are telltale signs all over the user interface. Text that didn’t get translated, inconsistent colors and fonts and missing buttons are all signs that specifications got overlooked, and there wasn’t enough time left to fix them.

When you have a team of ‘slap the paint on developers, they’re technically doing what’s needed to build the product. But they may not be doing all the critical extras at are needed to ensure the product is secure, usable, and completed on time. If you miss these points the product will likely fail.

For more QA help, visit TestTheTest.com

The Mopping Up

We’re at 90% with the product, so there’s only 90% left — project managers used to joke about their progress. This is the time when most of the features are completed but the product wasn’t yet tested by the QA team nor customers.

The “mop-up phase is the most critical time in the development process, and this is where the team will learn the truth about product quality. Great developers with well architected products generally allow for quicker bug fixes — while others scramble and keep introducing new bugs every time they fix one.

All this difference can be explained by how the team got to the 90%. With painting, if a hole in the wall wasn’t fixed before the paint job, that hole is probably best to leave there until the next time the wall needs painting. And similarly, fixing a small bug in badly architected software might be the start of a huge refactor job — and suddenly the mop-up phase is turned back into a full-on programming sprint.

Bringing Wallpapers to a Fight

Agencies do not react well to sudden changes from clients, and they also like to arrive at the job with a solution they have already used at other clients’ projects.

They are the painters who arrive at the job with a set of wallpapers that they know how to use and slap on in a short amount of time. Agencies have to work within their constraints to stay within budget and be profitable — so managing these details and changes is the most important job for an agency-facing project manager.

Slap it on

It’s on the home stretch where complicated issues start to pop up. Where the QA team uncover weird bugs. Where the client suddenly decides a new feature needs to be added. The last few sprints suddenly turn into eight!

In both painting and software, the little things can be some of the most time consuming. It’s getting everything prepared and cleaned up that takes time, so it’s no wonder that these are the bits and pieces that often get brushed under the rug. In software, putting unit tests, or a component library in place, setting up CI/CD, a separate development environment is just as important as prepping the wall for painters.

Every team needs members who can slap the paint on, but managers need to ensure to have people onboard who are willing to put that 90% of time in to finish off the final 10% in the best possible way.

Visit TestTheTest.com

Within a small company usually everyone is responsible for more than one role. At tech startups, you’ll often see CTOs write code, backend software developers run DevOps and frontend devs design part of the user interface.

When it comes to quality assurance, we see everybody and their mother put in time to test the app, and try to catch bugs before pushing new code to users.

It may be fun to test products once, but QA generally requires people to re-test every single feature after changes are made to the app. This tedious work is not fun anymore and certainly nobody’s passion — and nobody’s responsibility.

People will simply skip re-testing features “that always work anyway. Even worse, developers being developers tend to try and automate their part of the job by writing browser and automation tests — only to end up spending a big part of their time maintaining those.

Building a QA team from scratch

Testing is rarely where startups add value, so product managers in particular are annoyed when feature development gets stalled, just to make space for something as lowly as QA. To free up dev time, the next logical step is to bring in people to address the issue.

The first question then is: who can they hire to get quality assurance sorted? QA engineers? Analysts? Testers? Where does QA fit in the team’s workflows and release schedule?

Your mileage may vary, but the QA team should be able to:

• Perform regular auditing and testing to improve product quality

• Analyze results, share them with the business and provide actionable reports

• Help developers see where their products can be improved

• Keep products compliant and help reduce security risks and vulnerabilities

• Create, document, share, and repeat strong testing processes

At a startup, it’s only fair to assume that roles overlap, and some of the above will still be covered by existing employees. And so once you have an idea about the role, the next question is, finally: how to hire someone?

For more QA help, visit TestTheTest.com

Here are 5 top tips for hiring your first QA Engineer:

1. Attract the right people

You’re more likely to make a quality decision if your shortlist is full of quality candidates. However, while you may have a good idea of the role and responsibilities of a QA, it can be difficult to dig deeper into areas such as priorities and challenges unless you come from a testing background. If you don’t, reach out to QA experts for support in writing job specs that attract and appeal to the right people. Keep an eye out for workshops or mastermind sessions where you can ask questions of experienced QAs.

2. Consider different opinions

Who’s going to help you make a decision? Your IT team? OK, but they might not know about testing inside and out. Your developers? OK, but patch and move on devs don’t always understand the nitty gritty of testing. Basically, ask IT and they’ll back an IT-oriented candidate. Ask a dev, they’ll back a dev-oriented candidate. You need different perspectives, so be sure to involve a diverse group in the hiring process. The good news? Once you have a QA, they can help you find more!

3. Soft skills needed

A good QA doesn’t just have testing skills. They also have a range of skills that mean they’re able to withstand the challenges that testing throws at them. QAs need to be great communicators: they have to be able to share results and explain findings across all departments, so someone who can communicate both up and down is key. Similarly, testing can also be a repetitive and painstaking job, even pretty boring at times. It’s not a job for quitters, so it’s only fair to look for people who have strong motivation and are determined.

4. Take it slow

During these early stages, there’s no harm in starting with project-based hiring; hiring support to work on short term contracts to meet project demand. While this does mean you’ll need to deal with freelancers and HR again and again as new projects pop up, it also means that you have a great opportunity to identify the sort of person that works for your business – and the sort of person that doesn’t – without having to commit long term. Platforms like Upwork and Freelancer.com are great for dipping a toe in.

5. Build a good candidate experience

Not everyone you meet with will be your first QA Engineer. Or even your next QA Engineer. But they very well may play an important role in your business’ future. By working to create a positive candidate experience – by providing feedback, listening to feedback, and being respectful of the candidate’s time – you can help to improve your business reputation in the recruitment landscape. Hiring is very network-based. Treat candidates nicely, and they may refer a friend next time.

Don’t Lose Sight of What Matters

A report by the Recruitment and Employment Confederation shows that 85% of businesses have made a bad hire. A bad hire can mean many things, from hiring someone who lacks the necessary skills to do the job, to someone who simply doesn’t want to be there. Ultimately, this can have a big impact on organisations.

One mistake new businesses often make is hiring too late. Cash-strapped startups have to wait until a small skill gap becomes a major problem by which time they have to juggle recruitment, and revert to ad-hoc HR processes while also racing around trying to put out fires.

Hiring your first QA Engineer – and taking those next steps towards growing your business – can be an exciting time. But don’t lose sight of what really matters: your customers. Try to find a healthy balance between building your recruitment process and maintaining productivity and product quality. Hiring with care and attention is crucial to reduce the risk of a bad hire, but don’t let it become a distraction.

Visit TestTheTest.com