AI-powered test case generation: Myth or reality?

In recent years, we have heard about Artificial Intelligence everywhere in every field. The impact of AI is increasing day by day in software testing as well. It will be a game changer for the QA community. But can AI really generate test cases automatically?

Picture this: Can you imagine a QA feeding only a user story or requirements document into a tool and then getting the complete set of functional test cases (both positive and negative)? Even specific to non-functional testing like performance or security testing, as well. Maybe or maybe not?

The key point is whether it can generate test cases that will be beneficial for a QA or whether it’s just a waste of QA time in updating or verifying the generated test cases.

Traditional approach to designing test cases

Before seeing what AI can and cannot do, let us first evaluate the human approach to creating test cases, the way it has been done in the traditional way.

In this approach, the QA reads and understands the requirements document, user stories, and acceptance criteria. After that, it’s time to identify the following as per the QA professional’s domain knowledge and experience:

• Positive test scenarios
• Negative test scenarios
• Edge cases that will include unique flows for the application.
• Align with business logic.
• Ensure that the coverage of application is good so that most critical issues can be encountered.

A QA professional needs to think on many fronts - how normal users access the application, how smart users can use the application, and how even someone who has no knowledge of technology would access it.

What is AI-powered test case generation?

AI-powered test case generation means that it uses Machine Learning models or Natural Language Processing models to automatically generate test cases, its steps, and even any real/dummy test data, based on the Requirements Document, user stories, and acceptance criteria.

During a story refinement session, if a transcript is available, it can also be taken as an input to generate test cases. Test cases can be generated for functional scenarios and non-functional scenarios like performance testing and security testing.

What’s available today in the world of AI?

Today, there are several tools that can generate test cases for QA and help them with their day-to-day tasks.

1. ChatGPT, Microsoft Copilot, Google Gemini, Claude and many more LLM-based tools can take the user input and provide the required set of test cases, steps, and data.
2. Multiple commercial tools like Mabl, testRigor, and Testim use AI engines to generate test cases based on user actions. This is called recording of the user interaction with the application under test.
3. Tools like TOSCA copilot help users to write the flow of requirements in simple words. The tool uses AI to analyze the requirements and generate manual as well as automated tests within minutes, which can be directly used by the QA with minimal changes.

Limitations and challenges

Since AI is currently in a learning phase, it will naturally have its limitations and challenges that any QA professional will encounter while generating test cases.

1. AI can have challenges with domain knowledge. While it can generate test cases, it will not be able to provide complex user scenarios, edge cases, or business rule-specific scenarios. As AI expertise develops more, once can expect this challenge to be reduced with time, in future.
2. AI can hallucinate, which will lead to overthinking and fluffy output. This will generate test cases which are irrelevant to the application under test.
3. A lot of fine-tuning will be required to generate better results, which is a challenging task and will require more time and resources.
4. Given the lack of context engineering, the AI tool or LLM can generate incorrect results. It can also hallucinate as an unclear or ambiguous prompt for the AI might lead it astray.

How does AI-driven test case generation help?

While AI cannot fully replace human QA, it can surely help them accomplish their tasks in less time and assist them in being more productive.

• It will be helpful during Shift Left Testing as the requirements are available and the QA can start creating test cases which will help at later stages when Testing starts.
• As per the history of defects, AI can generate test cases, which will help in increasing the test coverage and reduce future bugs.
• AI can generate the basic scenarios of test cases, which will help in speeding up the test case creation. Basic tests include login scenarios, form validations, etc.

What is the future of test case generation in world of AI?

The future of AI in QA community is not only about generating test cases but also about integrating them into real world testing workflows. This is where MCP (Model Context Protocol) comes into picture.

AI can generate test cases, but the real challenge is getting those cases into test management and requirement tracking tools like JIRA, or into CI/CD pipelines. This is where MCP comes in — it acts as the bridge that integrates different tools and enables them to communicate seamlessly with each other.

Thanks to its ability to handle the repetitive tasks of generating test cases, QA teams can focus on more productive work like exploratory testing and domain/business-specific testing.

AI-powered test case generation checklist

Figure 1: AI-powered Test Case Generation Checklist

Step 1: Requirement Analysis

We must gather all the functional and non-functional requirements related to project that we will be testing. Then analyze these requirements to understand the scope, identify repetitive areas, and determine which requirements can be beneficial from AI-powered test case generation.

Step 2: AI Model Selection

Find out the available LLM models that are designed for test automation or can generate test cases based on good prompt. Select the one that best suits your requirements, testing needs, and integration capabilities. Before using it full fledge do some trial runs with few requirements so that you get the confidence of using it.

Step 3: Test Data Preparation

Carefully select, organize and prepare a variety of different data sets so that we can provide a good prompt to AI and accordingly it will generate good results and will not hallucinate. Include domain specific, real business flows and edge cases as well in the test data.

Step 4: Test Case Generation

Use the AI model to generate test cases that were selected based on trial runs that have given the feasible results. Review the test cases that are generated automatically to check if solves your purpose and provide related test cases that will be beneficial for the QA user.

Step 5: Validation & Refinement

Review the automatically generated test cases with the subject matter experts and business users so that they can provide the real feedback if this will help them to reduce effort and time. Or it will require more refinement. If it requires more updates as per SME feedback, then the prompt can be made better to get relevant results.

Step 6: Final Output

Once the SME and business users are satisfied with the output and they feel that it will improve the quality, reduce time and effort, you can export the optimized test cases. Use feedback from experts to enhance the test case generation for future as well. This is an ongoing activity that will improve the output as per feedback.

Myth or reality?

AI-powered test case generation is indeed a reality and not a myth. It is helping the QA community by generating a set of test cases. This reduces the time taken in creation of tests as teams don’t have to write everything from scratch. They already have a set of tests that can be analyzed and used. But there’s still a long way to go before we have more domain expertise and generate more robust test cases, which will include complex scenarios and edge cases. For complex business scenarios and edge test cases, test case generation can be improved drastically by providing better prompts with the help of feedback from the domain experts.

Stay tuned for more interesting reads related to MCP and Agentic AI which will be assisting the QA community to work much faster and will also improve the quality of their work. 

Explore Quality Copilot – your AI asssistant for smarter test design. Get demo access >> here!