The AI promise
AI is not a new technology. Recently, the field has made huge advancements.
Currently it seems like AI technology is mostly about using ML (machine learning) algorithms to train models with large volumes of data and use these trained computational models to make predictions or generate some outcomes.
From a testing and test automation point-of-view, the question in my mind still remains: Will AI be able to automatically generate and update test cases? Can it find contextual defects in the product? Can it eventually inspect code and the test coverage to prevent defects getting into production?
ICYMI: Read my recent article on this topic, AI: The magical helping hand in testing.
The promising future
Recently I came across a lot of interesting buzz created by ChatGPT, and I had to try it out.
Given I am a curious tester by heart, I signed up for it on https://chat.openai.com/ and tried to see the kind of responses generated by ChatGPT for a specific use case.
Live demo: Watch me demonstrate how to use ChatGPT for testing and development in the on-demand recording of Unlocking the Power of ChatGPT and AI in Testing: A Real-World Look.
What is the role of AI in testing?
I started with a simple question to ChatGPT. The answer was interesting. But the answer was nothing different than what I already knew.
So I thought of asking some more specific questions.
Create a test strategy using AI
I asked ChatGPT to create a test strategy for testing and automating Amazon India shopping app and website.
I was blown away by the first response. Though I can’t use this directly, as it is quite generic, the answer was actually pretty good as a starting point.
I was now hooked, and I had to keep going on now.
What test cases should I automate for my product?
Example: What test cases should I automate for Amazon India?
Like the test strategy, these are not to the level of details I am looking for. But it’s a great start.
That said, at the top of the test automation pyramid, I do not want to automate test cases, but I want to automate test scenarios. So, I asked ChatGPT about the important scenarios I should automate.
What test scenarios should I automate for my product?
Though not the specific test scenarios I was expecting, there is a clear difference between the identified test cases and the test scenarios.
Code generation: the first auto-generated test
I asked ChatGPT to give me the Selenium-Java automation code to automate the “search and add a One Plus phone to cart scenario” for Amazon India website.
ChatGPT not only corrected my question, but also gave me the exact code to implement the above scenario.
I updated the question to generate the test using Applitools Visual AI, and voila, here was the solution:
Is this code usable?
Not really!
If you run the test, there is a good chance it would fail. The reason being, the generated code assumes the page is “rendered” as soon as the actions are done. To make this code usable and consistent in execution, we need to update it with realistic and intelligent waits at relevant places to cater to the rendering and loading time required by the browser, and of-course, your environment.
NOTE: Intelligent waits are important as opposed to a hard wait to optimise the execution time. Also, being realistic in your wait durations is very important. You do not want to wait for a particular state of the application for more than what is expected to happen in production.
Also, typically in our test frameworks, we have various abstraction layers and utilities to manage different functionalities and responsibilities. But we can use snippets of this generated code and easily plug it in the right places in our frameworks. So it is still very valuable.
I also tried the same for some internal sites – i.e. not available to the general public. The test was still created, but not sure about the validity of those tests.
Test data generation
Lastly, I asked specific questions around test data – an aspect unfortunately ignored or forgotten until the last minute by most teams.
Example: What test data do I need to automate test scenarios for my product?
Again, I was blown away by the detailed test data characteristics that were given as a response. It was specific about the relevant data required for product, user, payment, and order to automate the important test scenarios that were highlighted in the earlier response.
I tried to trick ChatGPT to “search and add a product to cart” – but I didn’t specify what product.
It caught my bluff and gave very clear instructions that I should provide valid test data.
Try it yourself: Can ChatGPT accurately translate your tests from one framework to another? Try it out and let us know how it went @Applitools.
Other examples
Generating content
Instead of thinking and typing this blog post, I could have used ChatGPT to generate the post for me as well.
I didn’t even have time to step away to get a cup of coffee while this was being generated!
Planning a vacation
Another interesting use case of ChatGPT – getting help to plan a vacation!
NOTE: Be sure to work with your organization’s security team and @OpenAI before implementing ChatGPT into the organization’s official processes.
What’s next? What does it mean for using AI in testing?
In my examples, the IDEs are very powerful and make it very easy for programmers and automation engineers to write code in a better and more efficient way.
36.8% of our audience participants answered that they were worried about AI replacing their jobs. Others see AI as a tool to improve their skills and efficiency in their job.
Tools and technologies like ChatGPT will offer more assistance to such roles. Individuals can focus on logic and use cases. I have doubts on the applicability of these tools to provide answers based on contextual information. However, given very focused and precise questions, tools can provide information to help implement the same in an efficient and optimal fashion.
I am hooked!
To learn more, watch on-demand recording of Unlocking the Power of ChatGPT and AI in Testing: A Real-World Look. Be looking for a follow-on event in early April where I will go deeper into specific use cases and how ChatGPT and its use in testing are evolving. Sign up for an email notification when registration opens.
Quick Answers
AI can assist in test automation by generating test strategies, identifying potential test scenarios, and even creating automated test scripts. It helps reduce manual effort, improve efficiency, and ensure that tests align with user requirements and application behaviors.
ChatGPT can provide a solid starting point for a test strategy. While its outputs might be generic, they are helpful for brainstorming and structuring initial plans. However, human expertise is still essential to refine and tailor the strategy to specific requirements.
Not entirely. The generated code often lacks realistic waits, proper error handling, and integration with test frameworks. Manual updates and adjustments are needed to make the code reliable and executable in real-world scenarios.
ChatGPT can suggest detailed and relevant test data requirements, such as data for users, products, payments, and orders. It provides a strong foundation for generating the necessary data sets for automation, ensuring comprehensive testing coverage.
ChatGPT relies on the quality of the input provided. It struggles with highly contextual questions and cannot independently validate or execute tests. Additionally, domain-specific nuances and complexities still require human intervention.
Yes, ChatGPT can assist in translating tests between frameworks by generating corresponding code snippets. However, manual oversight is crucial to ensure functional accuracy and compatibility with the new framework.
AI is unlikely to replace human testers entirely. Instead, it complements human efforts by automating repetitive tasks and providing insights. Human testers will still be needed to handle complex scenarios, contextual assessments, and creative problem-solving in testing.
Applitools Autonomous offers purpose-built features like modular test design and Visual AI for comprehensive test automation, while ChatGPT provides basic assistance by generating test strategies or code snippets. Autonomous is tailored for precise, efficient workflows, while ChatGPT supports broader, conceptual tasks.
