Last week, Applitools hosted Unlocking the Power of ChatGPT and AI in Test Automation: Next Steps, where I explored how artificial intelligence, specifically ChatGPT, can revolutionize the field of test automation. In this article, I’ll share the audience Q&A as well as some of the results of the audience polls. Be sure to read my previous article, where I summarized the key takeaways from the webinar. You can also find the full recording, session materials, and more in our event archive.
Audience Q&A
Our audience asked various questions about the data and intellectual property when using AI and adding AI into their own test automation processes.
Intellectual properties when using ChatGPT
Question: To avoid disclosing company intellectual properties to ChatGPT, is it not better to build a “private” ChatGPT / large language model to use and train for test automation inside the company while securing the privacy?
My response: The way a lot of organizations are proceeding is setting up private ChatGPT-like infrastructure to get the value of AI. I think that’s a good way to proceed, at least for now.
Data privacy when using ChatGPT
Question: What do you think about feeding commercial data (requirements, code, etc.) to ChatGPT and/or OpenAI API (e.g. gpt-3.5-turbo) data privacy connected to the recent privacy issues like with Samsung, exposure of ChatGPT chats, and so forth?
My response: Feeding public data is okay, because it’s out in the public space anyway, but commercial data could be public or it could be private and that could become an issue. The problem is we do not understand enough about how ChatGPT is using the data or the questions that we are asking it. It is constantly learning, so if you feed a very unique type of question that it has never come across before, the algorithm is intelligent to learn from that. It might give you the wrong answer, but it is going to learn based on your follow-up questions, and it is going to use that information to answer someone else’s similar question.
Complying with data regulations
Question: How can we ensure that AI-driven test automation tools maintain compliance with data privacy regulations like GDPR and CCPA during the testing process?
My response: It’s a tough question. I don’t know how we can ensure that, but if you are going to use any AI tool, you must make sure you are asking very focused, specific questions that don’t disclose any confidential information. For example, in my demo, I had a piece of code pointing to a website asking it a very specific question how to implement it. That question could be implemented using some complex free algorithms or anything else, but taking that solution, you make it your own and then implement it in your organization. That might be safer than disclosing anything more. This is a very new area right now. It’s better to be on the side of caution.
Adding AI into the test automation process
Question: Any suggestions on how to embed ChatGPT/AI into automation testing efforts as a process more than individual benefit?
My response: I unfortunately do not have an answer to this yet. It is something that needs to be explored and figured out. One thing I will add is that even though it may be similar to many others, each product is different. The processes and tech stacks are going to vary for all these types of products you use for testing and automation, so one solution is not going to fit everyone. Auto-generated code will go up to a certain level, but at least as of now, the human mind is still very essential to use it correctly. So it’s not going to solve your problems; it is just going to make solving them easier. The examples I showed are ways to make it easier in your own case.
Using AI for API and NFRs
Question: How effective would AI be for API and NFR?
My response: I could ask a question to give me a performance test implementation approach for the Amazon website, and it gives me a performance test strategy. If I ask it a question to give me an implementation detail of what tool I should use, it is probably going to suggest a few tools. If I ask it to build an initial script for automating this performance test, it is probably going to do that for me as well. It all depends on the questions you are asking to proceed from there, and I’m sure you’ll get good insights for NFRs.
Using AI a dedicated private cloud instance
Question: Our organization is very particular about intellectual property protection, and they might deny us from using third-party cloud tools. What solution is there for this?
My response: Applitools uses the public cloud. I use a public cloud for my learning and training and demos that I do, but a lot of our customers actually use the dedicated cloud instance, which is hosted only for them. Only they have access to that, so that takes care of the security concerns that might be there. We also work with our customers to ensure from compliance and security perspectives that all the questions are answered and to make sure everything conforms as per their standards.
Using AI for mobile test automation
Question: Do you think AI can improve quality of life for automation engineers working on mobile app testing too? Or mostly web and API?
My response: Yes, it works for mobile. It works for anything that you want. You just have to try it out and be specific with your questions. What I learned from using ChatGPT is that you need to learn the art of asking the questions. It is very important in any communication, but now it is becoming very important in communicating with tools as well to get you the appropriate responses.
Audience poll results
In the live webinar, the audience was asked “Given the privacy concerns, how comfortable are you using AI tools for automation?” Of 105 votes, over half of the respondents would be somewhat or very comfortable with using AI tools for automation.
- Very comfortable: 16.95%
- Somewhat comfortable: 33.05%
- Not sure: 24.59%
- Somewhat comfortable: 14.41%
Next steps
You can take advantage of AI today by using Applitools to test web apps, mobile apps, desktop apps, PDFs, screenshots, and more. Applitools offers SDKs that support several popular testing frameworks in multiple languages, and these SDKs install directly into your projects for seamless integration. You can try it yourself by claiming a free account or request a demo.
Check out our events page to register for an upcoming session with our CTO and the inventor of visual AI Adam Carmi. For our clients and friends in the Asia-Pacific region, be sure to register to attend our upcoming encore presentation of Unlocking the Power of ChatGPT and AI in Test Automation happening on April 26th.
