TheTestMart - Leveraging ChatGPT
By David Hewit & James Farrier
Bridgewest Ventures portfolio company TheTestMart are leveraging ChatGPT to increase customer benefits and increase operational efficiency in software development & testing.
Demand for software developers and software testing experts has increased substantially over the last several years. However, the supply of software developers has not been able to keep pace leading to a shortage of these critical professionals and soaring costs for companies of all types, especially those engaged in software development.
“In the five years through 2019, US job postings for roles in software engineering increased +69%, machine learning increased +417%, data science increased +167%, and DevOps increased +443%”
Source: Bain & Company
The shortage of software developers and testing experts has had a significant impact on software companies' margins and profitability. Companies are facing increased costs due to higher salaries for skilled professionals and the need for additional recruitment and training expenses. In 2020 and 2021 the growth in average compensation for software engineers was over 20% higher than growth in compensation for all employees.
Despite the significant number of high-profile layoffs announcements by prominent technology companies over the last several months, the supply/demand imbalance for software developers has not changed. In the United States the unemployment rate in the technology sector decreased to 1.5% in January and is less than half that of the overall US unemployment rate (3.4%). According to Janco Associates, a management consulting firm focused on the technology industry, layoffs did not hit developers and were focused on data center operations, administrative, and HR roles. Recent studies conducted by ZipRecruiter showed that 79% of laid off tech-company workers were able to find a new job within three months of starting their employment search. There are currently over 109,000 unfilled IT related jobs in the United States, according to Janco.
Source: CompTIA
Over the long-term these trends are expected to persist. In the USA, the Bureau of Labor Statistics projects that demand for software developers will increase by +22% between 2021 and 2031, far higher than the +5% increase in overall employment they are projecting over the same period. Globally, the situation is no different, the World Economic Forum projects that over the next decade there will be a global shortfall of over 18M IT-related workers. To keep software development expenses under control, the world needs new tools to dramatically enhance the productivity of our existing pool of software development talent. Enter ChatGPT.
ChatGPT is a language generation model developed by Open AI that uses deep learning to generate human-like responses to text inputs. ChatGPT can be used for a variety of use cases from answering basic search queries, to writing blog posts, to building chat bots for customer service inquiries. Open AI has developed a version of ChatGPT specifically for software code generation called “GPT-3 Code Generation,” which can generate code in a variety of programing languages. In a recent test conducted by Google, ChatGPT passed a level 3 engineering exam, which would qualify ChatGPT for an entry-level engineering position at Google, with a total compensation package of $183,000.
At Bridgewest Ventures, we have seen our portfolio companies and other start-ups take advantage of ChatGPT in novel and interesting ways to enhance the quality of their products and services and improve their organization’s productivity. A wonderful and powerful example of this in action is the recent work done by TheTestMart, a Bridgewest Ventures portfolio company.
TheTestMart is a SaaS based platform that automates the testing of software upgrades and updates. Software testing is a labor intensive and time-consuming process. Less than 25% of companies have introduced any meaningful automation to their software testing processes and even the automated software testing tools available still require multiple full-time employees and weeks’ worth of manual work to complete routine software tests. TheTestMart’s first ever pre-built library of configurable tests has been a game changer, enabling their customers to reduce IT headcount dedicated to an ERP testing down to one employee and the amount of time it takes to complete testing down to one day.
Leveraging ChatGPT and other in-house developed AI technology, TheTestMart has been able to further increase their value proposition while simultaneously increasing the scalability and operating efficiency of their business. TheTestMart has been able to leverage ChatGPT to speed up the process of test creation.
“Approximately 90% of the tests that customers run are standard tests that can be leveraged across all TestMart customers, but 10% of tests are customized tests that cannot be scaled. It takes one minute to deploy a standardized test, but over an hour to deploy a customized test.”
This custom development work slows down the overall customer on-boarding process and is a significant cost in TheTestMart’s business, accounting for a mid-teens percentage of revenue.
TheTestMart has been able to automate this manual process by providing ChatGPT with multiple test templates from their existing library of tests and certain customer specific parameters and allowing ChatGPT to combine bits and pieces of multiple templates into a custom designed test for that customer. TheTestMart also uses ChatGPT to leverage tests built for certain customers for other customers with similar systems and testing requirements. Importantly, this creates two virtuous flywheels for TheTestMart and its customers, which continuously drives test quality higher while driving costs lower.
The adoption of SaaS software and its more frequent pace of customer updates has driven strong growth in the demand for software testers and developers. Currently approximately 30% of IT budgets go towards testing and other quality assurance activities. It is every CIO’s dream to simply type “test my system” into ChatGPT and have that testing activity automatically run.
TheTestMart is continuing to work on AI technology to make this dream a reality. In the future TheTestMart will be able to automatically edit tests and duplicate tests for different systems and datasets with ChatGPT. The cost savings and efficiency gains from the use of tools like ChatGPT are significant;
“We expect to decrease onboarding times from 20 hours to 8 hours through intelligent automation, and our early estimates suggest a service cost reduction of around 80%, with basic customer support being fully automated. ”
TheTestMart are already testing our prototype, and are working towards a general availability release. Imagine being able to test your system by typing in “Test my ERP system”. Well, that is where TheTestMart is heading.