AI can fail you, and in a big way: by delivering results, which nobody cares about, or providing results only if you invest millions, which might not guarantee final success. Measuring the ROI of the Artificial Intelligence project will put you on the right track.
Failures are there – Microsoft and IBM examples teach a lot
You might not even be able to demonstrate that AI is driving your investors and stakeholders to profit. Failures in the world of AI nowadays can be tiny or huge. For example, Microsoft, which announced its new chatbot, which was capable of communicating with the slang-laden voice of a teenager. Chatbot’s name was Tay, and it could automatically respond to people and join “casual and playful conversation” on Twitter. Less than 24 hours after Tay started, internet bots had completely “corrupted” Tay’s personality.
Another example comes from the tech giant, IBM. In 2013, IBM started to work with The University of Texas MD Anderson Cancer Center to design and build a new “Oncology Expert Advisor” platform. The system ultimately failed. Medical teams identified several examples of hazardous and inaccurate treatment paths, including a case where the platform recommended that doctors provide a patient with bleeding medicine that could worsen the bleeding. The university spent 62 million dollars and didn’t get any results (link to the report).
Capgemini’s study found that only 27% of data projects are regarded as successful, and only 8% of the big data projects are considered very auspicious. According to TechRepublic, 85% of AI projects eventually fail to bring their intended results to the business.
Because AI catches the eyeballs, and the term raises expectations, you want to be the person who can demonstrate that your use of Artificial Intelligence contributes to ROI, and you measure the ROI as well.
Step into the Artificial Intelligence now, or be late to the coffee
AI has taken over nearly every aspect of business nowadays. Every major player in every industry has Artificial Intelligence plotted into almost every project and precisely measured ROI. In automotive, BMW has used AI to recognize more than 203.000 unique parts numbers coming from 4.500 suppliers. AI helped to increase the number of manufactured cars. BMW produces a car every 56 seconds.
In eCommerce and restaurant, Yelp, which runs venue recommendation websites, uses Artificial Intelligence to experiment with buttons, scrolling, headlines, and everything that influences online conversions. Artificial Intelligence-powered platform runs 700 different experiments in total being run at any one time.
Then in retail Harley Davidson is using AI to automate the process of booking and selling motorcycles. Machine Learning models were trained to build micro-segments and target ads precisely. As a result, the company generated 2.930% more leads and sold 40% of motorcycles in a given period, without human touch.
Challenges in measuring ROI in Artificial Intelligence projects
You will have to face several challenges on your path to return on AI investments.
Funding as a part of the ROI measure
Over the years, I noticed that companies powered by automation were randomly using the term AI in their communication when they only focused on data analytics. In such cases, the product does not become more intelligent over time. Automation is easier and doesn’t require as much funding as Artificial Intelligence. You need to plan for fundraising; therefore, if you don’t have the capital to implement, monitor, and optimize your AI internally, you will find it hard to measure ROI. Conducting AI projects is a more resource-consuming process. You need to put more work in measuring models and algorithms, which is part of measuring ROI exercise.
Skills in Artificial Intelligence projects are different
You will also need a well-designed and executed plan for reskilling your employees and redesign team structures. A recent study by Accenture explains that the top three skills will shift to understanding digital, creative thinking, and data analysis. Your employees need to learn these skills and understand that data analysis and creative thinking are necessary to train Artificial Intelligence models. When you retrained your people, you need to run recruitment properly, to build the proper teams. Usually, every Artificial Intelligence project, which brings sustainable ROI requires the following squad:
- Software Engineer
- Machine Learning Researcher
- Data Scientist
- Machine Learning Engineer
- Product Manager of AI-powered products
- Conversational UX designer and researcher
- Growth Hacker
AI projects without robust data will never bring ROI
Developing a reliable data strategy before you start teaching and training AI models is crucial. You need to:
- map out the data you own,
- decide who you need to partner with to bring essential data,
- estimate how much data you will lack,
- plan out how to clean up and reconstruct your data pool
Obtaining the training data is the biggest challenge for the success of an AI project. I estimate that 96% of the AI projects fail or not started due to a lack of training data that guides the inability to train the ML algorithms resulting in the project’s failure.
What should you consider when measuring ROI in AI projects?
There are several, fundamental questions you need to ask before you start putting numbers into the excel spreadsheet:
What is the business case?
It is evident that the goal of calculating ROI is to determine the effectiveness of Artificial Intelligence project fundings. For instance, Domino’s Pizza realized that bringing robots to the kitchen can automate the repetitive task of mixing ingredients. Moreover, by collecting data, the chain of pizza stores, predicts the volume of purchases and the number of deliveries. An amazing example of influencing ROI of Artificial Intelligence effort. Eventually, the company redesigned contracts with suppliers and mitigate a lot of risks.
[elementor-template id=”3798″]
The above is the real business case, which consists of three elements: removing repetitive tasks (AI loves) and predicting financial impact and restructuring processes.
Is this the right Artificial Intelligence model to solve your business problem?
An organization must approach AI with a clear problem statement. For instance, we can’t deliver great insurance to older adults, as we don’t know their lifestyle. Rather than, “let’s do something for older folks and use AI for it,” Many businesses have not defined problem statements which leads to unrealistic expectations. You will not only miss your goals, but you have no idea what your funding and expected revenues should be – both are critical for measuring ROI.
What is your frequency of training AI models?
Frequency of measurement will be affected by many factors, one of the most important is how often you need to train your AI models. Facebook needs to do it at least daily, while Maersk (cargo company) can conduct retraining processes every month. Training models is required to build smarter algorithms but also costly. Any time you train the model, you will learn what is lacking. It might be a data record, data variable, or another line of code. Depends on the training frequency, your ROI will be calculated differently.
eBay developed the tool which clears the image backgrounds. It helps sellers quickly remove low-quality backgrounds from pictures, making the image more attractive for the buyer. Eventually, this tool improves CTR, important in eCommerce. This kind of model needs to be trained daily, which shapes the ROI measuring process significantly.
How will you measure the success of an AI deployment?
Step 1 – The costs of implementing Artificial Intelligence
At this stage, you should sketch all the expenses related to the implementation of Artificial Intelligence. Some of the positions are listed here:
- Team
- Licenses and tools
- Platforms and security
- Data (get, maintain, store, protect)
- Algorithms (choose, build, maintain)
- Models (choose, train, shift)
- Prototyping and designing
- Building and maintaining a final solution (incl. iterations)
From the licensing to the implementation, the annual maintenance, and the in-house recurring operation costs: all of these categories have particular expense items that need to be detailed and estimated.
Step 2 – The benefits of bringing AI under the roof
The benefits, or value that the AI-related investment will generate are more difficult to estimate than making expenses list. You want these benefits to be real and monetary, however, as mentioned earlier, they very well can be non-financial at the beginning. Imagine the Harley Davidson, which I mentioned above. Installing necessary infrastructure was costly, and nobody could predict that Harley would generate 40% of sales without human touch in the niche, which was not exploited before.
Here are categories of benefits, which any ROI calculation requires:
- Revenue
- Efficiency
- Productivity
- Reputation and Compliance
- Sustainability
- Resilience
- Assets
- Product and Service Quality
- Brand Identity
- Market Share
- Pricing Power
- Organizational Culture
- Knowledge
- Quality of work
Step 3 – The comparison
The final phase of the AI ROI measurement is to compare the costs to benefits and determine the ROI’s cash flow. Hopefully, this will deliver a great result for your planned AI investment. For an even better evaluation, you can estimate all of it for the best and the worst cases: you will enhance the awareness of the risks of investment. Stakeholders and investors love that, and they are right, I think.
A recent article on MIT Sloan Business Review says that new methods of working and the latest management strategies are among the most considerable factors keeping most AI initiatives from delivering planned ROI.
There is no guarantee that an AI project will not run above budget, given the uncertainty mentioned above and numerous experimentation which is needed in this kind of project.
Do’s and Dont’s for measuring ROI in Artificial Intelligence projects
DO: include business leaders when you have any results to show as most of them are skeptical of AI.
DON’T: begin a project without the knowledge of what specific ML models can do and what is there nature (check the article to learn about it)
DO: select problems that are easy to measure. For example, a focus on selecting low-quality photos in the stack is easy to measure the before and after. If you are looking to find frauds, that is far more difficult to measure and hard to prove ROI on any AI spend.
DON’T: Execute on a data strategy and forget about an analytics strategy. The first challenge ties to the reality that data quality and management is an ongoing journey itself. If you wait for your data to be perfect, then you will never launch your AI strategy.
DO: test new project ideas in small pilots. Think big, start small. GoPro started with one data scientist, and the first challenge focused on categorizing users’ pictures. Relatively easy, but very scalable.
DON’T: Overlook full integration requirements. Integration implies knowing where the model will live and execute. Can the performance of an API process 10,000 SKU outputs straight to a shop manager’s tablet? If not, the multi-SKU forecast might as well not exist. Without deployment, no data-driven decisions occur. In this case, stocking the wrong components could result in rush shipments for high-demand parts, resulting in a higher purchase price (lower ROI)
DO: Management and staff must be educated and trained in cross-functional teams in all processes and operations.
DO: Calculate the breakeven point, even if you can’t predict all costs and benefits. According to Investopedia, the breakeven point is the production level at which total revenues for a product equal total expenses. Calculate it upfront, and keep updating your files, so it becomes more realistic on the fly. You will experience better meetings and brainstorming sessions, avoid unnecessary risk, and make the process trackable.
Few final thoughts on measuring ROI of Artificial Intelligence projects
There are several decisions leaders can make to position their organizations to maximize the benefits of prediction machines and increase the chances for attractive ROI.
Consider premade blocks versus in-house solution
The problem with any Artificial Intelligence-powered products is that it’s time and money consuming. Only the model training phase can take from three months to a few years. There are too many unknown things starting from your data stack and ending at scaling the AI-fueled products. Fortunately, you can consider at least two approaches, hiring ready to go software or hardware (I call it Hire PreMade Blocks approach), versus building everything inside your company.
PreMade Blocks approach
The first process you can conduct is to examine whether a specific premade block meets your business requirement. It’s faster and less costly to integrate ready-to-use software (premade block) into a process than building one in-house. For instance, if you are in the entertainment business and own the library of movies, you might want to make them searchable. Instead of building an AI-powered engine from scratch, you can hire Valossa Video AI to perform the job.
PreMade blocks usually offer innovative business models like freemium, or all you can eat options. It gives your company agility and better cost control as well as a quality check.
Build it inside
You can consider building an in-house solution. If you choose to develop a custom AI-powered solution, you will get more accurate software, you will be the owner of the architecture, and you will control your data flow in 100%. This kind of project will require the different structure of resources, than PreMade Blocks strategy. At least you need to make sure you have an Artificial Intelligence team for that challenge and data strategy in place. If you must consider the bigger picture and ask yourself a question, why building inside is a better option. If you answer “because we always do like this,” you might squeeze your ROI and bring unnecessary cost into the equation.
Assessing data acquisition and labeling
Run the data test and answer at least these specific questions before you move further. Answers might give you a chance to improve data quality, which will result in building a well-deployed AI-powered system faster. Also, if you decide the data is not good enough, you might consider changing project objectives.
- Does the data contain information that can predict the business objective?
- Are the details of the training and forecast correct?
- Do you already have tagged data (i.e., training examples)?
- Is your data correct/accurate?
- Do you have enough data?
- Is the data easily accessible to the team and working ML models?
- Can you read the data fast enough?
- Do you have documentation for each data field?
- Are the missing values a small percentage of the fields we are interested in?
- If your data is periodic, do you have data for 3 x periods?
- If you want to predict n periods in advance, do +2 data periods?
- Do you know the timestamp where each data value was obtained, or is it updated?
- Is your data unbiased?
- If values are missing, do you know the reasons?
- If values are missing, are they random?
- Does the data have the same unit for each field (input or target)?
- Is the same value registered the same way everywhere?
The list was designed by Skuza Consulting and is based on Google AI’s best practices.
If the data quality is questionable at this stage, the Artificial Intelligence project squad needs to build a data acquisition strategy and consider possible data sources. Are the data available publicly? Should we buy data from any company? Should we acquire a specific company or partner with them?
Run Proof of Concept, it will help to collect more information for future ROI measurement.
According to Wikipedia, “Proof of concept or proof of principle is a realization of a certain method or idea in order to demonstrate its feasibility or a demonstration in principle with the aim of verifying that some concept or theory has practical potential. A proof of concept is usually small and may or may not be complete.”
POCs are timed-boxed (defined by # of hours), with clear KPIs (key performance indicators) for measuring your results. This exercise keeps costs low and provides rapid insights into what results can be expected before investing significant resources into the project.
Proof of concept can demonstrate a project’s ROI potential just as well as it proves its technical capability.