AI Analytics – Hype or Future?

Starting a new project to test AI Analytics solutions on the market today, the good the bad, and the ugly. Here is my view,

Assumptions:

1. AI will disrupt and transform virtually every area of tech (and life), just like digital disruption did before.
2. Yes, we need to start from the business goals to solve with AI. If we are building AI solutions for AI’s sake, we will crash and burn.
3. Making data-driven decisions with the help of analytics should be a perfect area for AI to help us solve business goals. After all, this is based on clear cut data. AI should be better and faster at turning data into insights than humans can be.
4. AI should reduce the need for manual effort from data teams

Open questions that I will be testing in this project:

1. First of all, can today’s AI analytics products live up to these assumptions?
   • Data is complex and messy.
   • Human data scientists first need to understand the data set and transform the data into the right data model before they can apply analysis.
   • Human data teams also need to understand the desired business or personal outcomes that are the end goal, not just the output of the analysis they are supposed to deliver
   • How will an AI know how to do that?
     
2. A practice of data-driven decision intelligence requires much more than just analysts and analysis. We also need the following, otherwise any ignorant (“blind”) analysis of the data will be mathematically correct but still garbage, in the end.
   • Data cleaning
   • Evaluation of the data for biases
   • Domain expertise
   • Judgement for taking the decision that the data is suggesting depending on the context, strategy, implications, risks, ability to execute, that is not visible in the data itself.
   • Automation for taking action. But that permission to take action will not be granted if the AI solution is working in a silo.

Decision making based on data is difficult even for humans because it needs to take into account the situational context of, … everything. Can AI go beyond being the analyst on the team?

Reality check?

There are a lot of Analytics solutions on the market that claim to include AI. They are forced to make these claims, because their competitors are making the same claims. But are they really more than partial solutions? To what degree do they actually save time? I am sure they provide good answers sometimes for some use case, but how often?

• It needs to be a true solution to an actual business or personal goal to be worth anything.
• A partial solution via AI is only useful if it is a co-pilot.
  • For example, programming code and marketing copy written by AI is a super useful partial solution,
  • It is not usually perfect enough to just use “as is”, but it saves tremendous time by getting the human part way to their goal,
• A solution that is not practical will be more frustrating than no solution at all
  • For example, see this review of the AI Pin.
  • If the review is accurate, here the AI-based device creates more practical headaches than actual value, unfortunately.

The difference is nuanced, e.g. an intern that requires more effort to train and manage than the value that he or she is providing is not helpful. But an intern that is actually helpful is a huge value. Usually, it takes time to ramp up employees, is that the case for AI too?

How will I go about this AI Analytics evaluation?

Here is the plan

1. Use a standard uniform data source that I will analyze with numerous AI Analytics products on the market today. I will be using the sample Google Analytics 4 (GA4) web analytics data set on Big Query that includes anonymous data from Google’s Merch store.
2. Start from the business goals and questions – see below
3. Use various AI Analytics tools on the market today
4. Evaluation criteria
   • Time savings? What effort and steps are needed for the tool to understand the data? Does it accelerate my analysis with practical help?
   • Insights generation? To what degree can the tool tell me things that I didn’t already know?
   • Explainable AI? Is the tool able to show its work?
   • Trust? Does the tool instill confidence and earn trust, e.g. by delivering consistent, repeatable answers?

What are the business goals for the AI analysis to deliver?

We’ll assume the business goals of a typical pure play ecommerce company, i.e. increase revenue and profitability by improving traffic, conversion, AOV, repeat purchases.

Descriptive analysis questions:

Bring me closer to my customers, i.e. help me understand

• What’s the profile and behavior of my best customers?
• How do they see and engage with my store?

Diagnostic analysis questions

Where are my biggest issues and why are they happening?

• What stops customers from purchasing?
• from coming back?

Prescriptive analysis questions

What is going to happen next?

• Who is going to come back and make a purchase?

Prescriptive questions

What should I do next?

• What should we promote better to increase return visits and purchases?
• What should we personalize for customers (e.g. in an email) to bring them back?
• What should we personalize for return visitors to make it more likely they will purchase?

It’s a tough challenge for the data set since it is only GA4 data, e.g. it doesn’t include data on friction and frustrations or behavior between clicks. But, let’s see what the AI Analytics tools can tickle out of the data.