Why is there still no Expert System for Web Analytics?

Isn’t it silly? Web analytics has been around for almost 15 years now. There are many books, many best practices, many consultancies, and many gurus.

But the common man’s approach to doing web analysis still seems a bit more like a teenager dabbling around in the dark vs. an engineer that systematically constructs or troubleshoots an issue. That is to say web analysis still seems very ad-hoc instead of being methodical.

No, the best practice of “set KPIs and implement a continuous cycle of improvement” is NOT enough.

Everyone who is in the space has heard of the best practice. It is a well taken recommendation. But it is only the beginning of a methodical approach.

As soon as one of your KPIs is down the “methodical” part ends and instead you are advised to check your funnels and your paths and segment and drill and compare and trend and experiment and yada, yada.

Where is the methodical best practice for that part, heh?

Car mechanics receive more complete training than web analysts

Imagine you towed your kaput car to the mechanic. The KPI of “starter” is clearly down, i.e. the car doesn’t start. But now the mechanic takes a screwdriver and randomly checks each of the gazillion screws on your car.

That would be ridiculous!

Clearly, the mechanic will be trained to apply their knowledge of what should be happening vs. where in the process the breakdown is occurring to narrow down the cause of the issue.

You might say that is exactly what we would do with funnel reporting in analytics. But not so fast!

Take an example.

Say, we are an eCommerce website and our KPI “revenue” is down this week

There ought to be an expert system that advises the web analyst in which order to investigate following interrelated questions:

  • Are conversion rates down?
  • Are average order values down?
  • Is traffic down?
  • Are new or repeat visitors down?
  • Are direct or campaign traffic sources down?
  • Which campaign channels are down?
  • Are click throughs on your search ads down?
  • Is search inventory down?
  • Is your paid search ad rank lower?
  • Are you running out of paid search budget during the day?
  • Are the direct competitor’s traffic and/or conversions up?
  • Are site errors up?

See what I mean?

There are too many questions for you to always have to derive from scratch in which order you should ask them.

Why should we need to reinvent the wheel every time instead of having an expert system walking the analyst through?

Similar to the car mechanic’s job, there is a logical order in which to investigate web site issues

For example:

Is Traffic down?

  • If Yes, then go on to troubleshoot traffic sources
  • If No, then go on to troubleshoot conversion rates

But this is not as simple as a “decision tree”. It will be more like a network of questions.

For example:

  • Traffic could be down because people may have shifted to clicking more on competitors’ search listings.
  • Conversion rate could be down because visitors may have shifted to dropping off and buying more from competitors

So, in both the Yes and No branches of the above question/answer case the logical path can reach a point where you ought to switch from your web analytics tool to your compIntel tool to answer the next logical question.

My point is though — did I miss something or

  • Where is there a white paper by our consulting gurus that has troubleshooting flow charts for each of the 5 site types and the most common 3 KPIs for each site type?
  • I haven’t gone through the UBC/WAA course ware so if it is in there, someone let me know please.
  • I have read most but not all web analytics books, so educate me if one of these has flow charts of questions in it?

Am I on crack?

I am sure you are yelling out loud by now: “Darn it, analytics are too complicated to turn into a static network of questions and web businesses are too different from each other even if they fall into just 5 common site types. “

  • eCommerce
  • lead gen
  • content / publishing
  • customer service
  • brand marketing / educational

But dude, medical Expert Systems could not eliminate medical doctors either — They assist doctors in doing their work more efficiently and more reliably!

And if the expert system runs out of ideas and can’t find the issue … you can always go back to applying your own brain.

That is all I am saying.

The expert system could never eliminate the need for having educated web analysts and consultancies. But it would make everyone more efficient and effective.

Road map

  • Phase 1: write down the trouble shooting algorithms on paper and try them out until they are ripened
  • Phase 2: turn them into an expert system for easier use
  • Phase 3: automate the connection to web analytics tools so that the expert system can check some of the answers itself and spit out a check list

An opportunity for the WAA, maybe?

Couldn’t this be a cool new online business for the Web Analytics Association?

  • The system would be open for members to upload their own expert rules
  • Members could select “trouble shooters” by site type and KPI
  • Members could rate other people’s trouble shooters so that the best ones will bubble to the top
  • An API would allow hooking up with the coolest web analytics tools, say Unica’s.

I suspect it would take too much work for volunteers though who may not have a real incentive for bottling up their expertise. So, I will be sure to talk to my colleagues at Unica and our consulting partners about this one.

11 Comments on “Why is there still no Expert System for Web Analytics?

  1. Akin,

    Interesting article, nicely written. This is also something I’ve been thinking about for a while – and while running consulting teams have always had the basics of such a system in my head (in other words while not documented – who has the time for such things – would always be there to guide a conversation with a consultant about where to look for answers).

    I agree that this would be too much work for a small group of volunteers, but built as a crowd-sourced application (think something between a Wiki, Linkedin Answers and Get Satisfaction) I’m pretty sure the global community could get together to build a searchable database of ‘what to do when X goes wrong with my campaign’.

    Something like this would be great for new folks looking for practical training, as well as a platform for ‘experts’, ‘gurus’ and the like to step in with their experiences. Could be fun as a twitter type application – people post questions and then through replies analyses are recommended.

    Now, to find the time for this…

    Cheers, James.

  2. Hello James

    Thanks much for the feedback and kind words. As professionals your team will have that expert system built into your brain. Probably only slightly more difficult for you than shifting gears in a stick shift car. .But anyone who isn’t doing this day to day would definitely have to think too hard.

    Isn’t there that design book “Don’t make me think”. While we can’t ever make thinking unnecessary we could make it easier.

    Nice to meet you on the blog!

  3. Hi Akin,

    it would take me 2 days to really comment on this post. You have touched something very important. As a student of epistemology, I have been devoting some time on WA theory and methodology, and I agree with you that there is a lot of work to be done.

    We need to be able to go beyond some kind of paradigm entertained by the blogosphere echo chamber, and really get to a point where we develop strong knowledge based on hard evidence. At least in good parts.

    This should not be that difficult, though, since we do play with tons of data.

  4. Hello Akin,
    very true. Building up some kind of knowledge base with checklists should be an integral part in a big WA project. On the one hand we can provide answers or todos for analysis questions. On the other hand the we have to keep record of changes in the website or campaigns.

    Another problem I noticed in bigger companies (at least in Germany) is, that the responsible person for website and analysis changes quite quite often. So keeping a history would improve the process.

    An important question I see is, where to locate such a knowledge tool. I think it has to be more integrated in a CMS or intranet, than just sitting inside a Web Analytics stand alone solution.

    Best regards, Markus

    • Hi Markus,

      Good to hear from good old Germany. Interesting point that you raise and makes total sense. I suppose Clicktracks once had a way to archive the previous state of pages so that you could compare before vs. current in the Overlay view. But your thinking goes more into a workflow, project, and team management system including versioning & all.

      Unica does offer a product called Marketing Operations OnDemand which comes close to this. It is an online tool for workflows, project management, digital archive with versioning, etc. It is typically used by marketing agencies that collaborate with clients on developing designs, etc. It hasn’t been promoted (nor thoroughly reviewed) under this valid use case for web analytics yet though.

      Thanks for adding the good point!

  5. Akin, it is just that easy 🙂

    I agree wholeheartedly that the WA industry as a whole has been focused more on reports and KPI’s than the true focus – measurable positve changes…and the methodologies to take.

    While being a fan and pro web analyst for over a decade now, the thought of some sort of Web Analytics AI has crossed my mind quite a few times.

    That said, I havealways landed on the same issues (as also mentioned in my blog: http://bosilytics.wordpress.com/2009/09/14/obstacles-to-preparing-data/)

    1) Preparation of the data (in your list you mention the web analytics tool, possible campaign management tool and\or paid search stats, competitive stats (compete, alexa) and good ole server log or operations management console.

    Not that I have not integrated those before, but it is not for the masses…yet. Solution = Data Warehousing + Web\Business Intelligence

    2) governance of metrics –
    Technology aside, for a group of individuals to gather together and dictate their key opjectives is still a task best performed by a consultant (strong biased here, it is what I do for the mortgage!). Once that is done, the organization as a whole has to buy into the decision tree … and adhere to the priority issues addressed.

    3) Advanced Analytics (multi-dimensional) –
    This is where the PHD’s need to step in. however, I do not think anyone wants to take the ENTIRE analyst role away, just allow them to do what they do best … advanced analytics.

    Cheers for addressing a pain point!

    • Thomas / Bosilytics — Well taken points, obviously straight from the front lines. I read the post too, thanks for the link! Looks like you also offer a solution in this area of pain: http://www.youranalyticssite.com .

      At Xchange last week, Bob Page from Yahoo (and formerly Accrue) made an interesting comment that resonated with everyone.

      “He said that web analytics is often expected to be like accounting wheras in reality it is more like statistics. And statistics means you can never say anything with certainty.” (Not that 2 accounts ever arrive at the same result on anything)

      This seemed a useful guideline for picking a piece of data that you can make sense of. Formulating a good enough theory and trying it out to see if it works.

      Another good guiding light is probably to focus energy on things you can control vs. trying to measure and measure and measure.

      Oh well, preaching to the choir.
      Thanks for sharing your experience here!

  6. Akin-

    We are 100% making this happen right now. The rules you outlined above are VERY similar to our internal modules that will provide insight and information from the data web analytic packages provide.

    You’re right – its amazing. For 15 years web business owners have had to hunt around web metric reports and deal with “chart porn” to try to find why something happened and if it’s significant.

    Just tell me WHEN something important happened and WHY it might have happened and WHAT might mean.

    That’s what Wripple will do.

    You want to help us create more modules? Everything we are doing is driving by potential user feedback, including deciding what to build.


  7. Thanks, this is a thoughtful and interesting article. As an entrepreneur I can see an opportunity to create something really useful here, integrated with a truly intuitive Web Analytics tool.


  8. Hi SEM Sensei,

    Thanks for the comment. And I agree it is inspiring and I have something coming too. Though content not tool/product.

    Interesting however was one of the other comments. A start up called Wripple is working on creating something that integrates with existing web analytics solutions. That I believe is the way to go because it is unlikely that enterprises will switch solutions only because of this factor.

Leave a Reply

Your email address will not be published. Required fields are marked *