Tracing an 85 percent KPI drop to broken tracking

The problem

Clicks on the site’s product recommendations appeared to be down 85% year over year. The number had been carried through several weekly business reviews on a slide of its own, and the owner of the recommendation programme was being asked what had gone wrong with personalization.

Attributed revenue from the same placements told a different story: it was down about 10%. And the conversion rate those two numbers implied had more than tripled, which no change in shopper behaviour could produce. Clicks, revenue and conversion cannot all be true at once unless something in the measurement is broken. But the three facts lived on three different slides, owned by three different people, and nobody had read them together.

How Augmeta helped

The Operator declined to explain the drop until it had tested whether the drop was real. It compared recommendation exposure, clicks, attributed revenue and implied conversion for the same period. Exposure was flat. Revenue was close to flat. Conversion was impossible. That pattern rules in a click-capture problem and rules out relevance or exposure before a single merchandising hypothesis needs entertaining.

The cause was a tracking variable that had stopped firing on recommendation interactions. It stayed broken while the fix was specified and shipped, and the Operator held the thread across that gap rather than assuming it fixed: it re-checked the first clean week of data and found recorded clicks up 209% against the trailing period. That was a recovery in measurement, and it was reported as one.

The convenient conclusion at that point was “measurement artifact, no business impact.” The Operator did not draw it. In the same clean week, attributed revenue was still down by roughly a quarter against the trailing period, and it had not recovered with the clicks. Two pieces of evidence that disagree were reported unreconciled, and the revenue decline was carried forward as its own open item.

Then it happened again, differently. A couple of months later every web placement stepped down by about two-thirds overnight, while the mobile app barely moved. A web-only signature. The investigation returned five confirmed hypotheses, one refuted and two explicitly inconclusive, and named the cause: a site-wide front-end framework release the previous night, recorded in the retailer’s own change-management system. Nobody had connected the release to the metric.

What changed

Several weeks of reporting were correctly discounted instead of acted on. The redesign that an 85% decline would reasonably have triggered never started, and the second break was attributed the day it appeared instead of becoming another multi-week hunt. The remaining revenue question stayed visible in every review until it was answered, instead of disappearing into a premature all-clear.

A KPI that is wrong and believed is more expensive than one that is missing. This is what owning a KPI takes: checking the evidence before explaining it, connecting it across systems, saying plainly when two facts do not agree, and staying with the work after the first answer.