From a product manager’s question to the real cause, the same day

The problem

A product manager for the retailer’s mobile app asked whether her conversion rate was reading low because push-notification traffic was inflating visits. It is a good hypothesis. It is also the kind that costs an analyst three days and usually comes back “inconclusive.”

The honest answer to the question as asked was “not separable.” Push traffic genuinely is mis-attributed in the retailer’s analytics, with nearly all of it landing in the direct channel. An analyst starting from her framing would have spent the three days and returned exactly that. The theory was untestable in the form it arrived in.

How Augmeta helped

The Operator tested the mechanism instead of the attribution, with two independent checks, both fast. First, the app’s share of all visits had moved by less than a tenth of a point week over week. At that weight, no mix effect of the claimed size was even available. Second, site-wide top-of-funnel conversion had fallen by almost exactly the same amount as web-only conversion, a gap of a hundredth of a point, when an app deteriorating faster than the web would have opened a visible one. The mechanism was not there.

Then it found the real one. Desktop accounted for 88% of the decline, and it had been hit twice: its conversion rate had slipped, and its share of web visits had slipped by about a point on a visit count down nearly a tenth. The year-over-year picture inverted as well. Desktop conversion was actually up close to a full point on the year while its share of visits had fallen from roughly 23% to 17%. Traffic reallocating from the best-converting surface to the worst was swamping a genuine rate gain.

Her stated premise also failed on the way through. She had assumed orders and visits had kept pace with each other. With conversion down about 3% on visits down about 3.5%, app orders had in fact fallen by roughly 6%.

What changed

The product manager got an answer the same day, and it redirected her from a dead end to a live one: the desktop mix shift, not the app’s attribution, was where the number was being lost.

The baseline this is measured against belongs to the retailer. Earlier in the year, a privacy-consent change had suppressed about 15% of daily web traffic for six weeks. Orders held steady, so conversion and click-through improved on every dashboard, the ratio illusion of a stable numerator over a collapsed denominator. The retailer’s own team diagnosed it, across five functions, in about a month. That is the honest before-picture, and the work was theirs.

The steady state since is a week to two days from question to resolution. On the daily revenue exposure of a KPI this size, the five days removed are worth roughly $200K for each investigation that turns out to matter, with the assumptions stated: exposure holds while the issue persists, and about one investigation in five is consequential. Both refuting checks are single queries against the retailer’s own analytics, and anyone with access can repeat them.