I'd been treating "drop the non-purchasers first" and "winsorize the top of the distribution" as basically the same cleanup step applied to a revenue metric. They aren't. Once I actually looked at what each one was protecting against, they turned out to be answering different questions — and conflating them was quietly breaking the conditional distribution I actually cared about.
The Muddled Starting Point
The scattered version of what I'd been poking at: whether to decompose conversion rate and expected revenue-given-conversion separately, why looking at the distribution mattered at all, how revenue metrics should be treated, and whether count data (orders, clicks, likes) needed the same treatment. None of it was written down as one coherent argument — just several related questions I hadn't yet separated from each other.
Why "Ignore Zero" Has to Happen Before Winsorization
Here's the concrete failure that "ignore zero" is protecting against. Suppose 96% of users are non-purchasers (revenue exactly zero) and the remaining 4% are purchasers whose revenue is roughly normally distributed. That's a bimodal distribution — a huge spike at zero, plus a separate normal-shaped hump sitting off to the side. If you winsorize the whole thing at the 99th percentile without separating out the zeros first, you end up clipping a full quarter of that 4% purchaser group, since the zero spike eats up 96 percentage points of the distribution before you even reach the purchasers. The top 25% of the purchaser distribution — a normal curve — gets flattened at the clip point. If the revenue from exactly those high-value purchasers is the part you actually care about, that's not a minor rounding effect. It's damage to the part of the distribution the metric exists to measure.
The reason to ignore zero, then, isn't just "zeros are annoying" — it's that once you set them aside, the remaining data can be checked against a specific distributional shape. Payment amounts, for instance, often look approximately normal once you exclude zeros and take a log — the same way income or stock prices are often modeled as log-normal. And after excluding zeros, if what you're counting is a count rather than an amount, you can check it against a zero-truncated Poisson or negative binomial instead. That diagnostic step is especially useful for count-type behavioral data — clicks, likes, saves — when you're trying to model user behavior rather than just clean up an outlier.
Not a Hurdle Model — Two Separate Corrections
It would be easy to read the conversion-rate / expected-revenue-given-conversion split as a hurdle or two-part model — decomposing the metric into (did they convert) times (how much did they spend, given that they did). That's not what's going on here. Winsorization's job is narrower: remove outliers from the conditional distribution — the purchaser-only distribution — and reduce its variance. The point of doing that, in this context, is specifically to get a tighter, less noisy estimator for A/B testing, not to build a structural model of the purchase decision itself.
Why Look at the Distribution At All?
The point of examining the distribution in the first place is just to see what the data actually looks like — whether anything about its shape is unusual. Going back to the bimodal example: if the data really is "0–96% one normal shape, 96–100% a second, separate normal shape," then straightforward winsorization at the 99th percentile isn't obviously the right tool — the two components need to be reasoned about separately, not clipped as if they were one continuous tail. And moving the threshold up to 99.9% doesn't automatically fix that either; it needs to be checked against the actual shape of the data, not assumed to solve the problem just because it's a higher percentile.
Why Revenue and Count Data Get Split
Count data — order counts, click counts — gets a different treatment than revenue, because its distributional shape is genuinely different, which points toward log transforms or Poisson/negative-binomial modeling rather than winsorization as the default fix.
The underlying reason is that count actions are bounded by how much effort a user is willing to spend clicking or tapping — there's a practical ceiling on how many times someone favorites or clicks something in a session. Revenue doesn't have that same ceiling: a single purchase event can vary enormously in size, whether it's a small everyday purchase or something two orders of magnitude larger. That's a structurally different kind of variability than "how many times did this person click," and it's why the two data types don't get the same fix.
What's Still Open
One thing I haven't worked through yet: whether applying the same winsorization threshold to both arms of an A/B test (pooled) versus computing it separately per arm creates its own problem. It's a real question — but a separate one from what's covered here. For now, the priority is getting the distribution diagnostics right: separating zero from the purchaser distribution, checking what shape the purchaser distribution actually takes, and picking between a 99% and 99.9% clip based on that shape rather than by default.
The Practical Habit This Left Me With
Before reaching for winsorization on a revenue metric, look at the distribution with zeros removed first — not to skip a step, but because a clip threshold chosen without seeing the actual shape of the purchaser distribution can quietly damage exactly the high-value tail the metric is supposed to capture. "Ignore zero" isn't a warm-up step before winsorization; it's what makes winsorization's job well-defined in the first place.