For a while I kept a running side-project going: use the posterior expected value of an experiment, instead of a p-value, to decide whether to roll it out. I never wrote it up cleanly, because I kept bolting extra pieces onto it — development cost, risk attitude, a second metric — without ever stating what problem each piece was solving. Naming those pieces separately, after the fact, is what finally made the whole thing make sense.
The Muddled Starting Point
The scattered version of what I'd actually been poking at, roughly: comparing transaction revenue against development cost, folding in something about risk attitude, and at some point trying to make the whole thing work with two metrics instead of one. None of it was written down as a coherent framework — it was a handful of half-finished threads that all touched the same underlying idea, without me ever separating what each thread was actually for.
A Decision Rule, Not a Better Estimate
The first thing to pin down: posterior expected value here is a decision rule for judging whether an experiment wins or loses — not an attempt to produce a better point estimate of the effect size. Concretely: you start with a prior distribution over the return of a change (say, on payment revenue per user), and once the experiment produces new data, you combine that prior with the new data to get a posterior distribution over the same return. If the expected value of that posterior is greater than zero, you roll out.
The more data you collect, the more the posterior leans on that new data rather than the prior; the less data you have, the more it leans back on the prior. That's the part that actually resolves the muddle: in a standard frequentist test, a small sample just means low power — a straightforward problem, with no built-in adjustment for it. Here, a small sample doesn't just weaken your evidence, it also automatically shifts weight back toward the prior, which softens how much you trust a shaky, low-sample estimate in the first place. Power loss doesn't get compensated for, exactly, but the framework has a built-in mechanism for not overreacting to noisy small-sample results the way a raw point estimate would.
Where Development Cost Enters
Comparing transaction revenue against development cost isn't a statistical-significance question at all — it's an expected net value question. The rule becomes: adopt the change if its expected revenue clears its development cost, not just if its effect is statistically distinguishable from zero.
Running this comparison myself made one thing obvious: relative to the revenue scale of an experiment at a company of meaningful size, the development cost side of that inequality can be a rounding error. Which is a useful thing to actually see in numbers, rather than assume — it changes how conservative you need to be about shipping marginal wins, since the cost side of the ledger is often smaller than intuition suggests.
Encoding Risk Attitude in the Loss Function
"Attitude toward risk" isn't a separate add-on bolted onto the expected-value rule — it belongs inside the loss function itself, as a risk-averse penalty rather than a plain expected-value maximization. The more risk-averse the loss function is, the more it's willing to trade away some expected value in exchange for lower variance across the portfolio of experiments you're running — fewer big wins, but also fewer big losses. That's a knob you set deliberately, not a side effect of anything else in the rule.
What Two Metrics Combine to Look Like
The two-metric version I'd been experimenting with mostly involved cases where both metrics were revenue-shaped — for instance, per-user payment revenue and per-user advertising revenue — which tend to be positively correlated rather than in tension with each other. How you combine them depends entirely on what each one actually implies downstream.
One approach is a weighted-sum utility, where the weights come from the actual economics. If the platform's revenue take is, say, 15% of transaction value, a natural combination weights the two metrics roughly 0.15-to-1 rather than treating them as equally important on their raw scales — the weighting isn't arbitrary, it's read directly off the revenue model.
The other approach drops the weighted sum entirely and treats it as a break-even condition instead: require both metrics to individually clear zero (or clear their own break-even point) rather than folding them into a single number. That version is less about maximizing a combined utility and more about a Pareto-style gate — you only ship if neither metric is being sacrificed for the other.
The Practical Habit This Left Me With
Every one of these threads — the decision rule itself, development cost, risk attitude, two metrics — answers a different question, and none of them are optional extensions of "just use posterior expected value." The posterior expected value rule replaces a significance test as the win/loss criterion. Development cost turns that into a net-value comparison. Risk attitude lives in the loss function, not in the rule itself. And combining two metrics is really a modeling choice about whether they trade off or move together, made explicit rather than assumed. Naming which question each piece is actually answering, before trying to combine them, would have saved me from treating the whole thing as one unfinished idea instead of four separate, already-workable ones.