There’s a philosophical statistics debate in the optimization world: Bayesian vs Frequentist.
This is not a new debate; Thomas Bayes wrote “An Essay towards solving a Problem in the Doctrine of Chances” in 1763, and it’s been an academic argument ever since.
Recently, the issue has become relevant in the CRO world – especially with the announcement that VWO will be using Bayesian decisions (Google Experiments also uses Thompson sampling, which is informed by a Bayesian perspective).
So what the hell does Bayesian statistics mean for a/b testing? First, let’s summarize Bayesian and Frequentist approaches, and what the difference between them is.