Speaker
Description
Online experimentation is a way of life for companies involved in information technology and e-commerce. These experiments allocate visitors to a website to different experimental conditions to identify conditions that optimize important performance metrics. Most online experiments are simple two-group comparisons with complete randomization. However, there is great potential for improvement from implementing multi-factor experiments and for accelerating exploitation of results by employing bandit allocation, in which an increasing fraction of traffic is directed toward successful variants. The natural implementation of this idea in a multi-factor experiment is to divert traffic to better levels of the factors that are the first to stand out as active. Logistical considerations may limit the number of variants that can be made available, so that full factorials are not possible. It is then appealing to use screening experiments (e.g. Plackett-Burman designs) to identify important factors. When bandit allocation is used with a screening experiment, declaring an active factor will reduce the number of factor combinations that get continuing use and may result in a singular design. We present here some simple, yet efficient, methods for online augmentation of the factor combinations to enhance the ability to identify additional active factors after one or more are shifted to bandit allocation.
| Type of presentation | Talk |
|---|---|
| Classification | Mainly methodology |
| Keywords | A/B Testing, Screening Experiments, Bandit Allocation |