A/B Testing With Nginx
Hone is an incredibly data-driven company. Whenever possible, we use analytics data (combined with customer feedback) to drive decision making about which features to add, modify, and remove — among other things.
We recently wanted to know how some proposed styling changes would affect user interaction rates of a widget in our web client. To accomplish this, we needed to run a few A/B tests.
Using our Nginx load-balancer, we decided to split incoming traffic in half: 50% of our visitors would be served the existing widget and the remaining 50% would be served the new widget (with the new styling/layout changes).
Simple A/B Testing Nginx Config (50%/50%)
The http server config above utilizes Nginx’s ngx_http_split_clients_module functionality to assign all incoming requests into one of n-buckets — and then redirects them to the corresponding test page.
We run A/B tests until a statistically-significant number of visitors have passed through them. For this particular widget being tested, we needed 10k visitors: ~5k going to each of our A- and B-test widgets.
What did we learn? After examining our analytics data, it was immediately clear: A much higher percentage of users interacted in the ways we wanted with the new, redesigned widget compared to the old widget. And — importantly — the higher-than-previous engagement remained steady during the following weeks and months — meaning it wasn’t merely a temporary lift.
More Advanced A/B Testing
If you’re interested in learning more, check out this article by Lawson Kurtz that details some more advanced configs and methods of A/B testing using Nginx.
Work At Hone
Interested in working at a small, ambitious startup? Check us out!