A/B testing cash payment button - Case study
1. The Problem
The app being A/B tested is an application that works on a Kiosk. This kiosk is placed in food courts, catering areas and/or specific restaurants or shops. It allows people to order the food they want to eat on-site or take-away.
Conversion rates were not optimal at this stage and a move needed to be made, so we decided to gather data instead of just guessing, we wanted to know.
2. The research
Interaction recordings made us realise that some customers were abandoning the purchase on the checkout screen. So we decided to contact registered users that abandoned the cart without checking out sending an SMS with a link to a survey. This survey completion would grant them access to a promo code that issued a $2 discount on the next purchase.
2.1. Survey
Using Google forms we created a survey to try to understand why users were abandoning.
2.1.1. Sample
In a week, we gathered 17 responses from registered users.
2.1.2. Results
- 58.8% of the users finds the price fair
- 68.8% actually wanted to order, against 31,3 % that were only experimenting
About payment only 11 respondents provided feedback:
- 54.5% Don’t want to use their credit card
- 27.3% Don’t have a credit card
- 18.2% Could not complete payment
All these 11 respondents fall into a category that could be solved with a cash payment option.
2.2. Hypothesis
A cash payment button on the checkout increases conversion rate.
2.3. Variables
Dependent variable: Number of checked out carts.
Independent variable: Integration of cash payment.
2.4. Methodology
2.4.1. A/B testing
Using optimizely we inserted a cash button on the checkout view to 50% of the users.
2.4.2. Duration of the experiment
Using the VWO A/B test duration calculator we found the number of days to run the experiment.
After 15 days we checked the results to find a 18% increase in conversion, which came close to the desired 20% increase.
