Online Grocery In Beta

Here at Walmart International we’ve been on a 18 month journey to refresh and improve the customer experience for our UK online grocery shopping platform, ASDA grocery home shopping. We started this initiative with the rebuilding of the customer experience for the checkout. We’ve since followed this with a much improved registration and sign-in process. Both projects providing considerable benefit to our users and to our platform. Whilst we continued to iterate on further improvements to the new sign-in and checkout flow, we have also been working on developing a new shopping experience for our users. This is the area where our users spend most of their time on the site adding their grocery items to the cart, it includes experiences such as search, homepage, favorites, taxonomy, shopping lists and our newly developed recipes section.

One of the key learnings of the work we did on the checkout was that we turned on the experiment before the experience was at a level of quality that was acceptable for our users. In the early stages of testing we experienced a number of false starts where we needed to turn off the experiment, fix problematic issues for our customers and turn the experience back on. Although we eventually got our KPIs and customer feedback to show positive results we lost a considerable amount of time and resource and frustrated some of our customers unnecessarily. We needed to rethink our approach…

As the initiative involved the creation of a new tech-stack, large experience changes to the taxonomy and new design system we need a way to get real world feedback from users outside of the lab environment which can sometimes come with biases whilst mitigating as much risk as possible. We also wanted to get feedback from users as early on in the development process as possible so that we had more time to adjust if necessary.

Our approach

Similar to the checkout redesign project we started with a private beta that was made available to our colleagues. Here the new app was very much in the alpha stage. Major features were still under development and we still had a large number of bugs to squash. Once we had our core feature set developed and most of the major bugs resolved we opened up the new experience to a select number of our users as a public beta.

For the public beta we targeted our most loyal customers, users who had completed 15+ shops with us in the last 6 months. We used our experimentation platform to randomly select 1% of our most loyal users. Initially we wanted to keep usage of the new experience fairly low to keep the risk low, but we up-weighted this percentage as we became more confident in the feedback we were getting.

We used our most loyal customer for 2 reasons. Firstly they are the most vocal and thus our chances of receiving feedback was higher. Secondly online grocery is largely a retention lead business with customers using the service every week. This is a much higher frequency than most eCommerce businesses so it was incredibly important to us that our most loyal customers had an early opportunity to voice their feedback so we could ensure we had time to adjust.

Those included in the experiment would see a button in the header or slide out navigation depending on the width of their device. Inviting them to try our ‘new look’. We made it clear to our user up front that we were interested in getting their feedback and that the site was still in development. We also let our users know that they could get back to the old site if they needed to using the same button in the header/navigation. Once in the new experience we had a clear feedback button always in view on any page to make it easy for the user to let us know their thoughts (opt-in survey). Also if the user did choose to revert back to the existing site after that action was completed and the user was back on the old site we asked them for their feedback as to why they chose to leave (opt-out survey). The feedback forms we used were very simple normally just 2 questions to encourage completion.

Finally we added some basic analytics tracking so that we could compare the qualitative (the customers written feedback) to the quantitive (the actions the users where taking). We ran the beta for 6 weeks analyzing the usage and feedback daily with our team.

Demo of a user moving from the old site to the new one and then back again

What we learn’t

Firstly users on the new experience were over a 100 times more likely to give feedback than we typically see on the old experience. This is because we selected a vocal user base, made giving feedback very easy with a simple 2 question feedback form and the fact it was was a big change to the experience itself.

We were able to the use the percentage of users who decided to exit the new experience back to the old site as an indicator of when the new experience was strong enough that we could start the AB test. The AB test wouldn’t have the option of users to return to the old site so we need to be confident in the quality of the experience before starting this phase. From the chart below you can see we initially started out at ~15% of users who went to the new site deciding to leave, rising to ~25% as we ran into performance issues. After a round of key performance and experience improvements we saw a significant decrease to ~5% which gave us the green light to start the experimentation phase of the project

The survey shown to the user as they exited back to the old site (opt-out survey) proved an incredibly valuable data source in order to understand where the opportunities lay in order to improve. Interestingly 6% of users leaving the new experience gave positive feedback and a further 9% of users left because they didn’t have enough time to explore the new experience and intended to come back later.

“Short of time today, will look when not so pressed with time.”

Our hypothesis was that bugs and features we hadn’t built yet would make up the majority of the reasons that users would choose to leave. However the 2 key area’s raised by users where the apps performance and feedback on the change in experience. This data point allowed us to change our approach, from focusing on feature development to fixing performance issues. The user experience feedback comments mainly focused on either users preferring the old site or users having difficult navigating the new taxonomy.

“The old site is much easier to navigate.”

However, when we checked this against our analytical data we saw a different trend. The new experience was showing a significant increase in cart additions from the taxonomy section compared to the old experience. What we learn’t was that users that were struggling the most where users that had been customers of ours the longest and thus where very used to shopping the old site and would take longer to adapt to the new changes. However if we could help our users transition easier there was value in customers using the new experience. This showed us the importance at not looking at qualitative and quantitate in isolation. Using both sources we were able to make changes to the experience to make the transition smoother for our most loyal users without losing the value that the new site offered.

Finally the public beta served two important internal benefits. Having an exit route was incredibly beneficial for both our users and the progress of our initiative. It enabled us keep the beta running and collect valuable data/feedback whilst us migrating against the risk of users being blocked and sales being impacted. Also it gave our stakeholders confidence in the initiative as we were able to show much earlier in the project lifecycle that there was value in the changes we were making and we could adjust quicker to the feedback our users were giving us. Support from business stakeholders is essential when completing large customer experience changes as often the development and experiment phase can take much longer than expected for both the internal team to fix issues and respond to customer feedback and for users to get accustomed to the new layout and design.

Into the future

The introduction of a public beta before the experimentation phased helped us vastly in achieving our project goals. We found it a useful tool in allowing us to shape the platform for the benefit of our users. We were able to get feedback much quicker and we were able to go into the experiment phase with much more confidence in our new product. We see this approach being useful in many more of our endeavors so much so we are considering building an early access / beta testing community. Here a user on our platform would be able to opt-in to the program. We could then use our experimentation platform to target these user to give them early access to new features that are in the later stages of development to get feedback earlier, make smarter decisions and ultimately make the experience much richer for our users.

Example approach for user to opt-in to the beta program

Online Grocery In Beta was originally published in Walmart Global Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Article Link: Online Grocery In Beta. Here at Walmart International we’ve… | by Richard Doyle | Walmart Global Tech Blog | Medium