I was lead UX Designer on the project, designing the assets to test. I was working amongst a team of other UX researchers.
We wanted to introduce recent journeys in the signed out state to improve
overall navigation / hand offs.
It's currently only available to logged in users, which there is a very small amount of users that create an account (less than 2%). Previous tests have showed how the "Recent Journey's" feature improves conversions, so making that feature only available to 2% of users greatly decreases the impact of its value.
We expect the recent journeys will increase overall usage in returning visitors without negatively impacting the performance of the new journey planner.
Key metrics we were tracking:
- Go button clicks
- Form interaction
- Hand off
What did we test?
We tried different combinations of the journey planner and recent journey modules, and also tested the removal of the “More options” widget in the planner.
Results and observations
- The presence of the Recent Journeys module diverted attention from the Go Button, but overall engagement with the Journey Planner/Recent Journeys section increased.
- All challengers were very successful at bringing more visitors to the Results Page.
- The Recent Journeys without More Options challenger brought the most visitors to the Results Page and was the top performer in terms of hand-off rates.
- This could produce an additional £1,397,000 in revenue over the course of the year
ROI based on 50,435 return visitors a month and average hand-off value of £40.
The challenger adding the recent journeys module while removing the “More options” widget showed the most promise.
The presence of the Recent Journeys module led to significantly fewer clicks on the Go button. But when we combine Go Button clicks with Recent Journeys module clicks, the Recent Journeys without More Options challenger saw significantly higher engagement rates.
All of the challengers brought more visitors to the Results page, and the Recent Journeys without More Options creative was most successful at bringing visitors there.
Fewer visitors arrived at the Results Page by clicking the Go Button when the Recent Journeys module was present, but visitors coming from Recent Journeys (and other pages) led to higher conversion to the Results Page overall.
Although removing the more options link proved most profitable, it’s not ideal to remove overall customisation of the journey
However, given the percentages of people tracked using the options, their voice will most likely be muted by the overall majority who has no use for it.
A good compromise would be to push with this creative with a less prominent more options section as the winner.