Transact’s revamped search experience feels more intuitive than ever, and customers are seeing 6% higher conversion rates.
At Atomic, we are relentlessly focused on improving our users’ search experience. After a recent UX improvement increased Atomic’s full funnel conversion by 14%, we identified an opportunity to improve our search algorithm with machine learning. Our machine learning team crunched thousands of logs of user search data to improve the relevance of search results. We then measure by how easy it is for users to find their company or payroll provider using our app. To explain how machine learning can improve search, we interviewed Dallan Quass, Atomic’s Principal Data Scientist, to learn more about how he thinks about search after years of experience with state-of-the-art algorithms.
Q: Why is the search experience hard?
A: Search is deceptively difficult. People often don’t search for a company’s official name and many companies and payroll providers have very similar names. We must also take into accoun typos, abbreviations, and acronyms. Bringing the specific company the user is looking for to the top of the search results under search-as-you-type latency limitations is challenging.
Q: How can the search experience be made better for users?
A: Whenever a user types another character when searching for a company or payroll provider, we want the one that is ultimately most likely to be selected to be ranked at the top of the results list. For example, if the user starts their search with “W,” then Workday is ultimately the most likely selection and it should be ranked highest. However, if the next letter after “w” is an “a,” then Wally’s is the most likely selection and it should be ranked highest. If the next letters are “rk,” Workday is again, the most likely selection. The “a” earlier was probably a typo and given the other conditions, we would still want Workday to be the top search result. There are two ways to make this happen: the old-fashioned way by manually testing different kinds of search algorithms until we get one that “seems to work well,” or using machine learning to identify the optimal search implementation with an approach called “learning to rank.”
Q: What is “learning to rank?”
A: “Learning to rank” uses machine learning to train a computer to assign a higher score to company A over company B whereby the end user sees and ultimately selects company A over company B. We fed the computer tens of thousands of historical searches containing all of the issues mentioned above along with the companies that the user ultimately selected. The computer then learned to rank companies that were selected more often higher in search results than those selected less often, as frequently as possible. Learning to rank is much more efficient and principled than manually trying different combinations of queries and weights until you finally find a combination that seems to give good search results most of the time.
Q: What were the results?
A: Our investment in machine learning is paying off. Our data-driven approach to search yielded three key insights about the importance of improved relevance.
1. Improved relevance increases conversion
Building off of our Search 3.0 gains, the algorithmic improvements demonstrated the following conversion improvements in A/B testing:
- +2.9% improvement in top of funnel search conversion
- +4.5% improvement in authentication attempt rate
- +5.8% improvement through the full funnel of completing a transaction
2. Improved relevance heightens the end user experience
Perhaps even more important than the improved conversion is the more intuitive feel of Transact search. Prior to our use of machine learning, companies that were selected more often appeared below companies selected less often 1/6 of the time. After our efforts, this rate dropped to 1/20 of the time. In other words, users have to scroll a lot less to find what they are looking for. Formulating it as a classification problem reduced our ranking error rate by a factor of three.
3. Improved search experience relevance accelerates discovery on coverage
Search logs are a rich source of data where users tell us unambiguously what they are looking for. This deepens our understanding of users, and our improved search analytics allow us to iterate more quickly on coverage expansion.
The “search” continues…
Transact Search isn’t the only new search experience we’ve rolled out recently. We’re excited to announce our new search within docs, which makes now the best time to be searching with Atomic.
