I’m a professional data + product leader trained in comp + info science. I code, write, take photos for fun. https://wilsonwong.co

In a previous 3-part series, we discussed the main hurdles limiting the value that businesses get from data. In particular, we highlighted the importance of the timing of investment into establishing data infrastructure, recruiting data professionals, etc. The reasons are clear. Any significant returns on investment that you make on data are likely to come from their use in conjunction with AI techniques such as the various ML algorithms in your products. This is when your Data Science/ML teams come in, to explore the data, build the ML models, and package them up into what I call AI services that your products can leverage. …


There has been a lot of talk in the recent years about the use of AI capabilities or machine-learned solutions to improve digital products or services and user experience. If we can cut through the hype and have the necessary building blocks in place as covered in Four Hurdles To Creating Value From Data, there is legitimate value to be had from the use of data and AI to make better products in many verticals. For tech companies that are founded on strong engineering practices and find themselves knee deep in data such as Uber, Twitter and Tesla, the use of AI techniques to extract value is likely to be already part of their culture. …


Note: This article was written and originally published in October 2016. I have reviewed and refreshed the content to reflect changes with the tools used in this article in 2020. The source code and data used and mentioned in this article are available from https://github.com/wyswilson/searchquality-in-practice

Good search is about ensuring that users find what they need. It can be something as simple as the contact details of a restaurant that someone needs to make a booking to something more life-defining such as exploring for new jobs. …


In the first and second part of this 3-part article, we covered the importance of knowledge about the different types of data and the roles that they play in data solutions. Equally as important is ensuring that your business invests in the people who have such knowledge and know what to do with that knowledge in building the right data infrastructure and assets for data professionals to work with. We also discussed the importance of ensuring that your products are ready with the right data and that your budget permits you to hire the right people to work on the data and not because it is simply cost effective. …


This is the second part of a 3-part article on the four hurdles to creating value from data. In the first part, we gave an overview of the four hurdles. We posited that the main challenges in extracting value from investments into all things data boil down to who (and to a lesser extent what) the businesses invest in and the order in which they happen. In this second part, we will discuss in detail the first two hurdles (of the four in total) that organisations need to overcome in order to get data working for them:

  • Lack of appreciation of the different types of data and what each…


Data provide organisations the new platform in the 21st century and beyond for innovation of products and processes — there is no arguing that. From identifying bottlenecks in departmental processes and reasons behind marketing campaigns under performing, to using data to inform and improve products and services in ways that were not previously possible, everyone knows data is important. However, that is where the good vibes end. Knowing that the future of your organisations rest on data versus being fully aware of what it entails are two very different stories.

Image for post
Image for post
Photo by Drew Beamer on Unsplash

When we consider reports such as this, this and many others, they tell us exactly the same thing, which is that investment into all things data have been increasingly steadily — so there’s nothing news worthy about that. Instead what is troubling and something that we should be paying more attention to is that the same businesses that have invested into data are not getting the most bang for their buck. More respondents than not in those surveys shared that they face challenges translating their investments in data into business outcomes. Do you want to make a guess as to what the main hurdles might be? It is not technology if that is what you are thinking. …


A previous post on retrieval, precision and recall discussed the distinction between the three important concepts in search. In another post, we described the two main approaches for retrieval, and saw that, for faceted search, retrieval with high precision is the foundation for a good search experience. In other words, if we look at the results in its entirety as a set and not be distracted by the ranking, how relevant is each and every single result. A highly precise retrieval contains very little to no irrelevant results. …


We have spoken about the fact that relevance is subjective. However, this does not mean that lines need not be drawn. In the area of search, there are generally two approaches of bringing results back given a user query.

The first one, which partitions the entire collection into relevant versus irrelevant documents, is known as set retrieval. If the system subsequently decides to order the relevant documents that it returns to you, it should be for the purpose of distinguishing the more relevant results from the less relevant ones.

“relevance ranking should help distinguish more relevant results from less relevant results, rather than distinguishing relevant results from irrelevant results”…


We make decisions on a daily basis, from something as innocent as figuring out what to have for lunch to deciding on whether to go for that last bid or not in a property auction. In business settings, sub-optimal decision making can lead to bad experiences for millions.

Image for post
Image for post
Photo by James Lee on Unsplash

It is worth emphasising at this stage that the core of the matter is not about avoiding mistakes at all cost or criticising incidents after the fact. Rather, sound decision making is about preventing the kind of undesirable outcomes that would have been picked up and avoided through (1) experience and subject matter expertise, (2) collective intelligence and (3) structured thinking. The reality however can be harsh. …


And it is especially true in the case of online search products. In a previous article, we discussed why figuring out and using proper metrics that are aligned with customer outcomes is important. In this article, we explore the traits of metrics which are fit for purpose. I will explain why not all metrics are created the same and describe some of the challenges in coming up with the right metrics to measure and test the performance of online search products.

Image for post
Image for post
Photo by Luke Chesser on Unsplash

Pick metrics that are fit for purpose, not what are easy to come by

There is a concept known as measurement inversion by Doug Hubbard, which is used to refer to organisations’ tendency to stick to the things which are immediately measurable. One of the main reason for this is things that really matter can often be perceived as harder to quantify and require thinking and investment from the business. For instance, almost all companies strive for customer satisfaction and loyalty for their products. However, these macro measures are not the easiest to pin down. …

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store