Last week I attended a session of Toronto Data Mining Forum that was hosted by SAS.
The Data Mining Forum is an opportunity for us to explore Data Mining related issues, share ideas and discuss industry standards and trends with your peers. Each session explores current trends, new technology and Data Mining techniques through presentations and discussions.
Last week session was an interesting mix of presentations on various topics in data mining. I was particularly interested in one of the topics – Recommender System In Retail that have been presented by Pramod Dogra and Iqbal Habib from Shoppers Drug Mart (SDM).
About Shoppers Drug Mart
Shoppers Drug Mart is Canada’s largest retail pharmacy chain and has more than 1,253 stores operating under the names Shoppers Drug Mart in nine provinces and two territories and Pharmaprix in Quebec. Shoppers Drug Mart is a unique and independent division of Loblaw Companies Limited. Shoppers Drug Mart has own exclusive loyalty card program – Shoppers Optimum. Shoppers’ Optimum members receive 10 points for every dollar spent at the store.
Shift to Customer-Centricity
The non-personalized channels and mass offers –billboard, radio, Newspaper, TV and direct mail flyers – had traditionally the strongest presence in the SDM media mix. The disadvantage of traditional media advertising that success is difficult to measure. In comparisons, addressable digital communication has traceable ROI. Merchants understand that future of retail advertising is a personalized communication with the focus on customer needs.
SDM is continuously increasing their attention to the digital channel and direct marketing. The shift to customer centricity combines targeted communications based on propensity to purchase products and personalization based on relevancy of customer interaction. Personalized messages significantly increase response rate and provide incremental revenue.
Now not only well-known global brands like Amazon and Netflix apply personalization but also Canadian retail brands like Metro, President’s Choice and Shoppers Drug Mart.
Recommender System and Personalization
We need a recommender system to find out which items are the most relevant to users.
The well-known examples of the websites with recommender systems are:
- Amazon
- Hulu
- Netflix
SDM uses recommender system to make decisions, which offers to be included in personalized direct mail or email messages. Each personalized communication has limited number of offers with promotional coupons. The job of the recommender system is to calculate the most relevant offers for each recipient.
The ultimate goal is to build personalization with 100% variability across all features (products, and types of offers). The approach leverages customer segments and predictions to drive offer type pairing.
The model takes into consideration 5 stages of customer’s lifecycle:
- Acquisition
- Onboard
- Grow
- Maintain
- Decline
Customers at each stage will receive different types of offers. Our knowledge about customers’ preferences is much more limited during the acquisition stage. We can use ‘look alike’ models for personalization and cannot rely on explicit preferences of past purchase history.
Shoppers Drug Mart explores two approaches to recommender systems: Content-based and Collaborative Filtering.
Content-Based Recommendation Approach
The approach uses similarity of purchased items to find common characteristics like product category, brand or style and recommend similar products for the customer. For instance, the recommender system may recommend baby products based on previously purchased baby products, or predict a cosmetic brand of choice based on the similar purchases in the past.
The real system has much more complex algorithms to predict the best matching product offer and which product characteristics to use for prediction. The content-based approach has the following advantages:
- No need for data on other users
- Able to recommend to users with unique tastes
- Able to recommend new & unpopular items
- Able to provide explanations and relate the recommendation to user preferences
The main limitation of the content-based system is that it cannot give good recommendations if the system does not have enough information to distinguish items based on the past customer purchases and user profile. It is not effective at the early stages of the customer lifecycle when we don’t have or have very little information about customer preferences.
The content-based approach may provide value in web personalization when we capture browsing history and may find about customer interest by click stream.
The content-based recommendation systems are very often used in combination with other methods such as collaborative filtering.
Collaborative Filtering
The Collaborative Filtering focuses on the relationship between users and items. The recommendations are based on a model of previous user behavior. The idea is that if we find that a group of people have similar tastes then we may provide a recommendation to new members based on preferences and purchase history of the existing members. We refer to this approach as “wisdom of crowds”.
Here is a simplistic example of the collaborative filtering rule:
If Alice and Bob like X and Alice like Y then Bob is more likely to like Y
The most known application of Collaborative Filtering is the book recommendation system by Amazon.
The collaborative filtering system uses two types of algorithms:
- Non-probabilistic
- User-based nearest neighbour – explores user to user similarities
- Item-based nearest neighbour – explores item-to-item similarities
- Dimension reduction – considers user preferences in multiple aspects of item instead of a single dimension such as user ratings
- Probabilistic
- Bayesian models
With the probabilistic approach, we are looking for products and product groups that have above average odds to be purchased based on the predictive model.
Measurement
It is essential to measure the effectiveness of the recommender system for both successful and wrong predictions. The measurement includes evaluation of predicted ratings, mean average error and root-mean-square errors (RMSE) and success rate of the Top-N predictions. The prediction of Top-N recommendation is one of the most common use cases. For instance, we often come across top 10 recommended books, top ten movies that we may like or top 5 recommended news.
The SDM recommender system has to predict the Top-N coupons in the direct mail or email communication. Web sites have also limited number of spots to display recommended items. Mobile websites and applications require even smaller numbers of items to be selected as the Top-N to display on a small screen. Successful predictions of Top-N items increase the response rate and conversion to purchase, which directly impact ROI.
Establishing measurement of the response rate of users who receive recommended items in their communications and comparing with a control group, provide us the depth of incremental contribution to sales and revenue by the recommender system.
Challenges
There are few problems with collaborative filtering method:
- Cold Start: new users have no history, and new items have no ratings
- Sparsity: if there are many items to be recommended, it is hard to find users that have rated or purchased the same items.
- First Rater: Cannot recommend an item that has not been previously rated
- Popularity Bias: The tendency for popular items to be recommended more frequently. As the result, the system cannot recommend items to someone with unique tastes.
The content-based method also has own challenges and limitations:
- To match item with user we need to extract meaningful features or terms from the item content to be used for predictions
- We also need to choose a learning algorithm that is able to learn the user tastes based on the content and make recommendations based on the user tastes
- Unable to recommend to new users without users profile and apply quality judgments of other users for predictions
Hybrid methods help solve some of the limitations of both methods.
Working together Collaborative Filtering with Content-based method can handle the new item problem and apply demographic to resolve the new user problem.
The new methodologies are still in earlier stages for SDM. The SDM team is working on integrating new models with the current recommender system and enhancing the contribution of Life Time Value models.
SDM presenter admitted that sparsity and size of data are the main challenges to building a successful recommendation system
Conclusion
Successfully implemented recommender systems provide benefits for customers and help them to find right products. Equally, the systems benefit businesses by generating more sales. The outcome is win-win for retailers and consumers.
Highly relevant offers may not only drive traffic to online store but increase traffic to the physical store with the additional opportunity to cross-sell and upsell customers.
