Ranking Models: NDCG: in Towards Data Science

Used this in practice ...  Good to see this overview for use. 

Demystifying NDCG        in Towards Data Science by Aparna Dhinakaran   

How to best use this important metric for monitoring ranking models

Ranking models underpin many aspects of modern digital life, from search results to music recommendations. Anyone who has built a recommendation system understands the many challenges that come from developing and evaluating ranking models to serve their customers.

While these challenges start in data preparation and model training and continue through model development and model deployment, often what tends to give data scientists and machine learning engineers the most trouble is maintaining their ranking models in production. It is notoriously difficult to maintain models in production because of how these models are constantly changing as they adapt to dynamic environments.

In order to break down how to monitor normalized discounted cumulative gain (NDCG) for ranking models in production, this post covers:

What is NDCG and where is it used?

The intuition behind NDCG

What is NDCG@K?

How does NDCG compare to other metrics?

How is NDCG used in model monitoring?

After tackling these main questions, your team will be able to achieve real time monitoring and root cause analysis using NGCG for ranking models in production.

What Is NDCG and Where Is It Used?

Normalized discounted cumulative gain is a measure of ranking quality. ML teams often use NDCG to evaluate the performance of a search engine, recommendation, or other information retrieval system. Search engines are popular for companies that have applications which directly interact with customers, like Alphabet, Amazon, Etsy, Netflix, and Spotify — just to name a few.

The value of NDCG is determined by comparing the relevance of the items returned by the search engine to the relevance of the item that a hypothetical “ideal” search engine would return. For example, if you search “Hero” on a popular music streaming app, you might get 10+ results with the word “Hero” in either the song, artist, or album.

The relevance of each song or artist is represented by a score (also known as a “grade”) that is assigned to the search query. The scores of these recommendations are then discounted based on their position in the search results — did they get recommended first or last? The discounted scores are then cumulated and divided by the maximum possible discounted score, which is the discounted score that would be obtained if the search engine returned the documents in the order of their true relevance.

If a user wants the song “My Hero” by Foo Fighters, for example, the closer that song is to the top for the recommendation the better the search will be for that user. Ultimately, the relative order of returned results or recommendations is important for customer satisfaction.  .... '   (more below at link) 

Mining the Wikipedia for Influential World Universities

In Technology Review: Interesting way to look at existing data.    An alternative approach to ranking. The subject attracts,  how accurate depends on the goal of the method.

Introduction to Bayesian Inference

Good technical introduction to Bayesian modeling.   Integrated with three generations of AI ...

 2 lectures from a summer school

http://research.microsoft.com/en-us/um/cambridge/projects/infernet/

Used with applications like XBox Trueskill ranking from scores of games and picking correct 'ads' for search engines uses

Christopher Bishop, Microsoft Research
published: Nov. 2, 2009,   recorded: August 2009,   views: 45824

An AI System Captions Cartoons

In the CACM:     I note that this is in part solved by crowdsourcing, not by a computer understanding humor.   But the integration of the crowdsourcing into the solution process is interesting,  We used the Mechanical Turk system in this way.  Use for choosing advertising copy/captions?    Interacting with consumers to tag their sentiments?

" ... Microsoft researchers aim to teach artificial intelligence (AI) software how humor works by training it on an archive of New Yorker cartoons and entries into the magazine's cartoon caption contest. Researcher Dafna Shahaf fed the cartoons and contest entries to the software and taught it to select the funniest choices among captions that make similar jokes, relying partly on crowdsourced input from contract workers via Amazon.com's Mechanical Turk. Ranking jokes was the next step, requiring the researchers to manually describe what was happening in each cartoon, and to categorize its context and anomalies.   ... " 

Analytics Speed, Context, Functionality

Some obvious and useful things to consider.

Analytics Leaders Target Speed, Context, Functionality
Among the emerging trends in the data analytics business is the desire for faster access to business insights with improved context along with greater functionality in areas like predictive analytics and cognitive intelligence.

Hence, the latest vendor rankings from market watcher Nucleus Research reveal that leading analytics vendors are pursuing both requirements with a mix of deep analytics architectures and flexible, cloud-based offerins.  ... " 

Gamification Leaders in the Enterprise

Enterprise Gamification is selling a report on:

Leaders & Visionaries Rankings of Gamification Platforms
We wanted to share some details from our first Gamification Industry Report that compared and ranked twelve gamification platforms. And you are in for some surprises! If you are in the market shopping for one to get your gamification strategy going, then this is certainly for you. ... " 

I have not bought the report, but it is most interesting because it lists vendors involved, most notably and large, SAP.   Probably worth a deep look if you are involved.  Still looking forward to examples involved with SAP and their enterprise systems.   Below from the report, link above to buy the full report.

Tweets vs Blog Posts

I blog first, and then write tweets (@FranzD) pointing to a select some of the blog entries.  Blogging itself is declining, because it is just easier to write little tweets.  Apparently Google has an algorithm where site posts that have more than 500 words get higher gravitas ranking.    And now, sites are interlacing blog post articles and tweets. 

CPG and Social Media

In Mediapost: 

" ... Good or bad, the social efforts of CPG brands tend to inspire the strongest consumer responses, according to Kate Dunham, a content marketing specialist.

“More than any other industry we’ve investigated, CPG brands elicit the most emotion on social due to funny and motivational campaigns and well… smell,” Dunham explains in a new blog post.

In the CPG category, Cottonelle and Avon also hit it out of the park — with positive social sentiment rankings of 52% and 50%, during the holiday season. Big losers included Clorox (with a negative social sentiment ranking of 21%); Pine Sol (with a negative ranking of 16%); and Vaseline (with a negative ranking of 11%). ... "

Google Large Scale Data Mining

From KDNuggets:  A peek into what Google is doing with large scale machine learning.   Good technical look with related numbers " ... Sibyl is one of the largest machine learning platforms at Google and is widely used in ranking and scoring of Google products like YouTube, Android app, Gmail, and Google+. Last month, Tushar Chandra, a Principal Engineer at Google Research, talked about how Sibyl is being used at IEEE/IFIP International Conference on Dependable Systems and Networks   ... " 
 

Gamification as a Mainstream Approach

Short piece in CIO Insight: Makes the case that gamification is in common internal and external use. And quotes:

" ... Accenture points out that gamification isn't only for Millennials and techies; it's now in the mainstream. Building an effective strategy requires a focus on some combination of seven key components: status, milestones, competition, rankings, social connectedness, immersion reality and personalization. According to Accenture, many older adults are also becoming digital device-savvy. "And they are often just as keen as the young to compete with their peers and publicize their accomplishments—the essential principles of gamification," the authors note  ... ".

I still think it is difficult to implement, based on our own experience.  Dependent on corporate culture and still seen suspiciously by much of management. Especially for internal applications.   Mainstream?  Not yet, and we can't expect to grow out of it with the Millennials.  But worth understanding.

Procter & Gamble and Unilever Honored

In Mediapost:

" ... P&G ranks highest for marketing; Wal-Mart named best retail partner  ... Procter & Gamble and Unilever are in the top spots among marketers of consumer packaged goods, and Wal-Mart Stores and Target are the most effective retail partners, according to Kantar Retail's annual PoweRanking list. "In a world of increasing eCommerce, brick-and-mortar retailers have never been more open than today in collaborating to build better shopping experiences," Kantar says in the report. ... 

Procter & Gamble came in No. 1 again this year, earning the highest marks for “clearest company strategy.” And Unilever moved into second place, powered by its flexibility in meeting retailer needs and improvement in its strategic alignments. It also scored highest on “best combination of growth and profitability.” And its shopper marketing programs ranked No. 1 overall, besting P&G. Kellogg, which rose to No. 7, is the only company with sales of less than $10 billion to crack the top 10.  ... " 

Reacting to Google Algorithm Changes

Another Algorithm Change.  Is there no other feasible reaction to the ongoing changes by Google?

" ... Every online publisher and her brother has since rushed to spill ink on the topic because we all want to know: how can we please Google? Not like souless robotic spammers, of course. But like people who sincerely believe they deserve to rank highly (even if that becomes a less meaningful metric by the day). Because, well, think about it …

If we all knew the ins-and-outs of how Google ranks content, and if we knew which factors they use and the exact weight each carries in ranking, then all we’d have to do is just tick off our check list, publish the post, and bask in the light of steady traffic pouring in from our high rankings! .. " 

NumberSense Reviewed

I have been looking for books that explain the difference of what I call analytics, in my long time consulting practice and the newly hyped arena of Big Data.   I am always also on the look out for books that I can use with client groups to explain how analytics is powerful.    The explanation has to be positioned with little or no complex mathematics.  The examples have to be clear, and beg for their simple reapplication to new business domains.

NumberSense includes these characteristics.   Easy to understand, non technical examples.  In the social/marketing/economic/sports domains.  Clear positioning about how the problems should be staged.  Not much about how the problems are technically solved, but that is for the data technologists.  Not a how to book, but sets up the crucial cautions very clearly.

I particularly liked the analysis of Groupon data, which clearly defines where claims and analyses can be wrong in marketing.   A good marketing analysis example.

Fung appreciates the fact that while having more, or 'big' data is useful, but it is more important to get the data and its analysis right, especially as it relates to the decision problem being addressed.  Numbersense is paying attention to the origin and context of the data involved, and knowing enough about how the analysis will be applied to the real problem.  Misinterpretation is the worst mistake you can make.

In the final chapter Fung describes a  day in his life as a data scientist.  This was painfully reminiscent of some of my own enterprise experiences.  Its often more difficult getting the data right than solving the technical problem.

As a decision oriented person you don't need to know the technical methods, any more than you need database expertise to create reports.  This books aims at the business problem and solutions, with a strong numerical focus.  Usually with basic math.   The title of the book NumberSense, is that quality of understanding when an analysis is right or going wrong, and what to do about that.   The data in an analysis does not have to be BIG or even complex, just correctly addressed.   The book and more about it:

NumberSense: How to Use Big Data to Your Advantage    by Kaiser Fung .

See also his Numbers Rule Your World blog site for day to day examples.

Examples covered: 

" ... How does the college ranking system really work?
Can an obesity measure solve America's biggest healthcare crisis?
Should you trust current unemployment data issued by the government?
How do you improve your fantasy sports team?
Should you worry about businesses that track your data?

Don't take for granted statements made in the media, by our leaders, or even by your best friend. We're on information overload today, and there's a lot of bad information out there.
Numbersense gives you the insight into how Big Data interpretation works--and how it too often doesn't work. You won't come away with the skills of a professional statistician. But you will have a keen understanding of the data traps even the best statisticians can fall into, and you'll trust the mental alarm that goes off in your head when something just doesn't seem to add up.... "

Gamification as Goal Concept Waning?

I was involved in a number of applications in the enterprise that would today be called gamification.  Involving people to engage with systems and data to compete for better value.  Using the concepts used in online games.  Since then I have given several talks on the subject, advised people on its use, even interpreted specific marketing examples and written about that here.   I detect somewhat of a decrease in its  use in the hype cycle of technology.    That is not uncommon.  Should we go into a project with the major goal being to gamify an interaction?  Or should it be considered one technical tool that can be added to the mix as needed?  Too much of the former has occurred.   More here.  In an E-Commerce Times article. " ... Using gamification to enhance the user experience must take different user motivations into account. For example, if the goal is to drive user to donate to a cause, a reward system should engage them to return more frequently and feel emotionally connected with the process. A ranking system might be implemented to encourage friendly competition between users.... " 

Power Retailers

In Progressive Retailer.  A near time view of who the power retailers of the future.  Those that operate in mutliple channels.  " ... Manufacturers expect Wal-Mart Stores, Target, Kroger and Costco Wholesale to be the top "power retailers" during the next 15 years, according to Kantar Retail's 2012 U.S. PoweRanking report. Publix Super Markets and ALDI also are in the top 10. The report reflects manufacturers' view that multichannel and discount stores have fared well. ... "

Stack Ranking Stifles Creativity

In Forbes:   Microsoft example.  I am inclined to agree, an approach that strictly forces grading on a curve tends to make people less creative, take fewer risks, and not talk openly to their management.  Not sure how much this really made MS less creative, but can only have hurt them.  In my own interaction with MS innovation groups I found them very creative, but too cautious in delivery.   In our own enterprise we were allowed to be creative, as long as that creativity did even slightly risk key touchstones of delivery.

Sands Research Nails VW Ad

An interesting result that starts to connect new ad analysis methods with classic methods: ------ Wall St. Journal and AdWeek Agree with  Sands Research #1 Ranking ------- El Paso, Texas - Sands Research Inc. (SRI), a leading provider of neuroscientific market research services and technology, announced today that the recent rankings by the Wall Street Journal and AdWeek of the best television commercials of 2011, confirmed SRI's rating of Volkswagen's "The Force" as the top advertisement in their Annual Super Bowl Neuro-Engagement Ad Study and the benefits of neuromarketing based market research.  ------ "Our Company revealed earlier this year that Volkswagen's "The Force" was the best, #1 ranked advertisement in our annual study of all the commercials run during the Super Bowl. The spot by Deutsch LA ran away with the highest Neuro-Engagement Score (NES) that we have ever obtained in the thousands of commercials we have tested," stated Ron Wright, President / CEO of Sands Research Inc.."

Web Semantics in the Cloud

Search and meaning in the Cloud, from Computing Now:

" ... In the last two years, the amount of structured data made available on the Web in semantic formats has grown by several orders of magnitude. On one side, the Linked Data effort has made available online hundreds of millions of entity descriptions based on the Resource Description Framework (RDF) in data sets such as DBPedia, Uniprot, and Geonames. On the other hand, the Web 2.0 community has increasingly embraced the idea of data portability, and the first efforts have already produced billions of RDF equivalent triples either embedded inside HTML pages using microformats or exposed directly using eRDF (embedded RDF) and RDFa (RDF attributes). Incentives for exposing such data are also finally becoming clearer. Yahoo!'s SearchMonkey, for example, makes Web sites containing structured data stand out from others by providing the most appropriate visualization for the end user in the search result page. It will not be long, we envision, before search engines will also directly use this information for ranking and relevance purposes—returning, for example, qualitatively better results for queries that involve everyday entities such as events, locations, and people.... " 

Webinar on Neuro-Ranking Superbowl Ads

This webinar looks to be very useful, especially for someone interested in the practical application of neuromarketing.  It will be held tomorrow March 8 at 2 PM EST, Here is the complete invitation, with lots more information, including the background behind why they believe VW won, excerpt below:

" ... Sands Research will demonstrate and discuss how we collect this key data for establishing the ranking using our EEG and SMI's Eye-tracking technology. Dr. Sands will also discuss our Neuro Engagement Score (NES) and  Emotional Valance Score (EVS) and the overall Ranking of all the 69 ads presented during this year's Super Bowl.

Learn the details by attending the webinar  When: Tuesday, March 8th  2:00 PM EST / 11:00 AM PST, How: For log-in information email - info@sandsresearch.com  and we will send you the details.
To view all the previous Sands Research Super Bowl Ad Studies follow this link ....  "

Sands Research Superbowl Ad Ranking

Sands Research has posted their most recent super bowl ad ranking information, based on their unique neuro marketing methods.  They have also included eye tracking information this year as well, which is further very useful in understanding consumer reactions to ads.   They write below, much more at the link above, including rankings of all 69 ads.  

" ... The wait is over. All 69 Super Bowl Commercials have been tested in-depth and as seen above, we now have a new format which includes our Emotional Valence Score (EVS)™ and has the eye-tracking data overlaid on the commercial.  Deutsch LA's Darth Vader Ad for Volkswagen not only tops the ranking but also achieved the highest Neuro Engagement Score (NES)™ since we began testing the Super Bowl advertisements four years ago. In 2010, Deutsch LA also achieved the top spot with their Volkswagen Punch Dub advertisement ... "