Alphabet DeepMind and NHS

Concrete example of the use of Big Data in creating a new prediction model.

Google will be using NHS data to warn doctors of potential kidney issues.

http://www.theverge.com/2016/11/23/13726280/deepmind-nhs-data-streams-app-new-deal

The power of prediction

Original post:  Oct 16, 2014

Carnac the Great was a skit that Johnny Carson used to perform regularly on the show. He would deliver the punch line first and then open the envelope to reveal the question. In that era, people who could predict the future were limited to carnivals and fortune telling booths. Today, we are inching closer to something like it.

We already see glimpses of this. Google is getting very good at autofill. It's uncanny how often it can "predict" what you are looking for through whatever algorithm they use to sift through the billions of previous searches to find others similar to yours. While it isn't perfect, it does seem to learn quickly. Witness how quickly today's hot topic becomes yesterday's fad (Gangnam style, anyone?). Google seems to be able to keep up rapidly whether you are searching for information on e-books or ebola.

In the same way, retailers are now attempting to duplicate the feat. The sales cycle seems to get shorter and shorter. People's tastes change rapidly. Think of how quickly the markets appear overnight for accessories to the latest hot gadget and then die off just as rapidly as we move to the next favorite. Case designers for smartphones have to come up with new tools and dies every six months just to match the crazy number of designs.

Predictive analytics is one potential way to stay ahead of the curve. From "Apparel" magazine, here is an excerpt from Melanie Nuce at GS1 US on how it works:

Gaining a Glimpse into the Future

Instead of simply analyzing sales, predictive analytics provide retailers with a glimpse into the future and an opportunity to identify patterns that lead to effective and highly personalized customer engagement strategies. Many retail and social commerce experts believe that predictive analytics are what make big data useful in a retail environment. As opposed to descriptive analytics, which measure what has already happened, predictive analytics apply statistical modeling and data mining to study recent and historical data, allowing for more accurate forecasting.

Every day consumers are willingly offering up pools of potential product data from a multitude of sources. Today it is common for consumers to share valuable, individualized opinions and purchasing habits via social media, product reviews, wish lists and online purchase histories; and they look to each other as influencers. Further, according to a 2013 survey from Dimensional Research, 90 percent of respondents claimed that positive online reviews influenced buying decisions, while 86 percent said buying decisions were influenced by negative online reviews. Earlier this year, Amazon revealed its plans for predictive shipping, in which they scan consumer-generated information to synthesize it with a sophisticated algorithm in order to optimize their fulfillment strategies.

She goes on to reveal her opinion on what is needed to capitalize on these trends:

Retailers and brand owners cultivating a predictive analytics strategy need to commit to at least two focus areas. First, they need to attain highly accurate inventory visibility to know which product will move in which channel to anticipate future trend shifts. The second focus should be on achieving industry-agreed upon best practices for sharing consumer data.

For both of these requirements there must be more collaboration and communication between trading partners — which will be more valuable than ever before. Retailers and brand owners using a standards-based framework can enable the real-time visibility needed to tap into rich information sources and effectively leverage more customer-centric strategies.

Our challenge is to try to find some way to adapt these retail-style trends to the healthcare industry. We've got to find some way to replicate these data sources and gain control over our inventory to match this level of performance in the future to meet our ambitious targets.

To learn more, here is the link to the full article:  Predictive Analytics and the Agile Supply Chain | News | Apparel Magazine(AM)

The limits of Big Data

Original post:  Apr 15, 2014

Today's post comes from an article on big data in the New York Times:  http://www.nytimes.com/2014/04/07/opinion/eight-no-nine-problems-with-big-data.html?_r=0

I am an enthusiastic supporter of the power of big data. On the one hand, I truly believe that there is significant power in the proper application of statistics to help provide the supporting data for business decisions. On the other hand, there are many times when "big data" just seems like another buzzword that is used as a substitute for the "and then a miracle happens" magical thinking in certain unwieldy process maps!

What attracted me to this article was the way that it tempers the great expectations often applied to big data. It's important to note some of its limitations in order to grasp its power more fully.

Here is one example:

The first thing to note is that although big data is very good at detecting correlations, especially subtle correlations that an analysis of smaller data sets might miss, it never tells us which correlations are meaningful. A big data analysis might reveal, for instance, that from 2006 to 2011 the United States murder rate was well correlated with the market share of Internet Explorer: Both went down sharply. But it’s hard to imagine there is any causal relationship between the two. Likewise, from 1998 to 2007 the number of new cases of autism diagnosed was extremely well correlated with sales of organic food (both went up sharply), but identifying the correlation won’t by itself tell us whether diet has anything to do with autism.

Another critical point:

A sixth worry is the risk of too many correlations. If you look 100 times for correlations between two variables, you risk finding, purely by chance, about five bogus correlations that appear statistically significant — even though there is no actual meaningful connection between the variables. Absent careful supervision, the magnitudes of big data can greatly amplify such errors.

In my opinion, the most important part of big data is having someone with the ability to look at the various correlations and draw meaningful conclusions from them that can make a significant business impact. Knowing that Netflix users really like sophisticated drama and Kevin Spacey means nothing until you make the $100 million investment to create the US version of "House of Cards"!

But do you really know me?

Original post:  Mar 20, 2014

I found a fascinating article by Derek Thompson in the Atlantic that discusses the new problems created by the rise of computer technology. Now that we have easy access to nearly limitless amounts of information, how do we zero in on the things we want the most?

Two leading companies, Facebook and Amazon, use special algorithms to help their customers navigate their vast treasure troves of data.

An algorithm is just a piece of code that solves a problem. Facebook's problem, with the News Feed, is that each day, there are 1,500 pieces of content—news articles, baby photos, engagement updates—and much of it is boring, dumb, or both. Amazon's problem is that it wants you to keep shopping after you buy what you came for, even though you don't need the vast majority of what Amazon's got to sell.

Both organizations narrow the aperture of discovery by using their best, fastest, most scalable formulas to bring to the fore the few things they think you'll want, all with the understanding that, online, you are always half a second away from closing the tab.

Facebook uses your "likes" and combines it with paid placements from their advertisers to customize your News Feed. Amazon found that there was nothing that existed to help them, so they created their own solution. It churns through millions of items and returns your search results--usually in less than a second! Amazon came up with an algorithm that was both fast and scalable. The article includes this simplified diagram from the patent application:

These are two successful approaches, yet wildly divergent.

The strengths and weaknesses of each algorithm is clear. Facebook knows more about its users; Amazon knows more about its inventory. Each could stand to learn a bit from the other. Facebook is desperately trying to better identify its higher quality inventory, while it's often obvious that Amazon doesn't know its users. Amazon knows what's good, because it knows (a) what's been bought and (b) what's been highly rated. Facebook has likes, which are similar to ratings, butpeople might not be reading most of the content that they like, as Chartbeat CEO Tony Haile suggested in Time. In short, Amazon and Facebook are solving the problem of abundance with similar, but conceptually opposite, formulas.

The article goes on to complain about a mediocre book purchase on Amazon. The next visit brought 19 new recommendations for books by the same author!

Here is the closing argument:

Maybe we like it that way. The equivalent knock on Facebook has often been that it knows us too personally and that its insinuation into our lives is creepy. But that's just the thing. For the age of algorithms to succeed on its own terms, we have to embrace a new version of intimacy that felt natural with the local newspaper and corner shop clerk who knew our name. The machines have to know us.

Here is the link to the full article:  The Algorithm Economy: Inside the Formulas of Facebook and Amazon - Derek Thompson - The Atlantic

Obvious in retrospect

Original post:  Feb 2, 2014

This post is going live just before 6 PM on Super Bowl Sunday. By the end of the night, the game will be over and a champion will be crowned. When we look back, there will be any number of people who will claim that they knew the outcome of the game before it started. Or they might claim that if we just analyzed this fact beforehand we would have obviously known who would have won.

It's funny how all of these specific variables will become so obvious in retrospect since they are still hidden now.

As for me, I am secretly hoping for an exciting game featuring at least one play or outcome that we have never seen before. Maybe it could be the first overtime game. Whatever the case, I don't have much of a rooting interest, but I do love the conflict because I am a huge fan of sports.

Whether you like American football or not, I hope that you can appreciate all of the effort by the unsung cast of millions who help to make this secular holiday such a spectacular event. From the players and coaches to the front office personnel constructing the teams to the entertainers and fans, there are so many who have dedicated so much to take part. You might even include the hosts of all of the parties that are taking place as I type or the creative geniuses who attempt to get us to pay attention to the widgets they hawk in between the numerous breaks in the action. This game is more than just a game--it is a celebration of the brash culture that America stands for!

UPDATE 2/3/2014:  Well, I did get something that I had never seen before. The safety on the first play from scrimmage was something completely unexpected! We also should have known that the Seahawks would play suffocating defense and win, 43-8. As they say, hindsight is 20/20.

Predicting the future is kind of hard

Original post:  Mar 20, 2013

"There is no reason anyone would want a computer in their home."    

Ken Olson, president, chairman and founder of Digital Equipment Corp., 1977

It is easy to look at the quote above with the benefit of hindsight and laugh. We can look at the world around us and see just how wrong this statement was.

Perhaps there is a simpler explanation at work. In 1977, computers cost tens or hundreds of thousands of dollars. There was little to no software available that would do much of anything. Mr. Olson may have made this statement while talking to analysts about his business. Digital probably focused on their core audience of major business enterprises. At that point in time, it may have been difficult to predict just how rapidly Moore's law might drive the evolution in computers from the hulking behemoths of the past to the mighty midgets of today. Perhaps Mr. Olson just didn't spend enough time with his engineers to know what was coming. Perhaps his engineers lacked the insight to imagine why you might want to market to a personal consumer. Whatever the case, these words look spectacularly silly today.

I think a similar analogy from a company perspective might be trying to imagine a world where an 840 ventilator might be placed into the home. At today's prices and with today's applications, that is improbable for anyone without a personal fortune. However, if we were able to continually improve the performance and miniaturize the technology, it may someday be possible to shrink the unit to a more portable size. Who knows? Fifty years from now, we might laugh at a prediction that stated no one would ever need a ventilator in their home!

At a recent workgroup meeting, we were discussing mobile technology. To give the attendees some sense of the rapid pace of change, the meeting opened with this photo below:

The world is morphing so quickly that it is difficult to imagine what the future will be like. We were discussing mobile applications. Ironically, up until a few years ago, mobile applications truly didn't exist in any meaningful numbers. Will it always be this way? Will the near term future rely on these streaking meteors that flash across the sky and disappear?

I'm not sure anyone truly knows the answer but it will certainly be entertaining to try and find out!