How P&G Got Hooked on Analytics: ‘High Value Problems People Were Willing to Pay For’ 

Procter & Gamble, the $76 billion consumer packaged goods giant with sales to 5 billion consumers in 180 countries, has lots of data. Four years ago, as the company found itself waist-deep in a rising tide of opaque data, P&G embarked on a voyage of big data discovery, one that provides a series of technology and organizational object lessons for the host of quietly desperate companies (75 percent, according to Forrester Research) that worry they're not wrestling enough insight from their data.

Walking point into the analytics jungle for P&G is Terry McFadden, associate director and enterprise architect (Decision Support Systems), who has been at the company for 30 years. Known around P&G as a “part-time comedian,” McFadden has an avuncular, competent and upbeat bearing that, one can imagine, puts colleagues at ease and promotes camaraderie in a team embarking on a daunting task.

The initial impulse toward data analytics at P&G began around 2012 when new sources of data converged into today’s data onslaught.

“Our business teams needed more access, more data,” McFadden said recently in a presentation at Strata + Hadoop World in New York. “When you think about it, it’s sales, it’s market measurements, it’s weather, its social. They wanted more granularity. We needed to wrap our arms around more of it and really improve the time-to-decision. We wanted to get deeper into the explanatory analytics, answering the age old question: ‘Why?’… We needed the ability to acquire and integrate many different types of data – structured, unstructured – we had that problem relative to the volume, the variety, the velocity, the variability of data.”

Traditional approaches for gaining insight from data just weren’t adequate, McFadden said. “The volumes were growing, the costs were growing, and the time-to-insight required in terms of the business event cycle was speeding up, and so this is a real challenge.”

As can happen when new technology takes the stage, the impetus to embark on a new strategy came from a senior P&G exec who sent an email to the IT group extolling the potential offered by the exciting new field of big data analytics. “I remember seeing an email from upper management and I about came out of my chair,” McFadden said. “It was one of those moments where – I’m teasing – somebody must have read something in Sky Magazine about Big Data, about what they ought to do and what products they ought to apply.”

Soon enough, a VP asked McFadden to come to his office. “We need to solve big data,” McFadden said to him.” “Fine,” said the VP, and began issuing battle orders: “You will do the architecture, you will do the evaluation of technologies, you will recommend the technology, you will drive the PoC, you will make a final recommendation, you will work with our folks and identify the key problems we’re going to go after.”

And he gave McFadden six months.

Terry McFadden of P&G

Where to begin? McFadden took a page out of a Gartner Group playbook that the consulting group recently published: “Mastering Data and Analytics: 'Table Stakes' for Digital Business.” In the initial phase, the idea is to go big and small simultaneously: take on strategic challenges that senior people are struggling with, in order to demonstrate value, but proceed on a baby-step basis:

“Start small,” says Gartner, “and build momentum by alleviating operational pain and deriving business value.” Small steps are more likely to proceed, and each success generates more organizational buy-in.

It was decided to go after category management as the broad and ambitious purview for data analytics at P&G. And the targeted beneficiaries were “embedded business analysts,” described by McFadden as advisers whom business executives rely on to “provide insight to daily problems, an ever increasing set of tough questions that are continually changing.” If ever there were a group at P&G what would value big data analytics, this was it.

McFadden said the embedded analysts were sent surveys asking them: “Imagine you had a magic 8-ball and you could ask it anything, what are the wicked problems you wish it would answer.” The answers helped guide the initial proof points for McFadden’s team.

“We were gated by our ability to sell services – and really, it’s sell solutions and solve problems – what are the high value problems that people were willing to pay for,” McFadden said, adding that an use case would be live “sense and respond” interactions for customer-facing managers involved in a new product launch. Another example: supporting better interfacing between manufacturing and retailers “to really create a win/win scenario. “It’s a challenging issue, it’s absolutely a big data problem, that in our proof of concept there were about two dozen data sets to bring together that we had never brought together before.”

This required the ability to quickly load and integrate structured, unstructured and semi-structured data. Within two weeks, McFadden’s team was able to load more than 25 data sources, including market signals, item sales, market share, surveys, social media and demographics, along with traditional sources, all within P&G’s data warehouse.

In building the infrastructure, McFadden said it was decided early on that a top priority, architecturally, was to build an analytics ecosystem that did not take the data off-platform. Back then, in 2013, the tools vendors assured customers that they did Big Data and that they had Hadoop, but that an outboard platform was involved, “and the size of the outboard motor and attendant costs of that model was not attractive to us,” McFadden said.”

After an extensive search, McFadden’s team found this capability at Arcadia Data, a visual analytics and BI platform for big data that enables users to create visualizations of Hadoop data.

“Yes, we have other tools associated with the stack,” McFadden said, “and we continue to look at and evaluate the marketplace, but we believe fundamentally in the model of moving the work to the data, and not moving the data off platform to the work. We think that’s a core, winning tenant of the Hadoop ecosystem. And when I say that, it’s everything that’s grown around…that ecosystem that morphs with different processing frameworks that is very attractive. Those that move the work to the data continues to show an ROI that allows you to build out more and more.”

McFadden characterized the P&G’s analytics physical landscape as largely a Cloudera stack, “a typical picture, the creatures in the zoo,” an infrastructure based on an appliance approach that enables high speed connectivity to our high speed d warehouse ecosystem.

He said results came quickly.

“Literally, inside of two weeks we were up and running,” he said. “We had a SWAT team deployed to help us. I didn’t expect anybody to say: ‘I’m just going to get a big data container, put data in it, shake it like the Magic 8-ball and get answers. But inside of two weeks, with a lot of help from our partners, we were able to start banging on the data and get after some of these (high value) questions.”

McFadden and his team went from there, building analytics capabilities use case by use case.

“It wasn’t a big bang,” he said. “No one told us, ‘Here’s the budget money, now move it all over.’ This was: We’re going to focus on high value problems that people are willing to pay for to be able to solve. There’s a dragon in their business and they want it slain, and they haven’t been able to do that with traditional approaches. “

The result has been making believers out of analysts and managers who might otherwise regard data analytics as a fad, “or who would chant ‘obsolete before plateau.’

“It’s sell a little, make a little, we hope learn a lot, grow and keep on investing,” McFadden said. “That’s been our approach.” And from a team management point of view, it’s “a lifetime of attention and study that’s going to be required here.”