Metrics for Managing Innovation: Lessons From Growth Leaders

Fall 2020 https://MackInstitute.Wharton.Upenn.Edu Contents Introduction ..............................................................................3 How Whirlpool Became an Innovator and Transformed an Industry .....................................................6 Why Is Innovation Performance So Hard to Measure? ..............................................................................8 How Do Innovation Metrics Contribute to Innovation Performance? ....................................................15 Which Innovation Metrics for a Dashboard? ............ 18 Pitfalls in Measuring Innovation .................................... 21 Overcoming the Pitfalls .....................................................25 Navigating Uncertainty With Metrics ........................ 30 Metrics for Managing Innovation Lessons From Growth Leaders

2 Metrics for Managing Innovation: Lessons From Growth Leaders ^Top “You can’t manage what you don’t measure.” – Edwards Deming Knowledge without quantification is of a “meagre and unsatisfactory kind.” “To measure is to know.” “If you cannot measure it, you cannot improve it.” – Lord Kelvin “Not everything that can be measured is important and not everything that is important can be measured.”1 – Albert Einstein 1 Donald R. Lehman and David J. Reibstein, Marketing Metrics and Financial Performance, Cambridge MA: Marketing Science Institute, 2006.

3 Metrics for Managing Innovation Lessons From Growth Leaders Authored by George S. Day, Geoffrey T. Boisi Emeritus Professor and Faculty Emeritus in Residence of the Mack Institute for Innovation Management of the Wharton School, University of Pennsylvania Introduction T he intense innovation activity ignited by the global pandemic shows that some elephants can dance when they must. Companies have moved faster and taken bigger risks than could have been imagined. A further impetus to rethinking established and cumbersome innovation approaches is the acceleration of many trends that are already underway. The lockdown has brought forward a shift to online work practices and team-sharing platforms while creating new opportunities. As one example, three-dimensional printing is getting a boost by helping to replace faraway suppliers with nearby three- dimensional printing contractors and making supply chains more resilient. As another, the penetration of e-commerce grew as much in the three months from April to June 2020, as in the previous 10 years.2 Also, advancing rapidly are bioengineering capabilities to reprogram human organisms, potentially improving disease prevention and treatment. In this fast-shifting innovation environment, some observers are counseling that “forecasts are out and dashboards are in. The notion that you can fore- cast the economy, healthcare and other aspects of what can disrupt life … is 2 Chris Bradley, Martin Hirt, Sara Hudson, Nicholas Northcote, and Suen Sunat, “The Great Accelera- tion,” McKinsey Quarterly, (July 2020).

4 Metrics for Managing Innovation: Lessons From Growth Leaders ^Top gone. Now we’re in an environment where we’ve also learned that what you really need to have a handle on are the metrics, insights and what’s actually happening on the ground—the dashboard of daily life.”3 For most firms a troublesome issue is whether their dashboard of innovation metrics will contribute to a solution, or reinforce endemic problems with their innovation practices. Can a well-designed innovation dashboard help avoid a reversion to cumbersome and cautious legacy practices that have hobbled innovation performance in many firms? The purpose of this White Paper is to address this question by drawing upon a stream of research that was funded by the Mack Institute for Innovation Management during the past 12 years, culminating in the 2019 Fall Conference, “Choosing Innovation Metrics for Success in the Digital Era.”4 Among the many challenges are choosing the best set of metrics from among the plethora of possibilities (we have identified 58 candidates), and then aligning the chosen metrics with group and individual incentives to motivate superior innovation activity. The scale of these challenges can be seen from the results of a 2015 Mack-sponsored study of 192 companies that asked, “How satisfied is your company with the innovation metrics in the dashboard?”5 The results were dispiriting: 36 percent were “very or somewhat dissatisfied,” 34 percent were “neutral” and 22 percent were “somewhat or very satisfied.” There was a further 8 percent of respondents who said “don’t know,” which is a refreshing admission of the state of mind of many innovation executives we have talked with. There are few areas of management practice where 76 per- cent of those with an opinion were neutral at best and are mainly somewhat or very dissatisfied. 3 Transcript of a video talk by Kevin Sneader, reported in, “The Future of Business: Reimagining 2020 and beyond," McKinsey & Co., July 2020. 4 Mack Institute 2019 Conference, "Choosing Innovation Metrics for Success in the Digital Era," https://mackinstitute.wharton.upenn.edu/2020/innovation-metrics-for-success-digital-era/ 5 The details of this study are reported in George S. Day, “Explaining Organic Growth Performance: Why Dynamic Capabilities Need Strategy Guidance” in David J. Teece (editor) The Oxford Handbook of Dynamic Capabilities, Oxford UK: Oxford University Press, 2019.

5 Metrics for Managing Innovation: Lessons From Growth Leaders ^Top This White Paper will tackle the managerial challenges of designing and im- plementing the dashboard of metrics for managing innovation, using the best available evidence and the insights from the participants in the 2019 Mack Fall Conference to address these questions: • Why is innovation performance hard to measure? • What is the contribution of innovation metrics to innovation performance? • Which innovation metrics should be used in a dashboard? • What are the primary pitfalls in measuring innovation and aligning the measures with incentives? • How can these measurement pitfalls be overcome? A dashboard of innovation metrics has many uses. It is essential for identi- fying the weak links in the overall innovation process, and costly disconnects between the growth strategy and the portfolio of growth initiatives. It is also needed to hold managers accountable, by setting targets for improvement and linking incentives to reaching their targets. An adroitly chosen metric with a challenging target is a strong signal of a shift in strategic priorities. A.G. Lafley successfully transformed the innovation process in Procter & Gamble by setting a goal of obtaining 50 percent of their innovations from outside the company.6 Establishing the right innovation metrics and linking improvements on these measures to rewards is an essential point of any initiative to improve the innovation prowess of a firm. This reality is highlighted in the story of how the staid Whirlpool Corporation became more innovative. The lessons are applicable to any firm seeking to use innovation to increase its organic growth performance.  6 A. G. Lafley and Roger L. Martin, Playing to Win: How Strategy Really Works, Boston MA: Harvard Busi- ness School Press, 2013.

6 Metrics for Managing Innovation: Lessons From Growth Leaders ^Top How Whirlpool Became an Innovator and Transformed an Industry W hen then Chairman and CEO of the company, David Whitwam looked ahead in 1999, he anticipated a possible stalemate in the major appliance industry, with shrinking differentiation, downward price pressures and anemic organic growth.7 To address the underlying “ocean- of-white” syndrome (the mass of confusingly similar boxes confronting a prospective buyer entering an appliance store), he and his leadership team committed to escaping this “commodity trap” by developing a new set of capabilities for continuously innovating Whirlpool products. The leadership team articulated a strategic vision and outlined a desired future narrative for Whirlpool. Historically, Whirlpool’s innovation efforts relied upon engineering and marketing to generate and develop their new product concepts and feature innovations. Whitwam envisioned a Whirlpool in which “Innovation would generate from everyone and from everywhere.” Realizing this vision would mean creating a new innovation narrative at Whirlpool, which in turn would require a broad-based organizational change. To overcome Whirlpool’s siloed approach to innovation, the leadership would have to equip as many employ- ees as possible with tools for identifying latent customer needs and emerging technologies, and then combine them into innovative new offerings. For start- ers, ideas were solicited from all of Whirlpool’s 61,000 employees. Whirlpool created a set of metrics that were distributed throughout the com- pany and included an emphasis on the innovation goal of a $1 billion addition 7 The Whirlpool change process is described in J.W. Rivkin, D. Leonard, and G. Hamel, “Change at Whirlpool Corporation (B),” Harvard Business School case 9-705-463, (March 2006): N. T. Snyder and Dr. I., Duarte Unleashing Innovation: How Whirlpool Transformed an Industry, (New York: Jossey-Bass 2008); G. Hamel and N. Tennant, “The 5 Requirements of a Truly Innovative Company,” Harvard Business Review, (April 27, 2015), and D. Dependable Results,” Business Strategy Series, 10/2 (2009).

7 Metrics for Managing Innovation: Lessons From Growth Leaders ^Top to revenue within three years. Every employee’s annual performance review was tied to short- and long-term success at meeting these goals and to the quality of the business plans and implementation work that went with them. For senior leaders, financial incentives were high; a third of their pay was directly linked to what came out of the innovation pipeline. For rank-and-file employees, the rewards were team based and designed to be mostly intrinsic. “The reward,” Whirlpool explained, “is recognition by your peers.” Learning Officer Nancy Snyder explained that Whirlpool employees were excited by the challenge. “We had no idea how motivating this would be … People at the bottom were saying, ‘Finally someone gets it!'” But having a lot of fresh ideas was only the start. As Snyder put it, “Our CEO would go out and talk to thousands of people and say ‘we are going to have innovation from everywhere and everyone. If you have a concept, put it for- ward.’ But we didn’t have the systems in place to react to this.” A persistent barrier to innovation was Whirlpool’s extremely conservative budget control process that helped control costs, but tended to place a stran- glehold on new ideas. To fund innovation, Whirlpool needed to change this overly bureaucratic process. As with most organizations of that era, budget- ing was done annually, and once the process was completed, the budget was locked in. This meant that if someone came up with a great idea, there was no money to fund it. To create flexible funding, Whitwam initially had each region set up a seed fund for innovation and told the senior team that they had to fund all of the ideas that came forward, with no exceptions. If they turned someone down, the CEO told employees to come directly to him. This “end run” created an information organization structure, alongside the formal organization structure. In just two years Whirlpool’s “innovation pipeline” went from $1.3 billion to $3.3 billion. By 2005, seven years after launch, Whirlpool’s share price was at an all-time high, and the company was posting record results. Roughly

8 Metrics for Managing Innovation: Lessons From Growth Leaders ^Top $3.6 billion of the $19 billion in revenue in 2011 came from their innovation areas. This has been an enduring change. In 2018, Whirlpool’s annual report announced the launching of 100 new products and in early 2019 the company reported 16 International Forum Design awards and five Consumer Electronic Show awards.  Why Is Innovation Performance So Hard to Measure? T here are four answers to this question: (1) Innovation is risky, (2) Innovation payoffs are in the future, and create real options that are hard to value, (3) Innovation metrics are imperfect, and (4) Innovation follows many pathways, including innovations in the value proposition and the business model. Each of these measurement challenges is rooted in the difference between ordinary capabilities (OC) and dynamic capabilities (DC). Ordinary capabil- ities are used to perform well-defined tasks and routines. They enable the efficient performance of core processes for production, supply chain man- agement, financial control and so on. They enable a firm to achieve “best practices.” Most performance metrics measure the short-term achievement of meeting goals for efficiency in these OC. Such metrics are inherently static and backward-looking. Dynamic capabilities represent the, “capability of an organization to purpose- fully create, extend or modify the resource base.”8 These capabilities enable organizations to sense opportunities sooner than rivals, seize them more 8 Constance F. Helfat (editor), Dynamic Capabilities: Understanding Strategic Change in Organizations, Malden MA: Blackwell, 2007; David J. Teece, "Explicating dynamic capabilities: The nature and mi- cro-foundations of (sustainable) enterprise performance," Strategic Management Journal, 28 (2007), 1319-1350 and David J. Teece, "The foundations of enterprise performance dynamic and ordinary capa- bilities in an (economic) theory of firms," The Academy of Management Perspective, 28 (2014), 328-352.

9 Metrics for Managing Innovation: Lessons From Growth Leaders ^Top effectively and support the organizational transformation needed to stay ahead. When guided by a clear strategic vision, they enable a firm to adapt to turbulent and uncertain market conditions.9 Innovation in its many forms enables this adaptation. Innovation Is Risky Innovations come in degrees, and each innovation initiative (i.e. new product development or R&D project) can be located along a continuum of degrees of innovation. The three categories of small i … adjacencies … BIG I, map closely to the familiar three horizons representation of levels of innovation: 9 George S. Day, “Is It Real? Can We Win? Is it Worth Doing?," Harvard Business Review (December 2007). small i (Horizon 1) Incremental improvements • better • faster • cheaper Line extensions • efcacy • convenience Adjacencies (Horizon 2) BIG I (Horizon 3) Similarity of market: • customer behavior • competitive set • brand promise • channels Applicability of Capabilities/Technology: • product development capability • technology platform • manufacturing/service delivery • knowledge/science (IP) Transformations = reframe existing categories: • home routers/ networks • liquid bandages • frozen yogurt Disruptions • online trading platforms vs financial advisers • autonomous cars Extend/Defend Core

10 Metrics for Managing Innovation: Lessons From Growth Leaders ^Top While “small i” projects are necessary for continuous improvement and to defend the core business, they don’t give companies a competitive edge or contribute much to profitability. It’s the moves into adjacent markets and beyond that generate the profits needed to close the growth gap. “Adjacencies” achieve a better balance of risk and reward by striking into new territory while drawing on the resources and market knowledge of the busi- ness. A market adjacency has some similarity to the currently served market in that the firm’s brand promise and customer relationships still have relevance in the new market, and distribu- tion and sales activities partially overlap. For instance, USAA found a profitable adjacency among the relatives of military members. (Members of the armed forces were their original market.) There will also be some similarities in the competitors in adjacent markets, so competitive moves can be better anticipated. An adjacency on the prod- uct/technology dimension has some overlap with the company’s value chain, technology and manufacturing competency, quality standards, and so forth, and helps the firm leverage its knowledge base. “Small i” innovations make up 85 to 90 percent of most companies’ innovation portfolios but rarely generate much additional growth. The result is internal traffic jams of safe, incremental initiatives that delay all projects, stress orga- nizations, and fail to achieve revenue or earnings goals. Any effort to measure innovation performance must account for the riskiness of the initiatives along this spectrum (as defined by the probability of failure). Each growth initiative can be plotted on two dimensions: how familiar the firm is with the intended market, and the similarity of the product/technol- ogy to existing offerings. This matrix has many sources, including long-buried "Small i" innovations make up 85 to 90 percent of most companies' innovation portfolios but rarely generate much additional growth.

11 Metrics for Managing Innovation: Lessons From Growth Leaders ^Top consulting reports and post-audits of product and service innovation I con- ducted years ago while advising a consortium of firms studying innovation and growth challenges. Failure is defined as missing by more than 50 percent the original financial and market forecasts used to justify the project. The results are consistent with recent surveys that place the overall failure rate of new products close to 40 percent. The ranges in probabilities within the “rainbow bands” are due mainly to differences in the ability of firms to manage risk and avoid unnecessary failures. Innovation Payoffs Are in the Future Most metrics used by companies are backward-looking (they assess past and present positions), whereas innovation metrics are necessarily forward-look- ing. The lag between an initial investment in R&D and the profitable launch in the market is often measured in years and not months. But the time lag New to Company Adjacent Product/ Technology Present Present Adjacent New to Company End-Use Market* P(f)=.45-.60 P(F)=.40-.50 Small i P(F)=.25-.40 P(F)=.60-.75 P(F)=.75-.95 BIG I Adjacencies The Innovation Risk Matrix

12 Metrics for Managing Innovation: Lessons From Growth Leaders ^Top depends on the type of innovation and this in turn influences the metrics dashboard. When a business considers investing in a "small i" innovation, close to their core they have a wealth of data to plug into the familiar net-present-value (NPV) and other discounted cash flow financial models. Out comes a (seem- ingly) rigorous answer to the question “Will the anticipated NPV return on investment in this project exceed our cost of capital?” Of course, all the cost, revenue, and other estimates are just that—assumptions that can be manipulated. The financial analysis of adja- cencies and "BIG(ger) I" growth opportunities is far more prob- lematic because there is so much uncertainty to be resolved. NPV approaches can actually impede strategic thinking. Instead what is needed is a wholly different approach to resource-allocation decisions, captured in the mantra “Think big … Start small … Fail cheap … Scale fast.” More formally, what is needed is a real-options approach.10 A real option is a relatively small investment that creates the right, but not the obligation, to make further investments as the future unfolds. For this approach to work there has to be asymmetry in the distribution of returns, with greater upside potential than downside exposure to failure and loss. This happens when you can terminate the investment or otherwise limit any negative outcomes, while retaining the right to make further investments if the initial foray seems promising. Your next step depends entirely on what you learn. 10 Avinash K. Dixit and R. S. Pindyck, “The Options Approach to Capital Investment,” Harvard Business Review (May/June 1995): 105-115; and Ian C. MacMillan and Rita Gunther McGrath, “Crafting R&D Proj- ect Portfolios,” Research Technology Management (September/October 2002): 48-59. A real option is a relatively small investment that creates the right, but not the obligation, to make further investments as the future unfolds.

13 Metrics for Managing Innovation: Lessons From Growth Leaders ^Top Innovation Metrics Are Imperfect All innovation metrics are inherently imperfect. Most are flawed proxies for what we would like, and are subject to availability and surrogation11 biases and manipulation. This will not be a surprise to any executive who has coped with the ambiguities of innovation. Indeed, I have met few executives who are the least surprised by the disturbing results of a study (by a consortium of 12 companies) of the results of the commercialization of 120 projects that survived seven years. The average project was forecast to break even in slightly less than two years. The actual performance was a median time-to-breakeven of four and a half years, with the worst decile of projects not breaking even for six years. Why the lack of surprise in the gap between what was promised and what was realized? Many explana- tions are offered; essentially they boil down to the inherent optimism of innovators, the difficulty of fore- casting competitive entries and counter-reactions, and the need to present a more attractive case for a share of scarce company resources than other projects. The effect of these pressures is exacerbated by the widespread use of spreadsheets, sometimes called “the fast food of strategic decision making” that enable analysts to build and manipulate financial forecasts. As Scott Cook12, the founder and Executive Chairman of Intuit, argues, a focus on financial rewards too early in the development process leads to a “withering 11 Michael Harris and Bill Taylor, “Don’t Let Metrics Undermine Your Business,” Harvard Business Review (September – October 2019), 63-69. 12 These quotations come from Clayton M. Christensen and Derek van Beur, “The Capitalist's Dilem- ma,” Harvard Business Review (June 2014), 62-68. The effect of these pressures is exacerbated by the widespread use of spreadsheets, sometimes called "the fast food of strategic decision making" …

14 Metrics for Managing Innovation: Lessons From Growth Leaders ^Top of ambition.” Worse, “Every one of the tragic and costly new business failures had a succession of great-looking financial spreadsheets.” Innovation Follows Many Pathways It is limiting to measure innovation activity only in the R&D, Product Develop- ment Group, or Innovation Department Center. What is needed are metrics that capture the full scope of growth possibilities to be realized through innovation. Too much of the innovation conversation (especially in growth laggards or average performers) is limited to a narrow set of possibilities from changing the design or functionality of the core offering. These innovations capture attention because they are obvious. But such a view of the avenues for innovations that drive organic growth is too narrow and ultimately con- strains the effort to grow. In order to break out this narrow range of thinking and searching for innovation that create shareholder and customer value, the full spectrum13 of possibilities needs to be considered. The full spectrum innovation approach has 14 growth pathways. These path- ways allow firms to stretch, push and reimagine every dimension of their strategy, including the customer value proposition and the business model.14 Each pathway can be combined with other pathways, and escapes the simple characterization found in most metrics dashboards that only address the of- ferings pathways and yield a very biased and incomplete picture of the drivers of innovation performance.  13 George S. Day, Innovation Prowess: Leadership Strategies for Accelerating Growth, Philadelphia, PA: Wharton Digital Press, 2013. 14 For other ways of specifying growth pathways see Mohanbir Sawhney, Robert C. Wolcott, and Inigo Arroniz, “The 12 Different Ways for Companies to Innovate,” MIT Sloan Management Review, Spring 2006, 75-81, and Geoffrey A. Moore, Dealing with Darwin, New York: Portfolio, 2005. Other valuable sources were Clayton Christensen and Michael Raynor, The Innovator’s Solution, and Rita Gunther Mc- Grath and Ian C. MacMillan, Market Busters: 40 Strategic Moves That Drive Exceptional Business Growth (Boston: Harvard Business School Press, 2005).

15 Metrics for Managing Innovation: Lessons From Growth Leaders ^Top How Do Innovation Metrics Contribute to Innovation Performance? A ddressing this issue is complicated by the number of possible prescriptions that companies can use to improve their innovation performance. These range from: change the incentives, listen to the “voice of the customers,” reorganize to become ambidextrous, learn to think like a start-up, and many more. We undertook a thorough survey of industry best practices and can- vassed the literature15 to identify 18 different prescriptions for innova- tion drivers and then saw whether they could explain differences in organic growth rates (relative to their industry peers) in a sample of 192 companies from diverse industries. We needed variance in the measures of the key constructs, which is difficult to achieve in single industry studies, even when a significant portion of industry players are surveyed. Surveys are also suited to this research question, as most of the constructs do not have objective and accessible referent. 15 An illustrative sampling of the sources we examined include: Edward D. Hess, Smarter Growth: Build- ing an Enduring Business by Managing the Risks of Growth, Columbia Business School Publishing, 2010; V. Govindarajan and Chris Trimble, Ten Rules for Strategic Innovators: From Idea to Execution, Harvard Busi- ness Press, 2005; Michael L. Tushman and Charles A. O’Reilly, Winning Through Innovation: A Practical Guide to Leading Organizational Change and Renewal, Harvard Business School Press, 1997; Clayton M. Christensen and Michael T. Raynor, The Innovator’s Solution: Creating and Sustaining Innovation, McKinsey Quarterly, April 2015; Michael Schrage, The Innovator’s Hypothesis: How Cheap Experiments are Worth More than Good Ideas, The MIT Press, 2014. We undertook a thorough survey of industry best practices and canvassed the literature to identify 18 different prescriptions for innovation drivers.

16 Metrics for Managing Innovation: Lessons From Growth Leaders ^Top We found four innovation drivers (and their associated behaviors) strongly associated with the relative rate of organic growth in our sample of 192 companies.16 • Investing in innovation talent: The leadership team signals a strong commitment to innovation through investments of resources and time to recruit, develop and retain innovation talent. • Encouraging prudent risk-taking: Innovative firms foster a tolerance for risk throughout the organization by encouraging learning from innovation disappointments. • Adopting an outside-in innovation process: Growth leaders start with deep insights into customers to anticipate emergent needs and likely responses to innovation. • Aligning metrics and incentives with innovation activity: An innova- tion dashboard creates a credible and transparent link to rewards and recognition for innovation accomplishments. The four bi-polar scales shown in Figure One, measure the extent to which each of the 192 companies applied the corresponding innovation driver to illustrate the differences between organic growth leaders, growth laggards or average performers. (These three groups were identified with a cluster anal- ysis of the measures of organic growth performance.) What is notable about the fourth scale on the alignment of metrics and incentives is the degree to 16 The data from the 192 respondents to the survey were first analyzed using bi-variate correlations to assess the ability of each of the 18 hypothesized variables to explain the variance in a dependent vari- able measuring the organic growth rate of each company, relative to their industry peers. This depen- dent variable was estimated as the optimum linear correlation of three measures: (1) Past performance as measured by the average annual rate of organic growth of revenues in the past five years, relative to competitors, (2) Present spending on innovation relative to competitors, and (3) Confidence of the management team that the organic growth targets in the coming three years could be achieved. There was some ecological correlation among the 18 hypothesized drivers due to the halo effect that besets all surveys, and we judged that combining the innovation drivers in multivariate regressions would obscure specific tests of our 18 hypotheses.

17 Metrics for Managing Innovation: Lessons From Growth Leaders ^Top which even growth leaders report there is not a strong linkage between re- wards and recognition. This confirms our earlier finding about dissatisfaction with innovation dashboards, and indicates the size of the opportunity for improvement. But first we need to dig deeper to understand what lies behind the frustration and dissatisfaction of innovation executives with the value of their innovation dashboards.  Profiling Growth Leaders Developing innovation talent is not a priority Our culture is highly risk-averse and does not cope well with failure We are poor at listening to customers and anticipating their needs Our incentives are not linked to innovation performance We invest heavily to develop and retain our talent in innovation activities Our culture encourages risk-taking and learning from disappointments Every senior manager is attuned to the voice of the customer Rewards and recognition are strongly linked 2.6 1 2 3 4 5 6 7 3.9 5.0 2.2 1 2 3 4 5 6 7 3.6 4.3 3.2 1 2 3 4 5 6 7 4.4 5.1 2.7 1 2 3 4 5 6 7 4.4 = Growth laggards = Average performers = Growth leaders 3.9 Figure One: Differences between means are significant > .01 (2-tailed test)

18 Metrics for Managing Innovation: Lessons From Growth Leaders ^Top Which Innovation Metrics for a Dashboard? D espite the benefits of a well-conceived and insightful innovation dashboard, most companies are poorly served by the haphazard array of innovation metrics they use. This conclusion comes from a survey of 1,075 senior managers in global companies the Mack Institute conducted in 2008 with McKinsey & Co.17 In this survey firms were asked about their innovation strategies and performance, the metrics they used, the connection between metrics, culture and incentives, and their satisfaction with their ability to manage their innovation portfolios with these metrics. The survey began with questions about the respondent’s innovation strategy (how important a priority) and their reasons for using metrics. Next the survey asked which innovation metrics they used from a list of 27 possibilities. (This list is shown in Figure Two, and was in turn drawn from a longer list of 58 metrics we had identified.) This list was compiled from best practice surveys and personal interviews with innovation leaders in diverse companies. These metrics were divided into three sequentially ordered categories: • Inputs such as R&D spending as a percentage of sales, number of R&D projects, number of ideas or concepts in the pipeline, and the percent of ideas sources from outside the company. • Process measures including patenting activity, percent of projects hit- ting their gates on time, budget verses actual spending, average time to market, and the percent of projects that are major improvements. 17 These data came from a sample of the McKinsey Quarterly global database of 18,700 executives (55% were C-suite). The survey was sent to 4,520 potential respondents, with a 26.6% response rate. Respondents said they were knowledgeable about the overall innovation portfolio and how it was as- sessed. This was truly a global study with 34% of respondents from North America, 27% from Europe and 28% from Asia. We are grateful for the assistance and inputs of the McKinsey organization in the design and implementation of this study.

19 Metrics for Managing Innovation: Lessons From Growth Leaders ^Top These measures were aimed at understanding the effectiveness of the innovation processes that yielded the performance outcomes. • Performance outcomes such as percent of sales from new products in past N years (this was usually three years) success rates, revenue growth, due to organic sources, customer satisfaction, net present value of the portfolio, and average time-to-breakeven. Finally, the survey asked how satisfied they were with the usefulness of their metrics for improving overall innovation performance, allocating resources, assessing spending effectiveness and holding their people accountable. Possible Metrics in an Innovation Dashboard Inputs Process efectiveness Performance outcomes 1. R&D spending (percent of sales) 2. Human resources devoted to innovation 3. Pipeline of ideas/concepts 4. Number of R&D projects in active development 5. Percent of ideas/concepts from outside the firm 6. Ratio of ideas from inside/outside 1. Development activities — Percent hitting gates on time — Percent meeting quality — Guidelines 2. Patenting activity — Number filed — Number commercialized — Percent covered by patents 3. Budget vs. actual — Time — Cost/investment 4. Average time to market — Number of new products launched 5. Percent of projects that are major improvements 1. Percent of sales from new products in past N years 2. Success ratio (percentage of meeting financial goals) 3. Revenue growth 4. Return on investment in innovation (ROIC) 5. Percent of profits from new customers (or occasions) 6. Percent of profits from new categories 7. Average time to break-even/cash 8. Customer satisfaction 9. Profit growth due to new products/services 10. Percent of profits from new products in a given period 11. NPV of portfolio 12. Potential of portfolio to meet growth targets Figure Two

20 Metrics for Managing Innovation: Lessons From Growth Leaders ^Top On average the companies in the sample used seven metrics—which was consistent with the results from other studies. These seven are shown in Figure Three. Some firms used fewer than four metrics, and a few used as many as 24. The majority used between six and 10 metrics to find the right balance between too few to be useful, and too many causing confusion. Per- haps unexpected, as we noted earlier, was the depth of disappointment with the usefulness of these metrics. Only a quarter of the senior managers agreed they were able to improve their overall innovation performance or assess their progress against their overall goals with the metrics they were using. These findings are from a survey done more than a decade ago. This raises the question of whether the results of a similar survey now would be sig- nificantly different, and show greater satisfaction with their dashboards? As the conference summary18 suggests “The challenge of identifying the best measures is becoming more difficult in a world of swiftly changing business models, increasing digitization and the pressure to show quarterly financial results.” Thus, there is no reason to believe that managers won’t be overloaded with data versus useful information, won’t continue to use what is available 18 Mack Institute 2019 Conference, op cit. Most Popular Innovation Metrics Inputs Process efectiveness Performance outcomes • Pipeline of ideas/concept (61%) • Number of new products launched (46%) • Revenue growth (60%) • Customer satisfaction (48%) • Return on investment in innovation (48%) • Profit growth (45%) • Percent of sales from new product in past N years (44%) Figure Three: Only metrics used by at least 40 percent of sample are shown.

21 Metrics for Managing Innovation: Lessons From Growth Leaders ^Top versus what is needed, or that the wrong behaviors won’t inadvertently be encouraged. Indeed, the case could be made that the problems may be get- ting worse and dissatisfaction is increasing. There are numerous pitfalls to be avoided if the innovation dashboard is to be a help rather than a hindrance to decision-makers.  Pitfalls in Measuring Innovation D isquiet and unease with the side effects and unintended consequences of a reliance on metrics is widespread. A recent book with the suggestive title The Tyranny of Metrics19 argues that metrics as a management tool may shift power away from front line personnel to distant managers who set targets and manage data, potentially direct behavior towards the measurable, and induce gaming and misplaced activity to meet targets, amongst other risks. These problems are exacerbated by advances in digital technologies so managers have to wade through a far larger volume of data—much of it noisy and irrelevant—to uncover relevant signals. As Nate Silver20 has noted, “Information is no longer a scarce commodity … but relatively little of it is useful, because useless data distracts us from the truth.” Innovation metrics are highly susceptible to all these problems. There are three pitfalls, in particular, that reduce their usefulness. The first is getting misleading signals from the innovation dashboard, due to having too many metrics that emphasize results over insights. The second is fostering a bias toward incremental or “small i” incremental innovation. The final pitfall is to inadvertently encourage the wrong behavior. 19 Jerry Z. Muller, Tyranny of Metrics, Princeton N.J.: Princeton University Press, 2018. 20 Nate Silver, The Signal and the Noise: Why So Many Predictions Fail – But Some Don’t, New York: Pen- guin Press, 2012.

22 Metrics for Managing Innovation: Lessons From Growth Leaders ^Top Pitfall One: Emphasizing Results Over Diagnostic Insights There was a strong bias towards the use of performance outcome measures. Five of the seven most popular innovation metrics measured either financial results or customer satisfaction. Outputs like revenue and profit growth due to innovations and the return on investment in innovation matter, for the promise of these results is used to justify investments in innovation. Customer satisfaction reveals the ability of the firm to create compelling offerings and please customers. Suppose customer satisfaction is poor and the financial returns are disappointing? Without process effectiveness or input metrics there is no gauge on the dashboard that will show the reason for these problems. Outcome measures are also hard to interpret because they are the result of a lengthy chain of decisions. As we noted earlier performance outcomes are lagging indicators. Conversely, metrics that precede and influence performance have several benefits. Managers can get signals of their progress towards growth goals well before the financial verdict is pronounced and the soundness of their investment decisions becomes moot. Employees can get more usable infor- mation on the actions needed to achieve the objectives, and can be incented to take those actions. Given the unclear signals from outcome measures, why are they the most often used? The usual answer is that they are familiar, and readily available from the management information or accounting systems. The underlying pitfall is not investing enough to find metrics that will provide diagnostic insights into the end-to-end process of converting innovation inputs into The best performers in our study were much better at making connections and relating their innovation investments to shareholder value creation.

23 Metrics for Managing Innovation: Lessons From Growth Leaders ^Top valuable outputs. This takes deep insights into the linkages between input, process and output measures. The best performers in our study were much better at making connections and relating their innovation investments to shareholder value creation. Pitfall Two: Encouraging Incremental Innovation Some of the most popular innovation metrics subtly, or overtly encourage a focus on “small i” incremental innovations. Such small projects are necessary for continuous improvement but don’t contribute much to material improve- ments in profit or revenue growth. Because they absorb scarce development resources they displace spending on riskier “BIG I” projects that take the firm into adjacent markets or new technologies that promise higher profits. Our survey respondents were sensitive to the problem, with half agreeing that their innovation metrics were much more useful for shorter-term, incremen- tal innovations than for longer-term breakthrough innovations. Two metrics with possible toxic side effects are the percent of sales from new products in the past N years (sometimes called the “vitality index”), and the new product success rate. The first measures the rate of change in the offer- ings, and emphasizes the need to keep the products and services up-to-date. With a simplistic emphasis on total revenue this metric fails to differentiate profitable and unprofitable investments. The second can be useful for mea- suring the effectiveness of the innovation process, but may encourage too many minor projects that “succeed” due to their low thresholds for success. Exactly what is a new product? Without keen oversight by leadership any “new and improved” tweak would qualify. The potential for mischief is magni- fied when bonuses are tied to this metric. 3M reputedly ran into this problem when it increased the corporate goal for sale of new products in the past three

24 Metrics for Managing Innovation: Lessons From Growth Leaders ^Top years, from 30 percent to 40 percent. The same problem affects the equally popular measure of number of new products launched.21 The new product success rate metric is also susceptible to gaming and ma- nipulation. On one hand it is an essential indicator of the ability of the firm to weed out unpromising projects as early as possible and bring the best ones to market ahead of the rivals. But what is “success”? Was success achieved by setting an easy target? A big problem is the under-emphasis on riskier “BIG I” innovations. It is tempting to defer spending on these projects when there is a bonus attached to reaching a target success rate. Pitfall Three: Encouraging the Wrong Behaviors Our conversations with senior managers responsible for innovation about their experience with innovation metrics, soon turned to their frustration with the misalignment of these metrics with individual performances incentives. This did not fully prepare us for the depth of dissatisfaction found in the sur- vey. Only 19 percent of the sample felt that incentives were effectively aligned with the innovation metrics, while 40 percent said they were not aligned. The level of dissatisfaction was less in firms with a supportive culture, and a strategy where innovation was the top strategic priority. But overall it was evident there was limited accountability for innovation performance because individual incentives could not be linked to improving some aspect of the innovation process. The role of a supportive culture was a major influence on all the judgments of the usefulness of metrics. For instance, the top two box score for satisfaction with the alignment of incentives was 27 percent when the culture was judged to be supportive, and only 5 percent when the culture was not supportive.  21 One of the consequences was to encourage the application of standard lean or Six Sigma processes to innovation within 3M, that eventually impeded innovation, as reposted in A. Canato, David Ravisi and N. Philips, “Coerced Practice Implementation in Cases of Low Cultural Fit: Cultural Change and Practice Adaptation During the Implementation of Six Sigma at 3M,” Academy of Management Journal, 56 (2013), 1724-1753.

25 Metrics for Managing Innovation: Lessons From Growth Leaders ^Top Overcoming the Pitfalls M ost managers are decidedly unhappy with the usefulness of their innovation dashboards for improving their innovation performance and better allocating resources. This frustration intensifies during uncertain times when resources become scarce and budgets are squeezed. Despite this frustration most firms seem content to use readily available metrics. Does convenience take precedence over insight? Why is there such a gap between what is needed for a useful dashboard to be useful and what is actually used? One reason is an unbalanced emphasis on flawed outcome measures that are far removed from the management choices and trade-off. Another is a desire to have just a small number of metrics to keep managers focused on a few strategic priorities. This research study and Mack Institute best practice enquiries offer three lessons that can help bridge the gap between need and practice. Lesson One: Emphasize Learning Over Score-Keeping It is essential to know whether the innovation investments and processes are delivering results. So, performance outcome measures such as revenue and profit growth from new products, customer satisfaction and new product suc- cess rates must be in the innovation dashboard. But score-keeping measures don’t yield actionable insights into what is working or not working. Were the poor results due to inadequate or unreliable inputs, or a cumbersome and slow stage-gate process that stalled projects? Were too many small projects absorbing scarce resources and creating traffic jams in the development process? A good starting place is to shift the balance away from score-keeping toward input and intermediate process effectiveness measures. But which measures are more useful? In our experience few companies are short of ideas—the real problem is a lack of ideas that are worth pursuing. Insightful metrics will

26 Metrics for Managing Innovation: Lessons From Growth Leaders ^Top reveal loose screening that keeps too many poor ideas in the pipeline, sloppy processes causing delays in hitting the stage-gates, or poor product quality that requires re-cycling the project back through development. Thus, Whirl- pool has a real-time dashboard, so any manager can see how many concepts are in process, which part of the globe they are coming from, and how many are headed for commercialization. A useful “rule-of-thumb” is to track no more than five to eight metrics with at least one metric about each of the sequential categories of inputs, process effectiveness and outcome measures. Finding the right balance between too few metrics to be revealing and too many metrics causing confusion and absorbing resources to measure and manage, is the challenge. A process for finding this balance has three steps. 1. Identify the best metrics. This requires both a top-down approach, to ensure the dashboard has strategically insightful metrics and a bottom-up approach that identifies the areas most in need of im- provement. For example, if leadership engagement is a concern, then measure the “amount of time the leadership team spends on innova- tion projects and developing innovation talent.” 2. Establish goals for each of the metrics in the dashboard. These goals should be set in light of the aspirations of the innovation strategy, but be attainable—otherwise they won’t motivate the organization. If the goal is to increase the number of projects in the pipeline, or their risk-adjusted value, are there enough resources now to support the achievement of the goal during the average development cycle? 3. Communicate the goals and monitor progress in real time. Be sure the organization is fully aware of the metrics, understands the goals and is kept informed about progress and possible shortfalls, so cor- rective action can be taken. After one year, assess whether the goals and time lines have been realistic and adjust as necessary. Be sure

27 Metrics for Managing Innovation: Lessons From Growth Leaders ^Top to look for bottlenecks, resource constraints and bureaucratic delays that cause the goals to be missed. A useful further step is to bench-mark key metrics in the dashboard against direct competitors as well as peer companies in similar industries that are non-competitive. This kind of bench-marking exercise is a valuable tool for understanding more deeply the chosen metrics, and might suggest others that would better motivate the organization. However, as we discuss in the next section, there is a limit to how much one can learn from others. Lesson Two: Customize the Dashboard One size does not fit all firms and there are no “silver bullet” metrics. Instead a useful innovation dashboard satisfies three requirements. First it reflects the strategic priorities of the business and is customized for the market. What is useful for a biosciences firm with lengthy development cycles won’t apply to a packaged goods firm—and vice versa. Second it gives a holistic picture of the entire innovation process and, third, it accepts the reality that all metrics are flawed and susceptible to gaming. Thus, a good metrics dashboard yields insights through a process of triangulation where several metrics taken to- gether yield a fuller picture. Firms can customize their innovation dashboards by choosing from a menu of metrics among the standard possibilities, and then adding specialized metrics that are customized to the innovation strategy. Most companies limit themselves to the available menu approach: only 6 percent used measures of inputs and process effectiveness other than the ones we gave them, and 2 percent used other measures of performance. Contrast the typical menu picking approach with the rigor that best practice companies apply to developing the metrics for their dashboards. These firms treat it as a research question and ask: Which metrics give the most useful

28 Metrics for Managing Innovation: Lessons From Growth Leaders ^Top insights, have a demonstrable impact on business results, can be influenced by management action, and will be trusted by the organization? The reality for Merck, as with all pharmaceutical companies, is that they need to evaluate 10,000 new compounds a year, with six to 12 years per drug launched spent in development, and 92 percent of their development bud- get is spent on failures. Their best metrics were: time spent in each development phase (compared with competition), mean time to failure and the full economic value of the drug pipeline. Henkel, the German packaged goods giant sees a very different market reality. Many small and new products are launched each year, failure rates after launch are very high, development times are short and compet- itors match successful moves quickly. Their choices of innovation metrics were strongly influenced by an internal study of 2,237 new products launched by themselves and four competitors over three and a half years. There were striking differences between companies in the returns from their new product efforts as revealed by measures of new product share, relative number of launches (actually an indicator of activity, but not necessarily progress), new product launches that gained more than 1 percent market share, and average brand share change, which showed whether new products were building share or just replacing their existing products. The innovation metrics dashboard is also a part of a firm’s overall dashboard of performance metrics, reflecting their strategic priorities and organizational imperatives. For example, Mack Institute partner JPMorgan Chase has HR metrics to assess their ability to attract and retain talent and promote an innovative mindset. One is JPMorgan Chase’s App Store rating, which is among the highest of all banks. Another metric is the extent to which senior They need to evaluate 10,000 new compounds a year … and 92 percent of their development budget is spent on failures.

29 Metrics for Managing Innovation: Lessons From Growth Leaders ^Top management spotlights technology in public statements. Such statements attract innovation talent by expressing what the company values. For instance, JPMorgan Chase CEO Jamie Dimon’s 2008 shareholder letter only mentioned technology once, but the most recent letter mentioned it 21 times, one leader noted during the Mack Institute conference on innovation metrics, “So as a bank, we have recognized, and the leadership has bought into the idea, that technology is the beginning of innovation.” Lesson Three: Put More Discipline Into the Innovation Culture To have accountability for innovation results the culture has to be supportive of innovation metrics. But notions of accountability and measurement rub many executives the wrong way. Eric Schmidt, when he was the CEO of Goo- gle believed that trying to measure his firm’s innovation process would choke it off entirely. At the extreme this view is expressed as “the more you measure and motivate the less likely you will have a truly innovation culture.” Our findings refuted the notion that an emphasis on metrics is inimical to innovation and growth. If anything, the converse is more accurate. This con- clusion is based on an analysis of the variables that best discriminated high versus low growth performers (in terms of their rate of organic growth relative to competitors over the past five years). The significant discriminators were: first, spending on innovation relative to the spending of major competitors, second, whether the culture was supportive toward the use of innovation metrics, and third, whether the innovation metrics were effectively aligned with individual performance incentives.22 There should be a healthy tension between the creative risk taking and ex- perimentation part of the innovation culture, and the disciplined, rigorous and 22 These relationships were significant with a probability of less than .02 in the database of 1,075 com- panies.

30 Metrics for Managing Innovation: Lessons From Growth Leaders ^Top results oriented part of the culture23: An innovative organization needs both right and left brain functions. If one dominates the other results will surely suffer. When divergent and creative thinking are celebrated by the culture, the ideas will flow but the development process becomes clogged. This was the plight of one firm that had so many projects in its pipeline that they joked they had “product escapes instead of product releases,” because few projects were properly completed before being launched. But an entirely buttoned-down and convergent culture may squelch outliers and experimentation. Finding the right balance is the leadership challenge. For example, the management team has to tolerate and encourage “well intentioned” failures that occurred for unexpected and unplanned reasons but brought valuable lessons. Without such tolerance the people working on individual projects will avoid risks. Then the culture subverts the process and all the outcome measures will suffer.  Navigating Uncertainty With Metrics T he foreseeable reality is that most companies will be forced to keep innovating with more projects and tighter budgets. The easy way to get quick savings is to cut the innovation budget across the board. Every project feels some pain—whether it is a high potential winner or a “zombie” project on life support. This is both inefficient and demoralizing to the organization. A more efficient and strategic approach is guided by the innovation dashboard to exercise discipline to cut out wasteful spending on low yield activities while streamlining the innovation process to improve productivity. The aim is to avoid mortgaging the future, by placing bets on high potential innovations. 23 This requires ambidexterity, which in a person is being equally adept with both hands. See Julian Birkinshaw, A. Zimmerman and S. Raisch, “How Do Firms Adapt to Discontinuous Change?: Balancing the Dynamic Capabilities and Ambidexterity Perspectives,” California Management Review, 58 (Summer 2016), 36-58.

31 Metrics for Managing Innovation: Lessons From Growth Leaders ^Top The first step is to compile all the growth initiatives that are underway. Because innovation portfolios tend to expand organically, with little central oversight, a full picture may be hard to capture. To be sure, R&D should know all about their technology initiatives and the new product pipeline. But other growth initiatives may be more dispersed: marketing may be exploring a new end-use market with a partner, while senior management may be investing in early-stage start-ups or considering a business model innovation. For each initiative there should be an expected value based on long-run potential, ad- justed for risk. Less than half the companies in our study were able to do this. The next step is to use the innovation dashboard to root out waste and find bottlenecks that persistently slow projects. The process effectiveness and input measures work in tandem with the outcome measures. If customer sat- isfaction with new products is persistently poor—which also compromises the success rate—one possible reason is that quality targets are being missed. Armed with this information, the innovation portfolio can be stress tested. This requires tough choices and broad organizational participation to make the outcomes palatable. The objective is to make decisions on which projects are really worthy, and which innovation activities can be done more cheaply, faster or better by development partners. The focus should not be solely on budget discipline, but also on cutting time-to-market and assets tied up in innovation activities that can be taken off the balance sheet. A central lesson from our study is the need to mindfully align incentives with the right metrics. If discipline is to be maintained then key managers have to be rewarded for their efforts. A mix of carrot and stick can help: rewards for improving the success rate, and penalties for exceeding budget. Organiza- tions with a supportive culture have a much better chance of finding the right balance of investments in the future and short-run discipline. 

Metrics for Managing Innovation: Lessons From Growth Leaders Mack Institute for Innovation Management 3620 Locust Walk Steinberg Hall-Dietrich Hall 3400 Philadelphia, PA 19104-6371 Phone: +1.215.746.4831 Email: mackinstitute@wharton.upenn.edu Acknowledgments The Mack Institute for Innovation Management provided continuing support for the research described in this White Paper. The industry partners of the Mack Institute were generous in sharing their experience, insights and frus- trations with their innovation metrics. A special debt of gratitude is due my colleague and friend Dave Reibstein who is surely the guru of marketing met- rics and has taught me much. Finally, the Wharton School and my marvelous colleagues have collectively created a robust and demanding research culture that brings out the potential of all researchers. About the Mack Institute for Innovation Management Emerging technologies and innovations can create and transform industries, while simultaneously introducing new risks and uncertainty to established organizations. The Mack Institute for Innovation Management is an exclusive network that connects business leaders, researchers, world-class Wharton faculty, and students. At the institute, thought leaders from across academic disciplines and industries come together to explore how companies survive, compete, and thrive through innovation management. The institute’s multidisciplinary faculty and researchers develop practical approaches to managing innovation and share this knowledge through thought-provoking conferences, workshops, and publications.