Is Stacked Review Really The Biggest Problem?

Several articles (including this recent Slate piece) talk about the Stacked Review process used in Microsoft. All indicate that the process is a factor in Microsoft’s current problems, and some say it is really the only problem. But is it really as big a contributor to the current state of things as some make it out to be?

Stacked Review Fundamentally Flawed

Let’s get this out of the way first: the Stacked Review (and it’s variations) is a fundamentally flawed way to determine compensation and whether someone has a job or not. It depends on statistical modelling that doesn’t scale down, because it assumes that all populations will conform to a bell curve when it comes to performance and productivity. This sort of thing might hold when it comes to large populations of people, measured in the thousands. It might even work in smaller populations measured in hundreds. It simply doesn’t work in the tiny populations under review inside a corporation.

Where it breaks down is that it assumes that, by applying a bell curve to the constituent parts of the company, it will “scale up” and sort the entire company as a whole. The problem, though, is that it doesn’t work in practice. Take, as an example, the situation cited in the Slate article. The manager had 3 people, two of whom should have been considered above average in terms of productivity and contribution, and one who as average. On small teams, this sort of thing isn’t unusual. It isn’t unusual to have a small team (of even up to a dozen people) who are all above average, or far above average. It happens, and it isn’t as much of an anomaly as some might think.

The idea that a small team of 3 people would conform to a productivity curve suited to thousands makes no logical sense. Assuming that, by applying the test to smaller groups, it would “scale up” is utter nonsense. It fails, in part, because the relative merit of everyone has to be compared to everyone else, not just the other members of a statistically insignificant sample of data. In a company of tens of thousands of employees, this isn’t feasible.

Is It The Underlying Problem?

It would be silly to try to minimize the impact that the stacked review has on everything a company does, including Microsoft. We had a slight variation of it at Bell-Northern Research (nowhere near as ruthless, but similar in some ways), and it did lead to some interesting politics at the end of each year. But ours was absolutely tame compared to the stories coming out of Microsoft over the past few days. None of them are a surprise. Rather than try to hook up with teams of top performers, and try to work with the very best, people would gravitate to teams  of underperformers where they felt they could shine. Managers had to play political games to get top performers into higher ratings “buckets”, often settling for a rating they knew was lower than they should have had. People knew that weaker teams had people that should have been rated “below average”, but because the stats had to work out, got a higher rating instead. This will have resulted in people leaving, including a lot of top people, either because they despised the process or hated the outcomes.

But there needs to be a few caveats here. When I was a consultant at Microsoft, they had a slightly different system for some parts of the HR process, and one that I thought was equally silly. Everything was considered a “project”, with defined boundaries. Whether it was developing a new product or technology, or upgrading an existing product, a “project” was defined, one or more program managers assigned, and then it was up to the management (dev, test, etc) to “hire” their team. That meant that Microsoft employees were basically applying for, and interviewing for, a new job as a project wound down. The result of not being “hired”? You didn’t work there anymore. It became a distraction at a critical time in the project, because employees on the project were as busy trying to find their next job as they were trying to get the current work out the door. Certainly not as destructive as the stacked review, but not exactly a brilliant way to try to retain talent.

The other caveat is that other companies do use similar processes, and some of them are successful. Supposedly Apple uses a similar sort of process, and this is the company so many other companies are determinedly trying to emulate. Comments left on various articles on the topic indicate that more companies than we might realize use this system, and some of them are still quite successful.

Trying to lay the blame primarily, or even solely, on the stacked review system paints an incomplete picture of Microsoft’s problems. While it blames senior management (for implementing such a dumb system), it also lets them off the hook. Bad management and strategic decisions are being ignored as people have latched onto the flawed stacked review like limpets.

No Single Cause

For whatever reason, people seem to like to find the One Wrong Thing to blame for a problem. Sometimes, it truly is one thing that is wrong. IBM’s obsession with the mainframe in a time when mainframes were on the decline. Coca-Cola’s release of New Coke, and the re-appeance of the original (as Coca-Cola Classic) 3 months later. GM’s near-manic dependence on focus groups for product design. In some companies, a single change can make a significant difference in terms of success. But these are the exceptions, and not the rules.

A company as big and complex as Microsoft isn’t going to be undone by a single cause, unless you want to count “bad management and planning” as a single event. But that isn’t a single event, because everything wrong in any company can devolve to “bad management and planning”. Microsoft’s condition, both good and bad, are a result of actions and decisions made over the course of the past 10-15 years. Microsoft didn’t suddenly “go bad”, and by some measures, Microsoft is still in objectively good shape. They still have a strong, and growing, presence in the enterprise market. Azure has seen tremendous success, and the XBox 360 is on the top of the game console world.

But Microsoft’s stumbles in mobile computing, and in trying to keep the traditional PC (the heart of their business) relevant point to more than just a lack of innovation. Actually, Microsoft continues to create all manner of innovative technologies. Their problem is that they aren’t inventing the “right” technologies. Even if they do, the company doesn’t know what to do with it, because they have been in a dominant position for so long, they forget what it takes to be truly competitive.

Despite actually being a sales and marketing company at its heart, Microsoft is also too wound up in the “technology” and forgets about other key elements, specifically “content”. When you are selling into the enterprise market, you can win deals by winning the “war of the checklists”. This means having a longer feature list that your competition, and making sure all of the customer’s items are covered in some way. And really, Microsoft has been more of an enterprise vendor than a retail vendor, at least at their roots. MS-DOS, and later Windows, PCs have been around since 1980. But home use didn’t really take off until around 1995-1996, with the release of Windows 95. Up until then, the biggest buyers of PCs, and Microsoft products, have been businesses. Microsoft won the home market more-or-less by default: people had a Windows PC at work, might as well get one for home. Windows Mobile was also primarily a business tool, as PDA’s were more common for business than consumer use. Smartphones trailed in that same groove, and when the phone version of Windows Mobile came along, the biggest users were business, not consumers.

Having a company bred to win the War Of The Feature Lists, it is easy to forget that consumers don’t buy feature lists. They don’t buy technology specifications. They buy based on some combination of desire, utility, price and content. The iPod is a good example. It wasn’t the cheapest MP3 player. It didn’t have the most features. It wasn’t the smallest. It didn’t have the best battery life. But it was one of the more attractive designs, was attached to a hip and modern image, and it had a music library that dwarfed all of its competitors. The content and the desire to own the device sold the iPod, not the technology. The technology was important. The fact that an iPod was easy to use was an important part of the attractive design. But that, by itself, isn’t enough. Being easy to use is irrelevant if there is nothing to use. The iPod dominated the MP3 player market, owning 85% of the market at one point, because of the design, content and desire it evoked.

Its About The Right Product

Even if Microsoft wasn’t burdened with the stacked review system, I’m not convinced they would be in any better shape. The stacked review didn’t result in the Zune. The stacked review didn’t create the “thing” that was MSN, and hasn’t held back their presence in search. The stacked review didn’t give away Microsoft’s brief lead in smartphones in North America back in the early 2000’s, didn’t result in the moribund mess that is Windows Phone, or the trainwreck-in-progress that is Windows RT and Windows 8. The stacked review didn’t help, for sure. But the people who created the underlying, fundamental designs, the business processes, the marketing processes and made the key decisions on things like presentation, position and content, are the ones near the top of the org chart. They aren’t being subjected to the same review process, because all of them are still working there. If they had, there would have been more turn-over at the top.

Even with the presence of something as toxic as the stacked review, Microsoft was able to create and build Azure. It was able to make the XBox 360 a very successful product, and one that dominates its space. It was able to create beautiful industrial design for the Surface (even if the software and resultant sales are unspectacular). Microsoft has built a significant and growing presence in the enterprise infrastructure market. They have seen success in some areas, even with the stacked review as part of the mix.

The stacked review isn’t helping. It will have an impact at some senior levels, because top people who should be improving existing product, or inventing new product, aren’t there. It will stifle communications and initiative within the company, because you want to do enough to be above average, but hold some back so you have more next year. You won’t cooperate with other groups because of the risk of helping another team beat you out at review time. The review process is one that ultimately celebrates and promotes mediocrity, and that will have an impact on product quality. But the decisions about what to make, when to make it, and how to sell it come from the top. And that has absolutely zip to do with the stacked review process.