The Risk in Polls

The provincial election here in Alberta is over, and to many people’s surprise, the Progressive Conservative (PC) party remains in power for another 4 years (extending a legacy going back to 1971), and does so with another convincing majority. The reason it was something of a surprise was that many polls taken in recent weeks were pointing to a Wild Rose Party (WRP) win, or at best a closer result between the two parties. But that didn’t happen. Why the difference? There were two elements at play: strategic voting and undecided voters. People either hadn’t made up their mind, weren’t willing to speak their mind, or said one thing while really thinking another. In the most recent case, undecided voters made up a significant portion of many polls (nearly half in many cases). To try to predict the outcome of a future event based on decisions by the public, but where nearly half of those haven’t made up their mind, is fraught with peril.

The disparity between opinion polls and actual results is an interesting lesson in the value of similar survey mechanisms, like market survey or focus group, that applie to product development.

People Don’t Know What They Want

Steve Jobs famously had no use for market surveys, the business equivalent of the opinion poll. Apple doesn’t use them. Why? Because when you are inventing something that is either brand new, or not widely known, it is hard for people to predict what they would really want. Basically, people think about the “here and now”, and about how things work today. Few either can, or are willing, to imagine how things could be different. Take as an example the iPhone: it was a smartphone that was very, very different from most smartphones at the time. It was built by a company with no real history in the mobile phone space. One Forbes analyst predicted it would be largely irrelevant. Can you blame them? Apple had no history of dealing with carriers, and it was notoriously hard to do so. Carriers expected the manufacturer to sell at a huge discount. Each expected heavily customized devices, and the carrier controlled the content on the device. The iPhone broke with all of that, but the conventional wisdom at the time said that you had to play by rules that were multiple decades old to succeed.

There are plenty of examples going back in history where, when consumers were asked, they said that they have no use for the product. The car, the telephone, the radio, the television, the airplane, the home computer. All of these, and others, are devices that have changed our lives and reshaped the economic and technology landscape. When they first arrived, few could see why you would want one, because the status quo was working just fine, thank you very much.

The alternative is the “I’m not sure” response that is typically expressed, which turns into “no, I wouldn’t want/use/need it”. This is the undecided voter which, in some market surveys, must make a choice. So they choose the safe, and quite frankly obvious, answer: nope, not interested.

People Say One Thing, Do Another

The other risk is a survey response where the individual says “sure, I’d buy that”, but in reality they wouldn’t. In the polling in Alberta, this effect seemed to skew the survey results. People said “yup, I’m voting Liberal/NDP/whoever isn’t PC or WRP”, but when it came time to actually mark a ballot, they chose the PCs. That may have been influenced by the talk of “strategic voting”, where people didn’t vote for who they really wanted, but instead voted for the candidate that was most likely able to defeat the WRP in a particular riding. The goal in promoting this effort was the experience of vote-splitting in past federal elections, where people voted for the old PC party or the new Reform party, and the result was a Liberal candidate win. The worry for many was that the same thing could happen in some Alberta ridings: a degree of dislike of the PC’s and the WRP would send votes to other parties, possibly resulting in a WRP win.

This sort of thing happens in market surveys: people say one thing, but when it comes time to vote with their wallet, they do otherwise. GM often fell into this trap, taking survey after survey on new models, and getting tremendously positive, “you bet I would buy that car!” feedback. But, when the model actually hit the lots, it was a sales disappointment, or outright failure. It’s hard to pick on any particular model, because it happened for so many in the 1980’s and 1990’s. The Aztek was probably one of the more notable example. It was designed by committee. Surveys taken indicated that people would buy it, and buy lots of them. On paper, it was economical, versatile and feature-rich. In reality, it was a sales dud. The design, which apparently tested out well with focus groups, was vilified by the press. It was received very negatively by the consumer. Part of the problem appear to be a phenomenon where survey participants basically just told the surveyor what they thought they wanted to hear. They wanted to be liked by the person running the focus group. They didn’t want to be perceived as negative by others there. They felt they owed them a positive answer, because after all, they bought them treats and had asked their opinion.

What Are The Lessons?

Using a market survey, focus group or opinion poll to determine future results or future success is never a sure thing. Between the “undecided/I’m-not-sure-so-I-say-no” responses, and misleading “sure I’ll buy it, but I really I won’t” results, a product’s future gets an inaccurate view of actual success. For anything that is new (or new enough), market surveys and similar mechanisms have little value. People won’t really know what to do with a novel product until they see it, get it or see others with it. Surveys can be skewed by people who either want the person performing the poll to like them, or be impressed with them, or by people who give the answer they think the surveyor wants to hear. They want to feed someone’s ego (either theirs or the person asking questions), and give what amounts to a dishonest answer.

I’m not convinced that a market survey or focus group has any real value. I question those that see them as important to validating a product before it goes to market. GM relied on them heavily to validate product, and the long-term results were disastrous from the company. As the Alberta election shows, reality will often differ from what a survey would predict. That drastically reduces their value in my mind.