CanadaOne Twitter CanadaOne Linkedin CanadaOne Facebook CanadaONe RSS


Exploring the Misuse of Statistics in Business: Is it Time to Do Away with Employee Engagement Surveys

By Julie King |

Definition of Statistics: The science of producing unreliable facts from reliable figures. - Evan Esar

Business leaders and managers love statistics, which often seem to provide a solution to challenging situations. We watch markets rise and fall and better yet, track indices that we hope will provide some indication of where our economy is headed.

As leaders and managers, we will often use a variety of employee engagement surveys to try to help motivate employees. Yet Robert Gerst, a Calgary-based statistician who specializes in employee surveys, believes that may not only be a mistake, but it could even be an unethical business practice.

Gerst is partner at Calgary-based Converge Consulting Group, where he is a partner working in operational excellence and research methods. He is also the author of the Performance Improvement Toolkit: The Guide to Knowledge-Based Improvement, an 860 page toolkit that has received rave reviews from leaders like Harry Truderung, president and chief operating officer at AT&T Canada.

What is wrong with employee engagement surveys? Why might they be unethical? CanadaOne’s editor, Julie King, caught up with Robert Gerst to find out. Here is their interview.

What are employee engagement surveys and how are they typically used?

Employee satisfaction and engagement surveys used to be about receiving employee feedback on what's working and what isn't in the employee/employer relationship from the employee’s perspective. Information that could be used to build better, more productive workplaces.

However, the employee survey has evolved into a process of misusing statistical technique in making exaggerated claims – literally creating statistical fairy tales about the level of employee engagement in the business and the impact this engagement has on performance. These surveys then, have moved from getting information useful for improvement to calculating meaningless statistical engagement indexes.

You have concerns that employee engagement surveys are not only incorrect, but that they also demotivate employees. What is going on?

Garbage in, garbage out. Employee engagement surveys typically use statistical significance tests to identify important findings – such as whether engagement is improving or getting worse, or whether departments have particularly high, or low levels of engagement.

But statistical significance isn't practical importance. When used in this way, statistical significance gets over 90 per cent of the answers wrong.

The result, management takes action on things that don't matter to employees while ignoring the things that do. It doesn't take long before employees conclude that management, or the HR department, just doesn't get it. This starts the decline in engagement.

Even worse, however, is when effective, efficient and highly engaged departments get labeled as having 'low engagement' determined by a statistical significance test. People don't know what's going on or what they should do. The numbers seem wrong, but people believe they are right, because statistical significance testing makes it look so scientific.

Many business owners and managers are not well versed in advanced concepts like statistics. Why do they need to be concerned about how the statistics and findings that are generated by engagement survey?

You are right. 'Stats' was the class most people simply muddled through in college. This is what has made it so easy for large HR consultancies to sell this statistical junk-science.

For example, engagement surveys today claim to be able to predict productivity, profitability, sales, absenteeism and even accident rates. Being able to relate employee engagement with business performance in this way may be described as the “Holy Grail” for HR. But few managers or HR professionals seem to take the time to think about how this could even be remotely possible. It isn't.

The best defense for managers and business owners is a little skepticism combined with that old bit of common folk wisdom: 'If it seems too good to be true, it is probably is.'

Is it fair to say that in your view, this really boils down to questions of ethics, and if so, why is that the case?

Absolutely. Edward Tufte, author of Beautiful Evidence, said that "a presentation is a moral act as well as an intellectual activity." When statistical technique is misused in the way it is, organizational performance suffers, real people are hurt. But the ethics of this go both ways. It is easy to point at those selling this junk science and say it shouldn't be done. But too many organizations are buying it and casting a blind eye to the ethical downside.

Perhaps an example will illustrate just how bad things have become. The statistical methods used to analyze data in engagement surveys is precisely the same as the analytical methods used in the 1994 best seller 'The Bell Curve' to 'prove' that blacks were intellectually inferior to whites. I am not saying sort of the same, or kind of like, but exactly the same.

I'd like to think that most organizations would be offended in knowing that this is the kind of thing they are inflicting on their people. They owe it to their people to do a better job at soliciting and analyzing employee feedback.

So if the most commonly used approaches are flawed, what should employers be doing as an alternative?

Get back to the basics. Stop trying to calculate statistically significant engagement indexes highly correlated to critical business performance metrics. Start asking your people what matters to them, what would help make this a better, more productive, and enjoyable workplace.

That means custom designing an employee survey. Steal questions you think are good from other surveys and develop a few of your own. Base these on informal conversations you have had with your people about what is working and what isn't.

Employee engagement and feedback surveys are great tools for getting feedback. Every business should use them. But what comes back must be given the respect it deserves. That means business leaders must listen to what is being said.

I know that not every business can afford a high priced consultant. To help, Converge has published a do it yourself guide on our Voice of the EmployeeTM website. This is a good guide even for those that can afford a consultant as it outlines what to do to conduct an employee survey properly and what to watch out for.

Smaller companies often take an ad hoc approach to the way they evaluate employee engagement and take steps to improve engagement. What are some simple things these business owners and managers can do to improve their approach?
Don't put all your eggs in one basket. The employee engagement survey is important, but it's not the only arrow in the quiver. Occasional town hall focus groups work well. So do individual and informal 'water cooler chats' with employees. The trick is to put the pieces together and use all the information to build the kind of workplaces we all want to work in.

On a personal basis, how did you come to be involved in this area of work?

As you can probably tell, I'm rather numbers oriented. My focus has been on improving operational performance. But my business partner in Converge, Shelley McLean, is a HR expert, specializing in change leadership.

She introduced me to employee engagement surveys and asked me what I thought from the scientific/statistical perspective. I was floored. I couldn't believe people were doing this. She challenged me to do it better.

It wasn't hard. Just adopt well known and valid statistical analysis techniques to the problem of gathering and gauging employee feedback. Real science doesn't use proprietary methods, magical black boxes or secret decoder rings. The techniques of sound data analysis are published and available.

Looking forward, if you were to look ahead five years into the future, what would you hope to see?
The Bell Curve data analysis gone from the face of the earth.

Canadian, Eh!

For over 15 years CanadaOne has helped Canadian businesses start-up and grow. All of the content on our site is created to help busineses get Canadian answers!

Featured Member

MemberZone. Get in the zone! Join Today!

CanadaOne Recommends

Bullies in the Boardroom: Covering the Legal Bases

Should I Start My Own Company?

Conversations with Entrepreneurs: Billy Blanks

Avoiding Legal Perils: Critical Insights into Canadian Franchise Law

Starting a Business: Choosing a Year-End