I kicked off a discussion about the hit and hope nature of employee engagement surveys. In summary it’s about why ask so many questions? Are our business leaders that far out of touch that they need to quiz us to that depth?
Jonathan Wilson offered some useful views which I share with you here:
Detailed social surveying produces at least three emotionally satisfying illusions. First is the illusion of activity. Just investing in the survey and getting everyone to complete it is a real activity that engages everyone and takes a lot of time. A survey of 50 questions will probably take a mean of 30-45 minutes to read, answer and submit. In a company of 100,000 people, if 80% respond that means it will takes about 28 person years, or more than £500,000 just to complete, before the costs of processing it.
The second is the illusion of precision. Few people read the small print that shows that any survey is a rough plus or minus estimate with distortions built in. Few surveys will claim retest reliability more than within 5%, 95% of the time. Because they include some percentages, which they choose to show to two or more decimal places, people are misled into thinking they are accurate to two decimal places. Can you measure your own attitude to two decimal places? So do you think they can measure thousands of peoples’ more accurately?
The illusion of precision leads to the illusions of tangibility, grasp and control. We think things we can measure are more ‘real’ than things we find harder to measure. We think we have a better grasp of them and can control them, or at least measure them again.
Other dangerous illusions include a mistaken belief in linearity, that the difference between 4 and 3 is the same as the difference between 5 and 4. On an attitudinal scale of 5, those two differences are huge, the difference between indifference, interest and passionate engagement.
I suggest that every manager should learn and be licensed in statistical literacy before being allowed to take or influence decisions based on survey responses – or anything else actually!
Statistics offer real insights, but rarely answers. Understanding them helps you see the questions better. The challenge is to meaningfully respond to the questions and the challenges they raise. What do you think?