Spray and Pray – The Base Coat

I kicked off a discussion about the hit and hope nature of employee engagement surveys. In summary it’s about why ask so many questions? Are our business leaders that far out of touch that they need to quiz us to that depth?

Jonathan Wilson offered some useful views which I share with you here:

Detailed social surveying produces at least three emotionally satisfying illusions. First is the illusion of activity. Just investing in the survey and getting everyone to complete it is a real activity that engages everyone and takes a lot of time. A survey of 50 questions will probably take a mean of 30-45 minutes to read, answer and submit. In a company of 100,000 people, if 80% respond that means it will takes about 28 person years, or more than £500,000 just to complete, before the costs of processing it.

The second is the illusion of precision. Few people read the small print that shows that any survey is a rough plus or minus estimate with distortions built in. Few surveys will claim retest reliability more than within 5%, 95% of the time. Because they include some percentages, which they choose to show to two or more decimal places, people are misled into thinking they are accurate to two decimal places. Can you measure your own attitude to two decimal places? So do you think they can measure thousands of peoples’ more accurately?

The illusion of precision leads to the illusions of tangibility, grasp and control. We think things we can measure are more ‘real’ than things we find harder to measure. We think we have a better grasp of them and can control them, or at least measure them again.

Other dangerous illusions include a mistaken belief in linearity, that the difference between 4 and 3 is the same as the difference between 5 and 4. On an attitudinal scale of 5, those two differences are huge, the difference between indifference, interest and passionate engagement.
I suggest that every manager should learn and be licensed in statistical literacy before being allowed to take or influence decisions based on survey responses – or anything else actually!

Statistics offer real insights, but rarely answers. Understanding them helps you see the questions better. The challenge is to meaningfully respond to the questions and the challenges they raise. What do you think?

Author: Doug Shaw

Artist and Consultant. Embracing uncertainty, sketching myself into existence. Helping people do things differently, through an artistic lens.

3 thoughts on “Spray and Pray – The Base Coat”

  1. You are hopeful! (I suggest that every manager should learn and be licensed in statistical literacy. As a nation we would appear to be innumerate which is why headlines writers are able to get away with some of their appalling copy!

  2. Had a good ‘dangerous conversation’ with Rob Briner of Birkbeck College about this last week. His recommendation, roughly speaking, is: ‘Don’t survey. And if you have to survey, don’t ask lots of dumb questions, and don’t do it any more times than you have to. And don’t mistake the results for facts of nature.’

    The conversation came around to the idea of asking just one thing: something like “do you love your job?”, but even then there were folks in the room pointing out that, if you have to ask, you must be organisationally incapable of normal human understanding.

    I’m sympathetic to all that, but … in the realpolitick of big organisations, the way you change things is with argument, backed by evidence of a kind that is recognised by the organisation. Surveys may be a necessary evil, for a limited period, if they’re managed as a supply of ammo for that good work. Oh, and while you’re there, a free comments box sometimes surfaces a neat idea.

    1. Good thinking AdHib. We should maybe have more dangerous conversations eh? And I like the comments box, we just have to remember to look in it, read what’s in there, then do something about it.

Leave a Reply to AdHib Cancel reply

Your email address will not be published. Required fields are marked *