Conclude, then Survey, DWP at Their Finest

The DWP has just issued a report, Attitudes to health and work amongst the working-age population on the public's perception of work as 'good' for us. This report necessarily has to be viewed against the DWP's continuing campaign to portray being out of work as bad for you and explicitly references boosting support for Dame Carol Black's report (which says precisely that) as part of its aims.

Um...

You're conducting research on attitudes towards work as good for you, and one of your expressed objectives is to further the perception of work as good for you. Which part of Anthropology 101 and the danger of predetermining your findings did the author miss?

The danger with the piece is actually that the conclusions seem so reasonable and logical. I am implacably opposed to the position of the DWP, but even I would agree that in most cases work is good for you. And there is the rub, in most cases, not all. For some disabled people, work can be very clearly bad for you. in most cases is inapplicable to the use the DWP will likely put this report, as we can already see from their splash page for the release of the report here. That talks about 80% believing work is good for physical and mental health, with no caveats whatsoever, and that is not at all what the survey says.

There is a major problem in comparing even the expressed aim of the research and the questions asked. Anthropological/Sociological survey design is science in a minefield, just the difference between one word and another of similar meaning can potentially completely change the answers that are given. The survey asked do you think that paid work is generally good or bad for physical or mental health? (my italics, see remainder of paragraph for the reason behind the emphasis). This is so wide a question as to be almost meaningless. If it had been asked in conjuction with do you think that paid work is always good for physical or mental health? then we might have had a useful comparison for understanding how the public actually view long-term health issues and whether they understand that disability really can interfere with the ability to work. But it doesn't ask that, and we have to ask why that might be. And disturbingly, buried in the data tables, three appendices after the final conclusions, is the innocuous statement ‘Don’t know’ responses to questions were removed for this analysis.

It might be argued that I am being unfair to the author by only quoting part of the question above, but I've split it because I want to address the separate problems with the rest of it. The full question actually reads Doing paid work can affect physical or mental health both positively and negatively. Taking everything into account, do you think that paid work is generally good or bad for physical or mental health? Let's look at that initial, italicised, part of the question, because a big part of probing people's understanding is recognising whether they have a good enough grasp on the subject to validate their opinion. It says can affect physical or mental health both positively and negatively, but how many people understand what that negatively might mean? Let's take me as an example; my spinal condition means that working left me feeling like I had a second degree burn over most of my lower body on pretty much a daily basis, frequently in so much pain that I could barely string two thoughts together. Hands-up if you think the survey population considered the possibility of 'negative' effects on that level when answering the question? And if they didn't consider it, then doesn't that specifically mean we have to throw out any conclusions based on the question, particularly any attempt to use the results in relation to disability and long-term illness?

We then progress from bad to worse. The author sets out to compare attitudes to short and long term conditions, but her examples of long term conditions are as follows: The first long-term scenario asked respondents to imagine they were in paid work, had long-term back pain and were currently in a phase when their back was particularly sore. The second long-term scenario asked respondents to imagine they were in paid work, had long-term depression and were currently in a phase of feeling particularly down. Back pain and depression, the two disabilities which everyone is convinced they understand, and almost no one does. I have already described what my experience of back pain feels like, is sore really an adequate descriptor for the possible extent of disability we are discussing here? And where understanding is bad for back pain, it is far, far worse for depression. In fact the report itself acknowledges this limitation, but only in the very last paragraph of Appendix A.

The survey then goes on to look at whether people have gone into work when feeling ill, and notes that people with long term health conditions are more likely to have done this than those without. Unfortunately the questions fail to allow for the every-damn-day factor of long term disability. Someone without a long term health condition may not go into work with what they perceive as severe backache, because they'll be back at work in a couple of days, but if, like me, you experience a far more severe level of pain on a daily basis, then you have no option but to go in, because tomorrow will not be better, nor the day after that, nor... If I want to work, I have to go in no matter how I feel, because I feel severe pain every damn day! Yet at the same time as someone disabled may be more likely to have gone in, through lack of any alternative, they are also far more likely to recognise that pain or depression may make it impossible to work, but the questions fail to present them with the option to say that. (Notably several tables include data indicating as many as 1 in 8 participants have spontaneously insisted on answering 'it depends', clearly indicating that a significant number of participants recognised major structural problems with the questions they were being asked).

Participants were then probed as to why they worked while feeling unwell (assuming they had), but again the questions fall apart when looked at in the light of experience of different levels of sickness/disability and the reality of long term conditions. The answers are unavoidably individual and simply cannot be extended to the general case.

Next up was whether people might have pulled sickies and why (6% had), and I actually don't have any problems with this question, just as long as it doesn't turn up being used to claim anyone going sick is faking it (and this is the DWP, so all bets are off).

The questions then move on to whether GPs and Employers should have an input into how long someone is off sick, with a disturbing 53% thinking employers should have a role. Again the issue of truly understanding the question arises and I suspect the 53% who think employers should have a say are people who have never faced an Occupational Health department who are part of the problem rather than part of the cure.

There is one very interesting breakdown that comes out of this section: Respondents with long-standing illness, disability or infirmity which did not limit activities were also much less likely (46 per cent) to agree that employers should have a say when compared to respondents with no long-standing illness, disability or infirmity (54 per cent). (my italics). As a first point this does suggest that experience of disability significantly changes views on how useful it is to involve OH in sickness for the worse, but as a second point, isn't which did not limit activities an absolutely fascinating qualification?

A string of questions then follow based on asking for help in the back pain and depression scenarios and the likelihood of getting it. All of the previous caveats apply, people who are not disabled are likely to severely underestimate the degree of disability potentially involved and are unlikely to have an educated understanding of possible employer attitudes to requesting reasonable adjustments. It isn't that the questions are invalid, rather that the survey population are incompetent (in a non-pejorative sense) to answer them, and this means that the conclusions absolutely cannot be used to represent a general case. Similar concerns apply to a following set of questions on the help expected from GPs.

A question on whether colleagues come into work when sick, which is reasonable, is then followed by a final question on whether colleagues call in sick when not sick. 'Objection, M'lud! Calls for speculation on the part of the witness.'

The report then moves on to its conclusions section and quickly states: To see improvements we would expect the percentages who agree (that work is good for you) to increase. The preset thinking that work is universally good for you is all too apparent and again casts major questions over the entire survey structure and the independence of its conclusions.

Amidst some relatively innocuous conclusions is a statement that: Far more respondents agreed that GPs should advise them (not to work) under the back pain scenario when compared to the depression scenario. No attention is drawn to the poor, if not downright biased, understanding of depression in the general population.

Alarmingly the report then states: the information will be of use as the health work and well-being agenda develops more widely. It also provides information that will be of use as the Government develops its plans for healthcare reform and also reforms to welfare benefits. Welfare benefits here undoubtedly refers to disability benefits, yet I have just demonstrated that the entire findings of the report are completely inapplicable to questions of disability due to the structure of the survey population and the inappropriate nature of the question design.

The conclusions then state: The findings are broadly positive and show that the health benefits of work are widely recognised among the working-age population That something is believed is not the same as it being positive. Much of the population once believed that the world was flat, that was a workable approximation if you never got more than five miles from the place you were born, but not exactly one we would now call positive. We now have the subtlety of understanding that tells us that the world can be considered flat over short distances, but over longer ones we must recognise that it is round. Similarly, the population may believe work is good for you, but we need them to understand the subtler distinction that work is good for most people, but for a significant number of people with disabilities, work is actually bad for them. Only that subtler understanding may genuinely be labelled positive, and that is not what the survey has measured.

Moving into the appendices, in Appendix A a worrying statement is made that: An initial set of survey questions was drafted by Health, Work and Well-being (HWWB) analysts and revised following discussion and consultation with policy colleagues and analysts from ONS, Independent Social Research (ISR), Health and Safety Executive (HSE), Department for Work and Pensions (DWP), Department of Health (DH), Scottish Government and Welsh Government. There appears to have been no independent review of the questions from outside of government, most particularly from Disabled People's Organistions or disability specialists in academia, and yet, as I have hopefully demonstrated, with only an interested amateur's knowledge of anthropology/sociology surveying, though considerable knowledge of disability, it is all too easy to raise major questions over the entire structure of the survey. Concern must also come from the attaching of the DWP survey to the ONS's omnibus survey, which could potentially confer stature on the DWP survey through association.

The appendices also note that around 35% of selected individuals declined to take part and that results have been weighted to reflect the absence of these individuals. No consideration is given to the potential that some of these individuals may have declined to take part specifically because their experience of DWP behaviour and use of statistics towards disabled people causes them to profoundly distrust the motivation and lack of bias within DWP, and that their answers to the questions might not have reflected the general distribution within their geographical/age/sex based sub-groups.

To reduce the problems with the report to an analogy that might be more generally accessible to non-disabled people, it's as though Nature had asked 2000 random people whether the Universe's 'missing mass' was best explained by a) the Higgs Boson, b) another WIMP, c) String Theory, d) the Cosmological Constant, or e) all of the above; and then published it as serious astrophysical research. Being able to offer an opinion is not the same as being able to offer an informed opinion and the conclusions of the survey can only be used within the scope of that limitation.

In the author's defence, the report states she has now moved on to work for the Department of Energy and Climate Change, strongly suggesting she is not a disability specialist and may genuinely not have recognised the weakness of her questions, but that excuses her, not the findings of her report and raises the further question of whether the DWP has any understanding of the scientific concept of independent peer review and why it is an absolute necessity?

Based on previous behaviour from the DWP we have to expect the likelihood that the findings will soon be mis-cited in support of a new set of attacks aimed at people with long term spinal conditions and/or clinical depression, implying that we should all be able to work through them and not need to access disability benefits, no matter that the overall structure and experience of the survey population renders them completely inappropriate to cast judgement on people with disabilities and that serious questions must be raised over the independence of the conclusions given the clear flaws in the survey design.