For a very convoluted chain of events that have resulted in me having multiple near mental breakdowns, I decide to write a post about opinion surveys. In particular, surveys that aim at finding out whether a school or department has a climate problem. I’d like to talk about just what sort of thing they are: what they do; how they do what they do; what they don’t do; etc.
It has occurred to me that climate surveys differ from the bulk of psychological tests in one not-immediately-noticeable aspect. Most psychological tests, such as Beck’s Depression Inventory or psychiatric assessment scales or intelligence tests such as the SAT, have some sort of a predictive goal. They are trying to predict whether someone will succeed in getting and keeping good jobs, or improve with certain types of treatment, or endanger themselves when left untreated, or re-offend, or relapse, etc. I’m using the word “prediction” in a broad way here. I don’t just mean your SAT score predicts your salary; I also have in mind classifications. One goal (“the” goal?) of classifying someone as severely depressed or mildly depressed is so that doctors have a better sense of what intensity of treatment we should be considering or how risks of undertreatment should be balanced with potential side effects. I’ll loosely refer to these as predictions because, in these cases, the goal of the test is to tell us something about the future behaviours of the test-taker.
For tests whose aim is to predict some other aspect of behaviour of test-takers, it is reasonable to assess the quality of these tests by how good they are at making these predictions. There is something that is external to the test that the test is trying to imitate. In fact, some people hold that predictions (or correlations with other measures in general; rather than through conceptual understanding) are the only scientifically valid ways of assessing a test’s validity. I don’t actually know how widespread this sentiment is since I’ve only read stuff arguing against it, although it is true that prediction-based validity measures are a lot more easily done rigorously than concept-based validity measures.
Some experience surveys can appear very similar to climate surveys, but also exhibit this feature of external criteria as well. Companies might use experience-focused surveys to predict whether certain employees will stay or leave; customer satisfaction surveys also ask of people’s experiences as a way of predicting whether they will buy again.
Climate surveys in academia, however, usually have different goals. It is not so much about predicting who will have a bad experience as it is about understanding what those bad experiences may be. In other words, it is very much about understanding and not really at all about prediction.
So, climate surveys are about understanding people’s experiences. How good is it at doing that? How do we assess? For a prediction-oriented test, it is as good of a test as it is good at doing its predictive job. For an understanding-oriented test, then, it must be as good of a test as it is at capturing people’s experiences, no?
I suspect that it’s thoughts like this that make people develop long and tedious and impossible-to-analyze climate surveys with only 30% response rate and hence no validity. If you’d like to capture someone’s experience as much as possible, you should go have a conversation with that person. Importantly, once you do, you will realize how complicated the task of “understanding someone’s experience” is. For each event, there is what happened, including who did what when. But there is also the person’s speculation as to why they did those things, which would involve a whole host of past events. If the person has spent a lot of time thinking about it, there may also be theories as to what those other people were trying to do and why it didn’t work out, etc. If you’d like to discuss the impact that event has had on the person, then you’d need more information about the person’s own circumstances, their personality, their past life experiences, etc. This is not to mention that you’d also have to confront your own biases in interpreting such accounts.
Imagine a conversation like that with someone about all the noteworthy events they have experienced as a part of some department. In what way might a survey be better, quality-wise? The only aspect in which a survey might be slightly better is that it’s anonymous (but, to be honest, it usually is not anonymous for people with more than one minority status). In all other aspects, surveys are worse at capturing someone’s experience than conversations. So, if you really want to just understand someone’s experience, you’re way better off just talking to them.
The major benefit of a survey is that it can be given to more people in a cost-efficient manner — which means that if not a lot of people responded or if it takes people longer to fill out than a normal conversation, you’d still be better off talking to people.
What do surveys do, then, if not capturing as much of people’s experience as possible? This is a question everyone who is thinking about doing a survey needs to think about and, in my experience, people don’t think about it. I suspect this is because surveys are cheap, so there is always a sense that “it couldn’t hurt, right? The worst case is that we wasted a little time and a few extra emails to get an imperfect picture of people’s experiences.”
While surveys can be cheap in dollar cost, they are often expensive in trust cost. People are asked to share personal stories in an impersonal way, which is already a toll. They are willing to do this in the hopes that something will get done. But if the survey results don’t have great quality (and I don’t just mean “some” quality; I mean “great” quality), nothing will be done. And you have just wasted people’s trust.
Why might surveys not get things done? It is important to note that surveys don’t generate magical insights; they just provide a channel for people to express them. For small, stable communities such as academic departments, many people know exactly what the problems are and what needs to be done. These shouldn’t be obscure knowledge that only a couple of people hold and keep in secrecy. Chances are, they have been voiced over and over again. If they haven’t been addressed, there will be reasons (probably widely recognized also): lack of resources; people holding power don’t take them seriously; problematic people having tenure, etc. Surveys will not make these reasons go away.
What can surveys do, then? I have argued that they aren’t good at helping you understand people’s experiences and they aren’t good at getting things done. Are they useless? To the contrary, they are very very useful — for very very specific purposes. The more specific the purpose is, the more useful a survey is. Suppose you just implemented a new meeting protocol and want to know if it’s effective at making discussions more demographic, then a survey with questions like “female students talk more under the new protocol — yes or no” and “female students talk just as much as male students under the new protocol — yes or no” are great at assessing the efficacy and sufficiency of this program.
A purpose can be specific without being as straightforward. You might be contemplating whether to implement a new meeting protocol, unsure whether it’s necessary or whether it would engender other problems. You can then write questions like “mandatory speaking time makes minority students unnecessarily anxious, rather than giving them a voice — yes or no” to assess those worries.
Once you have a specific purpose — a purpose that is more specific than “I just want to understand” and one that cannot be fulfilled more efficiently through other means — you can then write each question with that purpose in mind. This makes it much more likely to produce valid questions and the survey is likely to be much shorter. The answers will be much more interpretable, too.
- It might happen after all - May 14, 2023
- Another job market data point - December 17, 2022
- Our place in the fediverse - November 30, 2022