3 ms·
I realize everyone is a critic, and I work in ed tech so I love what you're doing here. I'm not a professional statistician, but I did write surveys professiona
by LEDThereBeLight 5y ago
I realize everyone is a critic, and I work in ed tech so I love what you're doing here. I'm not a professional statistician, but I did write surveys professionally for a few years, so I feel comfortable talking about some of the difficulties of writing good surveys. It's impossible to give much advice without knowing the goals of the survey and what you're trying to measure, but with that being said, I do have some comments and some concern about the structure of this survey.
Others have mentioned the problem of grouping children together into the same sentiment, so I won't comment on that. But I feel this survey suffers from some priming and conformity bias issues.
The survey is very "soft" in the sense of asking opinion questions, which can vary wildly with the respondent's mood, time of day, time of year (just had Christmas together as a family and everything's good), etc etc. A single survey at a point in time has basically no value, which is why I generally disregard surveys I see in the media. Surveys really get most of their value through asking the same questions repeatedly at different points in time so you can "average out" a lot of that bias. But in order to have results with any legitimacy, you almost need to split each question that you actually want to ask out into several different questions, some of which have numbers attached to them.
For example, the question "Does your child/children have friends (not including members of their immediate household)?":
A parent is more inclined to affirm their biases and assuage any guilt they might have over their child's loneliness than to help you get accurate survey data. It also depends on what a parent considers a friend versus what the child might consider a friend (someone to hang out with vs someone they feel a real connection with and are comfortable sharing things with, for example). I'm guessing you'd get very different answers if you asked the child the same question.
I can't imagine a parent is very willing to say their child has no friends at all, but you might split the "core" question out into multiple different questions - "how many close friends does your child have," "how many acquaintances," etc, with answers in different brackets (1 or fewer, etc). And then ask the general sentiment question after. The advantage of this is that you actually have something to compare across time. If you only have sentiment questions, it's almost impossible to know what the respondent has in mind for a "friend" when they answer the question.
I have the same feedback for the "How often does your child or children complain of feeling lonely or not having enough friends?" question. Different personalities will complain more or less, but that doesn't indicate how lonely a child actually feels.
A minor point, but without randomizing the question order or randomly flipping the answer options (ordering from negative sentiment to positive sentiment instead of positive to negative), you're very prone to influencing answers by "leading" the respondent to certain answers. You'd be surprised how much of a difference this makes.
Writing good surveys is very difficult, but you might pull up a list of common survey biases and run each question through the list. My 2c.
- actfrench 5y agoThis is such helpful feedback. Wow. I really appreciate the time you took to compile this - and you clearly have much more extensive experience in the field than me. If possible, would you be willing to continue the conversation with me and help me develop a survey that could lead to more accurate results? I think this information is important for people to have. If you'd like to speak further, my email is manisha[at]modulo[dot]app Either way, this gives me a lot to think about and I really appreciate your generosity in sharing it in such an eloquent and respectful manner.
- LEDThereBeLight 5y agoThank you for the kind words. I agree, it is important info. I'm a bit overwhelmed with things right now and don't have the time to help much, but I do have 2 points you can feel free to take or leave. The first is that when you're asking emotionally charged questions (as anything involving kids' happiness is), it can be a good idea to find ways to make it feel "okay" for someone to give a socially unacceptable answer. One common way is to apply some pressure for not telling the truth. For example, you could start by asking "does your child have any close friends," and then follow it up with, "if your answer was yes, how many close friends do they have?" It's sort of like being caught in a lie in real life - once people start asking follow-up questions, the pressure of keeping up the lie builds. In a survey, no one is watching you, so if you can find a way to make the sub-conscious pressure of not being totally truthful higher than the pressure of admitting something difficult, people are more inclined to revert their choices and give the truth. But this doesn't always work. The second part is to look at some popular survey makers' websites where they take common questions and re-word them in better ways. For example, https://www.medallia.com/wp-content/uploads/pdf/resources/Medallia-EX-Survey-Design-Guidance.pdf https://www.medallia.com/wp-content/uploads/pdf/resources/Me...