6 ms·

Why Aren’t SMS Interventions Designed to Boost College Success Working at Scale?

54 points by vitabenes 5y ago

saurik 5y ago
This title cut-down to "why aren't text messages designed to boost college success working at scale?" was extremely confusing until I looked more carefully at the original article and saw it says "text message INTERVENTIONS". To save characters--if we are ignoring the half-rule to accept the upstream author's title--I would suggest changing "text message" to "sms" instead of dropping the key noun "interventions".
- vitabenes 5y ago
  Changed.
- MauranKilom 5y ago
  I still can't actually parse the title. What is "college success working at scale"? Edit: Ok, halfway through the article it became clear: The proper reading is > "Why Aren’t SMS Interventions which are Designed to Boost College Success Working at Scale?" and not the way I read it initially: > "Why don't we design SMS interventions so that they boost college success working, at scale?" Funny enough, this is a case where languages with different capitalization for noun vs verb (working, in this case) would immediately resolve ambiguities. Well, except for Title Case, I guess.
  - contravariant 5y ago
    It's ((Why aren't) (SMS Interventions Designed to Boost College Success) (Working at Scale)) not ((Why aren't) (SMS Interventions) (Designed to Boost College Success Working at Scale))
    - rm445 5y ago
      (Why (aren't (Working (SMS Interventions Designed to Boost College Success) (at Scale))))
crooked-v 5y ago
The tl;dr from the article: > Implementing a texting campaign with hundreds of thousands of students may require organizations to forego advising and have texts provide just one-way content or to have large caseloads that limit how much engagement advisors can have with individual students. > Another trade-off is that students may have less trust in a state agency or national nonprofit sending the messages. > Finally, state- and national-level campaigns have typically sent content that is uniform across all students, whereas earlier, local campaigns often personalized information to students’ college plans or FAFSA (the United States’ government financial aid application) filing status. > Even local text campaigns may face obstacles to efficacy that similar campaigns did not encounter to the same degree a decade ago. [...] As message volume through text increases, we should reasonably expect that any one message won’t get the same attention—and eventually, that people will migrate to other communications channels.
orangeoxidation 5y ago
Do they not? "At scale" is somehow often synonymous with "with less effort and care per instance", but that doesn't have to be the case. Things work worse if you scale only one part of the equation (the "customers"), but do not scale the other side appropriately (the "service provider"). That should be obvious. > Text campaigns are most likely to be effective when they: Are sent by a school and even staff member who students know and trust Add meaningful and novel value above and beyond existing outreach and support efforts like giving students real-time information on the status of their financial aid applications Provide content that is personalized to the student and where they are in the college application and financial aid process Having a new org sending out mass messages to all students is not scaling The described. It is something different. Scaling it would be convincing more school staff and schools to make a similar effort. The positive effect was not caused by "getting a message", it was caused by "getting personal advice from someone you know". They scaled the wrong thing.
- Spooky23 5y ago
  Some problems can be improved with a simple contact point. I have a colleague who works in this space in a research capacity. They were looking at use of different systems to measure behavior and identify behaviors that are correlated with poor performance. In one scenario, they surveyed a cohort who was leaving and entering the dorm after 1AM, resulting in poor AM class attendance. 40% of the responses were essentially: “I went to meet my [girl/boy]friend and have sex.” Getting a text message to tell that kid to get his beauty rest isn’t going to work. Just treat people like adults, imo.
  - ineedasername 5y ago
    Some problems can be improved with a simple contact point Absolutely. Most schools don't employ enough advisors to have sit-down appointments with every student every semester. However, I did see a school implement a policy where 1st semester students-- a higher risk population-- were required to have a sit down like that before they were allowed to register for the next semester. That single effort resulted in a 15% decrease in 1st-year dropouts.
thrower123 5y ago
Yikes, the whole concept of this sounds horrifying to me. That text message vibration and notification ping engenders an immediate stress reaction.
tsumnia 5y ago
This isn't specific to the article per say, but something I've been discussing with my colleagues revolves around gamification and nudge theory in education. There's a lot of pressure on gamification in financial trading after the GameStop saga in January, and with the pre-existing pressures from loot boxes, I'm starting to ask "Why is it okay for education to do it?" I think nudge theory is a subset of all the elements that make up gamification, but still, the idea that something is okay to do because "we're doing it to help people" seems a bit odd.
- SpicyLemonZest 5y ago
  When you look closely, a whole ton of things are gamification. Even something as simple as a problem set is very gamified, if you choose to see it from that angle, since real challenges almost never come in the form of neat bundles with a single unambiguous solution. I'm honestly not sure whether I think that's a useful angle or not, but either way it seems inescapable to conclude that there's a category of healthy gamification towards good ends.
- tomjen3 5y ago
  If you have signed up for college, then I don't think it is wrong to get an SMS, similar to what a doctor might send. I think the key is "person really wanted this at some point in the recent past", which is not the case for gamefication.
- hoppyhoppy2 5y ago
  If you accept that the use of grades is just a way of gamifying individual learning to establish player scores and a competitive leaderboard, then schools been at it for a long, long time.
xwdv 5y ago
SMS used to be a very effective channel for getting conversions from people. But marketing channels are limited resources, they are most effective early in their discovery when organizations begin using it, but once their novelty is sucked dry the effectiveness tapers off until it’s no more effective than a flashing banner ad.
impendia 5y ago
I'm a professor. My university bought a big piece of enterprise software, designed precisely to track and boost student success in various ways. The software sent me an email asking me to do something, ostensibly from someone in our advising center, signed by my assistant chair instead, but obviously corporate-sounding and autogenerated. It reeked of fakery. Is this sort of thing supposed to accomplish something?
- myfavoritedog 5y ago
  Yes, the people who sold the software to the university made a ton of money.
- Loughla 5y ago
  That tells me you have a team of people who have not done their jobs, or let too many decisions be made by committee. We have a large piece of software at our institution that communicates automatically with students, staff, and faculty based on various triggers (grades, dates, etc). The hardest part of that process was writing the messages. It took 1 semester to get the system operational, and two years to dial in the messaging. The worst thing you can do is create a "specific" message for an individual with generic language targeted at groups. Which, subsequently, it seems is what most institutions do. No idea why. It's easier, I suppose?
- toast0 5y ago
  > The software sent me an email asking me to do something, ostensibly from someone in our advising center, signed by my assistant chair instead, but obviously corporate-sounding and autogenerated. It reeked of fakery. This is like the definition of phishing.
- fallingknife 5y ago
  Yes. What it accomplishes is to abuse the trust you have built with your students through genuine interaction to spread their corporate (ok university. Same shit) propaganda by making it seem like it comes from you. Most people can smell that from a mile away so you should absolutely not do it if you care about the respect of your students.
- 5y ago
calltrak 5y ago
Integrating SMS messages with your workflow and getting the message out there is a no brainer. You can setup something like Zapier and https://textgun.io https://textgun.io in 5 minutes. Response rates are still 5 times better than email.
dogman144 5y ago
At scale implies automation, especially guessing at budgets to support these programs. I think schools know there has to be a human at some point in the interaction chain. A frustrating experience I had with this is a school I was at (expensive uni) tried to farm out this responsibility to other students. In a mentorship program designed to do 1:1 student:student pairings and used “SMS intervention” this program asked students to take up to *10* student mentees, with the idea that that the real mentor load is pretty low (quick convos). To me, this felt like an outrageous ask, as if they were trying to sub in students paying to be there with the tasks paid advisors are responsible for. This was at grad school, and I think indicated something funding challenges behind the scenes. I imagine it’s similar/worse at high school levels. Maybe not the same exact problem, but it makes sense why this specific program linked has trouble.
michael_vo 5y ago
It does work "at scale" but you need to launch, fail, and iterate just like any product. Georgia State University is a great example of successfully implementing the goal of "college success" and continuing to innovate. They started by solving the problem of students accepting their offer letter, but never choosing classes during that crucial first summer. First tool they built used a combination of AI chat box and counsellors to answer common questions, help students fill out forms, and/or get financial aid. At each step of product development they asked "what's next?". For example choosing a degree is "hard" for first generation college students. So they built salary, and career earnings during the UX where you pick your courses. Another non-SMS solution they provide is automatically depositing money into student's accounts who are low and struggling. This especially helps 4th year students who may have burnt through their financial aid. https://www.youtube.com/watch?v=Rh20W2uXHCc&ab_channel=BillGates https://www.youtube.com/watch?v=Rh20W2uXHCc&ab_channel=BillG...
ineedasername 5y ago
I work in the industry. There are a couple of reasons they don't work, at least as currently used. 1) Over saturation/overused: Especially as High Schools begin using these, students get too many, too frequently, and it just becomes a sort of background noise. Back when students still had email as their prime mode of digital communication, we had the exact same issue. Studies on their effectiveness were based on relatively short time spans where nudges were new & novel: The novelty has worn off. 2) A simple nudge can't help a complex problem. A college may nudge "You haven't registered yet! Don't forget to register for the next semester". That's an extremely frustrating message to receive when you're poor and can't pay next semester's bill, or some other roadblock. 3) It has been used to justify not scaling human resources for students, so any benefit is offset by lesser real human contact with advisors. People complain about administrative bloat at universities, but that doesn't apply to advising: I've seen caseloads rise from ~300/advisor to ~500/advisor. This is especially problematic for overcoming #2 above: Advisors can often help with complex issues that student's believe are insurmountable. My observations on how to do it better: Only use nudges for low-stakes issues, and even then choose carefully so students don't become numb to them. High-stakes issues need human outreach. With limited bandwidth by advisors to do this work, it needs triage: Even if it's high stakes, don't waste time reaching out personally to low-risk students. Determining risk requires a two-pronged approach: 1) A statistical model that uses historical data to identify students at risk of poor performance or attrition. A competent data analyst can build a reasonable model without having to go too far into advanced data mining. "Deep Learning" tends to be fairly useless here: There's not enough data. There's also a few good vendors out there that will perform more advanced risk modeling, partitioning a school's population into a dozen or more groups and building models for each of them. These aren't cheap, but I've seen it pay off when correctly paired with one-to-one human outreach initiatives. In a single year I saw it reduce attrition by 1%, not to mention academic performance increases, and that more than paid for the cost. I've also seen schools pay the invoice and expect things to magically change without aligning their resources to the risk analysis output and waste a lot of money. 2) A rule-base system of identifying students at risk. Things like a late payment; a good student getting their first D or F in a class; delaying registration for the next semester; requesting transcripts to be sent to another school; and many other factors. Using the high risk population identified in #1 and #2, that is where you focus your one-to-one human outreach efforts.
jrm4 5y ago
Higher-ed Instructor Here. Because, and I am not exaggerating in the slightest, this is the worst idea I have ever seen directly related to my career. This concept is an idea that I am fighting all the time. Teachers and students alike around me sort of blindly "get used to" these notifications flying around you and even begin to expect them even when they don't work. Essentially, they become the equivalent of spam in your email that you ignore, except they're harder to ignore and people still have this wildly misguided faith that they can think for you. As a first measure -- I'd love to remove all "notification" type services from higher-ed; after a bit of a learning curve, as in -- yes, you must visit this website or call this person or read the syllabus to get things done. You develop your own system of remembering things, maybe it's paper, maybe you email yourself, whatever works for you
infogulch 5y ago
If nagging people to death with notifications doesn't help, what does? Maybe goal setting writing exercises: Using Future Authoring to Improve Student Outcomes - https://heqco.ca/wp-content/uploads/2020/03/HEQCO-Formatted_EPRI-Mohawk.pdf https://heqco.ca/wp-content/uploads/2020/03/HEQCO-Formatted_... - https://www.selfauthoring.com/research https://www.selfauthoring.com/research
b-more 5y ago
Because the vendor picked for these trials is terrible at understanding how to communicate with college students.

54 points by vitabenes 5y ago

This title cut-down to "why aren't text messages designed to boost college success working at scale?" was extremely confusing until I looked more carefully at the original article and saw it says "text message INTERVENTIONS". To save characters--if we are ignoring the half-rule to accept the upstream author's title--I would suggest changing "text message" to "sms" instead of dropping the key noun "interventions".

vitabenes 5y ago
Changed.
MauranKilom 5y ago
I still can't actually parse the title. What is "college success working at scale"? Edit: Ok, halfway through the article it became clear: The proper reading is > "Why Aren’t SMS Interventions which are Designed to Boost College Success Working at Scale?" and not the way I read it initially: > "Why don't we design SMS interventions so that they boost college success working, at scale?" Funny enough, this is a case where languages with different capitalization for noun vs verb (working, in this case) would immediately resolve ambiguities. Well, except for Title Case, I guess.
- contravariant 5y ago
  It's ((Why aren't) (SMS Interventions Designed to Boost College Success) (Working at Scale)) not ((Why aren't) (SMS Interventions) (Designed to Boost College Success Working at Scale))
  - rm445 5y ago
    (Why (aren't (Working (SMS Interventions Designed to Boost College Success) (at Scale))))

crooked-v 5y ago

The tl;dr from the article: > Implementing a texting campaign with hundreds of thousands of students may require organizations to forego advising and have texts provide just one-way content or to have large caseloads that limit how much engagement advisors can have with individual students. > Another trade-off is that students may have less trust in a state agency or national nonprofit sending the messages. > Finally, state- and national-level campaigns have typically sent content that is uniform across all students, whereas earlier, local campaigns often personalized information to students’ college plans or FAFSA (the United States’ government financial aid application) filing status. > Even local text campaigns may face obstacles to efficacy that similar campaigns did not encounter to the same degree a decade ago. [...] As message volume through text increases, we should reasonably expect that any one message won’t get the same attention—and eventually, that people will migrate to other communications channels.

orangeoxidation 5y ago

Do they not? "At scale" is somehow often synonymous with "with less effort and care per instance", but that doesn't have to be the case. Things work worse if you scale only one part of the equation (the "customers"), but do not scale the other side appropriately (the "service provider"). That should be obvious. > Text campaigns are most likely to be effective when they: Are sent by a school and even staff member who students know and trust Add meaningful and novel value above and beyond existing outreach and support efforts like giving students real-time information on the status of their financial aid applications Provide content that is personalized to the student and where they are in the college application and financial aid process Having a new org sending out mass messages to all students is not scaling The described. It is something different. Scaling it would be convincing more school staff and schools to make a similar effort. The positive effect was not caused by "getting a message", it was caused by "getting personal advice from someone you know". They scaled the wrong thing.

Spooky23 5y ago
Some problems can be improved with a simple contact point. I have a colleague who works in this space in a research capacity. They were looking at use of different systems to measure behavior and identify behaviors that are correlated with poor performance. In one scenario, they surveyed a cohort who was leaving and entering the dorm after 1AM, resulting in poor AM class attendance. 40% of the responses were essentially: “I went to meet my [girl/boy]friend and have sex.” Getting a text message to tell that kid to get his beauty rest isn’t going to work. Just treat people like adults, imo.
- ineedasername 5y ago
  Some problems can be improved with a simple contact point Absolutely. Most schools don't employ enough advisors to have sit-down appointments with every student every semester. However, I did see a school implement a policy where 1st semester students-- a higher risk population-- were required to have a sit down like that before they were allowed to register for the next semester. That single effort resulted in a 15% decrease in 1st-year dropouts.

thrower123 5y ago

Yikes, the whole concept of this sounds horrifying to me. That text message vibration and notification ping engenders an immediate stress reaction.

tsumnia 5y ago

This isn't specific to the article per say, but something I've been discussing with my colleagues revolves around gamification and nudge theory in education. There's a lot of pressure on gamification in financial trading after the GameStop saga in January, and with the pre-existing pressures from loot boxes, I'm starting to ask "Why is it okay for education to do it?" I think nudge theory is a subset of all the elements that make up gamification, but still, the idea that something is okay to do because "we're doing it to help people" seems a bit odd.

SpicyLemonZest 5y ago
When you look closely, a whole ton of things are gamification. Even something as simple as a problem set is very gamified, if you choose to see it from that angle, since real challenges almost never come in the form of neat bundles with a single unambiguous solution. I'm honestly not sure whether I think that's a useful angle or not, but either way it seems inescapable to conclude that there's a category of healthy gamification towards good ends.
tomjen3 5y ago
If you have signed up for college, then I don't think it is wrong to get an SMS, similar to what a doctor might send. I think the key is "person really wanted this at some point in the recent past", which is not the case for gamefication.
hoppyhoppy2 5y ago
If you accept that the use of grades is just a way of gamifying individual learning to establish player scores and a competitive leaderboard, then schools been at it for a long, long time.

xwdv 5y ago

SMS used to be a very effective channel for getting conversions from people. But marketing channels are limited resources, they are most effective early in their discovery when organizations begin using it, but once their novelty is sucked dry the effectiveness tapers off until it’s no more effective than a flashing banner ad.

impendia 5y ago

I'm a professor. My university bought a big piece of enterprise software, designed precisely to track and boost student success in various ways. The software sent me an email asking me to do something, ostensibly from someone in our advising center, signed by my assistant chair instead, but obviously corporate-sounding and autogenerated. It reeked of fakery. Is this sort of thing supposed to accomplish something?

myfavoritedog 5y ago
Yes, the people who sold the software to the university made a ton of money.
Loughla 5y ago
That tells me you have a team of people who have not done their jobs, or let too many decisions be made by committee. We have a large piece of software at our institution that communicates automatically with students, staff, and faculty based on various triggers (grades, dates, etc). The hardest part of that process was writing the messages. It took 1 semester to get the system operational, and two years to dial in the messaging. The worst thing you can do is create a "specific" message for an individual with generic language targeted at groups. Which, subsequently, it seems is what most institutions do. No idea why. It's easier, I suppose?
toast0 5y ago
> The software sent me an email asking me to do something, ostensibly from someone in our advising center, signed by my assistant chair instead, but obviously corporate-sounding and autogenerated. It reeked of fakery. This is like the definition of phishing.
fallingknife 5y ago
Yes. What it accomplishes is to abuse the trust you have built with your students through genuine interaction to spread their corporate (ok university. Same shit) propaganda by making it seem like it comes from you. Most people can smell that from a mile away so you should absolutely not do it if you care about the respect of your students.
5y ago

calltrak 5y ago

Integrating SMS messages with your workflow and getting the message out there is a no brainer. You can setup something like Zapier and https://textgun.io https://textgun.io in 5 minutes. Response rates are still 5 times better than email.

dogman144 5y ago

At scale implies automation, especially guessing at budgets to support these programs. I think schools know there has to be a human at some point in the interaction chain. A frustrating experience I had with this is a school I was at (expensive uni) tried to farm out this responsibility to other students. In a mentorship program designed to do 1:1 student:student pairings and used “SMS intervention” this program asked students to take up to *10* student mentees, with the idea that that the real mentor load is pretty low (quick convos). To me, this felt like an outrageous ask, as if they were trying to sub in students paying to be there with the tasks paid advisors are responsible for. This was at grad school, and I think indicated something funding challenges behind the scenes. I imagine it’s similar/worse at high school levels. Maybe not the same exact problem, but it makes sense why this specific program linked has trouble.

michael_vo 5y ago

It does work "at scale" but you need to launch, fail, and iterate just like any product. Georgia State University is a great example of successfully implementing the goal of "college success" and continuing to innovate. They started by solving the problem of students accepting their offer letter, but never choosing classes during that crucial first summer. First tool they built used a combination of AI chat box and counsellors to answer common questions, help students fill out forms, and/or get financial aid. At each step of product development they asked "what's next?". For example choosing a degree is "hard" for first generation college students. So they built salary, and career earnings during the UX where you pick your courses. Another non-SMS solution they provide is automatically depositing money into student's accounts who are low and struggling. This especially helps 4th year students who may have burnt through their financial aid. https://www.youtube.com/watch?v=Rh20W2uXHCc&ab_channel=BillGates https://www.youtube.com/watch?v=Rh20W2uXHCc&ab_channel=BillG...

ineedasername 5y ago

I work in the industry. There are a couple of reasons they don't work, at least as currently used. 1) Over saturation/overused: Especially as High Schools begin using these, students get too many, too frequently, and it just becomes a sort of background noise. Back when students still had email as their prime mode of digital communication, we had the exact same issue. Studies on their effectiveness were based on relatively short time spans where nudges were new & novel: The novelty has worn off. 2) A simple nudge can't help a complex problem. A college may nudge "You haven't registered yet! Don't forget to register for the next semester". That's an extremely frustrating message to receive when you're poor and can't pay next semester's bill, or some other roadblock. 3) It has been used to justify not scaling human resources for students, so any benefit is offset by lesser real human contact with advisors. People complain about administrative bloat at universities, but that doesn't apply to advising: I've seen caseloads rise from ~300/advisor to ~500/advisor. This is especially problematic for overcoming #2 above: Advisors can often help with complex issues that student's believe are insurmountable. My observations on how to do it better: Only use nudges for low-stakes issues, and even then choose carefully so students don't become numb to them. High-stakes issues need human outreach. With limited bandwidth by advisors to do this work, it needs triage: Even if it's high stakes, don't waste time reaching out personally to low-risk students. Determining risk requires a two-pronged approach: 1) A statistical model that uses historical data to identify students at risk of poor performance or attrition. A competent data analyst can build a reasonable model without having to go too far into advanced data mining. "Deep Learning" tends to be fairly useless here: There's not enough data. There's also a few good vendors out there that will perform more advanced risk modeling, partitioning a school's population into a dozen or more groups and building models for each of them. These aren't cheap, but I've seen it pay off when correctly paired with one-to-one human outreach initiatives. In a single year I saw it reduce attrition by 1%, not to mention academic performance increases, and that more than paid for the cost. I've also seen schools pay the invoice and expect things to magically change without aligning their resources to the risk analysis output and waste a lot of money. 2) A rule-base system of identifying students at risk. Things like a late payment; a good student getting their first D or F in a class; delaying registration for the next semester; requesting transcripts to be sent to another school; and many other factors. Using the high risk population identified in #1 and #2, that is where you focus your one-to-one human outreach efforts.

jrm4 5y ago

Higher-ed Instructor Here. Because, and I am not exaggerating in the slightest, this is the worst idea I have ever seen directly related to my career. This concept is an idea that I am fighting all the time. Teachers and students alike around me sort of blindly "get used to" these notifications flying around you and even begin to expect them even when they don't work. Essentially, they become the equivalent of spam in your email that you ignore, except they're harder to ignore and people still have this wildly misguided faith that they can think for you. As a first measure -- I'd love to remove all "notification" type services from higher-ed; after a bit of a learning curve, as in -- yes, you must visit this website or call this person or read the syllabus to get things done. You develop your own system of remembering things, maybe it's paper, maybe you email yourself, whatever works for you

infogulch 5y ago

If nagging people to death with notifications doesn't help, what does? Maybe goal setting writing exercises: Using Future Authoring to Improve Student Outcomes - https://heqco.ca/wp-content/uploads/2020/03/HEQCO-Formatted_EPRI-Mohawk.pdf https://heqco.ca/wp-content/uploads/2020/03/HEQCO-Formatted_... - https://www.selfauthoring.com/research https://www.selfauthoring.com/research

b-more 5y ago

Because the vendor picked for these trials is terrible at understanding how to communicate with college students.