8 ms·

I am one of the two PhD students that conducted this research with Boston Public Schools. I'll be glad to answer any question you may have.

keithpeter 8y ago
Did you think the project was useful in the sense of getting the parents to think about possible changes and rejecting them? So they as citizens are clear that the current system is sub-optimal but they are OK with that
- lodrein 8y ago
  That's a great question, that I asked myself a lot. I think this project was useful because it started this debate and brought more information to the table. For example, I found that a lot of people did not know how much the start times affected the district costs (tens of millions of dollars every year), or how the schools times seemed to be unequitable. And this Boston Globe article allows even more people to read about this issue and make their own opinion. Ultimately, a lot of people in the US want later start times for high schoolers. This work can provide some information about the ways to implement such a change, as well as the consequences.
opwieurposiu 8y ago
Did the algorithm assume all persons time was equally valuable? From an economics standpoint this is clearly not the case, the range of salaries of public school parents could vary by a factor of 10x or more. To me it seems obvious that the rich parents should be able to pay the poor parents to take the "worse" time slots. This should be beneficial for both parties, but human pride may preclude this as a solution.
- lodrein 8y ago
  Yes it did. What you are suggesting would be politically very complicated to implement. Also, one need to take into account the characteristics of the district: https://www.bostonpublicschools.org/cms/lib/MA01906464/Centricity/Domain/187/BPS%20at%20a%20Glance%202017-2018.pdf https://www.bostonpublicschools.org/cms/lib/MA01906464/Centr... 66% of the students are economically disadvantaged. Additionally, a lot of schools have both "rich" and "poor" parents, but still need to have a unique time.
- gbrown 8y ago
  Meanwhile, rich parents work jobs with flexible hours and poor parents are hourly with potential catastrophic penalties for missing work. Theoretically, you're correct (from a parent preference perspective, at least). In practice, even the rich parents would balk at the dollar amount required to make it worth it. That also completely ignores the fact that what you propose would deepen the existing structural inequalities for students, and presupposes that schools are and should remain stratified by income. I vehemently disagree with this unstated premise.
- AtlasBarfed 8y ago
  And there is the reminder that I'm reading hacker news. From an economics standpoint the ideal labor force is slavery. From an economics standpoint there is no point to the elderly they should be killed off once they are no longer viable laborers. Children are only useful if they show potential slave labor ability, and otherwise should be killed in adolescence if there are obvious shortcomings. From an economics standpoint murder, war, and violence are viable tools in competition, as illustrated by cartel turf wars. So it's OBVIOUS to me I should be allowed to kill you and your kid take your school start time slot if I am economically incentivized to do so / it's convenient aka economic to me and it is up to you to invest in the deterrent to prevent such things. Oh, you're not in favor of such things? I guess you are in favor of regulation, humanity, and other factors in things besides raw "economics".
  - twic 8y ago
    This is a huge overreaction - a sort of caricature of the slippery slope argument, attacking points that the parent did not make or even remotely imply, that was presumably intended to signal virtue, but does more to discredit is maker. Distasteful as the parent's comment might be, they are talking about Pareto optimality - finding a situation where both parties would agree it's an improvement.
  - PurpleBoxDragon 8y ago
    >From an economics standpoint the ideal labor force is slavery. How so? A happily paid worker with their own free time isn't likely to revolt and kill you. Even an unhappy worker who hates you will just quit. Is your calculations forgetting to count such costs? And a machine that replaces a person is far far more efficient. So machines backed by human operators/fixers/etc. end up outperforming humans, and free humans will have far better education possibilities. >From an economics standpoint there is no point to the elderly they should be killed off once they are no longer viable laborers. You are clearly zeroing out the cost of having all non-elderly realize this will be their fate as well. That seem an arbitrary cost to zero out and makes the results of your calculations unusable. >Children are only useful if they show potential slave labor ability A free child that is educated has a much higher return on investment on average. >otherwise should be killed in adolescence if there are obvious shortcomings Once again completely ignoring the costs of taking such a course of action. I'm not really having any confidence in your models.
- roel_v 8y ago
  That sort of arrangement incentives people in many unexpected ways, like people optimizing for the chance to get allocated a slot that can be sold at maximum price, rather than what is the 'best' for them overall. It then turns into a game theory exercise of course.
- MockObject 8y ago
  Excellent point, and a good real-world introduction to the idea of Pareto Efficiency. https://en.wikipedia.org/wiki/Pareto_efficiency https://en.wikipedia.org/wiki/Pareto_efficiency
- SamBam 8y ago
  One of the reasons that this would be a terrible idea is that the person who stands to make money by selling their good slot for a terrible one is not necessarily the person who has to wake up at 5:00 AM to bus across town. Adding money into the mix, when there's a large difference of means between families, adds a level of incentive that shouldn't be there -- one that has nothing to do with the district's goal of providing an equal education to all its students. When the district looks at the results at the end of the year, and all the poor kids have poor grades/attendance because they were all in the worst time slots, they can't say "well, at least the parents are slightly richer because of this, from the secondary trading market." Rather, based on the objectives they are concerned with, they may have just made inequality even worse.
- asr 8y ago
  Ah yes, I'm sure the whole thing would have worked politically if only they'd also tried to set up a system in which rich parents paid poor parents in order to get better start times. Conveniently, this would also mean that the students would be income-segregated so rich students would not have to be mixed in with poor students. </snark> ... I also took economics in college. To me it seems obvious that calling a fanciful solution like this "obvious" except for "human pride" is woefully inaccurate.
snerbles 8y ago
Do you feel that your research was sold to the public by the district as a panacea?
- lodrein 8y ago
  I think changing the start times is something the district had wanted to do for a long time. Therefore once the research and the algorithm suggested it was a good idea, they quickly jumped on board. Now I am sure we have done some communications mistakes. For example the algorithm does not "decide" the start times. You just give it a (long) list of preferences and objectives (e.g., less HS students starting early, more fairness across the population...), and then the algorithm generates thousands of solution that try to satisfy these objectives. Solutions are carefully reviewed, and the objective updated if we think we can do better. So in the end, this research is just a tool, it does not solve all the problems, and in the end people still need to find a way to agree. Hopefully they have a little bit more information to do so!
  - platz 8y ago
    > start times is something the district had wanted to do for a long time So, was the real motivation here one of efficiency and cost-savings then? That is a clear thorn they might have wanted to solve. Is the added sleep beneficial to students, all other things equal? I'm sure it is, but was the real motivation a cost savings and the sleep deprivation a nice cover for the real motivation?
    - lodrein 8y ago
      No, as far as I know having later school start times for high schoolers was the main objective, and that was the strongest one in the solution that was chosen by Boston when they used our algorithm. There has been a lot of research on high school students lack of sleep, and it seems that the school times do have a important impact on the students : https://www.cdc.gov/features/school-start-times/index.html https://www.cdc.gov/features/school-start-times/index.html
    - SamBam 8y ago
cowpig 8y ago
I just want to thank you for doing this work. Real-world problems are so much messier and more frustrating than the clean abstractions many of us choose to work with. But this is really meaningful work that should have concrete social benefit. We should all hope to do good as you have.
- lodrein 8y ago
  Thanks! I am actually very excited about this thread, our main hope is for this project to start a conversation !
Paul-ish 8y ago
Is there an arxiv version of your paper somewhere?
- lodrein 8y ago
  We submitted it to a journal that has a fast review process, and we are towards the end of it. We did not think soon enough to put it on arxiv, and we should be able to do it soon (it requires some editing to remove the journal-specific things). In the meantime, there is some available information that is listed on my website: https://sebmart.github.io/media.html https://sebmart.github.io/media.html
kwillets 8y ago
How transparent was the process? Are there standards for auditing the data and results, for instance?
- lodrein 8y ago
  The process went too fast to be entirely transparent. Boston tried to communicate as much as they could, and anything can be requested (Freedom of Information Law). We also have a paper that should not be too far from publication that details every detail of the thing. I would argue that although it could have been more transparent, it is way more transparent than the status quo, where each individual school can try to pressure the district to get the time that they want, without considering the "big picture" and the impact on other schools/neighborhood/budget. All decisions were made after a lot of group conversations with parents/staff/principals... But I am sure some things could have been better, in the end we were just the grad students trying to help :)
  - kwillets 8y ago
    Well, ideally people will look at this and say there are clear tradeoffs, but I suspect many will just blame the algorithm.
    - lodrein 8y ago
      This can happen and did happen before. Maybe being suspicious of an algorithm is not a bad things, we should have extremely high standards for them ! My hope is that some people will just have a better view of how complicated this problem is, and be able to construct their own opinion from there. And this would already be something I would be proud of!
coleca 8y ago
Was your project done as part of the A-Lab? http://mitsloan.mit.edu/actionlearning/labs/a-lab.php http://mitsloan.mit.edu/actionlearning/labs/a-lab.php My company had pitched the students of the A-Lab a couple years ago, but unfortunately we didn't get picked. BPS was one of the others pitching the students at the same event.
- lodrein 8y ago
  No, although the A-lab have a lot of interesting projects. We are part of the Operations Research Center: https://orc.mit.edu/ https://orc.mit.edu/
nhf 8y ago
Hello from across campus! Do you think more user and stakeholder engagement would have helped the public accept your research? I have no idea if BPS has the bandwidth for supporting this type of engagement, but I imagine a lot more fears would be assuaged if parents could have some role in "co-creation" of the algorithm, rather than just accepting results. A lot of research from the fields of citizen science, public understanding of technology, and user experience show that participatory design techniques can enhance acceptance. E.g., https://www.bmj.com/content/319/7212/774.short https://www.bmj.com/content/319/7212/774.short
- lodrein 8y ago
  Thanks for sharing! I am sure more engagement is always a good thing! One of the tricky tasks of BPS is to make sure that this engagement is "balanced", i.e., everyone's voice is equally heard. And you can imagine how hard that may be in practice. For example, Boston Public Schools had this survey asking people for their preferences on the start times. But most of the answers from these survey came from schools with less economically disadvantaged families. One way to do it is to call the families directly or have a lot of local meetings. But it is still a complicated problem.
  - nhf 8y ago
    Thanks for the response. I'm in the technology and policy program within IDSS and we have discussions about these "politics of technology" issues a lot. I'm sure you could become a case study in one of our classes if you wanted :)
dmckeon 8y ago
Did BPS consider applying the changes to only one or a few high schools and their feeder schools? Perhaps the realities of distribution of students among schools in Boston not allow doing such a beta test?
- lodrein 8y ago
  That could be a way to do it, and Boston decided to do it all at once. I think their reasoning was the fairness of the process (it's hard/unfair to use beta testing on public schools). Also, it is such a hard change that maybe it is easier to do it all at once. You may also want to consider the political implications that are very important when making this kind of decisions.
aero142 8y ago
It seems the amount of change was a large problem. Was it attempted to minimize either number of student start times changed or total delta of minutes changed? Was it discussed how this could be done incrementally to slowly move towards more optimal start times?
- lodrein 8y ago
  Yes incremental changes looks like a good solution. I think it is hard to do it with the delta minutes changed, as you could have dramatic consequences on the costs (the ripples on the transportation systems can be quite surprising, in that changing the time of one school can have a big impact on the number of buses needed). As to the number of student start time, the main question is how do you choose the schools that change first? How do you make sure that your changes are not interrupted in the middle for political reasons, which would be unfair for the first schools and maybe the whole system. Also, it is an algorithmic challenge to find a "path" of good solutions that go step by step to the good final start times.
jhayward 8y ago
The engagement of the MIT team is interesting, but this reads to me like a classic failure of a "big-bang" rollout, rather than a gradual adoption. Was there any back and forth to create a slower-paced rollout plan?
- lodrein 8y ago
  We talked about it at some point, and it seems like a good idea. There are two main issues with it: first it can be hard to do a step by step change that is (a) fair (how do you choose the first schools to change?) and (b) works for every step (make sure that the transportation costs do not explode in between).
  - jhayward 8y ago
    > it can be hard to do a step by step change that is (a) fair (...) and (b) works for every step Yes. This is what separates the pros from the not-pros in major systems design and operation. The transition plan is the major key to plan, not the algorithm (which plays the role of Shiny Object in this case). It is lots harder, lots, to get smoothly from A to B than to simply big-bang B. It takes enormous attention to detail and thinking through of every angle, and when done well is a thing of wonder. I think in this case your team was not well served by how it was tasked. Thanks again for responding, and for your team's efforts in the design.
drewvolpe 8y ago
One of the changes was moving start times of high schoolers later, which often came at the expense of younger students who would've had to start earlier. In my neighborhood, having to put 5 year olds on the bus at 6am or earlier was a big cause for the uproar from parents. Can you talk about how you made this tradeoff? There's research that suggests high schoolers do better with later start times. But is there any research that backs up doing it at the expense of younger students? ie, is it possible that all students benefit from a later start time?
- lodrein 8y ago
  I am no specialist of children health, but my understanding is the following: I guess starting too early is never good for any child, but there is something specific with teenagers that makes it a lot worse. There is much more medical literature documenting this fact than for the younger kids. So yes, in a way, given the constraints of costs, the algorithm had to choose the lesser of two evils.
- arkh 8y ago
  https://www.ncbi.nlm.nih.gov/books/NBK222804/ https://www.ncbi.nlm.nih.gov/books/NBK222804/ > Research findings suggest that changes occur in the “biological clock” during adolescence. As a result, teenagers have a natural tendency to fall asleep later and to wake up later. This is referred to as sleep phase delay. > A correlation was found between subjects' melatonin secretion and their stage of development. The results indicated that melatonin onset occurs later in adolescents, making it difficult for them to go to sleep earlier at night. At the same time, the hormone “turns off” later in the morning, making it harder for them to wake up early (Carskadon et al., 1998, 1999). It's like things are happening to the body during puberty.
siosonel 8y ago
Thanks for working on the technical side of this social-equity project. Was the possibility of parent offering car pools or "walking groups" for kids included in the algorithm? I'm assuming that if volunteer parents are compensated to give carpool rides or lead walking buddies, then the negative impact would be lessened to parents who has to handle drastically different school start/end times. At the same time, it may be cheaper to compensate volunteer parents than for the district to procure or maintain more buses. Another way to ask this question is: were there volunteer transit, ride share, or pre/post-school care incentives included in your algorithm, to balance the potential negative impacts to parents?
- lodrein 8y ago
  Yes and no. The algorithm runs with historical student data and therefore only considers students using the bus system in order to compute the effects of school start times on transportation. On the other hand I think the district had planned to used the saved transportation money (~18 million/year) in order to help the most impacted parents (pre/post school care...) I guess volunteer transit is always a positive thing for the parents and the districts. The algorithm was just trying to fulfil the objectives given by the district (costs, later students, fairness...). The compensations are what should happen after the algorithm: what do you do with the saved money, how do you help the most impacted parents? These negative effects were expected to disappear with the years as new parents start to choose the schools after the start times have changed (and therefore choose schools that can accomodate their schedules).
lazyasciiart 8y ago
Did anyone suggest making the change, but announcing it today and having it come into effect a year (or even two) from now? That would give parents time to plan adjustments to their schedule, and would give a lot of people the chance to move schools if they really wanted to.
- kodablah 8y ago
  Or altering the ideal results to be less drastic at least initially to incrementally work towards the goal. Clearly the problem is the size of the change, not what the change is.
- mrcodedude 8y ago
  > District officials expected some pushback when they released the new school schedule on a Thursday night in December, with plans to implement in the fall of 2018. That's a ~9 month notice?
  - lazyasciiart 8y ago
    The first round of applications for schools in Boston is January, so it's only a few weeks (over Christmas!) for families to figure out if they can make it work, if it's worth moving schools, which school they want to move to, etc. (Your chances of getting into your first choice school decline rapidly after the first round).
fdr 8y ago
Don't get discouraged, if you were feeling that way. It was important you substantiated the envelope of what kind of improvements are theoretically possible. And although the wealthier residents, who really ought to have the property to absorb inconvenience, may have been self-serving, they might reasonably have a point that sudden upheaval can cause problems for others that cannot attend meetings. Maybe some will remember this if money for student transportation comes up to ballot.
levthedev 8y ago
Is it not possible to limit your algorithm's maximum time change for each school? Say, capping the change at 1 hour, and finding the best result of that set of times. Even if it is not as good as the 100% optimized solution, it seems like more palatable while also being an improvement. Presumably, you are already doing that, by limiting the start/end times from 7am-10am, or something similar. In this case, instead of a global time restraint, there would be a local time change restraint +-1 hour for each school.
abalone 8y ago
Not about your research, but any thoughts on SFUSD's assignment algorithm? I ask because it was also designed in consultation with MIT (in part) and also is an example of a very thoughtfully designed school district algorithm[1] that has been met with widespread displeasure in practice and is beginning to be dismantled.[2] [1] http://web.stanford.edu/~niederle/SFUSDBoardPresentationFeb.17.2010.pdf http://web.stanford.edu/~niederle/SFUSDBoardPresentationFeb.... [2] https://news.ycombinator.com/item?id=18063863 https://news.ycombinator.com/item?id=18063863
traviscj 8y ago
I’m curious if an additional constraint like “total (absolute) time shift from this year to next (over all families in the district) <= K” or “max shift in first bell over all schools in district <= MS” was considered, along with the acknowledgement that this is e.g. an N year process to get from the current schedule S_0 to the optimal schedule S^*. In my mind, this alleviates a lot of the “would have to change jobs this year” concerns, because the people making changes would ALREADY be making bigger changes (starting to take their kids to school), and the most extreme schedule changes would be deferred until the children on the existing schedule had aged out of that grade’s current schedule. (Done enough optimization modeling to know this isn’t an easy path... but seems like it has some potential.) Similarly, (maybe this was just the article dumbing down reality a bit, but it sounded like the transit expenditure vs parent happiness units were somewhat fungible. Any possibility of offsetting a bit of upfront expenditure with future savings of the more efficient (say from year N/2 onward) future state?
- traviscj 8y ago
  (Eh, lots of other more succinct discussion along these lines. That’s what I get for commenting before reading! Good luck w/ this work!)

keithpeter 8y ago

Did you think the project was useful in the sense of getting the parents to think about possible changes and rejecting them? So they as citizens are clear that the current system is sub-optimal but they are OK with that

lodrein 8y ago
That's a great question, that I asked myself a lot. I think this project was useful because it started this debate and brought more information to the table. For example, I found that a lot of people did not know how much the start times affected the district costs (tens of millions of dollars every year), or how the schools times seemed to be unequitable. And this Boston Globe article allows even more people to read about this issue and make their own opinion. Ultimately, a lot of people in the US want later start times for high schoolers. This work can provide some information about the ways to implement such a change, as well as the consequences.

opwieurposiu 8y ago

Did the algorithm assume all persons time was equally valuable? From an economics standpoint this is clearly not the case, the range of salaries of public school parents could vary by a factor of 10x or more. To me it seems obvious that the rich parents should be able to pay the poor parents to take the "worse" time slots. This should be beneficial for both parties, but human pride may preclude this as a solution.

lodrein 8y ago
Yes it did. What you are suggesting would be politically very complicated to implement. Also, one need to take into account the characteristics of the district: https://www.bostonpublicschools.org/cms/lib/MA01906464/Centricity/Domain/187/BPS%20at%20a%20Glance%202017-2018.pdf https://www.bostonpublicschools.org/cms/lib/MA01906464/Centr... 66% of the students are economically disadvantaged. Additionally, a lot of schools have both "rich" and "poor" parents, but still need to have a unique time.
gbrown 8y ago
Meanwhile, rich parents work jobs with flexible hours and poor parents are hourly with potential catastrophic penalties for missing work. Theoretically, you're correct (from a parent preference perspective, at least). In practice, even the rich parents would balk at the dollar amount required to make it worth it. That also completely ignores the fact that what you propose would deepen the existing structural inequalities for students, and presupposes that schools are and should remain stratified by income. I vehemently disagree with this unstated premise.
AtlasBarfed 8y ago
And there is the reminder that I'm reading hacker news. From an economics standpoint the ideal labor force is slavery. From an economics standpoint there is no point to the elderly they should be killed off once they are no longer viable laborers. Children are only useful if they show potential slave labor ability, and otherwise should be killed in adolescence if there are obvious shortcomings. From an economics standpoint murder, war, and violence are viable tools in competition, as illustrated by cartel turf wars. So it's OBVIOUS to me I should be allowed to kill you and your kid take your school start time slot if I am economically incentivized to do so / it's convenient aka economic to me and it is up to you to invest in the deterrent to prevent such things. Oh, you're not in favor of such things? I guess you are in favor of regulation, humanity, and other factors in things besides raw "economics".
- twic 8y ago
  This is a huge overreaction - a sort of caricature of the slippery slope argument, attacking points that the parent did not make or even remotely imply, that was presumably intended to signal virtue, but does more to discredit is maker. Distasteful as the parent's comment might be, they are talking about Pareto optimality - finding a situation where both parties would agree it's an improvement.
- PurpleBoxDragon 8y ago
  >From an economics standpoint the ideal labor force is slavery. How so? A happily paid worker with their own free time isn't likely to revolt and kill you. Even an unhappy worker who hates you will just quit. Is your calculations forgetting to count such costs? And a machine that replaces a person is far far more efficient. So machines backed by human operators/fixers/etc. end up outperforming humans, and free humans will have far better education possibilities. >From an economics standpoint there is no point to the elderly they should be killed off once they are no longer viable laborers. You are clearly zeroing out the cost of having all non-elderly realize this will be their fate as well. That seem an arbitrary cost to zero out and makes the results of your calculations unusable. >Children are only useful if they show potential slave labor ability A free child that is educated has a much higher return on investment on average. >otherwise should be killed in adolescence if there are obvious shortcomings Once again completely ignoring the costs of taking such a course of action. I'm not really having any confidence in your models.
roel_v 8y ago
That sort of arrangement incentives people in many unexpected ways, like people optimizing for the chance to get allocated a slot that can be sold at maximum price, rather than what is the 'best' for them overall. It then turns into a game theory exercise of course.
MockObject 8y ago
Excellent point, and a good real-world introduction to the idea of Pareto Efficiency. https://en.wikipedia.org/wiki/Pareto_efficiency https://en.wikipedia.org/wiki/Pareto_efficiency
SamBam 8y ago
One of the reasons that this would be a terrible idea is that the person who stands to make money by selling their good slot for a terrible one is not necessarily the person who has to wake up at 5:00 AM to bus across town. Adding money into the mix, when there's a large difference of means between families, adds a level of incentive that shouldn't be there -- one that has nothing to do with the district's goal of providing an equal education to all its students. When the district looks at the results at the end of the year, and all the poor kids have poor grades/attendance because they were all in the worst time slots, they can't say "well, at least the parents are slightly richer because of this, from the secondary trading market." Rather, based on the objectives they are concerned with, they may have just made inequality even worse.
asr 8y ago
Ah yes, I'm sure the whole thing would have worked politically if only they'd also tried to set up a system in which rich parents paid poor parents in order to get better start times. Conveniently, this would also mean that the students would be income-segregated so rich students would not have to be mixed in with poor students. </snark> ... I also took economics in college. To me it seems obvious that calling a fanciful solution like this "obvious" except for "human pride" is woefully inaccurate.

snerbles 8y ago

Do you feel that your research was sold to the public by the district as a panacea?

lodrein 8y ago
I think changing the start times is something the district had wanted to do for a long time. Therefore once the research and the algorithm suggested it was a good idea, they quickly jumped on board. Now I am sure we have done some communications mistakes. For example the algorithm does not "decide" the start times. You just give it a (long) list of preferences and objectives (e.g., less HS students starting early, more fairness across the population...), and then the algorithm generates thousands of solution that try to satisfy these objectives. Solutions are carefully reviewed, and the objective updated if we think we can do better. So in the end, this research is just a tool, it does not solve all the problems, and in the end people still need to find a way to agree. Hopefully they have a little bit more information to do so!
- platz 8y ago
  > start times is something the district had wanted to do for a long time So, was the real motivation here one of efficiency and cost-savings then? That is a clear thorn they might have wanted to solve. Is the added sleep beneficial to students, all other things equal? I'm sure it is, but was the real motivation a cost savings and the sleep deprivation a nice cover for the real motivation?
  - lodrein 8y ago
    No, as far as I know having later school start times for high schoolers was the main objective, and that was the strongest one in the solution that was chosen by Boston when they used our algorithm. There has been a lot of research on high school students lack of sleep, and it seems that the school times do have a important impact on the students : https://www.cdc.gov/features/school-start-times/index.html https://www.cdc.gov/features/school-start-times/index.html
  - SamBam 8y ago

cowpig 8y ago

I just want to thank you for doing this work. Real-world problems are so much messier and more frustrating than the clean abstractions many of us choose to work with. But this is really meaningful work that should have concrete social benefit. We should all hope to do good as you have.

lodrein 8y ago
Thanks! I am actually very excited about this thread, our main hope is for this project to start a conversation !

Paul-ish 8y ago

Is there an arxiv version of your paper somewhere?

lodrein 8y ago
We submitted it to a journal that has a fast review process, and we are towards the end of it. We did not think soon enough to put it on arxiv, and we should be able to do it soon (it requires some editing to remove the journal-specific things). In the meantime, there is some available information that is listed on my website: https://sebmart.github.io/media.html https://sebmart.github.io/media.html

kwillets 8y ago

How transparent was the process? Are there standards for auditing the data and results, for instance?

lodrein 8y ago
The process went too fast to be entirely transparent. Boston tried to communicate as much as they could, and anything can be requested (Freedom of Information Law). We also have a paper that should not be too far from publication that details every detail of the thing. I would argue that although it could have been more transparent, it is way more transparent than the status quo, where each individual school can try to pressure the district to get the time that they want, without considering the "big picture" and the impact on other schools/neighborhood/budget. All decisions were made after a lot of group conversations with parents/staff/principals... But I am sure some things could have been better, in the end we were just the grad students trying to help :)
- kwillets 8y ago
  Well, ideally people will look at this and say there are clear tradeoffs, but I suspect many will just blame the algorithm.
  - lodrein 8y ago
    This can happen and did happen before. Maybe being suspicious of an algorithm is not a bad things, we should have extremely high standards for them ! My hope is that some people will just have a better view of how complicated this problem is, and be able to construct their own opinion from there. And this would already be something I would be proud of!

coleca 8y ago

Was your project done as part of the A-Lab? http://mitsloan.mit.edu/actionlearning/labs/a-lab.php http://mitsloan.mit.edu/actionlearning/labs/a-lab.php My company had pitched the students of the A-Lab a couple years ago, but unfortunately we didn't get picked. BPS was one of the others pitching the students at the same event.

lodrein 8y ago
No, although the A-lab have a lot of interesting projects. We are part of the Operations Research Center: https://orc.mit.edu/ https://orc.mit.edu/

nhf 8y ago

Hello from across campus! Do you think more user and stakeholder engagement would have helped the public accept your research? I have no idea if BPS has the bandwidth for supporting this type of engagement, but I imagine a lot more fears would be assuaged if parents could have some role in "co-creation" of the algorithm, rather than just accepting results. A lot of research from the fields of citizen science, public understanding of technology, and user experience show that participatory design techniques can enhance acceptance. E.g., https://www.bmj.com/content/319/7212/774.short https://www.bmj.com/content/319/7212/774.short

lodrein 8y ago
Thanks for sharing! I am sure more engagement is always a good thing! One of the tricky tasks of BPS is to make sure that this engagement is "balanced", i.e., everyone's voice is equally heard. And you can imagine how hard that may be in practice. For example, Boston Public Schools had this survey asking people for their preferences on the start times. But most of the answers from these survey came from schools with less economically disadvantaged families. One way to do it is to call the families directly or have a lot of local meetings. But it is still a complicated problem.
- nhf 8y ago
  Thanks for the response. I'm in the technology and policy program within IDSS and we have discussions about these "politics of technology" issues a lot. I'm sure you could become a case study in one of our classes if you wanted :)

dmckeon 8y ago

Did BPS consider applying the changes to only one or a few high schools and their feeder schools? Perhaps the realities of distribution of students among schools in Boston not allow doing such a beta test?

lodrein 8y ago
That could be a way to do it, and Boston decided to do it all at once. I think their reasoning was the fairness of the process (it's hard/unfair to use beta testing on public schools). Also, it is such a hard change that maybe it is easier to do it all at once. You may also want to consider the political implications that are very important when making this kind of decisions.

aero142 8y ago

It seems the amount of change was a large problem. Was it attempted to minimize either number of student start times changed or total delta of minutes changed? Was it discussed how this could be done incrementally to slowly move towards more optimal start times?

lodrein 8y ago
Yes incremental changes looks like a good solution. I think it is hard to do it with the delta minutes changed, as you could have dramatic consequences on the costs (the ripples on the transportation systems can be quite surprising, in that changing the time of one school can have a big impact on the number of buses needed). As to the number of student start time, the main question is how do you choose the schools that change first? How do you make sure that your changes are not interrupted in the middle for political reasons, which would be unfair for the first schools and maybe the whole system. Also, it is an algorithmic challenge to find a "path" of good solutions that go step by step to the good final start times.

jhayward 8y ago

The engagement of the MIT team is interesting, but this reads to me like a classic failure of a "big-bang" rollout, rather than a gradual adoption. Was there any back and forth to create a slower-paced rollout plan?

lodrein 8y ago
We talked about it at some point, and it seems like a good idea. There are two main issues with it: first it can be hard to do a step by step change that is (a) fair (how do you choose the first schools to change?) and (b) works for every step (make sure that the transportation costs do not explode in between).
- jhayward 8y ago
  > it can be hard to do a step by step change that is (a) fair (...) and (b) works for every step Yes. This is what separates the pros from the not-pros in major systems design and operation. The transition plan is the major key to plan, not the algorithm (which plays the role of Shiny Object in this case). It is lots harder, lots, to get smoothly from A to B than to simply big-bang B. It takes enormous attention to detail and thinking through of every angle, and when done well is a thing of wonder. I think in this case your team was not well served by how it was tasked. Thanks again for responding, and for your team's efforts in the design.

drewvolpe 8y ago

One of the changes was moving start times of high schoolers later, which often came at the expense of younger students who would've had to start earlier. In my neighborhood, having to put 5 year olds on the bus at 6am or earlier was a big cause for the uproar from parents. Can you talk about how you made this tradeoff? There's research that suggests high schoolers do better with later start times. But is there any research that backs up doing it at the expense of younger students? ie, is it possible that all students benefit from a later start time?

lodrein 8y ago
I am no specialist of children health, but my understanding is the following: I guess starting too early is never good for any child, but there is something specific with teenagers that makes it a lot worse. There is much more medical literature documenting this fact than for the younger kids. So yes, in a way, given the constraints of costs, the algorithm had to choose the lesser of two evils.
arkh 8y ago
https://www.ncbi.nlm.nih.gov/books/NBK222804/ https://www.ncbi.nlm.nih.gov/books/NBK222804/ > Research findings suggest that changes occur in the “biological clock” during adolescence. As a result, teenagers have a natural tendency to fall asleep later and to wake up later. This is referred to as sleep phase delay. > A correlation was found between subjects' melatonin secretion and their stage of development. The results indicated that melatonin onset occurs later in adolescents, making it difficult for them to go to sleep earlier at night. At the same time, the hormone “turns off” later in the morning, making it harder for them to wake up early (Carskadon et al., 1998, 1999). It's like things are happening to the body during puberty.

siosonel 8y ago

Thanks for working on the technical side of this social-equity project. Was the possibility of parent offering car pools or "walking groups" for kids included in the algorithm? I'm assuming that if volunteer parents are compensated to give carpool rides or lead walking buddies, then the negative impact would be lessened to parents who has to handle drastically different school start/end times. At the same time, it may be cheaper to compensate volunteer parents than for the district to procure or maintain more buses. Another way to ask this question is: were there volunteer transit, ride share, or pre/post-school care incentives included in your algorithm, to balance the potential negative impacts to parents?

lodrein 8y ago
Yes and no. The algorithm runs with historical student data and therefore only considers students using the bus system in order to compute the effects of school start times on transportation. On the other hand I think the district had planned to used the saved transportation money (~18 million/year) in order to help the most impacted parents (pre/post school care...) I guess volunteer transit is always a positive thing for the parents and the districts. The algorithm was just trying to fulfil the objectives given by the district (costs, later students, fairness...). The compensations are what should happen after the algorithm: what do you do with the saved money, how do you help the most impacted parents? These negative effects were expected to disappear with the years as new parents start to choose the schools after the start times have changed (and therefore choose schools that can accomodate their schedules).

lazyasciiart 8y ago

Did anyone suggest making the change, but announcing it today and having it come into effect a year (or even two) from now? That would give parents time to plan adjustments to their schedule, and would give a lot of people the chance to move schools if they really wanted to.

kodablah 8y ago
Or altering the ideal results to be less drastic at least initially to incrementally work towards the goal. Clearly the problem is the size of the change, not what the change is.
mrcodedude 8y ago
> District officials expected some pushback when they released the new school schedule on a Thursday night in December, with plans to implement in the fall of 2018. That's a ~9 month notice?
- lazyasciiart 8y ago
  The first round of applications for schools in Boston is January, so it's only a few weeks (over Christmas!) for families to figure out if they can make it work, if it's worth moving schools, which school they want to move to, etc. (Your chances of getting into your first choice school decline rapidly after the first round).

fdr 8y ago

Don't get discouraged, if you were feeling that way. It was important you substantiated the envelope of what kind of improvements are theoretically possible. And although the wealthier residents, who really ought to have the property to absorb inconvenience, may have been self-serving, they might reasonably have a point that sudden upheaval can cause problems for others that cannot attend meetings. Maybe some will remember this if money for student transportation comes up to ballot.

levthedev 8y ago

Is it not possible to limit your algorithm's maximum time change for each school? Say, capping the change at 1 hour, and finding the best result of that set of times. Even if it is not as good as the 100% optimized solution, it seems like more palatable while also being an improvement. Presumably, you are already doing that, by limiting the start/end times from 7am-10am, or something similar. In this case, instead of a global time restraint, there would be a local time change restraint +-1 hour for each school.

abalone 8y ago

Not about your research, but any thoughts on SFUSD's assignment algorithm? I ask because it was also designed in consultation with MIT (in part) and also is an example of a very thoughtfully designed school district algorithm[1] that has been met with widespread displeasure in practice and is beginning to be dismantled.[2] [1] http://web.stanford.edu/~niederle/SFUSDBoardPresentationFeb.17.2010.pdf http://web.stanford.edu/~niederle/SFUSDBoardPresentationFeb.... [2] https://news.ycombinator.com/item?id=18063863 https://news.ycombinator.com/item?id=18063863

traviscj 8y ago

I’m curious if an additional constraint like “total (absolute) time shift from this year to next (over all families in the district) <= K” or “max shift in first bell over all schools in district <= MS” was considered, along with the acknowledgement that this is e.g. an N year process to get from the current schedule S_0 to the optimal schedule S^*. In my mind, this alleviates a lot of the “would have to change jobs this year” concerns, because the people making changes would ALREADY be making bigger changes (starting to take their kids to school), and the most extreme schedule changes would be deferred until the children on the existing schedule had aged out of that grade’s current schedule. (Done enough optimization modeling to know this isn’t an easy path... but seems like it has some potential.) Similarly, (maybe this was just the article dumbing down reality a bit, but it sounded like the transit expenditure vs parent happiness units were somewhat fungible. Any possibility of offsetting a bit of upfront expenditure with future savings of the more efficient (say from year N/2 onward) future state?

traviscj 8y ago
(Eh, lots of other more succinct discussion along these lines. That’s what I get for commenting before reading! Good luck w/ this work!)