5 ms·

Our CI system broke as a result - I really like this "brownout" idea to help us find it before it turns off for good, but a 24-hour period for us to be broken o

Our CI system broke as a result - I really like this "brownout" idea to help us find it before it turns off for good, but a 24-hour period for us to be broken or scramble to fix is kind of a PITA. I imagine it would be much more technical effort, but a way for us to opt certain repos out of a brownout would be really nice, so that once it happens, we could easily disable the brownout for our repo & schedule working on a fix, while letting us continue working with the existing infra.

vtbassmatt 5y ago
Noted! As you surmise, that's a MUCH bigger lift, but it's worth considering.
- ocdtrekkie 5y ago
  An interesting additional benefit here, is that rather than a brownout, you could just have a "soft cutover", where people can reenable the old protocols for two months. There's good reason for this: - People who aren't using the old methods can turn them off now and leave them off, benefiting from the new security change sooner. - People who need to fix something can temporarily repair their workflow at the time of their choosing. - People who need more than 24 hours to fix their workflow can re-disable the old methods to test that they are now good, at a time of their leisure, between today and March.
humanwhosits 5y ago
Brownouts make me lose confidence in the product, simply because it manifests as a failure that needs to be debugged at unknown cost
- whimsicalism 5y ago
  So do deprecations.
  - dragonwriter 5y ago
    > So do deprecations. Well, feature removals do. Deprecations are just declaring that a feature should not be used and either will or may be removed in the future, which causes no operational problems (and in fact is done specifically to help avoid the operational problems of feature removals.)
    - deleted 5y ago
      [deleted]
- mplewis 5y ago
  You could have avoided this failure by upgrading at any point after September 1 when this change was announced.
- Too 5y ago
  What do you suggest instead? This was announced long ago. Though have to admit I didn’t see it back then myself.
ghshephard 5y ago
One way to opt out of the brownout would have been to switch to new Auth back in September of last year. Or November of last year after the first brown out. Because the next brownout is permanent - and that will be an even bigger PITA. I think it's extraordinarily powerful to have these brownouts for organizations that don't make the change when they should have (which was last year).
- daenney 5y ago
  > One way to opt out of the brownout would have been to switch to new Auth back in September of last year. Or November of last year after the first brown out. The point of the brown out is to help people find cases where they’ve missed this. So “opt out by doing it beforehand” isn’t a viable solution. The last brownout was a quarter ago. A lot of new things can get introduced in that time that still do it wrong.
  - whimsicalism 5y ago
    So why would it be better to learn about those things when the brownout was permanent? Seems like very short-sighted thinking.
    - deleted 5y ago
      [deleted]
    - thayne 5y ago
      I think the idea is that once you know what is going on you can opt out, so that you can put fixing it on your next sprint instead of having to drop everything and fix it right now so you don't lose 24 hours of productivity.
      - ghshephard 5y ago
        I 100% understand that - The point I'm trying to make is- the next "brown out" is permanent - so instead of 24 hours, next time you will have a permanent loss of productivity. I.E. Why are there devices being deployed that are still broken? Because in a few months, they are never coming back until they are fixed. The idea is to make brownouts increasingly painful - just letting them be a short period of time, or let people "Opt. Out" doesn't service the purpose here - which is to make it absolutely clear that the service is going away.
dragonwriter 5y ago
> a 24-hour period for us to be broken or scramble to fix is kind of a PITA. Brownouts need to be a PITA otherwise people are too likely to miss them or write them off as transient errors. > I imagine it would be much more technical effort, but a way for us to opt certain repos out of a brownout would be really nice That's not a bad idea, though.

vtbassmatt 5y ago

Noted! As you surmise, that's a MUCH bigger lift, but it's worth considering.

ocdtrekkie 5y ago
An interesting additional benefit here, is that rather than a brownout, you could just have a "soft cutover", where people can reenable the old protocols for two months. There's good reason for this: - People who aren't using the old methods can turn them off now and leave them off, benefiting from the new security change sooner. - People who need to fix something can temporarily repair their workflow at the time of their choosing. - People who need more than 24 hours to fix their workflow can re-disable the old methods to test that they are now good, at a time of their leisure, between today and March.

humanwhosits 5y ago

Brownouts make me lose confidence in the product, simply because it manifests as a failure that needs to be debugged at unknown cost

whimsicalism 5y ago
So do deprecations.
- dragonwriter 5y ago
  > So do deprecations. Well, feature removals do. Deprecations are just declaring that a feature should not be used and either will or may be removed in the future, which causes no operational problems (and in fact is done specifically to help avoid the operational problems of feature removals.)
  - deleted 5y ago
    [deleted]
mplewis 5y ago
You could have avoided this failure by upgrading at any point after September 1 when this change was announced.
Too 5y ago
What do you suggest instead? This was announced long ago. Though have to admit I didn’t see it back then myself.

ghshephard 5y ago

One way to opt out of the brownout would have been to switch to new Auth back in September of last year. Or November of last year after the first brown out. Because the next brownout is permanent - and that will be an even bigger PITA. I think it's extraordinarily powerful to have these brownouts for organizations that don't make the change when they should have (which was last year).

daenney 5y ago
> One way to opt out of the brownout would have been to switch to new Auth back in September of last year. Or November of last year after the first brown out. The point of the brown out is to help people find cases where they’ve missed this. So “opt out by doing it beforehand” isn’t a viable solution. The last brownout was a quarter ago. A lot of new things can get introduced in that time that still do it wrong.
- whimsicalism 5y ago
  So why would it be better to learn about those things when the brownout was permanent? Seems like very short-sighted thinking.
  - deleted 5y ago
    [deleted]
  - thayne 5y ago
    I think the idea is that once you know what is going on you can opt out, so that you can put fixing it on your next sprint instead of having to drop everything and fix it right now so you don't lose 24 hours of productivity.
    - ghshephard 5y ago
      I 100% understand that - The point I'm trying to make is- the next "brown out" is permanent - so instead of 24 hours, next time you will have a permanent loss of productivity. I.E. Why are there devices being deployed that are still broken? Because in a few months, they are never coming back until they are fixed. The idea is to make brownouts increasingly painful - just letting them be a short period of time, or let people "Opt. Out" doesn't service the purpose here - which is to make it absolutely clear that the service is going away.

dragonwriter 5y ago

> a 24-hour period for us to be broken or scramble to fix is kind of a PITA. Brownouts need to be a PITA otherwise people are too likely to miss them or write them off as transient errors. > I imagine it would be much more technical effort, but a way for us to opt certain repos out of a brownout would be really nice That's not a bad idea, though.