5 ms·

This mistake happened at Twitter ( https://twitter.com/mattklein123/status/984587433646764032 https://twitter.com/mattklein123/status/984587433646764032 ) - and

This mistake happened at Twitter ( https://twitter.com/mattklein123/status/984587433646764032 https://twitter.com/mattklein123/status/984587433646764032 ) - and people lost jobs.

albertgoeswoof 8y ago
Reading this thread, it looks like the reason management got pissed was because android users didn't log back in after the bug logged them out. So they fired people because of a bug that caused users who don't find enough value in the app login again to stop using the app that they weren't using already. Sounds like twitter has been dying a slow death for a long time.
- ape4 8y ago
  If all programmers get fired for bugs that make it into production ... well you know.
- cosmie 8y ago
  Pretty much. > it looks like the reason management got pissed was because android users didn't log back in after the bug logged them out Q4 of 2015 was the first time they reported[1] a negative figure for Monthly Active Users. They had squeaked by with 0% and 1% growth before, but that quarter broke the dam and reported a negative figure. That last few days of 2015 could have been leveraged to annoy the piss out of people with NYE tweet notifications to get users to inadvertently click on a link and squeak their MAU figure up to the flatline they were used to. With the 40m logged out Android accounts, it probably would have been fairly easy to get that to a flatline number with enough notifications. It's basically hovered around 0-1% ever since, but even then it's only had 2 more quarters of negative quarter over quarter MAU figures reported[2]. Not that it's any excuse. And from the next quarter's results, a lot of users appeared to log back in eventually. Just not in time for that NYE dash to pump up the quarterly numbers. Which is likely a reason management threw such a tantrum to assign blame. [1] The QoQ% for MAUs in the US: http://files.shareholder.com/downloads/AMDA-2F526X/6263295295x0x887900/00A282F9-CADE-410F-92C5-9A185F407667/Q116_Selected_Company_Metrics_and_Financials.pdf http://files.shareholder.com/downloads/AMDA-2F526X/626329529... [2] Q2'17 and Q4'17, also for QoQ% for US only: http://files.shareholder.com/downloads/AMDA-2F526X/6263295295x0x978175/05B95296-0113-41FE-9F7F-FE0B0E4932CE/Q1_2018_Selected_Company_Metrics_and_Financials.pdf http://files.shareholder.com/downloads/AMDA-2F526X/626329529...
- rachelbythebay 8y ago
  "Don't ever log people out" is pretty much the credo for a lot of big things I've seen. A surprising number of folks will never manage to get logged back in. Passwords? They have no idea what they are, or how to reset them. They'll either set up a new account or churn entirely.
  - squiggleblaz 8y ago
    But that doesn't justify firing the programmer who wrote "%G" insetad of "%Y". If they chose to create a stupidly complex login process that involves memorising some string you're never supposed to use (because you should never be logged out), that's a design bug. What kind of idiot designs a system and says "the way to access the system is by using a memorised string" and "you should never log any users out". It's frustrating and it makes no sense. Passwords. They're even stupider than the idiot who works for every single news company, who thought they should resize videos shot on a phone so they can be displayed on a widescreen tv, and then using that on your news app that people only ever use on a phone. It's shocking how few companies will let you log in by sending a token to your phone or your email. It's how I bloody log in anyway, I just have to put up with your stupid password reset process and then be able to three times type some gibberish string that I'm going to forget. As for those who log people out - well, my email is logged in on my phone anyway. If I've lost my phone no-one cares about your dingy little app. And even if they did, they have access to it by virtue of having my email account. Bleh.
pronoiac 8y ago
This, and the thread from the parent tweet, is a fun read!
mtkd 8y ago
If that account is a true reflection of events - it's really disappointing that engineers got fired (edit: or resulted in engineers leaving). A company needing hours to identify/fix a minor code issue that affected 40M people is a senior management failure - not engineering. When I read the initial %G post - I couldn't see how you could/would even test for it - unless you'd heard about such a scenario previously. Rolling heads in engineering for understandable errors just creates a toxic environment for building anything remotely interesting going forward.
- jwilk 8y ago
  Were they actually fired? I assumed that they got tired of the shitshow and quit.
  - Volt 8y ago
    "Blameless postmortems anyone?"
- masklinn 8y ago
  > When I read the initial %G post - I couldn't see how you could/would even test for it - unless you'd heard about such a scenario previously. Interestingly, because IIRC the Gregorian calendar has a 400 years "full" cycle (hence the "Doomsday Rule") depending on the "expense" of your processes you could just use exhaustive testing: there are less than 150k days in 400 gregorian years. Considering exhaustive testing is feasible across 32b ranges (https://randomascii.wordpress.com/2014/01/27/theres-only-four-billion-floatsso-test-them-all/ https://randomascii.wordpress.com/2014/01/27/theres-only-fou...), ~17 bits isn't much of an issue. You would still need to define properties which hold across the board, but then you could just check literally every date of the cycle. This get significantly less feasible if you need to check full calendaring (datetime x timezones).
deleted 8y ago
[deleted]
deleted 8y ago
[deleted]
rachelbythebay 8y ago
Nice find! "Probably because it comes first in the man page". Amazing! The Twitter situation is in fact what got me looking into the problem where I worked, and I patched a great many potential holes. I also left behind things that'll hopefully keep it from being re-introduced.

This mistake happened at Twitter ( https://twitter.com/mattklein123/status/984587433646764032 https://twitter.com/mattklein123/status/984587433646764032 ) - and people lost jobs.

albertgoeswoof 8y ago

Reading this thread, it looks like the reason management got pissed was because android users didn't log back in after the bug logged them out. So they fired people because of a bug that caused users who don't find enough value in the app login again to stop using the app that they weren't using already. Sounds like twitter has been dying a slow death for a long time.

ape4 8y ago
If all programmers get fired for bugs that make it into production ... well you know.
cosmie 8y ago
Pretty much. > it looks like the reason management got pissed was because android users didn't log back in after the bug logged them out Q4 of 2015 was the first time they reported[1] a negative figure for Monthly Active Users. They had squeaked by with 0% and 1% growth before, but that quarter broke the dam and reported a negative figure. That last few days of 2015 could have been leveraged to annoy the piss out of people with NYE tweet notifications to get users to inadvertently click on a link and squeak their MAU figure up to the flatline they were used to. With the 40m logged out Android accounts, it probably would have been fairly easy to get that to a flatline number with enough notifications. It's basically hovered around 0-1% ever since, but even then it's only had 2 more quarters of negative quarter over quarter MAU figures reported[2]. Not that it's any excuse. And from the next quarter's results, a lot of users appeared to log back in eventually. Just not in time for that NYE dash to pump up the quarterly numbers. Which is likely a reason management threw such a tantrum to assign blame. [1] The QoQ% for MAUs in the US: http://files.shareholder.com/downloads/AMDA-2F526X/6263295295x0x887900/00A282F9-CADE-410F-92C5-9A185F407667/Q116_Selected_Company_Metrics_and_Financials.pdf http://files.shareholder.com/downloads/AMDA-2F526X/626329529... [2] Q2'17 and Q4'17, also for QoQ% for US only: http://files.shareholder.com/downloads/AMDA-2F526X/6263295295x0x978175/05B95296-0113-41FE-9F7F-FE0B0E4932CE/Q1_2018_Selected_Company_Metrics_and_Financials.pdf http://files.shareholder.com/downloads/AMDA-2F526X/626329529...
rachelbythebay 8y ago
"Don't ever log people out" is pretty much the credo for a lot of big things I've seen. A surprising number of folks will never manage to get logged back in. Passwords? They have no idea what they are, or how to reset them. They'll either set up a new account or churn entirely.
- squiggleblaz 8y ago
  But that doesn't justify firing the programmer who wrote "%G" insetad of "%Y". If they chose to create a stupidly complex login process that involves memorising some string you're never supposed to use (because you should never be logged out), that's a design bug. What kind of idiot designs a system and says "the way to access the system is by using a memorised string" and "you should never log any users out". It's frustrating and it makes no sense. Passwords. They're even stupider than the idiot who works for every single news company, who thought they should resize videos shot on a phone so they can be displayed on a widescreen tv, and then using that on your news app that people only ever use on a phone. It's shocking how few companies will let you log in by sending a token to your phone or your email. It's how I bloody log in anyway, I just have to put up with your stupid password reset process and then be able to three times type some gibberish string that I'm going to forget. As for those who log people out - well, my email is logged in on my phone anyway. If I've lost my phone no-one cares about your dingy little app. And even if they did, they have access to it by virtue of having my email account. Bleh.

pronoiac 8y ago

This, and the thread from the parent tweet, is a fun read!

mtkd 8y ago

If that account is a true reflection of events - it's really disappointing that engineers got fired (edit: or resulted in engineers leaving). A company needing hours to identify/fix a minor code issue that affected 40M people is a senior management failure - not engineering. When I read the initial %G post - I couldn't see how you could/would even test for it - unless you'd heard about such a scenario previously. Rolling heads in engineering for understandable errors just creates a toxic environment for building anything remotely interesting going forward.

jwilk 8y ago
Were they actually fired? I assumed that they got tired of the shitshow and quit.
- Volt 8y ago
  "Blameless postmortems anyone?"
masklinn 8y ago
> When I read the initial %G post - I couldn't see how you could/would even test for it - unless you'd heard about such a scenario previously. Interestingly, because IIRC the Gregorian calendar has a 400 years "full" cycle (hence the "Doomsday Rule") depending on the "expense" of your processes you could just use exhaustive testing: there are less than 150k days in 400 gregorian years. Considering exhaustive testing is feasible across 32b ranges (https://randomascii.wordpress.com/2014/01/27/theres-only-four-billion-floatsso-test-them-all/ https://randomascii.wordpress.com/2014/01/27/theres-only-fou...), ~17 bits isn't much of an issue. You would still need to define properties which hold across the board, but then you could just check literally every date of the cycle. This get significantly less feasible if you need to check full calendaring (datetime x timezones).

deleted 8y ago

[deleted]

deleted 8y ago

[deleted]

rachelbythebay 8y ago

Nice find! "Probably because it comes first in the man page". Amazing! The Twitter situation is in fact what got me looking into the problem where I worked, and I patched a great many potential holes. I also left behind things that'll hopefully keep it from being re-introduced.