8 ms·

I don't understand the social structure inside a software company where this kind of thing can go from some intern's 3am idea to production, without passing man

I don't understand the social structure inside a software company where this kind of thing can go from some intern's 3am idea to production, without passing many layers of gatekeepers, any one of which should have swiftly flagged this down. It's not that the string replacement was implemented wrongly (that too)—it's that they're touching, in any manner at all, one of the most obviously-sensitive UX things in their product. Without a commensurate amount of security review. Like, in my imagination, within five minutes of anyone seeing this, a person with responsibility would have stepped in and said "No, you can't do this. And if you insist on doing this, here's five layers of audits and sign-offs that this needs to go through first, because the thing you're proposing is potentially really dangerous". Am I thinking it about it wrongly? I cannot understand at all.

MattGaiser 3y ago
> without passing many layers of gatekeepers Given all the Twitter bugs and issues, it seems they have all been laid off.
scblock 3y ago
> some intern's 3am idea I think you mean "Elon Musks random demand".
greenthrow 3y ago
Anyone who would tell Elon "this isn't a good idea" left or was fired.
zettabomb 3y ago
>I don't understand the social structure There is none. Stop trying to understand - it's a fool's errand. >a person with responsibility would have stepped in There are none left. They were either laid off, or they left before that could happen. >"No, you can't do this" ... is the last thing anyone says to Elon Musk before getting fired.
- deleted 3y ago
  [deleted]
rtkwe 3y ago
Well it starts with the bad idea probably coming from the top, Musk saying I'm tired of seeing twitter.com links change them all to look like x.com links, that plus his gutting of the company when he took over means there's less people around to be the person to say no you can't do this this way go back and start over (or not at all).
- quandrum 3y ago
  It's probably worse with Musk. His executive style seems to be, from his biography, ignore a thing for a while until he gets in a maniac phase and then over the shoulder manage a thing until it's done, regardless of time or context. I can just take the scene of Musk being on a roof yelling at the crew to change how they install solar tiles late in the night and translate it to him berating a programmer in the office to make it look like x.com and not caring about the details.
pjc50 3y ago
> Like, in my imagination, within five minutes of anyone seeing this, a person with responsibility would have stepped in and said "No, you can't do this. And if you insist on doing this, here's five layers of audits and sign-offs that this needs to go through first, because the thing you're proposing is potentially really dangerous". Am I thinking it about it wrongly? Which part of "anyone who is not a Musk yes-man has already been fired or quit" are you having trouble with?
- lenerdenator 3y ago
  It's worth remembering, there are two kinds of yes-men: 1) the sycophant who loves authoritarian institutions; the "true believers" and 2) the young, brilliant visa holder who was the talk of his parents' social circle in Hyderabad three years ago when he graduated and was able to get on board at a household-name NorCal tech company, but who is now being abused by the employer who sponsors the thing that lets him stay in the US. You'll always have type one; some humans simply love following a dolt. The second type is a result of our laws, and laws can be changed to keep people like Elon from taking advantage of workers.
  - triceratops 3y ago
    The visa holder thing might have been true for maybe 3 months before and after the takeover. The job market was absolutely on fire at the time (early 2022). Anyone who wanted to leave should have been able to, especially considering they were good enough to be hired at Twitter.
    - clipsy 3y ago
      > The visa holder thing might have been true for maybe 3 months before and after the takeover. The job market was absolutely on fire at the time (early 2022). The Twitter takeover (and subsequent layoffs, ultimatum, etc) happened in late 2022.[0] [0]: https://en.wikipedia.org/wiki/Acquisition_of_Twitter_by_Elon_Musk https://en.wikipedia.org/wiki/Acquisition_of_Twitter_by_Elon...
deleted 3y ago
[deleted]
segasaturn 3y ago
Well, that is the specific outcome Elon wanted when he laid off three-quarters of the company. He very clearly stated that he didn't see the value of the "trust and safety" teams that would have been the ones to flag something like this down.
- ceejayoz 3y ago
  This wouldn't have fallen within "trust and safety" purview. This should've been caught in even the most cursory of code reviews.
  - somenameforme 3y ago
    It seems this may have only affected the X on iOS App. [1] That greatly expands the range of possible causes. It also makes this quite odd in another way as well, because it suggests this was not a server side change. [1] - https://news.ycombinator.com/item?id=39991312 https://news.ycombinator.com/item?id=39991312
  - sangnoir 3y ago
    Why wouldn't phishing fall under trust and safety? Perhaps not primarily - but it's definitely a trust and safety event.
    - ceejayoz 3y ago
      They wouldn't "have been the ones to flag something like this down". It should've been caught long prior; the code shouldn't have been written, it shouldn't have survived code review, and it should've failed automated tests. It's a trust and safety incident now, but it never should have been.
    - blasphemers 3y ago
      The trust and safety team was only responsible for censoring wrongthink
lokar 3y ago
Go re-read all the comments here during the take over and layoff where people claimed it could not possibly take more then a handful of people to run such a simple site.
- misiti3780 3y ago
  he fired a huge number of the staff and the site is still running, so how was that assumption not proven correct ?
  - lokar 3y ago
    [flagged]
  - smith7018 3y ago
    I use twitter daily and the site is a shell of its former self. It's slow, prone to bugs, filled with bots, the amount of real users has cratered, user reports go nowhere, there's no support team, the ads are now bot accounts posting crap like "Today is a good day, be sure to make it advantageous", there are no new features besides previously in-flight projects pre-Musk, they've actually removed a lot of features (like Circles, block lists, etc), and much more. He took an otherwise functioning social media service and forced it into maintenance mode. He also fired all of the people that keep the user base alive so now it's flooded with bots (which he presumably likes so he can boast about engagement being up). So yes it's still around but it's dying and the skeleton crew he has left can't do anything. In other words, he destroyed it.
    - mschuster91 3y ago
      The bot plague is atrocious. Like, there are tons of "keyword watcher" bots... write "onlyfans" and you'll get ~5-10 spambots in under half a minute, and for stuff involving popular politicians or political events (anything Russia, Ukraine, Israel, Palestine, Covid) you'll get Russian fake newspaper clones. On top of that come the "human bots" - write the name of infamous German youtuber "Drachenlord" and you'll get that vile hater bunch and it's just the same.
    - deleted 3y ago
rsynnott 3y ago
I mean, I assume anyone who could get another job has already left, so they're probably running low on competent gatekeepers.
swid 3y ago
Don't take this as a defense of what is a harebrained idea; but this kind of replacement should be easy to do correctly. You know; in such a way where only real twitter.com links are changed to x.com. Honestly it is only somewhat surprising to me that no one noticed the error ahead of time. On the one hand, this is the type of mistake I do see in reviews from time to time... usually in the form of a regex that is not anchored to the start of the string, or perhaps it uses a non escaped period which of course means "any character" in regex. On the other hand, it is revealing about the kinds of controls in place that it got through.
- notatoad 3y ago
  even things that are easy (or maybe, especially things that are easy) can benefit from having a review to make sure you didn't miss something obvious. nothing more dangerous than "oh, that's easy, let's push it straight to prod"
  - seanhunter 3y ago
    Example: one of the worst and most costly bugs I ever saw during my time in finance involved a code review [1] with the exchange: "Looks good generally but this will serialize incorrectly if a date is ever negative." <... pause while everyone digests that...> "Why would a date ever be negative?" Noone could think of a reason a date would ever be negative. The resulting code push cost millions in a single day. [1] I was on the call because I was the SRE[2] responsible for doing the release [2] We didn't call ourselves SREs because it was like 2001 and that terminology hadn't caught on yet. We called ourselves "Installmeisters" believe it or not.
    - zestyping 3y ago
      How did it turn out that a date was negative, and how did it cost millions?
      - seanhunter 3y ago
        Iirc it turned out some 20+ year old code was relying on an undocumented/unknown behaviour of dates when they are negative to calculate cashflows on very old perpetual fixed income instruments[1]. If you have a codebase that's large enough and worked on by enough people, for every possible hack/undocumented feature you'll have at least one person who will rely on it in some way. It cost millions, because the serialization corrupted the USD treasury curve when we wrote it to the database, which meant that even though the code was reverted quickly, for a while no derivatives (like literally no derivatives) in the whole of the trading part of a big IB could price (because essentially all derivatives were priced in dollars using some sort of discounted cash flow mechanism which relies on the USD treasury curve). No price means no risk, because how you calculate risk of a derivative is by finding the underlyers, shocking those underlyers a little bit and calculating the new price (essentially an empirical method of finding the derivative of price with respect to each of those things). [1] https://en.wikipedia.org/wiki/Perpetual_bond https://en.wikipedia.org/wiki/Perpetual_bond
deleted 3y ago
[deleted]
deleted 3y ago
[deleted]
renewiltord 3y ago
Well, that’s because any org that has five layers of audits for this just has five layers of audits for everything and so rarely gets anything done. This is a clbuttic bug. It’s silly and damaging but easy to fix and move on with. Not particularly different from Bluesky allowing one guy to own all of S3.
sangnoir 3y ago
> I don't understand the social structure inside a software company where this kind of thing can go from some intern's 3am idea to production This is what happens when you "cut the fat" and are left with an adversely-selected[1], skeleton crew of "hard core engineers." The site was never going to fail all at once, instead, it's a death by a thousand cuts and suboptimal engineering. 1. No disrespect to current Twitter engineers who can't leave easily, or believe in the mew mission. However those who survived layoffs but could leave have left.
SmartJerry 3y ago
This article was 5 hours old at the time I'm viewing it and the bug is supposedly already fixed per the article itself. So yea, seems like this was probably fixed within 5 minutes of anyone noticing.

MattGaiser 3y ago

> without passing many layers of gatekeepers Given all the Twitter bugs and issues, it seems they have all been laid off.

scblock 3y ago

> some intern's 3am idea I think you mean "Elon Musks random demand".

greenthrow 3y ago

Anyone who would tell Elon "this isn't a good idea" left or was fired.

zettabomb 3y ago

>I don't understand the social structure There is none. Stop trying to understand - it's a fool's errand. >a person with responsibility would have stepped in There are none left. They were either laid off, or they left before that could happen. >"No, you can't do this" ... is the last thing anyone says to Elon Musk before getting fired.

deleted 3y ago
[deleted]

rtkwe 3y ago

Well it starts with the bad idea probably coming from the top, Musk saying I'm tired of seeing twitter.com links change them all to look like x.com links, that plus his gutting of the company when he took over means there's less people around to be the person to say no you can't do this this way go back and start over (or not at all).

quandrum 3y ago
It's probably worse with Musk. His executive style seems to be, from his biography, ignore a thing for a while until he gets in a maniac phase and then over the shoulder manage a thing until it's done, regardless of time or context. I can just take the scene of Musk being on a roof yelling at the crew to change how they install solar tiles late in the night and translate it to him berating a programmer in the office to make it look like x.com and not caring about the details.

pjc50 3y ago

> Like, in my imagination, within five minutes of anyone seeing this, a person with responsibility would have stepped in and said "No, you can't do this. And if you insist on doing this, here's five layers of audits and sign-offs that this needs to go through first, because the thing you're proposing is potentially really dangerous". Am I thinking it about it wrongly? Which part of "anyone who is not a Musk yes-man has already been fired or quit" are you having trouble with?

lenerdenator 3y ago
It's worth remembering, there are two kinds of yes-men: 1) the sycophant who loves authoritarian institutions; the "true believers" and 2) the young, brilliant visa holder who was the talk of his parents' social circle in Hyderabad three years ago when he graduated and was able to get on board at a household-name NorCal tech company, but who is now being abused by the employer who sponsors the thing that lets him stay in the US. You'll always have type one; some humans simply love following a dolt. The second type is a result of our laws, and laws can be changed to keep people like Elon from taking advantage of workers.
- triceratops 3y ago
  The visa holder thing might have been true for maybe 3 months before and after the takeover. The job market was absolutely on fire at the time (early 2022). Anyone who wanted to leave should have been able to, especially considering they were good enough to be hired at Twitter.
  - clipsy 3y ago
    > The visa holder thing might have been true for maybe 3 months before and after the takeover. The job market was absolutely on fire at the time (early 2022). The Twitter takeover (and subsequent layoffs, ultimatum, etc) happened in late 2022.[0] [0]: https://en.wikipedia.org/wiki/Acquisition_of_Twitter_by_Elon_Musk https://en.wikipedia.org/wiki/Acquisition_of_Twitter_by_Elon...

deleted 3y ago

[deleted]

segasaturn 3y ago

Well, that is the specific outcome Elon wanted when he laid off three-quarters of the company. He very clearly stated that he didn't see the value of the "trust and safety" teams that would have been the ones to flag something like this down.

ceejayoz 3y ago
This wouldn't have fallen within "trust and safety" purview. This should've been caught in even the most cursory of code reviews.
- somenameforme 3y ago
  It seems this may have only affected the X on iOS App. [1] That greatly expands the range of possible causes. It also makes this quite odd in another way as well, because it suggests this was not a server side change. [1] - https://news.ycombinator.com/item?id=39991312 https://news.ycombinator.com/item?id=39991312
- sangnoir 3y ago
  Why wouldn't phishing fall under trust and safety? Perhaps not primarily - but it's definitely a trust and safety event.
  - ceejayoz 3y ago
    They wouldn't "have been the ones to flag something like this down". It should've been caught long prior; the code shouldn't have been written, it shouldn't have survived code review, and it should've failed automated tests. It's a trust and safety incident now, but it never should have been.
  - blasphemers 3y ago
    The trust and safety team was only responsible for censoring wrongthink

lokar 3y ago

Go re-read all the comments here during the take over and layoff where people claimed it could not possibly take more then a handful of people to run such a simple site.

misiti3780 3y ago
he fired a huge number of the staff and the site is still running, so how was that assumption not proven correct ?
- lokar 3y ago
  [flagged]
- smith7018 3y ago
  I use twitter daily and the site is a shell of its former self. It's slow, prone to bugs, filled with bots, the amount of real users has cratered, user reports go nowhere, there's no support team, the ads are now bot accounts posting crap like "Today is a good day, be sure to make it advantageous", there are no new features besides previously in-flight projects pre-Musk, they've actually removed a lot of features (like Circles, block lists, etc), and much more. He took an otherwise functioning social media service and forced it into maintenance mode. He also fired all of the people that keep the user base alive so now it's flooded with bots (which he presumably likes so he can boast about engagement being up). So yes it's still around but it's dying and the skeleton crew he has left can't do anything. In other words, he destroyed it.
  - mschuster91 3y ago
    The bot plague is atrocious. Like, there are tons of "keyword watcher" bots... write "onlyfans" and you'll get ~5-10 spambots in under half a minute, and for stuff involving popular politicians or political events (anything Russia, Ukraine, Israel, Palestine, Covid) you'll get Russian fake newspaper clones. On top of that come the "human bots" - write the name of infamous German youtuber "Drachenlord" and you'll get that vile hater bunch and it's just the same.
  - deleted 3y ago

rsynnott 3y ago

I mean, I assume anyone who could get another job has already left, so they're probably running low on competent gatekeepers.

swid 3y ago

Don't take this as a defense of what is a harebrained idea; but this kind of replacement should be easy to do correctly. You know; in such a way where only real twitter.com links are changed to x.com. Honestly it is only somewhat surprising to me that no one noticed the error ahead of time. On the one hand, this is the type of mistake I do see in reviews from time to time... usually in the form of a regex that is not anchored to the start of the string, or perhaps it uses a non escaped period which of course means "any character" in regex. On the other hand, it is revealing about the kinds of controls in place that it got through.

notatoad 3y ago
even things that are easy (or maybe, especially things that are easy) can benefit from having a review to make sure you didn't miss something obvious. nothing more dangerous than "oh, that's easy, let's push it straight to prod"
- seanhunter 3y ago
  Example: one of the worst and most costly bugs I ever saw during my time in finance involved a code review [1] with the exchange: "Looks good generally but this will serialize incorrectly if a date is ever negative." <... pause while everyone digests that...> "Why would a date ever be negative?" Noone could think of a reason a date would ever be negative. The resulting code push cost millions in a single day. [1] I was on the call because I was the SRE[2] responsible for doing the release [2] We didn't call ourselves SREs because it was like 2001 and that terminology hadn't caught on yet. We called ourselves "Installmeisters" believe it or not.
  - zestyping 3y ago
    How did it turn out that a date was negative, and how did it cost millions?
    - seanhunter 3y ago
      Iirc it turned out some 20+ year old code was relying on an undocumented/unknown behaviour of dates when they are negative to calculate cashflows on very old perpetual fixed income instruments[1]. If you have a codebase that's large enough and worked on by enough people, for every possible hack/undocumented feature you'll have at least one person who will rely on it in some way. It cost millions, because the serialization corrupted the USD treasury curve when we wrote it to the database, which meant that even though the code was reverted quickly, for a while no derivatives (like literally no derivatives) in the whole of the trading part of a big IB could price (because essentially all derivatives were priced in dollars using some sort of discounted cash flow mechanism which relies on the USD treasury curve). No price means no risk, because how you calculate risk of a derivative is by finding the underlyers, shocking those underlyers a little bit and calculating the new price (essentially an empirical method of finding the derivative of price with respect to each of those things). [1] https://en.wikipedia.org/wiki/Perpetual_bond https://en.wikipedia.org/wiki/Perpetual_bond

deleted 3y ago

[deleted]

deleted 3y ago

[deleted]

renewiltord 3y ago

Well, that’s because any org that has five layers of audits for this just has five layers of audits for everything and so rarely gets anything done. This is a clbuttic bug. It’s silly and damaging but easy to fix and move on with. Not particularly different from Bluesky allowing one guy to own all of S3.

sangnoir 3y ago

> I don't understand the social structure inside a software company where this kind of thing can go from some intern's 3am idea to production This is what happens when you "cut the fat" and are left with an adversely-selected[1], skeleton crew of "hard core engineers." The site was never going to fail all at once, instead, it's a death by a thousand cuts and suboptimal engineering. 1. No disrespect to current Twitter engineers who can't leave easily, or believe in the mew mission. However those who survived layoffs but could leave have left.

SmartJerry 3y ago

This article was 5 hours old at the time I'm viewing it and the bug is supposedly already fixed per the article itself. So yea, seems like this was probably fixed within 5 minutes of anyone noticing.