5 ms·

One of the replies is an engineer at Twitter, who confirmed that someone managed to achieve this 7 years ago and it caused issues in the backend.

by Closi 6y ago

OisinMoran 6y ago
The tweet Closi is referring to: https://twitter.com/noradio/status/1333558513314922497 https://twitter.com/noradio/status/1333558513314922497
- meowface 6y ago
  And the content, for anyone too lazy to click: Commenter >what's really amazing is that twitter programmers thought about this edge case and made sure the tweet would not display itself Twitter Engineer >We didn't think of this edge case. Someone did this about 7 years ago and the recursive hydration would make a tweet service crash by simply loading the tweet in a browser. It took a principal engineer an entire day of wading through heap dumps to figure out what was happening.
  - tantalor 6y ago
    Hydration?
    - plantain 6y ago
      As in, https://ssr.vuejs.org/guide/hydration.html https://ssr.vuejs.org/guide/hydration.html
    - GauntletWizard 6y ago
      Hydrate: (verb, jargon) To populate with metadata or subobjects. i.e. The FriendsList service hydrates each friend object with a list of friends you have in common
  - nullsense 6y ago
    TIL: debugging via memory dumps is a Principal Engineer level skill. Anyone here actually do this? I read about it in Release It and it sounds by far like the closest thing there is to a super power when it comes to solving production incidents. I've never actually seen anyone do it though. Recently saw a video on this technique from Dotnet Conf. Piqued my curiosity again, and now this. I've really gotta learn this.
    - voltagex_ 6y ago
      I have done it once successfully in 10 years (.NET dev). Would recommend having any other kind of logging or instrumentation in place so you don't have to do it. It's still worth learning WinDbg and sosclr.
      - tsimionescu 6y ago
        In my company, we used to have a plugin for our bug tracker to automatically analyze .NET core dumps with WinDbg (if they were attached to a bug) and extract some useful information. We used to do this relatively often, for a shipped product, not a live service, especially if we found memory leaks.
        nullsense 6y ago
        Would you say something like that is worth to set up? I noticed EC2 now has an API to get memory dumps. Theoretically you could automate collecting memory dumps when an unhealthy instance is pulled out of a load balancer. Then some automated analysis could happen, and allow further manual analysis.
        tsimionescu 6y ago
        Not sure how much it cost, but it was definitely helpful - even the fact that it was obvious which team needed to take a look first based on the objects that had leaked often made it worth it.
    - voltagex_ 6y ago
koolba 6y ago
It’d be a simple check that anything referenced has to have a lower ID (and hence time stamp). I find this bug more interesting: > Also, it seems like Twitter doesn't actually care about the username and just resolves URLs based on the tweet ID. I'm sure lots of people already knew that but it's new to me. They’re not validating the parent directory matches the actual tweet. I wonder if that’s an actual bug or intentional to allow for handle renaming to not break existing links.
- munk-a 6y ago
  That seems incredibly brittle due to making assumptions about the ID format that twitter has no obligations to keep. It just so happens that if your logic had been if(newId > olderId) you would have survived their new format (due to the fact that the timestamp leads the integer) but that'd be a win based on pure luck. The example the engineer came up with was seven years old so it had no way of foreseeing the ID format change.
  - amichal 6y ago
    It's a good point but I would suspect that the timestamp being prefix is no accident. If your IDs were sortable in the past it's probably a good a idea to keep them sortable. And since tweets more often than not refer to tweets in the same time range or are paginated together with tweets in the same time range (roughly) having time as a prefix has other advantages
    - zimpenfish 6y ago
      > I would suspect that the timestamp being prefix is no accident. Yeah, it's what they created Snowflake for. https://blog.twitter.com/engineering/en_us/a/2010/announcing-snowflake.html https://blog.twitter.com/engineering/en_us/a/2010/announcing...
- numpad0 6y ago
  Can’t be unintentional, else they can’t keep up with users changing IDs.
- 6y ago

One of the replies is an engineer at Twitter, who confirmed that someone managed to achieve this 7 years ago and it caused issues in the backend.

by Closi 6y ago

One of the replies is an engineer at Twitter, who confirmed that someone managed to achieve this 7 years ago and it caused issues in the backend.

OisinMoran 6y ago

The tweet Closi is referring to: https://twitter.com/noradio/status/1333558513314922497 https://twitter.com/noradio/status/1333558513314922497

meowface 6y ago
And the content, for anyone too lazy to click: Commenter >what's really amazing is that twitter programmers thought about this edge case and made sure the tweet would not display itself Twitter Engineer >We didn't think of this edge case. Someone did this about 7 years ago and the recursive hydration would make a tweet service crash by simply loading the tweet in a browser. It took a principal engineer an entire day of wading through heap dumps to figure out what was happening.
- tantalor 6y ago
  Hydration?
  - plantain 6y ago
    As in, https://ssr.vuejs.org/guide/hydration.html https://ssr.vuejs.org/guide/hydration.html
  - GauntletWizard 6y ago
    Hydrate: (verb, jargon) To populate with metadata or subobjects. i.e. The FriendsList service hydrates each friend object with a list of friends you have in common
- nullsense 6y ago
  TIL: debugging via memory dumps is a Principal Engineer level skill. Anyone here actually do this? I read about it in Release It and it sounds by far like the closest thing there is to a super power when it comes to solving production incidents. I've never actually seen anyone do it though. Recently saw a video on this technique from Dotnet Conf. Piqued my curiosity again, and now this. I've really gotta learn this.
  - voltagex_ 6y ago
    I have done it once successfully in 10 years (.NET dev). Would recommend having any other kind of logging or instrumentation in place so you don't have to do it. It's still worth learning WinDbg and sosclr.
    - tsimionescu 6y ago
      In my company, we used to have a plugin for our bug tracker to automatically analyze .NET core dumps with WinDbg (if they were attached to a bug) and extract some useful information. We used to do this relatively often, for a shipped product, not a live service, especially if we found memory leaks.
      - nullsense 6y ago
        Would you say something like that is worth to set up? I noticed EC2 now has an API to get memory dumps. Theoretically you could automate collecting memory dumps when an unhealthy instance is pulled out of a load balancer. Then some automated analysis could happen, and allow further manual analysis.
        tsimionescu 6y ago
        Not sure how much it cost, but it was definitely helpful - even the fact that it was obvious which team needed to take a look first based on the objects that had leaked often made it worth it.
  - voltagex_ 6y ago

koolba 6y ago

It’d be a simple check that anything referenced has to have a lower ID (and hence time stamp). I find this bug more interesting: > Also, it seems like Twitter doesn't actually care about the username and just resolves URLs based on the tweet ID. I'm sure lots of people already knew that but it's new to me. They’re not validating the parent directory matches the actual tweet. I wonder if that’s an actual bug or intentional to allow for handle renaming to not break existing links.

munk-a 6y ago
That seems incredibly brittle due to making assumptions about the ID format that twitter has no obligations to keep. It just so happens that if your logic had been if(newId > olderId) you would have survived their new format (due to the fact that the timestamp leads the integer) but that'd be a win based on pure luck. The example the engineer came up with was seven years old so it had no way of foreseeing the ID format change.
- amichal 6y ago
  It's a good point but I would suspect that the timestamp being prefix is no accident. If your IDs were sortable in the past it's probably a good a idea to keep them sortable. And since tweets more often than not refer to tweets in the same time range or are paginated together with tweets in the same time range (roughly) having time as a prefix has other advantages
  - zimpenfish 6y ago
    > I would suspect that the timestamp being prefix is no accident. Yeah, it's what they created Snowflake for. https://blog.twitter.com/engineering/en_us/a/2010/announcing-snowflake.html https://blog.twitter.com/engineering/en_us/a/2010/announcing...
numpad0 6y ago
Can’t be unintentional, else they can’t keep up with users changing IDs.
6y ago