4 ms·

The tweet Closi is referring to: https://twitter.com/noradio/status/1333558513314922497 https://twitter.com/noradio/status/1333558513314922497

meowface 6y ago
And the content, for anyone too lazy to click: Commenter >what's really amazing is that twitter programmers thought about this edge case and made sure the tweet would not display itself Twitter Engineer >We didn't think of this edge case. Someone did this about 7 years ago and the recursive hydration would make a tweet service crash by simply loading the tweet in a browser. It took a principal engineer an entire day of wading through heap dumps to figure out what was happening.
- tantalor 6y ago
  Hydration?
  - plantain 6y ago
    As in, https://ssr.vuejs.org/guide/hydration.html https://ssr.vuejs.org/guide/hydration.html
  - GauntletWizard 6y ago
    Hydrate: (verb, jargon) To populate with metadata or subobjects. i.e. The FriendsList service hydrates each friend object with a list of friends you have in common
- nullsense 6y ago
  TIL: debugging via memory dumps is a Principal Engineer level skill. Anyone here actually do this? I read about it in Release It and it sounds by far like the closest thing there is to a super power when it comes to solving production incidents. I've never actually seen anyone do it though. Recently saw a video on this technique from Dotnet Conf. Piqued my curiosity again, and now this. I've really gotta learn this.
  - voltagex_ 6y ago
    I have done it once successfully in 10 years (.NET dev). Would recommend having any other kind of logging or instrumentation in place so you don't have to do it. It's still worth learning WinDbg and sosclr.
    - tsimionescu 6y ago
      In my company, we used to have a plugin for our bug tracker to automatically analyze .NET core dumps with WinDbg (if they were attached to a bug) and extract some useful information. We used to do this relatively often, for a shipped product, not a live service, especially if we found memory leaks.
      - nullsense 6y ago
        Would you say something like that is worth to set up? I noticed EC2 now has an API to get memory dumps. Theoretically you could automate collecting memory dumps when an unhealthy instance is pulled out of a load balancer. Then some automated analysis could happen, and allow further manual analysis.
        tsimionescu 6y ago
        Not sure how much it cost, but it was definitely helpful - even the fact that it was obvious which team needed to take a look first based on the objects that had leaked often made it worth it.
  - voltagex_ 6y ago
    Any chance you remember which video this was? I can't see it in the dotnet conf 2020 playlist.
    - nullsense 6y ago
      Analyzing Memory Dumps of .NET Applications. https://channel9.msdn.com/Events/dotnetConf/2020/Analyzing-Memory-Dumps-of-NET-Applications https://channel9.msdn.com/Events/dotnetConf/2020/Analyzing-M... I found another one where more detail is gone into on how to script WinDbg to have breakpoints that run code to do stuff. Sounds pretty powerful.

meowface 6y ago

And the content, for anyone too lazy to click: Commenter >what's really amazing is that twitter programmers thought about this edge case and made sure the tweet would not display itself Twitter Engineer >We didn't think of this edge case. Someone did this about 7 years ago and the recursive hydration would make a tweet service crash by simply loading the tweet in a browser. It took a principal engineer an entire day of wading through heap dumps to figure out what was happening.

tantalor 6y ago
Hydration?
- plantain 6y ago
  As in, https://ssr.vuejs.org/guide/hydration.html https://ssr.vuejs.org/guide/hydration.html
- GauntletWizard 6y ago
  Hydrate: (verb, jargon) To populate with metadata or subobjects. i.e. The FriendsList service hydrates each friend object with a list of friends you have in common
nullsense 6y ago
TIL: debugging via memory dumps is a Principal Engineer level skill. Anyone here actually do this? I read about it in Release It and it sounds by far like the closest thing there is to a super power when it comes to solving production incidents. I've never actually seen anyone do it though. Recently saw a video on this technique from Dotnet Conf. Piqued my curiosity again, and now this. I've really gotta learn this.
- voltagex_ 6y ago
  I have done it once successfully in 10 years (.NET dev). Would recommend having any other kind of logging or instrumentation in place so you don't have to do it. It's still worth learning WinDbg and sosclr.
  - tsimionescu 6y ago
    In my company, we used to have a plugin for our bug tracker to automatically analyze .NET core dumps with WinDbg (if they were attached to a bug) and extract some useful information. We used to do this relatively often, for a shipped product, not a live service, especially if we found memory leaks.
    - nullsense 6y ago
      Would you say something like that is worth to set up? I noticed EC2 now has an API to get memory dumps. Theoretically you could automate collecting memory dumps when an unhealthy instance is pulled out of a load balancer. Then some automated analysis could happen, and allow further manual analysis.
      - tsimionescu 6y ago
        Not sure how much it cost, but it was definitely helpful - even the fact that it was obvious which team needed to take a look first based on the objects that had leaked often made it worth it.
- voltagex_ 6y ago
  Any chance you remember which video this was? I can't see it in the dotnet conf 2020 playlist.
  - nullsense 6y ago
    Analyzing Memory Dumps of .NET Applications. https://channel9.msdn.com/Events/dotnetConf/2020/Analyzing-Memory-Dumps-of-NET-Applications https://channel9.msdn.com/Events/dotnetConf/2020/Analyzing-M... I found another one where more detail is gone into on how to script WinDbg to have breakpoints that run code to do stuff. Sounds pretty powerful.