6 ms·

Reminds me of the HFT shop that built in Java and simply turned the garbage collector off. Then when the market closed they would restart the process for the ne

Reminds me of the HFT shop that built in Java and simply turned the garbage collector off. Then when the market closed they would restart the process for the next day.

HideousKojima 3y ago
Or the missile firmware where the missile is going to explode before they run out of memory: https://devblogs.microsoft.com/oldnewthing/20180228-00/?p=98125 https://devblogs.microsoft.com/oldnewthing/20180228-00/?p=98...
- jnwatson 3y ago
  There was a certain Space Shuttle program that never returned from interrupt. At the end of its interrupt handler, it would just spin. The next interrupt would just add to the call stack. What happens when it runs out of stack? It just resets the stack pointer back to the start.
- tetromino_ 3y ago
  Using warhead explosion as garbage collector might seem like a clever hack, but all it takes is an upgrade by a different team (say, adding a new engine, or longer-range sensors, or repurposing a surface-to-air missile for a surface-to-surface role), and suddenly your guidance software runs out of memory before it explodes, and your missile falls onto an an elementary school or hospital.
  - HideousKojima 3y ago
    You can probably double the ram (or more) again by then.
    - LtWorf 3y ago
      Assuming you know you have to do it.
  - shortrounddev2 3y ago
    I assume they would test the missiles before attesting to its range
  - wegfawefgawefg 3y ago
    But sir!! They were nazi orphans!
  - zlsa 3y ago
    Related: the Ariane 5 rocket failed on its maiden flight because it reused code from its predecessor, which made an assumption that didn't apply to the higher-performance Ariane 5. https://en.wikipedia.org/wiki/Ariane_flight_V88 https://en.wikipedia.org/wiki/Ariane_flight_V88
    - h0l0cube 3y ago
      Seems like a mitigation for this kind of issue would be to simulate the inputs to the system for the expected flight duration.
      - kitd 3y ago
        IIRC it was the higher acceleration value of Ariane5 that the old code wasn't expecting that caused the problem. Some fuzz testing could have worked, I suppose.
        h0l0cube 3y ago
        I would imagine a simulation would factor in the higher acceleration
        mpweiher 3y ago
        Except it was worse: 1. The module that failed actually was no longer in use for that part of the flight. 2. The acceleration for the part of the flight where it was in use was within the parameters. 3. Instead of just ignoring / clamping the out-of-bounds values, the no-longer-needed module sent a big error dump. 4. That error dump was sent to the next module in-line, which was expecting...I think numbers, but certainly not big textual error dumps. And so that failed as well. 5. I don't know if that second module was needed. So had the module just (a) processed the incoming data normally or (b) clamped the values silently or (c) dropped the out-of-bounds values silently, the Ariane 5 would not have exploded. Instead it did the "make it impossible to represent invalid states"-thing and exploded. While I understand the appeal of that idea, I think it is overrated with any software that has to interact in some way shape or form with the real world, however indirectly. Because programmers tend to have a pretty limited understanding of what states are valid or invalid in the real world. (See also: the "falsehoods programmers believe about XXX" series)
  - LtWorf 3y ago
    > and your missile falls onto an an elementary school or hospital. Then you just say terrorists were hiding inside…
- queuebert 3y ago
  God, that email takes me back to days when email was much higher signal-to-noise and sig lines were for color, not credentials.
deleted 3y ago
[deleted]
ahoka 3y ago
It’s available from Java 11 as the Epsilon GC.
tombert 3y ago
I had a friend who worked for one of the big Market Makers, and he told me that they would indeed turn the GC off, but what they'd do is just pre-allocate everything into bigass arrays before-hand, and have incrementers to simulate the "new" keyword. They might do this in something more or less like a threadlocal to avoid having to deal with locks or race conditions or anything like that.
- shortrounddev2 3y ago
  A lot of game engines do this, avoiding the cost of a malloc lookup by just allocating everything as a stack
  - tarnith 3y ago
    Seen some audio engines that do this too. You can come up with an elegant non-blocking lock free allocator and pass updates from the user thread out to system and reserve the memory, or you can just up front allocate everything that's needed. If it's a fixed function synth, sometimes just allocating everything up front makes more sense.
    - gabesk 3y ago
      Kernel drivers, too. Especially the DMA buffers for devices that don’t support scatter gather: if you don’t allocate them at driver startup, you might not be able to find enough free contiguous regions later. Although maybe that’s not as big of a deal these days, since both Windows and Linux can relocate physical pages.
  - wtracy 3y ago
    I remember seeing a Java game engine that included a container that you were supposed to dump unused objects into to prevent GC from running in the middle of a level. At the end of the level, you could empty the container, and let the GC run during the loading screen or whatever. It was a deliberate design choice that was surprisingly close to what the original article describes.
- 3y ago
bluGill 3y ago
it is somewhat common in garbage collected langages to fight the garrbage collector like that. Sure manual free probably adds up to more CPU time, but it is more spread out and thus not noticable (normally, real time still cannot allocate in the sensitive areas)
lifthrasiir 3y ago
Many online games regularly schedule a downtime often to simply restart the process and "solve" moderate memory leaks.
JB_Dev 3y ago
I’ve done something similar in one of our production services. There was a problem with extremely long GC pauses during Gen2 garbage collection (.NET uses a multigenerational GC design). Pauses could be many seconds long or more than a minute in extreme cases. We found the underlying issue that caused memory pressure in the Gen2 region but the fix was to change some very fundamental aspects of the service and would need to have some significant refactoring. Since this was a legacy service (.net framework) that we were refactoring anyway to run in new .NET (5+), we decided to ignore the issue. Instead we adjusted the GC to just never do the expensive Gen2 collections (GCLatencyMode) and moved the service to run on higher memory VMs. It would hit OOM every 3 days or so, so we just set instances to auto-restart once a day. Then 1 year later we deployed the replacement for the legacy service and the problem was solved.
aidenn0 3y ago
Early lisp machines had garage collectors that were so slow, that this was done on workstations.

Reminds me of the HFT shop that built in Java and simply turned the garbage collector off. Then when the market closed they would restart the process for the next day.

HideousKojima 3y ago

Or the missile firmware where the missile is going to explode before they run out of memory: https://devblogs.microsoft.com/oldnewthing/20180228-00/?p=98125 https://devblogs.microsoft.com/oldnewthing/20180228-00/?p=98...

jnwatson 3y ago
There was a certain Space Shuttle program that never returned from interrupt. At the end of its interrupt handler, it would just spin. The next interrupt would just add to the call stack. What happens when it runs out of stack? It just resets the stack pointer back to the start.
tetromino_ 3y ago
Using warhead explosion as garbage collector might seem like a clever hack, but all it takes is an upgrade by a different team (say, adding a new engine, or longer-range sensors, or repurposing a surface-to-air missile for a surface-to-surface role), and suddenly your guidance software runs out of memory before it explodes, and your missile falls onto an an elementary school or hospital.
- HideousKojima 3y ago
  You can probably double the ram (or more) again by then.
  - LtWorf 3y ago
    Assuming you know you have to do it.
- shortrounddev2 3y ago
  I assume they would test the missiles before attesting to its range
- wegfawefgawefg 3y ago
  But sir!! They were nazi orphans!
- zlsa 3y ago
  Related: the Ariane 5 rocket failed on its maiden flight because it reused code from its predecessor, which made an assumption that didn't apply to the higher-performance Ariane 5. https://en.wikipedia.org/wiki/Ariane_flight_V88 https://en.wikipedia.org/wiki/Ariane_flight_V88
  - h0l0cube 3y ago
    Seems like a mitigation for this kind of issue would be to simulate the inputs to the system for the expected flight duration.
    - kitd 3y ago
      IIRC it was the higher acceleration value of Ariane5 that the old code wasn't expecting that caused the problem. Some fuzz testing could have worked, I suppose.
      - h0l0cube 3y ago
        I would imagine a simulation would factor in the higher acceleration
      - mpweiher 3y ago
        Except it was worse: 1. The module that failed actually was no longer in use for that part of the flight. 2. The acceleration for the part of the flight where it was in use was within the parameters. 3. Instead of just ignoring / clamping the out-of-bounds values, the no-longer-needed module sent a big error dump. 4. That error dump was sent to the next module in-line, which was expecting...I think numbers, but certainly not big textual error dumps. And so that failed as well. 5. I don't know if that second module was needed. So had the module just (a) processed the incoming data normally or (b) clamped the values silently or (c) dropped the out-of-bounds values silently, the Ariane 5 would not have exploded. Instead it did the "make it impossible to represent invalid states"-thing and exploded. While I understand the appeal of that idea, I think it is overrated with any software that has to interact in some way shape or form with the real world, however indirectly. Because programmers tend to have a pretty limited understanding of what states are valid or invalid in the real world. (See also: the "falsehoods programmers believe about XXX" series)
- LtWorf 3y ago
  > and your missile falls onto an an elementary school or hospital. Then you just say terrorists were hiding inside…
queuebert 3y ago
God, that email takes me back to days when email was much higher signal-to-noise and sig lines were for color, not credentials.

deleted 3y ago

[deleted]

ahoka 3y ago

It’s available from Java 11 as the Epsilon GC.

tombert 3y ago

I had a friend who worked for one of the big Market Makers, and he told me that they would indeed turn the GC off, but what they'd do is just pre-allocate everything into bigass arrays before-hand, and have incrementers to simulate the "new" keyword. They might do this in something more or less like a threadlocal to avoid having to deal with locks or race conditions or anything like that.

shortrounddev2 3y ago
A lot of game engines do this, avoiding the cost of a malloc lookup by just allocating everything as a stack
- tarnith 3y ago
  Seen some audio engines that do this too. You can come up with an elegant non-blocking lock free allocator and pass updates from the user thread out to system and reserve the memory, or you can just up front allocate everything that's needed. If it's a fixed function synth, sometimes just allocating everything up front makes more sense.
  - gabesk 3y ago
    Kernel drivers, too. Especially the DMA buffers for devices that don’t support scatter gather: if you don’t allocate them at driver startup, you might not be able to find enough free contiguous regions later. Although maybe that’s not as big of a deal these days, since both Windows and Linux can relocate physical pages.
- wtracy 3y ago
  I remember seeing a Java game engine that included a container that you were supposed to dump unused objects into to prevent GC from running in the middle of a level. At the end of the level, you could empty the container, and let the GC run during the loading screen or whatever. It was a deliberate design choice that was surprisingly close to what the original article describes.
3y ago

bluGill 3y ago

it is somewhat common in garbage collected langages to fight the garrbage collector like that. Sure manual free probably adds up to more CPU time, but it is more spread out and thus not noticable (normally, real time still cannot allocate in the sensitive areas)

lifthrasiir 3y ago

Many online games regularly schedule a downtime often to simply restart the process and "solve" moderate memory leaks.

JB_Dev 3y ago

I’ve done something similar in one of our production services. There was a problem with extremely long GC pauses during Gen2 garbage collection (.NET uses a multigenerational GC design). Pauses could be many seconds long or more than a minute in extreme cases. We found the underlying issue that caused memory pressure in the Gen2 region but the fix was to change some very fundamental aspects of the service and would need to have some significant refactoring. Since this was a legacy service (.net framework) that we were refactoring anyway to run in new .NET (5+), we decided to ignore the issue. Instead we adjusted the GC to just never do the expensive Gen2 collections (GCLatencyMode) and moved the service to run on higher memory VMs. It would hit OOM every 3 days or so, so we just set instances to auto-restart once a day. Then 1 year later we deployed the replacement for the legacy service and the problem was solved.

aidenn0 3y ago

Early lisp machines had garage collectors that were so slow, that this was done on workstations.