3 ms·

This vaguely reminded of the time where our self-hosted installation of JVM-based app worked fine in DEV and QA (on vmware-based VMs), but in PROD it would erro

This vaguely reminded of the time where our self-hosted installation of JVM-based app worked fine in DEV and QA (on vmware-based VMs), but in PROD it would error out the time with "out of memory", even though the machines (bare metal) had far more RAM. It turned out they also had lots more CPU cores (they were repurposed former DB machines), and by default the memory manager (of the JVM, I suppose?) scaled the allocation block size the number of CPUs, so it tried to allocate far more memory than necessary, and since this was still a 32-bit JVM it quickly ran out of space for its allocations. Once we found out what the issue was, it was pretty easy to tune with an environment variable.

kevingadd 3y ago
32 bit and high core count is definitely a nasty combo. Many applications spawn 2-4 threads per core which is a great way to eat up that address space.
- nine_k 3y ago
  But don't all threads of a process share mostly the same address space? I could imagine that starting many processes would exhaust address space, or allocating a lot of virtual memory / mmap without committing.
  - simcop2387 3y ago
    That's one of the fun bits, other processes don't share the memory space. That's one reason you can have 16GiB of ram on a 32bit x86 server without much issue. This was done on x86 through a tech called PAE (Physical Address Extension) that let the OS know there was more memory and allocate it around, but any one process was limited to 4GiB (usually lower due to kernel/userspace barriers) because of 32 bit pointers. So if those threads all allocated memory it was easier to hit the limit but if you used separate processes then they could each reach 4GiB (or whatever the limit is) independently without causing problems.
jon-wood 3y ago
Off topic, so apologies in advance. Why do environments so often get capitalised like that? Prod isn’t an acronym, it’s an abbreviation of Production. Similarly Dev is an abbreviation of Development. I know in the grand scheme of things it’s meaningless, this is just one of those things that’s like someone scratching their fingers down a blackboard when I see it day to day.
- mikebenfield 3y ago
  I don't know, but I see unnecessary all caps in other words too - it's not infrequent that I see JAVA and sometimes RUST (for the programming languages). Kind of irks me, but obviously doesn't really matter. Ultimately I think it's just a weird niche cultural holdover from the very early days of computing when lots of random stuff was arbitrarily capitalized, like FORTRAN was originally all caps, maybe related to early character sets that only had capital letters.
  - mike_hock 3y ago
    And some all caps languages are still around. C, D, R, C++, C#, F#.
    - burnished 3y ago
      Those follow normal punctuation rules in that the first letter of the proper noun is capitalized.
- alexstore06 3y ago
  My first guess for that particular case is that PROD and DEV are often either parts of hostnames for those particular machines or are referring to hostnames, which are often written in all caps.
- Terr_ 3y ago
  > Prod isn’t an acronym, it’s an abbreviation of Production. Tangentially, "Prod" is justified since it's being treated as a proper name, like "Paris", "Roderick", or "Mount Rushmore." But yes, "PROD" is harder to explain, and I suspect it might be bleed-over from looking at code or config files where it gets capitalized.

kevingadd 3y ago

32 bit and high core count is definitely a nasty combo. Many applications spawn 2-4 threads per core which is a great way to eat up that address space.

nine_k 3y ago
But don't all threads of a process share mostly the same address space? I could imagine that starting many processes would exhaust address space, or allocating a lot of virtual memory / mmap without committing.
- simcop2387 3y ago
  That's one of the fun bits, other processes don't share the memory space. That's one reason you can have 16GiB of ram on a 32bit x86 server without much issue. This was done on x86 through a tech called PAE (Physical Address Extension) that let the OS know there was more memory and allocate it around, but any one process was limited to 4GiB (usually lower due to kernel/userspace barriers) because of 32 bit pointers. So if those threads all allocated memory it was easier to hit the limit but if you used separate processes then they could each reach 4GiB (or whatever the limit is) independently without causing problems.

jon-wood 3y ago

Off topic, so apologies in advance. Why do environments so often get capitalised like that? Prod isn’t an acronym, it’s an abbreviation of Production. Similarly Dev is an abbreviation of Development. I know in the grand scheme of things it’s meaningless, this is just one of those things that’s like someone scratching their fingers down a blackboard when I see it day to day.

mikebenfield 3y ago
I don't know, but I see unnecessary all caps in other words too - it's not infrequent that I see JAVA and sometimes RUST (for the programming languages). Kind of irks me, but obviously doesn't really matter. Ultimately I think it's just a weird niche cultural holdover from the very early days of computing when lots of random stuff was arbitrarily capitalized, like FORTRAN was originally all caps, maybe related to early character sets that only had capital letters.
- mike_hock 3y ago
  And some all caps languages are still around. C, D, R, C++, C#, F#.
  - burnished 3y ago
    Those follow normal punctuation rules in that the first letter of the proper noun is capitalized.
alexstore06 3y ago
My first guess for that particular case is that PROD and DEV are often either parts of hostnames for those particular machines or are referring to hostnames, which are often written in all caps.
Terr_ 3y ago
> Prod isn’t an acronym, it’s an abbreviation of Production. Tangentially, "Prod" is justified since it's being treated as a proper name, like "Paris", "Roderick", or "Mount Rushmore." But yes, "PROD" is harder to explain, and I suspect it might be bleed-over from looking at code or config files where it gets capitalized.