3 ms·

In the article they talk about how printing an error from the anyhow crate in debug format creates a full backtrace, which leads to an OOM error. This happens e

by xgb84j 2y ago

In the article they talk about how printing an error from the anyhow crate in debug format creates a full backtrace, which leads to an OOM error. This happens even with 4 GB of memory. Why does creating a backtrace need such a large amount of memory? Is there a memory leak involved as well?

oguz-ismail 2y ago
[flagged]
prerok 2y ago
Well, based on the article, if there was a memory leak then they should see the steady increase in memory consumption, which was not the case. The only explanation I can see (if their conclusion is accurate) is that the end result of the symbolization is more than 400MB additional memory consumption (which is a lot in my opinion), however the process of the symbolization requires more than 2GB additional memory (which is incredibly a lot).
- prerok 2y ago
  The author replied with additional explanations, so it seems that the additional 400MB were needed because the debug symbols were compressed.
CodesInChaos 2y ago
I don't think the 4 GiB instance actually ran into an OOM error. They merely observed a 400 MiB memory spike. The crashing instances were limited to 256 and later 512 MiB. (Assuming that the article incorrectly used Mib when they meant MiB. Used correctly b=bit, B=byte)
erebe__ 2y ago
Sorry if the article is misleading. The first increase of the memory limit was not 4G, but something roughly around 300Mb/400Mb, and the OOM did happen again with this setting. Thus leading to a 2nd increase to 4Gi to be sure the app would not get OOM killed when the behavior get triggered. We needed the app to be alive/running for us to investigate the memory profiling. Regarding the increase of 400MiB, yeah it is a lot, and it was a surprise to us too. We were not expecting such increase. There are, I think 2 reasons behind this. 1. This service is a grpc server, which has a lot of code generated, so lots of symbols 2. we compile the binary with debug symbols and a flag to compress the debug symbols sections to avoid having huge binary. Which may part be of this issue.
- CodesInChaos 2y ago
  > we compile the binary with debug symbols and a flag to compress the debug symbols sections to avoid having huge binary. How big are the uncompressed debug symbols? I'd expected processing uncompressed debug symbols to happen via a memory mapped file, while compressed debug symbols probably need to be extracted to anonymous memory. https://github.com/llvm/llvm-project/issues/63290 https://github.com/llvm/llvm-project/issues/63290
  - erebe__ 2y ago
    Normal build cargo build --bin engine-gateway --release Finished `release` profile [optimized + debuginfo] target(s) in 1m 00s ls -lh target/release/engine-gateway .rwxr-xr-x erebe erebe 198 MB Sun Jan 19 12:37:35 2025  target/release/engine-gateway what we ship export RUSTFLAGS="-C link-arg=-Wl,--compress-debug-sections=zlib -C force-frame-pointers=yes" cargo build --bin engine-gateway --release Finished `release` profile [optimized + debuginfo] target(s) in 1m 04s ls -lh target/release/engine-gateway .rwxr-xr-x erebe erebe 61 MB Sun Jan 19 12:39:13 2025  target/release/engine-gateway The diff is more impressive on some bigger projects
    - adastra22 2y ago
      The compressed symbols sounds like the likely culprit. Do you really need a small executable? The uncompressed symbols need to be loaded into RAM anyway, and if it is delayed until it is needed then you will have to allocate memory to uncompress them.
      - erebe__ 2y ago
        I will give it a shot next week to try out ;P For this particular service, the size does not matter really. For others, it makes more diff (several hundred of Mb) and as we deploy on customers infra, we want images' size to stay reasonable. For now, we apply the same build rules for all our services to stay consistent.

by xgb84j 2y ago

oguz-ismail 2y ago

[flagged]

prerok 2y ago

Well, based on the article, if there was a memory leak then they should see the steady increase in memory consumption, which was not the case. The only explanation I can see (if their conclusion is accurate) is that the end result of the symbolization is more than 400MB additional memory consumption (which is a lot in my opinion), however the process of the symbolization requires more than 2GB additional memory (which is incredibly a lot).

prerok 2y ago
The author replied with additional explanations, so it seems that the additional 400MB were needed because the debug symbols were compressed.

CodesInChaos 2y ago

I don't think the 4 GiB instance actually ran into an OOM error. They merely observed a 400 MiB memory spike. The crashing instances were limited to 256 and later 512 MiB. (Assuming that the article incorrectly used Mib when they meant MiB. Used correctly b=bit, B=byte)

erebe__ 2y ago

Sorry if the article is misleading. The first increase of the memory limit was not 4G, but something roughly around 300Mb/400Mb, and the OOM did happen again with this setting. Thus leading to a 2nd increase to 4Gi to be sure the app would not get OOM killed when the behavior get triggered. We needed the app to be alive/running for us to investigate the memory profiling. Regarding the increase of 400MiB, yeah it is a lot, and it was a surprise to us too. We were not expecting such increase. There are, I think 2 reasons behind this. 1. This service is a grpc server, which has a lot of code generated, so lots of symbols 2. we compile the binary with debug symbols and a flag to compress the debug symbols sections to avoid having huge binary. Which may part be of this issue.

CodesInChaos 2y ago
> we compile the binary with debug symbols and a flag to compress the debug symbols sections to avoid having huge binary. How big are the uncompressed debug symbols? I'd expected processing uncompressed debug symbols to happen via a memory mapped file, while compressed debug symbols probably need to be extracted to anonymous memory. https://github.com/llvm/llvm-project/issues/63290 https://github.com/llvm/llvm-project/issues/63290
- erebe__ 2y ago
  Normal build cargo build --bin engine-gateway --release Finished `release` profile [optimized + debuginfo] target(s) in 1m 00s ls -lh target/release/engine-gateway .rwxr-xr-x erebe erebe 198 MB Sun Jan 19 12:37:35 2025  target/release/engine-gateway what we ship export RUSTFLAGS="-C link-arg=-Wl,--compress-debug-sections=zlib -C force-frame-pointers=yes" cargo build --bin engine-gateway --release Finished `release` profile [optimized + debuginfo] target(s) in 1m 04s ls -lh target/release/engine-gateway .rwxr-xr-x erebe erebe 61 MB Sun Jan 19 12:39:13 2025  target/release/engine-gateway The diff is more impressive on some bigger projects
  - adastra22 2y ago
    The compressed symbols sounds like the likely culprit. Do you really need a small executable? The uncompressed symbols need to be loaded into RAM anyway, and if it is delayed until it is needed then you will have to allocate memory to uncompress them.
    - erebe__ 2y ago
      I will give it a shot next week to try out ;P For this particular service, the size does not matter really. For others, it makes more diff (several hundred of Mb) and as we deploy on customers infra, we want images' size to stay reasonable. For now, we apply the same build rules for all our services to stay consistent.