4 ms·

Using 512gb of ram for a 22gb remote file does feel a bit weird for a benchmark but maybe they couldn’t get a large number of cores without lots of memory?

• otterley 2mo ago
  Most cloud providers start with a 2:1 ratio of memory in GiB to CPU cores and go up from there. Databases also are the most common workload for large-memory systems because they benefit so much from large buffer caches.
• marginalia_nu 2mo ago
  Even so you could use cgroups to cap the memory for the process (including its caches).
• pdet 2mo ago
  The main reason I decided to use a beefier machine is that it gives me flexibility when benchmarking, without the need to set up different environments. The CSV data, for example, is >80 GB. We can also “scale down the machine” for experiments where we want to stress-test lower-memory scenarios or use fewer threads by configuring DuckDB’s settings (e.g., SET memory_limit = '10GB'; or SET threads TO 1;). (Disclaimer: I’m the author of the blog post.)
  • mahogany 2mo ago
    Note that SET memory_limit is a soft limit and can be completely ignored for some tasks, so this wouldn’t be the same as having a limit on physical memory.
    • michaelmdresser 2mo ago
      In past experience with DuckDB I have found this soft-limit distinction to be very important.