3 ms·

> DuckDB is designed from day one to use all the CPU cores in your machine Hi. No thank you? I'm tired of greedy software that I have to literally pin down by

> DuckDB is designed from day one to use all the CPU cores in your machine Hi. No thank you? I'm tired of greedy software that I have to literally pin down by shackling it to a cgroup via systemd or docker. 1 core is fine. We have a group of 20, and limited resources, and there are no fires.

notpushkin 3y ago
I think having an option to utilize multiple cores is better than not having one. Would be nice to have an option to limit the number of cores used by DuckDB though, if it doesn't have one already. (`duckdb -j4`?)
marginalia_nu 3y ago
Yeah, this is kind of a nuisance on high-core count CPUs, especially SMP.
_flux 3y ago
Why can't the scheduler just do its job and let applications enjoy the highest performance they can achieve? I assume DuckDB is making good use of those cores, not just spinning wildly ;).
- tetris11 3y ago
  True, the scheduler usually does do fine job CPU-wise. In terms of RAM though, if an app sees 120GB it will expand to 120GB before considering paging. I'm just not a fan of software that immediately tries to use all possible resources for 2% gain in a performance benchmark over peers.
  - diroussel 3y ago
    That’s not how most apps work. They tend to allocate memory for specific purposes and free it when done. Only more heavy apps will retain large buffer pools, like databases but they are usually tunable. And apps themselves don’t page to disk, usually. The OS does that. Before we had MMUs apps like photoshop used to manage the page file manually. But again an app doing that is doing that for a specific purpose, not just to be greedy. Is there some specific software you are referring to?
    - tetris11 3y ago
      No, but a general rant at analysis software in general which seeks to maximise all resources. I agree that database software likely does not behave so aggressively, but the terms I saw in the duckdb post triggered me to believe they were trending in that direction.
szarnyasg 3y ago
You can limit the number of threads used by DuckDB via `SET threads = x`, see https://duckdb.org/docs/configuration/pragmas#threads https://duckdb.org/docs/configuration/pragmas#threads (Disclaimer: I work at DuckDB Labs)

notpushkin 3y ago

I think having an option to utilize multiple cores is better than not having one. Would be nice to have an option to limit the number of cores used by DuckDB though, if it doesn't have one already. (`duckdb -j4`?)

marginalia_nu 3y ago

Yeah, this is kind of a nuisance on high-core count CPUs, especially SMP.

_flux 3y ago

Why can't the scheduler just do its job and let applications enjoy the highest performance they can achieve? I assume DuckDB is making good use of those cores, not just spinning wildly ;).

tetris11 3y ago
True, the scheduler usually does do fine job CPU-wise. In terms of RAM though, if an app sees 120GB it will expand to 120GB before considering paging. I'm just not a fan of software that immediately tries to use all possible resources for 2% gain in a performance benchmark over peers.
- diroussel 3y ago
  That’s not how most apps work. They tend to allocate memory for specific purposes and free it when done. Only more heavy apps will retain large buffer pools, like databases but they are usually tunable. And apps themselves don’t page to disk, usually. The OS does that. Before we had MMUs apps like photoshop used to manage the page file manually. But again an app doing that is doing that for a specific purpose, not just to be greedy. Is there some specific software you are referring to?
  - tetris11 3y ago
    No, but a general rant at analysis software in general which seeks to maximise all resources. I agree that database software likely does not behave so aggressively, but the terms I saw in the duckdb post triggered me to believe they were trending in that direction.

szarnyasg 3y ago

You can limit the number of threads used by DuckDB via `SET threads = x`, see https://duckdb.org/docs/configuration/pragmas#threads https://duckdb.org/docs/configuration/pragmas#threads (Disclaimer: I work at DuckDB Labs)