3 ms·

I have a major project (a general-purpose data management system) that I have been working on for 10 years. https://www.Didgets.com https://www.Didgets.com I h

I have a major project (a general-purpose data management system) that I have been working on for 10 years. https://www.Didgets.com https://www.Didgets.com I have had a few friends help out with some pieces, but 99%+ of the work I have done alone. I also have some people who I can discuss the architecture with but I don't need anyone's permission to make a radical change. The project is so big that I can't possibly do it all by myself and I have had to learn quite a bit outside my area of expertise (file systems and databases). Even though I could have done it faster with a big team, it is liberating to have full control over a project.

tarr11 4y ago
Curious about your project - you have some claims that your engine is multiple times faster than Postgres. You also mention that you are “using all the cores” in your system. Is it correct to assume that the core innovation here that you are parallelizing your queries? If so, have you tried benchmarking with parallel queries enabled in Postgres (eg., one test would be to increase the number of workers allowed per query) https://www.postgresql.org/docs/current/how-parallel-query-works.html https://www.postgresql.org/docs/current/how-parallel-query-w...
- didgetmaster 4y ago
  There are a number of innovations that make Didgets extremely fast and parallelizing queries is certainly one of them. When I performed the benchmarks against Postgres, I assumed that parallelization was on by default. I certainly saw output when I performed the EXPLAIN ANALYZE operations that indicated it was working (e.g. 'Workers Planned', 'Workers Launched' and 'Parallel Bitmap Heap Scan...'). I haven't worked extensively with Postgres since version 9.5 so I haven't kept up on all the configuration settings. After getting this message, I went into the Postgresql.conf file and bumped up the values for max_worker_processes and max_parallel_workers. This seemed to make some of the queries a bit faster, but I have not re-run the benchmarks in the video.
- didgetmaster 4y ago
  I would love to have someone who is very familiar with the parallel query features of Postgres to independently test out my claims. If you have the data for a decent sized table (e.g. a dozen or more columns with at least a couple million rows) and run some queries on both Postgres and Didgets to see which is truly faster. Is that something you are interested in?

tarr11 4y ago

Curious about your project - you have some claims that your engine is multiple times faster than Postgres. You also mention that you are “using all the cores” in your system. Is it correct to assume that the core innovation here that you are parallelizing your queries? If so, have you tried benchmarking with parallel queries enabled in Postgres (eg., one test would be to increase the number of workers allowed per query) https://www.postgresql.org/docs/current/how-parallel-query-works.html https://www.postgresql.org/docs/current/how-parallel-query-w...

didgetmaster 4y ago
There are a number of innovations that make Didgets extremely fast and parallelizing queries is certainly one of them. When I performed the benchmarks against Postgres, I assumed that parallelization was on by default. I certainly saw output when I performed the EXPLAIN ANALYZE operations that indicated it was working (e.g. 'Workers Planned', 'Workers Launched' and 'Parallel Bitmap Heap Scan...'). I haven't worked extensively with Postgres since version 9.5 so I haven't kept up on all the configuration settings. After getting this message, I went into the Postgresql.conf file and bumped up the values for max_worker_processes and max_parallel_workers. This seemed to make some of the queries a bit faster, but I have not re-run the benchmarks in the video.
didgetmaster 4y ago
I would love to have someone who is very familiar with the parallel query features of Postgres to independently test out my claims. If you have the data for a decent sized table (e.g. a dozen or more columns with at least a couple million rows) and run some queries on both Postgres and Didgets to see which is truly faster. Is that something you are interested in?