4 ms·

My question is - why does Postgres need its own scheduler? Shouldn't that be the job of the OS? Is it a legacy thing or just something to squeeze out a tiny bit

My question is - why does Postgres need its own scheduler? Shouldn't that be the job of the OS? Is it a legacy thing or just something to squeeze out a tiny bit of extra performance?

josephlord 14y ago
It will have more information about what is trying to be achieved than can be given to the kernel with the available (across all platforms where possible) API. More knowledge/more specialised means it can have greater optimisation.
jeltz 14y ago
PostgreSQL's spinlock implementation is very fast and scalable and not all supported platforms have good implementations of mutexes. The Linux futexes might be fast enough to replace PostgreSQL's spinlock implementation, but even that is not known yet.
- LnxPrgr3 14y ago
  Here—have a microbenchmark! Here are two really simple mutexes: a wrapper around the pthread functions, and a no-thought-required spinlock: class Spinlock { private: bool state; public: Spinlock() { state = false; } void lock() { while(!__sync_bool_compare_and_swap(&state, false, true)); } void unlock() { state = false; __sync_synchronize(); } }; class PthreadMutex { private: pthread_mutex_t mutex; public: PthreadMutex() { pthread_mutex_init(&mutex, NULL); } void lock() { pthread_mutex_lock(&mutex); } void unlock() { pthread_mutex_unlock(&mutex); } ~PthreadMutex() { pthread_mutex_destroy(&mutex); } }; I used these to protect a not-optimized-at-all dispatch queue built on top of std::deque (single reader, single writer, reader busy-waits until work is available, work is an empty function). I did this test on Mac OS X 10.6.8 (with Apple s gcc 4.2.1 build). A run using PthreadMutex: 0,5944405 1,1627173 2,135598 3,363801 4,3360217 5,5773374 6,5727638 7,5953505 8,4567648 9,5106300 Using Spinlock: 0,154817 1,187507 2,145171 3,102454 4,180604 5,155360 6,128448 7,82418 8,49538 9,39560 All times are in microseconds to run 1,000,000 iterations of the test. Only time to enqueue is measured. I've seen similar results with a very similar test on FreeBSD, though I don't have a box to retest on at the moment. I can only conclude it's not at all unreasonable for PostgreSQL to use its own spinlocks.
  - deleted 14y ago
    [deleted]
barrkel 14y ago
For similar reasons to why DB engines often implement their own caching and disable OS disk caching; a combination of more information about what needs to be done, and a very competitive marketplace.
- chawco 14y ago
  Indeed, there are lessons here, especially from people like PHK and projects like Varnish. Understanding the underlying system is critical for these sorts of projects. For 99% of users the OS will do a much better job of handling your resources than you ever will, but there will always remain a segment where additional knowledge about how those resources are going to be used can yield significant performance gains.
nuje 14y ago
PostgreSQL does not have its own scheduler. Here's a comment at LWN that tries to correct the article a bit: http://lwn.net/Articles/518405/ http://lwn.net/Articles/518405/
chris_wot 14y ago
Everyone does this. Even SQL Server, which runs on only Windows and is made by that very same company, had to have Fibres invented in the kernel so it can do its own user mode scheduling. http://msdn.microsoft.com/en-us/library/aa175385(v=sql.80).aspx http://msdn.microsoft.com/en-us/library/aa175385(v=sql.80).a...

My question is - why does Postgres need its own scheduler? Shouldn't that be the job of the OS? Is it a legacy thing or just something to squeeze out a tiny bit of extra performance?

josephlord 14y ago

It will have more information about what is trying to be achieved than can be given to the kernel with the available (across all platforms where possible) API. More knowledge/more specialised means it can have greater optimisation.

jeltz 14y ago

PostgreSQL's spinlock implementation is very fast and scalable and not all supported platforms have good implementations of mutexes. The Linux futexes might be fast enough to replace PostgreSQL's spinlock implementation, but even that is not known yet.

LnxPrgr3 14y ago
Here—have a microbenchmark! Here are two really simple mutexes: a wrapper around the pthread functions, and a no-thought-required spinlock: class Spinlock { private: bool state; public: Spinlock() { state = false; } void lock() { while(!__sync_bool_compare_and_swap(&state, false, true)); } void unlock() { state = false; __sync_synchronize(); } }; class PthreadMutex { private: pthread_mutex_t mutex; public: PthreadMutex() { pthread_mutex_init(&mutex, NULL); } void lock() { pthread_mutex_lock(&mutex); } void unlock() { pthread_mutex_unlock(&mutex); } ~PthreadMutex() { pthread_mutex_destroy(&mutex); } }; I used these to protect a not-optimized-at-all dispatch queue built on top of std::deque (single reader, single writer, reader busy-waits until work is available, work is an empty function). I did this test on Mac OS X 10.6.8 (with Apple s gcc 4.2.1 build). A run using PthreadMutex: 0,5944405 1,1627173 2,135598 3,363801 4,3360217 5,5773374 6,5727638 7,5953505 8,4567648 9,5106300 Using Spinlock: 0,154817 1,187507 2,145171 3,102454 4,180604 5,155360 6,128448 7,82418 8,49538 9,39560 All times are in microseconds to run 1,000,000 iterations of the test. Only time to enqueue is measured. I've seen similar results with a very similar test on FreeBSD, though I don't have a box to retest on at the moment. I can only conclude it's not at all unreasonable for PostgreSQL to use its own spinlocks.
- deleted 14y ago
  [deleted]

barrkel 14y ago

For similar reasons to why DB engines often implement their own caching and disable OS disk caching; a combination of more information about what needs to be done, and a very competitive marketplace.

chawco 14y ago
Indeed, there are lessons here, especially from people like PHK and projects like Varnish. Understanding the underlying system is critical for these sorts of projects. For 99% of users the OS will do a much better job of handling your resources than you ever will, but there will always remain a segment where additional knowledge about how those resources are going to be used can yield significant performance gains.

nuje 14y ago

PostgreSQL does not have its own scheduler. Here's a comment at LWN that tries to correct the article a bit: http://lwn.net/Articles/518405/ http://lwn.net/Articles/518405/

chris_wot 14y ago

Everyone does this. Even SQL Server, which runs on only Windows and is made by that very same company, had to have Fibres invented in the kernel so it can do its own user mode scheduling. http://msdn.microsoft.com/en-us/library/aa175385(v=sql.80).aspx http://msdn.microsoft.com/en-us/library/aa175385(v=sql.80).a...