5 ms·

In this case nginx is the 'application'. the requests is still going to be expressed as a SQL query, sent to the PG, parsed, compiled, optimized, executed, then

by rusanu 9y ago

In this case nginx is the 'application'. the requests is still going to be expressed as a SQL query, sent to the PG, parsed, compiled, optimized, executed, then the tabular response formatted as the HTTP response. Many more steps compared to a file-on-disk response. But I second that is an interesting nginx module

icebraining 9y ago
Ah, but the filesystem has to do all that as well! It must receive a path, parse it, and then execute the query, with possible optimizations (eg. ext4 even has indexes implemented with hashed b-trees). A filesystem is just an hierarchical database.
- fnord123 9y ago
  >A filesystem is just an hierarchical database. Loosely. Today's file systems aren't transactional (i.e. acid compliance), which is a basic property that most people consider necessary for a database.
  - phamilton 9y ago
    That's not really a true statement. I use plenty of databases that aren't transactional beyond a single record.
    - _jal 9y ago
      It is a true statement. You're not a member of the set of "most people": > a basic property that most people consider necessary I use non-ACID stores for some purposes, but don't consider them real databases.
  - the8472 9y ago
    Many filesystems are logged and provide atomic operations. You can build multi-step transactions on top of them with various operations. And with CoW filesystems you can even get read consistency. Oh, and optimistic locking also is an option.
    - fnord123 9y ago
      Are we talking about the POSIX interface to files or is there a new transactional API that allows for atomic writes (all pages are written to disk or none are), consistency, isolation (opening a file in another process which is currently being written won't show the pages that are being written), and makes the writes always durable (sync; not fsync)?
      - the8472 9y ago
        > Are we talking about the POSIX interface to files Those give you these atomic ops and optimistic or cooperative locking: https://rcrowley.org/2010/01/06/things-unix-can-do-atomically.html https://rcrowley.org/2010/01/06/things-unix-can-do-atomicall... Multi-file transactions can be built by moving whole directories over symlinks to the previous version. The linux-specific RENAME_EXCHANGE flag can simplify this. > or is there a new transactional API CoW filesystems give you snapshots and reflink copies via ioctls. Under heavy concurrency this can provide cheaper isolation than locking. > and makes the writes always durable (sync; not fsync)? For durability fsync is sufficient if you do it on the file and the directory. To combine atomic and durable you can do the write, fsync, rename, fsync dir dance. https://lwn.net/Articles/457667/ https://lwn.net/Articles/457667/
        fnord123 9y ago
        >For durability fsync is sufficient if you do it on the file and the directory. To combine atomic and durable you can do the write, fsync, rename, fsync dir dance. Right. This allows for the correct behaviour but I wouldn't conclude that POSIX is a transactional API. I would regard this as being able to build a transactional API on top of it; but not transactional in itself. I mean, if you had to do the same dance in SQL (insert to dummy pkey and then update the entry) you would rightly dismiss it.
        the8472 9y ago
        Which is why I said > You can build multi-step transactions on top of them with various operations.
- cat199 9y ago
  Does your database transparently DMA the buffer cache to the NIC after loading the file in the background using read-ahead without any process context switching or upcalls to userland? That's what I thought. https://www.freebsd.org/cgi/man.cgi?query=sendfile&sektion=2 https://www.freebsd.org/cgi/man.cgi?query=sendfile&sektion=2 http://man7.org/linux/man-pages/man2/sendfile.2.html http://man7.org/linux/man-pages/man2/sendfile.2.html
  - coldtea 9y ago
    Does your filesystem provide for its files that are referenced by your DB the same versioning, consistent backups, and ACID guarantees as the DB does? That's what I thought.

by rusanu 9y ago

icebraining 9y ago

Ah, but the filesystem has to do all that as well! It must receive a path, parse it, and then execute the query, with possible optimizations (eg. ext4 even has indexes implemented with hashed b-trees). A filesystem is just an hierarchical database.

fnord123 9y ago
>A filesystem is just an hierarchical database. Loosely. Today's file systems aren't transactional (i.e. acid compliance), which is a basic property that most people consider necessary for a database.
- phamilton 9y ago
  That's not really a true statement. I use plenty of databases that aren't transactional beyond a single record.
  - _jal 9y ago
    It is a true statement. You're not a member of the set of "most people": > a basic property that most people consider necessary I use non-ACID stores for some purposes, but don't consider them real databases.
- the8472 9y ago
  Many filesystems are logged and provide atomic operations. You can build multi-step transactions on top of them with various operations. And with CoW filesystems you can even get read consistency. Oh, and optimistic locking also is an option.
  - fnord123 9y ago
    Are we talking about the POSIX interface to files or is there a new transactional API that allows for atomic writes (all pages are written to disk or none are), consistency, isolation (opening a file in another process which is currently being written won't show the pages that are being written), and makes the writes always durable (sync; not fsync)?
    - the8472 9y ago
      > Are we talking about the POSIX interface to files Those give you these atomic ops and optimistic or cooperative locking: https://rcrowley.org/2010/01/06/things-unix-can-do-atomically.html https://rcrowley.org/2010/01/06/things-unix-can-do-atomicall... Multi-file transactions can be built by moving whole directories over symlinks to the previous version. The linux-specific RENAME_EXCHANGE flag can simplify this. > or is there a new transactional API CoW filesystems give you snapshots and reflink copies via ioctls. Under heavy concurrency this can provide cheaper isolation than locking. > and makes the writes always durable (sync; not fsync)? For durability fsync is sufficient if you do it on the file and the directory. To combine atomic and durable you can do the write, fsync, rename, fsync dir dance. https://lwn.net/Articles/457667/ https://lwn.net/Articles/457667/
      - fnord123 9y ago
        >For durability fsync is sufficient if you do it on the file and the directory. To combine atomic and durable you can do the write, fsync, rename, fsync dir dance. Right. This allows for the correct behaviour but I wouldn't conclude that POSIX is a transactional API. I would regard this as being able to build a transactional API on top of it; but not transactional in itself. I mean, if you had to do the same dance in SQL (insert to dummy pkey and then update the entry) you would rightly dismiss it.
        the8472 9y ago
        Which is why I said > You can build multi-step transactions on top of them with various operations.
cat199 9y ago
Does your database transparently DMA the buffer cache to the NIC after loading the file in the background using read-ahead without any process context switching or upcalls to userland? That's what I thought. https://www.freebsd.org/cgi/man.cgi?query=sendfile&sektion=2 https://www.freebsd.org/cgi/man.cgi?query=sendfile&sektion=2 http://man7.org/linux/man-pages/man2/sendfile.2.html http://man7.org/linux/man-pages/man2/sendfile.2.html
- coldtea 9y ago
  Does your filesystem provide for its files that are referenced by your DB the same versioning, consistent backups, and ACID guarantees as the DB does? That's what I thought.