4 ms·

But your durability goes down the drain on power loss or similar. Files are _really_ complicated to get durable.

0xbadcafebee 4y ago
Sorry, I don't follow? SQLite also has durability problems on power loss. You can use exactly the same method that SQLite uses to flush data to disk. There is no need for a WAL because the queue operations are atomic, using rename() on POSIX filesystems (including NFS). There's no rollback of course, but you can recover items that are stalled in the queue. With a journaled filesystem, fdatasync() after each atomic operation, and battery-backed RAID, I can't imagine it getting more durable [in comparison to SQLite]. Interestingly, journaled rollback mode in SQLite is probably worse durability than the file/directory queue method. (https://www.sqlite.org/lockingv3.html#how_to_corrupt https://www.sqlite.org/lockingv3.html#how_to_corrupt) (https://www.sqlite.org/atomiccommit.html#sect_9_0 https://www.sqlite.org/atomiccommit.html#sect_9_0) It depends on the same system-level guarantees about the filesystem and hardware, except it also depends on POSIX advisory locks, which are fallible and implementation-specific; while the file/directory queue solely depends on rename() being atomic, which it always is.
- Sesse__ 4y ago
  > Sorry, I don't follow? SQLite also has durability problems on power loss. No, it doesn't. Once COMMIT returns, your data is durable. > You can use exactly the same method that SQLite uses to flush data to disk. Good luck with that! It took them about ten years to get it right, and the SQLite people are world class. (You can't just copy their code wholesale, since they have one file to worry about and your scheme has tons.) > With a journaled filesystem, fdatasync() after each atomic operation, and battery-backed RAID, I can't imagine it getting more durable [in comparison to SQLite]. The problem is lack of imagination. :-) See https://danluu.com/deconstruct-files/ https://danluu.com/deconstruct-files/ (and the linked papers) for an overview of all the things that can go wrong. In particular, if you only only ever fdatasync(), your files are not even guaranteed to show up after a power loss.
mst 4y ago
Solution: use a qmail install as the queue manager, and a custom mail server it relays to as a job handler. There's already a spec for transferring SOAP over SMTP so the request/response part is already built for you. (couldn't resist, sorry)
- vidarh 4y ago
  Been there, done that. In production. That is, without the custom mail server you describe. You can feed every incoming message to a custom command with ".qmail", or forward with ".forward", so we used that for "push" delivery of messages and plain old POP3 for pull. E-mail really does have everything you need to build complex routing topologies. We ran a mail provider, and so had a heavily customised Qmail setup, and when we needed a queue we figured we might as well use that. Meant we could trivially do things like debugging by cc:'ing messages to a mailbox someone connected a mail client to, for example.
  - mst 4y ago
    My ISP days were not unsimilar, and had we a TARDIS it might be fun to go back to when both setups were still there and compare notes. However, I'd hope the SOAP part made it clear I was exaggerating a little for effect :) ... also because the idea of the custom server meaning you could have the main system's qmail handle spaced retries based on SMTP status codes amused me.
    - vidarh 4y ago
      I mean SOAP fit right in with the zeitgeist of that era... I seem to remember we were sending XML (shudder), as we had quite a lot of other stuff using XML, but nothing as verbose as SOAP. Our main CGI (yes...) written in C++ (I can hear you squirm from here) was also loosely inspired by CORBA (stop that screaming) in terms of "sort-of" exposing objects via router that automatically mapped URLs to objects, and which used XML for persistence and an XML based rendering pipeline (not that different from React components, actually, except all C++ and server side). Hit all the 90's buzz words.

Sorry, I don't follow? SQLite also has durability problems on power loss. You can use exactly the same method that SQLite uses to flush data to disk. There is no need for a WAL because the queue operations are atomic, using rename() on POSIX filesystems (including NFS). There's no rollback of course, but you can recover items that are stalled in the queue. With a journaled filesystem, fdatasync() after each atomic operation, and battery-backed RAID, I can't imagine it getting more durable [in comparison to SQLite]. Interestingly, journaled rollback mode in SQLite is probably worse durability than the file/directory queue method. (https://www.sqlite.org/lockingv3.html#how_to_corrupt https://www.sqlite.org/lockingv3.html#how_to_corrupt) (https://www.sqlite.org/atomiccommit.html#sect_9_0 https://www.sqlite.org/atomiccommit.html#sect_9_0) It depends on the same system-level guarantees about the filesystem and hardware, except it also depends on POSIX advisory locks, which are fallible and implementation-specific; while the file/directory queue solely depends on rename() being atomic, which it always is.

Sesse__ 4y ago
> Sorry, I don't follow? SQLite also has durability problems on power loss. No, it doesn't. Once COMMIT returns, your data is durable. > You can use exactly the same method that SQLite uses to flush data to disk. Good luck with that! It took them about ten years to get it right, and the SQLite people are world class. (You can't just copy their code wholesale, since they have one file to worry about and your scheme has tons.) > With a journaled filesystem, fdatasync() after each atomic operation, and battery-backed RAID, I can't imagine it getting more durable [in comparison to SQLite]. The problem is lack of imagination. :-) See https://danluu.com/deconstruct-files/ https://danluu.com/deconstruct-files/ (and the linked papers) for an overview of all the things that can go wrong. In particular, if you only only ever fdatasync(), your files are not even guaranteed to show up after a power loss.

mst 4y ago

Solution: use a qmail install as the queue manager, and a custom mail server it relays to as a job handler. There's already a spec for transferring SOAP over SMTP so the request/response part is already built for you. (couldn't resist, sorry)

vidarh 4y ago
Been there, done that. In production. That is, without the custom mail server you describe. You can feed every incoming message to a custom command with ".qmail", or forward with ".forward", so we used that for "push" delivery of messages and plain old POP3 for pull. E-mail really does have everything you need to build complex routing topologies. We ran a mail provider, and so had a heavily customised Qmail setup, and when we needed a queue we figured we might as well use that. Meant we could trivially do things like debugging by cc:'ing messages to a mailbox someone connected a mail client to, for example.
- mst 4y ago
  My ISP days were not unsimilar, and had we a TARDIS it might be fun to go back to when both setups were still there and compare notes. However, I'd hope the SOAP part made it clear I was exaggerating a little for effect :) ... also because the idea of the custom server meaning you could have the main system's qmail handle spaced retries based on SMTP status codes amused me.
  - vidarh 4y ago
    I mean SOAP fit right in with the zeitgeist of that era... I seem to remember we were sending XML (shudder), as we had quite a lot of other stuff using XML, but nothing as verbose as SOAP. Our main CGI (yes...) written in C++ (I can hear you squirm from here) was also loosely inspired by CORBA (stop that screaming) in terms of "sort-of" exposing objects via router that automatically mapped URLs to objects, and which used XML for persistence and an XML based rendering pipeline (not that different from React components, actually, except all C++ and server side). Hit all the 90's buzz words.