3 ms·

Dang some cool ideas in here. Yeah trying to figure out how to annotate the schema was a big motivator for inventing the meta identifier system. Thought about

Dang some cool ideas in here. Yeah trying to figure out how to annotate the schema was a big motivator for inventing the meta identifier system. Thought about using COMMENT for documentation, but once you have meta-ids, I thought it was cleaner to just put schema annotations in a different table. Yeah I think the LISTEN/NOTIFY system in PostgreSQL is a bit primitive. Still working on that part of the project. We got NOTIFYs to propagate up through nginx over a WebSocket and get them into web-world that way, but that section of the project is still fairly immature.

cryptonector 7y ago
Thanks. You can find some of that code here: https://github.com/twosigma/postgresql-contrib https://github.com/twosigma/postgresql-contrib I need to finish the open sourcing of tailfhttpd though. It's... an open-coded HTTP server, written in C, specifically tailored for tailing files over HTTP. It uses epoll and inotify, and is C10K, and blazingly fast. You have to front it with nginx or envoy to get TLS support, naturally. Because it's so simple, open-coding HTTP/1.1 seemed reasonable at the time, though nowadays writing this in Rust with some reasonable framework would be better. The key to tailing files over HTTP is to have a server that supports: - weak ETags (st_dev, st_ino, generation number) - If-Match and If-None-Match conditional request support - Range requests - when the right end of the requested range is left unspecified, use chunked transfer-encoding and don't send the terminator chunk until a) EOF is reached, and b) the file is renamed out of the way or unlinked (which the server can detect using inotify or similar) With this clients can tail a file. Heartbeats can only be supported by the application itself -- HTTP/1.1 does not have the ability to send empty non-terminating chunks (HTTP/2.0 doesn't have that problem). If client loses its tail, it can simply resume by using If-Match and Range starting at the next byte offset after the last byte received before. So recovery is trivial. And presto, super cheap, simple, and reliable pub/sub.
cryptonector 7y ago
What I like about using COMMENT with JSON content is that it makes it trivial to extend the schema in powerful ways, and then Aquameta's meta can be stateless views (and instead of triggers). That makes it much easier to swallow meta.
- erichanson 7y ago
  Oh, we actually split up meta into just the views or a additional extension for the update triggers. Yeah the updatable views are a giant foot gun. Essential in our case but not generally. https://github.com/aquametalabs/meta https://github.com/aquametalabs/meta

cryptonector 7y ago

Thanks. You can find some of that code here: https://github.com/twosigma/postgresql-contrib https://github.com/twosigma/postgresql-contrib I need to finish the open sourcing of tailfhttpd though. It's... an open-coded HTTP server, written in C, specifically tailored for tailing files over HTTP. It uses epoll and inotify, and is C10K, and blazingly fast. You have to front it with nginx or envoy to get TLS support, naturally. Because it's so simple, open-coding HTTP/1.1 seemed reasonable at the time, though nowadays writing this in Rust with some reasonable framework would be better. The key to tailing files over HTTP is to have a server that supports: - weak ETags (st_dev, st_ino, generation number) - If-Match and If-None-Match conditional request support - Range requests - when the right end of the requested range is left unspecified, use chunked transfer-encoding and don't send the terminator chunk until a) EOF is reached, and b) the file is renamed out of the way or unlinked (which the server can detect using inotify or similar) With this clients can tail a file. Heartbeats can only be supported by the application itself -- HTTP/1.1 does not have the ability to send empty non-terminating chunks (HTTP/2.0 doesn't have that problem). If client loses its tail, it can simply resume by using If-Match and Range starting at the next byte offset after the last byte received before. So recovery is trivial. And presto, super cheap, simple, and reliable pub/sub.

cryptonector 7y ago

What I like about using COMMENT with JSON content is that it makes it trivial to extend the schema in powerful ways, and then Aquameta's meta can be stateless views (and instead of triggers). That makes it much easier to swallow meta.

erichanson 7y ago
Oh, we actually split up meta into just the views or a additional extension for the update triggers. Yeah the updatable views are a giant foot gun. Essential in our case but not generally. https://github.com/aquametalabs/meta https://github.com/aquametalabs/meta