5 ms·

Postgres's extension APIs are pretty good and Rust's interoperability with C makes it quite feasible to extend the database without rewriting it. Some months ag

Postgres's extension APIs are pretty good and Rust's interoperability with C makes it quite feasible to extend the database without rewriting it. Some months ago we wrote a logical decoder -- to capture and forward the write-ahead log as JSON -- in Rust and later were able to pull out some helpful bindings for logical decoders, foreign data wrappers and so forth. The logical decoder: http://pgxn.org/dist/jsoncdc http://pgxn.org/dist/jsoncdc The bindings: https://github.com/posix4e/rpgffi https://github.com/posix4e/rpgffi The extensibility of Postgres is really profound -- there are so many things you can do, and so much internal functionality you can access, without having to recompile the database. The way extensions are loaded also mitigates the risk of what can be admittedly tricky issues relating to safety. Extensions run in backend processes, not the postmaster. If rewriting Postgres happens, I hope it would be gradual. You can replace a lot of the core with extensions, and the scope for extension is ever increasing. So say you wrote a storage engine in Rust. Writing a foreign data wrapper for it would give you a way to use it from Postgres for all your tables; but you wouldn't have to write a SQL parser, client manager, permissions system, or any of the rest of that stuff. Although the C stuff is a little gritty, a lot of what goes on in a database is gritty "because it is" and that low-level control is needed to deliver on tail latency, fault tolerance or simple storage efficiency. For example, Postgres has code to recover from malloc failing.

masklinn 10y ago
I'm not sure there's much reason to rewrite Postgres as a whole. If there are specific subsystems which are common sources of CVE they might be targets, but looking at the list[0] no specific system really seems to jump out. Extensions could be neat, there doesn't seem to be a high-level Rust API for it (in the style of ruru or rust-cpython) yet though, only incomplete-looking bindgen: https://github.com/posix4e/rpgffi https://github.com/posix4e/rpgffi [0] https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=postgresql https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=postgresql
- stymaar 10y ago
  I'm not advocating for a complete rewrite of Postgres in C, but I'd like to emphasize the fact that security is not the only good reason to justify the transition of a project from C to Rust: Nowadays most developers comes from memory-managed languages => for us, C is a scary land where it's really hard to get things right when you don't have all the prerequisite knowledge (what's the stack ?) => the language is really old, you don't have all the fancy and useful tools and syntactic sugar (package manager, proper error handling, generics) => then we don't want to invest a lot of time to learn a difficult language that is so unproductive. => and even if we do out of curiosity, we'll most likely never feel confident enough to dig into an open-source project and contribute. Rust on the other hand, is like C but with a personal IA assistant (the compiler) and all the syntactic sugar you're used to (and even more if you're not familiar with the ML family). That's what's attracting so many web developers to Rust ! In my opinion it means that, in a somewhat near future, the amount of potential contributors for projects written in Rust could be higher than for projects written in C. And those contributor are also likely to be more productive, thanks to the language features and tooling.
  - thesmallestcat 10y ago
    You're describing qualities that are meaningless to a database user. A Rust implementation of PostgreSQL, with its runtime checks, would almost certainly be slower than the existing implementation. So a Rust port is a non-starter. Database users don't care about your syntactic sugar, that you find manual memory management a "scary land" due to background in managed languages, or how many contributors you have. Speaking of which, the PostgreSQL project is healthy and has many, many contributors. And while we're discussing developer productivity, I can't even imagine the build time of a Rust port. What's annoying about OP's email is that they clearly did no research on the matter. You can find many discussions on the list about "upgrading" to C++, and there has been much wringing of hands. The last I heard, it sounded like a faction was considering having the project build with a C++ compiler rather than a C compiler for better type checks and as a step on that path.
    - stymaar 10y ago
      > A Rust implementation of PostgreSQL, with its runtime checks, would almost certainly be slower than the existing implementation. So a Rust port is a non-starter. Wait, which runtime checks are you talking about ? This is a recurring FUD we often see on HN about Rust and I'd like to clarify this point. IIRC, there is no mandatory runtime check in idiomatic Rust except UTF8 validation for strings[0]. There is no overflow checks on numbers, and no bounds checking when using iterators[1]. Another known overhead is the default use of siphash in hashmaps for DoS prevention[2], but this is not really a «runtime check». Do you have anything else in mind ? > Speaking of which, the PostgreSQL project is healthy and has many, many contributors. Indeed, and as I said, I'm not advocating for a port of PostgreSQL in Rust. But for other projects that suffer from a lack of attention[3], it could be a good move. > And while we're discussing developer productivity, I can't even imagine the build time of a Rust port. touché, but compile time is one of the biggest focus of the Rust core team atm, so I'm confident things will improve in 2017 :) [0]: which can be worked around if need be by using &[u8] instead of &str (see: https://news.ycombinator.com/item?id=13268051 https://news.ycombinator.com/item?id=13268051 from /u/burntsushi) [1]: there are bounds checking when you access array elements through indexes, but most of the time you don't do that since iterators are great. If you really need to, and this have a performance impact, you can opt-out the bound checking in unsafe block. [2]: But this is opt-out by using a different hash function, if you use your hashmap in a non adversarial environment. [3]: like librsvg which was chronically under-maintained.

masklinn 10y ago

I'm not sure there's much reason to rewrite Postgres as a whole. If there are specific subsystems which are common sources of CVE they might be targets, but looking at the list[0] no specific system really seems to jump out. Extensions could be neat, there doesn't seem to be a high-level Rust API for it (in the style of ruru or rust-cpython) yet though, only incomplete-looking bindgen: https://github.com/posix4e/rpgffi https://github.com/posix4e/rpgffi [0] https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=postgresql https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=postgresql

stymaar 10y ago
I'm not advocating for a complete rewrite of Postgres in C, but I'd like to emphasize the fact that security is not the only good reason to justify the transition of a project from C to Rust: Nowadays most developers comes from memory-managed languages => for us, C is a scary land where it's really hard to get things right when you don't have all the prerequisite knowledge (what's the stack ?) => the language is really old, you don't have all the fancy and useful tools and syntactic sugar (package manager, proper error handling, generics) => then we don't want to invest a lot of time to learn a difficult language that is so unproductive. => and even if we do out of curiosity, we'll most likely never feel confident enough to dig into an open-source project and contribute. Rust on the other hand, is like C but with a personal IA assistant (the compiler) and all the syntactic sugar you're used to (and even more if you're not familiar with the ML family). That's what's attracting so many web developers to Rust ! In my opinion it means that, in a somewhat near future, the amount of potential contributors for projects written in Rust could be higher than for projects written in C. And those contributor are also likely to be more productive, thanks to the language features and tooling.
- thesmallestcat 10y ago
  You're describing qualities that are meaningless to a database user. A Rust implementation of PostgreSQL, with its runtime checks, would almost certainly be slower than the existing implementation. So a Rust port is a non-starter. Database users don't care about your syntactic sugar, that you find manual memory management a "scary land" due to background in managed languages, or how many contributors you have. Speaking of which, the PostgreSQL project is healthy and has many, many contributors. And while we're discussing developer productivity, I can't even imagine the build time of a Rust port. What's annoying about OP's email is that they clearly did no research on the matter. You can find many discussions on the list about "upgrading" to C++, and there has been much wringing of hands. The last I heard, it sounded like a faction was considering having the project build with a C++ compiler rather than a C compiler for better type checks and as a step on that path.
  - stymaar 10y ago
    > A Rust implementation of PostgreSQL, with its runtime checks, would almost certainly be slower than the existing implementation. So a Rust port is a non-starter. Wait, which runtime checks are you talking about ? This is a recurring FUD we often see on HN about Rust and I'd like to clarify this point. IIRC, there is no mandatory runtime check in idiomatic Rust except UTF8 validation for strings[0]. There is no overflow checks on numbers, and no bounds checking when using iterators[1]. Another known overhead is the default use of siphash in hashmaps for DoS prevention[2], but this is not really a «runtime check». Do you have anything else in mind ? > Speaking of which, the PostgreSQL project is healthy and has many, many contributors. Indeed, and as I said, I'm not advocating for a port of PostgreSQL in Rust. But for other projects that suffer from a lack of attention[3], it could be a good move. > And while we're discussing developer productivity, I can't even imagine the build time of a Rust port. touché, but compile time is one of the biggest focus of the Rust core team atm, so I'm confident things will improve in 2017 :) [0]: which can be worked around if need be by using &[u8] instead of &str (see: https://news.ycombinator.com/item?id=13268051 https://news.ycombinator.com/item?id=13268051 from /u/burntsushi) [1]: there are bounds checking when you access array elements through indexes, but most of the time you don't do that since iterators are great. If you really need to, and this have a performance impact, you can opt-out the bound checking in unsafe block. [2]: But this is opt-out by using a different hash function, if you use your hashmap in a non adversarial environment. [3]: like librsvg which was chronically under-maintained.