7 ms·

RustgreSQL

81 points by desiderantes 10y ago

purpleidea 10y ago
A lot of people aren't being very nice ITT and elsewhere and criticizing the guy for asking. If you don't ask you won't learn. Remember to be nice!
- saurik 10y ago
  This person could have and absolutely should have ran this past some other people before bringing it to the upstream PostgreSQL developers, not just wasting their time but putting them in the awkward position of having to respond for an audience.
flukus 10y ago
Aside from a complete OS, a database implementation that is performant seems like it would be the ultimate proof that rust can replace c. I did find a basic one: https://github.com/nukep/llamadb/blob/master/Usage.md https://github.com/nukep/llamadb/blob/master/Usage.md
nl 10y ago
My motivation is primarily I don't want to learn all the over-complicated details of C If the author finds C overcomplicated I look forward to finding out what they think of a complete database system.
- nathancahill 10y ago
  As a Postgres user, I like the idea. But their post comes off sounding like a JS dev wanting to rewrite an app in the latest popular language.
  - flukus 10y ago
    In this case there are potential performance and safety improvements by rewriting things in rust, the exact opposite of a JS rewrite.
    - deleted 10y ago
      [deleted]
    - SwellJoe 10y ago
      Why would Rust be faster than C? And, is potential safety 5+ years in the future (that's an absurdly optimistic estimate of a rewrite for an OSS project the size and scope of PostgreSQL) worth effectively standing still for all that time? Is the security/stability history of PostgreSQL poor? I'm not under that impression.
      - cornstalks 10y ago
        Rust isn't inherently faster than C, but it can be easier in some cases for the optimizer to optimize Rust code better than C code. In particular, I have in mind Rust's reference types and the fact that they all "follow LLVM’s scoped noalias model"[1]. You can, of course, get this same benefit in C by using restrict, but most of the C code I've seen makes little use of restrict (and it's quite easy to accidentally alias a restricted pointer, so it definitely imposes a mental burden, even if it's very slight). Rust's safety model allows you to get the optimization of C's restrict without the footgun. That can be a big win. But of course, just because something is written in Rust doesn't mean it'll be faster than the equivalent C code. Rust isn't magic, after all. [1]: https://doc.rust-lang.org/reference.html#behavior-considered-undefined https://doc.rust-lang.org/reference.html#behavior-considered...
        Sanddancer 10y ago
        This has been touted, but has there been a non-trivial project that shows this? Something like a media manipulation library where there are lots of places for optimization. It's been stable for a few years now, so I imagine someone's seen performance benefits if it's happened.
- hobarrera 10y ago
  I belive that this reply says it all > Well that's going to be a show-stopper right there. For a proper > port, a deep understanding of the current source code is necessary. > You'd need a team expert in both C and Rust to pull it off.
- rcthompson 10y ago
  Also, they don't seem to understand that Corrode doesn't generate idiomatic Rust, nor does it remove the "over-complicated details of C". From the Corrode README: "This tool does not fully automate the job because its output is only as safe as the input was; you should clean up the output afterward to use Rust features and idioms where appropriate." So keeping the Rust rewrite in sync with the C codebase via automatic translation with Corrode is mutually exclusive with actually rewriting it in safe idiomatic Rust. The weird thing is that they clearly have the sense to recognize that a rewrite would be a non-trivial project, but they seem to completely misunderstand why.
- Matthias247 10y ago
  Add to that that you basically need to learn everything about C too when you learn Rust. C (without macros) is pretty much a subset of Rust, and there's a lot to learn on top of that subset. The advantage of Rust is not that it's easier, it is that you will more likely get a compiler error than a (possibly hidden) runtime bug if you don't have fully understood something.
  - stymaar 10y ago
    > C (without macros) is pretty much a subset of Rust Well, yes and no. On the one hand unsafe Rust is roughly similar to C[1], and unsafe Rust is a subset of Rust. But unsafe Rust is not the most widely used subset of Rust, stable Rust is ! And stable Rust is arguably way easier to understand and work with than C. And you can write fairly complicated software without writing a single line of unsafe code. [1]: even though, the more expressive syntax and type system of Rust makes a lot of things less error-prone in Rust, like pattern matching or Result<> for error handling instead of switch and goto.
    - steveklabnik 10y ago
      Unsafe Rust is a _superset_ of safe Rust: it's the exact same thing, with some extra abilities. For that reason, "roughly similar to C" is also not exactly true; unsafe Rust, for example, does not remove any safety checks, it gives you access to extra unchecked things. A reference inside of an unsafe block is still borrow checked. But you can also use a raw pointer, that isn't.
      - stymaar 10y ago
        Thanks for your clarification. Your dedication to provide answer and explanations to absolutely everybody everywhere on the internet is incredible. How many people do you keep enslaved in your basement to do all that work ? :p
        steveklabnik 10y ago
        You're welcome. And I don't have a basement ;)
- dbaupp 10y ago
  I think the author is making a good point but possibly couching it in the wrong terms: C shifts a lot of the inherent complexity of systems programming and manual memory management onto the programmer, with the language offering little to no assistance. If you squint hard enough around the reems of specification and piles of undefined/implementation-defined behaviour in innocuous places, the language itself might indeed be something you could call simple, but actually writing correct code in it is not. There's a reason many, many person-years have been devoted to tooling: offloading as much of the "simplicity" as we can onto computers is the best way we know how to write correct C at scale. (That said, PostgreSQL clearly works well enough and has a system for writing new code that works well enough, so proposing a wholesale rewrite into a different language seems... unproductive, even moreso than usual.)
akst 10y ago
If you want to write extensions in Rust, here's a library for rust that utilises Postgresqls own allocators https://github.com/michaelsproul/postgres-alloc-rs https://github.com/michaelsproul/postgres-alloc-rs
cwyers 10y ago
> Porting PostgreSQL to Rust would be a multi-year project, That's... underselling it somewhat a little.
diziet 10y ago
I am not sure what to make of this: the author is a cto of a 100~ person company, but he is thinking that a project like pg can be realistically rewritten in rust. I just don't get it. Even if you could somehow make the 'complicated' and 'scary' parts of the code base magically transform into rust (much less handle performance and other issues due to memory management being different between c and rust), how would you maintain that project? Ai driven code translation from c to rust is a singularity too soon.
- rtpg 10y ago
  What's the argument against possible Rust porting of Postgres? Because of the interop ease, it's not hard to imagine a multi-year peicemeal rewrite. Postgres' code has some pretty good idioms that could be nicely encoded in Rust, and a good separation of concerns
  - jeffdavis 10y ago
    The biggest argument against it is falling into the "pure technical" trap. Postgres has a great focus on solving real problems for real people. Losing that focus would be a disaster.
  - coldtea 10y ago
    >What's the argument against possible Rust porting of Postgres? That's putting the cart before the horse. The right question is: a) What's the argument in favor of porting Postgres to Rust? and b) What's the opportunity cost?
- ww520 10y ago
  This is worrisome for the company, to have someone as clueless about technical work scope and estimate to be in charge of the technical department. Or is that more of a political post?
  - nickez 10y ago
    > clueless about technical work scope Do you mean that it would be impossible for a team of experts in C and rust to rewrite pg in 5 years? That is essentially what he is proposing. (Which probably isn't a great idea, but I think the work scope estimate is in the right ball park.)
hendzen 10y ago
More importantly, when are we going to see a port of PostgreSQL to React?
andrewchambers 10y ago
This person doesn't seem to know much about reality. Or maybe I'm just a jaded jerk.
- michaelvoz 10y ago
  Agreed. Not discounting that you are a jaded jerk, as I think I may be one myself.
iagooar 10y ago
I would suggest that the author creates a clone of a "simpler" DB like SQLite or Redis. Those have, as far as I understand, good C codebases, so understanding the system should be much easier. This way you could also say: hey! I have successfully ported a DB written in C in the past, and it has been an extremely positive experience, how about trying the same with Postgres?
- nathancahill 10y ago
  SQLite for sure. The test suite is top tier.
  - dllthomas 10y ago
    Hopefully you could even reuse it!
    - rosser 10y ago
      If you wanted to write a drop-in SQLite replacement in Rust, using the original project's (incredible) test suite would be on the mandatory side, wouldn't it?
      - mathw 10y ago
        Definitely! And Rust is capable of that kind of thing (producing a drop-in replacement for a C library) so there'd be no excuses either.
      - dllthomas 10y ago
        To a degree. Certainly the end-to-end tests should be something you can expect to reuse. Other tests may be hit or miss. In the case of SQLite, where there's >100x as much test code as application code, there's a large range between "can use some of it" and "can use most of it" - as mathw gets at I think we should be able to expect the latter here.
- riffraff 10y ago
  redis-in-rust : https://github.com/seppo0010/rsedis https://github.com/seppo0010/rsedis it seems many people re-implement redis (I recall at least also Go and Scala). It looks like it's the new "backend language todo list project".
kibwen 10y ago
Come now, this is just silly. And I say this as someone who's been using Rust for years now! Let's avoid upvoting stuff this insubstantial.
- stymaar 10y ago
  HN is just weird … A guy randomly asks a question on a mailing list, and then his mail goes in the front page, and then people start insulting him: «blah blah, stupid politic CTO … understand nothing, blah blah blah».
  - ww520 10y ago
    Because this reminds us of the damage done by technical decisions made by ignorant people in charge. The resonance draws out the overwhelming reaction.
Jweb_Guru 10y ago
It's perfectly feasible to write something like Postgres in Rust, and it's a good idea, too! But Postgres is not going to be that database, for a huge number of reasons. Even Mozilla isn't replacing Gecko wholesale with Servo.
alexnewman 10y ago
This isn't a port of postgresql, more of an attempt to extend it's functionality into new spaces with the power of rust. github.com/posaix4e/rpgffi (logical decoders and foreign data wrappers in wrust) github.com/posix4e/jsoncdc ( an example logical decoder in rust) We are going to be adding bigtable and s3 fdw soon.
andrewvijay 10y ago
Is the C language more complicated than building a database?! That's looking to be way more exaggerated.
- stymaar 10y ago
  For people coming from managed languages (and in 2017, the huge majority of us is), contributing to a C project is way more complicated than contributing to a database project. For open-source projects, being written in a scary[1] language is a liability in the long run. For many tasks, we used to have no good alternative than using C (or C++ which is even scarier), but this is no longer the case now thanks to Rust. I'm obviously not saying we should rewrite PostgreSQL in Rust, but the ability to grow/maintain the contributor base is an argument that should not be underestimated. [1]: no matter how difficult or not it is in practice, the C is scary for many people and most of them will never feel confident enough to suggest changes in a C code base.
  - pjmlp 10y ago
    As someone coding since the 80's I never understood what is scary about programming languages. Scary is having to run from a lion to save our life.
  - riffraff 10y ago
    > For people coming from managed languages[...] contributing to a C project is way more complicated than contributing to a database project. I think people do have this feeling, but likely it's because they don't actually know much about writing databases, so they see one issue but not the other. OTOH it might be possible to contribute small things (e.g. improving the database client program, adding a test) if one can write some code in $LANG_MORE_ACCESSIBLE_THAN_C, even without knowing what a btree is or what "cache locality" means.
  - thesmallestcat 10y ago
    Out of curiosity, if you find C and C++ to be scary and unapproachable, how on earth can you feel confident in your assessment that Rust is a good replacement for it?
prymitive 10y ago
My motivation is primarily I don't want to learn all the over-complicated details of C I don't understand this statement. Is he saying he want to contribute to postgres, but he doesn't know C well enough, so he kindly asks the team to consider the idea of a rewrite in Rust to make it easy for him?
solidsnack9000 10y ago
Postgres's extension APIs are pretty good and Rust's interoperability with C makes it quite feasible to extend the database without rewriting it. Some months ago we wrote a logical decoder -- to capture and forward the write-ahead log as JSON -- in Rust and later were able to pull out some helpful bindings for logical decoders, foreign data wrappers and so forth. The logical decoder: http://pgxn.org/dist/jsoncdc http://pgxn.org/dist/jsoncdc The bindings: https://github.com/posix4e/rpgffi https://github.com/posix4e/rpgffi The extensibility of Postgres is really profound -- there are so many things you can do, and so much internal functionality you can access, without having to recompile the database. The way extensions are loaded also mitigates the risk of what can be admittedly tricky issues relating to safety. Extensions run in backend processes, not the postmaster. If rewriting Postgres happens, I hope it would be gradual. You can replace a lot of the core with extensions, and the scope for extension is ever increasing. So say you wrote a storage engine in Rust. Writing a foreign data wrapper for it would give you a way to use it from Postgres for all your tables; but you wouldn't have to write a SQL parser, client manager, permissions system, or any of the rest of that stuff. Although the C stuff is a little gritty, a lot of what goes on in a database is gritty "because it is" and that low-level control is needed to deliver on tail latency, fault tolerance or simple storage efficiency. For example, Postgres has code to recover from malloc failing.
- masklinn 10y ago
  I'm not sure there's much reason to rewrite Postgres as a whole. If there are specific subsystems which are common sources of CVE they might be targets, but looking at the list[0] no specific system really seems to jump out. Extensions could be neat, there doesn't seem to be a high-level Rust API for it (in the style of ruru or rust-cpython) yet though, only incomplete-looking bindgen: https://github.com/posix4e/rpgffi https://github.com/posix4e/rpgffi [0] https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=postgresql https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=postgresql
  - stymaar 10y ago
    I'm not advocating for a complete rewrite of Postgres in C, but I'd like to emphasize the fact that security is not the only good reason to justify the transition of a project from C to Rust: Nowadays most developers comes from memory-managed languages => for us, C is a scary land where it's really hard to get things right when you don't have all the prerequisite knowledge (what's the stack ?) => the language is really old, you don't have all the fancy and useful tools and syntactic sugar (package manager, proper error handling, generics) => then we don't want to invest a lot of time to learn a difficult language that is so unproductive. => and even if we do out of curiosity, we'll most likely never feel confident enough to dig into an open-source project and contribute. Rust on the other hand, is like C but with a personal IA assistant (the compiler) and all the syntactic sugar you're used to (and even more if you're not familiar with the ML family). That's what's attracting so many web developers to Rust ! In my opinion it means that, in a somewhat near future, the amount of potential contributors for projects written in Rust could be higher than for projects written in C. And those contributor are also likely to be more productive, thanks to the language features and tooling.
timClicks 10y ago
I really hope that this doesn't mean we've already hit peak Rust. The recent hype of the language is great, but I think it's really important to start seeing substantial large projects receiving productivity gains otherwise it'll attract derision and then loose its momentum.
- mathw 10y ago
  Firefox should be the poster child for this, with the Oxidation project now in full swing. I'm hoping to see similar work appearing elsewhere though. There's no reason why any large C or C++ project couldn't decide to write some new component in Rust, if they felt it would benefit them. But they would need to have a contributor and maintainer base sufficiently motivated to learn Rust and stick with it. Which will happen more and more if Firefox can be visibly shown to benefit from including Rust in their toolkit.
rcthompson 10y ago
I was immediately reminded of this: http://thedailywtf.com/articles/unpythonic http://thedailywtf.com/articles/unpythonic (TLDR: CTO comes in and says "Angular and TypeScript is old. Python is new, and Google is writing everything with it. Python is the best practices for development." Tells IT department to rewrite everything in Python. “If our team still needs to use Angular, just use the Python version.” After failing to convince CTO this is absurd, IT realizes they can do nothing and tell the CTO they totally rewrote everything in Python, since he won't know the difference anyway.) This case isn't quite that bad, obviously. The request is more reasonable, but still seems to betray a pretty fundamental misunderstanding of the situation.
- smnscu 10y ago
  > Python is new I just realised Python is roughly my age.
- sebcat 10y ago
  The sender of the e-mail is also a CTO. A CTO of a payment company, so the inquiry to see if it would be possible to replace a large C project in their stack with a drop-in replacement written in a more safe language makes sense from where he's standing.
deleted 10y ago
[deleted]
nostrademons 10y ago
I assumed this was satire of the various "Let's rewrite X in Rust!" posts, but perhaps that's just Poe's Law talking.
protomikron 10y ago
Ok, I really like the approach Rust is taking and appreciate their effort to make low-level system programming safer, but it really is getting ridiculous. Not that implementing a DB system in Rust would be infeasible, but advocating to just rewrite some large projects in Rust is probably not the step forward. Waiting for a flamewar on the LKML on this. "Hey, Linux devs. I heard there is this nice language called Rust. Do you mind rewriting your OS in this? Ok, might not be possible over the weekend, but maybe over a year? Maybe use some Deep Learning to automagically transform the code?" I smell an upcoming Linus Rust rant.
- mrkgnao 10y ago
  And a corresponding NVIDIA video?
- anarazel 10y ago
  > Ok, I really like the approach Rust is taking and appreciate their effort to make low-level system programming safer, but it really is getting ridiculous. Not that implementing a DB system in Rust would be infeasible, but advocating to just rewrite some large projects in Rust is probably not the step forward. Note that the OP has, afaik, no involvement in the rust project. Nor has he done much hacking on postgres. Don't think the rust people can be blamed for that one.
  - protomikron 10y ago
    Yeah, that was probably phrased poorly. I wanted to criticize the "just rewrite it in Rust" evangelists, not the Rust makers.
belvoran 10y ago
I'm getting sick of that Hype Driven Development.
bwblabs 10y ago
A port to a way more strict type and memory safe language would be great, but as Craig Ringer from 2ndquadrant replies: the postgres project is pretty conservative & this could only happen in an incremental way.
perlgeek 10y ago
FWIW if somebody were to port a DBMS like PostgreSQL to Rust, or any other programming language, it would only be feasibly to do it component by component. Much like Mozilla now plans to use Servo for CSS rendering first, and maybe for more tasks later.
deleted 10y ago
[deleted]
pikzen 10y ago
That's it, we've reached peak circlejerk. Sorry steveklabnik, Manishearth, any rustacean, we're gonna have to shut it down.
myf01d 10y ago
> Let's rewrite & test a 20 year old already working & popular project with ~1,000,000 lines of code just because I don't fucking know what I am saying. Please engineers try to figure that out and come up with solution before Monday.
titraprutr 10y ago
This feels more like a troll than a genuine question.
aibottle 10y ago
You jsut went full retard. Never go full retard!

81 points by desiderantes 10y ago

A lot of people aren't being very nice ITT and elsewhere and criticizing the guy for asking. If you don't ask you won't learn. Remember to be nice!

saurik 10y ago
This person could have and absolutely should have ran this past some other people before bringing it to the upstream PostgreSQL developers, not just wasting their time but putting them in the awkward position of having to respond for an audience.

flukus 10y ago

Aside from a complete OS, a database implementation that is performant seems like it would be the ultimate proof that rust can replace c. I did find a basic one: https://github.com/nukep/llamadb/blob/master/Usage.md https://github.com/nukep/llamadb/blob/master/Usage.md

nl 10y ago

My motivation is primarily I don't want to learn all the over-complicated details of C If the author finds C overcomplicated I look forward to finding out what they think of a complete database system.

nathancahill 10y ago
As a Postgres user, I like the idea. But their post comes off sounding like a JS dev wanting to rewrite an app in the latest popular language.
- flukus 10y ago
  In this case there are potential performance and safety improvements by rewriting things in rust, the exact opposite of a JS rewrite.
  - deleted 10y ago
    [deleted]
  - SwellJoe 10y ago
    Why would Rust be faster than C? And, is potential safety 5+ years in the future (that's an absurdly optimistic estimate of a rewrite for an OSS project the size and scope of PostgreSQL) worth effectively standing still for all that time? Is the security/stability history of PostgreSQL poor? I'm not under that impression.
    - cornstalks 10y ago
      Rust isn't inherently faster than C, but it can be easier in some cases for the optimizer to optimize Rust code better than C code. In particular, I have in mind Rust's reference types and the fact that they all "follow LLVM’s scoped noalias model"[1]. You can, of course, get this same benefit in C by using restrict, but most of the C code I've seen makes little use of restrict (and it's quite easy to accidentally alias a restricted pointer, so it definitely imposes a mental burden, even if it's very slight). Rust's safety model allows you to get the optimization of C's restrict without the footgun. That can be a big win. But of course, just because something is written in Rust doesn't mean it'll be faster than the equivalent C code. Rust isn't magic, after all. [1]: https://doc.rust-lang.org/reference.html#behavior-considered-undefined https://doc.rust-lang.org/reference.html#behavior-considered...
      - Sanddancer 10y ago
        This has been touted, but has there been a non-trivial project that shows this? Something like a media manipulation library where there are lots of places for optimization. It's been stable for a few years now, so I imagine someone's seen performance benefits if it's happened.
hobarrera 10y ago
I belive that this reply says it all > Well that's going to be a show-stopper right there. For a proper > port, a deep understanding of the current source code is necessary. > You'd need a team expert in both C and Rust to pull it off.
rcthompson 10y ago
Also, they don't seem to understand that Corrode doesn't generate idiomatic Rust, nor does it remove the "over-complicated details of C". From the Corrode README: "This tool does not fully automate the job because its output is only as safe as the input was; you should clean up the output afterward to use Rust features and idioms where appropriate." So keeping the Rust rewrite in sync with the C codebase via automatic translation with Corrode is mutually exclusive with actually rewriting it in safe idiomatic Rust. The weird thing is that they clearly have the sense to recognize that a rewrite would be a non-trivial project, but they seem to completely misunderstand why.
Matthias247 10y ago
Add to that that you basically need to learn everything about C too when you learn Rust. C (without macros) is pretty much a subset of Rust, and there's a lot to learn on top of that subset. The advantage of Rust is not that it's easier, it is that you will more likely get a compiler error than a (possibly hidden) runtime bug if you don't have fully understood something.
- stymaar 10y ago
  > C (without macros) is pretty much a subset of Rust Well, yes and no. On the one hand unsafe Rust is roughly similar to C[1], and unsafe Rust is a subset of Rust. But unsafe Rust is not the most widely used subset of Rust, stable Rust is ! And stable Rust is arguably way easier to understand and work with than C. And you can write fairly complicated software without writing a single line of unsafe code. [1]: even though, the more expressive syntax and type system of Rust makes a lot of things less error-prone in Rust, like pattern matching or Result<> for error handling instead of switch and goto.
  - steveklabnik 10y ago
    Unsafe Rust is a _superset_ of safe Rust: it's the exact same thing, with some extra abilities. For that reason, "roughly similar to C" is also not exactly true; unsafe Rust, for example, does not remove any safety checks, it gives you access to extra unchecked things. A reference inside of an unsafe block is still borrow checked. But you can also use a raw pointer, that isn't.
    - stymaar 10y ago
      Thanks for your clarification. Your dedication to provide answer and explanations to absolutely everybody everywhere on the internet is incredible. How many people do you keep enslaved in your basement to do all that work ? :p
      - steveklabnik 10y ago
        You're welcome. And I don't have a basement ;)
dbaupp 10y ago
I think the author is making a good point but possibly couching it in the wrong terms: C shifts a lot of the inherent complexity of systems programming and manual memory management onto the programmer, with the language offering little to no assistance. If you squint hard enough around the reems of specification and piles of undefined/implementation-defined behaviour in innocuous places, the language itself might indeed be something you could call simple, but actually writing correct code in it is not. There's a reason many, many person-years have been devoted to tooling: offloading as much of the "simplicity" as we can onto computers is the best way we know how to write correct C at scale. (That said, PostgreSQL clearly works well enough and has a system for writing new code that works well enough, so proposing a wholesale rewrite into a different language seems... unproductive, even moreso than usual.)

akst 10y ago

If you want to write extensions in Rust, here's a library for rust that utilises Postgresqls own allocators https://github.com/michaelsproul/postgres-alloc-rs https://github.com/michaelsproul/postgres-alloc-rs

cwyers 10y ago

> Porting PostgreSQL to Rust would be a multi-year project, That's... underselling it somewhat a little.

diziet 10y ago

I am not sure what to make of this: the author is a cto of a 100~ person company, but he is thinking that a project like pg can be realistically rewritten in rust. I just don't get it. Even if you could somehow make the 'complicated' and 'scary' parts of the code base magically transform into rust (much less handle performance and other issues due to memory management being different between c and rust), how would you maintain that project? Ai driven code translation from c to rust is a singularity too soon.

rtpg 10y ago
What's the argument against possible Rust porting of Postgres? Because of the interop ease, it's not hard to imagine a multi-year peicemeal rewrite. Postgres' code has some pretty good idioms that could be nicely encoded in Rust, and a good separation of concerns
- jeffdavis 10y ago
  The biggest argument against it is falling into the "pure technical" trap. Postgres has a great focus on solving real problems for real people. Losing that focus would be a disaster.
- coldtea 10y ago
  >What's the argument against possible Rust porting of Postgres? That's putting the cart before the horse. The right question is: a) What's the argument in favor of porting Postgres to Rust? and b) What's the opportunity cost?
ww520 10y ago
This is worrisome for the company, to have someone as clueless about technical work scope and estimate to be in charge of the technical department. Or is that more of a political post?
- nickez 10y ago
  > clueless about technical work scope Do you mean that it would be impossible for a team of experts in C and rust to rewrite pg in 5 years? That is essentially what he is proposing. (Which probably isn't a great idea, but I think the work scope estimate is in the right ball park.)

hendzen 10y ago

More importantly, when are we going to see a port of PostgreSQL to React?

andrewchambers 10y ago

This person doesn't seem to know much about reality. Or maybe I'm just a jaded jerk.

michaelvoz 10y ago
Agreed. Not discounting that you are a jaded jerk, as I think I may be one myself.

iagooar 10y ago

I would suggest that the author creates a clone of a "simpler" DB like SQLite or Redis. Those have, as far as I understand, good C codebases, so understanding the system should be much easier. This way you could also say: hey! I have successfully ported a DB written in C in the past, and it has been an extremely positive experience, how about trying the same with Postgres?

nathancahill 10y ago
SQLite for sure. The test suite is top tier.
- dllthomas 10y ago
  Hopefully you could even reuse it!
  - rosser 10y ago
    If you wanted to write a drop-in SQLite replacement in Rust, using the original project's (incredible) test suite would be on the mandatory side, wouldn't it?
    - mathw 10y ago
      Definitely! And Rust is capable of that kind of thing (producing a drop-in replacement for a C library) so there'd be no excuses either.
    - dllthomas 10y ago
      To a degree. Certainly the end-to-end tests should be something you can expect to reuse. Other tests may be hit or miss. In the case of SQLite, where there's >100x as much test code as application code, there's a large range between "can use some of it" and "can use most of it" - as mathw gets at I think we should be able to expect the latter here.
riffraff 10y ago
redis-in-rust : https://github.com/seppo0010/rsedis https://github.com/seppo0010/rsedis it seems many people re-implement redis (I recall at least also Go and Scala). It looks like it's the new "backend language todo list project".

kibwen 10y ago

Come now, this is just silly. And I say this as someone who's been using Rust for years now! Let's avoid upvoting stuff this insubstantial.

stymaar 10y ago
HN is just weird … A guy randomly asks a question on a mailing list, and then his mail goes in the front page, and then people start insulting him: «blah blah, stupid politic CTO … understand nothing, blah blah blah».
- ww520 10y ago
  Because this reminds us of the damage done by technical decisions made by ignorant people in charge. The resonance draws out the overwhelming reaction.

Jweb_Guru 10y ago

It's perfectly feasible to write something like Postgres in Rust, and it's a good idea, too! But Postgres is not going to be that database, for a huge number of reasons. Even Mozilla isn't replacing Gecko wholesale with Servo.

alexnewman 10y ago

This isn't a port of postgresql, more of an attempt to extend it's functionality into new spaces with the power of rust. github.com/posaix4e/rpgffi (logical decoders and foreign data wrappers in wrust) github.com/posix4e/jsoncdc ( an example logical decoder in rust) We are going to be adding bigtable and s3 fdw soon.

andrewvijay 10y ago

Is the C language more complicated than building a database?! That's looking to be way more exaggerated.

stymaar 10y ago
For people coming from managed languages (and in 2017, the huge majority of us is), contributing to a C project is way more complicated than contributing to a database project. For open-source projects, being written in a scary[1] language is a liability in the long run. For many tasks, we used to have no good alternative than using C (or C++ which is even scarier), but this is no longer the case now thanks to Rust. I'm obviously not saying we should rewrite PostgreSQL in Rust, but the ability to grow/maintain the contributor base is an argument that should not be underestimated. [1]: no matter how difficult or not it is in practice, the C is scary for many people and most of them will never feel confident enough to suggest changes in a C code base.
- pjmlp 10y ago
  As someone coding since the 80's I never understood what is scary about programming languages. Scary is having to run from a lion to save our life.
- riffraff 10y ago
  > For people coming from managed languages[...] contributing to a C project is way more complicated than contributing to a database project. I think people do have this feeling, but likely it's because they don't actually know much about writing databases, so they see one issue but not the other. OTOH it might be possible to contribute small things (e.g. improving the database client program, adding a test) if one can write some code in $LANG_MORE_ACCESSIBLE_THAN_C, even without knowing what a btree is or what "cache locality" means.
- thesmallestcat 10y ago
  Out of curiosity, if you find C and C++ to be scary and unapproachable, how on earth can you feel confident in your assessment that Rust is a good replacement for it?

prymitive 10y ago

My motivation is primarily I don't want to learn all the over-complicated details of C I don't understand this statement. Is he saying he want to contribute to postgres, but he doesn't know C well enough, so he kindly asks the team to consider the idea of a rewrite in Rust to make it easy for him?

solidsnack9000 10y ago

Postgres's extension APIs are pretty good and Rust's interoperability with C makes it quite feasible to extend the database without rewriting it. Some months ago we wrote a logical decoder -- to capture and forward the write-ahead log as JSON -- in Rust and later were able to pull out some helpful bindings for logical decoders, foreign data wrappers and so forth. The logical decoder: http://pgxn.org/dist/jsoncdc http://pgxn.org/dist/jsoncdc The bindings: https://github.com/posix4e/rpgffi https://github.com/posix4e/rpgffi The extensibility of Postgres is really profound -- there are so many things you can do, and so much internal functionality you can access, without having to recompile the database. The way extensions are loaded also mitigates the risk of what can be admittedly tricky issues relating to safety. Extensions run in backend processes, not the postmaster. If rewriting Postgres happens, I hope it would be gradual. You can replace a lot of the core with extensions, and the scope for extension is ever increasing. So say you wrote a storage engine in Rust. Writing a foreign data wrapper for it would give you a way to use it from Postgres for all your tables; but you wouldn't have to write a SQL parser, client manager, permissions system, or any of the rest of that stuff. Although the C stuff is a little gritty, a lot of what goes on in a database is gritty "because it is" and that low-level control is needed to deliver on tail latency, fault tolerance or simple storage efficiency. For example, Postgres has code to recover from malloc failing.

masklinn 10y ago
I'm not sure there's much reason to rewrite Postgres as a whole. If there are specific subsystems which are common sources of CVE they might be targets, but looking at the list[0] no specific system really seems to jump out. Extensions could be neat, there doesn't seem to be a high-level Rust API for it (in the style of ruru or rust-cpython) yet though, only incomplete-looking bindgen: https://github.com/posix4e/rpgffi https://github.com/posix4e/rpgffi [0] https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=postgresql https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=postgresql
- stymaar 10y ago
  I'm not advocating for a complete rewrite of Postgres in C, but I'd like to emphasize the fact that security is not the only good reason to justify the transition of a project from C to Rust: Nowadays most developers comes from memory-managed languages => for us, C is a scary land where it's really hard to get things right when you don't have all the prerequisite knowledge (what's the stack ?) => the language is really old, you don't have all the fancy and useful tools and syntactic sugar (package manager, proper error handling, generics) => then we don't want to invest a lot of time to learn a difficult language that is so unproductive. => and even if we do out of curiosity, we'll most likely never feel confident enough to dig into an open-source project and contribute. Rust on the other hand, is like C but with a personal IA assistant (the compiler) and all the syntactic sugar you're used to (and even more if you're not familiar with the ML family). That's what's attracting so many web developers to Rust ! In my opinion it means that, in a somewhat near future, the amount of potential contributors for projects written in Rust could be higher than for projects written in C. And those contributor are also likely to be more productive, thanks to the language features and tooling.

timClicks 10y ago

I really hope that this doesn't mean we've already hit peak Rust. The recent hype of the language is great, but I think it's really important to start seeing substantial large projects receiving productivity gains otherwise it'll attract derision and then loose its momentum.

mathw 10y ago
Firefox should be the poster child for this, with the Oxidation project now in full swing. I'm hoping to see similar work appearing elsewhere though. There's no reason why any large C or C++ project couldn't decide to write some new component in Rust, if they felt it would benefit them. But they would need to have a contributor and maintainer base sufficiently motivated to learn Rust and stick with it. Which will happen more and more if Firefox can be visibly shown to benefit from including Rust in their toolkit.

rcthompson 10y ago

I was immediately reminded of this: http://thedailywtf.com/articles/unpythonic http://thedailywtf.com/articles/unpythonic (TLDR: CTO comes in and says "Angular and TypeScript is old. Python is new, and Google is writing everything with it. Python is the best practices for development." Tells IT department to rewrite everything in Python. “If our team still needs to use Angular, just use the Python version.” After failing to convince CTO this is absurd, IT realizes they can do nothing and tell the CTO they totally rewrote everything in Python, since he won't know the difference anyway.) This case isn't quite that bad, obviously. The request is more reasonable, but still seems to betray a pretty fundamental misunderstanding of the situation.

smnscu 10y ago
> Python is new I just realised Python is roughly my age.
sebcat 10y ago
The sender of the e-mail is also a CTO. A CTO of a payment company, so the inquiry to see if it would be possible to replace a large C project in their stack with a drop-in replacement written in a more safe language makes sense from where he's standing.

deleted 10y ago

[deleted]

nostrademons 10y ago

I assumed this was satire of the various "Let's rewrite X in Rust!" posts, but perhaps that's just Poe's Law talking.

protomikron 10y ago

Ok, I really like the approach Rust is taking and appreciate their effort to make low-level system programming safer, but it really is getting ridiculous. Not that implementing a DB system in Rust would be infeasible, but advocating to just rewrite some large projects in Rust is probably not the step forward. Waiting for a flamewar on the LKML on this. "Hey, Linux devs. I heard there is this nice language called Rust. Do you mind rewriting your OS in this? Ok, might not be possible over the weekend, but maybe over a year? Maybe use some Deep Learning to automagically transform the code?" I smell an upcoming Linus Rust rant.

mrkgnao 10y ago
And a corresponding NVIDIA video?
anarazel 10y ago
> Ok, I really like the approach Rust is taking and appreciate their effort to make low-level system programming safer, but it really is getting ridiculous. Not that implementing a DB system in Rust would be infeasible, but advocating to just rewrite some large projects in Rust is probably not the step forward. Note that the OP has, afaik, no involvement in the rust project. Nor has he done much hacking on postgres. Don't think the rust people can be blamed for that one.
- protomikron 10y ago
  Yeah, that was probably phrased poorly. I wanted to criticize the "just rewrite it in Rust" evangelists, not the Rust makers.

belvoran 10y ago

I'm getting sick of that Hype Driven Development.

bwblabs 10y ago

A port to a way more strict type and memory safe language would be great, but as Craig Ringer from 2ndquadrant replies: the postgres project is pretty conservative & this could only happen in an incremental way.

perlgeek 10y ago

FWIW if somebody were to port a DBMS like PostgreSQL to Rust, or any other programming language, it would only be feasibly to do it component by component. Much like Mozilla now plans to use Servo for CSS rendering first, and maybe for more tasks later.

deleted 10y ago

[deleted]

pikzen 10y ago

That's it, we've reached peak circlejerk. Sorry steveklabnik, Manishearth, any rustacean, we're gonna have to shut it down.

myf01d 10y ago

> Let's rewrite & test a 20 year old already working & popular project with ~1,000,000 lines of code just because I don't fucking know what I am saying. Please engineers try to figure that out and come up with solution before Monday.

titraprutr 10y ago

This feels more like a troll than a genuine question.

aibottle 10y ago

You jsut went full retard. Never go full retard!