29 ms·

Rust – A hard decision pays off

474 points by muizelaar 4y ago

fb03 4y ago
Would you guys start a GraphQL Api today using Rust? I'm very tempted to hint at that but am still insecure about my 'options'. This is coming from a very comfortable python + fastapi + sqlalchemy ecossystem
jraph 4y ago
The title is "Inside the Pinecone". "Rust – A hard decision pays off" is only one of the three major subsections of this article.
- gk1 4y ago
  Interesting to see the part about Rust getting the most attention. We're giving a talk about it in NYC next week[1] and will definitely follow up with an in-depth writeup. [1] https://www.meetup.com/rust-nyc/events/287821884/ https://www.meetup.com/rust-nyc/events/287821884/
  - dfinninger 4y ago
    The submitted link auto-scrolls me to the section on Rust, so I guess the submitter wanted to promote that specific section of the article?
- 3a2d29 4y ago
  Gotta mention rust, how else will you get upvotes on HN?
- eterm 4y ago
  The title of the section posted is "Rust - A hard decision...". It's been posted here with the anchor included making it clear that it is that section that has been submitted.
  - jraph 4y ago
    Indeed, I didn't notice. Sorry!
amelius 4y ago
Nice to see that Rust is working for your particular usecase!
benreesman 4y ago
So I was one of the 3 hackers who wrote the LSM that predated Rocks at FB, and as a result I distinctly remember when Rocks overtook it in features by trading off performance, and again when it met and exceeded the performance while keeping the greater capability. RocksDB kicks ass and has for a long time, I’d recommend it to anyone in the market for that sort of thing. FAISS is also great, and has been for years. I don’t doubt that someone has done a meaningful rev on it. But as someone who likes Rust and wants to like it more: don’t lead with that. The software speaks for itself. Or should.
- bluejekyll 4y ago
  > But as someone who likes Rust and wants to like it more: don’t lead with that. The original title appears to be “Inside the Pinecone”, but seeing as they submitted it themselves I’m guessing they wanted the uptick on the Rust. The article itself doesn’t discuss their transition to Rust until the end. So I wouldn’t say they led with it originally, but perhaps didn’t get traction on the original title.
  - gk1 4y ago
    > seeing as they submitted it themselves I'm grateful to muizelaar but I don't know who that is, so we can't take credit for this one.
    - bluejekyll 4y ago
      Oh, sorry. I misattributed that. I saw the domain origin and confused that for the submitter. In that case, the submission actually is against the rules of HN, as I understand them (use the original title rule).
  - benreesman 4y ago
    I guess I’m rooting for Rust in the long game sense, which is a different set of imperatives than the hang the hashtag on peripheral stuff sense. I use a lot of great software written in Rust, it’s demonstrably a good vehicle for great software. But too many of its fans are advocates, this can start to seem like an agenda. Don’t take engineering advice from people with an agenda.
endisneigh 4y ago
Maybe it’s just me but it seems irresponsible to move your entire team to a programming language none of them knows. Rust is great, but modern C++ is great too.
- moffkalast 4y ago
  > modern C++ is great Great in the way that Alexander the Great was great. Achieved big things and killed a lot of people. Though in the case of C++ they're just dead inside.
  - benreesman 4y ago
    There are better threads to trash C++ hackers on than this one. It’s literally about insanely successful software written in C++, maintained successfully for a decade, that seems to have survived a port to another language.
  - FpUser 4y ago
    >"Though in the case of C++ they're just dead inside." I think this is pure and memory safe BS
- cratermoon 4y ago
  > it seems irresponsible to move your entire team to a programming language none of them knows Based on my experiences, I would agree. There seems to be a very important difference between my experiences, which probably align with most of the industry, and the Pinecone team. My career has been working with average developers doing ordinary business and internet things. Stuff that takes organization and teamwork, but not necessarily advanced or esoteric knowledge. So the COBOL programmers that went to the "Java in 21 Days" training that I consulted with, or the C# programmers who went with Go, they did poor-to-OK, but that was Good Enough. The Pinecone team, though. They were already writing C/C++ to solve Hard Problems. As experienced programmers, they probably already understood the landscape of systems languages with pointers and having to manage their own memory. Rust, while imposing, would not be a huge leap for them. Another aspect worth noting: they did they re-write incrementally, never completely abandoning the existing system. At least it seems so. I'll be interested in seeing the presentation from the upcoming talk about the rewrite that's mentioned in the story.
0xbeefeed 4y ago
[flagged]
throwaway642012 4y ago
I don't want to be dismissive but how much of this success belongs to the team of rock solid senior engineers with enough experience under their belt rather than rewrite in Rust?
- hawk_ 4y ago
  I would let the same team of "rock solid senior engineers" be the judge of that. They seem to think it mattered significantly enough.
  - amelius 4y ago
    Or ... since they are now betting fully on Rust, they now want everybody else on Rust too. Because more people == a higher probability that if they run into issues with Rust someone will fix it.
    - farresito 4y ago
      That's a really cynical take on the matter...
- robertlagrant 4y ago
  The writer seems to give massive credit to the team.
- runevault 4y ago
  You can give some credit to the fact they were writing it a second time certainly, but as far as we know at least a lot of the people were the same ones who were central to writing V1. So certainly there was some lift from already doing it once, but if they went from a lot of subtle bugs they struggled to track down to having far fewer of them, and those bugs tended to be the types of issue the rust compiler and/or linter normally point out to you, it will certainly make it easier to mitigate the issues. Rust cannot make a bad programmer a good programmer, but the fact it points out several classes of mistakes (sometimes too aggressively, hopefully someday we get Polonia and other improvements to the borrow checker to remove some things that are hard only because of the current implementation of various checking systems) allows far more confidence. I don't care how good someone is, having an automated way to validate large scales of issues aren't there is a big deal, and unlike unit/integration tests you don't have to write these.
mhaberl 4y ago
> ..complex runtime issues which were almost impossible to reproduce or isolate > That’s when internal murmurs about a complete rewrite started brewing… > We decided to move our entire codebase to Rust > there was still one minor problem - no one on the team knew Rust "We have runtime issues so we will fix them by a complete rewrite in a language that no one on the team knows." I would not call that a "hard decision", rather "rolling the dice". It awesome that they managed to make it work.
- moonchrome 4y ago
  I mean when the tools you know are failing you, you might as well roll the dice.
  - draw_down 4y ago
  - remram 4y ago
    They could have had some team members learn Rust before deciding that Rust was the right language for the re-write. It feels really risky to do such an important decision only on a high-level view of what you think the language will provide.
- bluejekyll 4y ago
  I think of rolling the dice as taking a chance without any knowledge of the chance for it paying of. Rust fit the need they had pretty well (on paper anyway since they didn’t have experience with it), and the industry is showing it to be a good choice for solving these types of problems. That’s not rolling the dice, it’s taking a chance based on research.
  - meragrin_ 4y ago
    > I think of rolling the dice as taking a chance without any knowledge of the chance for it paying of. I think hoping a Rust rewrite would fix "complex runtime issues which were almost impossible to reproduce or isolate" as rolling the dice. If you cannot reproduce or isolate an issue, how can you know a rewrite in another language will fix the issue?
ramoz 4y ago
> Simultaneous fetches of thousands (sometimes tens of thousands) of objects started becoming inefficient, especially when fetching from collections of tens of millions of objects Why not try leveldb. We're doing random reads of 170,000 vectors (/130M) a second. No startup needed.
- pas 4y ago
  Isn't RocksDB a fork (reimplementation) of LevelDB? :o
- benreesman 4y ago
  RocksDB started as a fork of LevelDB (Dean/Ghemawat). The performance imperatives have changed with the hardware, but NAND flash at that time had an asymmetry between reads and writes in terms of the amount of data: once you were writing even one byte you had effectively paid for writing a whole block and were therefore incentivized to get your money’s worth and write to a “log”, which then would be “merged”, in some “structured” way, hence LSM. This is old news these days but it was quite the novelty at the time!
ncmncm 4y ago
Leading with several pages of what amounts to smug self-congratulation does not inspire finishing. Rust deserves mention in the title only if actually getting it done in Rust was particularly difficult.
RcouF1uZ4gsC 4y ago
> Dev velocity, which was supposed to be the claim to fame of Python, improved dramatically with Rust. Built-in testing, CI/CD, benchmarking, and an overzealous compiler increased engineers’ confidence in pushing changes, and enabled them to work on the same code sections and contribute simultaneously without breaking the code base. Most impressively though, real time operational events dropped almost to zero overnight after the original release. I think Rust is really shaping up to be in a sweet spot. One thing that is not talked about enough with Rust, is that you do not need to start with fighting the borrow checker from day one. Making copies and using Rc can bypass a lot of the trickier parts of the borrow checker and still allow you to access much of the benefits of Rust in terms of performance and correctness, especially compared to a dynamic language like Python. As you get more experience, you can then use references and other techniques to eliminated unnecessary copies, etc.
- ncmncm 4y ago
  Using Rc unnecessarily is a code smell commonly called Java Disease.
- klabb3 4y ago
  > Making copies and using Rc can bypass a lot of the trickier parts of the borrow checker and still allow you to access much of the benefits of Rust in terms of performance and correctness, especially compared to a dynamic language like Python. > As you get more experience, you can then use references and other techniques to eliminated unnecessary copies, etc. I've always found this advice to be infeasible in practice, despite sounding like a reasonable pitch. Arc allows you to avoid thinking about ownership, which is largely what dictates the architecture when writing idiomatic Rust. Plastering on ownership later is going to change the code so much you might as well rewrite it . Moreover, if you want to prototype (with Arcs, clones) a new part of the code, while integrating with a more mature part of the code base, you still need to adhere to the ownership structure. In fact, this is one of my main issues with Rust, that it's a terrible language for prototyping. I don't have a solution, and I'm not blaming Rust – it's still a great language. But you have to have a very solid design/mental model before you start coding.
liminal 4y ago
Any plans to open source the kv store?
haskellandchill 4y ago
so how are graphs used for vector search?
- gk1 4y ago
  See: https://www.pinecone.io/learn/hnsw/ https://www.pinecone.io/learn/hnsw/ Although our graph-based vector index does not use HNSW, the concept is similar.
throwaway894345 4y ago
> Dev velocity, which was supposed to be the claim to fame of Python, improved dramatically with Rust. I don't doubt this in the least. I've been a professional Python developer for 15 years, and I can't believe Python ever had the reputation for "high dev velocity" beyond toy examples. In every real world code base I've worked in, Python has been a strict liability and the promise that you can "just rewrite the slow parts in C/C++/Numpy/etc" has never held (you very often spend at least as much time marshaling the data to the target language format than you gain by processing in the faster language and you have all of the maintainability problems of C/C++/Numpy). Python trades developer velocity for poor runtime performance. I don't think Rust is the paragon of high dev velocity languages either, but it seems to be more predictable. You don't have something that works as a prototype, but then you run into a bunch of performance problems (which virtually cannot be worked around) when you go to productionize, nor do you get all of the emergent quality and maintainability issues that come about from a dynamic type system (and last I checked, Python's optional static type system was still pretty alpha-grade). I strongly recommend avoiding Python for new projects. Use Go if you're looking for an easy GC language with max productivity and a decent performance ceiling. Use Rust if you're writing really high performance or correctness-is-paramount software. I'm sure there are other decent options as well (I've heard generally good things about TypeScript, but I'd be concerned about its performance even if it is better than Python).
- benreesman 4y ago
  I think Rust deserves a lot of credit for popularizing the notion of “constraints as power”, but excessive enthusiasm around it does seem to obscure the rich traditions of PLT that it has more than liberally borrowed from. There are more rungs on the ladder.
  - Kinrany 4y ago
    "Constraints as power" describes the proposition of all statically typed languages. But the compilers of popular languages weren't smart enough to cover many common scenarios, forcing language designers to offer many unsafe escape hatches.
kartoolOz 4y ago
Sidenote, been using pinecone for semantic search recently, and it's been a joyful experience.
nu11ptr 4y ago
This doesn't surprise me and matches my own experience. Rust literally makes a codebase nearly void of most bug classes with the exception of logic bugs (Unfortunately, in a huge codebase, there can still be tons and tons of logic bugs). Still, when I migrated my Python codebase to Rust I got rid of whole classes of bugs and honestly code faster in Rust on a "per debugged line of code" basis. In Python, every line MUST be run or could trigger an exception (and so many do and it is hard to see). This now requires a test harness that tests every single line which is a huge pain. In Rust, the things that can obviously panic are pretty easy to see and are limited (unwrap, expect, indexing, etc.) making it much easier to write robust code, even with a more limited test suite.
- brightball 4y ago
  That’s why Elixir developers like to make NIF callouts with Rust. It’s dangerous to do anything outside of the BEAM but Rust negates all of those concerns.
  - johnisgood 4y ago
    Is it possible to use Ada instead of Rust?
    - brightball 4y ago
      I believe you can made a NIF with just about anything.
      - johnisgood 4y ago
        I wonder if there are any examples though. Would be helpful. I suppose I could just make Ada bindings.
  - Xorlev 4y ago
    Maybe not all concerns, you're still allocating memory outside of BEAM and you still need to be careful to yield control on longer-running NIFs.
api 4y ago
We're rusting at the moment too. Encouraging. :)
t43562 4y ago
Writing something for the second time is a significantly different experience so that makes the conclusion much less definitive to me. Did they need python at all the first time - maybe not. Python "velocity" is, to me, more about the ease with which you can make changes without breaking other code. You can still break other code and if you don't have a proper test suite then you're wasting your time - which is something that smells slightly in the story.
- jnwatson 4y ago
  Yeah. Having runtime errors in a non-trivial Python code base is indicative of poor test coverage.
- obviouslynotme 4y ago
  This is my feeling as well. The biggest problem with new projects is that you don't know what the product will look like in advance, even if you think you do. Restructuring code, especially with full 20/20 hindsight, will make that code much better.
nazka 4y ago
I like this "and an overzealous compiler increased engineers’ confidence in pushing changes". If you think about it, you can try to optimize a Rust program through trying brute force stuff and trying weird things that you are not even sure will break something or not. But you can try ~without 100% knowledge or confidence~ and boom the compiler will tell you if you are shooting yourself in the foot. Now try to that in C++. Even with 95% knowledge and confidence I will be very worried as a Lead or CTO.
AtNightWeCode 4y ago
Dev velocity with new projects is often higher so I am not convinced by that argument. With that said. I learned Rust recently. I think it is a viable choice for a business core. Rust is not at all that difficult to learn as some claim. The tuts are great. Some things are very different though.
FpUser 4y ago
Frankly on larger projects overall time to create and test the whole thing in modern C++ or (I assume) in Rust should not take longer than in Python. I do not understand this idea of mixing languages where one is enough. In my own case I rewrote decent size Python app in C++ and it had taken me about the same time as for original developers. And no I was not porting code but rather working with the specs so did my own design.
t6jvcereio 4y ago
> We decided to move our entire codebase to Rust (and Go for the k8s control plane). > there was still one minor problem - no one on the team knew Rust. Is this real or satire? I can't tell.
- drexlspivey 4y ago
  How hard can it be for C/C++ devs to learn Rust? Python devs on the other hand..
  - t6jvcereio 4y ago
    That's not the point. If no one knows the language, no one knows if it's appropriate. All you know is the language marketing claims
    - tinco 4y ago
      Even though it's software engineering, which supposedly isn't really engineering, you can still apply engineering principles. Disregarding all the bullshit debates about whether Rust is better than Go, there are actually differences between the languages that you can objectively research. Maybe no one in that specific company really knew the language, but there's loads of people actually using the language, it's not like it dropped from the clear blue sky. It's not like there's a Rust salesperson that knocked on their door with a leaflet. They're not even the first persons to write a database engine in Rust. I wonder if other engineering domains have this same silliness. Do the architects have intense debates over steel H beams vs wooden LVL beams, and at some point newbie architects believe you should only switch from one to the other as an architecture firm if your architects have experience with it, because the only way to know if it would work is reading the marketing claims. A compiler is just a tool. If you're doing a rewrite you better pick your tools correctly, and it seems like they did. You could theorise they were lucky somehow, or you could recognise that they're experienced, educated and skilled and made the right choice based on their researched.
      - t6jvcereio 4y ago
jhgg 4y ago
Writing Rust at scale, I concur with some of these findings. Nowadays, my team exclusively writes Rust. Our rationale is two-fold. 1) We want to write software that is fast and inexpensive to run. Our rust services are quickly growing, but cumulatively, they remain in the sub-1000 core count, while processing many millions of requests per second across our service with excellent latency. 2) We want to write software that won't break and page us in the middle of the night, or blow up when we deploy it. We operate some of the core persistence primitives at our company, so the consequence for error in the worst case could be corruption or loss of user data. Our experience thusfar with Rust is that we're able to confidently iterate and deploy our services. The developer velocity is indeed higher, and continues to grow with your experience in the language. Our team of 10 (with 3 engineers having started in the last 2 weeks) owns all of the database clusters, and all of the services surrounding them. Our on-call is surprisingly quiet, with maybe 0-3 low sev incidents a week, with most of them being due to the underlying database/hardware, and not the rust services. We are actually writing a Rust control plane for our databases soon, which seeks to automate most of the response and remediation of most of the incidents we experience in a given week, and also to automate a bunch of manual work that we currently do around operations of the databases. The learning curve can be a bit rough, but our team of engineers is motivated to learn, and we have a great culture of mentorship and teaching from the engineers who are familiar with Rust.
- nazka 4y ago
  I am curious which web framework do you use? Also do you use an ORM like diesel?
  - faitswulff 4y ago
    You could probably get a pretty good idea from reading the Discord engineering blog.
    - nazka 4y ago
      Yep. I checked his bio only after commenting.
  - 4y ago
bfrog 4y ago
This very closely matches my own experience nearly 4 or 5 years ago now rewriting an ad server in rust from another scripty language.
kissgyorgy 4y ago
Expectation: this article. Reality: It's insanely hard to find talented developers, but Rust makes people excited, so they are just using it as a marketing tool.
theptip 4y ago
I'd be interested in a bit more detail on the technical migration strategy. Did they just cut over to a version of the project built in Rust? (I.e. vN is Python+C/C++, vN+1 is Rust?) Or something more gradual? How did they verify that the new code was conformant with the old app's behavior/logic? > To make matters worse, we would discover issues only after deploying (or in production!) due to Python’s run time nature. I don't want to infer too much, but this makes it sound like perhaps they didn't have a very robust set of E2E/Acceptance tests, which would make a full-cutover migration scary to me. If you're finding Python bugs only after deploy, how can you find the inevitable rewritten-Rust-code incompatibilities before deploying/production? I've been digging into Rust/Python interop recently using https://github.com/PyO3/pyo3 https://github.com/PyO3/pyo3 and maturin, and this points to an interesting migration strategy; PyO3 makes it quite easy to write a Python module in Rust, or even call back and forth between them, so you could gradually move code from Python to Rust. A migration path to full-Rust might be: 1. Incrementally replace all your fast-path C/C++ code with Rust, still having Python call these compiled modules. End-state: You now have a Python/Rust project instead of Python/C/C++. 2. Gradually grow the surface area of your Rust packages, moving logic from Python into Rust. Your existing Python tests can still run, and your existing entrypoints are the same Python. 3. At some point, you presumably need to cut over the entrypoint layer (the API?) to use pure Rust, instead of Python. This should be a much less scary migration since both versions are calling into the same underlying Rust library code. Depending on your architecture, if you have an API Gateway you can split your service backends to migrate one endpoint at a time to the new Rust API service, while keeping the old Python API service around to fail back to. (They are using k8s so you can do this with your Ingress for example). I'm interested in others' experiences with Rust/Python interop, are there any rough edges worth knowing about?
lmeyerov 4y ago
Something I still struggle to understand with these systems is why indexing 1-10M things is hard / their sweet spot, when that is basically a small dataframe in pandas or a small SQLite, vs the 10B+ rowa // GBs+ I was expecting them to be discussing
jmartin2683 4y ago
This is all directly in line with my experience having ported several production applications to Rust.
lucidguppy 4y ago
Python has built in testing and C++ has libraries like catch2. I would have liked some initial comment that said "we had a good suite of tests". Python has plenty of benchmarking tools available. IMHO - a db is all core. Everything should be fast. Even the CLI tools - Rust is pretty good at cli tools. Why is he building a database out of python? How much python is in this repo anyway? Also this is a Python + C++ app - not python. And the article critiques python, not c++. Which is weird, because I would have thought the source of problems would have been C++ not python. Don't get me wrong - Rust is great. Cargo is great too. Compiled programs are great to deploy rather than python (pip hell) and C++ ({CMAKE_HELL}). But this is apples to oranges comparison. The things that make Rust better than C++ / Python are not the things this article talks about.
linsomniac 4y ago
The steps some people to go to to avoid writing tests is shocking. (I say this as someone who has written Python programs and been tired of deploying them only to find a runtime bug, and solved it by writing tests)
shrubble 4y ago
Has anyone ever compared Rust with Modula-2/3 or Oberon family of languages? I would think it would be an interesting comparison.
deleted 4y ago
[deleted]
whoomp12342 4y ago
;lkj
4m1rk 4y ago
> we would discover issues only after deploying (or in production!) due to Python’s run time nature I love Rust but seems like they just needed to write some test before shipping to prod? > Built-in test, CI/CD Does it have anything to do with the language?

474 points by muizelaar 4y ago

fb03 4y ago

Would you guys start a GraphQL Api today using Rust? I'm very tempted to hint at that but am still insecure about my 'options'. This is coming from a very comfortable python + fastapi + sqlalchemy ecossystem

jraph 4y ago

The title is "Inside the Pinecone". "Rust – A hard decision pays off" is only one of the three major subsections of this article.

gk1 4y ago
Interesting to see the part about Rust getting the most attention. We're giving a talk about it in NYC next week[1] and will definitely follow up with an in-depth writeup. [1] https://www.meetup.com/rust-nyc/events/287821884/ https://www.meetup.com/rust-nyc/events/287821884/
- dfinninger 4y ago
  The submitted link auto-scrolls me to the section on Rust, so I guess the submitter wanted to promote that specific section of the article?
3a2d29 4y ago
Gotta mention rust, how else will you get upvotes on HN?
eterm 4y ago
The title of the section posted is "Rust - A hard decision...". It's been posted here with the anchor included making it clear that it is that section that has been submitted.
- jraph 4y ago
  Indeed, I didn't notice. Sorry!

amelius 4y ago

Nice to see that Rust is working for your particular usecase!

benreesman 4y ago

So I was one of the 3 hackers who wrote the LSM that predated Rocks at FB, and as a result I distinctly remember when Rocks overtook it in features by trading off performance, and again when it met and exceeded the performance while keeping the greater capability. RocksDB kicks ass and has for a long time, I’d recommend it to anyone in the market for that sort of thing. FAISS is also great, and has been for years. I don’t doubt that someone has done a meaningful rev on it. But as someone who likes Rust and wants to like it more: don’t lead with that. The software speaks for itself. Or should.

bluejekyll 4y ago
> But as someone who likes Rust and wants to like it more: don’t lead with that. The original title appears to be “Inside the Pinecone”, but seeing as they submitted it themselves I’m guessing they wanted the uptick on the Rust. The article itself doesn’t discuss their transition to Rust until the end. So I wouldn’t say they led with it originally, but perhaps didn’t get traction on the original title.
- gk1 4y ago
  > seeing as they submitted it themselves I'm grateful to muizelaar but I don't know who that is, so we can't take credit for this one.
  - bluejekyll 4y ago
    Oh, sorry. I misattributed that. I saw the domain origin and confused that for the submitter. In that case, the submission actually is against the rules of HN, as I understand them (use the original title rule).
- benreesman 4y ago
  I guess I’m rooting for Rust in the long game sense, which is a different set of imperatives than the hang the hashtag on peripheral stuff sense. I use a lot of great software written in Rust, it’s demonstrably a good vehicle for great software. But too many of its fans are advocates, this can start to seem like an agenda. Don’t take engineering advice from people with an agenda.

endisneigh 4y ago

Maybe it’s just me but it seems irresponsible to move your entire team to a programming language none of them knows. Rust is great, but modern C++ is great too.

moffkalast 4y ago
> modern C++ is great Great in the way that Alexander the Great was great. Achieved big things and killed a lot of people. Though in the case of C++ they're just dead inside.
- benreesman 4y ago
  There are better threads to trash C++ hackers on than this one. It’s literally about insanely successful software written in C++, maintained successfully for a decade, that seems to have survived a port to another language.
- FpUser 4y ago
  >"Though in the case of C++ they're just dead inside." I think this is pure and memory safe BS
cratermoon 4y ago
> it seems irresponsible to move your entire team to a programming language none of them knows Based on my experiences, I would agree. There seems to be a very important difference between my experiences, which probably align with most of the industry, and the Pinecone team. My career has been working with average developers doing ordinary business and internet things. Stuff that takes organization and teamwork, but not necessarily advanced or esoteric knowledge. So the COBOL programmers that went to the "Java in 21 Days" training that I consulted with, or the C# programmers who went with Go, they did poor-to-OK, but that was Good Enough. The Pinecone team, though. They were already writing C/C++ to solve Hard Problems. As experienced programmers, they probably already understood the landscape of systems languages with pointers and having to manage their own memory. Rust, while imposing, would not be a huge leap for them. Another aspect worth noting: they did they re-write incrementally, never completely abandoning the existing system. At least it seems so. I'll be interested in seeing the presentation from the upcoming talk about the rewrite that's mentioned in the story.

0xbeefeed 4y ago

[flagged]

throwaway642012 4y ago

I don't want to be dismissive but how much of this success belongs to the team of rock solid senior engineers with enough experience under their belt rather than rewrite in Rust?

hawk_ 4y ago
I would let the same team of "rock solid senior engineers" be the judge of that. They seem to think it mattered significantly enough.
- amelius 4y ago
  Or ... since they are now betting fully on Rust, they now want everybody else on Rust too. Because more people == a higher probability that if they run into issues with Rust someone will fix it.
  - farresito 4y ago
    That's a really cynical take on the matter...
robertlagrant 4y ago
The writer seems to give massive credit to the team.
runevault 4y ago
You can give some credit to the fact they were writing it a second time certainly, but as far as we know at least a lot of the people were the same ones who were central to writing V1. So certainly there was some lift from already doing it once, but if they went from a lot of subtle bugs they struggled to track down to having far fewer of them, and those bugs tended to be the types of issue the rust compiler and/or linter normally point out to you, it will certainly make it easier to mitigate the issues. Rust cannot make a bad programmer a good programmer, but the fact it points out several classes of mistakes (sometimes too aggressively, hopefully someday we get Polonia and other improvements to the borrow checker to remove some things that are hard only because of the current implementation of various checking systems) allows far more confidence. I don't care how good someone is, having an automated way to validate large scales of issues aren't there is a big deal, and unlike unit/integration tests you don't have to write these.

mhaberl 4y ago

> ..complex runtime issues which were almost impossible to reproduce or isolate > That’s when internal murmurs about a complete rewrite started brewing… > We decided to move our entire codebase to Rust > there was still one minor problem - no one on the team knew Rust "We have runtime issues so we will fix them by a complete rewrite in a language that no one on the team knows." I would not call that a "hard decision", rather "rolling the dice". It awesome that they managed to make it work.

moonchrome 4y ago
I mean when the tools you know are failing you, you might as well roll the dice.
- draw_down 4y ago
- remram 4y ago
  They could have had some team members learn Rust before deciding that Rust was the right language for the re-write. It feels really risky to do such an important decision only on a high-level view of what you think the language will provide.
bluejekyll 4y ago
I think of rolling the dice as taking a chance without any knowledge of the chance for it paying of. Rust fit the need they had pretty well (on paper anyway since they didn’t have experience with it), and the industry is showing it to be a good choice for solving these types of problems. That’s not rolling the dice, it’s taking a chance based on research.
- meragrin_ 4y ago
  > I think of rolling the dice as taking a chance without any knowledge of the chance for it paying of. I think hoping a Rust rewrite would fix "complex runtime issues which were almost impossible to reproduce or isolate" as rolling the dice. If you cannot reproduce or isolate an issue, how can you know a rewrite in another language will fix the issue?

ramoz 4y ago

> Simultaneous fetches of thousands (sometimes tens of thousands) of objects started becoming inefficient, especially when fetching from collections of tens of millions of objects Why not try leveldb. We're doing random reads of 170,000 vectors (/130M) a second. No startup needed.

pas 4y ago
Isn't RocksDB a fork (reimplementation) of LevelDB? :o
benreesman 4y ago
RocksDB started as a fork of LevelDB (Dean/Ghemawat). The performance imperatives have changed with the hardware, but NAND flash at that time had an asymmetry between reads and writes in terms of the amount of data: once you were writing even one byte you had effectively paid for writing a whole block and were therefore incentivized to get your money’s worth and write to a “log”, which then would be “merged”, in some “structured” way, hence LSM. This is old news these days but it was quite the novelty at the time!

ncmncm 4y ago

Leading with several pages of what amounts to smug self-congratulation does not inspire finishing. Rust deserves mention in the title only if actually getting it done in Rust was particularly difficult.

RcouF1uZ4gsC 4y ago

> Dev velocity, which was supposed to be the claim to fame of Python, improved dramatically with Rust. Built-in testing, CI/CD, benchmarking, and an overzealous compiler increased engineers’ confidence in pushing changes, and enabled them to work on the same code sections and contribute simultaneously without breaking the code base. Most impressively though, real time operational events dropped almost to zero overnight after the original release. I think Rust is really shaping up to be in a sweet spot. One thing that is not talked about enough with Rust, is that you do not need to start with fighting the borrow checker from day one. Making copies and using Rc can bypass a lot of the trickier parts of the borrow checker and still allow you to access much of the benefits of Rust in terms of performance and correctness, especially compared to a dynamic language like Python. As you get more experience, you can then use references and other techniques to eliminated unnecessary copies, etc.

ncmncm 4y ago
Using Rc unnecessarily is a code smell commonly called Java Disease.
klabb3 4y ago
> Making copies and using Rc can bypass a lot of the trickier parts of the borrow checker and still allow you to access much of the benefits of Rust in terms of performance and correctness, especially compared to a dynamic language like Python. > As you get more experience, you can then use references and other techniques to eliminated unnecessary copies, etc. I've always found this advice to be infeasible in practice, despite sounding like a reasonable pitch. Arc allows you to avoid thinking about ownership, which is largely what dictates the architecture when writing idiomatic Rust. Plastering on ownership later is going to change the code so much you might as well rewrite it . Moreover, if you want to prototype (with Arcs, clones) a new part of the code, while integrating with a more mature part of the code base, you still need to adhere to the ownership structure. In fact, this is one of my main issues with Rust, that it's a terrible language for prototyping. I don't have a solution, and I'm not blaming Rust – it's still a great language. But you have to have a very solid design/mental model before you start coding.

liminal 4y ago

Any plans to open source the kv store?

haskellandchill 4y ago

so how are graphs used for vector search?

gk1 4y ago
See: https://www.pinecone.io/learn/hnsw/ https://www.pinecone.io/learn/hnsw/ Although our graph-based vector index does not use HNSW, the concept is similar.

throwaway894345 4y ago

> Dev velocity, which was supposed to be the claim to fame of Python, improved dramatically with Rust. I don't doubt this in the least. I've been a professional Python developer for 15 years, and I can't believe Python ever had the reputation for "high dev velocity" beyond toy examples. In every real world code base I've worked in, Python has been a strict liability and the promise that you can "just rewrite the slow parts in C/C++/Numpy/etc" has never held (you very often spend at least as much time marshaling the data to the target language format than you gain by processing in the faster language and you have all of the maintainability problems of C/C++/Numpy). Python trades developer velocity for poor runtime performance. I don't think Rust is the paragon of high dev velocity languages either, but it seems to be more predictable. You don't have something that works as a prototype, but then you run into a bunch of performance problems (which virtually cannot be worked around) when you go to productionize, nor do you get all of the emergent quality and maintainability issues that come about from a dynamic type system (and last I checked, Python's optional static type system was still pretty alpha-grade). I strongly recommend avoiding Python for new projects. Use Go if you're looking for an easy GC language with max productivity and a decent performance ceiling. Use Rust if you're writing really high performance or correctness-is-paramount software. I'm sure there are other decent options as well (I've heard generally good things about TypeScript, but I'd be concerned about its performance even if it is better than Python).

benreesman 4y ago
I think Rust deserves a lot of credit for popularizing the notion of “constraints as power”, but excessive enthusiasm around it does seem to obscure the rich traditions of PLT that it has more than liberally borrowed from. There are more rungs on the ladder.
- Kinrany 4y ago
  "Constraints as power" describes the proposition of all statically typed languages. But the compilers of popular languages weren't smart enough to cover many common scenarios, forcing language designers to offer many unsafe escape hatches.

kartoolOz 4y ago

Sidenote, been using pinecone for semantic search recently, and it's been a joyful experience.

nu11ptr 4y ago

This doesn't surprise me and matches my own experience. Rust literally makes a codebase nearly void of most bug classes with the exception of logic bugs (Unfortunately, in a huge codebase, there can still be tons and tons of logic bugs). Still, when I migrated my Python codebase to Rust I got rid of whole classes of bugs and honestly code faster in Rust on a "per debugged line of code" basis. In Python, every line MUST be run or could trigger an exception (and so many do and it is hard to see). This now requires a test harness that tests every single line which is a huge pain. In Rust, the things that can obviously panic are pretty easy to see and are limited (unwrap, expect, indexing, etc.) making it much easier to write robust code, even with a more limited test suite.

brightball 4y ago
That’s why Elixir developers like to make NIF callouts with Rust. It’s dangerous to do anything outside of the BEAM but Rust negates all of those concerns.
- johnisgood 4y ago
  Is it possible to use Ada instead of Rust?
  - brightball 4y ago
    I believe you can made a NIF with just about anything.
    - johnisgood 4y ago
      I wonder if there are any examples though. Would be helpful. I suppose I could just make Ada bindings.
- Xorlev 4y ago
  Maybe not all concerns, you're still allocating memory outside of BEAM and you still need to be careful to yield control on longer-running NIFs.

api 4y ago

We're rusting at the moment too. Encouraging. :)

t43562 4y ago

Writing something for the second time is a significantly different experience so that makes the conclusion much less definitive to me. Did they need python at all the first time - maybe not. Python "velocity" is, to me, more about the ease with which you can make changes without breaking other code. You can still break other code and if you don't have a proper test suite then you're wasting your time - which is something that smells slightly in the story.

jnwatson 4y ago
Yeah. Having runtime errors in a non-trivial Python code base is indicative of poor test coverage.
obviouslynotme 4y ago
This is my feeling as well. The biggest problem with new projects is that you don't know what the product will look like in advance, even if you think you do. Restructuring code, especially with full 20/20 hindsight, will make that code much better.

nazka 4y ago

I like this "and an overzealous compiler increased engineers’ confidence in pushing changes". If you think about it, you can try to optimize a Rust program through trying brute force stuff and trying weird things that you are not even sure will break something or not. But you can try ~without 100% knowledge or confidence~ and boom the compiler will tell you if you are shooting yourself in the foot. Now try to that in C++. Even with 95% knowledge and confidence I will be very worried as a Lead or CTO.

AtNightWeCode 4y ago

Dev velocity with new projects is often higher so I am not convinced by that argument. With that said. I learned Rust recently. I think it is a viable choice for a business core. Rust is not at all that difficult to learn as some claim. The tuts are great. Some things are very different though.

FpUser 4y ago

Frankly on larger projects overall time to create and test the whole thing in modern C++ or (I assume) in Rust should not take longer than in Python. I do not understand this idea of mixing languages where one is enough. In my own case I rewrote decent size Python app in C++ and it had taken me about the same time as for original developers. And no I was not porting code but rather working with the specs so did my own design.

t6jvcereio 4y ago

> We decided to move our entire codebase to Rust (and Go for the k8s control plane). > there was still one minor problem - no one on the team knew Rust. Is this real or satire? I can't tell.

drexlspivey 4y ago
How hard can it be for C/C++ devs to learn Rust? Python devs on the other hand..
- t6jvcereio 4y ago
  That's not the point. If no one knows the language, no one knows if it's appropriate. All you know is the language marketing claims
  - tinco 4y ago
    Even though it's software engineering, which supposedly isn't really engineering, you can still apply engineering principles. Disregarding all the bullshit debates about whether Rust is better than Go, there are actually differences between the languages that you can objectively research. Maybe no one in that specific company really knew the language, but there's loads of people actually using the language, it's not like it dropped from the clear blue sky. It's not like there's a Rust salesperson that knocked on their door with a leaflet. They're not even the first persons to write a database engine in Rust. I wonder if other engineering domains have this same silliness. Do the architects have intense debates over steel H beams vs wooden LVL beams, and at some point newbie architects believe you should only switch from one to the other as an architecture firm if your architects have experience with it, because the only way to know if it would work is reading the marketing claims. A compiler is just a tool. If you're doing a rewrite you better pick your tools correctly, and it seems like they did. You could theorise they were lucky somehow, or you could recognise that they're experienced, educated and skilled and made the right choice based on their researched.
    - t6jvcereio 4y ago

jhgg 4y ago

Writing Rust at scale, I concur with some of these findings. Nowadays, my team exclusively writes Rust. Our rationale is two-fold. 1) We want to write software that is fast and inexpensive to run. Our rust services are quickly growing, but cumulatively, they remain in the sub-1000 core count, while processing many millions of requests per second across our service with excellent latency. 2) We want to write software that won't break and page us in the middle of the night, or blow up when we deploy it. We operate some of the core persistence primitives at our company, so the consequence for error in the worst case could be corruption or loss of user data. Our experience thusfar with Rust is that we're able to confidently iterate and deploy our services. The developer velocity is indeed higher, and continues to grow with your experience in the language. Our team of 10 (with 3 engineers having started in the last 2 weeks) owns all of the database clusters, and all of the services surrounding them. Our on-call is surprisingly quiet, with maybe 0-3 low sev incidents a week, with most of them being due to the underlying database/hardware, and not the rust services. We are actually writing a Rust control plane for our databases soon, which seeks to automate most of the response and remediation of most of the incidents we experience in a given week, and also to automate a bunch of manual work that we currently do around operations of the databases. The learning curve can be a bit rough, but our team of engineers is motivated to learn, and we have a great culture of mentorship and teaching from the engineers who are familiar with Rust.

nazka 4y ago
I am curious which web framework do you use? Also do you use an ORM like diesel?
- faitswulff 4y ago
  You could probably get a pretty good idea from reading the Discord engineering blog.
  - nazka 4y ago
    Yep. I checked his bio only after commenting.
- 4y ago

bfrog 4y ago

This very closely matches my own experience nearly 4 or 5 years ago now rewriting an ad server in rust from another scripty language.

kissgyorgy 4y ago

Expectation: this article. Reality: It's insanely hard to find talented developers, but Rust makes people excited, so they are just using it as a marketing tool.

theptip 4y ago

I'd be interested in a bit more detail on the technical migration strategy. Did they just cut over to a version of the project built in Rust? (I.e. vN is Python+C/C++, vN+1 is Rust?) Or something more gradual? How did they verify that the new code was conformant with the old app's behavior/logic? > To make matters worse, we would discover issues only after deploying (or in production!) due to Python’s run time nature. I don't want to infer too much, but this makes it sound like perhaps they didn't have a very robust set of E2E/Acceptance tests, which would make a full-cutover migration scary to me. If you're finding Python bugs only after deploy, how can you find the inevitable rewritten-Rust-code incompatibilities before deploying/production? I've been digging into Rust/Python interop recently using https://github.com/PyO3/pyo3 https://github.com/PyO3/pyo3 and maturin, and this points to an interesting migration strategy; PyO3 makes it quite easy to write a Python module in Rust, or even call back and forth between them, so you could gradually move code from Python to Rust. A migration path to full-Rust might be: 1. Incrementally replace all your fast-path C/C++ code with Rust, still having Python call these compiled modules. End-state: You now have a Python/Rust project instead of Python/C/C++. 2. Gradually grow the surface area of your Rust packages, moving logic from Python into Rust. Your existing Python tests can still run, and your existing entrypoints are the same Python. 3. At some point, you presumably need to cut over the entrypoint layer (the API?) to use pure Rust, instead of Python. This should be a much less scary migration since both versions are calling into the same underlying Rust library code. Depending on your architecture, if you have an API Gateway you can split your service backends to migrate one endpoint at a time to the new Rust API service, while keeping the old Python API service around to fail back to. (They are using k8s so you can do this with your Ingress for example). I'm interested in others' experiences with Rust/Python interop, are there any rough edges worth knowing about?

lmeyerov 4y ago

Something I still struggle to understand with these systems is why indexing 1-10M things is hard / their sweet spot, when that is basically a small dataframe in pandas or a small SQLite, vs the 10B+ rowa // GBs+ I was expecting them to be discussing

jmartin2683 4y ago

This is all directly in line with my experience having ported several production applications to Rust.

lucidguppy 4y ago

Python has built in testing and C++ has libraries like catch2. I would have liked some initial comment that said "we had a good suite of tests". Python has plenty of benchmarking tools available. IMHO - a db is all core. Everything should be fast. Even the CLI tools - Rust is pretty good at cli tools. Why is he building a database out of python? How much python is in this repo anyway? Also this is a Python + C++ app - not python. And the article critiques python, not c++. Which is weird, because I would have thought the source of problems would have been C++ not python. Don't get me wrong - Rust is great. Cargo is great too. Compiled programs are great to deploy rather than python (pip hell) and C++ ({CMAKE_HELL}). But this is apples to oranges comparison. The things that make Rust better than C++ / Python are not the things this article talks about.

linsomniac 4y ago

The steps some people to go to to avoid writing tests is shocking. (I say this as someone who has written Python programs and been tired of deploying them only to find a runtime bug, and solved it by writing tests)

shrubble 4y ago

Has anyone ever compared Rust with Modula-2/3 or Oberon family of languages? I would think it would be an interesting comparison.

deleted 4y ago

[deleted]

whoomp12342 4y ago

;lkj

4m1rk 4y ago

> we would discover issues only after deploying (or in production!) due to Python’s run time nature I love Rust but seems like they just needed to write some test before shipping to prod? > Built-in test, CI/CD Does it have anything to do with the language?