4 ms·

We did a similar thing with a Scala -> Rust rewrite for the http://prisma.io http://prisma.io query engine. By rewriting small components and integrating them

We did a similar thing with a Scala -> Rust rewrite for the http://prisma.io http://prisma.io query engine. By rewriting small components and integrating them into the existing project using Javas native interface, our small team of 5 developers were able to pull off this massive rewrite in just under a year. The resulting code base is rearchitected in a few very important ways, but mostly follows the same structure. And because we kept and evolved our old Scala based test suite, we have a very high confidence in the rewrite. When Async/.await finally landed, we could switch over very quickly, and it has been a joy to focus on benchmarks and performance over the last month. Spoiler: Rust is faster than Scala :-D

sorenbs 7y ago
I should add that the Rust community has been extraordinarily welcoming, and our existing Scala engineers were able to relatively quickly become proficient in Rust. Huge shoutout to everyone working on Rust Async, especially the Berlin crew, who has been very helpful.
tombert 7y ago
I promise that this is asked genuinely and isn't some sort of veiled "gotcha!" (it's tough to tell on the internet sometimes); what was the reason for a change from Scala to Rust? I ask because Scala already has a good type system and the JVM typically has good performance nowadays, particularly with something like GraalVM, so I am actually really curious to why you felt a Rust rewrite was a good idea.
- gameswithgo 7y ago
  Just some reason I might make a switch from Java/C# to Rust: * you can keep memory use quite a bit lower * you can still sometimes get large constant factors of performance improvements over the JVM in some kinds of problem domains. If this means you can run on 1 server instead of say, 5, you have a much simpler infrastructure. * startup time, especially if you are doing 'serverless' or similar * tail latency - even a good GC language will have occasional long pauses. * data race protection at compile time * easier deployment - no need to install a jvm and keep it updated
  - tombert 7y ago
    I agree with all of your points except the "startup time" and "easier deployment"; GraalVM is pretty sweet and produces nice, self-contained executables.
    - pimeys 7y ago
      Prisma engineer here who's been part of the rewrite since the beginning. We tried GraalVM, but the binary size was huge and we anyways needed to write parts, such as the JDBC drivers in Rust, C or C++ due to GraalVM not being able to compile certain JVM code to a native binary. We distribute the binary with our NodeJS package and hundreds of megabytes of binary size will not work that well for our users.
      - tombert 7y ago
        Oh! That's a valid reason; I'm actually surprised that GraalVM has issues not supporting JVM features, since I've been using it for Clojure code locally (though admittedly just for fun, nothing serious). Definitely makes sense if you're stuck rewriting things anyway, might as well just do it in a more modern language. Thanks for the insight!
        pimeys 7y ago
        Yep. I remember how at least JDBC and JWT libs were needed to rewrite using a native language (which we did until some point!). And the GraalVM has a weird API that is not JNA/JNI when you want to use the native-image. It is not very robust system for our needs, but it was a good learning experience. What we did in the end is we rewrote parts of the system with Rust, plugged that to the JVM package and we had all our tests ready to use. First the database connectors and at the same time the other part of the team was writing the graphql parsing in Rust. We could all utilize our Scala integration tests which was crucial for our success. And btw. we still have our tests in JVM, although the rest of the stack is Rust now.
      - deepsun 7y ago
        That's interesting, would love to read more (blog post maybe?) on issues with GraalVM.
michael_j_ward 7y ago
Do you have any write-ups on this? I might be looking to do this for a large java codebase soon.
Scarbutt 7y ago
How many lines of code? (old/new)
- pimeys 7y ago
  The Rust code is here: https://github.com/prisma/prisma-engine/ https://github.com/prisma/prisma-engine/ And the old Scala codebase here: https://github.com/prisma/prisma/ https://github.com/prisma/prisma/ The old codebase has parts in Rust, so counting the lines is not that straightforward.
yann_s 7y ago
It'd be great to hear from your experience about using Javas native interface. AFAIK, using native call can have a performance impact: https://en.wikipedia.org/wiki/Java_Native_Interface#Performance https://en.wikipedia.org/wiki/Java_Native_Interface#Performa... I guess the plan what to make native is pretty important. Maybe you want to disclose more details about that?

sorenbs 7y ago

I should add that the Rust community has been extraordinarily welcoming, and our existing Scala engineers were able to relatively quickly become proficient in Rust. Huge shoutout to everyone working on Rust Async, especially the Berlin crew, who has been very helpful.

tombert 7y ago

I promise that this is asked genuinely and isn't some sort of veiled "gotcha!" (it's tough to tell on the internet sometimes); what was the reason for a change from Scala to Rust? I ask because Scala already has a good type system and the JVM typically has good performance nowadays, particularly with something like GraalVM, so I am actually really curious to why you felt a Rust rewrite was a good idea.

gameswithgo 7y ago
Just some reason I might make a switch from Java/C# to Rust: * you can keep memory use quite a bit lower * you can still sometimes get large constant factors of performance improvements over the JVM in some kinds of problem domains. If this means you can run on 1 server instead of say, 5, you have a much simpler infrastructure. * startup time, especially if you are doing 'serverless' or similar * tail latency - even a good GC language will have occasional long pauses. * data race protection at compile time * easier deployment - no need to install a jvm and keep it updated
- tombert 7y ago
  I agree with all of your points except the "startup time" and "easier deployment"; GraalVM is pretty sweet and produces nice, self-contained executables.
  - pimeys 7y ago
    Prisma engineer here who's been part of the rewrite since the beginning. We tried GraalVM, but the binary size was huge and we anyways needed to write parts, such as the JDBC drivers in Rust, C or C++ due to GraalVM not being able to compile certain JVM code to a native binary. We distribute the binary with our NodeJS package and hundreds of megabytes of binary size will not work that well for our users.
    - tombert 7y ago
      Oh! That's a valid reason; I'm actually surprised that GraalVM has issues not supporting JVM features, since I've been using it for Clojure code locally (though admittedly just for fun, nothing serious). Definitely makes sense if you're stuck rewriting things anyway, might as well just do it in a more modern language. Thanks for the insight!
      - pimeys 7y ago
        Yep. I remember how at least JDBC and JWT libs were needed to rewrite using a native language (which we did until some point!). And the GraalVM has a weird API that is not JNA/JNI when you want to use the native-image. It is not very robust system for our needs, but it was a good learning experience. What we did in the end is we rewrote parts of the system with Rust, plugged that to the JVM package and we had all our tests ready to use. First the database connectors and at the same time the other part of the team was writing the graphql parsing in Rust. We could all utilize our Scala integration tests which was crucial for our success. And btw. we still have our tests in JVM, although the rest of the stack is Rust now.
    - deepsun 7y ago
      That's interesting, would love to read more (blog post maybe?) on issues with GraalVM.

michael_j_ward 7y ago

Do you have any write-ups on this? I might be looking to do this for a large java codebase soon.

Scarbutt 7y ago

How many lines of code? (old/new)

pimeys 7y ago
The Rust code is here: https://github.com/prisma/prisma-engine/ https://github.com/prisma/prisma-engine/ And the old Scala codebase here: https://github.com/prisma/prisma/ https://github.com/prisma/prisma/ The old codebase has parts in Rust, so counting the lines is not that straightforward.

yann_s 7y ago

It'd be great to hear from your experience about using Javas native interface. AFAIK, using native call can have a performance impact: https://en.wikipedia.org/wiki/Java_Native_Interface#Performance https://en.wikipedia.org/wiki/Java_Native_Interface#Performa... I guess the plan what to make native is pretty important. Maybe you want to disclose more details about that?