29 ms·

We've been using Rust in production for Skylight (https://www.skylight.io https://www.skylight.io) for many months now, and we've been very happy with it. Bein

by tomdale 12y ago

We've been using Rust in production for Skylight (https://www.skylight.io https://www.skylight.io) for many months now, and we've been very happy with it. Being one of the first to deploy a new programming language into production is scary, and keeping up with the rapid changes was painful at times, but I'm extremely impressed with the Rust team's dedication to simplifying the language. It's much easier to pick up today than it was 6 months ago. The biggest win for us is how low-resource the compiled binaries are. Skylight relies on running an agent that collects performance information from our customers' Rails apps (à la New Relic, if you're more familiar with that). Previously, we wrote the agent in Ruby, because it was interacting with Ruby code and we were familiar with the language. However, Ruby's memory and CPU performance are not great, especially in long-running processes like ours. What's awesome about Rust is that it combines low-level performance with high-level memory safety. We end up being able to do many more stack allocations, with less memory fragmentation and more predictable performance, while never having to worry about segfaults. Put succinctly, we get the memory safety of a GCed language with the performance of a low-level language like C. Given that we need to run inside other people's processes, the combination of these guarantees is extremely powerful. Because Rust is so low-level, and makes guarantees about how memory is laid out (unlike e.g. Go), we can build Rust code that interacts with Ruby using the Ruby C API. I'm excited to see Rust continue to improve. Its combination of high-level expressiveness with low-level control is unique, and for many use cases where you'd previously use C or C++, I think Rust is a compelling alternative.

alelefant 12y ago
I've never built a Rails app, so perhaps there's something simple that I'm missing (I'm mostly a C# guy), but how would I use Skylight to monitor my on-premise application? The pricing makes it seems like a hosted service. I would have expected some sort of profiler to be needed on the server and then perhaps a centralized location for the data to be pushed to for display. The design of the site is great and I'm mostly curious because it sounds like something I wish was available in the .NET world!
- tomdale 12y ago
  Thanks for asking. If you're interested in the nitty-gritty, Yehuda and I gave a talk on the architecture behind Skylight at RailsConf: http://www.confreaks.com/videos/3394-railsconf-how-to-build-a-smart-profiler-for-rails http://www.confreaks.com/videos/3394-railsconf-how-to-build-... The short version is that the agent runs on your servers and collects information from your Rails app using the ActiveSupport::Notifications instrumentation built in to the framework. We serialize that into a protobuf that's transmitted via IPC to a background daemon (written in Rust). That daemon batches multiple reports into a single payload that is sent to the Skylight servers, where we use Storm and Cassandra to process the requests, and periodically do aggregate roll-ups. Unlike New Relic, Skylight gives you access to the entire distribution of response times, not just the average response time. (According to DHH, averages are "useless.")[1] This ends up being a lot of data, and a lot of CPU-intensive processing, which is why we sell it as a hosted service. [1]: https://signalvnoise.com/posts/1836-the-problem-with-averages https://signalvnoise.com/posts/1836-the-problem-with-average...
  - yazaddaruvala 12y ago
    Just an FYI, while I think, to set up skylight I just add a gem to my project, I'm still not really sure. It would have been nice to go to your homepage and have it confirm that it is easy to configure/deploy into your application.
  - grumbles 12y ago
    Minor point, but New Relic has had built-in support for collecting response time distributions for over a year. Please don't lie about this. https://docs.newrelic.com/docs/apm/applications-menu/features/histograms-viewing-data-distribution https://docs.newrelic.com/docs/apm/applications-menu/feature...
  - creatio 12y ago
    Is it possible to get the names of the Phd thesis read?
- deleted 12y ago
  [deleted]
hit8run 12y ago
Hey Tom, did you also consider googles golang? Currently I'm building my backends with golang and I am pretty satisfied with the performance and the whole flow. Maybe it is sometimes a bit cumbersome to check on errors like a paranoid but in the long term it helps to predict what happens in error cases. How about rust? Is the workflow comparable to golang? Is there an equivalent to gofmt and some ide support for code completion? Thx and regards Bijan
- dbaupp 12y ago
  I don't speak for tomdale/wycats, but embedding a language with a GC (Go) inside another GC'd language (Ruby) only leads to worlds of pain, especially when trying to efficiently transfer data between them. In languages without compulsory GCs (like Rust, which doesn't have one at all) this works because it's easy to have complete control about when memory is freed, but a GC'd language may free memory that was passed back into Ruby since the GC can no longer find any references to it. I'm sure there's ways around this (e.g. using unsafe/raw pointers), but this control is the default in Rust, and there are a pile of language mechanisms (ownership, in particular) that make making this safe much easier. (Also, AIUI, Go's FFI story is relatively inefficient, and, has some rather bad bugs; apparently https://code.google.com/p/go/issues/detail?id=7978 https://code.google.com/p/go/issues/detail?id=7978 is the Go GC trying to free FFI memory.)
  - Xorlev 12y ago
    From what I understood, it's a Rust agent communicating protobuf structs over IPC, not embedded into the gem. In such a case, memory safety and GC is less of an issue.
    - bjz_ 12y ago
      Memory safety is still good for long term code maintenance and refactorability. wycats also mentions that these processes are reasonably long lived - memory leaks can become a big problem over time if you are not careful. These can be mostly purged by static analysis and Valgrind, but are still a headache to deal with. I would also add that Rust is much easier to hack on safely if you don't have a great deal of prior systems programming experience.
Xorlev 12y ago
Love that you guys are using Rust in production. Very elegant design (AS::Notification IPC -> Rust). I'd dump the part of the marketing site talking about the fast UI on Ember. It's cool that you guys use Ember and you're proud of it, but I want to see more about your product. Show me how easy it is to drop into an app, or reports you generate, alerts you flag, etc.. Ultimately, while I nerd out on what a product is built with, I only care about what it can offer me. Fast UI isn't a feature, it's an expectation.
aaronbrethorst 12y ago
Somewhat OT, but... I'm looking at your pricing, and I don't have any idea how you define the notion of a "request". Right now, I can't even begin to guess what pricing bucket my apps might fit into, and that reduces my motivation to try out your service. I speculate I'm not the only person who's felt this way. (also, I know why you include the Ember mention at the bottom, but I really don't care. I'd rather see your awesome UI than hear what technology it's based on.) Edit: Seriously, I'm getting downvoted for offering someone feedback on their product experience?
- annnnd 12y ago
  Don't let it get to you - upvote and downvote are very close for mobile users. Here you have a +1 to compensate. :)
chrisseaton 12y ago
"We end up being able to do many more stack allocations, with less memory fragmentation and more predictable performance, while never having to worry about segfaults." But isn't this stuff the job of the VM? There's no reason why a program written in Ruby can't do the same stack allocations, reduced memory fragmentation and predictable performance automatically - if the Ruby VM was better designed. If Ruby had a better VM would you chose to use it over Rust? In Rust are you doing things that the VM could be doing for you?
- dbaupp 12y ago
  It is very hard to have all this done reliably, especially in a rather dynamic language like Ruby. Maybe the Ruby VM could be better designed to perform these sort of optimisations, but it will never give the same control as a language like Rust, C or C++; i.e. a small code change could cause the optimiser to fail, leading to slow-down and bloat. Furthermore, the hypothetical "sufficiently smart VM" isn't much value for code written now.
  - vidarh 12y ago
    [Chris has done some fantastic work on a Truffle / Graal backend for jruby; for my part I'm (slowly) working on a "mostly-ahead-of-time" Ruby compiler] I'm not sure I'd agree with "never", though I do agree Ruby is a hard language to optimize. There are two challenges with optimizing Ruby: What people do and don't know the potential cost of, and what people pretty much never do, but that the compiler / VM must be prepared for. The former includes things like "accidentally" doing things that triggers lots of copying (e.g. String#+ vs String#<< - the former creates a new String object every time); the latter includes things like overriding Fixnum#+, breaking all attempts at inlining maths, for example. The former is a matter of education, but a lot of the benefits for many things are masked by slow VMs today that in many cases makes it pointless to care about specific methods, and an expectation not to think much about performance (incidentally, it's not that many years ago that C++ programmers tended to make the same mistakes en-masse) The latter can be solved (there are other alternatives too), or at least substantially alleviated, in the general case by providing means of making a few small annotations in an app. E.g. we could provide a gem with methods that are no-ops for MRI but that signals to compilers that you guarantee not to ever do certain things, allowing many safeguards and fallbacks to be dropped. Ruby's dynamic nature is an asset here in that there are many things where we can easily build a gem that provides assertions that on e.g. MRI throws an error if violated or turns into a no-op, but that on specific implementations "toggle" additional optimizations that will break if the assertions are not met. E.g. optional type assertions to help guide the optimizer for subsets of the app. In other words: how optimized Ruby implementations we get depends entirely on whether people start trying to use Ruby for things where they badly need them.

by tomdale 12y ago

alelefant 12y ago

I've never built a Rails app, so perhaps there's something simple that I'm missing (I'm mostly a C# guy), but how would I use Skylight to monitor my on-premise application? The pricing makes it seems like a hosted service. I would have expected some sort of profiler to be needed on the server and then perhaps a centralized location for the data to be pushed to for display. The design of the site is great and I'm mostly curious because it sounds like something I wish was available in the .NET world!

tomdale 12y ago
Thanks for asking. If you're interested in the nitty-gritty, Yehuda and I gave a talk on the architecture behind Skylight at RailsConf: http://www.confreaks.com/videos/3394-railsconf-how-to-build-a-smart-profiler-for-rails http://www.confreaks.com/videos/3394-railsconf-how-to-build-... The short version is that the agent runs on your servers and collects information from your Rails app using the ActiveSupport::Notifications instrumentation built in to the framework. We serialize that into a protobuf that's transmitted via IPC to a background daemon (written in Rust). That daemon batches multiple reports into a single payload that is sent to the Skylight servers, where we use Storm and Cassandra to process the requests, and periodically do aggregate roll-ups. Unlike New Relic, Skylight gives you access to the entire distribution of response times, not just the average response time. (According to DHH, averages are "useless.")[1] This ends up being a lot of data, and a lot of CPU-intensive processing, which is why we sell it as a hosted service. [1]: https://signalvnoise.com/posts/1836-the-problem-with-averages https://signalvnoise.com/posts/1836-the-problem-with-average...
- yazaddaruvala 12y ago
  Just an FYI, while I think, to set up skylight I just add a gem to my project, I'm still not really sure. It would have been nice to go to your homepage and have it confirm that it is easy to configure/deploy into your application.
- grumbles 12y ago
  Minor point, but New Relic has had built-in support for collecting response time distributions for over a year. Please don't lie about this. https://docs.newrelic.com/docs/apm/applications-menu/features/histograms-viewing-data-distribution https://docs.newrelic.com/docs/apm/applications-menu/feature...
- creatio 12y ago
  Is it possible to get the names of the Phd thesis read?
deleted 12y ago
[deleted]

hit8run 12y ago

Hey Tom, did you also consider googles golang? Currently I'm building my backends with golang and I am pretty satisfied with the performance and the whole flow. Maybe it is sometimes a bit cumbersome to check on errors like a paranoid but in the long term it helps to predict what happens in error cases. How about rust? Is the workflow comparable to golang? Is there an equivalent to gofmt and some ide support for code completion? Thx and regards Bijan

dbaupp 12y ago
I don't speak for tomdale/wycats, but embedding a language with a GC (Go) inside another GC'd language (Ruby) only leads to worlds of pain, especially when trying to efficiently transfer data between them. In languages without compulsory GCs (like Rust, which doesn't have one at all) this works because it's easy to have complete control about when memory is freed, but a GC'd language may free memory that was passed back into Ruby since the GC can no longer find any references to it. I'm sure there's ways around this (e.g. using unsafe/raw pointers), but this control is the default in Rust, and there are a pile of language mechanisms (ownership, in particular) that make making this safe much easier. (Also, AIUI, Go's FFI story is relatively inefficient, and, has some rather bad bugs; apparently https://code.google.com/p/go/issues/detail?id=7978 https://code.google.com/p/go/issues/detail?id=7978 is the Go GC trying to free FFI memory.)
- Xorlev 12y ago
  From what I understood, it's a Rust agent communicating protobuf structs over IPC, not embedded into the gem. In such a case, memory safety and GC is less of an issue.
  - bjz_ 12y ago
    Memory safety is still good for long term code maintenance and refactorability. wycats also mentions that these processes are reasonably long lived - memory leaks can become a big problem over time if you are not careful. These can be mostly purged by static analysis and Valgrind, but are still a headache to deal with. I would also add that Rust is much easier to hack on safely if you don't have a great deal of prior systems programming experience.

Xorlev 12y ago

Love that you guys are using Rust in production. Very elegant design (AS::Notification IPC -> Rust). I'd dump the part of the marketing site talking about the fast UI on Ember. It's cool that you guys use Ember and you're proud of it, but I want to see more about your product. Show me how easy it is to drop into an app, or reports you generate, alerts you flag, etc.. Ultimately, while I nerd out on what a product is built with, I only care about what it can offer me. Fast UI isn't a feature, it's an expectation.

aaronbrethorst 12y ago

Somewhat OT, but... I'm looking at your pricing, and I don't have any idea how you define the notion of a "request". Right now, I can't even begin to guess what pricing bucket my apps might fit into, and that reduces my motivation to try out your service. I speculate I'm not the only person who's felt this way. (also, I know why you include the Ember mention at the bottom, but I really don't care. I'd rather see your awesome UI than hear what technology it's based on.) Edit: Seriously, I'm getting downvoted for offering someone feedback on their product experience?

annnnd 12y ago
Don't let it get to you - upvote and downvote are very close for mobile users. Here you have a +1 to compensate. :)

chrisseaton 12y ago

"We end up being able to do many more stack allocations, with less memory fragmentation and more predictable performance, while never having to worry about segfaults." But isn't this stuff the job of the VM? There's no reason why a program written in Ruby can't do the same stack allocations, reduced memory fragmentation and predictable performance automatically - if the Ruby VM was better designed. If Ruby had a better VM would you chose to use it over Rust? In Rust are you doing things that the VM could be doing for you?

dbaupp 12y ago
It is very hard to have all this done reliably, especially in a rather dynamic language like Ruby. Maybe the Ruby VM could be better designed to perform these sort of optimisations, but it will never give the same control as a language like Rust, C or C++; i.e. a small code change could cause the optimiser to fail, leading to slow-down and bloat. Furthermore, the hypothetical "sufficiently smart VM" isn't much value for code written now.
- vidarh 12y ago
  [Chris has done some fantastic work on a Truffle / Graal backend for jruby; for my part I'm (slowly) working on a "mostly-ahead-of-time" Ruby compiler] I'm not sure I'd agree with "never", though I do agree Ruby is a hard language to optimize. There are two challenges with optimizing Ruby: What people do and don't know the potential cost of, and what people pretty much never do, but that the compiler / VM must be prepared for. The former includes things like "accidentally" doing things that triggers lots of copying (e.g. String#+ vs String#<< - the former creates a new String object every time); the latter includes things like overriding Fixnum#+, breaking all attempts at inlining maths, for example. The former is a matter of education, but a lot of the benefits for many things are masked by slow VMs today that in many cases makes it pointless to care about specific methods, and an expectation not to think much about performance (incidentally, it's not that many years ago that C++ programmers tended to make the same mistakes en-masse) The latter can be solved (there are other alternatives too), or at least substantially alleviated, in the general case by providing means of making a few small annotations in an app. E.g. we could provide a gem with methods that are no-ops for MRI but that signals to compilers that you guarantee not to ever do certain things, allowing many safeguards and fallbacks to be dropped. Ruby's dynamic nature is an asset here in that there are many things where we can easily build a gem that provides assertions that on e.g. MRI throws an error if violated or turns into a no-op, but that on specific implementations "toggle" additional optimizations that will break if the assertions are not met. E.g. optional type assertions to help guide the optimizer for subsets of the app. In other words: how optimized Ruby implementations we get depends entirely on whether people start trying to use Ruby for things where they badly need them.