5 ms·

Crunchy Bridge's Ruby Back End: Sorbet, Tapioca, and Parlour

fny 4y ago
Something I've always wondered is if there's any reason to have so many different typing engines. TypeScript, for example, seems feature complete enough to provide a mechanism to type any language. (I'm almost sure this can be proven.) The real challenge instead would be to design the syntax extensions for typing so they feel natural. `sig { returns(Blah) }` feels ugly.
- emptysea 4y ago
  Infer kind of does this, it has multiple frontends and provides linting capabilities https://fbinfer.com https://fbinfer.com
ht85 4y ago
Doesn't the article make the whole thing sound like an incredibly painful experience?
newaccount2021 4y ago
ch4s3 4y ago
> But Ruby by itself has a major challenge in that it lacks any kind of built-in mechanism for expressing variable and method type signatures. Every one of us had managed large Ruby codebases in the past and wanted to avoid the quagmire of uncertainty around what the types of anything are supposed to be Ruby has a lot of ways to express the type of information you're passing around. The problem is when you don't validate things coming from users, APIs, and the database, or if you muddy you state in some overburdened god-object. I think the problem is that its so easy to write Ruby code and it less you do almost anything you want, so it is very easy to make a mess. Types may or may not help. I've worked in some big Rails/Ruby code bases and usually any mess was the result of someone who didn't know the language writing a bunch of code alone with little to no code review or guidance.
- karmakaze 4y ago
  > easy to write Ruby code and it less you do almost anything Types do help in that this would be visible as it's happening with the parameter list and allowed types list getting longer and longer or uses of untyped. A similar thing with highly overloaded types could happen in type-inferred languages but compile errors catch when expectations don't match.
  - ch4s3 4y ago
    I would definitely agree in a typed language, and I see how identifying this "smell" early could be nice in Ruby. However you can do that with validations too and get basically the same results. If your validations are overflowing the page, maybe you're doing too much. For something like Ruby types feel bolted on and for the trouble I'm not sure the payoff is worth it personally. But this is all just my opinion, clearly a lot of smart folks a Shopify disagree.
    - karmakaze 4y ago
      It is bolted on, and personally I might not use it either. But at scale it makes a big difference. The other difference is that I can not add a validation and the code passes existing tests and runs. Typecheck won't pass without making the messy longer type sig or writing the untyped smell.
      - ch4s3 4y ago
        I always hear the at scale claim, but have never seen it even in larger code based. The academic literature also seems far from conclusive. I get why people like types, but I think they’re a bit oversold.
        deleted 4y ago
        [deleted]
        karmakaze 4y ago
        The literature I read was about comparable defect rates between different programming languages producing comparable functionality. I hadn't read anything about productivity of different languages and team sizes--if you have any references I'd be interested in reading them. One story I did read was about how Twitter switched to Scala (and later Java IIRC) mainly for the static typing and not for performance or other reasons. I don't understand the not seeing it in a large codebase point. I work on a large Ruby code base that started without type annotations. Since I joined, we're now all adding type annotations for everyone's benefit, so they definitely exist. The reason for not wholly rewriting such a large codebase in a static typed language should generally be considered a no go. Many devs complained at the time and now most (if not all, I don't know of exceptions) now appreciate having the type checking and visible annotations for reading. I don't work on front-end but I expect that there was a some of that too in the React TypeScript codebases.
brightball 4y ago
Big fan of Sequel. Haven’t ever tried out the type checking stuff for Ruby but I might give it a look after reading this.
weatherlight 4y ago
Im not sure if Crystal had hit it's 1.0 release by the time they selected ruby for this project, other than a ruby-like repl, it seems like it hits all the marks
- nobleach 4y ago
  I've said this before (and been downvoted for it). Crystal is such a phenomenal language in that it brings a HEAVY spirit of Ruby (some Ruby stuff even runs unmodified) yet has wonderful type-safety. It's also compiled and relatively fast. It seems like such a win/win.
- willlll 4y ago
  We use Crystal a lot too at Crunchy Bridge. It is used for all of the on-instance programs, the per-region monitoring service, as well as the customer CLI. I'm a big fan of Crystal, and wrote the postgres driver for the language :) However a main reason we used Ruby was for the production REPL which lets us manage our entire server fleet with like 1/3 of the staff we'd need otherwise.
  - kureikain 4y ago
    thank willl. When I investigate in Crystal I found your github and your Posgres driver.
  - rafBM 4y ago
    Amen to that. We reached $2M ARR as 3 engineers at Missive [1]. As the CTO, I can tell life wouldn’t quite be the same without the ability to save the day in the production console at the speed of light. [1] https://missiveapp.com/ https://missiveapp.com/
  - theonething 4y ago
    > production REPL which lets us manage our entire server fleet with like 1/3 of the staff I'm intrigued. Are you talking about the `rails console` REPL? How did that allow you to manage your fleet with 1/3 staff?
skykooler 4y ago
Crunchy Bridge sounds like a new Intel chipset name.
fdr 4y ago
I work on Crunchy Bridge. Mostly, the Ruby parts discussed. I personally find Sorbet more marginal in its benefits than perhaps conveyed here, but the Parlour-Tapioca additions are key parts of making it practical. I'd recommend those to anyone looking to use Sorbet. Sorbet has been moderately expensive to implement and maintain, definitely not entirely a set-and-forget piece of infrastructure (compare: Sequel, which is), and periodically someone has to re-familiarize themselves with the details to address changes in Sorbet or Tapioca (somewhat less often, these days, perhaps?). It also has some unfortunate interaction (for load performance) with the lazy loading in Zeitwerk (also a good program), since you will tend to load files that you refer to types in only. And you can't really use Dependabot or similar if you need an rbi generation step, so someone needed to write a custom github action. Plus we needed to stumble around a bit with rbi generation (parlour and tapioca, which this post hopefully invites you to use). Also, the stack traces, as instrumented by the sorbet runtime, are pretty ugly, and I get to filter through those every single time. And it broke pry's show-source, which I've had to learn to live without. Code reloading is also a lot more busted than it otherwise needs to be. This kind of friction definitely adds up. My favorite part of sorbet is catching unbound symbol typos extremely quickly, often simple ones local to a file. That would probably be the 80/20 for me, if a simpler program to replace Sorbet's value to me were to be integrated. Probably people who really like LSP-style features will have a more positive assessment of sorbet than I convey here. Though I have lsp enabled, I don't find most of its features amazingly compelling, so, perhaps I am the odd one. I am perhaps overly influenced by xcscope.el as close to my ideal. Emacs's projectile, ivy, and counsel projects are a partial substitute, along with the hippie-expand feature. In my opinion, the 100% branch coverage briefly mentioned has more to do with the tenor of developing on the project. This was an adapted practice from Sequel and Roda, which are Jeremy Evans projects, which I have long admired in their methodology: Evans makes a huge scope of features look so natural to write and maintain, and I think the test coverage is part of that. My previous work at Citus Cloud also implemented a similar technique, based on line coverage, so I've been doing something like this for a number of years. Brandur, who wrote this, has more experience with large, fluctuating Ruby code bases that slide in cogency. I tend to think that the high coverage requirements carries more weight in how I experience daily development life than Sorbet does.
quechimba 4y ago
Sorbet is really good. Makes refactoring really easy.

fny 4y ago

Something I've always wondered is if there's any reason to have so many different typing engines. TypeScript, for example, seems feature complete enough to provide a mechanism to type any language. (I'm almost sure this can be proven.) The real challenge instead would be to design the syntax extensions for typing so they feel natural. `sig { returns(Blah) }` feels ugly.

emptysea 4y ago
Infer kind of does this, it has multiple frontends and provides linting capabilities https://fbinfer.com https://fbinfer.com

ht85 4y ago

Doesn't the article make the whole thing sound like an incredibly painful experience?

newaccount2021 4y ago

ch4s3 4y ago

> But Ruby by itself has a major challenge in that it lacks any kind of built-in mechanism for expressing variable and method type signatures. Every one of us had managed large Ruby codebases in the past and wanted to avoid the quagmire of uncertainty around what the types of anything are supposed to be Ruby has a lot of ways to express the type of information you're passing around. The problem is when you don't validate things coming from users, APIs, and the database, or if you muddy you state in some overburdened god-object. I think the problem is that its so easy to write Ruby code and it less you do almost anything you want, so it is very easy to make a mess. Types may or may not help. I've worked in some big Rails/Ruby code bases and usually any mess was the result of someone who didn't know the language writing a bunch of code alone with little to no code review or guidance.

karmakaze 4y ago
> easy to write Ruby code and it less you do almost anything Types do help in that this would be visible as it's happening with the parameter list and allowed types list getting longer and longer or uses of untyped. A similar thing with highly overloaded types could happen in type-inferred languages but compile errors catch when expectations don't match.
- ch4s3 4y ago
  I would definitely agree in a typed language, and I see how identifying this "smell" early could be nice in Ruby. However you can do that with validations too and get basically the same results. If your validations are overflowing the page, maybe you're doing too much. For something like Ruby types feel bolted on and for the trouble I'm not sure the payoff is worth it personally. But this is all just my opinion, clearly a lot of smart folks a Shopify disagree.
  - karmakaze 4y ago
    It is bolted on, and personally I might not use it either. But at scale it makes a big difference. The other difference is that I can not add a validation and the code passes existing tests and runs. Typecheck won't pass without making the messy longer type sig or writing the untyped smell.
    - ch4s3 4y ago
      I always hear the at scale claim, but have never seen it even in larger code based. The academic literature also seems far from conclusive. I get why people like types, but I think they’re a bit oversold.
      - deleted 4y ago
        [deleted]
      - karmakaze 4y ago
        The literature I read was about comparable defect rates between different programming languages producing comparable functionality. I hadn't read anything about productivity of different languages and team sizes--if you have any references I'd be interested in reading them. One story I did read was about how Twitter switched to Scala (and later Java IIRC) mainly for the static typing and not for performance or other reasons. I don't understand the not seeing it in a large codebase point. I work on a large Ruby code base that started without type annotations. Since I joined, we're now all adding type annotations for everyone's benefit, so they definitely exist. The reason for not wholly rewriting such a large codebase in a static typed language should generally be considered a no go. Many devs complained at the time and now most (if not all, I don't know of exceptions) now appreciate having the type checking and visible annotations for reading. I don't work on front-end but I expect that there was a some of that too in the React TypeScript codebases.

brightball 4y ago

Big fan of Sequel. Haven’t ever tried out the type checking stuff for Ruby but I might give it a look after reading this.

weatherlight 4y ago

Im not sure if Crystal had hit it's 1.0 release by the time they selected ruby for this project, other than a ruby-like repl, it seems like it hits all the marks

nobleach 4y ago
I've said this before (and been downvoted for it). Crystal is such a phenomenal language in that it brings a HEAVY spirit of Ruby (some Ruby stuff even runs unmodified) yet has wonderful type-safety. It's also compiled and relatively fast. It seems like such a win/win.
willlll 4y ago
We use Crystal a lot too at Crunchy Bridge. It is used for all of the on-instance programs, the per-region monitoring service, as well as the customer CLI. I'm a big fan of Crystal, and wrote the postgres driver for the language :) However a main reason we used Ruby was for the production REPL which lets us manage our entire server fleet with like 1/3 of the staff we'd need otherwise.
- kureikain 4y ago
  thank willl. When I investigate in Crystal I found your github and your Posgres driver.
- rafBM 4y ago
  Amen to that. We reached $2M ARR as 3 engineers at Missive [1]. As the CTO, I can tell life wouldn’t quite be the same without the ability to save the day in the production console at the speed of light. [1] https://missiveapp.com/ https://missiveapp.com/
- theonething 4y ago
  > production REPL which lets us manage our entire server fleet with like 1/3 of the staff I'm intrigued. Are you talking about the `rails console` REPL? How did that allow you to manage your fleet with 1/3 staff?

skykooler 4y ago

Crunchy Bridge sounds like a new Intel chipset name.

fdr 4y ago

I work on Crunchy Bridge. Mostly, the Ruby parts discussed. I personally find Sorbet more marginal in its benefits than perhaps conveyed here, but the Parlour-Tapioca additions are key parts of making it practical. I'd recommend those to anyone looking to use Sorbet. Sorbet has been moderately expensive to implement and maintain, definitely not entirely a set-and-forget piece of infrastructure (compare: Sequel, which is), and periodically someone has to re-familiarize themselves with the details to address changes in Sorbet or Tapioca (somewhat less often, these days, perhaps?). It also has some unfortunate interaction (for load performance) with the lazy loading in Zeitwerk (also a good program), since you will tend to load files that you refer to types in only. And you can't really use Dependabot or similar if you need an rbi generation step, so someone needed to write a custom github action. Plus we needed to stumble around a bit with rbi generation (parlour and tapioca, which this post hopefully invites you to use). Also, the stack traces, as instrumented by the sorbet runtime, are pretty ugly, and I get to filter through those every single time. And it broke pry's show-source, which I've had to learn to live without. Code reloading is also a lot more busted than it otherwise needs to be. This kind of friction definitely adds up. My favorite part of sorbet is catching unbound symbol typos extremely quickly, often simple ones local to a file. That would probably be the 80/20 for me, if a simpler program to replace Sorbet's value to me were to be integrated. Probably people who really like LSP-style features will have a more positive assessment of sorbet than I convey here. Though I have lsp enabled, I don't find most of its features amazingly compelling, so, perhaps I am the odd one. I am perhaps overly influenced by xcscope.el as close to my ideal. Emacs's projectile, ivy, and counsel projects are a partial substitute, along with the hippie-expand feature. In my opinion, the 100% branch coverage briefly mentioned has more to do with the tenor of developing on the project. This was an adapted practice from Sequel and Roda, which are Jeremy Evans projects, which I have long admired in their methodology: Evans makes a huge scope of features look so natural to write and maintain, and I think the test coverage is part of that. My previous work at Citus Cloud also implemented a similar technique, based on line coverage, so I've been doing something like this for a number of years. Brandur, who wrote this, has more experience with large, fluctuating Ruby code bases that slide in cogency. I tend to think that the high coverage requirements carries more weight in how I experience daily development life than Sorbet does.

quechimba 4y ago

Sorbet is really good. Makes refactoring really easy.