17 ms·

Rewriting the Ruby parser

520 points by kddeisz 3y ago

noelwelsh 3y ago
It feels to me like Shopify is single-handedly keeping Ruby alive. A little bit like Jane Street and O'Caml.
- giovannibonetti 3y ago
  Github is built with Ruby on Rails and they are heavily investing on it, too. See this recent blog post [1] for more details. [1] https://github.blog/2023-04-06-building-github-with-ruby-and-rails/ https://github.blog/2023-04-06-building-github-with-ruby-and...
- jrochkind1 3y ago
  Github employees also send a lot of commits to Ruby and Rails. But probably not as many as Shoppify. There might be one or two additional big companies that are similarly funding employees to contribute to core infrastructure in significant ways. But not too many more than a handful, I agree. And they are carrying a lot of weight in ruby ecosystem for sure. I feel like this era of open source in general is one of very shifting patterns of contribution for sustainability. One or a few big companies paying people to keep the thing alive is definitely one that seems to be increasing. Perhaps since the company(ies) in question didn't originate the product(s) in this case, it doesn't feel like they "own" them exactly (not like "open source" products originated and developed by only one company where the product is their business itself -- not sure which category O'Caml fits in), but the risk in depending on only one or two companies (where the product is _not_ their actual business itself) is that if the company decides it no longer wants to make the investment, it can definitely be disastrous for the product. Shoppify just did a bunch of layoffs -- I don't think they hit the people contributing to ruby too hard, but they easily could have, except perhaps Shopify too realizes that if they stopped keeping Ruby alive it would be disastrous for their own business.
- ksec 3y ago
  Github is doing some as well. Although with resources from Microsoft I do wish they do more.
- lagniappe 3y ago
  You're going to incur the wrath for saying the silent part, but you're not wrong. People have the same 5 rails example corps every time someone says one of these two things: - without rails nobody would use ruby - without X corp, ruby's dead Fact is, we're seeing less and less usage, and more and more distillation of the current userbase along the golden paths laid out by DHH. Is it wrong for rails and thus ruby adoption to slow down? Not at all, people should use what they like, however I think Rails and Ruby are in this negative spiral where: - rails is mostly needed for prototyping and crud apps - this work is typically done by juniors - rails devs are at this point largely seniors, not juniors - rails devs pay the bills with other tech or by maintaining legacy rails apps. There are startlingly few deviations from this, and either everyone majors in rails with a minor in javascript and C++ or they just get happy with their current gig and settle. I wish it weren't the case, but Ruby just hasn't done enough to differentiate itself from Rails, and when compared to neo-PHP or JS there's just not a lot of attractive parts of the golden path Rails provides. It's off-tune for this generation of choice and ubiquity. We don't need that level of scaffolding anymore and in many cases there are other tools that handle that with more versatility.
  - rgrieselhuber 3y ago
    I mentally put Ruby / Rails and Python with Flask / Django (and also “neo” PHP) in the same category vs all of the 70 million JS frameworks. When you talk about alternatives, what are the major ones you have in mind?
  - toolz 3y ago
    Are you suggesting php and JS are stealing from what would've otherwise been ruby projects? I've been a ruby dev for a decade and I've never heard of a ruby shop migrating to using php for new work and maybe one or two moving to JS for new work. It's golang and elixir that are taking the place of ruby for new work in my experience. I suspect some python too, but I haven't seen that.
    - runjake 3y ago
      I think they're suggesting that neo-PHP and JS adopted the best parts and patterns of Rails (eg. Laravel).
- hardwaregeek 3y ago
  This is wholly irrelevant, but I love the spelling of O'Caml like an Irish last name. I'd definitely frequent a pub called O'Caml's.
  - noelwelsh 3y ago
    OCaml was originally a contraction of Objective Caml, and spelled O'Caml. At some point this changed and the spelling without the apostrophe was adopted. What comes out of my fingers hasn't caught up, though.
  - jjgreen 3y ago
    Given the early logo for the language was Joe Camel, it would need a smoking room ...
- brightball 3y ago
  I don't know why, there are tons of smaller companies using it. I have a lot of languages and experience on my resume and the one that consistently gets me the most inquiries at the highest pay grades is still Ruby. Anecdotal I know, but from the moment that it appeared on my resume in 2012 it's been non-stop. Probably 80% of everything I hear about. Ruby and it's ecosystem brings the closest thing to natural Aspect Oriented Programming that I've seen in the wild, which is why it's so much more productive than everything else I've tried.
  - clairity 3y ago
    i think if rails continues to push hotwire (turbo + stimulus, and perhaps strada?) and get a coherent story on view components, it will continue to take mindshare from the js hype of the last decade. mobile dev is in decline, and browser makers just released web notifications, web app support, webtransport, page transitions, etc., so the backend has largely reached parity for cross-platform development. no longer is json the natural data exchange medium for apps, but rather chucks of html that can be plopped right into the dom without js having to massage the response into shape on the frontend. js can return to being a frontend scripting language, its natural habitat, rather than being shoehorned into being a do-it-all platform language.
    - d3nj4l 3y ago
      I've been building a webapp with Rails on and off over weekends. Several times over this process, I thought through some of the architectural decisions and naturally realized that the "Rails way" was the best option to pursue. It's not just because I'm using Rails - my most recent webdev experience was with a SPA driven by a Java backend. I'm sure there are tradeoffs involved (what doesn't?) but with every passing day I use Rails the more I appreciate the decisions it makes for you.
      - graypegg 3y ago
        I’ve had the exact same experience! I’m getting to use some new JS SSR frameworks at work (Remix) but I keep using rails for my own things. Gets out of my way.
- deleted 3y ago
  [deleted]
- kjghkjghkjgh 3y ago
  This is spoken like someone who knows nothing about Ruby.
- riffraff 3y ago
  stripe seemed to be doing quite a few interesting things (e.g. Sorbet, and there was some work adopting TUF for rubygems IIRC) but it seems to have dialled down things a bit.
- ljm 3y ago
  I think ‘is Ruby dying?’ is little more than a meme that has stuck around for longer than it deserves. The continuing work on the language and its performance is impressive, but Ruby (and Rails) themselves have the honour of being stable, tried-and-tested solutions for rapid application development. Is it as exciting as the latest and greatest serverless lambda framework in Typescript? Not really. Is it a dependable workhorse? Absolutely. At some point you might be successful enough to justify a rewrite into something else, but a simple Rails app will take you a long way with little effort.
  - sergiotapia 3y ago
    I used to get invites for jobs and see listings for jobs for Ruby and Rails a lot. Now I don't see any at all. There's most definitely a "dead" feeling to the platform. I hardly hear about new projects being started with Rails as well...
    - scarnz 3y ago
      My co-founder and I are using RoR at our new company! It's definitely not dead and allows you to build SAAS very quickly.
    - 1123581321 3y ago
      It might just be the economy. I was getting about 20 inquiries a week from Rails positions until around December 2022. It's used in many companies, new and legacy.
      - fknorangesite 3y ago
        Yeah same story here. Until about 6-7 months ago I was hearing from recruiters for Rails jobs daily, both old and new projects. I'd be very surprised if the decrease since then is out of line with the rest of the industry, regardless of tech stack.
    - ljm 3y ago
      Probably depends where you’re based. Ruby is still lucrative in London and there is a healthy market for it both in startups and more established businesses. While I’ve branches out to other languages (not just JS as a full stack engineer) my career is still boosted by my Ruby experience. I don’t think this makes it dead or dying though. It’s stable and entrenched while JS has taken the place of the golden child. One complaint I’ll grant myself is that library development is a little less prolific this days. Again, there are well-established solutions to a lot of problems in Ruby so you’ll have a go-to collection of gems, but it’s more often the case these days that something doesn’t have much library support and you’ve got to roll your own.
- sam0x17 3y ago
  People forget Ruby is also massively popular in Japan to this day
  - ramesh31 3y ago
    >People forget Ruby is also massively popular in Japan to this day So are fax machines.
- johnisom2001 3y ago
  This couldn't be further from the truth. There are so many people using Ruby, so many modern companies deciding to use it and so many large companies that continue to use it. The demand for Ruby developers is higher than the supply.
  - deleted 3y ago
    [deleted]
  - ksec 3y ago
    Your parent is talking about investing in Ruby the language and its ecosystem. Not just using it. The new GC, and JIT, along with a few Ruby Cores are all Shopify employees.
- paulddraper 3y ago
  Github: "Am I a joke to you?"
SushiHippie 3y ago
Somewhat fitting xkcd: https://xkcd.com/927/ https://xkcd.com/927/
- revscat 3y ago
  While this xkcd is frequently germane, in this case it is not particularly apt. The core CRuby team, who maintain the current “standard”, are in agreement that YARP — the new “standard” — will one day replace it.
  - Dylan16807 3y ago
    Even putting the word "standard" in quotes is buying too much into the argument. This is a new implementation of an existing standard.
- wiseowise 3y ago
  It’s not fitting at all. > We recently got approval to merge this work into CRuby, and are very excited to share our work with the community.
soulbadguy 3y ago
I love how pragmatic the approach is here. We still don't have parser generators which generate human understandable/readable recursive descent parser with good error recovery baked in. But i am guessing the ruby syntax is too complicated/irregular anyway. On a macro side, i am always surprised with the length and effort company will go to keep working with solution design and selected at their young age, and try to scale them well beyond the braking point. And i feel like most of this effort to keep ruby alive could have been used to slowly transition part of the infra to something JVM/.Net/Rust based for much more bang for engineering buck.
- ecshafer 3y ago
  (Disclaimer: Shopify employee but not part of this team) Ruby is my favorite language so far to write "business logic" in, and Rails is a very developer friendly web framework that makes it very easy to set up websites. So I think this is a pretty great strategy overall. The end of the day I think the way to get tech too scale, is for companies / academics / etc to really just put the work in. Java was very slow on release, but now the JVM is blazingly fast, and that is because of the huge amount of work put in by Sun, Oracle, IBM, Redhat, Google, Azul, etc. over the years. For what its worth though, we do also have Go, Rust, Scala, etc. in different places in our stack where its needed.
  - soulbadguy 3y ago
    Disclaimer : my career has mainly been on compiler/runtime and other infrastructure related stuff, so not much of a business logic/web dev kind of guy. > The end of the day I think the way to get tech too scale, is for companies / academics / etc to really just put the work in. This is true in the abstract, but my point is mainly about the comparative effectiveness of different strategies when it come to building scalable infrastructure (or scaling up infra): Keep the bulk of the existing code and invest in making the underlying compiler/interpreter/tool chain better or progressively migrate the code to a tool chain with better scaling capabilities from the get go. From experience, the nature and semantic of a language severely what is "reasonably" possible in the runtime in term of safety (like code loader in java), performance (both CPU and memory) and tooling. Now improvement are possible, but they tend become exponentially expensive as time goes on. Now, obviously there is always a tension between developer productivity and infra concerns, but i do believe that we have better compromise point on that line with newer language and framework. I totally agree that in the 2000's the experience of most "enterprise framework" sucked very hard, and the emergence of language such ruby/python etc... was a god send : a response to the overly rigid and ceremonial way of the past. But with time, we have been able to understand better what makes a good programming experience, distill that into better designed languages and framework which offer "better" compromises. For example : - Instead of dynamic vs static type , we have progressively typed and type inference - Instead of GC vs non-GC we have rust borrow-checker - Instead of runtime meta programing with have DSL and macro's Even more important, i believe that a lot of the experience come from the tooling around the language, and there again the "cargo/dotnet/go" cli approach with a single coherent entry point for both package management and framework scaffolding ease a lot of the pain of the old way. With all of that we now have languages which offer a better compromise on the dev. prod vs infra/performance... > JVM is blazingly fast I would say blazingly faster... But compared to C++ (or even rust) java is still quite slow. Especially for anything compute intensive.
jrochkind1 3y ago
As a devoted rubyist, it seems clear to me in retrospect that ruby's "pretty complex to parse is just fine" approach has been a real challenge to the language's success. For reasons related to discussion in OP. This project could be one approach to ameliorating that challenge, I'm very pleased to see shopify willing to invest in this, and hope it ends up effecting the ecosystem as hoped.
- munificent 3y ago
  I disagree. There are many languages that are hard to parse that are very successful. C and C++ are the obvious examples: there are many dark corners around the preprocessor, type annotations, etc. C++ is an absolute nightmare to parse. C# is pretty complex too. Python has its weirdness around indentation. Honestly, most popular languages end up acquiring a decent amount of grammatical complexity over the years and it doesn't seem to significantly hinder adoption. Humans are quite good at reading complex text and syntax. I think the real tax on Ruby is its pervasive use of runtime metaprogramming. It's Ruby's most exciting strength and enables much of the joy and excitement that Ruby is known for. But it makes static analysis so hard and becomes less and less valuable as team and codebase size increases.
  - nine_k 3y ago
    I'd say that C and C++ were successful despite their being hard to parse, and other unfortunate hacks, like null-terminated strings and include files instead if modules There was little choice for other features they offered (like Unix being written in C and bundling a C compiler) which.were the compelling reason to use them. JavaScript was a huge success not because it was a great language, but because everyone wanted to write for the browser.
    - munificent 3y ago
      I think you and I are saying essentially the same thing: grammatical complexity seems to have little negative impact on language success. > There was little choice for other features they offered (like Unix being written in C and bundling a C compiler) which.were the compelling reason to use them. Pascal was around at the same time, is grammatically infinitely more elegant... and is dead.
e4m2 3y ago
Interestingly, the actual syntax tree and related structures/functions are generated from the config.yml file and the templates inside the bin directory. They are using a custom template language written in Ruby, here's an example of how they do enum stringification: https://github.com/ruby/yarp/blob/main/bin/templates/src/token_type.c.erb https://github.com/ruby/yarp/blob/main/bin/templates/src/tok.... Obviously this isn't a novel idea, but IMO this kind of design goes a long way to support their maintainability argument, especially in C. Also, > CRuby actually ships with 90 encodings (as of 3.3) This is asinine.
- kddeisz 3y ago
  The templating is definitely an effort to keep the maintainability in check. It's also because we're planning on keeping a grammar around to generate test cases for us with fuzzing/other algorithms. The encodings is a bit historical, to my knowledge Ruby is the only major language that was developed by someone whose language did not work with ASCII. So the first encodings written in Ruby were the old Windows pages.
- cremno 3y ago
  >> CRuby actually ships with 90 encodings (as of 3.3) >This is asinine. Overall there are even more: 103. But then again 1.9.2 only has 85 and 95 overall. Also one of those new 'overall' ones is the EBCDIC code page for US/Canada.
- munificent 3y ago
  > Interestingly, the actual syntax tree and related structures/functions are generated from the config.yml file and the templates inside the bin directory. That makes a lot of sense. ASTs tend to be a very dumb but fairly verbose data structure. And, in C in particular where everything is more verbose, there ends up being a ton of boilerplate in the AST nodes. In a language like Ruby with a very rich syntax, you end up needed a ton of different AST nodes. Generating those from a simple declarative format makes maintaining that much easier.
- vidarh 3y ago
  > a custom template language written in Ruby This makes it sound like it's some template language specific to this project, but ERB is the dominant templating language for Ruby because an ERB implementation is in the standard library.
davexunit 3y ago
I think Ruby is a pretty good language and I spent many years using it professionally but I can't imagine having enough motivation to work on the language implementation when it's so difficult to even parse the code. The amount of engineering time spent on this topic is bonkers!
- kddeisz 3y ago
  Maybe I'm a masochist, but to be honest it's a dream job. Working on such a complex language and history is really really fun.
chubot 3y ago
Nice article! In this sentence ... open parenthesis character is ambiguous in this context. To get around it they made their grammar more ambiguous and then enforced that the actual grammar was enforced in their tree builder. I'd change the second "more ambiguous" to "more lenient" i.e. lenient meaning "grammar accepts more strings", ambiguous meaning "grammar is invalid" I have seen this issue in Python's grammar as well, and it's mitigated by the new PEG parser
- awestroke 3y ago
  Ambigious does not mean invalid
  - chubot 3y ago
    Yes, technically it means "there's more than one derivation" But Python's pgen rejects such grammars as invalid ... The other strategy is just to pick an arbitrary interpretation
  - hinkley 3y ago
    For people who see parsing as model checking, ambiguity means something vastly different (inferior) to leniency. Fast parsers that use a schema are also doing model checking.
Alifatisk 3y ago
What a impressive work & effort! Well done.
cout 3y ago
With the mention of the parser's performance, I am reminded of Rich Kilmer's 2004 RubyConf presentation. Rich used Ruby to test reliability of a distributed system with hundreds of Java VMs. Some of the serialized data was stored as XML (over 1M lines!), which was slow to parse and load. Rich modified the program to serialize the data as Ruby code, which loaded much faster (https://www.infoq.com/news/2007/06/infoq-interview-rich-kilmer/ https://www.infoq.com/news/2007/06/infoq-interview-rich-kilm...): > Chad Fowler and I over basically 2 weeks, took the Java Debug Wire Protocol specification [...] turned it into a DSL in Ruby, and used that DSL to generate the packets for sending and receiving data. So we used the DSL in Ruby as a generator to generate Ruby code, as the whole protocol and then I used that at Darpa. They were trying to say “we could freeze the agent society from within the agent society will send messages, and it took about 7-8 minutes for all the messages to propagate and everything to freeze and then go quiet. I had a Ruby process that was running all 300 VM’s were underneath it, and I could freeze it in about a half of second. All 300 of them! And you actually could watch the CPU use because we had a monitor and the CPU use has dropped to zero. And it freaked them out. And what was great was you could turn it back on, and all the agents came back on. But time had been lost. And it was like alien abduction lost time, ten minutes went away, “what happened to us?” It was a bizarre thing because they were agents and they were planning on things and all of sudden 10 minutes just went away. But it was interesting to show how Ruby could actually be used as this kind of harnesses to wrap around things like systems.
- srgpqt 3y ago
  Nice cover story to hide your ~totally existing~ time machine ;)
- RhodesianHunter 3y ago
  >Some of the serialized data was stored as XML (over 1M lines!), which was slow to parse and load. Rich modified the program to serialize the data as Ruby code, which loaded much faster So he took a data format designed for human readability and converted it to a data format that's designed purely to be read by Ruby and people are surprised that it's faster?
  - 3y ago
fareesh 3y ago
Will this change impact Solargraph / Rubocop? They are painfully slow / unusable in their current state on large-ish projects
- kddeisz 3y ago
  Depends on what you mean by Solargraph, because it's kind of a large project. If you mean their typechecking, then definitely not, because it's not using a Ruby parser. If you mean the general feedback, then yeah potentially. Rubocop yes if it ends up using YARP as a new backend. Either way, I would suggest you check out ruby-lsp, which is definitely going to benefit from this speed, and soon.
- pianoben 3y ago
  The article mentions that they are building a compatibility layer around YARP that tools can use to transform its new tree format into the legacy Ripper format. They don't call out Rubocop specifically but I can't think of another OSS tool that so prominently uses the parser APIs.
  - kddeisz 3y ago
    rubocop uses rubocop-ast uses parser, so it will eventually make its way down to rubocop once we finish the compat layer for the parser gem.
- dabears 3y ago
  I made a Ruby LSP because of this problem. It's not perfect but incase it's helpful for you. It can parse a large project with all of its gems in a few minutes. That data is indexed in an in-memory db with Tantivy. https://github.com/pheen/fuzzy_ruby_server https://github.com/pheen/fuzzy_ruby_server
fareesh 3y ago
I use ruby/rails for professional consulting work. It helps get startups off the ground and into a MVP stage quite quickly, and there is high productivity. You can also use whatever React/Vue on the frontend if you really need to. I would like to use some Typescript framework but none are "there yet" with regard to productivity. Scaling rails in some scenarios is quite challenging, but in most cases you can leverage caching to solve performance challenges.
- justanother_guy 3y ago
  Nest.Js seems to be promising, but like you said, it has a long way to go.
- weaksauce 3y ago
  with the improvements to hotwire you might not even need to get to a heavy frontend like react
fireweed 3y ago
[dead]
soulbadguy 3y ago
While on the subject, anybody has good reference on parsing error recovery ?
- di4na 3y ago
  If you find it, i am interested. But it seems to be a topic that is still open to research and niche to a small group of devs
  - soulbadguy 3y ago
    I suspect that most of the knowledge on this topic is embedded in source code of most prominent compiler tool chain and the head of their dev. :( I think they might an interesting intersection here we ML, where can could learn the comment mistake pattern made by real user and either error correct better, or at least provide pin point accurate error messages.
    - di4na 3y ago
      Not really no.
johnisom2001 3y ago
This is big. Ruby syntax errors are some of the most frustrating things to track down. So much worse than C-like languages like JavaScript. I'm glad to see big steps towards this.
progitter 3y ago
Well described. Also I found this on github. https://github.com/whitequark/parser https://github.com/whitequark/parser
- vidarh 3y ago
  That is one of the many parsers listed in the article.
freedomben 3y ago
It's been really great watching Ruby over the last few years. I had the privilege of having dinner with Matz several years ago near the beginning of Ruby 3.0 work. I was giving a talk about Elixir at a Ruby conference, and he was interested in things I (as a self-professed fanboy of both languages) liked better about Elixir than Ruby. We talked for quite some time, and it became very clear to me that he knew that Ruby was stagnating and wanted to make some changes to keep it relevant (without wrecking the language). He was incredibly open-minded (and nice!) and was willing to listen to anyone who had things to say. Flash forward several years, and the amount of changes in Ruby are huge! Ruby is as fit as ever for modern development, and I'm really happy about that.
- ljm 3y ago
  Pattern matching alone is a huge feature and an absolute delight to use when the opportunity presents itself. I’d love to see where Ractor goes but I worry it will remain niche, like with Refinements.
  - freedomben 3y ago
    Agreed. Pattern matching and pipe operator were the two that I hoped for most. The pipe operator implementation proposal that they team came up with was wrong though and I'm glad they didn't do it. We don't need alternate syntax for `.`, we need ability to chain arguments together in a syntactically pleasant way in a functional style so we don't have to write `first(second(third(arg)))` we can write `arg |> third() |> second() |> first` which is much cleaner and reads left to right like it should
    - zverok 3y ago
      Ruby-idiomatic pipe operator is Object#then. After a lot of design proposals and discussions it is more or less evident no solution other than method would integrate naturally with the rest of the code. So it is just `arg.then{ third(_1)}.then{ second(_1)}.then{ first(_1)}` Would've been a bit more concise with method references, almost introduced in 2.7, but alas. (But, well, people tend to want "something that looks like operator, preferably something that looks exactly like |>" and reject every other possibility)
PaulHoule 3y ago
It’s little appreciated how much current parser generators are holding the industry back. That is, LISP maintains a lead in metaprogramming because it bypasses Chomskyism alltogether. It really should be a few lines of code to add an “unless(X) {}” statement to “if(!X) ()” to Java, Python, Ruby but the very idea that you could patch an existing grammar with a separate file is like technology that fell off a UFO. Also anything that you generate a parser for should automatically generate an unparser. Sphinx is a fraction of the framework it could be because it can’t process markup and turn it back to Sphinx. I think a PEG framework with a few features (like an easy way to implement operator precedence just by stating it with either numeric values or X > Y statements) could be revolutionary. Python is almost there but not quite.
- barrkel 3y ago
  Custom grammars are a cool idea. They are terrible in practice and one of the biggest reasons why Lisp failed to take off, why it's almost noone's first choice of language for building something big. The reason is that new grammar creates a new language, and the divergence of the new language from the base language creates a cultural barrier that inhibits communication. It's harder for new engineers to be productive, and it's harder to collaborate. Customizing your grammar works best if you're a lone wolf, a one man band. You can lever up your productivity by custom-designing something perfectly suited to both you and your preferred problem domain. But nobody else will understand it.
- G3rn0ti 3y ago
  If you’re into language parsing and how its patterns could be used in a modern programming language you should take look on Raku (fka Perl 6): It has native support for „Grammars“ [1] which are basically specialized classes to put Regexes in for defining tokens and also combinations of such tokens. Once you‘re done you can use a grammar object to parse text returning an AST object. Raku‘s own syntax is defined in Grammars being a subset of Raku‘s syntax. So in a nutshell Raku‘s „Grammar“ construct is like RegExes on steroids and renders defining DSLs or other special purpose languages so much easier than in any other modern language. [1] https://docs.raku.org/language/grammars https://docs.raku.org/language/grammars
- 3y ago
IshKebab 3y ago
People always say language choice doesn't matter, but how many companies get stuck with a language where they resort to rewriting the parser for it?
- nirvdrum 3y ago
  No one is "stuck" or "resorting" to anything. It's an effort to improve tooling, performance, error reporting, and working with Ruby in general. We could maintain the status quo (which does indeed work) or we could improve it. If we don't improve the status quo, people call the language stagnant and outdated. If we do, it's considered a criticism of its maturity. How many languages would even entertain such a contribution? How many languages are so tightly bound to their parser that writing a new one wouldn't even be workable? How many languages have user tooling that works extensively with parsed code snippets? How many languages have multiple implementations? Of those, how many are working together on an effort to share a common parser? There's plenty of valid reasons to use something other than Ruby. An open source project that grew organically over 25 years deciding that it's time to pay down some technical debt and improve the ecosystem at the same time is probably not one of them.
nusaru 3y ago
> This is the final step before merging YARP into CRuby, and we’re very excited to see it come to fruition. This work will be done in the next couple of work days. It continues to amaze me the pace at which kddnewton and co. are able to work. His tweet[1] from a few months ago is really relevant here. Being full-time employees and not just open-source volunteers makes a huge difference! [1]: https://twitter.com/kddnewton/status/1639258413120073730 https://twitter.com/kddnewton/status/1639258413120073730
tester756 3y ago
A lot of cool work! Congrats!
nsajko 3y ago
Tangential: > Over the years, processors and C compilers have gotten much better using a couple of techniques. These include pipelining, inlining functions, and branch prediction. Unfortunately, the parsers generated by most parser generators make it difficult for any of these techniques to apply. Most generated parsers operate with a combination of jump tables and gotos, rendering some of the more advanced optimization techniques impotent. Because of this, generated parsers have a maximum performance cliff that is extremely difficult to overcome without significant effort. Although generating parsers, and finite automata in general, using a table-based approach is common, it has long been recognized that using tables/data for this purpose (as opposed to generating executable source code directly) is not a good idea, precisely because it inhibits compiler optimizations. I think the current situation is simply a consequence of the fact that parsing abruptly stopped being sexy multiple decades ago. Much better parser generators are possible, and LR, specifically, has much untapped potential left.
ramesh31 3y ago
So Shopify spends millions of dollars in engineering salary to get a substandard performing language up to par with a dozen other contemporary better choices? The choice to stick so hard with Ruby just doesn't seem reasonable at all with their scale. It must be something dogmatic from the top.
- agotterer 3y ago
  Facebook did the same when they developed HipHop for PHP. I imagine the reasoning is the same in both cases: spend a few million dollars and a handful of engineers improving the language and tools, or spend hundreds of millions and years of time rewriting everything into a more performant language. I honestly think the math pencils and makes sense.
stevebmark 3y ago
"Rails at scale" is a funny title. Shopify had to rebuild the core parser. Github has a department dedicated to working off the bleeding edge of Rails (and has core Ruby and Rails maintainers on staff, Shopify probably does too). Both Shopify and Github spent literal engineering years upgrading Rails, and still for some reason think it's worthy of boasting about in engineering blog posts, despite the burnout, not-shipping-features, and turnover they suffered because of it. If your company has enough engineering overhead and capacity to stop delivering features and try to fix your core language for a year, then you're in a different ballpark from most other tech companies. If you're NOT in that boat, then "Rails scaled for Github and Shopify" is not a message you should take home.
- vidarh 3y ago
  If you're not in that boat, odds are you don't need to care. And they didn't stop delivering features for a year.
tiffanyh 3y ago
Hack? I wonder if Shopify will ever give up on trying to fix Ruby and instead, do what FB did with PHP and just create their own derivative (Ruby) language that addresses their needs better.
- kayodelycaon 3y ago
  Why? Shopify was able to work with the core team and make the whole language better. Facebook has far more employees and wasn’t going to get PHP improved to their liking.
  - tiffanyh 3y ago
    The irony is that PHP has improved massively in perf vs the relative gains of Ruby. https://onlinephp.io/benchmarks https://onlinephp.io/benchmarks https://serpapi.com/blog/benchmarking-ruby-3-1-yjit-ruby-2-7/amp/ https://serpapi.com/blog/benchmarking-ruby-3-1-yjit-ruby-2-7...
    - vidarh 3y ago
      Apart from measuring very different things, the PHP page compares PHP releases over something like 25 years vs. about 3 years of Ruby releases. Look at PHP 7.2 onwards, which is a similar timeframe, and the improvements are fairly modest.
tiffanyh 3y ago
GraalVM? What’s the progress on Ruby/GraalVM that Shopify has heavily invested in?
- ximus 3y ago
  GraalVM Ruby, more commonly known as TruffleRuby, is actively developed still. As the article states, it adopted YARP already.
  - tiffanyh 3y ago
    My understanding is that Shopify still hasn’t been able to migrate to Graal/TruffleRuby, even after 10-years of its development.
    - nirvdrum 3y ago
      Your phrasing could be misconstrued by those not familiar with the history. Shopify hasn't been developing TruffleRuby for 10 years. The entire TruffleRuby project is 10 years old, including the first 12 - 18 months where it was a humble intern project. TruffleRuby (previously called JRuby+Truffle) was initially a research project for testing out optimizations, not a production-quality implementation suitable for deployment. I think it's now a viable deployment target, but that wasn't true 10 years ago. As you might imagine, there's a lot more to production deployments than simple Ruby compatibility. At Shopify, we've recently modified our CI system to support TruffleRuby and are actively running projects against TruffleRuby in CI now [1]. The TruffleRuby 23.0.0 release coming out in the next day or so includes quite a few compatibility issues we've worked out getting a large Rails app booting. That project is on the order of months, not decades. YARP will make adoption of TruffleRuby easier. Absent a language specification, implementations like JRuby and TruffleRuby have to match what CRuby is shipping and that by necessity means they lag after CRuby releases. Parser changes are amongst the hardest things to port over. YARP eliminates most of the challenges there. [1] -- https://railsatscale.com/2023-06-12-truffleruby-in-shopify-ci/ https://railsatscale.com/2023-06-12-truffleruby-in-shopify-c... Disclaimer: I'm on the TruffleRuby team at Shopify.
thiht 3y ago
Great read. I’ve never written a single line of code in Ruby but I love reading stuff related to compilers.
mingodad 3y ago
It's not true "the best chance you have is reading the 14 thousand-line parse.y file and trying to understand it", there are several tools to navigate yacc/bnf style grammars like https://www.bottlecaps.de/convert/ https://www.bottlecaps.de/convert/ and it's companion https://www.bottlecaps.de/convert/ https://www.bottlecaps.de/convert/ that make relatively easy to understand/document/debug/compare the grammar. Just added several ruby grammars here https://github.com/mingodad/plgh/tree/main/ruby https://github.com/mingodad/plgh/tree/main/ruby they are converted (mainly using https://www.bottlecaps.de/convert/ https://www.bottlecaps.de/convert/) to an EBNF understood by https://bottlecaps.de/rr/ui https://bottlecaps.de/rr/ui to generate navigable railroad diagrams. Copy and paste the EBNF on https://www.bottlecaps.de/rr/ui https://www.bottlecaps.de/rr/ui on the tab "Edit Grammar" the click on the tab "View Diagram" to see/download a navigable railroad diagram.

520 points by kddeisz 3y ago

noelwelsh 3y ago

It feels to me like Shopify is single-handedly keeping Ruby alive. A little bit like Jane Street and O'Caml.

giovannibonetti 3y ago
Github is built with Ruby on Rails and they are heavily investing on it, too. See this recent blog post [1] for more details. [1] https://github.blog/2023-04-06-building-github-with-ruby-and-rails/ https://github.blog/2023-04-06-building-github-with-ruby-and...
jrochkind1 3y ago
Github employees also send a lot of commits to Ruby and Rails. But probably not as many as Shoppify. There might be one or two additional big companies that are similarly funding employees to contribute to core infrastructure in significant ways. But not too many more than a handful, I agree. And they are carrying a lot of weight in ruby ecosystem for sure. I feel like this era of open source in general is one of very shifting patterns of contribution for sustainability. One or a few big companies paying people to keep the thing alive is definitely one that seems to be increasing. Perhaps since the company(ies) in question didn't originate the product(s) in this case, it doesn't feel like they "own" them exactly (not like "open source" products originated and developed by only one company where the product is their business itself -- not sure which category O'Caml fits in), but the risk in depending on only one or two companies (where the product is _not_ their actual business itself) is that if the company decides it no longer wants to make the investment, it can definitely be disastrous for the product. Shoppify just did a bunch of layoffs -- I don't think they hit the people contributing to ruby too hard, but they easily could have, except perhaps Shopify too realizes that if they stopped keeping Ruby alive it would be disastrous for their own business.
ksec 3y ago
Github is doing some as well. Although with resources from Microsoft I do wish they do more.
lagniappe 3y ago
You're going to incur the wrath for saying the silent part, but you're not wrong. People have the same 5 rails example corps every time someone says one of these two things: - without rails nobody would use ruby - without X corp, ruby's dead Fact is, we're seeing less and less usage, and more and more distillation of the current userbase along the golden paths laid out by DHH. Is it wrong for rails and thus ruby adoption to slow down? Not at all, people should use what they like, however I think Rails and Ruby are in this negative spiral where: - rails is mostly needed for prototyping and crud apps - this work is typically done by juniors - rails devs are at this point largely seniors, not juniors - rails devs pay the bills with other tech or by maintaining legacy rails apps. There are startlingly few deviations from this, and either everyone majors in rails with a minor in javascript and C++ or they just get happy with their current gig and settle. I wish it weren't the case, but Ruby just hasn't done enough to differentiate itself from Rails, and when compared to neo-PHP or JS there's just not a lot of attractive parts of the golden path Rails provides. It's off-tune for this generation of choice and ubiquity. We don't need that level of scaffolding anymore and in many cases there are other tools that handle that with more versatility.
- rgrieselhuber 3y ago
  I mentally put Ruby / Rails and Python with Flask / Django (and also “neo” PHP) in the same category vs all of the 70 million JS frameworks. When you talk about alternatives, what are the major ones you have in mind?
- toolz 3y ago
  Are you suggesting php and JS are stealing from what would've otherwise been ruby projects? I've been a ruby dev for a decade and I've never heard of a ruby shop migrating to using php for new work and maybe one or two moving to JS for new work. It's golang and elixir that are taking the place of ruby for new work in my experience. I suspect some python too, but I haven't seen that.
  - runjake 3y ago
    I think they're suggesting that neo-PHP and JS adopted the best parts and patterns of Rails (eg. Laravel).
hardwaregeek 3y ago
This is wholly irrelevant, but I love the spelling of O'Caml like an Irish last name. I'd definitely frequent a pub called O'Caml's.
- noelwelsh 3y ago
  OCaml was originally a contraction of Objective Caml, and spelled O'Caml. At some point this changed and the spelling without the apostrophe was adopted. What comes out of my fingers hasn't caught up, though.
- jjgreen 3y ago
  Given the early logo for the language was Joe Camel, it would need a smoking room ...
brightball 3y ago
I don't know why, there are tons of smaller companies using it. I have a lot of languages and experience on my resume and the one that consistently gets me the most inquiries at the highest pay grades is still Ruby. Anecdotal I know, but from the moment that it appeared on my resume in 2012 it's been non-stop. Probably 80% of everything I hear about. Ruby and it's ecosystem brings the closest thing to natural Aspect Oriented Programming that I've seen in the wild, which is why it's so much more productive than everything else I've tried.
- clairity 3y ago
  i think if rails continues to push hotwire (turbo + stimulus, and perhaps strada?) and get a coherent story on view components, it will continue to take mindshare from the js hype of the last decade. mobile dev is in decline, and browser makers just released web notifications, web app support, webtransport, page transitions, etc., so the backend has largely reached parity for cross-platform development. no longer is json the natural data exchange medium for apps, but rather chucks of html that can be plopped right into the dom without js having to massage the response into shape on the frontend. js can return to being a frontend scripting language, its natural habitat, rather than being shoehorned into being a do-it-all platform language.
  - d3nj4l 3y ago
    I've been building a webapp with Rails on and off over weekends. Several times over this process, I thought through some of the architectural decisions and naturally realized that the "Rails way" was the best option to pursue. It's not just because I'm using Rails - my most recent webdev experience was with a SPA driven by a Java backend. I'm sure there are tradeoffs involved (what doesn't?) but with every passing day I use Rails the more I appreciate the decisions it makes for you.
    - graypegg 3y ago
      I’ve had the exact same experience! I’m getting to use some new JS SSR frameworks at work (Remix) but I keep using rails for my own things. Gets out of my way.
deleted 3y ago
[deleted]
kjghkjghkjgh 3y ago
This is spoken like someone who knows nothing about Ruby.
riffraff 3y ago
stripe seemed to be doing quite a few interesting things (e.g. Sorbet, and there was some work adopting TUF for rubygems IIRC) but it seems to have dialled down things a bit.
ljm 3y ago
I think ‘is Ruby dying?’ is little more than a meme that has stuck around for longer than it deserves. The continuing work on the language and its performance is impressive, but Ruby (and Rails) themselves have the honour of being stable, tried-and-tested solutions for rapid application development. Is it as exciting as the latest and greatest serverless lambda framework in Typescript? Not really. Is it a dependable workhorse? Absolutely. At some point you might be successful enough to justify a rewrite into something else, but a simple Rails app will take you a long way with little effort.
- sergiotapia 3y ago
  I used to get invites for jobs and see listings for jobs for Ruby and Rails a lot. Now I don't see any at all. There's most definitely a "dead" feeling to the platform. I hardly hear about new projects being started with Rails as well...
  - scarnz 3y ago
    My co-founder and I are using RoR at our new company! It's definitely not dead and allows you to build SAAS very quickly.
  - 1123581321 3y ago
    It might just be the economy. I was getting about 20 inquiries a week from Rails positions until around December 2022. It's used in many companies, new and legacy.
    - fknorangesite 3y ago
      Yeah same story here. Until about 6-7 months ago I was hearing from recruiters for Rails jobs daily, both old and new projects. I'd be very surprised if the decrease since then is out of line with the rest of the industry, regardless of tech stack.
  - ljm 3y ago
    Probably depends where you’re based. Ruby is still lucrative in London and there is a healthy market for it both in startups and more established businesses. While I’ve branches out to other languages (not just JS as a full stack engineer) my career is still boosted by my Ruby experience. I don’t think this makes it dead or dying though. It’s stable and entrenched while JS has taken the place of the golden child. One complaint I’ll grant myself is that library development is a little less prolific this days. Again, there are well-established solutions to a lot of problems in Ruby so you’ll have a go-to collection of gems, but it’s more often the case these days that something doesn’t have much library support and you’ve got to roll your own.
sam0x17 3y ago
People forget Ruby is also massively popular in Japan to this day
- ramesh31 3y ago
  >People forget Ruby is also massively popular in Japan to this day So are fax machines.
johnisom2001 3y ago
This couldn't be further from the truth. There are so many people using Ruby, so many modern companies deciding to use it and so many large companies that continue to use it. The demand for Ruby developers is higher than the supply.
- deleted 3y ago
  [deleted]
- ksec 3y ago
  Your parent is talking about investing in Ruby the language and its ecosystem. Not just using it. The new GC, and JIT, along with a few Ruby Cores are all Shopify employees.
paulddraper 3y ago
Github: "Am I a joke to you?"

SushiHippie 3y ago

Somewhat fitting xkcd: https://xkcd.com/927/ https://xkcd.com/927/

revscat 3y ago
While this xkcd is frequently germane, in this case it is not particularly apt. The core CRuby team, who maintain the current “standard”, are in agreement that YARP — the new “standard” — will one day replace it.
- Dylan16807 3y ago
  Even putting the word "standard" in quotes is buying too much into the argument. This is a new implementation of an existing standard.
wiseowise 3y ago
It’s not fitting at all. > We recently got approval to merge this work into CRuby, and are very excited to share our work with the community.

soulbadguy 3y ago

I love how pragmatic the approach is here. We still don't have parser generators which generate human understandable/readable recursive descent parser with good error recovery baked in. But i am guessing the ruby syntax is too complicated/irregular anyway. On a macro side, i am always surprised with the length and effort company will go to keep working with solution design and selected at their young age, and try to scale them well beyond the braking point. And i feel like most of this effort to keep ruby alive could have been used to slowly transition part of the infra to something JVM/.Net/Rust based for much more bang for engineering buck.

ecshafer 3y ago
(Disclaimer: Shopify employee but not part of this team) Ruby is my favorite language so far to write "business logic" in, and Rails is a very developer friendly web framework that makes it very easy to set up websites. So I think this is a pretty great strategy overall. The end of the day I think the way to get tech too scale, is for companies / academics / etc to really just put the work in. Java was very slow on release, but now the JVM is blazingly fast, and that is because of the huge amount of work put in by Sun, Oracle, IBM, Redhat, Google, Azul, etc. over the years. For what its worth though, we do also have Go, Rust, Scala, etc. in different places in our stack where its needed.
- soulbadguy 3y ago
  Disclaimer : my career has mainly been on compiler/runtime and other infrastructure related stuff, so not much of a business logic/web dev kind of guy. > The end of the day I think the way to get tech too scale, is for companies / academics / etc to really just put the work in. This is true in the abstract, but my point is mainly about the comparative effectiveness of different strategies when it come to building scalable infrastructure (or scaling up infra): Keep the bulk of the existing code and invest in making the underlying compiler/interpreter/tool chain better or progressively migrate the code to a tool chain with better scaling capabilities from the get go. From experience, the nature and semantic of a language severely what is "reasonably" possible in the runtime in term of safety (like code loader in java), performance (both CPU and memory) and tooling. Now improvement are possible, but they tend become exponentially expensive as time goes on. Now, obviously there is always a tension between developer productivity and infra concerns, but i do believe that we have better compromise point on that line with newer language and framework. I totally agree that in the 2000's the experience of most "enterprise framework" sucked very hard, and the emergence of language such ruby/python etc... was a god send : a response to the overly rigid and ceremonial way of the past. But with time, we have been able to understand better what makes a good programming experience, distill that into better designed languages and framework which offer "better" compromises. For example : - Instead of dynamic vs static type , we have progressively typed and type inference - Instead of GC vs non-GC we have rust borrow-checker - Instead of runtime meta programing with have DSL and macro's Even more important, i believe that a lot of the experience come from the tooling around the language, and there again the "cargo/dotnet/go" cli approach with a single coherent entry point for both package management and framework scaffolding ease a lot of the pain of the old way. With all of that we now have languages which offer a better compromise on the dev. prod vs infra/performance... > JVM is blazingly fast I would say blazingly faster... But compared to C++ (or even rust) java is still quite slow. Especially for anything compute intensive.

jrochkind1 3y ago

As a devoted rubyist, it seems clear to me in retrospect that ruby's "pretty complex to parse is just fine" approach has been a real challenge to the language's success. For reasons related to discussion in OP. This project could be one approach to ameliorating that challenge, I'm very pleased to see shopify willing to invest in this, and hope it ends up effecting the ecosystem as hoped.

munificent 3y ago
I disagree. There are many languages that are hard to parse that are very successful. C and C++ are the obvious examples: there are many dark corners around the preprocessor, type annotations, etc. C++ is an absolute nightmare to parse. C# is pretty complex too. Python has its weirdness around indentation. Honestly, most popular languages end up acquiring a decent amount of grammatical complexity over the years and it doesn't seem to significantly hinder adoption. Humans are quite good at reading complex text and syntax. I think the real tax on Ruby is its pervasive use of runtime metaprogramming. It's Ruby's most exciting strength and enables much of the joy and excitement that Ruby is known for. But it makes static analysis so hard and becomes less and less valuable as team and codebase size increases.
- nine_k 3y ago
  I'd say that C and C++ were successful despite their being hard to parse, and other unfortunate hacks, like null-terminated strings and include files instead if modules There was little choice for other features they offered (like Unix being written in C and bundling a C compiler) which.were the compelling reason to use them. JavaScript was a huge success not because it was a great language, but because everyone wanted to write for the browser.
  - munificent 3y ago
    I think you and I are saying essentially the same thing: grammatical complexity seems to have little negative impact on language success. > There was little choice for other features they offered (like Unix being written in C and bundling a C compiler) which.were the compelling reason to use them. Pascal was around at the same time, is grammatically infinitely more elegant... and is dead.

e4m2 3y ago

Interestingly, the actual syntax tree and related structures/functions are generated from the config.yml file and the templates inside the bin directory. They are using a custom template language written in Ruby, here's an example of how they do enum stringification: https://github.com/ruby/yarp/blob/main/bin/templates/src/token_type.c.erb https://github.com/ruby/yarp/blob/main/bin/templates/src/tok.... Obviously this isn't a novel idea, but IMO this kind of design goes a long way to support their maintainability argument, especially in C. Also, > CRuby actually ships with 90 encodings (as of 3.3) This is asinine.

kddeisz 3y ago
The templating is definitely an effort to keep the maintainability in check. It's also because we're planning on keeping a grammar around to generate test cases for us with fuzzing/other algorithms. The encodings is a bit historical, to my knowledge Ruby is the only major language that was developed by someone whose language did not work with ASCII. So the first encodings written in Ruby were the old Windows pages.
cremno 3y ago
>> CRuby actually ships with 90 encodings (as of 3.3) >This is asinine. Overall there are even more: 103. But then again 1.9.2 only has 85 and 95 overall. Also one of those new 'overall' ones is the EBCDIC code page for US/Canada.
munificent 3y ago
> Interestingly, the actual syntax tree and related structures/functions are generated from the config.yml file and the templates inside the bin directory. That makes a lot of sense. ASTs tend to be a very dumb but fairly verbose data structure. And, in C in particular where everything is more verbose, there ends up being a ton of boilerplate in the AST nodes. In a language like Ruby with a very rich syntax, you end up needed a ton of different AST nodes. Generating those from a simple declarative format makes maintaining that much easier.
vidarh 3y ago
> a custom template language written in Ruby This makes it sound like it's some template language specific to this project, but ERB is the dominant templating language for Ruby because an ERB implementation is in the standard library.

davexunit 3y ago

I think Ruby is a pretty good language and I spent many years using it professionally but I can't imagine having enough motivation to work on the language implementation when it's so difficult to even parse the code. The amount of engineering time spent on this topic is bonkers!

kddeisz 3y ago
Maybe I'm a masochist, but to be honest it's a dream job. Working on such a complex language and history is really really fun.

chubot 3y ago

Nice article! In this sentence ... open parenthesis character is ambiguous in this context. To get around it they made their grammar more ambiguous and then enforced that the actual grammar was enforced in their tree builder. I'd change the second "more ambiguous" to "more lenient" i.e. lenient meaning "grammar accepts more strings", ambiguous meaning "grammar is invalid" I have seen this issue in Python's grammar as well, and it's mitigated by the new PEG parser

awestroke 3y ago
Ambigious does not mean invalid
- chubot 3y ago
  Yes, technically it means "there's more than one derivation" But Python's pgen rejects such grammars as invalid ... The other strategy is just to pick an arbitrary interpretation
- hinkley 3y ago
  For people who see parsing as model checking, ambiguity means something vastly different (inferior) to leniency. Fast parsers that use a schema are also doing model checking.

Alifatisk 3y ago

What a impressive work & effort! Well done.

cout 3y ago

With the mention of the parser's performance, I am reminded of Rich Kilmer's 2004 RubyConf presentation. Rich used Ruby to test reliability of a distributed system with hundreds of Java VMs. Some of the serialized data was stored as XML (over 1M lines!), which was slow to parse and load. Rich modified the program to serialize the data as Ruby code, which loaded much faster (https://www.infoq.com/news/2007/06/infoq-interview-rich-kilmer/ https://www.infoq.com/news/2007/06/infoq-interview-rich-kilm...): > Chad Fowler and I over basically 2 weeks, took the Java Debug Wire Protocol specification [...] turned it into a DSL in Ruby, and used that DSL to generate the packets for sending and receiving data. So we used the DSL in Ruby as a generator to generate Ruby code, as the whole protocol and then I used that at Darpa. They were trying to say “we could freeze the agent society from within the agent society will send messages, and it took about 7-8 minutes for all the messages to propagate and everything to freeze and then go quiet. I had a Ruby process that was running all 300 VM’s were underneath it, and I could freeze it in about a half of second. All 300 of them! And you actually could watch the CPU use because we had a monitor and the CPU use has dropped to zero. And it freaked them out. And what was great was you could turn it back on, and all the agents came back on. But time had been lost. And it was like alien abduction lost time, ten minutes went away, “what happened to us?” It was a bizarre thing because they were agents and they were planning on things and all of sudden 10 minutes just went away. But it was interesting to show how Ruby could actually be used as this kind of harnesses to wrap around things like systems.

srgpqt 3y ago
Nice cover story to hide your ~totally existing~ time machine ;)
RhodesianHunter 3y ago
>Some of the serialized data was stored as XML (over 1M lines!), which was slow to parse and load. Rich modified the program to serialize the data as Ruby code, which loaded much faster So he took a data format designed for human readability and converted it to a data format that's designed purely to be read by Ruby and people are surprised that it's faster?
- 3y ago

fareesh 3y ago

Will this change impact Solargraph / Rubocop? They are painfully slow / unusable in their current state on large-ish projects

kddeisz 3y ago
Depends on what you mean by Solargraph, because it's kind of a large project. If you mean their typechecking, then definitely not, because it's not using a Ruby parser. If you mean the general feedback, then yeah potentially. Rubocop yes if it ends up using YARP as a new backend. Either way, I would suggest you check out ruby-lsp, which is definitely going to benefit from this speed, and soon.
pianoben 3y ago
The article mentions that they are building a compatibility layer around YARP that tools can use to transform its new tree format into the legacy Ripper format. They don't call out Rubocop specifically but I can't think of another OSS tool that so prominently uses the parser APIs.
- kddeisz 3y ago
  rubocop uses rubocop-ast uses parser, so it will eventually make its way down to rubocop once we finish the compat layer for the parser gem.
dabears 3y ago
I made a Ruby LSP because of this problem. It's not perfect but incase it's helpful for you. It can parse a large project with all of its gems in a few minutes. That data is indexed in an in-memory db with Tantivy. https://github.com/pheen/fuzzy_ruby_server https://github.com/pheen/fuzzy_ruby_server

fareesh 3y ago

I use ruby/rails for professional consulting work. It helps get startups off the ground and into a MVP stage quite quickly, and there is high productivity. You can also use whatever React/Vue on the frontend if you really need to. I would like to use some Typescript framework but none are "there yet" with regard to productivity. Scaling rails in some scenarios is quite challenging, but in most cases you can leverage caching to solve performance challenges.

justanother_guy 3y ago
Nest.Js seems to be promising, but like you said, it has a long way to go.
weaksauce 3y ago
with the improvements to hotwire you might not even need to get to a heavy frontend like react

fireweed 3y ago

[dead]

soulbadguy 3y ago

While on the subject, anybody has good reference on parsing error recovery ?

di4na 3y ago
If you find it, i am interested. But it seems to be a topic that is still open to research and niche to a small group of devs
- soulbadguy 3y ago
  I suspect that most of the knowledge on this topic is embedded in source code of most prominent compiler tool chain and the head of their dev. :( I think they might an interesting intersection here we ML, where can could learn the comment mistake pattern made by real user and either error correct better, or at least provide pin point accurate error messages.
  - di4na 3y ago
    Not really no.

johnisom2001 3y ago

This is big. Ruby syntax errors are some of the most frustrating things to track down. So much worse than C-like languages like JavaScript. I'm glad to see big steps towards this.

progitter 3y ago

Well described. Also I found this on github. https://github.com/whitequark/parser https://github.com/whitequark/parser

vidarh 3y ago
That is one of the many parsers listed in the article.

freedomben 3y ago

It's been really great watching Ruby over the last few years. I had the privilege of having dinner with Matz several years ago near the beginning of Ruby 3.0 work. I was giving a talk about Elixir at a Ruby conference, and he was interested in things I (as a self-professed fanboy of both languages) liked better about Elixir than Ruby. We talked for quite some time, and it became very clear to me that he knew that Ruby was stagnating and wanted to make some changes to keep it relevant (without wrecking the language). He was incredibly open-minded (and nice!) and was willing to listen to anyone who had things to say. Flash forward several years, and the amount of changes in Ruby are huge! Ruby is as fit as ever for modern development, and I'm really happy about that.

ljm 3y ago
Pattern matching alone is a huge feature and an absolute delight to use when the opportunity presents itself. I’d love to see where Ractor goes but I worry it will remain niche, like with Refinements.
- freedomben 3y ago
  Agreed. Pattern matching and pipe operator were the two that I hoped for most. The pipe operator implementation proposal that they team came up with was wrong though and I'm glad they didn't do it. We don't need alternate syntax for `.`, we need ability to chain arguments together in a syntactically pleasant way in a functional style so we don't have to write `first(second(third(arg)))` we can write `arg |> third() |> second() |> first` which is much cleaner and reads left to right like it should
  - zverok 3y ago
    Ruby-idiomatic pipe operator is Object#then. After a lot of design proposals and discussions it is more or less evident no solution other than method would integrate naturally with the rest of the code. So it is just `arg.then{ third(_1)}.then{ second(_1)}.then{ first(_1)}` Would've been a bit more concise with method references, almost introduced in 2.7, but alas. (But, well, people tend to want "something that looks like operator, preferably something that looks exactly like |>" and reject every other possibility)

PaulHoule 3y ago

It’s little appreciated how much current parser generators are holding the industry back. That is, LISP maintains a lead in metaprogramming because it bypasses Chomskyism alltogether. It really should be a few lines of code to add an “unless(X) {}” statement to “if(!X) ()” to Java, Python, Ruby but the very idea that you could patch an existing grammar with a separate file is like technology that fell off a UFO. Also anything that you generate a parser for should automatically generate an unparser. Sphinx is a fraction of the framework it could be because it can’t process markup and turn it back to Sphinx. I think a PEG framework with a few features (like an easy way to implement operator precedence just by stating it with either numeric values or X > Y statements) could be revolutionary. Python is almost there but not quite.

barrkel 3y ago
Custom grammars are a cool idea. They are terrible in practice and one of the biggest reasons why Lisp failed to take off, why it's almost noone's first choice of language for building something big. The reason is that new grammar creates a new language, and the divergence of the new language from the base language creates a cultural barrier that inhibits communication. It's harder for new engineers to be productive, and it's harder to collaborate. Customizing your grammar works best if you're a lone wolf, a one man band. You can lever up your productivity by custom-designing something perfectly suited to both you and your preferred problem domain. But nobody else will understand it.
G3rn0ti 3y ago
If you’re into language parsing and how its patterns could be used in a modern programming language you should take look on Raku (fka Perl 6): It has native support for „Grammars“ [1] which are basically specialized classes to put Regexes in for defining tokens and also combinations of such tokens. Once you‘re done you can use a grammar object to parse text returning an AST object. Raku‘s own syntax is defined in Grammars being a subset of Raku‘s syntax. So in a nutshell Raku‘s „Grammar“ construct is like RegExes on steroids and renders defining DSLs or other special purpose languages so much easier than in any other modern language. [1] https://docs.raku.org/language/grammars https://docs.raku.org/language/grammars
3y ago

IshKebab 3y ago

People always say language choice doesn't matter, but how many companies get stuck with a language where they resort to rewriting the parser for it?

nirvdrum 3y ago
No one is "stuck" or "resorting" to anything. It's an effort to improve tooling, performance, error reporting, and working with Ruby in general. We could maintain the status quo (which does indeed work) or we could improve it. If we don't improve the status quo, people call the language stagnant and outdated. If we do, it's considered a criticism of its maturity. How many languages would even entertain such a contribution? How many languages are so tightly bound to their parser that writing a new one wouldn't even be workable? How many languages have user tooling that works extensively with parsed code snippets? How many languages have multiple implementations? Of those, how many are working together on an effort to share a common parser? There's plenty of valid reasons to use something other than Ruby. An open source project that grew organically over 25 years deciding that it's time to pay down some technical debt and improve the ecosystem at the same time is probably not one of them.

nusaru 3y ago

> This is the final step before merging YARP into CRuby, and we’re very excited to see it come to fruition. This work will be done in the next couple of work days. It continues to amaze me the pace at which kddnewton and co. are able to work. His tweet[1] from a few months ago is really relevant here. Being full-time employees and not just open-source volunteers makes a huge difference! [1]: https://twitter.com/kddnewton/status/1639258413120073730 https://twitter.com/kddnewton/status/1639258413120073730

tester756 3y ago

A lot of cool work! Congrats!

nsajko 3y ago

Tangential: > Over the years, processors and C compilers have gotten much better using a couple of techniques. These include pipelining, inlining functions, and branch prediction. Unfortunately, the parsers generated by most parser generators make it difficult for any of these techniques to apply. Most generated parsers operate with a combination of jump tables and gotos, rendering some of the more advanced optimization techniques impotent. Because of this, generated parsers have a maximum performance cliff that is extremely difficult to overcome without significant effort. Although generating parsers, and finite automata in general, using a table-based approach is common, it has long been recognized that using tables/data for this purpose (as opposed to generating executable source code directly) is not a good idea, precisely because it inhibits compiler optimizations. I think the current situation is simply a consequence of the fact that parsing abruptly stopped being sexy multiple decades ago. Much better parser generators are possible, and LR, specifically, has much untapped potential left.

ramesh31 3y ago

So Shopify spends millions of dollars in engineering salary to get a substandard performing language up to par with a dozen other contemporary better choices? The choice to stick so hard with Ruby just doesn't seem reasonable at all with their scale. It must be something dogmatic from the top.

agotterer 3y ago
Facebook did the same when they developed HipHop for PHP. I imagine the reasoning is the same in both cases: spend a few million dollars and a handful of engineers improving the language and tools, or spend hundreds of millions and years of time rewriting everything into a more performant language. I honestly think the math pencils and makes sense.

stevebmark 3y ago

"Rails at scale" is a funny title. Shopify had to rebuild the core parser. Github has a department dedicated to working off the bleeding edge of Rails (and has core Ruby and Rails maintainers on staff, Shopify probably does too). Both Shopify and Github spent literal engineering years upgrading Rails, and still for some reason think it's worthy of boasting about in engineering blog posts, despite the burnout, not-shipping-features, and turnover they suffered because of it. If your company has enough engineering overhead and capacity to stop delivering features and try to fix your core language for a year, then you're in a different ballpark from most other tech companies. If you're NOT in that boat, then "Rails scaled for Github and Shopify" is not a message you should take home.

vidarh 3y ago
If you're not in that boat, odds are you don't need to care. And they didn't stop delivering features for a year.

tiffanyh 3y ago

Hack? I wonder if Shopify will ever give up on trying to fix Ruby and instead, do what FB did with PHP and just create their own derivative (Ruby) language that addresses their needs better.

kayodelycaon 3y ago
Why? Shopify was able to work with the core team and make the whole language better. Facebook has far more employees and wasn’t going to get PHP improved to their liking.
- tiffanyh 3y ago
  The irony is that PHP has improved massively in perf vs the relative gains of Ruby. https://onlinephp.io/benchmarks https://onlinephp.io/benchmarks https://serpapi.com/blog/benchmarking-ruby-3-1-yjit-ruby-2-7/amp/ https://serpapi.com/blog/benchmarking-ruby-3-1-yjit-ruby-2-7...
  - vidarh 3y ago
    Apart from measuring very different things, the PHP page compares PHP releases over something like 25 years vs. about 3 years of Ruby releases. Look at PHP 7.2 onwards, which is a similar timeframe, and the improvements are fairly modest.

tiffanyh 3y ago

GraalVM? What’s the progress on Ruby/GraalVM that Shopify has heavily invested in?

ximus 3y ago
GraalVM Ruby, more commonly known as TruffleRuby, is actively developed still. As the article states, it adopted YARP already.
- tiffanyh 3y ago
  My understanding is that Shopify still hasn’t been able to migrate to Graal/TruffleRuby, even after 10-years of its development.
  - nirvdrum 3y ago
    Your phrasing could be misconstrued by those not familiar with the history. Shopify hasn't been developing TruffleRuby for 10 years. The entire TruffleRuby project is 10 years old, including the first 12 - 18 months where it was a humble intern project. TruffleRuby (previously called JRuby+Truffle) was initially a research project for testing out optimizations, not a production-quality implementation suitable for deployment. I think it's now a viable deployment target, but that wasn't true 10 years ago. As you might imagine, there's a lot more to production deployments than simple Ruby compatibility. At Shopify, we've recently modified our CI system to support TruffleRuby and are actively running projects against TruffleRuby in CI now [1]. The TruffleRuby 23.0.0 release coming out in the next day or so includes quite a few compatibility issues we've worked out getting a large Rails app booting. That project is on the order of months, not decades. YARP will make adoption of TruffleRuby easier. Absent a language specification, implementations like JRuby and TruffleRuby have to match what CRuby is shipping and that by necessity means they lag after CRuby releases. Parser changes are amongst the hardest things to port over. YARP eliminates most of the challenges there. [1] -- https://railsatscale.com/2023-06-12-truffleruby-in-shopify-ci/ https://railsatscale.com/2023-06-12-truffleruby-in-shopify-c... Disclaimer: I'm on the TruffleRuby team at Shopify.

thiht 3y ago

Great read. I’ve never written a single line of code in Ruby but I love reading stuff related to compilers.

mingodad 3y ago

It's not true "the best chance you have is reading the 14 thousand-line parse.y file and trying to understand it", there are several tools to navigate yacc/bnf style grammars like https://www.bottlecaps.de/convert/ https://www.bottlecaps.de/convert/ and it's companion https://www.bottlecaps.de/convert/ https://www.bottlecaps.de/convert/ that make relatively easy to understand/document/debug/compare the grammar. Just added several ruby grammars here https://github.com/mingodad/plgh/tree/main/ruby https://github.com/mingodad/plgh/tree/main/ruby they are converted (mainly using https://www.bottlecaps.de/convert/ https://www.bottlecaps.de/convert/) to an EBNF understood by https://bottlecaps.de/rr/ui https://bottlecaps.de/rr/ui to generate navigable railroad diagrams. Copy and paste the EBNF on https://www.bottlecaps.de/rr/ui https://www.bottlecaps.de/rr/ui on the tab "Edit Grammar" the click on the tab "View Diagram" to see/download a navigable railroad diagram.