7 ms·

Rebuilding the Racket Compiler with Chez Scheme

106 points by AlexeyBrin 6y ago

bloopernova 6y ago
Very cool that HackerNews is using "their own language called Arc that is programmed in Racket." http://arclanguage.org/ http://arclanguage.org/ I had no idea I was using Racket already, via this website! Makes me wonder if Terraform's HCL could be re-written using Racket.
- opnitro 6y ago
  I'm sure they could (don't know they if should). If you'd like to learn more about Racket's language building infrastructure, check out: https://beautifulracket.com/ https://beautifulracket.com/
jhoechtl 6y ago
What are the reasons Racket CS is, or at least used to be slower than Racket BS? Was the whole transition towards CS not primaritly to boost performance? Were expectations to high in this regard?
- tux1968 6y ago
  They took the opportunity to move some of the infrastructure from C to Racket itself: Mostly, we did reimplement the C stuff in Racket. The I/O subsystem, the concurrency subsystem (which includes the scheduler for “green” threads, Concurrent ML-style events, and custodians), and the regexp matcher were all rewritten in Racket. Those pieces followed the rewrite of the macro expander in Racket. So there was a slowdown until those pieces matured. But now they get the benefit of having all those pieces in higher level of abstraction and more accessible to Racket programmers.
- samth 6y ago
  1. The big performance issues early on were compilation speed -- Racket had a simple JIT compiler that ran almost instantly. 2. Some rewritten parts of the system, such as IO, were initially slower, but have now been improved to catch up to the previous implementation. 3. Chez Scheme didn't support some performance features that Racket BC had, such as unboxed floating point arithmetic, parallel GC, or continuation marks. Those have been added to (Racket's variant of) Chez Scheme. 4. Racket's prior compiler was somewhat more competitive with Chez Scheme than we expected, particularly for larger and more Racket-idiomatic programs.
- dreamcompiler 6y ago
  Flatt has written elsewhere that the motivation to use Chez was not speed per se but better maintainability and portability, and that required rewriting more of the C core in Racket itself [0]. Doing so would have slowed down all of Racket considerably, but by using Chez it became possible without a big speed penalty. At first there was such a penalty because the rewritten stuff was still slower than C, but with tuning that has mostly gone away. [0] https://blog.racket-lang.org/2018/01/racket-on-chez-status.html https://blog.racket-lang.org/2018/01/racket-on-chez-status.h...
- ywei3410 6y ago
  To add to the other reasons - the CS compiler optimizes different forms and Racket is currently optimized for it's own compiler. To give a concrete example, taken directly from the racket blog [1]: > For example, Racket BC makes dormant code relatively cheap, so Racket libraries generate a lot of code. Future Racket libraries will likely shift to take advantage of Racket CS’s cheaper function calls and dramatically cheaper continuations. One day, probably, Racket BC will no longer be a viable alternative to Racket CS for most programs. [1] https://blog.racket-lang.org/2020/02/racket-on-chez-status.html https://blog.racket-lang.org/2020/02/racket-on-chez-status.h...
pmoriarty 6y ago
"Chez Scheme is one of the oldest Scheme implementations, and its evolution informed many parts of the Scheme standard through R6RS. (Racket's influence on the Scheme standard, in contrast, is limited to aspects of the R6RS library design.)..." Why did Chez and Racket embrace R6RS, a standard widely rejected by the rest of the Scheme community? Also, why did Racket choose Chez over some other high-performance Schemes, like Chicken? Finally, how many Racket and Chez users are there compared to those of other Schemes?
- rscho 6y ago
  Isn't chicken actually slower than Racket was before the switch? IMO its strong point is more seamless CFFI than performance. Chez also has a small C core and the rest in scheme, which I think was the main motive for the switch.
- yakubin 6y ago
  > Why did Chez and Racket embrace R6RS, a standard widely rejected by the rest of the Scheme community? Why wouldn't they? The usual rationale for rejecting R6RS is that it's somehow "too big". But Racket already has a library whose size blows R6RS out of the water. So this rationale is irrelevant in this context. Also, Chez Scheme is an implementation detail. R6RS is only one of many languages that you may use through Racket. And what is this "Scheme community" that rejected R6RS? It's always been my impression that it was rejected by authors of toy implementations. Is there actually any data on what portion of people who use Scheme (as opposed to write another implementation of it) reject R6RS?
  - bitwize 6y ago
    > Why wouldn't they? The usual rationale for rejecting R6RS is that it's somehow "too big". But Racket already has a library whose size blows R6RS out of the water. So this rationale is irrelevant in this context. No one begrudges Racket for having a huge library. The Racket team have obviously had enormous success building large, comprehensive real-world systems. But Racket is an implementation -- not the Scheme standard. The historic value of the Scheme standard is its smallness and ease of implementation -- a small base upon which to build large systems. Large, comprehensive implementations -- such as T -- have been built on top of this small basis for almost as long as there's been an RnRS. R6RS represented a fundamental shift in philosophy for what is Scheme. It de-emphasized the "small language kernel" approach and emphasized a comprehensive, software-engineering approach for building "real world systems". But the Lisp community already has a comprehensive, software-engineering-oriented language, Common Lisp (which borrowed some of its best features, like lexical scoping, from Scheme). So what's the point of building Scheme into a large software-engineering language to compete with Common Lisp? It's a bit hard to fathom today, when most modern programming languages -- excluding ECMAScript but including, for example, Rust -- are defined in terms of a single reference implementation, but there are advantages to keeping the language standard small while allowing implementations to grow to arbitrary size and accrete features. Advantages which Scheme has turned into its own little niche among the family of Lisp programming languages. > And what is this "Scheme community" that rejected R6RS? It's always been my impression that it was rejected by authors of toy implementations. Of course, because Gambit, Chicken, and Gauche -- among others -- are mere toys. A toy implementation that implements all of R5RS is at least possible. You can't build a toy implementation of Common Lisp, Python, or Rust, that covers even a small fraction of the core language. But you can build a Scheme that's nearly or entirely R5RS-compliant in maybe a few thousand lines of C, and from there build out the language to incorporate Python- or Rust-like features. "Toy" Schemes can even be useful in production, as an embedded scripting layer to a larger system.
patterns 6y ago
I wonder whether the migration to Chez Scheme will improve the performance of Dr. Racket which consumes a lot of memory and feels very sluggish. I have been experimenting with Racket for quite a while and I appreciate the effort that went into making the language extensible. That being said, I wish that the community would have embraced "object-oriented" techniques for building the VM - Ian Piumarta's COLA (Combined Object-Lambda Architecture) comes to mind [1]. I think this would have saved them a lot of performance troubles (but this is mere speculation) and would make the system far more flexible and pleasant to use. While it's comforting to have a very powerful macro system at your fingertips to change the semantics of the language when needed, many of the macros in the base language and libraries could be eliminated with a more convenient default notation for closures and message-passing (to objects simulating control structures like in Smalltalk). Racket has one of the best module systems I have worked with but modules cannot be (easily) parameterized. Research on systems such as Self and more recently Newspeak [2] gives ample of demonstration of the benefits (conceptual integrity, security) of having modules as objects. What Racket also lacks is a good meta-object system with highly reflective capabilities (of say a CLOS or Smalltalk system). This makes it difficult to build tooling such as inspectors or browsers. I hope that in the future these issues will be addressed. [1] https://www.piumarta.com/papers/colas-whitepaper.pdf https://www.piumarta.com/papers/colas-whitepaper.pdf [2] https://newspeaklanguage.org/ https://newspeaklanguage.org/
- rscho 6y ago
  I have no CL experience but doesn't a CLOS system require multiple dispatch?
  - Pet_Ant 6y ago
    With macros you can easily build your own multiple dispatch, it's really not a concern.
    - rscho 6y ago
      Well, there's at least the concern that multiple dispatch results in unsafe edge cases across module boundaries, apparently. So, multiple dispatch as a library yes (and racket has that), but probably not ok in the core language?
Janet29 6y ago
In my opinion, Dr Racket is the most bloated and most sluggish programming editor ever written. And the editor on windows constantly display strange unwanted lines (see for example this screenshot [1]. I had complained numerous times to Racket's news group because of this, but this has never been fixed. Very strange community, very poor tool. [1] https://pasteboard.co/JCkb0tM.png https://pasteboard.co/JCkb0tM.png
- deleted 6y ago
  [deleted]
- mitch_br 6y ago
  My experience with Dr Racket is the same.
- andrewflnr 6y ago
  That looks like it's working as intended, to mark the end of the function. The lines from a symbol to its definition look even weirder, but I can see the appeal. A setting to toggle them would be reasonable, but they're not going to get rid of them, if that's what you asked for.
- shakna 6y ago
  That looks like underline has been enabled for parenthesis. Dr. Racket is _highly_ configurable. I believe you'll find the option you're looking for at: Preferences->Colors->Racket->Parenthesis (Underline)
- nyanpasu64 6y ago
  I think the issue is because you're running on 125%-ish DPI scaling, and the program doesn't handle it well.
  - Janet29 6y ago
    Yes, I do have 125% DPI on my computer. When I set back to 100% DPI than strange lines doesn't show. But I want to keep 125% DPI and the point is that Dr Racket should respect that. But, Dr Racket is the only program that display strange artifacts on 125% DPI. Neither VS Code nor Sublime nor Notepad++ do that. As I already mentioned, I reported this bug many times during past few years to the Racket group, but the damn lines are still there, and that made the editor totally unusable. What should I say? Strange community, they don't care for this serious bug, they are preocupied with the wrong stuff of "speeding" the language with Chez Scheme. The result is crappy, unusable tool and angry users.

106 points by AlexeyBrin 6y ago

Very cool that HackerNews is using "their own language called Arc that is programmed in Racket." http://arclanguage.org/ http://arclanguage.org/ I had no idea I was using Racket already, via this website! Makes me wonder if Terraform's HCL could be re-written using Racket.

opnitro 6y ago
I'm sure they could (don't know they if should). If you'd like to learn more about Racket's language building infrastructure, check out: https://beautifulracket.com/ https://beautifulracket.com/

jhoechtl 6y ago

What are the reasons Racket CS is, or at least used to be slower than Racket BS? Was the whole transition towards CS not primaritly to boost performance? Were expectations to high in this regard?

tux1968 6y ago
They took the opportunity to move some of the infrastructure from C to Racket itself: Mostly, we did reimplement the C stuff in Racket. The I/O subsystem, the concurrency subsystem (which includes the scheduler for “green” threads, Concurrent ML-style events, and custodians), and the regexp matcher were all rewritten in Racket. Those pieces followed the rewrite of the macro expander in Racket. So there was a slowdown until those pieces matured. But now they get the benefit of having all those pieces in higher level of abstraction and more accessible to Racket programmers.
samth 6y ago
1. The big performance issues early on were compilation speed -- Racket had a simple JIT compiler that ran almost instantly. 2. Some rewritten parts of the system, such as IO, were initially slower, but have now been improved to catch up to the previous implementation. 3. Chez Scheme didn't support some performance features that Racket BC had, such as unboxed floating point arithmetic, parallel GC, or continuation marks. Those have been added to (Racket's variant of) Chez Scheme. 4. Racket's prior compiler was somewhat more competitive with Chez Scheme than we expected, particularly for larger and more Racket-idiomatic programs.
dreamcompiler 6y ago
Flatt has written elsewhere that the motivation to use Chez was not speed per se but better maintainability and portability, and that required rewriting more of the C core in Racket itself [0]. Doing so would have slowed down all of Racket considerably, but by using Chez it became possible without a big speed penalty. At first there was such a penalty because the rewritten stuff was still slower than C, but with tuning that has mostly gone away. [0] https://blog.racket-lang.org/2018/01/racket-on-chez-status.html https://blog.racket-lang.org/2018/01/racket-on-chez-status.h...
ywei3410 6y ago
To add to the other reasons - the CS compiler optimizes different forms and Racket is currently optimized for it's own compiler. To give a concrete example, taken directly from the racket blog [1]: > For example, Racket BC makes dormant code relatively cheap, so Racket libraries generate a lot of code. Future Racket libraries will likely shift to take advantage of Racket CS’s cheaper function calls and dramatically cheaper continuations. One day, probably, Racket BC will no longer be a viable alternative to Racket CS for most programs. [1] https://blog.racket-lang.org/2020/02/racket-on-chez-status.html https://blog.racket-lang.org/2020/02/racket-on-chez-status.h...

pmoriarty 6y ago

"Chez Scheme is one of the oldest Scheme implementations, and its evolution informed many parts of the Scheme standard through R6RS. (Racket's influence on the Scheme standard, in contrast, is limited to aspects of the R6RS library design.)..." Why did Chez and Racket embrace R6RS, a standard widely rejected by the rest of the Scheme community? Also, why did Racket choose Chez over some other high-performance Schemes, like Chicken? Finally, how many Racket and Chez users are there compared to those of other Schemes?

rscho 6y ago
Isn't chicken actually slower than Racket was before the switch? IMO its strong point is more seamless CFFI than performance. Chez also has a small C core and the rest in scheme, which I think was the main motive for the switch.
yakubin 6y ago
> Why did Chez and Racket embrace R6RS, a standard widely rejected by the rest of the Scheme community? Why wouldn't they? The usual rationale for rejecting R6RS is that it's somehow "too big". But Racket already has a library whose size blows R6RS out of the water. So this rationale is irrelevant in this context. Also, Chez Scheme is an implementation detail. R6RS is only one of many languages that you may use through Racket. And what is this "Scheme community" that rejected R6RS? It's always been my impression that it was rejected by authors of toy implementations. Is there actually any data on what portion of people who use Scheme (as opposed to write another implementation of it) reject R6RS?
- bitwize 6y ago
  > Why wouldn't they? The usual rationale for rejecting R6RS is that it's somehow "too big". But Racket already has a library whose size blows R6RS out of the water. So this rationale is irrelevant in this context. No one begrudges Racket for having a huge library. The Racket team have obviously had enormous success building large, comprehensive real-world systems. But Racket is an implementation -- not the Scheme standard. The historic value of the Scheme standard is its smallness and ease of implementation -- a small base upon which to build large systems. Large, comprehensive implementations -- such as T -- have been built on top of this small basis for almost as long as there's been an RnRS. R6RS represented a fundamental shift in philosophy for what is Scheme. It de-emphasized the "small language kernel" approach and emphasized a comprehensive, software-engineering approach for building "real world systems". But the Lisp community already has a comprehensive, software-engineering-oriented language, Common Lisp (which borrowed some of its best features, like lexical scoping, from Scheme). So what's the point of building Scheme into a large software-engineering language to compete with Common Lisp? It's a bit hard to fathom today, when most modern programming languages -- excluding ECMAScript but including, for example, Rust -- are defined in terms of a single reference implementation, but there are advantages to keeping the language standard small while allowing implementations to grow to arbitrary size and accrete features. Advantages which Scheme has turned into its own little niche among the family of Lisp programming languages. > And what is this "Scheme community" that rejected R6RS? It's always been my impression that it was rejected by authors of toy implementations. Of course, because Gambit, Chicken, and Gauche -- among others -- are mere toys. A toy implementation that implements all of R5RS is at least possible. You can't build a toy implementation of Common Lisp, Python, or Rust, that covers even a small fraction of the core language. But you can build a Scheme that's nearly or entirely R5RS-compliant in maybe a few thousand lines of C, and from there build out the language to incorporate Python- or Rust-like features. "Toy" Schemes can even be useful in production, as an embedded scripting layer to a larger system.

patterns 6y ago

I wonder whether the migration to Chez Scheme will improve the performance of Dr. Racket which consumes a lot of memory and feels very sluggish. I have been experimenting with Racket for quite a while and I appreciate the effort that went into making the language extensible. That being said, I wish that the community would have embraced "object-oriented" techniques for building the VM - Ian Piumarta's COLA (Combined Object-Lambda Architecture) comes to mind [1]. I think this would have saved them a lot of performance troubles (but this is mere speculation) and would make the system far more flexible and pleasant to use. While it's comforting to have a very powerful macro system at your fingertips to change the semantics of the language when needed, many of the macros in the base language and libraries could be eliminated with a more convenient default notation for closures and message-passing (to objects simulating control structures like in Smalltalk). Racket has one of the best module systems I have worked with but modules cannot be (easily) parameterized. Research on systems such as Self and more recently Newspeak [2] gives ample of demonstration of the benefits (conceptual integrity, security) of having modules as objects. What Racket also lacks is a good meta-object system with highly reflective capabilities (of say a CLOS or Smalltalk system). This makes it difficult to build tooling such as inspectors or browsers. I hope that in the future these issues will be addressed. [1] https://www.piumarta.com/papers/colas-whitepaper.pdf https://www.piumarta.com/papers/colas-whitepaper.pdf [2] https://newspeaklanguage.org/ https://newspeaklanguage.org/

rscho 6y ago
I have no CL experience but doesn't a CLOS system require multiple dispatch?
- Pet_Ant 6y ago
  With macros you can easily build your own multiple dispatch, it's really not a concern.
  - rscho 6y ago
    Well, there's at least the concern that multiple dispatch results in unsafe edge cases across module boundaries, apparently. So, multiple dispatch as a library yes (and racket has that), but probably not ok in the core language?

Janet29 6y ago

In my opinion, Dr Racket is the most bloated and most sluggish programming editor ever written. And the editor on windows constantly display strange unwanted lines (see for example this screenshot [1]. I had complained numerous times to Racket's news group because of this, but this has never been fixed. Very strange community, very poor tool. [1] https://pasteboard.co/JCkb0tM.png https://pasteboard.co/JCkb0tM.png

deleted 6y ago
[deleted]
mitch_br 6y ago
My experience with Dr Racket is the same.
andrewflnr 6y ago
That looks like it's working as intended, to mark the end of the function. The lines from a symbol to its definition look even weirder, but I can see the appeal. A setting to toggle them would be reasonable, but they're not going to get rid of them, if that's what you asked for.
shakna 6y ago
That looks like underline has been enabled for parenthesis. Dr. Racket is _highly_ configurable. I believe you'll find the option you're looking for at: Preferences->Colors->Racket->Parenthesis (Underline)
nyanpasu64 6y ago
I think the issue is because you're running on 125%-ish DPI scaling, and the program doesn't handle it well.
- Janet29 6y ago
  Yes, I do have 125% DPI on my computer. When I set back to 100% DPI than strange lines doesn't show. But I want to keep 125% DPI and the point is that Dr Racket should respect that. But, Dr Racket is the only program that display strange artifacts on 125% DPI. Neither VS Code nor Sublime nor Notepad++ do that. As I already mentioned, I reported this bug many times during past few years to the Racket group, but the damn lines are still there, and that made the editor totally unusable. What should I say? Strange community, they don't care for this serious bug, they are preocupied with the wrong stuff of "speeding" the language with Chez Scheme. The result is crappy, unusable tool and angry users.