8 ms·

An update on Dart macros and data serialization

143 points by oltmang 2y ago

deleted 2y ago
[deleted]
eseidel 2y ago
Former Eng. Dir for Dart, co-founder of Flutter, here. I'd like to believe this is a good thing for the Dart project, but only time will tell. My hot take here: https://shorebird.dev/blog/dart-macros/ https://shorebird.dev/blog/dart-macros/
- bloqs 2y ago
  Thanks!
- elcritch 2y ago
  Thanks, good take. Especially if some of the pieces are still being added. In my humble opinion, you can handle many cases like serialization better with 'compileTime' or comptime features though I'm partial to macros. Especially with core compile time constructs like 'fields' [1, 2]. Though those require some abilities dart's compiler may not have or be able to do efficiently. That'd be a bummer, as even C++ is finally getting compile time reflection. 1: https://nim-lang.org/docs/iterators.html#fieldPairs.i%2CT https://nim-lang.org/docs/iterators.html#fieldPairs.i%2CT 2: https://www.openmymind.net/Basic-MetaProgramming-in-Zig/ https://www.openmymind.net/Basic-MetaProgramming-in-Zig/
ElliotH 2y ago
This is good news. The dart language has been getting more complicated without corresponding quality of life improvements. A first class record object without messing around with macros would be a great start.
- msie 2y ago
  Why is it that many languages, at the start, don't have support for records/plain structs?
  - refulgentis 2y ago
    Because it's arguably syntactic sugar and, IMHO, it's worked out better for developers for Dart to model it as a 3rd party library problem. i.e. have a JSONSerializable protocol, and enable easy code generation by libraries. i.e. I annotate my models with @freezed, which also affords you config of the N different things people differ on with json (are fields snake case? camel case? pascal case?) and if a new N+1 became critical, I could hack it in myself in 2-4 hours. I'm interested to see how this'd integrate with the language while affording the same functionality. Or maybe that's the point: it won't, but you can always continue using the 3rd party libraries. But now it's built into the language, so it is easier to get from 0 to 1.
  - kazinator 2y ago
    Because implementing them is tedious, and you can always simulate them with simpler aggregation methods, or possibly lexical closures. When the language implementors start making larger programs, it will soon become apparent how the program organization is hampered without named, defined data structures. I didn't add structs to TXR Lisp until August 2015, a full six years from the start of the project. I don't remember it being all that much fun, except when I changed my mind about single inheritance and went multiple. The first commit for that was in December 2019. Another fun thing was inventing a macro system for defstruct, allowing new kinds of clauses to be written that can be used inside defstruct. Then using them to write a delegation mechanism in the form of :delegate and :mass-delegate clauses, whereby you can declare individual methods, or a swath of them, to delegate through another object.
- suddenexample 2y ago
  I might be missing something here, don't records already exist? https://dart.dev/language/records https://dart.dev/language/records
  - cageface 2y ago
    They exist but they're still missing a lot of things you'd want in a data class like copyWith, serialization etc. In practice in a Dart app you usually use freezed or something similar: https://pub.dev/packages/freezed https://pub.dev/packages/freezed
ajross 2y ago
Maybe it's time to just recognize that lisp-style macros-as-language-syntax features just aren't worth the struggle and grief? The big metaprogramming feature traditionally implemented in macros, type generation, is already provided in some form by all major languages already. And an awful lot (and I mean an awful lot) of good work can be done at the string replacement level with cpp. And generating code upstream of the compiler entirely via e.g. python scripts or templating engines is a very reasonable alternative too. And at lower levels generating code programmatically via LLVM and GPU shaders is well-trodden and mature. Basically, do "macros" really have a home as a first class language feature anymore?
- thuuuomas 2y ago
  In another world, macros could have filled the role that programmable yaml fills today.
  - DiggyJohnson 2y ago
    |- help me
- elcritch 2y ago
  Oh how I enjoy trying to compile and use projects where they use some complex home brew codegen system often written in a different language entirely [1]. Luckily they often use Python as part of some core build step which never breaks compatability in their regex librwry [2]. </sarcasm> Yes macros can be a pain and should be limited, but in my experience, a couple hundred lines of macros replaces many thousands of lines code generators with complicated baroque build system integrations (ahem ROS2). The tradeoff is even worse when the language supports templates and compile time operations which can usually replace macros with even less code and are easier to understand. Though at least Go supports codegen properly with support in its official tooling. 1: https://github.com/google/flatbuffers/blob/master/src/idl_gen_dart.cpp https://github.com/google/flatbuffers/blob/master/src/idl_ge... 2: https://github.com/python/cpython/issues/94675 https://github.com/python/cpython/issues/94675
  - 2y ago
anothername12 2y ago
Why is it that Lisp family macros are easy to implement and use, but not so in other languages?
- lolinder 2y ago
  Are Lisp macros easy to use? My understanding was that Lisp code is notoriously difficult to understand if you didn't write it, largely because of the obscenely powerful macro system that makes it too easy to be too clever. Which is essentially the same complaint that everyone has about every macro system.
  - db48x 2y ago
    It’s possible to do that, but in practice it’s quite uncommon. Especially since Lisps offer great tools for programmers to learn what the macros that the are using actually do.
  - fiddlerwoaroof 2y ago
    I’ve been working with Common Lisp for about ten years now and I’ve never found that this criticism matches the reality of working on lisp codebases.
- behnamoh 2y ago
  in Scheme you can redefine `define` to be number 5. Easy to implement, but a nightmare in real world scenarios [0]. That's why languages like Go became popular, they're trashy, boring, and dumb, but that's exactly what's needed in big projects. [0]: imagine your colleague wrote a macro that redefines for loops because at the time, it made life easier for him.
  - codemac 2y ago
    > in Scheme you can redefine `define` to be number 5. This is like asking "what if your coworker named all errs as `ok`" so everything was `if ok { return errors.New("Not ok!!"); }`. It's possible but no one does it. This is why `defmacro` and `gensym` in common lisp are awesome, and similarly why Go's warts don't matter. Much of programming language ugliness is an "impact x frequency" calculation, rather than one or the other. It's also why javascript is so terrible, you run into it's warts constantly all day long.
Alifatisk 2y ago
What will the Dart team focus on instead? I wish the cross-compilation issue was taken to a higher priority, I mean Flutter already kinda of solved it.
- eseidel 2y ago
  They talk about it some in the post, but my understanding is they're going to see if the can solve some of the motivating problems (e.g. json serialization) with simpler one-off solutions rather than a big general language feature.
  - Alifatisk 2y ago
    Oh okey
deleted 2y ago
[deleted]
leecommamichael 2y ago
Good. Dart already has good support for code-generation. It would just encourage package authors and app developers to waste their time golfing.
- ollysb 2y ago
  It is incredibly slow though. I have a project with 40k lines of code which takes a minute to generate on an m1. It's a far cry from incremental compilation. It's enough that I generally avoid adding anything new that would require generation.
  - eseidel 2y ago
    I agree, Dart's public-facing codegen system (build_runner) leaves a lot to be desired. (In part the problem is that Dart uses a separate system inside Google.) However, this is a topic of active work for the Dart team: https://github.com/dart-lang/build/issues/3800 https://github.com/dart-lang/build/issues/3800. I'm sure they would welcome your feedback, particularly if you have examples you can share. You're also always welcome to reach out to me if you have Flutter/Dart concerns. I founded the Flutter project (and briefly led the Dart team) and care a great deal about customer success with both. eric@shorebird.dev reaches me.
  - cageface 2y ago
    It takes a minute to build from scratch or to update when running "build_runner watch"? My app is over 40k lines and watch updates almost instantaneously.
  - nonsense867 2y ago
    You should be able to leverage generate_for in your build.yaml with include/exclude to reduce those build times significantly. You should be able to get it back down to a few seconds including building the graph and then you should be able to just run watch instead of build. It may be worth mentioning that build_runner's graph contains every single asset that might be generated. So when selecting what's included and excluded you can reduce the graph size dramatically.
zigzag312 2y ago
They should have just use C# for Flutter. Without investing significant time, like they did with Dart, they would have a language with a much bigger ecosystem that is faster, already has compile time code generation and better support for data than Dart. It supports ahead-of-time compilation and hot-reload. The only feature missing in C# is compilation to JS, but with WASM is that really needed? Biggest downside of C# is probably that it's not invented at Google.
- kernal 2y ago
  And Microsoft should have just used Java.
  - refulgentis 2y ago
    And Java should have just been C++ with a really nice garbage collector
    - donatj 2y ago
      And C++ should have just been Objective-C with saner invocation syntax.
      - refulgentis 2y ago
        And Objective-C should have just been Smalltalk without the C baggage (is this the root NIH syndrome?! I'm guessing no, I'm only 36. maybe LISP enters the picture here?)
  - zachrip 2y ago
    Genuine question, is this comparison really apples to apples? Microsoft wanted to compete with sun right? Does google want to compete with programming languages like this? My gut tells me this is NIH not wanting to compete.
    - refulgentis 2y ago
jcstk 2y ago
FWIW, I enjoyed the hundreds of hours I spent with dart:mirrors to automate serialization, and the code-generation heavy approach always felt like kind of a bummer. But I feel like AI-assisted programming solves the majority of use cases this feature was meant for.
mindwok 2y ago
I always feel better about the stewardship of a project when you see a thoughtfully written reason for saying no to a feature, especially when there’s already sunk cost. Props to the team.
lutherqueen 2y ago
Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away
- nrclark 2y ago
  "You have discovered ENGINEERING"
mdhb 2y ago
I think after reading through the blog post the reasons they have made a whole lot of sense and sounded like that of a mature engineering team to me. There are a bunch of other interesting approaches here they can look at. Improving the code generation story more generally, shopping the augmentations feature (basically C#’s partial classes) and getting more serious about serialization all feel like sensible directions from here. There is a really interesting community proposal at the moment on the serialization front that I think would solve a lot of the issues that got people so excited about macros in the first place here: https://github.com/schultek/codable/blob/main/docs/rfc.md https://github.com/schultek/codable/blob/main/docs/rfc.md
- CharlieDigital 2y ago
  This sounds like they were going for a Roslyn analogue (using Dart to generate Dart the same way Roslyn uses C# to generate C#). Definitely a big time investment. It's a big bite to chew, but I think Roslyn has paid big dividends.
Decabytes 2y ago
> Runtime introspection (e.g., reflection) makes it difficult to perform the tree-shaking optimizations that allow us to generate smaller binaries. Does anyone have any more information on How Dart actually does Tree Shaking? And what is "Tree Shakeable"? This issue is still open on Github https://github.com/Dart-lang/sdk/issues/33920 https://github.com/Dart-lang/sdk/issues/33920. I think this quote accurately sums things up > In fact the only references I can find anywhere to this feature is on the Dart2JS page: > Don’t worry about the size of your app’s included libraries. The dart2js tool performs tree shaking to omit unused classes, functions, methods, and so on. Just import the libraries you need, and let dart2js get rid of what you don’t need. > This has led customers to wild assumptions around what is and what is not tree-shakeable, and without any clear guidance to patterns that allow or disallow tree-shaking. For example internally, many large applications chose to store configurable metadata in a hash-map:
- eseidel 2y ago
  I don't have a full answer for you, but I know a little. I've hacked on the Dart compiler some, but my relationship with Dart has mostly been as a creator of Flutter and briefly Eng Dir for the Dart project. Dart has multiple layers where it does tree shaking. The first one is when building the "dill" (dart intermediate language) file, which is essentially the "front-end" processing step of the compiler which takes .dart files and does amount of processing. At that step things like entire unused libraries and classes are removed I believe. When compiling to an ahead of time compiled binary (e.g. for releasing to iOS or Android) Dart does additional steps where it collects a set of roots and walks from those roots to related objects in the graph and discards all the rest. Not unlike a garbage collection. There are several passes of this for different parts of the compile, including as Dart is even writing the binary it will drop things like class names for unused classes (but keep their id in the snapshot so as not to re-number all the other classes). I have no experience with tree shaking in the dart2js compiler, but there are experts on Discord who might be able to answer: https://github.com/flutter/flutter/blob/master/docs/contributing/Chat.md https://github.com/flutter/flutter/blob/master/docs/contribu... What exactly all this means as a dev using Dart, I don't know. In general I just assume the tree shaking works and ignore it. :) The Dart tech lead has done some writings, but none seem to cover the exact details of treeshaking: https://mrale.ph/dartvm/ https://mrale.ph/dartvm/ https://github.com/dart-lang/sdk/blob/main/runtime/docs/README.md https://github.com/dart-lang/sdk/blob/main/runtime/docs/READ...
grahar64 2y ago
If the trade off for macros is speed, size and usability. I am glad they didn't merge macros just to tick a box and leave everyone saddled with a bad decision going forward.
chad1n 2y ago
Sounds like a good thing overall, my biggest annoyance when I was writing a flutter app was the codegen for annotations (which sure it's better iteratively, but the first one was taking minutes), but if you move these seconds that happen once in a while to seconds during "hot" reload, you're just losing. Honestly, I think they should try to come with a faster codegen, maybe write it in c++ or rust and fix these problems, because macros aren't a silver bullet. They introduce complexity, a new "thing to learn" and sometimes lead to Turing complete machines.
- munificent 2y ago
  > I think they should try to come with a faster codegen, maybe write it in c++ or rust and fix these problems Language execution speed isn't the fundamental blocker for code generation in Dart. Dart isn't quite as fast as C++ or Rust, but it's in roughly the same ballpark as other statically typed GC languages like C#, Java, and Go. The performance challenges in code generation are more architectural and are around cache invalidation, modularity, and other tricky stuff like that.
  - brookman64k 2y ago
    I belive there are many low hanging fruits to improve the speed. In my project the code generation took more than 3 minutes. This is rather annoying if you have just renamed a field in a single “freezed“ class. We could speed up this process by 3x by hacking together a script which greps all .dart files for the relevant annotations like “@freezed“ and then only feeds those to the build_runner via a on-the-fly-generated config file.
    - munificent 2y ago
      Agreed, there is a lot of opportunity to improve build_runner. I hope now that we've freed up a lot of engineering resources from macros that we can dig into that some.
    - TsonnReopev 2y ago
      https://github.com/dart-lang/build/issues/3800 https://github.com/dart-lang/build/issues/3800 is where the "low hanging fruit search" is happening :)
jdonaldson 2y ago
Macros give their own kind of power, and it's a tough call to give that up for runtime hot-reloading. Languages like Haxe have macros, but also have hot reloading capabilities that typically are supported in certain game frameworks. You probably don't want to mix them together, but it's also a good development process to have simpler compilation targets that enable more rapid R&D, and then save macros for larger/more comprehensive builds. https://haxe.org/manual/macro.html https://haxe.org/manual/macro.html https://github.com/RblSb/KhaHotReload https://github.com/RblSb/KhaHotReload
kazinator 2y ago
> Semantic introspection, unfortunately, turned out to introduce large compile-time costs which made it difficult to keep stateful hot reload hot. They must have done something wrong. Macros are expanded when you ahead-of-time compile your code, which doesn't take place in the run-time environment where you hot load, but in the build environment. It doesn't matter whether the macro are simple, or whether they can inspect lexical environments and look up type info and whatnot. Compile-time costs should never factor into hot reload, because the stuff being loaded should already be compiled. Maybe they aren't explaining it; there could be certain semantic problems preventing existing state from being re-used on what should be a hot reload. Macros create certain issues in reloading. If you change a macro such that the expansion requires different run-time support which is incompatible with existing expansions, you have problems. One option may be to reload all the code which depends on those macros, so that everything cuts over to the new run-time support. If you need to support a mixture: hot-reloaded modules using the new versions of the macros, side by side with code made using the old versions, then the old version of the run-time support has to coexist with the old. If the run-time support for the macros is something which manages state that needs to be preserved on reloads, then that can cause difficulties. The old and new macro expansions want to appear to be sharing the same state, not different silos.
- munificent 2y ago
  > Macros are expanded when you ahead-of-time compile your code, which doesn't take place in the run-time environment where you hot load, but in the build environment. The user experience with hot reload is: 1. They hit "run". 2. The compiler compiles the app. 3. The app starts running on their device. 4. They change some code in their IDE. 5. They click "hot reload". 6. The compiler compiles the changed code. 7. The IDE sends the updated code to the running app. 8. The runtime loads the changed code. 9. They see the changed behavior in their running app. Steps 6-8 determine the total time between "user requests a hot reload" and "user sees their updated app". Compilation doesn't happen on the device, but it still takes time and is in the critical path for that experience. Making the compiler slower makes hot reload slower. We measure hot reload time in milliseconds, so it doesn't take much for us to consider it an unacceptable performance regression.
munificent 2y ago
Hi, I work on Dart and was one of the people working on this feature. Reposting my Reddit comment to provide a little context: I'm bummed that it's canceled because of the lost time, but also relieved that we decided to cancel it. I feel it was the right decision. We knew the macros feature was a big risky gamble when we took a shot at it. But looking at other languages, I saw that most started out with some simple metaprogramming feature (preprocessor macros in C/C++, declarative macros in Rust, etc.) and then later outgrew them and added more complex features (C++ template metaprogramming, procedural macros in Rust). I was hoping we could leapfrog that whole process and get to One Metaprogramming Feature to Rule Them All. Alas, it is really hard to be able to introspect on the semantics of a program while it is still being modified in a coherent way without also seriously regressing compiler performance. It's probably not impossible, but it increasingly felt like the amount of work to get there was unbounded. I'm sad we weren't able to pull it off but I'm glad that we gave it a shot. We learned a lot about the problem space and some of the hidden sharp edges. I'm looking forward to working on a few smaller more targeted features to deal with the pain points we hoped to address with macros (data classes, serialization, stateful widget class verbosity, code generation UX, etc.).
- zigzag312 2y ago
  Sounds like performance is the biggest issue. I'm guessing the macros need to run after every keystroke. That creates big time constraint that while nice it's not really needed. Due to AOT compilation, some form of (pre)compile time code generation is needed, but it doesn't need to be macros. It doesn't need to be instantaneous, but it also shouldn't take minutes. Adding features directly into the language removes the need for some code generation. Augmentations will already make code generation much nicer to use. build_runner needs to become more integrated so that IDEs would read build_runner's config and run it automatically.
  - cjrome 2y ago
    That's what you really believe huh?
  - krzat 2y ago
deleted 2y ago
[deleted]

143 points by oltmang 2y ago

deleted 2y ago

[deleted]

Former Eng. Dir for Dart, co-founder of Flutter, here. I'd like to believe this is a good thing for the Dart project, but only time will tell. My hot take here: https://shorebird.dev/blog/dart-macros/ https://shorebird.dev/blog/dart-macros/

bloqs 2y ago
Thanks!
elcritch 2y ago
Thanks, good take. Especially if some of the pieces are still being added. In my humble opinion, you can handle many cases like serialization better with 'compileTime' or comptime features though I'm partial to macros. Especially with core compile time constructs like 'fields' [1, 2]. Though those require some abilities dart's compiler may not have or be able to do efficiently. That'd be a bummer, as even C++ is finally getting compile time reflection. 1: https://nim-lang.org/docs/iterators.html#fieldPairs.i%2CT https://nim-lang.org/docs/iterators.html#fieldPairs.i%2CT 2: https://www.openmymind.net/Basic-MetaProgramming-in-Zig/ https://www.openmymind.net/Basic-MetaProgramming-in-Zig/

ElliotH 2y ago

This is good news. The dart language has been getting more complicated without corresponding quality of life improvements. A first class record object without messing around with macros would be a great start.

msie 2y ago
Why is it that many languages, at the start, don't have support for records/plain structs?
- refulgentis 2y ago
  Because it's arguably syntactic sugar and, IMHO, it's worked out better for developers for Dart to model it as a 3rd party library problem. i.e. have a JSONSerializable protocol, and enable easy code generation by libraries. i.e. I annotate my models with @freezed, which also affords you config of the N different things people differ on with json (are fields snake case? camel case? pascal case?) and if a new N+1 became critical, I could hack it in myself in 2-4 hours. I'm interested to see how this'd integrate with the language while affording the same functionality. Or maybe that's the point: it won't, but you can always continue using the 3rd party libraries. But now it's built into the language, so it is easier to get from 0 to 1.
- kazinator 2y ago
  Because implementing them is tedious, and you can always simulate them with simpler aggregation methods, or possibly lexical closures. When the language implementors start making larger programs, it will soon become apparent how the program organization is hampered without named, defined data structures. I didn't add structs to TXR Lisp until August 2015, a full six years from the start of the project. I don't remember it being all that much fun, except when I changed my mind about single inheritance and went multiple. The first commit for that was in December 2019. Another fun thing was inventing a macro system for defstruct, allowing new kinds of clauses to be written that can be used inside defstruct. Then using them to write a delegation mechanism in the form of :delegate and :mass-delegate clauses, whereby you can declare individual methods, or a swath of them, to delegate through another object.
suddenexample 2y ago
I might be missing something here, don't records already exist? https://dart.dev/language/records https://dart.dev/language/records
- cageface 2y ago
  They exist but they're still missing a lot of things you'd want in a data class like copyWith, serialization etc. In practice in a Dart app you usually use freezed or something similar: https://pub.dev/packages/freezed https://pub.dev/packages/freezed

ajross 2y ago

Maybe it's time to just recognize that lisp-style macros-as-language-syntax features just aren't worth the struggle and grief? The big metaprogramming feature traditionally implemented in macros, type generation, is already provided in some form by all major languages already. And an awful lot (and I mean an awful lot) of good work can be done at the string replacement level with cpp. And generating code upstream of the compiler entirely via e.g. python scripts or templating engines is a very reasonable alternative too. And at lower levels generating code programmatically via LLVM and GPU shaders is well-trodden and mature. Basically, do "macros" really have a home as a first class language feature anymore?

thuuuomas 2y ago
In another world, macros could have filled the role that programmable yaml fills today.
- DiggyJohnson 2y ago
  |- help me
elcritch 2y ago
Oh how I enjoy trying to compile and use projects where they use some complex home brew codegen system often written in a different language entirely [1]. Luckily they often use Python as part of some core build step which never breaks compatability in their regex librwry [2]. </sarcasm> Yes macros can be a pain and should be limited, but in my experience, a couple hundred lines of macros replaces many thousands of lines code generators with complicated baroque build system integrations (ahem ROS2). The tradeoff is even worse when the language supports templates and compile time operations which can usually replace macros with even less code and are easier to understand. Though at least Go supports codegen properly with support in its official tooling. 1: https://github.com/google/flatbuffers/blob/master/src/idl_gen_dart.cpp https://github.com/google/flatbuffers/blob/master/src/idl_ge... 2: https://github.com/python/cpython/issues/94675 https://github.com/python/cpython/issues/94675
- 2y ago

anothername12 2y ago

Why is it that Lisp family macros are easy to implement and use, but not so in other languages?

lolinder 2y ago
Are Lisp macros easy to use? My understanding was that Lisp code is notoriously difficult to understand if you didn't write it, largely because of the obscenely powerful macro system that makes it too easy to be too clever. Which is essentially the same complaint that everyone has about every macro system.
- db48x 2y ago
  It’s possible to do that, but in practice it’s quite uncommon. Especially since Lisps offer great tools for programmers to learn what the macros that the are using actually do.
- fiddlerwoaroof 2y ago
  I’ve been working with Common Lisp for about ten years now and I’ve never found that this criticism matches the reality of working on lisp codebases.
behnamoh 2y ago
in Scheme you can redefine `define` to be number 5. Easy to implement, but a nightmare in real world scenarios [0]. That's why languages like Go became popular, they're trashy, boring, and dumb, but that's exactly what's needed in big projects. [0]: imagine your colleague wrote a macro that redefines for loops because at the time, it made life easier for him.
- codemac 2y ago
  > in Scheme you can redefine `define` to be number 5. This is like asking "what if your coworker named all errs as `ok`" so everything was `if ok { return errors.New("Not ok!!"); }`. It's possible but no one does it. This is why `defmacro` and `gensym` in common lisp are awesome, and similarly why Go's warts don't matter. Much of programming language ugliness is an "impact x frequency" calculation, rather than one or the other. It's also why javascript is so terrible, you run into it's warts constantly all day long.

Alifatisk 2y ago

What will the Dart team focus on instead? I wish the cross-compilation issue was taken to a higher priority, I mean Flutter already kinda of solved it.

eseidel 2y ago
They talk about it some in the post, but my understanding is they're going to see if the can solve some of the motivating problems (e.g. json serialization) with simpler one-off solutions rather than a big general language feature.
- Alifatisk 2y ago
  Oh okey

deleted 2y ago

[deleted]

leecommamichael 2y ago

Good. Dart already has good support for code-generation. It would just encourage package authors and app developers to waste their time golfing.

ollysb 2y ago
It is incredibly slow though. I have a project with 40k lines of code which takes a minute to generate on an m1. It's a far cry from incremental compilation. It's enough that I generally avoid adding anything new that would require generation.
- eseidel 2y ago
  I agree, Dart's public-facing codegen system (build_runner) leaves a lot to be desired. (In part the problem is that Dart uses a separate system inside Google.) However, this is a topic of active work for the Dart team: https://github.com/dart-lang/build/issues/3800 https://github.com/dart-lang/build/issues/3800. I'm sure they would welcome your feedback, particularly if you have examples you can share. You're also always welcome to reach out to me if you have Flutter/Dart concerns. I founded the Flutter project (and briefly led the Dart team) and care a great deal about customer success with both. eric@shorebird.dev reaches me.
- cageface 2y ago
  It takes a minute to build from scratch or to update when running "build_runner watch"? My app is over 40k lines and watch updates almost instantaneously.
- nonsense867 2y ago
  You should be able to leverage generate_for in your build.yaml with include/exclude to reduce those build times significantly. You should be able to get it back down to a few seconds including building the graph and then you should be able to just run watch instead of build. It may be worth mentioning that build_runner's graph contains every single asset that might be generated. So when selecting what's included and excluded you can reduce the graph size dramatically.

zigzag312 2y ago

They should have just use C# for Flutter. Without investing significant time, like they did with Dart, they would have a language with a much bigger ecosystem that is faster, already has compile time code generation and better support for data than Dart. It supports ahead-of-time compilation and hot-reload. The only feature missing in C# is compilation to JS, but with WASM is that really needed? Biggest downside of C# is probably that it's not invented at Google.

kernal 2y ago
And Microsoft should have just used Java.
- refulgentis 2y ago
  And Java should have just been C++ with a really nice garbage collector
  - donatj 2y ago
    And C++ should have just been Objective-C with saner invocation syntax.
    - refulgentis 2y ago
      And Objective-C should have just been Smalltalk without the C baggage (is this the root NIH syndrome?! I'm guessing no, I'm only 36. maybe LISP enters the picture here?)
- zachrip 2y ago
  Genuine question, is this comparison really apples to apples? Microsoft wanted to compete with sun right? Does google want to compete with programming languages like this? My gut tells me this is NIH not wanting to compete.
  - refulgentis 2y ago

jcstk 2y ago

FWIW, I enjoyed the hundreds of hours I spent with dart:mirrors to automate serialization, and the code-generation heavy approach always felt like kind of a bummer. But I feel like AI-assisted programming solves the majority of use cases this feature was meant for.

mindwok 2y ago

I always feel better about the stewardship of a project when you see a thoughtfully written reason for saying no to a feature, especially when there’s already sunk cost. Props to the team.

lutherqueen 2y ago

Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away

nrclark 2y ago
"You have discovered ENGINEERING"

mdhb 2y ago

I think after reading through the blog post the reasons they have made a whole lot of sense and sounded like that of a mature engineering team to me. There are a bunch of other interesting approaches here they can look at. Improving the code generation story more generally, shopping the augmentations feature (basically C#’s partial classes) and getting more serious about serialization all feel like sensible directions from here. There is a really interesting community proposal at the moment on the serialization front that I think would solve a lot of the issues that got people so excited about macros in the first place here: https://github.com/schultek/codable/blob/main/docs/rfc.md https://github.com/schultek/codable/blob/main/docs/rfc.md

CharlieDigital 2y ago
This sounds like they were going for a Roslyn analogue (using Dart to generate Dart the same way Roslyn uses C# to generate C#). Definitely a big time investment. It's a big bite to chew, but I think Roslyn has paid big dividends.

Decabytes 2y ago

> Runtime introspection (e.g., reflection) makes it difficult to perform the tree-shaking optimizations that allow us to generate smaller binaries. Does anyone have any more information on How Dart actually does Tree Shaking? And what is "Tree Shakeable"? This issue is still open on Github https://github.com/Dart-lang/sdk/issues/33920 https://github.com/Dart-lang/sdk/issues/33920. I think this quote accurately sums things up > In fact the only references I can find anywhere to this feature is on the Dart2JS page: > Don’t worry about the size of your app’s included libraries. The dart2js tool performs tree shaking to omit unused classes, functions, methods, and so on. Just import the libraries you need, and let dart2js get rid of what you don’t need. > This has led customers to wild assumptions around what is and what is not tree-shakeable, and without any clear guidance to patterns that allow or disallow tree-shaking. For example internally, many large applications chose to store configurable metadata in a hash-map:

eseidel 2y ago
I don't have a full answer for you, but I know a little. I've hacked on the Dart compiler some, but my relationship with Dart has mostly been as a creator of Flutter and briefly Eng Dir for the Dart project. Dart has multiple layers where it does tree shaking. The first one is when building the "dill" (dart intermediate language) file, which is essentially the "front-end" processing step of the compiler which takes .dart files and does amount of processing. At that step things like entire unused libraries and classes are removed I believe. When compiling to an ahead of time compiled binary (e.g. for releasing to iOS or Android) Dart does additional steps where it collects a set of roots and walks from those roots to related objects in the graph and discards all the rest. Not unlike a garbage collection. There are several passes of this for different parts of the compile, including as Dart is even writing the binary it will drop things like class names for unused classes (but keep their id in the snapshot so as not to re-number all the other classes). I have no experience with tree shaking in the dart2js compiler, but there are experts on Discord who might be able to answer: https://github.com/flutter/flutter/blob/master/docs/contributing/Chat.md https://github.com/flutter/flutter/blob/master/docs/contribu... What exactly all this means as a dev using Dart, I don't know. In general I just assume the tree shaking works and ignore it. :) The Dart tech lead has done some writings, but none seem to cover the exact details of treeshaking: https://mrale.ph/dartvm/ https://mrale.ph/dartvm/ https://github.com/dart-lang/sdk/blob/main/runtime/docs/README.md https://github.com/dart-lang/sdk/blob/main/runtime/docs/READ...

grahar64 2y ago

If the trade off for macros is speed, size and usability. I am glad they didn't merge macros just to tick a box and leave everyone saddled with a bad decision going forward.

chad1n 2y ago

Sounds like a good thing overall, my biggest annoyance when I was writing a flutter app was the codegen for annotations (which sure it's better iteratively, but the first one was taking minutes), but if you move these seconds that happen once in a while to seconds during "hot" reload, you're just losing. Honestly, I think they should try to come with a faster codegen, maybe write it in c++ or rust and fix these problems, because macros aren't a silver bullet. They introduce complexity, a new "thing to learn" and sometimes lead to Turing complete machines.

munificent 2y ago
> I think they should try to come with a faster codegen, maybe write it in c++ or rust and fix these problems Language execution speed isn't the fundamental blocker for code generation in Dart. Dart isn't quite as fast as C++ or Rust, but it's in roughly the same ballpark as other statically typed GC languages like C#, Java, and Go. The performance challenges in code generation are more architectural and are around cache invalidation, modularity, and other tricky stuff like that.
- brookman64k 2y ago
  I belive there are many low hanging fruits to improve the speed. In my project the code generation took more than 3 minutes. This is rather annoying if you have just renamed a field in a single “freezed“ class. We could speed up this process by 3x by hacking together a script which greps all .dart files for the relevant annotations like “@freezed“ and then only feeds those to the build_runner via a on-the-fly-generated config file.
  - munificent 2y ago
    Agreed, there is a lot of opportunity to improve build_runner. I hope now that we've freed up a lot of engineering resources from macros that we can dig into that some.
  - TsonnReopev 2y ago
    https://github.com/dart-lang/build/issues/3800 https://github.com/dart-lang/build/issues/3800 is where the "low hanging fruit search" is happening :)

jdonaldson 2y ago

Macros give their own kind of power, and it's a tough call to give that up for runtime hot-reloading. Languages like Haxe have macros, but also have hot reloading capabilities that typically are supported in certain game frameworks. You probably don't want to mix them together, but it's also a good development process to have simpler compilation targets that enable more rapid R&D, and then save macros for larger/more comprehensive builds. https://haxe.org/manual/macro.html https://haxe.org/manual/macro.html https://github.com/RblSb/KhaHotReload https://github.com/RblSb/KhaHotReload

kazinator 2y ago

> Semantic introspection, unfortunately, turned out to introduce large compile-time costs which made it difficult to keep stateful hot reload hot. They must have done something wrong. Macros are expanded when you ahead-of-time compile your code, which doesn't take place in the run-time environment where you hot load, but in the build environment. It doesn't matter whether the macro are simple, or whether they can inspect lexical environments and look up type info and whatnot. Compile-time costs should never factor into hot reload, because the stuff being loaded should already be compiled. Maybe they aren't explaining it; there could be certain semantic problems preventing existing state from being re-used on what should be a hot reload. Macros create certain issues in reloading. If you change a macro such that the expansion requires different run-time support which is incompatible with existing expansions, you have problems. One option may be to reload all the code which depends on those macros, so that everything cuts over to the new run-time support. If you need to support a mixture: hot-reloaded modules using the new versions of the macros, side by side with code made using the old versions, then the old version of the run-time support has to coexist with the old. If the run-time support for the macros is something which manages state that needs to be preserved on reloads, then that can cause difficulties. The old and new macro expansions want to appear to be sharing the same state, not different silos.

munificent 2y ago
> Macros are expanded when you ahead-of-time compile your code, which doesn't take place in the run-time environment where you hot load, but in the build environment. The user experience with hot reload is: 1. They hit "run". 2. The compiler compiles the app. 3. The app starts running on their device. 4. They change some code in their IDE. 5. They click "hot reload". 6. The compiler compiles the changed code. 7. The IDE sends the updated code to the running app. 8. The runtime loads the changed code. 9. They see the changed behavior in their running app. Steps 6-8 determine the total time between "user requests a hot reload" and "user sees their updated app". Compilation doesn't happen on the device, but it still takes time and is in the critical path for that experience. Making the compiler slower makes hot reload slower. We measure hot reload time in milliseconds, so it doesn't take much for us to consider it an unacceptable performance regression.

munificent 2y ago

Hi, I work on Dart and was one of the people working on this feature. Reposting my Reddit comment to provide a little context: I'm bummed that it's canceled because of the lost time, but also relieved that we decided to cancel it. I feel it was the right decision. We knew the macros feature was a big risky gamble when we took a shot at it. But looking at other languages, I saw that most started out with some simple metaprogramming feature (preprocessor macros in C/C++, declarative macros in Rust, etc.) and then later outgrew them and added more complex features (C++ template metaprogramming, procedural macros in Rust). I was hoping we could leapfrog that whole process and get to One Metaprogramming Feature to Rule Them All. Alas, it is really hard to be able to introspect on the semantics of a program while it is still being modified in a coherent way without also seriously regressing compiler performance. It's probably not impossible, but it increasingly felt like the amount of work to get there was unbounded. I'm sad we weren't able to pull it off but I'm glad that we gave it a shot. We learned a lot about the problem space and some of the hidden sharp edges. I'm looking forward to working on a few smaller more targeted features to deal with the pain points we hoped to address with macros (data classes, serialization, stateful widget class verbosity, code generation UX, etc.).

zigzag312 2y ago
Sounds like performance is the biggest issue. I'm guessing the macros need to run after every keystroke. That creates big time constraint that while nice it's not really needed. Due to AOT compilation, some form of (pre)compile time code generation is needed, but it doesn't need to be macros. It doesn't need to be instantaneous, but it also shouldn't take minutes. Adding features directly into the language removes the need for some code generation. Augmentations will already make code generation much nicer to use. build_runner needs to become more integrated so that IDEs would read build_runner's config and run it automatically.
- cjrome 2y ago
  That's what you really believe huh?
- krzat 2y ago

deleted 2y ago

[deleted]