5 ms·

> 1. Codegen is powerful, and often easier to understand than a mountain of typelevel magic. I disagree here. And I also don't like your framing. Yes, type-le

> 1. Codegen is powerful, and often easier to understand than a mountain of typelevel magic. I disagree here. And I also don't like your framing. Yes, type-level logic is harder to learn than understanding code you already read everyday. It's also easier to count and calculate with your fingers and use concrete examples rather than learning abstract math. But we still do it, because in the end you learn it once and have a powerful tool that makes you more productive for the rest of your life and allow you do to things that you otherwise would never have been able to. The same is true for code-generation in my opinion. And I believe it will change for Go as well. Go is not getting generics for no reason. It is because the push is strong - Go has more inexperierenced developers using it compared to other languages. But over time, they will turn into experienced developers. They learn more concept and want to become more productive. So they will gradually ask for more and more features until Go is nothing like it was before. > 2. A good ORM solution should really build in permissions and composing mutations at a deep level. The previous generation of ORMs like ActiveRecord left this stuff to user space to everyone’s detriment. Totally disagree. You can never solve those problems in a generic way for any domain. Just make it easy for the user to do it themselves. > 3. Embedded DSL as query language is a blessing and a curse. It’s better than forcing engineers to learn a weird alternative to SQL strings, but more frustrating to experts. Use these tradeoffs with caution! That's because SQL as a language has very very limited capabilities for abstraction. So every language will create a DSL to deal with it, even languages that can parse all the SQL and validate it at compile-time. Same reasons that I gave in 1.)

rowanG077 4y ago
Totally agree with you. Especially about point 1. Codegen is one of the bluntest tools there is and should be almost always avoided.
- mbrodersen 4y ago
  Codegen is a super power. It has consistently made me 5 to 10 times more productive compared with experienced developers I have worked with. I have more than once implemented production code where 80% of the code is generated. With only the biz logic hand coded. Including a complex 50+ screen Typescript application developed from scratch in just 3 months. However, as with any other tool, you have to use Codegen intelligently to get the benefits. I have more than once seen Codegen being used really badly. Making the people using it think that Codegen is always a bad idea. Codegen is a complex tool and there are many ways to use it incorrectly.
- jitl 4y ago
  Why is that? I’ve done a few toy attempts to build out ORM-like model classes in Typescript that wrap an input schema type using typelevel features, and my profiles of the Typescript compiler show a lot of slowdowns coming from combinatoric explosions in the typechecker. In this language specifically, the type system has quite advanced expressiveness but using those features in practice runs into limitations of the checker’s implementation frequently, and more so once you have many input types. We have ~80 table types and often a single table type is a union of variants - our biggest, the type for a Notion block is a union with 64 variants. I wrote a type that takes such row type as input, and outputs an OO-style Model interface for it with accessor methods, relational shortcuts, etc, and a matching Proxy that does the same at runtime. It works, and for smaller types seems like a fine solution. But in practice all the magic gums up Typescript’s inference and makes the language service crash into memory limits. Or the compiler just gives up and says “this union type is too complex to represent”. If I move the same type magic into codegen, the compiler is much happier; plus stack traces lead directly to a simple method implementation instead of jumping into difficult-to-follow proxy machinery. The downside is that the codegen logic that consumes the Typescript compiler API to understand the schema types is more complex than the mapped types before, but it does make it feasible to have my cake (derive an ORM/model API from a schema) and eat it too (not make the compiler so unstable and touchy).
  - spion 4y ago
    While in that specific case codegen does sound like the right approach, I think in most regular cases you would be able to get away without it. Honestly, a 64-variant type is not really that common. I think codegen is OK as long as - its designed in a way where it doesn't cause users to want or need to modify the generated code (this is the biggest downside of codegen) - its against "relatively stable" API thing (protobuf / swagger descriptions, DB schema etc).
  - rowanG077 4y ago
    I said almost never not never, 64 variants is quite esoteric I'd say. The reason why I dislike codegen so much is that you fix one problem but introduce multiple new problems. How do you handle the generated code? Do you check it into source control? If so, how do you ensure people do not touch it. Manually modified code generated code is one of the worst places to be in for maintenance. You mentioned you wanted language server support this means the generated code must not be ephemeral and continually visible to the rest of the project. When will this generated code need to be regenerated? On a schema change? How do you ensure the generated code always matches the schema? Like I said all of these question can be answered, but it's quite a burden. In fact if I were you I would have looked into fixing the typescript compiler since the problems your ran into probably weren't fundamental. -
eternalban 4y ago
> You can never solve those problems in a generic way for any domain. This is rather final. Honestly curious if this was shown to be the case or just the result of failed earlier attempts.
jitl 4y ago
None of the industrial languages I’ve looked at (Kotlin, Typescript, Go, Java, Swift, Rust) can do the things I want to do using only typelevel features, especially at practical scale. Eg, transform a schema type that describes my database tables into an easy-to-use query builder and data abstraction API with rich method chaining. Often in Java/Kotlin projects people end up writing compiler plugins / annotation processors which is like moving the codegen to build time and making it harder to inspect/understand, while simultaneously spending more developer time on it since those systems may need to run for every compiler invocation. The same goes for Rust; rust’s various macro systems target the problem I have, but Rust users report long compile times due to macro magic. More esoteric languages like Scala and Haskell might have the expressive power I want, but don’t seem practical to implement for other reasons. Of course there’s always the runtime only approach in Ruby/ActiveRecord but that runs into correctness problems at practical scales. Codegen is usually brittle and is often ugly, but is guaranteed to have more metaprogramming power than in-language typelevel features. Ultimately any such system can be complex, I just hope to build one where the leverage I payed for with complexity is well worth it.
- valenterry 4y ago
  > More esoteric languages like Scala and Haskell might have the expressive power I want, but don’t seem practical to implement for other reasons. Well, if you can't use languages that make it possible/feasible, then code generation might indeed be the best option. It's just that your claim sounded quite general, so I had to jump in. :)
- spion 4y ago
  > Eg, transform a schema type that describes my database tables into an easy-to-use query builder and data abstraction API with rich method chaining. TypeScript can do this. It was done already halfway-well 6 years ago with way fewer features in TypeScript (https://github.com/brianc/node-sql/blob/master/lib/types.d.ts https://github.com/brianc/node-sql/blob/master/lib/types.d.t...) - today we can do much better.
  - spion 4y ago
    FWIW I've given up the SQL query builder route and instead went with GraphQL query builder / Hasura route (https://typed-graphql-builder.spion.dev/ https://typed-graphql-builder.spion.dev/). Its technically codegen but only from the schema; queries are fully typed on-the-fly.
xyzzy_plugh 4y ago
> But we still do it, because in the end you learn it once and have a powerful tool that makes you more productive for the rest of your life and allow you do to things that you otherwise would never have been able to. Your own words are apt: I disagree here. And I also don't like your framing. > Go has more inexperierenced developers using it compared to other languages. But over time, they will turn into experienced developers. They learn more concept and want to become more productive. So they will gradually ask for more and more features until Go is nothing like it was before. Ignoring the rest of your comment, as well as the possibly snide remark regarding the experience of engineers who use Go, I believe this neither reflects reality nor the reality I wish to participate in. Go eschews complexity, and is particularly paranoid of complexity sneaking in under the guise of convenience. Productivity is rarely hindered by having to hand-write a for-loop or some boilerplate to handle errors. Go is designed to be easy to read and easy to write, to be comprehensible by both green and sage developers, and most importantly maintainable. This means preferring explicit behavior over implicit magic. The toolchain prioritizes fast compilation times over aggressive optimization. I don't write Go to be productive (but I am very productive). I write Go so everyone can be productive. I don't want anyone spending time unraveling your life's work masterpiece. I don't want anyone to ever have to understand Scala or Haskell. It's cool that you feel it necessary to reach for more powerful, expressive language constructs to be productive. I do not.

rowanG077 4y ago

Totally agree with you. Especially about point 1. Codegen is one of the bluntest tools there is and should be almost always avoided.

mbrodersen 4y ago
Codegen is a super power. It has consistently made me 5 to 10 times more productive compared with experienced developers I have worked with. I have more than once implemented production code where 80% of the code is generated. With only the biz logic hand coded. Including a complex 50+ screen Typescript application developed from scratch in just 3 months. However, as with any other tool, you have to use Codegen intelligently to get the benefits. I have more than once seen Codegen being used really badly. Making the people using it think that Codegen is always a bad idea. Codegen is a complex tool and there are many ways to use it incorrectly.
jitl 4y ago
Why is that? I’ve done a few toy attempts to build out ORM-like model classes in Typescript that wrap an input schema type using typelevel features, and my profiles of the Typescript compiler show a lot of slowdowns coming from combinatoric explosions in the typechecker. In this language specifically, the type system has quite advanced expressiveness but using those features in practice runs into limitations of the checker’s implementation frequently, and more so once you have many input types. We have ~80 table types and often a single table type is a union of variants - our biggest, the type for a Notion block is a union with 64 variants. I wrote a type that takes such row type as input, and outputs an OO-style Model interface for it with accessor methods, relational shortcuts, etc, and a matching Proxy that does the same at runtime. It works, and for smaller types seems like a fine solution. But in practice all the magic gums up Typescript’s inference and makes the language service crash into memory limits. Or the compiler just gives up and says “this union type is too complex to represent”. If I move the same type magic into codegen, the compiler is much happier; plus stack traces lead directly to a simple method implementation instead of jumping into difficult-to-follow proxy machinery. The downside is that the codegen logic that consumes the Typescript compiler API to understand the schema types is more complex than the mapped types before, but it does make it feasible to have my cake (derive an ORM/model API from a schema) and eat it too (not make the compiler so unstable and touchy).
- spion 4y ago
  While in that specific case codegen does sound like the right approach, I think in most regular cases you would be able to get away without it. Honestly, a 64-variant type is not really that common. I think codegen is OK as long as - its designed in a way where it doesn't cause users to want or need to modify the generated code (this is the biggest downside of codegen) - its against "relatively stable" API thing (protobuf / swagger descriptions, DB schema etc).
- rowanG077 4y ago
  I said almost never not never, 64 variants is quite esoteric I'd say. The reason why I dislike codegen so much is that you fix one problem but introduce multiple new problems. How do you handle the generated code? Do you check it into source control? If so, how do you ensure people do not touch it. Manually modified code generated code is one of the worst places to be in for maintenance. You mentioned you wanted language server support this means the generated code must not be ephemeral and continually visible to the rest of the project. When will this generated code need to be regenerated? On a schema change? How do you ensure the generated code always matches the schema? Like I said all of these question can be answered, but it's quite a burden. In fact if I were you I would have looked into fixing the typescript compiler since the problems your ran into probably weren't fundamental. -

eternalban 4y ago

> You can never solve those problems in a generic way for any domain. This is rather final. Honestly curious if this was shown to be the case or just the result of failed earlier attempts.

jitl 4y ago

None of the industrial languages I’ve looked at (Kotlin, Typescript, Go, Java, Swift, Rust) can do the things I want to do using only typelevel features, especially at practical scale. Eg, transform a schema type that describes my database tables into an easy-to-use query builder and data abstraction API with rich method chaining. Often in Java/Kotlin projects people end up writing compiler plugins / annotation processors which is like moving the codegen to build time and making it harder to inspect/understand, while simultaneously spending more developer time on it since those systems may need to run for every compiler invocation. The same goes for Rust; rust’s various macro systems target the problem I have, but Rust users report long compile times due to macro magic. More esoteric languages like Scala and Haskell might have the expressive power I want, but don’t seem practical to implement for other reasons. Of course there’s always the runtime only approach in Ruby/ActiveRecord but that runs into correctness problems at practical scales. Codegen is usually brittle and is often ugly, but is guaranteed to have more metaprogramming power than in-language typelevel features. Ultimately any such system can be complex, I just hope to build one where the leverage I payed for with complexity is well worth it.

valenterry 4y ago
> More esoteric languages like Scala and Haskell might have the expressive power I want, but don’t seem practical to implement for other reasons. Well, if you can't use languages that make it possible/feasible, then code generation might indeed be the best option. It's just that your claim sounded quite general, so I had to jump in. :)
spion 4y ago
> Eg, transform a schema type that describes my database tables into an easy-to-use query builder and data abstraction API with rich method chaining. TypeScript can do this. It was done already halfway-well 6 years ago with way fewer features in TypeScript (https://github.com/brianc/node-sql/blob/master/lib/types.d.ts https://github.com/brianc/node-sql/blob/master/lib/types.d.t...) - today we can do much better.
- spion 4y ago
  FWIW I've given up the SQL query builder route and instead went with GraphQL query builder / Hasura route (https://typed-graphql-builder.spion.dev/ https://typed-graphql-builder.spion.dev/). Its technically codegen but only from the schema; queries are fully typed on-the-fly.

xyzzy_plugh 4y ago

> But we still do it, because in the end you learn it once and have a powerful tool that makes you more productive for the rest of your life and allow you do to things that you otherwise would never have been able to. Your own words are apt: I disagree here. And I also don't like your framing. > Go has more inexperierenced developers using it compared to other languages. But over time, they will turn into experienced developers. They learn more concept and want to become more productive. So they will gradually ask for more and more features until Go is nothing like it was before. Ignoring the rest of your comment, as well as the possibly snide remark regarding the experience of engineers who use Go, I believe this neither reflects reality nor the reality I wish to participate in. Go eschews complexity, and is particularly paranoid of complexity sneaking in under the guise of convenience. Productivity is rarely hindered by having to hand-write a for-loop or some boilerplate to handle errors. Go is designed to be easy to read and easy to write, to be comprehensible by both green and sage developers, and most importantly maintainable. This means preferring explicit behavior over implicit magic. The toolchain prioritizes fast compilation times over aggressive optimization. I don't write Go to be productive (but I am very productive). I write Go so everyone can be productive. I don't want anyone spending time unraveling your life's work masterpiece. I don't want anyone to ever have to understand Scala or Haskell. It's cool that you feel it necessary to reach for more powerful, expressive language constructs to be productive. I do not.