4 ms·

I think Ent for Go has a ton of potential. Although I haven’t used this library, I have spent a lot of time studying the design space of ORMs because I’m curren

by jitl 4y ago

I think Ent for Go has a ton of potential. Although I haven’t used this library, I have spent a lot of time studying the design space of ORMs because I’m currently iterating on an internal library that does ORM-like things at Notion. I really like the Ent approach because it allows working with a graph-based data model directly in the embedding language, instead of forcing developers to learn a new query language like Datomic datalog or SPARQL, while simultaneously avoiding lock-in to a specific data store backend. I would like to build something like Ent that is ergonomic and usable on both the client and the server in Typescript, and facilitates easily moving business logic between the two. The context at Notion is that we have a very smart client using something ORM-like and a somewhat dumb server using plain row record values, so while we do share some code between the two, it’s difficult to port larger features from the client to the server. It’s kind of the reverse situation that most people have, which is a smart server pushing GraphQL to a relatively dumb client. There are a lot of people looking into client-focused or end-to-end database abstractions right now, so the space exciting. What the upstarts should learn from Ent is: 1. Codegen is powerful, and often easier to understand than a mountain of typelevel magic. 2. A good ORM solution should really build in permissions and composing mutations at a deep level. The previous generation of ORMs like ActiveRecord left this stuff to user space to everyone’s detriment. 3. Embedded DSL as query language is a blessing and a curse. It’s better than forcing engineers to learn a weird alternative to SQL strings, but more frustrating to experts. Use these tradeoffs with caution!

Fire-Dragon-DoL 4y ago
I keep thinking that the query portion should be entirely split from the ORM portion. Query builder have specific usage, but are not the norm. None of them can model an advanced postgres query, but at the same time, they provide little (or nothing) utilities to map the data coming from a complex query, back to objects, which is funny, to me that would be the definition of "object relational mapper". I know why activerecord does this: it's way easier to do the join in memory, so that one row is obtained per object and not an enormous row representing many, but not providing any utility for raw queries seems a big missing feature. Edit: to be clear, by query I refer to read operations (select) that are not trivial (find one by id would be trivial)
- jitl 4y ago
  In a typical ORM I agree about the query builder. Most ORMs encourage the typical use of a relational database where SQL fits the domain well. But Ent is modeling data as a graph, and the DB behind the scenes is more of an implementation detail. The production version of Ent used at Meta stores data in TAO (https://engineering.fb.com/2013/06/25/core-data/tao-the-power-of-the-graph/); https://engineering.fb.com/2013/06/25/core-data/tao-the-powe... my impression is that product engineers do not write any SQL when working with Ent at Meta.
  - Fire-Dragon-DoL 4y ago
    I might be misreading here, but are you suggesting that the DB behind the scenes is not modeling things in the traditional sense but rather as a graph. Thinking of postgres, this could be by dumping a big json object in a generic "graph" table. Is that the case? That would be acceptable from my point of view. The problem is using SQL as a graph produces very inefficient queries once hitting a certain scale, but if the DB is already a graph, the problem should be limited.
    - jitl 4y ago
      The way TAO (and Notion to some degree) deal with this is by tightly coupling the persisted (SQL?) data store to an in-memory cache that make KV style reads like “get foo by foo_id” extremely fast. They also limit the kinds of queries developers can write to the ones more likely to be very fast on the infrastructure. Pushing graph queues down to a single SQL query so that the SQL DB spends massive CPU time on joins can be an issue. Instead these systems can avoid talking to the SQL DB at all for example if they’re on the happy path just chasing graph edges.
      - Fire-Dragon-DoL 4y ago
        That makes sense. In a certain way, it's doing something similar to react-query
  - guggle 4y ago
valenterry 4y ago
> 1. Codegen is powerful, and often easier to understand than a mountain of typelevel magic. I disagree here. And I also don't like your framing. Yes, type-level logic is harder to learn than understanding code you already read everyday. It's also easier to count and calculate with your fingers and use concrete examples rather than learning abstract math. But we still do it, because in the end you learn it once and have a powerful tool that makes you more productive for the rest of your life and allow you do to things that you otherwise would never have been able to. The same is true for code-generation in my opinion. And I believe it will change for Go as well. Go is not getting generics for no reason. It is because the push is strong - Go has more inexperierenced developers using it compared to other languages. But over time, they will turn into experienced developers. They learn more concept and want to become more productive. So they will gradually ask for more and more features until Go is nothing like it was before. > 2. A good ORM solution should really build in permissions and composing mutations at a deep level. The previous generation of ORMs like ActiveRecord left this stuff to user space to everyone’s detriment. Totally disagree. You can never solve those problems in a generic way for any domain. Just make it easy for the user to do it themselves. > 3. Embedded DSL as query language is a blessing and a curse. It’s better than forcing engineers to learn a weird alternative to SQL strings, but more frustrating to experts. Use these tradeoffs with caution! That's because SQL as a language has very very limited capabilities for abstraction. So every language will create a DSL to deal with it, even languages that can parse all the SQL and validate it at compile-time. Same reasons that I gave in 1.)
- rowanG077 4y ago
  Totally agree with you. Especially about point 1. Codegen is one of the bluntest tools there is and should be almost always avoided.
  - mbrodersen 4y ago
    Codegen is a super power. It has consistently made me 5 to 10 times more productive compared with experienced developers I have worked with. I have more than once implemented production code where 80% of the code is generated. With only the biz logic hand coded. Including a complex 50+ screen Typescript application developed from scratch in just 3 months. However, as with any other tool, you have to use Codegen intelligently to get the benefits. I have more than once seen Codegen being used really badly. Making the people using it think that Codegen is always a bad idea. Codegen is a complex tool and there are many ways to use it incorrectly.

by jitl 4y ago

Fire-Dragon-DoL 4y ago

I keep thinking that the query portion should be entirely split from the ORM portion. Query builder have specific usage, but are not the norm. None of them can model an advanced postgres query, but at the same time, they provide little (or nothing) utilities to map the data coming from a complex query, back to objects, which is funny, to me that would be the definition of "object relational mapper". I know why activerecord does this: it's way easier to do the join in memory, so that one row is obtained per object and not an enormous row representing many, but not providing any utility for raw queries seems a big missing feature. Edit: to be clear, by query I refer to read operations (select) that are not trivial (find one by id would be trivial)

jitl 4y ago
In a typical ORM I agree about the query builder. Most ORMs encourage the typical use of a relational database where SQL fits the domain well. But Ent is modeling data as a graph, and the DB behind the scenes is more of an implementation detail. The production version of Ent used at Meta stores data in TAO (https://engineering.fb.com/2013/06/25/core-data/tao-the-power-of-the-graph/); https://engineering.fb.com/2013/06/25/core-data/tao-the-powe... my impression is that product engineers do not write any SQL when working with Ent at Meta.
- Fire-Dragon-DoL 4y ago
  I might be misreading here, but are you suggesting that the DB behind the scenes is not modeling things in the traditional sense but rather as a graph. Thinking of postgres, this could be by dumping a big json object in a generic "graph" table. Is that the case? That would be acceptable from my point of view. The problem is using SQL as a graph produces very inefficient queries once hitting a certain scale, but if the DB is already a graph, the problem should be limited.
  - jitl 4y ago
    The way TAO (and Notion to some degree) deal with this is by tightly coupling the persisted (SQL?) data store to an in-memory cache that make KV style reads like “get foo by foo_id” extremely fast. They also limit the kinds of queries developers can write to the ones more likely to be very fast on the infrastructure. Pushing graph queues down to a single SQL query so that the SQL DB spends massive CPU time on joins can be an issue. Instead these systems can avoid talking to the SQL DB at all for example if they’re on the happy path just chasing graph edges.
    - Fire-Dragon-DoL 4y ago
      That makes sense. In a certain way, it's doing something similar to react-query
- guggle 4y ago

valenterry 4y ago

> 1. Codegen is powerful, and often easier to understand than a mountain of typelevel magic. I disagree here. And I also don't like your framing. Yes, type-level logic is harder to learn than understanding code you already read everyday. It's also easier to count and calculate with your fingers and use concrete examples rather than learning abstract math. But we still do it, because in the end you learn it once and have a powerful tool that makes you more productive for the rest of your life and allow you do to things that you otherwise would never have been able to. The same is true for code-generation in my opinion. And I believe it will change for Go as well. Go is not getting generics for no reason. It is because the push is strong - Go has more inexperierenced developers using it compared to other languages. But over time, they will turn into experienced developers. They learn more concept and want to become more productive. So they will gradually ask for more and more features until Go is nothing like it was before. > 2. A good ORM solution should really build in permissions and composing mutations at a deep level. The previous generation of ORMs like ActiveRecord left this stuff to user space to everyone’s detriment. Totally disagree. You can never solve those problems in a generic way for any domain. Just make it easy for the user to do it themselves. > 3. Embedded DSL as query language is a blessing and a curse. It’s better than forcing engineers to learn a weird alternative to SQL strings, but more frustrating to experts. Use these tradeoffs with caution! That's because SQL as a language has very very limited capabilities for abstraction. So every language will create a DSL to deal with it, even languages that can parse all the SQL and validate it at compile-time. Same reasons that I gave in 1.)

rowanG077 4y ago
Totally agree with you. Especially about point 1. Codegen is one of the bluntest tools there is and should be almost always avoided.
- mbrodersen 4y ago
  Codegen is a super power. It has consistently made me 5 to 10 times more productive compared with experienced developers I have worked with. I have more than once implemented production code where 80% of the code is generated. With only the biz logic hand coded. Including a complex 50+ screen Typescript application developed from scratch in just 3 months. However, as with any other tool, you have to use Codegen intelligently to get the benefits. I have more than once seen Codegen being used really badly. Making the people using it think that Codegen is always a bad idea. Codegen is a complex tool and there are many ways to use it incorrectly.