4 ms·

> No need to just have just 1 model that maps 1:1 to your DB row. This applies to all languages One reason is to avoid copying data constantly. I don't just me

> No need to just have just 1 model that maps 1:1 to your DB row. This applies to all languages One reason is to avoid copying data constantly. I don't just mean this from an efficiency perspective, but also (and maybe more so) from a simplicity one. If you have a library for shoving data into a struct mechanistically, but you then take the data from that struct and shove it into an additional struct, what's the point of the library? You're writing the code move the data anyway. In my dayjob I see this tendency constantly to have a lot of different very narrow structs that somehow integrate into some library, and then a TON of supporting code to copy between those structs. Only to then do very little actually useful work with any of the data at the end. I generally think you'd be happier with fatter structs that integrated less with weird "struct-filling" libraries.

masklinn 1y ago
> In my dayjob I see this tendency constantly to have a lot of different very narrow structs that somehow integrate into some library, and then a TON of supporting code to copy between those structs. Maybe that's the problem to solve, rather than exposing the entire internal world to the outside? Because different views of the same entities is pretty critical otherwise it's way too easy to start e.g. returning PII to public endpoints because some internal process needed it.
- delusional 1y ago
  > exposing the entire internal world to the outside That's not at all what I said. You don't need a struct to avoid exposing internal data. If you're building a JSON object, you can just not write the code to format some fields out. You don't need a new data layout for that.
  - CactusRocket 1y ago
    That's the whole point of the first item in the article, and the original comment you were replying to. In Go (and some other languages) that formatting is implicit and automatic. So you need to write to code to NOT format the fields out. Which leads to the potential security issues where data is leaked, or you can inject data into "hidden" fields. So since it's implicit and automatic, it's safer to, as a rule, define separate structs for the data input and map them, so that there is absolutely no chance (implicit or explicit) to leak or inject data.
  - masklinn 1y ago
    > If you're building a JSON object, you can just not write the code to format some fields out. Did you fail to read the article somehow? Few if any modern language requires you to write the code to format individual fields out. Even Go does not, though when it comes to json most fields need to be annotated because of the choices it made to tie casing and visibility, but then it doesn't even require you to opt the type itself into being serializable, every struct is serializable by default. Skipping fields on a serializable structure is what requires extra work.
TeMPOraL 1y ago
> also (and maybe more so) from a simplicity one. If you have a library for shoving data into a struct mechanistically, but you then take the data from that struct and shove it into an additional struct, what's the point of the library? You're writing the code move the data anyway. Super annoying if you need to do it by hand, and wastes compute and memory if you actually need to do copies of copies, but this is the mapping part of "object relational mapping", the M in ORM. Skipping it is a bad idea. Your business/domain model should not be tied directly to your persistence model. It's a common mistake that's responsible for like half of the bad rep ORMs get. Data structures may look superficially similar, but they represent different concepts with different semantics and expectations. If you skip on that, you'll end up with tons of stupid mistakes like 'masklinn mentions, and more subtle bugs when the concepts being squashed together start pulling in opposite directions over time.
kgeist 1y ago
1. As shown in the article, exposing the internal model to APIs directly is a security footgun. I've seen sites get hacked because the developer serialized internal objects without any oversight just like in the article and accidentally exposed secrets. 2. Exposing internal models to APIs directly also makes it hard to refactor code because refactoring would change APIs, which would require updating the clients (especially problematic when the clients are owned by other teams). I've seen this firsthand too in a large legacy project, people were afraid to refactor the code because whenever they tried, it broke the clients downstream. So instead of refactoring, they just added various complex hacks to avoid touching the old core code (and of course, their models also mapped directly to the UI). In the end, codebases like that, with no layer separation, become really hard to maintain and full of security problems. All because they thought it was "simpler" to skip writing ~10 lines of extra boilerplate per model to map models to DTOs. Lack of layer separation becomes a problem in the long term. When you're just starting out, it may seem like overengineering, but it isn't
- delusional 1y ago
  > Lack of layer separation becomes a problem in the long term. When you're just starting out, it may seem like overengineering, but it isn't I actually agree, but you're setting of a false dichotomy. I do believe in strong layering at the interfaces, for exactly the reasons you line up. What I don't believe in is what I might call "struct annotation based parsing" at those interfaces. Typically, you don't want to pass DTO's around your code. Usually, you take in that struct, and then immediatly have some code to poke it into the appropriate places in your actual data structures. It's very often much easier to simply take a well structured but more direct and generic interpretation of the input data, and write the code to poke it into the correct places directly. It is not that you should define your inputs separately from your internal data storage. It's that the specification of your input structure shouldn't exist as a struct, it should exist as the consequence of your parsing code. > When you're just starting out, it may seem like overengineering, but it isn't It's a real shame that the internet has driven us to assume everybody is a novice.
  - kgeist 1y ago
homebrewer 1y ago
I don't know about Go, but in Java and .NET world this is trivially solvable with libraries like MapStruct. If you have a model with 20 fields and need to create a tiny slice of it (with let's say three fields), you need a few lines of boilerplate: create a record with those fields (1-3 LOC): record Profile(int id, String login, boolean isAdmin) {} create a mapper for it: interface UserMapper { // arrays are just one example, plain models and plenty // of other data structures are supported Profile[] usersToProfiles(User[] user); // other mappers... } and then use it: class UserController { // @GET("/profiles") Profile[] getUserProfiles() { var users = userRepo.getUsers(); return userMapper.usersToProfiles(users); } } As long as fields' names match, everything will be handled for you. Adding another "view" of your users requires creating that "view" (as a record or as a plain class) and adding just one line to the mapper interface, even if that class contains all User's fields but one. So no need to write and maintain 19+ lines of copying data around. It also handles nested/recursive entities, nulls, etc. It's also using codegen, not reflection, so performance is exactly the same as if you had written it by hand, and the code is easy to read. https://mapstruct.org https://mapstruct.org Go developers usually "don't need these complications", so this is just another self-inflicted problem. Or maybe it's solved, look around.
- kgeist 1y ago
  >you have a model with 20 fields and need to create a tiny slice of it (with let's say three fields), you need a few lines of boilerplate: create a record with those fields (1-3 LOC): >create a mapper for it: > ... >Go developers usually "don't need these complications", so this is just another self-inflicted problem. In Go: type DTO struct { A, B, C string } Somewhere in your API layer: // copy the fields to the DTO return DTO{A: o.A, B: o.B, C: o.C} I fail to see where the "self-inflicted problem" is and why it requires a whole library? (which seems to require around the same number of lines of code at the end of the day, if you count the imports, the additional mapper interface)
- jerf 1y ago
  Actually it's even easier in Go with struct embedding: type SmallerThing struct { Id int Login string IsAdmin bool } type UserController struct { SmallerThing OtherField Whatever OtherField2 SomethingElse } In principle this could break down if you need super, super complicated non-overlapping mappings, in practice I have yet to need that.
eadmund 1y ago
> In my dayjob I see this tendency constantly to have a lot of different very narrow structs that somehow integrate into some library, and then a TON of supporting code to copy between those structs. Only to then do very little actually useful work with any of the data at the end. Don’t think of it as doing a little useful work at the end; think of it as doing all the useful work in the centre. Your core logic should be as close to a pure implementation without external libraries as possible (ideally zero, but that is often not easily achievable), but call out to external libraries and services to get its work done as appropriate. That does mean a fair amount of data copying, but IMHO it’s worth it. Testing copies is easy and localised, whereas understand the implications of a JSON (or Protobuf, or Django, or whatever) object carried deep into one’s core logic and passed into other services and libraries is very very difficult. There’s a common theme with the ORM trap here. The cost of a little bit of magic is often higher than the cost of a little bit of copying.

masklinn 1y ago

> In my dayjob I see this tendency constantly to have a lot of different very narrow structs that somehow integrate into some library, and then a TON of supporting code to copy between those structs. Maybe that's the problem to solve, rather than exposing the entire internal world to the outside? Because different views of the same entities is pretty critical otherwise it's way too easy to start e.g. returning PII to public endpoints because some internal process needed it.

delusional 1y ago
> exposing the entire internal world to the outside That's not at all what I said. You don't need a struct to avoid exposing internal data. If you're building a JSON object, you can just not write the code to format some fields out. You don't need a new data layout for that.
- CactusRocket 1y ago
  That's the whole point of the first item in the article, and the original comment you were replying to. In Go (and some other languages) that formatting is implicit and automatic. So you need to write to code to NOT format the fields out. Which leads to the potential security issues where data is leaked, or you can inject data into "hidden" fields. So since it's implicit and automatic, it's safer to, as a rule, define separate structs for the data input and map them, so that there is absolutely no chance (implicit or explicit) to leak or inject data.
- masklinn 1y ago
  > If you're building a JSON object, you can just not write the code to format some fields out. Did you fail to read the article somehow? Few if any modern language requires you to write the code to format individual fields out. Even Go does not, though when it comes to json most fields need to be annotated because of the choices it made to tie casing and visibility, but then it doesn't even require you to opt the type itself into being serializable, every struct is serializable by default. Skipping fields on a serializable structure is what requires extra work.

TeMPOraL 1y ago

> also (and maybe more so) from a simplicity one. If you have a library for shoving data into a struct mechanistically, but you then take the data from that struct and shove it into an additional struct, what's the point of the library? You're writing the code move the data anyway. Super annoying if you need to do it by hand, and wastes compute and memory if you actually need to do copies of copies, but this is the mapping part of "object relational mapping", the M in ORM. Skipping it is a bad idea. Your business/domain model should not be tied directly to your persistence model. It's a common mistake that's responsible for like half of the bad rep ORMs get. Data structures may look superficially similar, but they represent different concepts with different semantics and expectations. If you skip on that, you'll end up with tons of stupid mistakes like 'masklinn mentions, and more subtle bugs when the concepts being squashed together start pulling in opposite directions over time.

kgeist 1y ago

1. As shown in the article, exposing the internal model to APIs directly is a security footgun. I've seen sites get hacked because the developer serialized internal objects without any oversight just like in the article and accidentally exposed secrets. 2. Exposing internal models to APIs directly also makes it hard to refactor code because refactoring would change APIs, which would require updating the clients (especially problematic when the clients are owned by other teams). I've seen this firsthand too in a large legacy project, people were afraid to refactor the code because whenever they tried, it broke the clients downstream. So instead of refactoring, they just added various complex hacks to avoid touching the old core code (and of course, their models also mapped directly to the UI). In the end, codebases like that, with no layer separation, become really hard to maintain and full of security problems. All because they thought it was "simpler" to skip writing ~10 lines of extra boilerplate per model to map models to DTOs. Lack of layer separation becomes a problem in the long term. When you're just starting out, it may seem like overengineering, but it isn't

delusional 1y ago
> Lack of layer separation becomes a problem in the long term. When you're just starting out, it may seem like overengineering, but it isn't I actually agree, but you're setting of a false dichotomy. I do believe in strong layering at the interfaces, for exactly the reasons you line up. What I don't believe in is what I might call "struct annotation based parsing" at those interfaces. Typically, you don't want to pass DTO's around your code. Usually, you take in that struct, and then immediatly have some code to poke it into the appropriate places in your actual data structures. It's very often much easier to simply take a well structured but more direct and generic interpretation of the input data, and write the code to poke it into the correct places directly. It is not that you should define your inputs separately from your internal data storage. It's that the specification of your input structure shouldn't exist as a struct, it should exist as the consequence of your parsing code. > When you're just starting out, it may seem like overengineering, but it isn't It's a real shame that the internet has driven us to assume everybody is a novice.
- kgeist 1y ago

homebrewer 1y ago

I don't know about Go, but in Java and .NET world this is trivially solvable with libraries like MapStruct. If you have a model with 20 fields and need to create a tiny slice of it (with let's say three fields), you need a few lines of boilerplate: create a record with those fields (1-3 LOC): record Profile(int id, String login, boolean isAdmin) {} create a mapper for it: interface UserMapper { // arrays are just one example, plain models and plenty // of other data structures are supported Profile[] usersToProfiles(User[] user); // other mappers... } and then use it: class UserController { // @GET("/profiles") Profile[] getUserProfiles() { var users = userRepo.getUsers(); return userMapper.usersToProfiles(users); } } As long as fields' names match, everything will be handled for you. Adding another "view" of your users requires creating that "view" (as a record or as a plain class) and adding just one line to the mapper interface, even if that class contains all User's fields but one. So no need to write and maintain 19+ lines of copying data around. It also handles nested/recursive entities, nulls, etc. It's also using codegen, not reflection, so performance is exactly the same as if you had written it by hand, and the code is easy to read. https://mapstruct.org https://mapstruct.org Go developers usually "don't need these complications", so this is just another self-inflicted problem. Or maybe it's solved, look around.

kgeist 1y ago
>you have a model with 20 fields and need to create a tiny slice of it (with let's say three fields), you need a few lines of boilerplate: create a record with those fields (1-3 LOC): >create a mapper for it: > ... >Go developers usually "don't need these complications", so this is just another self-inflicted problem. In Go: type DTO struct { A, B, C string } Somewhere in your API layer: // copy the fields to the DTO return DTO{A: o.A, B: o.B, C: o.C} I fail to see where the "self-inflicted problem" is and why it requires a whole library? (which seems to require around the same number of lines of code at the end of the day, if you count the imports, the additional mapper interface)
jerf 1y ago
Actually it's even easier in Go with struct embedding: type SmallerThing struct { Id int Login string IsAdmin bool } type UserController struct { SmallerThing OtherField Whatever OtherField2 SomethingElse } In principle this could break down if you need super, super complicated non-overlapping mappings, in practice I have yet to need that.

eadmund 1y ago

> In my dayjob I see this tendency constantly to have a lot of different very narrow structs that somehow integrate into some library, and then a TON of supporting code to copy between those structs. Only to then do very little actually useful work with any of the data at the end. Don’t think of it as doing a little useful work at the end; think of it as doing all the useful work in the centre. Your core logic should be as close to a pure implementation without external libraries as possible (ideally zero, but that is often not easily achievable), but call out to external libraries and services to get its work done as appropriate. That does mean a fair amount of data copying, but IMHO it’s worth it. Testing copies is easy and localised, whereas understand the implications of a JSON (or Protobuf, or Django, or whatever) object carried deep into one’s core logic and passed into other services and libraries is very very difficult. There’s a common theme with the ORM trap here. The cost of a little bit of magic is often higher than the cost of a little bit of copying.