3 ms·

using schemas rather than APIs to share information between components seems right IMO one of the reasons CRUD is so hard now is that the schema is different a

using schemas rather than APIs to share information between components seems right IMO one of the reasons CRUD is so hard now is that the schema is different at every layer of the product Slight differences between layers are necessary for permissions / privacy, but there are probably better ways to get that done than to reimplement the schema at every layer.

dathinab 7y ago
I was thinking about this recently a lot as in my experience the difference in the API is often a major source of overhead (not necessary complexity). - I think a major problem is in the difference between the way you can layout thinks in the storage layer and your application. - Another is where to evaluate correctness (e.g. in service checks + DB-system constraints, etc.). - Another major think is that different actions/taks take different slices of the same data. One area where dynamic typed languages can have a clear benefit. - Different actions/tasks in the same system work better with different representations. --- What I currently think is helpful is to: - learn from Entity-Component Systems (from Games) for slicing data of the same entity. (but can lead to problems with consistency across slices, transactions can help if doable). - I would love to have a DB which can somehow do algebraic data types/sum types/tagged union/rust enum (all different words for roughly the same concept). - Be very strict about preventing coupling of internal state, mixup of service responsibilities in logic/endpoint _and_ mixup of this responsibilities in data. Which e.g. means that e.g. you avoid any FK between schemas owned by different services, even through it often seems usefull at the beginning. - Have a _system wide_ schema for all entities split up into small slices which if combined together from the entity and only use that schemas in the system no service specific schemas. - Specify this schema somewhere, preferably generate data types and similar from it, preferably have some opt-in strict schema validation (enabled during part of testing). - Use events for communication, use the schemas from above here, too. - Have a well defined way to represent "patch"/"update" queries, I have run to often into the problem with JSON of "reseting/deleting" a values vs. just not changing it (null value vs. field not given) and/or nested optionality.
- dathinab 7y ago
  Also depending on what you do using event sourcing can be helpful, too. Or can be a big chunk of unneeded additional work. Be aware that there are 2 ways to do event sourcing one focused on traceability and maybe even replay-ability which has a choke point in that you have a single sequential log, i.e. it doesn't scale horizontally. Or the one which uses it to archive huge horizontal scalability but on the cost of problems with (more) eventual consistency and a very very hard time to get system wide transactions right. (Best of both worlds is if you can shard your system into _independent_ subsystems and have a use case where you can rely on it to not need too much throughput per shard so that you can go with sequential. E.g. like a chat system which can shard per channel all messages and user management per workspace/group).

dathinab 7y ago

I was thinking about this recently a lot as in my experience the difference in the API is often a major source of overhead (not necessary complexity). - I think a major problem is in the difference between the way you can layout thinks in the storage layer and your application. - Another is where to evaluate correctness (e.g. in service checks + DB-system constraints, etc.). - Another major think is that different actions/taks take different slices of the same data. One area where dynamic typed languages can have a clear benefit. - Different actions/tasks in the same system work better with different representations. --- What I currently think is helpful is to: - learn from Entity-Component Systems (from Games) for slicing data of the same entity. (but can lead to problems with consistency across slices, transactions can help if doable). - I would love to have a DB which can somehow do algebraic data types/sum types/tagged union/rust enum (all different words for roughly the same concept). - Be very strict about preventing coupling of internal state, mixup of service responsibilities in logic/endpoint _and_ mixup of this responsibilities in data. Which e.g. means that e.g. you avoid any FK between schemas owned by different services, even through it often seems usefull at the beginning. - Have a _system wide_ schema for all entities split up into small slices which if combined together from the entity and only use that schemas in the system no service specific schemas. - Specify this schema somewhere, preferably generate data types and similar from it, preferably have some opt-in strict schema validation (enabled during part of testing). - Use events for communication, use the schemas from above here, too. - Have a well defined way to represent "patch"/"update" queries, I have run to often into the problem with JSON of "reseting/deleting" a values vs. just not changing it (null value vs. field not given) and/or nested optionality.

dathinab 7y ago
Also depending on what you do using event sourcing can be helpful, too. Or can be a big chunk of unneeded additional work. Be aware that there are 2 ways to do event sourcing one focused on traceability and maybe even replay-ability which has a choke point in that you have a single sequential log, i.e. it doesn't scale horizontally. Or the one which uses it to archive huge horizontal scalability but on the cost of problems with (more) eventual consistency and a very very hard time to get system wide transactions right. (Best of both worlds is if you can shard your system into _independent_ subsystems and have a use case where you can rely on it to not need too much throughput per shard so that you can go with sequential. E.g. like a chat system which can shard per channel all messages and user management per workspace/group).