3 ms·

> Data retrieval generally does not require any custom business logic, while data-modifying requests do This just doesn't seem to be the case in my experience,

> Data retrieval generally does not require any custom business logic, while data-modifying requests do This just doesn't seem to be the case in my experience, at least not in most cases. Perhaps for internal services which are completely locked down and you can freely just expose the data via a REST API, but for public facing services I just haven't found this to be the case in practice, except in extremely limited circumstances.

danpalmer 2y ago
Yeah this is it. It's all fine until a product manager asks for analytics on this via their analytics tool of choice, and you have to say "sorry can't do that", or you have to build a complex data pipeline all because you can't do an HTTP POST to an external service. Also database migrations are notoriously hard to get right, at scale of traffic, at scale of development pace, at scale of team, and often require a bunch of tooling. This pattern pushes even more into database transactions. I'd rather take a boring ORM plus web framework, where you need a little boilerplate, but you get a stateless handler in Python/whatever to handle this. So much more flexibility for very little extra cost.
- dventimi 2y ago
  I don't see what the problem is with producing analytics. As for data database migrations, if anything about them is "notoriously hard" it isn't changing views and procedures.
  - danpalmer 2y ago
    Analytics would need to be written to a table, then you'd need a batch job to come along and read them out and send them into the analytics system of choice in the company. Now you've got extra database load, write load, you can't do it on a read replica but it needs to be on the database primary, rolling back a transaction now has bigger implications than a read-only process, and you're hosting another job to post to the analytics system. I'm assuming external analytics here because almost every product team I've seen wants them. As for database migrations, views and procedures are easier than data migrations, sure, but I've not seen anything around progressive rollout, canarying, etc. Do you do that separately on different replicas? All of these things are kind of solved problems in basic web apps, but all of these would require a bunch of "odd" stuff to make it work hosting in a database. Is any one part bad? Not necessarily, but I generally like to minimise things I have to apologise to new starters for, and this would cause many of those sorts of conversations.
    - dventimi 2y ago
      > then you'd need a batch job to come along and read them out and send them into the analytics system of choice in the I wouldn't need that. I would just use the logs for analytics. > I've not seen anything around progressive rollout, canarying, etc. Do you do that separately on different replicas? No, I would put the new feature into a new view or procedure, and use database roles to manage progressive rollout, canary releases, and feature flags. > I generally like to minimise things I have to apologise to new starters You do "you" as is your right. But what I hear when I read this is not that there are technical obstacles to the approach under consideration, but rather that there are social obstacles. Using the database this way "isn't how you're supposed to do it." I grant that has been the prevailing view for about thirty years or so, but to me it's unexamined received wisdom that deserves scrutiny every now and then.
      - danpalmer 2y ago
        Well you do you too, but in an org shipping a product I couldn't see this sort of thing working. > I grant that has been the prevailing view for about thirty years or so This is true, and in fact it's something I have tried to push back on. I've tried this sort of thing. I've tried to push more things into the database. I've tried things like pushing more logic into the database, or pushing more into the frontend webserver, etc, and my conclusion is that it doesn't work. > this is not that there are technical obstacles to the approach under consideration, but rather that there are social obstacles The way I see it is that there are no technical obstacles if you assume zero cost of development. If we value our time at zero we can make all of this work, and in some cases in nicer ways. This is how I feel as a hobbyist software developer, but I learnt early on in my career as a software engineer that this is not viable at work. Optimising for development time, and optimising for product flexibility or requirements changes, are almost always worth doing (and yes I realise those are in tension with each other). Moving more logic to the database like this, or eschewing a logic layer entirely in favour of configuration, may feel like an elegant solution, but I've been bitten by these sorts of things too many times. I'll take a little boilerplate, something a junior engineer will understand, and a bunch of flexibility for product direction, any day.

danpalmer 2y ago

Yeah this is it. It's all fine until a product manager asks for analytics on this via their analytics tool of choice, and you have to say "sorry can't do that", or you have to build a complex data pipeline all because you can't do an HTTP POST to an external service. Also database migrations are notoriously hard to get right, at scale of traffic, at scale of development pace, at scale of team, and often require a bunch of tooling. This pattern pushes even more into database transactions. I'd rather take a boring ORM plus web framework, where you need a little boilerplate, but you get a stateless handler in Python/whatever to handle this. So much more flexibility for very little extra cost.

dventimi 2y ago
I don't see what the problem is with producing analytics. As for data database migrations, if anything about them is "notoriously hard" it isn't changing views and procedures.
- danpalmer 2y ago
  Analytics would need to be written to a table, then you'd need a batch job to come along and read them out and send them into the analytics system of choice in the company. Now you've got extra database load, write load, you can't do it on a read replica but it needs to be on the database primary, rolling back a transaction now has bigger implications than a read-only process, and you're hosting another job to post to the analytics system. I'm assuming external analytics here because almost every product team I've seen wants them. As for database migrations, views and procedures are easier than data migrations, sure, but I've not seen anything around progressive rollout, canarying, etc. Do you do that separately on different replicas? All of these things are kind of solved problems in basic web apps, but all of these would require a bunch of "odd" stuff to make it work hosting in a database. Is any one part bad? Not necessarily, but I generally like to minimise things I have to apologise to new starters for, and this would cause many of those sorts of conversations.
  - dventimi 2y ago
    > then you'd need a batch job to come along and read them out and send them into the analytics system of choice in the I wouldn't need that. I would just use the logs for analytics. > I've not seen anything around progressive rollout, canarying, etc. Do you do that separately on different replicas? No, I would put the new feature into a new view or procedure, and use database roles to manage progressive rollout, canary releases, and feature flags. > I generally like to minimise things I have to apologise to new starters You do "you" as is your right. But what I hear when I read this is not that there are technical obstacles to the approach under consideration, but rather that there are social obstacles. Using the database this way "isn't how you're supposed to do it." I grant that has been the prevailing view for about thirty years or so, but to me it's unexamined received wisdom that deserves scrutiny every now and then.
    - danpalmer 2y ago
      Well you do you too, but in an org shipping a product I couldn't see this sort of thing working. > I grant that has been the prevailing view for about thirty years or so This is true, and in fact it's something I have tried to push back on. I've tried this sort of thing. I've tried to push more things into the database. I've tried things like pushing more logic into the database, or pushing more into the frontend webserver, etc, and my conclusion is that it doesn't work. > this is not that there are technical obstacles to the approach under consideration, but rather that there are social obstacles The way I see it is that there are no technical obstacles if you assume zero cost of development. If we value our time at zero we can make all of this work, and in some cases in nicer ways. This is how I feel as a hobbyist software developer, but I learnt early on in my career as a software engineer that this is not viable at work. Optimising for development time, and optimising for product flexibility or requirements changes, are almost always worth doing (and yes I realise those are in tension with each other). Moving more logic to the database like this, or eschewing a logic layer entirely in favour of configuration, may feel like an elegant solution, but I've been bitten by these sorts of things too many times. I'll take a little boilerplate, something a junior engineer will understand, and a bunch of flexibility for product direction, any day.