3 ms·

> Data outlives code, and a valuable database supports many applications over time. Probably an unpopular opinion, but I think having a central database that d

> Data outlives code, and a valuable database supports many applications over time. Probably an unpopular opinion, but I think having a central database that directly interfaces with multiple applications is an enormous source of technical debt and other risks, and unnecessary for most organizations. Read-only users are fine for exploratory/analytical stuff, but multiple independent writers/cooks is a recipe for disaster. I prefer an architecture where the central "database" is a central, monolithic Django/Rails/NodeJS/Spring app that totally owns the actual database, and if someone needs access to the data, you whip up an HTTPS API for them. Yes, it is a tiny bit of effort to "whip up an API" but it deals with so many of the footguns implied by this article. "I need X+Y tables formatted as Z JSON" is a 5 minute dev task in a modern framework.

wrs 3y ago
In that architecture, the central API effectively becomes “the database”, because apps depend on it just as much. Most of the same rules would apply to the central API’s schema.
- pphysch 3y ago
  Conceptually, yes they are the same unit. Which is part of the beauty of it; it's not a radical paradigm shift like monolith->microservices or systemd->k8s. But the developer experience/productivity of bending Django or Rails in different ways is far superior to bending e.g. MySQL or PostgreSQL*. HTTPS is plainly a much better, and more widely available, integration protocol than N different SQL dialects. *much respect to what the Crunchy Data folks have accomplished
  - sgarland 3y ago
    There are pros and cons to the abstraction that an ORM brings. Assuming it’s decent (like Django), you can be guaranteed that the queries it generates are good enough. They’re not always perfect, and sometimes need help depending on how convoluted your schema and desired query are, but they’re decent. The downside is this can cause devs to never have to think in terms of what the DB is capable of, so they may not consider writing the code such that the ORM decides to write the query differently. I’ve specifically seen a dearth of semijoins in output, even though they’re a common construct in RDBMS, and often far faster for the desired end goal. Sometimes the SQL planner itself will produce them, of course, but that’s not a guarantee either.
- cturner 3y ago
  You equate the central API to the database but they are not equivalent. The issue is not how much apps depend on them, but maintenance options: in a central-database model, refactoring is high-friction. Whereas an API model can be built so that refactoring is low-friction. When we talk about composition, we distinguish between data structures that are private to their codebase, and data structures that have a contract between codebases. In the shared database model, everything is shared, so any change can affect all stakeholders. The API model respects composition. This allows you to make changes behind the perimeter without the permission of stakeholders. If you want to make a major change to internal data structures, you can retain the old API, offer a new API, and then grandfather apps from the old endpoints to the new endpoints.
  - wrs 3y ago
    I didn’t mean they’re exactly equivalent, but they share the fundamental concern that client code relies on the data structures being passed. In the case of an API, it’s the request parameters and response body — though in a GraphQL API it would look even closer to a DB query. So, many of these rules still apply: You should only grow an existing API, not shrink it. You shouldn’t rename things in an API. You shouldn’t use the same name with different meanings except in different namespaces/endpoints. Etc.
gwn7 3y ago
> I think having a central database that directly interfaces with multiple applications is an enormous source of technical debt and other risks, and unnecessary for most organizations. I think that the operative word here is "over time". So what is meant is not necessarily supporting many applications at the same time, but rather serially. So the message is supposed to be: Apps come and go as they can be rewritten for so many reasons, but there will be a lot less reasons to redesign / replace a "valuable" database.
timeagain 3y ago
Agreed. When a company outgrows its monolith this component you describe is normally the hardest part to figure out. All those sneaky little joins and stored procedures and undocumented use cases come pouring out. If your data layer has an API from the start it makes everyone act honestly about data access and provides more accountability. An API layer also allows for more options when dealing with caching, noisy neighbors, retention, consistency, and security.
- iamwil 3y ago
  what is noisy neighbors, and what's the issue with consistency?
vb-8448 3y ago
> is an enormous source of technical debt I totally agree with you, but I think in the real world (mostly in monolithic apps, microservices shouldn't be affected) at some point someone will try to access directly the database. There are several reasons for doing this: API are too slow, it's simpler and more immediate writing some SQL vs a http client, the team responsible for APIs it's no more around and similar.
- butlerm 3y ago
  Every non-trivial API has a data model, and API stability requires that it be as carefully designed as any database, unless you are going to start rolling incompatible API versions off the photocopier, which is no small task. The entire idea of an unanticipated JOIN is beyond the ken of most APIs as well, unfortunately. For an external API that may not be much of a problem, but for an internal one you might end up creating a new de facto schema with a new query language.
plandis 3y ago
In my experience, it’s less risk to take the advice in the article because even if you only have a monolith as the sole producer/consumer you could still mess up and make a backwards incompatible change. Humans are fallible, so try to minimize human error. One way to minimize the human error is by only extending the schema rather than changing it and forcing your monolith to correctly make changes to existing queries. I’m not saying adding an API is bad, because it’s not. I just think it’s solving a different set of problems.
bigger_cheese 3y ago
>Probably an unpopular opinion, but I think having a central database that directly interfaces with multiple applications is an enormous source of technical debt and other risks, and unnecessary for most organizations. Read-only users are fine for exploratory/analytical stuff, but multiple independent writers/cooks is a recipe for disaster. In my org I've felt the pain of having centralized DBs (with many writers and many readers) a lot of our woes come because of legacy debt some of these databases are quite old - a number date back to the mid 90's so over time they've ballooned considerably. The Architecture I've found which makes things less painful is to transition the the centralized database into two databases. On Database A you keep the legacy schemas etc and restrict access only to the DB writers (in our case we have A2A messaging queues as well as some compiled binaries which directly write to the DB). Then you have data replicated from database A into database B. Database B is where the data consumers (BI tools, reporting, etc) interface with the data. You can exercise greater control over the schema on B which is exposed to the data consumers without needing to mass recompile binaries which can continue writing to Database A. I'm not sure how "proper" DBAs feel about this split but it works for my usecase and has helped control ballooning legacy databases somewhat.
- rawgabbit 3y ago
  What you did is CQRS architect ure pattern and is the recommended path. It separates command writes from query reads.
  - bigger_cheese 3y ago
    Thankyou - good to know there is a name for the pattern I'm reading more about it now.

wrs 3y ago

In that architecture, the central API effectively becomes “the database”, because apps depend on it just as much. Most of the same rules would apply to the central API’s schema.

pphysch 3y ago
Conceptually, yes they are the same unit. Which is part of the beauty of it; it's not a radical paradigm shift like monolith->microservices or systemd->k8s. But the developer experience/productivity of bending Django or Rails in different ways is far superior to bending e.g. MySQL or PostgreSQL*. HTTPS is plainly a much better, and more widely available, integration protocol than N different SQL dialects. *much respect to what the Crunchy Data folks have accomplished
- sgarland 3y ago
  There are pros and cons to the abstraction that an ORM brings. Assuming it’s decent (like Django), you can be guaranteed that the queries it generates are good enough. They’re not always perfect, and sometimes need help depending on how convoluted your schema and desired query are, but they’re decent. The downside is this can cause devs to never have to think in terms of what the DB is capable of, so they may not consider writing the code such that the ORM decides to write the query differently. I’ve specifically seen a dearth of semijoins in output, even though they’re a common construct in RDBMS, and often far faster for the desired end goal. Sometimes the SQL planner itself will produce them, of course, but that’s not a guarantee either.
cturner 3y ago
You equate the central API to the database but they are not equivalent. The issue is not how much apps depend on them, but maintenance options: in a central-database model, refactoring is high-friction. Whereas an API model can be built so that refactoring is low-friction. When we talk about composition, we distinguish between data structures that are private to their codebase, and data structures that have a contract between codebases. In the shared database model, everything is shared, so any change can affect all stakeholders. The API model respects composition. This allows you to make changes behind the perimeter without the permission of stakeholders. If you want to make a major change to internal data structures, you can retain the old API, offer a new API, and then grandfather apps from the old endpoints to the new endpoints.
- wrs 3y ago
  I didn’t mean they’re exactly equivalent, but they share the fundamental concern that client code relies on the data structures being passed. In the case of an API, it’s the request parameters and response body — though in a GraphQL API it would look even closer to a DB query. So, many of these rules still apply: You should only grow an existing API, not shrink it. You shouldn’t rename things in an API. You shouldn’t use the same name with different meanings except in different namespaces/endpoints. Etc.

gwn7 3y ago

> I think having a central database that directly interfaces with multiple applications is an enormous source of technical debt and other risks, and unnecessary for most organizations. I think that the operative word here is "over time". So what is meant is not necessarily supporting many applications at the same time, but rather serially. So the message is supposed to be: Apps come and go as they can be rewritten for so many reasons, but there will be a lot less reasons to redesign / replace a "valuable" database.

timeagain 3y ago

Agreed. When a company outgrows its monolith this component you describe is normally the hardest part to figure out. All those sneaky little joins and stored procedures and undocumented use cases come pouring out. If your data layer has an API from the start it makes everyone act honestly about data access and provides more accountability. An API layer also allows for more options when dealing with caching, noisy neighbors, retention, consistency, and security.

iamwil 3y ago
what is noisy neighbors, and what's the issue with consistency?

vb-8448 3y ago

> is an enormous source of technical debt I totally agree with you, but I think in the real world (mostly in monolithic apps, microservices shouldn't be affected) at some point someone will try to access directly the database. There are several reasons for doing this: API are too slow, it's simpler and more immediate writing some SQL vs a http client, the team responsible for APIs it's no more around and similar.

butlerm 3y ago
Every non-trivial API has a data model, and API stability requires that it be as carefully designed as any database, unless you are going to start rolling incompatible API versions off the photocopier, which is no small task. The entire idea of an unanticipated JOIN is beyond the ken of most APIs as well, unfortunately. For an external API that may not be much of a problem, but for an internal one you might end up creating a new de facto schema with a new query language.

plandis 3y ago

In my experience, it’s less risk to take the advice in the article because even if you only have a monolith as the sole producer/consumer you could still mess up and make a backwards incompatible change. Humans are fallible, so try to minimize human error. One way to minimize the human error is by only extending the schema rather than changing it and forcing your monolith to correctly make changes to existing queries. I’m not saying adding an API is bad, because it’s not. I just think it’s solving a different set of problems.

bigger_cheese 3y ago

>Probably an unpopular opinion, but I think having a central database that directly interfaces with multiple applications is an enormous source of technical debt and other risks, and unnecessary for most organizations. Read-only users are fine for exploratory/analytical stuff, but multiple independent writers/cooks is a recipe for disaster. In my org I've felt the pain of having centralized DBs (with many writers and many readers) a lot of our woes come because of legacy debt some of these databases are quite old - a number date back to the mid 90's so over time they've ballooned considerably. The Architecture I've found which makes things less painful is to transition the the centralized database into two databases. On Database A you keep the legacy schemas etc and restrict access only to the DB writers (in our case we have A2A messaging queues as well as some compiled binaries which directly write to the DB). Then you have data replicated from database A into database B. Database B is where the data consumers (BI tools, reporting, etc) interface with the data. You can exercise greater control over the schema on B which is exposed to the data consumers without needing to mass recompile binaries which can continue writing to Database A. I'm not sure how "proper" DBAs feel about this split but it works for my usecase and has helped control ballooning legacy databases somewhat.

rawgabbit 3y ago
What you did is CQRS architect ure pattern and is the recommended path. It separates command writes from query reads.
- bigger_cheese 3y ago
  Thankyou - good to know there is a name for the pattern I'm reading more about it now.