5 ms·

Databases are "heavy", something once chosen, seldom changes. Many companies just stick to whatever DB stack they chose very early into the development. Many t

Databases are "heavy", something once chosen, seldom changes. Many companies just stick to whatever DB stack they chose very early into the development. Many tech blogs say you can use NoSQL to solve A, relax your consistency requirements to solve B and use offline pipelines to solve C, but these solutions only alleviate the problem, not solve the fundamental problem, the DB. Fast-growing companies really need to plan this from day one. It's already 2022 and we have scalable distributed RDBMS like Spanner / TiDB / Cockroach, and some of them can do a bit of analytical workload as well. Plan ahead.

drewcoo 4y ago
But if we use ORMs, they're "services" we can swap, right? Sort of? If we design architectures that hide db access (should it exist) behind a service, better yet. A FOO data service for FOO data and callers don't care how it's stored . . . then it's not heavy, is it? Not that the caller cares about. You're talking actual implementation details before we talk about partitioning responsibility.
- Hermitian909 4y ago
  > But if we use ORMs, they're "services" we can swap, right? Sort of? That's the dream, but often the answer is "no, not even sort of". Plenty of companies with super smart and experienced engineers laying the DB foundations for a company early in its life have failed at this goal. A few problems come into effect, two immediately spring to mind: 1. Eventually some engineer comes along with a reasonably compelling business case to either pass by the abstraction layer, or extend it so that it's no longer general (e.g. now it only works for Postgres). 2. As the company scales Hyrum's law comes into effect and a thousand pieces of code come to have subtle dependencies on implementation details of the DB. (2) is particularly hard to avoid because DBs are often at least the nominal source of performance bottlenecks and engineers will do almost anything to make performance acceptable no matter how horribly it wrecks abstraction layers.
- azurezyq 4y ago
  If DB works beautifully, it should not be a problem. But in reality it doesn't. Also DB doesn't live in vacuum, so you are facing these challenges: * ORM bypass, for better performance. * Sharding and application level handling of sharding (making DB not transparent) * Replication (and obviously, delays) And there are always tons of application level code just written because DB just cannot do it efficiently.
- bcrosby95 4y ago
  Actually abstracting your database would look like taking the lowest common denominator and assuming your database can only support that. Unless you're building a piece of software where database agnosticism is actually a required feature, then I generally find it to be a very poor tradeoff. Databases are incredibly powerful, you should be using them to their fullest extent rather than purposefully handicapping yourself.
ransom1538 4y ago
"Spanner / TiDB / Cockroach" Eh. Use old boring dbs: mysql, etc. Systems with decades of fixes, stackoverflow posts, work arounds, migration tools, etc. Otherwise - you will learn Spanner doesn't support deletes of 100k rows in 2022 and it is discontinued in 2023 (lol made up prediction). New db tech is pretty terrible (excluding vitess db - which is now industry standard). I wouldn't treat it like a JS framework.
- yehosef 4y ago
  you should check out singlestore.
- KronisLV 4y ago
  TiDB also claims MySQL compatibility at the protocol level, but the amount of unsupported features is slightly demotivating: https://docs.pingcap.com/tidb/stable/mysql-compatibility#unsupported-features https://docs.pingcap.com/tidb/stable/mysql-compatibility#uns... Here's some of what's not supported: Stored procedures and functions Triggers Events User-defined functions FOREIGN KEY constraints #18209 FULLTEXT syntax and indexes #1793 SPATIAL (also known as GIS/GEOMETRY) functions, data types and indexes #6347 Character sets other than ascii, latin1, binary, utf8, utf8mb4, and gbk. SYS schema Optimizer trace ... Not being able to use FOREIGN KEY constraints makes me distrust the viability of something like that, because in my experience DB schemas and how they're utilized without foreign key end up being dumpsterfires. Then again, at a certain scale you kind of have to look into alternative ways of how to look at working with your data, so there's that. That said, the idea of giving people something that's compatible (for the most part) with the drivers and syntax that they already utilize is an excellent one!
- billythemaniam 4y ago
  Spanner can delete >100k rows if partitioned DML is used. TiDB is already what Vitess is trying to be. I think you would be surprised the number of companies who touted Vitess publicly that either went to TiDB or back to vanilla MySQL.
- tmountain 4y ago
  Case in point, Drizzle. It had backing from Google, RedHat, Sun, HP, etc. The fork had over 10k commits, and it was supposed to unseat MySQL as the SaaS cloud DB. Then, the project was just canceled because, “nobody had time to work on it”. Picking boring tech for crucial areas of a business is almost always the best plan. https://en.m.wikipedia.org/wiki/Drizzle_(database_server) https://en.m.wikipedia.org/wiki/Drizzle_(database_server)
mathgladiator 4y ago
I'm believe there has to be a better way where planning can be more freeform. I've seen both sides of NoSQL versus a real DB, and I'm using a real DB for my current product which is where I package a small DB inside a Document within a NoSQL design and I have a real DB to index the documents. An interesting aspect is that the Database is a large part of the data pipe, but it often is insufficient for the product. The key issue being torn writes between the database and a service like twilio or elastic-search. A good way to broker this is via a queue, or to use a tailer via kafka for your data pipeline. My gut is that so many things are just held together by bail wire, and the complexity is depressing to me. So, here I am, inventing a new thing. I've invented a language called Adama ( https://www.adama-platform.com/ https://www.adama-platform.com/ ) which allow a document to have a schema. Accidentally, I added transformation logic to the schema and then hooked it up to a socket to discover that many things were possible. I had intentions of using this thing to help get my state under control with node.js for a complex board game, but the language took over such that the entire game was eventually built within the language. Kind of neat. My thesis is that the NoSQL pattern is fantastic for building products quickly if you don't mind the mess that schemaless evolves into, and a key weakness of what I have is that the document is limited in size in memory (which is then made tighter due to the reactive elements). However, I can leverage a logger much like a NoSQL solution to produce many of a stream of changes which I can shred into various solutions. For instance, I could put data changes directly into snowflake to provide massive scale queries. There are some neat possibilities with this approach, but limits as well. However, I'm not sure if planning ahead is the most effective thing for a startup. I believe a new discipline over the coming decades will emerge regarding how to pick the appropriate level atomic unit of data, and then the skills to break down that unit will be more common place.
dominotw 4y ago
>some of them can do a bit of analytical workload as well. Analytical dbs seem to adding transactional workload support to them. Snowflake announced something to that effect this week. https://siliconangle.com/2022/06/14/snowflake-bids-transactional-data-science-workloads-broad-set-enhancements/ https://siliconangle.com/2022/06/14/snowflake-bids-transacti...

drewcoo 4y ago

But if we use ORMs, they're "services" we can swap, right? Sort of? If we design architectures that hide db access (should it exist) behind a service, better yet. A FOO data service for FOO data and callers don't care how it's stored . . . then it's not heavy, is it? Not that the caller cares about. You're talking actual implementation details before we talk about partitioning responsibility.

Hermitian909 4y ago
> But if we use ORMs, they're "services" we can swap, right? Sort of? That's the dream, but often the answer is "no, not even sort of". Plenty of companies with super smart and experienced engineers laying the DB foundations for a company early in its life have failed at this goal. A few problems come into effect, two immediately spring to mind: 1. Eventually some engineer comes along with a reasonably compelling business case to either pass by the abstraction layer, or extend it so that it's no longer general (e.g. now it only works for Postgres). 2. As the company scales Hyrum's law comes into effect and a thousand pieces of code come to have subtle dependencies on implementation details of the DB. (2) is particularly hard to avoid because DBs are often at least the nominal source of performance bottlenecks and engineers will do almost anything to make performance acceptable no matter how horribly it wrecks abstraction layers.
azurezyq 4y ago
If DB works beautifully, it should not be a problem. But in reality it doesn't. Also DB doesn't live in vacuum, so you are facing these challenges: * ORM bypass, for better performance. * Sharding and application level handling of sharding (making DB not transparent) * Replication (and obviously, delays) And there are always tons of application level code just written because DB just cannot do it efficiently.
bcrosby95 4y ago
Actually abstracting your database would look like taking the lowest common denominator and assuming your database can only support that. Unless you're building a piece of software where database agnosticism is actually a required feature, then I generally find it to be a very poor tradeoff. Databases are incredibly powerful, you should be using them to their fullest extent rather than purposefully handicapping yourself.

ransom1538 4y ago

"Spanner / TiDB / Cockroach" Eh. Use old boring dbs: mysql, etc. Systems with decades of fixes, stackoverflow posts, work arounds, migration tools, etc. Otherwise - you will learn Spanner doesn't support deletes of 100k rows in 2022 and it is discontinued in 2023 (lol made up prediction). New db tech is pretty terrible (excluding vitess db - which is now industry standard). I wouldn't treat it like a JS framework.

yehosef 4y ago
you should check out singlestore.
KronisLV 4y ago
TiDB also claims MySQL compatibility at the protocol level, but the amount of unsupported features is slightly demotivating: https://docs.pingcap.com/tidb/stable/mysql-compatibility#unsupported-features https://docs.pingcap.com/tidb/stable/mysql-compatibility#uns... Here's some of what's not supported: Stored procedures and functions Triggers Events User-defined functions FOREIGN KEY constraints #18209 FULLTEXT syntax and indexes #1793 SPATIAL (also known as GIS/GEOMETRY) functions, data types and indexes #6347 Character sets other than ascii, latin1, binary, utf8, utf8mb4, and gbk. SYS schema Optimizer trace ... Not being able to use FOREIGN KEY constraints makes me distrust the viability of something like that, because in my experience DB schemas and how they're utilized without foreign key end up being dumpsterfires. Then again, at a certain scale you kind of have to look into alternative ways of how to look at working with your data, so there's that. That said, the idea of giving people something that's compatible (for the most part) with the drivers and syntax that they already utilize is an excellent one!
billythemaniam 4y ago
Spanner can delete >100k rows if partitioned DML is used. TiDB is already what Vitess is trying to be. I think you would be surprised the number of companies who touted Vitess publicly that either went to TiDB or back to vanilla MySQL.
tmountain 4y ago
Case in point, Drizzle. It had backing from Google, RedHat, Sun, HP, etc. The fork had over 10k commits, and it was supposed to unseat MySQL as the SaaS cloud DB. Then, the project was just canceled because, “nobody had time to work on it”. Picking boring tech for crucial areas of a business is almost always the best plan. https://en.m.wikipedia.org/wiki/Drizzle_(database_server) https://en.m.wikipedia.org/wiki/Drizzle_(database_server)

mathgladiator 4y ago

I'm believe there has to be a better way where planning can be more freeform. I've seen both sides of NoSQL versus a real DB, and I'm using a real DB for my current product which is where I package a small DB inside a Document within a NoSQL design and I have a real DB to index the documents. An interesting aspect is that the Database is a large part of the data pipe, but it often is insufficient for the product. The key issue being torn writes between the database and a service like twilio or elastic-search. A good way to broker this is via a queue, or to use a tailer via kafka for your data pipeline. My gut is that so many things are just held together by bail wire, and the complexity is depressing to me. So, here I am, inventing a new thing. I've invented a language called Adama ( https://www.adama-platform.com/ https://www.adama-platform.com/ ) which allow a document to have a schema. Accidentally, I added transformation logic to the schema and then hooked it up to a socket to discover that many things were possible. I had intentions of using this thing to help get my state under control with node.js for a complex board game, but the language took over such that the entire game was eventually built within the language. Kind of neat. My thesis is that the NoSQL pattern is fantastic for building products quickly if you don't mind the mess that schemaless evolves into, and a key weakness of what I have is that the document is limited in size in memory (which is then made tighter due to the reactive elements). However, I can leverage a logger much like a NoSQL solution to produce many of a stream of changes which I can shred into various solutions. For instance, I could put data changes directly into snowflake to provide massive scale queries. There are some neat possibilities with this approach, but limits as well. However, I'm not sure if planning ahead is the most effective thing for a startup. I believe a new discipline over the coming decades will emerge regarding how to pick the appropriate level atomic unit of data, and then the skills to break down that unit will be more common place.

dominotw 4y ago

>some of them can do a bit of analytical workload as well. Analytical dbs seem to adding transactional workload support to them. Snowflake announced something to that effect this week. https://siliconangle.com/2022/06/14/snowflake-bids-transactional-data-science-workloads-broad-set-enhancements/ https://siliconangle.com/2022/06/14/snowflake-bids-transacti...