5 ms·

It’s amazing that this isn’t a solved problem, but we have all of this crazy language model stuff. Unfortunately Spanner isn’t open source. Yugabyte and Citus

It’s amazing that this isn’t a solved problem, but we have all of this crazy language model stuff. Unfortunately Spanner isn’t open source. Yugabyte and Citus are close but have annoying issues. Cockroach isn’t 100% compatible (and has its own issues) and things like FoundationDB which are truly HA and comparable to Spanner in terms of consistency and fault tolerance are not easily plugged into Postgres as the underlying storage engine since sadly it’s only a key value store. edit: when I say close, I'm talking strictly about HA, not general functionality. lately I've been thinking of using FoundationDB, which is closest to Spanner in terms of ACID and serializability and mvsqlite. Then, I was thinking, since SQLite doesn't have online schema changes (nor does mvsqlite) to have a schema such as: [UUID, Data, Version, CreatedAt, UpdatedAt] Where Data is a JSON or Proto and Version is an integer. You then could mimic an online schema change by in your application code supporting two adjacent "versions", and then in an eventually consistent manner run [small] transactions to update the Data to the new Version as necessary. You would index Version, and UpdatedAt as necessary to find the rows in the table that are not "migrated." In SQLite you can also create indexes on expressions so technically all of your JSON or Proto could also have indexes.

roncesvalles 3y ago
Honestly, just use Spanner. It's the perfect database*. If you need to trade freshness for savings, put a cache in front of it. If I were building a new startup in 2023, I would need a mountain of evidence against using Spanner. It's ugly that it locks you into GCP but hey an iPhone locks you into Apple's ecosystem, that's just the price you pay to get good things. * unless you need timeseries, columnar, FTS, geospatial, graph or something special like that
- hot_gril 3y ago
  I've built my own little pile of evidence against Spanner using it for several years, after previously being on Postgres. It has good potential but isn't there yet. And if you're making a startup, you're probably a long way from the scale where sharded DBs become necessary.
  - brigadier132 3y ago
    I hate this tired argument. If I'm building a startup I'm building it with the intention to hit scale. If I can use cockroach or yugabyte and get 90% of the benefits of postgres without significant additional cost I'm going to use it. I've been at multiple startups that have had trouble scaling their database and the cost to switch database technologies when you hit scale is massive. And worst of all, the limitations of the technology always impact product development. I don't care what cool features postgres has if I can't do online schema changes. So yeah, I'd rather spend a few more hours picking an actual scalable technology now than spend what could be 7 figures (it can be a lot more, I've heard of this migration uber had to do that was costing them half a billion a year) and massively impact product development moving to scalable technology..
    - tmpz22 3y ago
      Different teams have different abilities. You can't really make general rules around any of this. Should you use spanner as a default selection for your company's database needs? Maybe.
      - audioheavy 3y ago
        both good points, including the devs having different abilities and using a ready-made scalable solution. We're living in the world of having something working that you can test in a few days shrug
      - hot_gril 3y ago
        Some general rules make sense for common use cases. Like, I'll say that a relational DBMS is generally what you want rather than a graph one. For a single-node DB, I'll always use Postgres if I have the choice. For a webserver, can't really go wrong with NodeJS, though you might have reasons to use something else (e.g. cost to run). But sharding is too use-case specific to prescribe something broadly like Spanner.
    - hot_gril 3y ago
      > If I can use cockroach or yugabyte and get 90% of the benefits of postgres without significant additional cost I'm going to use it Here's the problem, it's not 90%. Idk about those two, but you sacrifice a lot using Spanner, maybe enough to delay your launch. Any slightly advanced query becomes hard to optimize, for one. Not to dis Spanner, it's just a very different beast. CEO won't want to hear "launch is delayed, but it'll scale better later," and it's also not really true. In the initial phases, you can't tell what your actual needs will be at scale, so you'll probably have to rework things anyway. Just using Spanner doesn't solve your scaling problems. Maybe you'll need to totally redo your Spanner schema to actually scale, maybe you find you're better off sharding at the application layer than at the DB.
- endisneigh 3y ago
  Spanner is ridiculously expensive. Though I get your point, it’s not like an iPhone. An iPhone is not actually that expensive compared to top end android phones. Spanner is very expensive even compared to hosted cockroach and RDS (Aurora). That being said I’m sure if you're enterprise it's substantially cheaper. Spanner generally is about twice the cost as cockroach which seems cheap but the gap in nominal cost only grows with usage. The other issue is that since there’s no source to self host you also have to pay, unlike cockroach. Generally with cockroach I’d expect one to self host all instances except a staging/pre-pod and prod. With Spanner all your instances necessarily will be hosted, which means $$. That being all said, Spanner is worth it if money isn’t an issue
  - roncesvalles 3y ago
    The thing is, there is nothing else like Spanner, unless you want to trust a startup with your database, which opens up Cockroach, Yuga, and TiDB. Databases like RDS don't compare because they don't offer the same guarantees. Nothing even in AWS' and Azure's portfolios is like Spanner. Back of napkin math I did a couple of months ago: Spanner costs about $1/month for 2 writes/second and 10 reads/second. The minimum is $88 which is 176wps and 880rps. If your startup gets more traffic than that, you have good problems. The simplified guarantee model (fully linearizable and serializable everything; automagic sharding, replication and failover) could save you costs elsewhere. Low-value high-volume data like logs and search queries can be sent to a cheaper self-hosted timeseries/fts database.
    - ddorian43 3y ago
      You forgot storage costs and especially bandwidth costs and lockin costs and probably range-queries costs.
- aadvark69 3y ago
  Spanner is great if you have unlimited money
- bob1029 3y ago
  > perfect Doesn't spanner introduce new (very unlikely) failure modes that other databases are not impacted by? The reliance on an external consistency model feels to me like a complete outsourcing of liability that warrants thorough investigation. Hypothetically, if GPS went down for a prolonged period and/or a bug was found in the TrueTime system, what would happen to the consistency model around Spanner? I feel like some applications and customers would much rather wait for a synchronous acknowledgement from the actual, live system. An extra 150ms when you are confirming a 6-figure wire transfer could easily be framed as a good thing in most circles.
  - deleted 3y ago
    [deleted]
  - roncesvalles 3y ago
    Spanner offers both linearizability and serializability. They use the non-standard term "external consistency" to describe the union of those two properties. >Doesn't spanner introduce new (very unlikely) failure modes that other databases are not impacted by? Curious if you could go into that.
franckpachot 3y ago
Citus is not close to Spanner. No global secondary indexes, no cross-shard ACID (the commit status is eventually consistent). No auto-resharding. Yugabyte and Cockroach have a spanner-like architecture, but different open-source model and postgres-compatibility. What are those annoying issues?
- endisneigh 3y ago
  This is true, but the article is talking about HA and Citus is HA, no?
  - franckpachot 3y ago
    by itself, Citus is just sharding to multiple databases. Then each of this database can be protected like any database with a standby
hot_gril 3y ago
There are some fundamentally really hard problems with multi-node DBs, and the usable solutions depend a lot on your use case, so I'm not surprised at the current state.
gen220 3y ago
Doesn't Spanner depend on access to globally-synchronized custom atomic clock hardware? i.e. even if it were open source, it wouldn't really be valuable unless you're a data center operator. As I understand it, it's not a solved problem because there is no silver bullet, but rather trade-offs in every direction, and which solution works for you (including Spanner) is heavily dependent on your use case.
- endisneigh 3y ago
  the clocks help but are not strictly required. FoundationDB also achieves external consistency without the clocks, but has limitations, such as a 5 second window for a transaction, due to optimistic concurrency. the clocks remove such limitations.
- deleted 3y ago
  [deleted]
deleted 3y ago
[deleted]
panda888888 3y ago
Out of curiosity, what's wrong with cockroach?

roncesvalles 3y ago

Honestly, just use Spanner. It's the perfect database*. If you need to trade freshness for savings, put a cache in front of it. If I were building a new startup in 2023, I would need a mountain of evidence against using Spanner. It's ugly that it locks you into GCP but hey an iPhone locks you into Apple's ecosystem, that's just the price you pay to get good things. * unless you need timeseries, columnar, FTS, geospatial, graph or something special like that

hot_gril 3y ago
I've built my own little pile of evidence against Spanner using it for several years, after previously being on Postgres. It has good potential but isn't there yet. And if you're making a startup, you're probably a long way from the scale where sharded DBs become necessary.
- brigadier132 3y ago
  I hate this tired argument. If I'm building a startup I'm building it with the intention to hit scale. If I can use cockroach or yugabyte and get 90% of the benefits of postgres without significant additional cost I'm going to use it. I've been at multiple startups that have had trouble scaling their database and the cost to switch database technologies when you hit scale is massive. And worst of all, the limitations of the technology always impact product development. I don't care what cool features postgres has if I can't do online schema changes. So yeah, I'd rather spend a few more hours picking an actual scalable technology now than spend what could be 7 figures (it can be a lot more, I've heard of this migration uber had to do that was costing them half a billion a year) and massively impact product development moving to scalable technology..
  - tmpz22 3y ago
    Different teams have different abilities. You can't really make general rules around any of this. Should you use spanner as a default selection for your company's database needs? Maybe.
    - audioheavy 3y ago
      both good points, including the devs having different abilities and using a ready-made scalable solution. We're living in the world of having something working that you can test in a few days shrug
    - hot_gril 3y ago
      Some general rules make sense for common use cases. Like, I'll say that a relational DBMS is generally what you want rather than a graph one. For a single-node DB, I'll always use Postgres if I have the choice. For a webserver, can't really go wrong with NodeJS, though you might have reasons to use something else (e.g. cost to run). But sharding is too use-case specific to prescribe something broadly like Spanner.
  - hot_gril 3y ago
    > If I can use cockroach or yugabyte and get 90% of the benefits of postgres without significant additional cost I'm going to use it Here's the problem, it's not 90%. Idk about those two, but you sacrifice a lot using Spanner, maybe enough to delay your launch. Any slightly advanced query becomes hard to optimize, for one. Not to dis Spanner, it's just a very different beast. CEO won't want to hear "launch is delayed, but it'll scale better later," and it's also not really true. In the initial phases, you can't tell what your actual needs will be at scale, so you'll probably have to rework things anyway. Just using Spanner doesn't solve your scaling problems. Maybe you'll need to totally redo your Spanner schema to actually scale, maybe you find you're better off sharding at the application layer than at the DB.
endisneigh 3y ago
Spanner is ridiculously expensive. Though I get your point, it’s not like an iPhone. An iPhone is not actually that expensive compared to top end android phones. Spanner is very expensive even compared to hosted cockroach and RDS (Aurora). That being said I’m sure if you're enterprise it's substantially cheaper. Spanner generally is about twice the cost as cockroach which seems cheap but the gap in nominal cost only grows with usage. The other issue is that since there’s no source to self host you also have to pay, unlike cockroach. Generally with cockroach I’d expect one to self host all instances except a staging/pre-pod and prod. With Spanner all your instances necessarily will be hosted, which means $$. That being all said, Spanner is worth it if money isn’t an issue
- roncesvalles 3y ago
  The thing is, there is nothing else like Spanner, unless you want to trust a startup with your database, which opens up Cockroach, Yuga, and TiDB. Databases like RDS don't compare because they don't offer the same guarantees. Nothing even in AWS' and Azure's portfolios is like Spanner. Back of napkin math I did a couple of months ago: Spanner costs about $1/month for 2 writes/second and 10 reads/second. The minimum is $88 which is 176wps and 880rps. If your startup gets more traffic than that, you have good problems. The simplified guarantee model (fully linearizable and serializable everything; automagic sharding, replication and failover) could save you costs elsewhere. Low-value high-volume data like logs and search queries can be sent to a cheaper self-hosted timeseries/fts database.
  - ddorian43 3y ago
    You forgot storage costs and especially bandwidth costs and lockin costs and probably range-queries costs.
aadvark69 3y ago
Spanner is great if you have unlimited money
bob1029 3y ago
> perfect Doesn't spanner introduce new (very unlikely) failure modes that other databases are not impacted by? The reliance on an external consistency model feels to me like a complete outsourcing of liability that warrants thorough investigation. Hypothetically, if GPS went down for a prolonged period and/or a bug was found in the TrueTime system, what would happen to the consistency model around Spanner? I feel like some applications and customers would much rather wait for a synchronous acknowledgement from the actual, live system. An extra 150ms when you are confirming a 6-figure wire transfer could easily be framed as a good thing in most circles.
- deleted 3y ago
  [deleted]
- roncesvalles 3y ago
  Spanner offers both linearizability and serializability. They use the non-standard term "external consistency" to describe the union of those two properties. >Doesn't spanner introduce new (very unlikely) failure modes that other databases are not impacted by? Curious if you could go into that.

franckpachot 3y ago

Citus is not close to Spanner. No global secondary indexes, no cross-shard ACID (the commit status is eventually consistent). No auto-resharding. Yugabyte and Cockroach have a spanner-like architecture, but different open-source model and postgres-compatibility. What are those annoying issues?

endisneigh 3y ago
This is true, but the article is talking about HA and Citus is HA, no?
- franckpachot 3y ago
  by itself, Citus is just sharding to multiple databases. Then each of this database can be protected like any database with a standby

hot_gril 3y ago

There are some fundamentally really hard problems with multi-node DBs, and the usable solutions depend a lot on your use case, so I'm not surprised at the current state.

gen220 3y ago

Doesn't Spanner depend on access to globally-synchronized custom atomic clock hardware? i.e. even if it were open source, it wouldn't really be valuable unless you're a data center operator. As I understand it, it's not a solved problem because there is no silver bullet, but rather trade-offs in every direction, and which solution works for you (including Spanner) is heavily dependent on your use case.

endisneigh 3y ago
the clocks help but are not strictly required. FoundationDB also achieves external consistency without the clocks, but has limitations, such as a 5 second window for a transaction, due to optimistic concurrency. the clocks remove such limitations.
deleted 3y ago
[deleted]

deleted 3y ago

[deleted]

panda888888 3y ago

Out of curiosity, what's wrong with cockroach?