24 ms·

> If you exceed the capacity of a single node, sharding your data can allow you to scale horizontally to multiple nodes. It's so simple, just don't have a lot

by cfors 6y ago

> If you exceed the capacity of a single node, sharding your data can allow you to scale horizontally to multiple nodes. It's so simple, just don't have a lot of data that doesn't need to interact with other data and your problem is solved! /s

benbjohnson 6y ago
This is take seems a little intellectually dishonest. The very next sentence is, "this works particularly well for SaaS applications where each customer is isolated from one another." Sharding doesn't work for all applications. Social networks are one example where all data can potentially interact with all other data. Sarcasm noted though. :)
- aidenn0 6y ago
  Social networks are also write-heavy so probably not a good fit for SQLite anyways, right?
  - benbjohnson 6y ago
    Social networks have a lot of lurkers who mostly read so I'm not sure about the balance of reads vs writes.
    - rapnie 6y ago
      I was getting enthusiastic about Fediverse (ActivityPub) use case, for small-ish federated instances. But these probably are still rather write-heavy even with lurker users due to the synching of federation msgs with other instances.
- cfors 6y ago
  Awesome product, btw. Definitely see where it fits some niches. Actual concerns about running a real application with real uptime requirements. 1. Say your EC2 or docker container that's hosting this goes down. Is that left up to the user to deal with? RDS handles this for you 2. No ACID transactions if you ever outgrow a DB. You talk about in your pitch that the vertical scaling, so you have to just keep bumping the VPS/container memory. 3. Sure a SaaS application where a customer specific DB is isolated, but as soon as you hit any _real_ scaling limits you immediately are back to the entire problem statement you are aiming to (at least you hint at that in your pitch) solve which is the crazy n-tier architectures we have. While I was being sarcastic, I was not being intellectually dishonest. There are entire hosts of problems that you call out you are trying to solve without providing any real solution.
  - benbjohnson 6y ago
    > Awesome product, btw. Definitely see where it fits some niches. Thanks! I appreciate it. > Say your EC2 or docker container that's hosting this goes down. Is that left up to the user to deal with? RDS handles this for you Yes, that's out of scope for Litestream since there are a lot of ways to manage that depending on your application. I agree that RDS wins here for simplicity. > No ACID transactions if you ever outgrow a DB. You talk about in your pitch that the vertical scaling, so you have to just keep bumping the VPS/container memory. You still have ACID transactions but they are just per-shard. For a SaaS application, that seems reasonable since they're localized to the customer (assuming you're sharding by customer). > Sure a SaaS application where a customer specific DB is isolated, but as soon as you hit any _real_ scaling limits you immediately are back to the entire problem statement you are aiming to (at least you hint at that in your pitch) solve which is the crazy n-tier architectures we have. [...] There are entire hosts of problems that you call out you are trying to solve without providing any real solution. I'm not trying to solve an infinite scaling problem. If you're seeing sustained 100K request/sec on your application then you'll need specific solutions. But I'd argue that 98% of applications never come near that threshold and those are the applications that could benefit from simpler architecture. Thanks for all the feedback. I hope I'm not coming off as argumentative.

by cfors 6y ago

benbjohnson 6y ago

This is take seems a little intellectually dishonest. The very next sentence is, "this works particularly well for SaaS applications where each customer is isolated from one another." Sharding doesn't work for all applications. Social networks are one example where all data can potentially interact with all other data. Sarcasm noted though. :)

aidenn0 6y ago
Social networks are also write-heavy so probably not a good fit for SQLite anyways, right?
- benbjohnson 6y ago
  Social networks have a lot of lurkers who mostly read so I'm not sure about the balance of reads vs writes.
  - rapnie 6y ago
    I was getting enthusiastic about Fediverse (ActivityPub) use case, for small-ish federated instances. But these probably are still rather write-heavy even with lurker users due to the synching of federation msgs with other instances.
cfors 6y ago
Awesome product, btw. Definitely see where it fits some niches. Actual concerns about running a real application with real uptime requirements. 1. Say your EC2 or docker container that's hosting this goes down. Is that left up to the user to deal with? RDS handles this for you 2. No ACID transactions if you ever outgrow a DB. You talk about in your pitch that the vertical scaling, so you have to just keep bumping the VPS/container memory. 3. Sure a SaaS application where a customer specific DB is isolated, but as soon as you hit any _real_ scaling limits you immediately are back to the entire problem statement you are aiming to (at least you hint at that in your pitch) solve which is the crazy n-tier architectures we have. While I was being sarcastic, I was not being intellectually dishonest. There are entire hosts of problems that you call out you are trying to solve without providing any real solution.
- benbjohnson 6y ago
  > Awesome product, btw. Definitely see where it fits some niches. Thanks! I appreciate it. > Say your EC2 or docker container that's hosting this goes down. Is that left up to the user to deal with? RDS handles this for you Yes, that's out of scope for Litestream since there are a lot of ways to manage that depending on your application. I agree that RDS wins here for simplicity. > No ACID transactions if you ever outgrow a DB. You talk about in your pitch that the vertical scaling, so you have to just keep bumping the VPS/container memory. You still have ACID transactions but they are just per-shard. For a SaaS application, that seems reasonable since they're localized to the customer (assuming you're sharding by customer). > Sure a SaaS application where a customer specific DB is isolated, but as soon as you hit any _real_ scaling limits you immediately are back to the entire problem statement you are aiming to (at least you hint at that in your pitch) solve which is the crazy n-tier architectures we have. [...] There are entire hosts of problems that you call out you are trying to solve without providing any real solution. I'm not trying to solve an infinite scaling problem. If you're seeing sustained 100K request/sec on your application then you'll need specific solutions. But I'd argue that 98% of applications never come near that threshold and those are the applications that could benefit from simpler architecture. Thanks for all the feedback. I hope I'm not coming off as argumentative.