6 ms·

At my previous startup we put everything we could into the database (work queues and pup/sub included). It was magical. It minimized complexity (moving parts,

by sgk284 7y ago

At my previous startup we put everything we could into the database (work queues and pup/sub included). It was magical. It minimized complexity (moving parts, boundaries, etc...), everything was transactional, and a single dump of the database gave you a copy of the entire persistent state of your system at that moment (across all services). We had several services running, but any notion of persistent state was stored in a single database. We didn't have particularly high demands, but processed ~140M jobs per month in addition to our normal query load. Postgres handled this like a champ, had far lower p50 and p95 latency than SNS/SQS, and rarely went above single-digit percentage CPU usage on a fairly cheap DigitalOcean box. I've worked at a lot of big tech companies (Google, Microsoft, Twitter, Salesforce) in many varieties of giant distributed systems and the most valuable lesson I've learned over and over again is: Distributed systems are hard, avoid them until you can't. That's the best advice I can give to any startup. Until you're regularly sustaining 1,000's of TPS or have grown to dozens of TB of data, it is generally a distraction to even think about anything other than a single relational database.

gigatexal 7y ago
Fascinating, was there ever more written about your tech stack?
- kiwicopple 7y ago
  Not OP, but we do something similar it our startup. PostgREST (http://postgrest.org http://postgrest.org) definitely helps if you want to go down this path as it exposes everything in the database (including functions etc) via a RestFUL api. A bit outdated now, but some details here: https://paul.copplest.one/blog/nimbus-tech-2019-04.html#tech-stack https://paul.copplest.one/blog/nimbus-tech-2019-04.html#tech...
manigandham 7y ago
That's great to read, and a perfect example of solving real problems instead of wasting time on exotic infrastructure. The vast majority of companies and their scale will never go above a single midsize database server. All of the transactional, backup, and querying functionalities you list make it much more productive than using fancy AWS/cloud services.
simplecto 7y ago
> Distributed systems are hard, avoid them until you can't. thank you. May I borrow this?
- sgk284 7y ago
  Please do!
- AtlasBarfed 7y ago
  Yes, but why architect yourself into a more painful corner when you reach "can't"?
djrobstep 7y ago
We did a very similar thing where I used to work. Didn't want to bother with a separate task queue and wanted a consistent SQL interface for everything. We wrote a very simple task queue (<500 lines of python) using a postgres table and inbuilt pub/sub. Ran millions of jobs, scheduled and unscheduled, without a hitch.
barrkel 7y ago
Of course, at the point where you do need to break out beyond a single DB, life is more painful because you've got dependencies on the database all over the place...
- y4mi 7y ago
  That's incorrect. It's trivial If you have a clearly defined API with which to extract or insert data. It only starts to leak if everyone accesses the the DB through raw sql, mishmashing modules and creating insane dependencies. Give each module ownership to a table and demand to only access it through there and the backend storage system becomes irrelevant
  - barrkel 7y ago
    If you put everything in the database, and use it for all it's able to do, as the comment I replied to suggested, it is practically inevitable under startup feature pressure conditions that total modularity won't hold. And in fact I think it would be irresponsible to pursue total modularity; it would court failure. I believe you're being deeply naive about modules owning single tables. You forego relational integrity and almost the whole power of relational algebra if you take that approach. That's heaps of functionality to leave behind when you and one or two other guys need to crank out a feature a day.
    - y4mi 7y ago
      You obviously need to write queries which span tables (I.e. joins). I can see how you misunderstood me there however, it was admittedly poorly worded. What I was talking about was basic data hygine as it's often called. If you need specific data, you define a clear way to get this data and only access it through that API. This can be a class, method or anything your language of choice prefers. If you skip that step and directly access the DB everywhere in your code, you'll create another unmaintainable dumbsterfire as soon as your team goes beyond the initial programmers.
      - barrkel 7y ago
BenoitEssiambre 7y ago
To add to this, the numerous, impossible to solve issues that happen when your data starts hopping all over the network are proven mathematically: https://groups.csail.mit.edu/tds/papers/Lynch/podc89.pdf https://groups.csail.mit.edu/tds/papers/Lynch/podc89.pdf
MaxBarraclough 7y ago
For some perhaps-off-topic contrarianism, what do you make of Uncle Bob railing against database-oriented architectures? [0] (He takes a good few minutes to get his point across.) [0] https://youtu.be/o_TH-Y78tt4?t=2566 https://youtu.be/o_TH-Y78tt4?t=2566
- ianamartin 7y ago
  I can't watch that at work right now. But is it pretty much this? https://blog.cleancoder.com/uncle-bob/2012/05/15/NODB.html https://blog.cleancoder.com/uncle-bob/2012/05/15/NODB.html I've heard of him before but never really read anything much by him. Is this guy for real? He worked at one startup, got angry about a database, became a consultant, and screams about everyone being wrong about everything?
  - MaxBarraclough 7y ago
    Yes he makes similar points there. A few scattered critical thoughts: * I don't see any problem with stored procedures. They can make good sense for, say, auditing, as well as for performance. * People describe their software systems as "Using Oracle" because it matters, not because their design is stupid. It tells me that Oracle skills are relevant, for instance. * NoSQL is generally, in my experience, awful and chaotic, rather than liberating. Lots of good work has gone into making serious grown-up relational databases. Schemas, normal forms, constraints, rigorous work on the ACID properties. Something like MongoDB is just sloppy amateur-hour by comparison. * In the YouTube video, he says that solid-state drives render relational databases obsolete. This strikes me as absurd. Not even ultra-fast SSD storage technologies will do that. The relational model is effective, and the associated DBMSs still well justified. Joins belong in a query language, not in imperative code. Why would you want to try managing a huge complex dataset manually? [0] https://en.wikipedia.org/wiki/3D_XPoint https://en.wikipedia.org/wiki/3D_XPoint (Edit: formatting)
    - NotSammyHagar 7y ago
      Yeah, seems like he vastly overstates things. ssds are great for relational databases. They all use them.
Thaxll 7y ago
Running a PG instance on DO for a startup is a really bad idea, I don't understand how can choose that over a managed solution. If you're a startup just use SNS / Kinesis / Google pub sub ect ...
- tyri_kai_psomi 7y ago
  Can you elaborate on why it is a really bad idea?
  - Thaxll 7y ago
    Because running you're own DB on a VPS means: - Dealing with sec upgrade for PG and Linux - Dealing with backups - Dealing with HA, so settings up slaves / replica, then what do you do when something goes wrong? Do you manually SSH and do some magic? - Network security / usernames / passwords - ect ... Clearly what startup don't want to do and usually lacks expertise into.
- sbellware 7y ago
  Or managed Postgres on AWS Aurora, AWS RDS, Google Cloud SQL, Heroku, etc :) SNS, Kenesis (Kafka), Google Pub/Sub are awesome. Not the same problem/solution fit as an event store, but awesome for the scenarios and architectures they're targeted at.
  - ziftface 7y ago
    Is there a difference from a programming perspective between, say Kafka, and the event store linked in this post? I understand that Kafka can scale horizontally and can handle crazy throughput, but I mean from a programming point of view, the idea of a unified log as a data model applies to both, correct?
    - ethangarofolo 7y ago
      There are some similarities, and there are definitely worse technologies you could choose for a message store than Kafka. It's worth calling out the difference between event-sourced and event-based. The former is necessarily the latter, but that doesn't go in the opposite direction. Event-based just means that communication happens over events. Event-sourced means that the authoritative state of the system is sourced from events. If the events are literally the state, then how those are retrieved begins to matter. Kafka breaks down as a message store in 2 key ways that I mentioned elsewhere in all these threads. > The first is that one generally has a separate stream for each entity in an event-sourced system. Streams are sort of like topics in Kafka, but it would be quite challenging to, say, make a topic per user in Kafka. The second is Kafka's lack of optimistic concurrency support (see https://issues.apache.org/jira/browse/KAFKA-2260 https://issues.apache.org/jira/browse/KAFKA-2260). The decision to not support expected offsets makes perfect sense for what Kafka is, but it does make it unsuitable for event sourcing. If my only tool were Kafka, then I wouldn't be able to use the messages in the same way that I can with something like Message DB. And that's okay, different tools for different jobs.
traviscj 7y ago
> put everything we could into the database (work queues and pup/sub included). It was magical. I've been building systems on some pretty similar ideas for the last 5+ years, and have a similarly optimistic outlook on it. Just to add another datapoint here, we've been able to scale them up to 20-40x higher than your figure (a couple thousand jobs/sec) on a single (albeit beefy) MySQL instance. It definitely takes a fair bit of care & we certainly have some time invested in it, but it can be done. I guess my services haven't had (gotten to? :-) ) to share databases with other services for quite some time -- they did when I started my job! -- but even with that concession, the benefits of DB guarantees have been amazing. There's nothing like a transaction for preserving your sanity. The best tactic I've learned from these systems is to think really hard about the batching & partitioning behavior to keep msgs/sec high enough & transactions/sec low enough to stay afloat.
vvern 7y ago
This! So many wasted years of engineering making correct applications on top of Cassandra/Kafka/Elasticsearch when a single Postgres server could have handled everything no problem. Scalability for a tech product should be a concern at some point, but if you don’t have product market fit it doesn’t matter. The complexity of building a distributed system right out of the gate will cripple rapid iteration or lead to a broken product.

by sgk284 7y ago

gigatexal 7y ago

Fascinating, was there ever more written about your tech stack?

kiwicopple 7y ago
Not OP, but we do something similar it our startup. PostgREST (http://postgrest.org http://postgrest.org) definitely helps if you want to go down this path as it exposes everything in the database (including functions etc) via a RestFUL api. A bit outdated now, but some details here: https://paul.copplest.one/blog/nimbus-tech-2019-04.html#tech-stack https://paul.copplest.one/blog/nimbus-tech-2019-04.html#tech...

manigandham 7y ago

That's great to read, and a perfect example of solving real problems instead of wasting time on exotic infrastructure. The vast majority of companies and their scale will never go above a single midsize database server. All of the transactional, backup, and querying functionalities you list make it much more productive than using fancy AWS/cloud services.

simplecto 7y ago

> Distributed systems are hard, avoid them until you can't. thank you. May I borrow this?

sgk284 7y ago
Please do!
AtlasBarfed 7y ago
Yes, but why architect yourself into a more painful corner when you reach "can't"?

djrobstep 7y ago

We did a very similar thing where I used to work. Didn't want to bother with a separate task queue and wanted a consistent SQL interface for everything. We wrote a very simple task queue (<500 lines of python) using a postgres table and inbuilt pub/sub. Ran millions of jobs, scheduled and unscheduled, without a hitch.

barrkel 7y ago

Of course, at the point where you do need to break out beyond a single DB, life is more painful because you've got dependencies on the database all over the place...

y4mi 7y ago
That's incorrect. It's trivial If you have a clearly defined API with which to extract or insert data. It only starts to leak if everyone accesses the the DB through raw sql, mishmashing modules and creating insane dependencies. Give each module ownership to a table and demand to only access it through there and the backend storage system becomes irrelevant
- barrkel 7y ago
  If you put everything in the database, and use it for all it's able to do, as the comment I replied to suggested, it is practically inevitable under startup feature pressure conditions that total modularity won't hold. And in fact I think it would be irresponsible to pursue total modularity; it would court failure. I believe you're being deeply naive about modules owning single tables. You forego relational integrity and almost the whole power of relational algebra if you take that approach. That's heaps of functionality to leave behind when you and one or two other guys need to crank out a feature a day.
  - y4mi 7y ago
    You obviously need to write queries which span tables (I.e. joins). I can see how you misunderstood me there however, it was admittedly poorly worded. What I was talking about was basic data hygine as it's often called. If you need specific data, you define a clear way to get this data and only access it through that API. This can be a class, method or anything your language of choice prefers. If you skip that step and directly access the DB everywhere in your code, you'll create another unmaintainable dumbsterfire as soon as your team goes beyond the initial programmers.
    - barrkel 7y ago

BenoitEssiambre 7y ago

To add to this, the numerous, impossible to solve issues that happen when your data starts hopping all over the network are proven mathematically: https://groups.csail.mit.edu/tds/papers/Lynch/podc89.pdf https://groups.csail.mit.edu/tds/papers/Lynch/podc89.pdf

MaxBarraclough 7y ago

For some perhaps-off-topic contrarianism, what do you make of Uncle Bob railing against database-oriented architectures? [0] (He takes a good few minutes to get his point across.) [0] https://youtu.be/o_TH-Y78tt4?t=2566 https://youtu.be/o_TH-Y78tt4?t=2566

ianamartin 7y ago
I can't watch that at work right now. But is it pretty much this? https://blog.cleancoder.com/uncle-bob/2012/05/15/NODB.html https://blog.cleancoder.com/uncle-bob/2012/05/15/NODB.html I've heard of him before but never really read anything much by him. Is this guy for real? He worked at one startup, got angry about a database, became a consultant, and screams about everyone being wrong about everything?
- MaxBarraclough 7y ago
  Yes he makes similar points there. A few scattered critical thoughts: * I don't see any problem with stored procedures. They can make good sense for, say, auditing, as well as for performance. * People describe their software systems as "Using Oracle" because it matters, not because their design is stupid. It tells me that Oracle skills are relevant, for instance. * NoSQL is generally, in my experience, awful and chaotic, rather than liberating. Lots of good work has gone into making serious grown-up relational databases. Schemas, normal forms, constraints, rigorous work on the ACID properties. Something like MongoDB is just sloppy amateur-hour by comparison. * In the YouTube video, he says that solid-state drives render relational databases obsolete. This strikes me as absurd. Not even ultra-fast SSD storage technologies will do that. The relational model is effective, and the associated DBMSs still well justified. Joins belong in a query language, not in imperative code. Why would you want to try managing a huge complex dataset manually? [0] https://en.wikipedia.org/wiki/3D_XPoint https://en.wikipedia.org/wiki/3D_XPoint (Edit: formatting)
  - NotSammyHagar 7y ago
    Yeah, seems like he vastly overstates things. ssds are great for relational databases. They all use them.

Thaxll 7y ago

Running a PG instance on DO for a startup is a really bad idea, I don't understand how can choose that over a managed solution. If you're a startup just use SNS / Kinesis / Google pub sub ect ...

tyri_kai_psomi 7y ago
Can you elaborate on why it is a really bad idea?
- Thaxll 7y ago
  Because running you're own DB on a VPS means: - Dealing with sec upgrade for PG and Linux - Dealing with backups - Dealing with HA, so settings up slaves / replica, then what do you do when something goes wrong? Do you manually SSH and do some magic? - Network security / usernames / passwords - ect ... Clearly what startup don't want to do and usually lacks expertise into.
sbellware 7y ago
Or managed Postgres on AWS Aurora, AWS RDS, Google Cloud SQL, Heroku, etc :) SNS, Kenesis (Kafka), Google Pub/Sub are awesome. Not the same problem/solution fit as an event store, but awesome for the scenarios and architectures they're targeted at.
- ziftface 7y ago
  Is there a difference from a programming perspective between, say Kafka, and the event store linked in this post? I understand that Kafka can scale horizontally and can handle crazy throughput, but I mean from a programming point of view, the idea of a unified log as a data model applies to both, correct?
  - ethangarofolo 7y ago
    There are some similarities, and there are definitely worse technologies you could choose for a message store than Kafka. It's worth calling out the difference between event-sourced and event-based. The former is necessarily the latter, but that doesn't go in the opposite direction. Event-based just means that communication happens over events. Event-sourced means that the authoritative state of the system is sourced from events. If the events are literally the state, then how those are retrieved begins to matter. Kafka breaks down as a message store in 2 key ways that I mentioned elsewhere in all these threads. > The first is that one generally has a separate stream for each entity in an event-sourced system. Streams are sort of like topics in Kafka, but it would be quite challenging to, say, make a topic per user in Kafka. The second is Kafka's lack of optimistic concurrency support (see https://issues.apache.org/jira/browse/KAFKA-2260 https://issues.apache.org/jira/browse/KAFKA-2260). The decision to not support expected offsets makes perfect sense for what Kafka is, but it does make it unsuitable for event sourcing. If my only tool were Kafka, then I wouldn't be able to use the messages in the same way that I can with something like Message DB. And that's okay, different tools for different jobs.

traviscj 7y ago

> put everything we could into the database (work queues and pup/sub included). It was magical. I've been building systems on some pretty similar ideas for the last 5+ years, and have a similarly optimistic outlook on it. Just to add another datapoint here, we've been able to scale them up to 20-40x higher than your figure (a couple thousand jobs/sec) on a single (albeit beefy) MySQL instance. It definitely takes a fair bit of care & we certainly have some time invested in it, but it can be done. I guess my services haven't had (gotten to? :-) ) to share databases with other services for quite some time -- they did when I started my job! -- but even with that concession, the benefits of DB guarantees have been amazing. There's nothing like a transaction for preserving your sanity. The best tactic I've learned from these systems is to think really hard about the batching & partitioning behavior to keep msgs/sec high enough & transactions/sec low enough to stay afloat.

vvern 7y ago

This! So many wasted years of engineering making correct applications on top of Cassandra/Kafka/Elasticsearch when a single Postgres server could have handled everything no problem. Scalability for a tech product should be a concern at some point, but if you don’t have product market fit it doesn’t matter. The complexity of building a distributed system right out of the gate will cripple rapid iteration or lead to a broken product.