4 ms·

All valid points. I guess I’m hesitant to put time into documentation and similar, if I can’t somehow find a steady stream of sales prospects. > even though ou

All valid points. I guess I’m hesitant to put time into documentation and similar, if I can’t somehow find a steady stream of sales prospects. > even though our writes far outpace our reads, this pricing makes your system about ten times more expensive than an RDS cluster That indeed sounds off… Are you sure you’re comparing the total cost to that of an RDS cluster? I am aware that reads will be more expensive (due to the write optimization), but I was hoping most customers would make it back on cheap writes. Also the storage itself ($0.23 per GB-month) should be much cheaper than RDS.

bastawhiz 2y ago
My total database is maybe 400 gigs. Most of the writes overwrite existing data, so storage cost isn't a concern. With the cost of an upfront RI for the year on RDS (with basically as many iops as I can use), your solution gives me ~100 million reads. That's...like a month of usage at best. At least I'm my case, the fundamental problem you're facing is that reads are just too expensive. Writes and reads tend to grow at the same pace in many products: there's a ratio that tends to stay the same as you scale. $20/million reads is just a _lot_. The ratio of writes to reads for your pricing needs to be 100:1 or more for it to make sense for me, but I'm more like 10-20:1. > I guess I’m hesitant to put time into documentation and similar, if I can’t somehow find a steady stream of sales prospects. This is part of why a database company is hard to build. You will simply not find anyone willing to give you money, because the alternative is going to be a solution your customers already know and understand and which is likely extremely mature. You're competing with Postgres and Mongo. You can't ship a database product that doesn't work: you're asking people to build on you for their storage primitive. If you fuck up, that's a business-ending event for your customer. You've either got to come to the table with an extremely compelling product ("I couldn't build my business without this") or you've got to show why someone should trust you over an established but somewhat more expensive alternative.
- bjornsing 2y ago
  > The ratio of writes to reads for your pricing needs to be 100:1 or more for it to make sense for me Correct. I bet Uber’s use case here is something like 1000:1. I’ve worked on systems that were over 1000000:1. That’s where HaystackDB makes sense. > but I'm more like 10-20:1. Then RDS is hard to beat.
  - erik_seaberg 2y ago
    You might be surprised how often a deep graph of microservices ends up rereading the same prior transactions over the course of stateful payment processing and on-demand payouts. DynamoDB can give you 2.6 million short reads of base load (1 RCU/s provisioned) for $0.12 per month, which would make a $65 alternative (2.6 * $5 + 2.6 * $20) a hard sell.
    - bjornsing 2y ago
      Typically you have a “hot-cold” architecture where transactions stay in “hot” storage for a number of days (I think 12 days in the OP if I remember correctly), then are moved to “cold” storage. HaystackDB is primarily designed for the “cold” portion of that architecture. But I am considering adding an open source / source available “frontend” that would provide low-cost “hot” storage.

bastawhiz 2y ago

My total database is maybe 400 gigs. Most of the writes overwrite existing data, so storage cost isn't a concern. With the cost of an upfront RI for the year on RDS (with basically as many iops as I can use), your solution gives me ~100 million reads. That's...like a month of usage at best. At least I'm my case, the fundamental problem you're facing is that reads are just too expensive. Writes and reads tend to grow at the same pace in many products: there's a ratio that tends to stay the same as you scale. $20/million reads is just a _lot_. The ratio of writes to reads for your pricing needs to be 100:1 or more for it to make sense for me, but I'm more like 10-20:1. > I guess I’m hesitant to put time into documentation and similar, if I can’t somehow find a steady stream of sales prospects. This is part of why a database company is hard to build. You will simply not find anyone willing to give you money, because the alternative is going to be a solution your customers already know and understand and which is likely extremely mature. You're competing with Postgres and Mongo. You can't ship a database product that doesn't work: you're asking people to build on you for their storage primitive. If you fuck up, that's a business-ending event for your customer. You've either got to come to the table with an extremely compelling product ("I couldn't build my business without this") or you've got to show why someone should trust you over an established but somewhat more expensive alternative.

bjornsing 2y ago
> The ratio of writes to reads for your pricing needs to be 100:1 or more for it to make sense for me Correct. I bet Uber’s use case here is something like 1000:1. I’ve worked on systems that were over 1000000:1. That’s where HaystackDB makes sense. > but I'm more like 10-20:1. Then RDS is hard to beat.
- erik_seaberg 2y ago
  You might be surprised how often a deep graph of microservices ends up rereading the same prior transactions over the course of stateful payment processing and on-demand payouts. DynamoDB can give you 2.6 million short reads of base load (1 RCU/s provisioned) for $0.12 per month, which would make a $65 alternative (2.6 * $5 + 2.6 * $20) a hard sell.
  - bjornsing 2y ago
    Typically you have a “hot-cold” architecture where transactions stay in “hot” storage for a number of days (I think 12 days in the OP if I remember correctly), then are moved to “cold” storage. HaystackDB is primarily designed for the “cold” portion of that architecture. But I am considering adding an open source / source available “frontend” that would provide low-cost “hot” storage.