4 ms·

Everyone seems fixated on the word database and the engineering cost of writing one. This is a log file. You write data to the end of it. You flush it to dis

Everyone seems fixated on the word database and the engineering cost of writing one. This is a log file. You write data to the end of it. You flush it to disk whenever you've filled up some unit of storage that is efficient to write to disk. Every query is a full table scan. If you have multiple writers, this works out very nicely when you have one API server per disk; each server writes its own files (with a simple mutex gating the write out of a batch of records), and queries involve opening all the files in parallel and aggregating the result. (Map, shuffle, reduce.) Atomic: not applicable, as there are no transactions. Consistent: no, as there is no protection about losing the tail end of writes (consider "no space left on device" halfway through a record). Independent: not applicable, as there are no transactions. Durable: no, the data is buffered in memory before being written to the network (EBS is the network, not a disk). So with all of this in mind, the engineering cost is not going to be higher than $10,000 a month. It's a print statement. If it sounds like I'm being negative, I'm not. Log files are one of my favorite types of time series data storage. A for loop that reads every record is one of my favorite query plans. But this is not what things like Postgres or Aurora aim to do, they aim for things like "we need to edit past data several times per second and derive some of those edits from data that is also being edited". Now you have some complexity and a big old binary log file and some for loops isn't really going to get you there. But if you don't need those things, then you don't need those things, and you don't need to pay for them. The question you always have to ask, though, is have you reasoned about the business impacts of losing data through unhandled transactional conflicts? "read committed" or "non-durable writes" are often big customer service problems. "You deducted this bill payment twice, and now I can't pay the rent!" Does it matter to your end users? If not, you can save a lot of time and money. If it does, well, then the best-effort log file probably isn't going to be good for business.

bradleyjg 3y ago
If you only need those things there’s also an off the shelf solution for log files. Time you spend reinventing the wheel is time you aren’t spending finding product-market fit (if you’ve already found it you wouldn’t even consider it because you’d be too busy servicing the flood of customers.) Unless your company is so far past product market fit that it hires qualified applicants by the classfull or whatever-it-is is their product, they have no business coding up custom infra bits. The opportunity cost alone is sufficient argument against, though far from the only one.
- jrockway 3y ago
  I think that EBS is the difficult engineering problem that they purchased instead of built from scratch here. Writing binary records to a file and reading them all into memory is not going to be a time sink that prevents you from finding product/market fit. The $120,000/year burn rate on Aurora they had seems alarming; an alarm that strongly implies "we didn't use the right system for this problem". My guess for "why didn't they use something off the shelf" is that no existing software would be satisfied with the tradeoffs they made here. Nobody else wants this.
deleted 3y ago
[deleted]
happymellon 3y ago
It's also nonsense. If those were your requirements, why on earth are you using Aurora? Aurora is a multi-region, failover protected, backup managed service. This isn't. It would have been cheaper and quicker to install an OpenSource logging DB on an EC2. Like Elastic.
- jrockway 3y ago
  I mean, that's basically my default. If I have some problem involving storing data, I pretty much 100% of the time type "pgx.Connect(" and go from there. If that's not good enough, only then do I start thinking about the problem. 99.9% of the time, PostgreSQL is going to get the job done for you. Aurora is a nice drop-in replacement for Postgres. Kind of a good way of dealing "oh shit, I built way too much stuff on top of a database that can't scale to this workload." It was good enough to find product/market fit, but now it's too slow. Solution: pay Amazon some money. If that works, that works. If it doesn't, then you need to take a different approach. At this point, you're like 8 levels of weirdness down the stack, and that's when it's time for innovation. Nobody says "I need to make a CRUD app, so I'm going to build a team to build a database first." No! You just use Postgres! When that starts failing in a measurable business metric kind of way, then your creative juices start flowing and you start thinking about making your own database. (An aside: I make a database at work. We use Postgres for all the stuff that doesn't need to scale. "Is this auth token one associated with an authorized user?" That's Postgres. "Are these bytes being written to object storage approximately the same as bytes that have been written before?" That's our thing.) (Another aside: My all-time favorite database is Spanner. I used that a lot when I worked at Google. I remember someone on my team working on the Spanner integration with our product, which was basically an in-memory database, but we needed durability. I've never seen a piece of software laugh at our 20,000 writes per second workload so hard. I imagined outages, rollbacks, furrowed brows... nope. I thought there was no way a durable replicated storage system could handle our workload. I was wrong. It just accepted all of our data and never caused a problem. So to me, that was the solution to every problem. But after I left Google, my thought was to use Cloud Spanner for everything. I quickly realized I did not have enough money to pay for Spanner! Nobody had enough money. So postgres is what I settled on. I have enough money to afford it ($0 is the starting cost) and it has never done me wrong. So I don't blame OP for starting there. It's what I would have done, anyway. BTW, I think they lowered the price a lot since I last looked, so if it sounds like the kind of thing you need, you should probably just buy it. But never underestimate "beefy computer with Postgres". By the time that fails you, you will probably have a lot of money if people like the thing you're making.)

bradleyjg 3y ago

If you only need those things there’s also an off the shelf solution for log files. Time you spend reinventing the wheel is time you aren’t spending finding product-market fit (if you’ve already found it you wouldn’t even consider it because you’d be too busy servicing the flood of customers.) Unless your company is so far past product market fit that it hires qualified applicants by the classfull or whatever-it-is is their product, they have no business coding up custom infra bits. The opportunity cost alone is sufficient argument against, though far from the only one.

jrockway 3y ago
I think that EBS is the difficult engineering problem that they purchased instead of built from scratch here. Writing binary records to a file and reading them all into memory is not going to be a time sink that prevents you from finding product/market fit. The $120,000/year burn rate on Aurora they had seems alarming; an alarm that strongly implies "we didn't use the right system for this problem". My guess for "why didn't they use something off the shelf" is that no existing software would be satisfied with the tradeoffs they made here. Nobody else wants this.

deleted 3y ago

[deleted]

happymellon 3y ago

It's also nonsense. If those were your requirements, why on earth are you using Aurora? Aurora is a multi-region, failover protected, backup managed service. This isn't. It would have been cheaper and quicker to install an OpenSource logging DB on an EC2. Like Elastic.

jrockway 3y ago
I mean, that's basically my default. If I have some problem involving storing data, I pretty much 100% of the time type "pgx.Connect(" and go from there. If that's not good enough, only then do I start thinking about the problem. 99.9% of the time, PostgreSQL is going to get the job done for you. Aurora is a nice drop-in replacement for Postgres. Kind of a good way of dealing "oh shit, I built way too much stuff on top of a database that can't scale to this workload." It was good enough to find product/market fit, but now it's too slow. Solution: pay Amazon some money. If that works, that works. If it doesn't, then you need to take a different approach. At this point, you're like 8 levels of weirdness down the stack, and that's when it's time for innovation. Nobody says "I need to make a CRUD app, so I'm going to build a team to build a database first." No! You just use Postgres! When that starts failing in a measurable business metric kind of way, then your creative juices start flowing and you start thinking about making your own database. (An aside: I make a database at work. We use Postgres for all the stuff that doesn't need to scale. "Is this auth token one associated with an authorized user?" That's Postgres. "Are these bytes being written to object storage approximately the same as bytes that have been written before?" That's our thing.) (Another aside: My all-time favorite database is Spanner. I used that a lot when I worked at Google. I remember someone on my team working on the Spanner integration with our product, which was basically an in-memory database, but we needed durability. I've never seen a piece of software laugh at our 20,000 writes per second workload so hard. I imagined outages, rollbacks, furrowed brows... nope. I thought there was no way a durable replicated storage system could handle our workload. I was wrong. It just accepted all of our data and never caused a problem. So to me, that was the solution to every problem. But after I left Google, my thought was to use Cloud Spanner for everything. I quickly realized I did not have enough money to pay for Spanner! Nobody had enough money. So postgres is what I settled on. I have enough money to afford it ($0 is the starting cost) and it has never done me wrong. So I don't blame OP for starting there. It's what I would have done, anyway. BTW, I think they lowered the price a lot since I last looked, so if it sounds like the kind of thing you need, you should probably just buy it. But never underestimate "beefy computer with Postgres". By the time that fails you, you will probably have a lot of money if people like the thing you're making.)