10 ms·

Spending $5k to learn how database indexes work

236 points by anglinb 5y ago

NicoJuicy 5y ago
0,15 $ per query... The world has gone insane.
- cperciva 5y ago
  It's not quite so crazy when you phrase it as $0.15 for reading a million items from the database.
  - williamdclt 5y ago
    No that's still crazy. Scanning a million items isn't a big workload.
    - friedman23 5y ago
      I agree, it might be distributed in some way which is driving up the cost?
      - Nextgrid 5y ago
        The greed is distributed across the cloud provider's C-suite for sure. To be fair to them they refunded the bill afterwards according to the article but IMO we should not be accepting this kind of pricing models as normal. A million items isn't a big deal, distributed or not. If anything, if your distributed architecture makes reading a million items more costly than a single machine doing it then it's time to go back to the drawing board.
      - williamdclt 5y ago
        They mention that PlanetScale uses Vitess, so indeed it might be distributed
    - cperciva 5y ago
      I'm not saying that it's not crazy. Just that it's less crazy.
      - williamdclt 5y ago
        Fair enough!
  - Spooky23 5y ago
    25 years ago i built and ran a biggish database system that supported a reservation system. Even given the limitations of the time (RDBMS cost, 9GB disks, Sun kit, etc), our cost of good sold for that type of workload was exponentially less. (At scale) Today, I could probably run that company off my MacBook Pro and have room to spare. That said, the rationale for choosing this technology is cute: “After seeing a ton of the best GitHub engineers end up at PlanetScale and seeing the process GitHub went through to issue simple migrations, we chose to use their service.” If you use the same methodology to choose a database that the public uses to choose between Bud Light, Miller Lite, and Coors Light, expect a suboptimal outcome.
    - Nextgrid 5y ago
      I have seen so much over-engineered startup shit costing thousands in AWS fees that could run faster on my laptop.
      - Spooky23 5y ago
        Lol funny that you say that. I’m not a developer by trade (mostly an email/excel/PowerPoint jockey these days), but am a local sme on a few things, one of which was absolutely critical for a very key project. I was asked to mock-up a prototype of a core process that produced correct outputs. Dusted off my old toolbox and mocked it up in a combo of python and bash. Probably a total of 900-1000 lines of “code”. The mock-up, running on some little vm, ended up outperforming the production solution for quite some time! :)
  - akudha 5y ago
    One of my employers was using BigQuery. I was so scared that I might accidentally run queries and get a big bill, even though our tables weren’t that big. It is funny to look back, but getting huge bills without even realizing that we’re doing something wrong is very real possibility. Cloud vendors happily make their pricing opaque as it benefits them. I’d avoid even the best product in the industry, if their pricing is opaque. Or if there is a “Contact Us” button when there needn’t be.
  - XCSme 5y ago
    Still crazy, charging $0.15 for probably less than a few ms of computing power.
    - cperciva 5y ago
      If those million items need to be read from different servers -- as they might well, in a distributed database -- it's definitely not just a few ms of computing power. For reference, reading a million items of up to 1 kB each costs $0.125 with on-demand dynamodb.
      - rustc 5y ago
        > For reference, reading a million items of up to 1 kB each costs $0.125 with on-demand dynamodb. Is that the same counting method as PlanetScale's "row read"? That is, `select title from posts order by title limit 10` on a table with 10 million rows and no index on `title` would cost $1.25 per query?
        cperciva 5y ago
        DynamoDB doesn't have SQL queries; but yes, if you're performing an operation which reads 10 million items from ddb it will be absurdly expensive. It will also take an absurdly long time; by default ddb is limited to 40k read request units (= 80k eventually consistent reads of up to 1 kB) per second. Being so slow would probably make users realize they're doing something wrong.
        deleted 5y ago
        [deleted]
  - NicoJuicy 5y ago
    20 of those queries give me a VPS at Hetzner for a month. Or 1 query = 3 gb / snapshot storage for a month.
max_hammer 5y ago
I hate this pricing model My company in boarded `fivetran` to source data from different tools. Budget got exhausted in sourcing `iterable` data
smsm42 5y ago
TLDR: author forgot to create indexes in cloud-based MySQL database and paid too much for the queries which were run as full-table scans. Interestingly enough, some DBs (like Cassandra) would refuse scan-type queries unless specifically asked to. I wonder if cloud-based DBs which charge per row inspected could have such mode... Though of course it's their incentive not to.
mekster 5y ago
Why do people get on stuff some known people use blindly? That is such a bad habit like everyone getting on git and getting burned and now it's irreversible with all the existing ecosystem. How hard is it to just spin up a beefy cloud instance and run a MySQL of your own with whatever backup strategy you got and do things the way it is than getting bitten by using stuff you're not even familiar with.
deleted 5y ago
[deleted]
smoldesu 5y ago
Huh, learning about this "Superwall" product constitutes as my horror-story-of-the-day. It's paywalling as a service, just what the industry needed. Thankfully it appears to be quarantined to iOS right now, but God does it feel like we're headed right back into Stallman's predictions about how SAASS will ruin the landscape of commercial technology.
- jakemor 5y ago
  Two sides to every coin. Ethical developers need ways to make low prices work, which is impossible without good testing suites.
  - smoldesu 5y ago
    These days, if you're developing something for profit it's pretty hard to see your software as ethical. You're either trying to empower your user or trying to monetize them, the two will always fight one another and snuff the other out unless you, the developer, take a stand.I fully understand the market for proprietary software, but trying to define some ethical middle ground is just blatant lip service, nothing else.
- Nextgrid 5y ago
  It's better than ads though.
  - andybak 5y ago
    It's better than targetted ads built on intrusive tracking that also enables several other abusive business practices. It's not better than "good old fashioned ads". Heck I'd even be ok with targeted ads if it could be done without the rest of the "destroy civil society" that seems to come along for the ride.
    - Nextgrid 5y ago
      Even if you solve the privacy problem, there's still a problem with advertising which is that it's inherently at odds with the user's interests. An advertising-funded product will always prioritize engagement - they want you to "engage" with the product even if it means degrading the experience intentionally such as making a process manual or take more steps than necessary (so that you are exposed to more ads). The "destroy civil society" problem you mention is a direct consequence of the pursuit of engagement. In contrast, with a paid product, the company's interests are directly aligned with yours and they have no incentive to intentionally degrade the experience or get in your way any more than necessary. They don't care about how much you "engage" with the product as long as the bill gets paid (if anything, the less you engage the better as it uses less server resources).
rabuse 5y ago
Gotta love cloud pricing. This is why I colocate.
bigbillheck 5y ago
I'm not a DB expert, but "750k users in a month." doesn't sound like a quantity that you'd need to use some kind of fancy special tooling for.
- mkl95 5y ago
  In a world where Juicero raised $120m, selling overengineered solutions for simple problems is not necessarily a bad idea.
  - Nextgrid 5y ago
    The problem isn't selling (there's plenty of dubious or badly-priced products being around), it's that someone thought buying said product was a good idea.
- dahdum 5y ago
  It's far from big data even if they grow 10x in the next year, but if you are unfamiliar with database migrations and branches I can see the appeal of the product.
- tdrdt 5y ago
  I was thinking the same. It sound like this could be done on a private virtual server for less than $100/month.
xupybd 5y ago
No foreign keys to make migrations easier. That doesn't sound like the best trade off to me. Having the database constrained as much as possible makes maintenance so much easier. Many bugs don't escape into production as they're caught by the database constraints. Those that do get out do less damage to the data. I know scale comes with trade offs but that seems extreme to me.
- derekperkins 5y ago
  I'm a Vitess maintainer and I feel the same way. I don't plan to use any of the Online DDL because you'll have to pry my foreign keys out of my cold, dead hands. I understand the reasoning and limitations, but like you, the trade-off isn't worth it to me.
  - anglinb 5y ago
    I'm so curious, so you maintain Vitess but don't use it personally?
    - derekperkins 5y ago
      I do use it in production and have for years, just not the online schema changes. It's fantastic and FKs are supported in a single shard, which we use heavily.
      - romero-jk 5y ago
        If you take out online schema changes and sharding, what's the use case for vitess?
        derekperkins 5y ago
        Architected correctly, there's minimal need for cross-shard foreign keys. A common use case is sharding by tenant/customer id, which means that all records for a single customer live on a single shard. That lets you have all the FKs that you want, and any operations for that customer happen on a single shard, which gives you maximum speed and transactional guarantees.
Grimm1 5y ago
I feel like indexes are a pretty fundamental type of DB knowledge. In fact I'd say it's table stakes knowledge you should have if you're working with them. Further more, knowing that ForeignKeys typically apply an index to that column is also in my head basic knowledge. I'm sorry you got burnt, and congrats on learning a lesson, but you could have gotten the same knowledge by ever googling MySql ForeignKeys and saved yourself a headache. In fact it's like a big bullet point near the top of the docs page. "MySQL requires indexes on foreign keys and referenced keys so that foreign key checks can be fast and not require a table scan. In the referencing table, there must be an index where the foreign key columns are listed as the first columns in the same order. Such an index is created on the referencing table automatically if it does not exist. This index might be silently dropped later if you create another index that can be used to enforce the foreign key constraint. index_name, if given, is used as described previously." I'm not entirely sure why buzz around "developer learns basic knowledge" has this on the front page.
- Nextgrid 5y ago
  > I'm not entirely sure why buzz around "developer learns basic knowledge" has this on the front page. The problem is that in the old days, not knowing about indexes left you with an underperforming system or downtime. But in The Cloud™ it leaves you with an unreasonably huge bill and that somehow as an industry we're accepting this as normal.
  - aspenmayer 5y ago
    Using money to solve business problems is good business sense, but only if that’s the best way to spend that money. I agree with you that the status quo is normal, but nonsensical.
  - Grimm1 5y ago
    Which really is a head scratcher. You'd figure especially as a startup seeing a 5k oopsie isn't really as acceptable. Mistakes do happen and I don't mean any shade to this particular person (they'll never make this mistake again) but as an industry the aggregate consequence of this is you have a lot of waste and stupid choices that then have to be cleaned up when more knowledgeable (read highly paid) people are introduced later on. They'll have to clean up the mess which causes real business consequences that, and I've personally seen this, will directly impact bottom line and have no quick or easy solution to wiggle out of. Maybe it's acceptable for products like this because the balance between good engineering and company health probably aren't as cut and clear but stuff like this always makes me sad because it's such low hanging fruit, it doesn't require any real effort, just basic curiosity around your job.
AnotherGoodName 5y ago
I've seen things you people wouldn't believe. Millions burnt on consultants and licensing Oracle. I watched C series startups throwing it all away in a move to NoSQL. All those Amazon RDS fees will be lost in time.
- vincentpants 5y ago
  Like tears in the rain
  - phoenixdblack 5y ago
    These two might just be the best comments i have ever seen on HN
    - redisman 5y ago
      Time to die
  - xiphias2 5y ago
    That last metaphore was added by the actor himself...the director asked him to have some human showing in him, and now it's history.
    - utopcell 5y ago
      Actually, the whole part was added by the actor, not just the last metaphor.
      - xiphias2 5y ago
        I think we're both right :) https://en.wikipedia.org/wiki/Tears_in_rain_monologue?wprov=sfla1 https://en.wikipedia.org/wiki/Tears_in_rain_monologue?wprov=...
- deleted 5y ago
  [deleted]
samlambert 5y ago
This is definitely a lesson in the importance of indexes in general. We are well aware of the potential pitfalls with our current pricing. I’m happy to say we are nearly done modeling different metering rates for the product which would mean significantly lower bills for our users and avoid issues like this. It’s core to our mission that our product’s pricing is accessible and friendly to small teams. Part of being in beta was us wanting to figure out the best pricing based on usage patterns. That work is nearly done. As the post mentions we’ve credited back the amount.
- anglinb 5y ago
  Thanks Sam! As mentioned in the post, the PlantScale team was quick to credit our account for the overages and help us figure out what was going on. I'm personally super bullish on PlanetScale! With any new product there will be tradeoffs and rough edges but the positives, like easy migrations and database branches have definitely outweighed any difficulties.
  - brasetvik 5y ago
    Kudos for being open about your mistakes. Could you share a little bit about what your thought process was in general when picking a database technology? You call out "easy migrations and database branches" outweighing other quirks, so some pros and cons weighing must've happened :) Is it easy, for example, to test things in your dev environment with realistic amounts of data, and to get an understanding of how the queries will execute, etc? These seem somewhat basic, and would've probably caught this (also kinda basic, sorry :) problem early. (As in discovering "why is this query that should be a few ms with an index lookup taking so long?" early on)
    - anglinb 5y ago
      Thanks! The decision making process was pretty unsophisticated tbh. Basically I spent the last couple years working at GitHub as a security engineer and had been pretty comfortable with MySQL so wanted to stick with that. I had heard from our database team how annoying migrations were and I had previously locked a postgres database multiple times in production trying to deploy a migration, so MySQL + safe migrations + some of the best engineers I've worked with pouring all their time into PlanetScale, made a ton of sense. So basically a combination of proven underlying tech + believing in the team. The migration workflow is really cool, basically when we create a PR we branch our production database and apply any migrations that are included in that PR and then that branch is used in our per-PR review environment. (Just Heroku Review Apps), then when we merge the PR, we also merge the deploy request in PlanetScale. Database branching is a super powerful concept once you've leaned into it. We don't really do any sort of load testing in a dev environment. We have one customer who is also a co-founder of our company so we just deploy whatever changes we're unsure about (after automated testing) to his application and see what happens. If anything looks off in Grafana we'll make take a look but it's usually "good enough" or "totally broken", very rarely do we take time to make something 25% better if it already works. The time to fix vs speed of shipping features tradeoff doesn't make sense for us. In this specific case, the query was taking place in a background job so 10ms - 500ms didn't really matter to us so we didn't really measure the timing, if we had we may have noticed it was slow but kinda a testament to PlanetScale that we didn't even notice ;)
dreyfan 5y ago
Don’t use DB providers that charge for rows/data scanned. Use Amazon RDS or Google Cloud SQL or just install it yourself on a VM. Pay for CPU, memory, and storage instead.
- ldoughty 5y ago
  Rows returned model works really well for certain data loads (where all data customers use is customer-keyed).... This model also scales DOWN really well .. while still providing good scalable availability... That said, I DO agree with the sentiment of paying for a set performance level (clu, memory, storage), to provide predictable pricing.. obviously these guys were bit by the scaling capability. I do a lot of pet projects, and I find DynamoDB works really well because my pet projects cost $0 most months... And I don't have to worry about servers, maintenance, or what not... I'm happy to do that at work, but I don't want that for my friends & fun projects... And I've not seen a decent DB managed RDS for <$5/month
  - boulos 5y ago
    Disclosure: I used to work on Google Cloud. This is why BigQuery offers both models and lets you control the caps [1]. Buying fixed compute is effectively buying a throughput cap. Hard Quotas provide a similar function, but aren't a useful budgeting tool if you can't set them yourself. "Serverless" without limits is basically "infinite throughput, infinite budget" (though App Engine had quotas since day 1 and then budgets once charging was added). The default quotas give you some of that budget / throughput capping, but again if you can't lower them they might not help you. Either way, BQ won't drop ingestion or storage though because almost nobody wants their data deleted. As a provider, implementing strict budgets is impossible without having a fairly complex policy "if over $X/second stop all activity, oh except let me still do admin work, like adding indexes? Over $Y/second delete everything". I think having user adjustable quotas and throughput caps per "dimension" makes more sense but it puts the burden on the user and no provider offers good enough user control over quota. tl;dr: true budgets are hard to do, but every provider should strive to offer better quota/throughput controls. [1] https://cloud.google.com/bigquery/pricing https://cloud.google.com/bigquery/pricing [2] https://cloud.google.com/bigquery/docs/reservations-workload-management https://cloud.google.com/bigquery/docs/reservations-workload...
foreigner 5y ago
The real answer here is cost limiting. I don't want my cloud provider to keep working at the cost of an order of magnitude higher bill than I was expecting because of a bug in my code. I want to be able to set a billing limit and have them degrade or stop there service if I exceed the limit. AFAIK AWS doesn't have that. They do have the ability to send me alerts if my bill is unexpectedly high, but they still keep working until I go bankrupt. It's possible to use those alerts to implement your own "broke man's switch", but they don't have it built in.
- babayega2 5y ago
  That's why we use DigitalOcean a lot in Africa. You know upfront how much you will spend.
  - miyuru 5y ago
    You can calculate how much RDS is gonna cost you per month beforehand. In fact, it is slightly cheaper at AWS. Ondemand PostgreSQL, Single Node, 1vCPU, 1GB MEM, 10GB Storage is $15 at DO Ondemand PostgreSQL, Single Node, 2vCPU, 1GB MEM, 10GB Storage is $14.29 at AWS (db.t3.micro at us-east-2) if reserved for 1yr no upfront Reserved PostgreSQL, Single Node, 2vCPU, 1GB MEM, 10GB Storage is $10.57 at AWS (db.t3.micro at us-east-2) Or you can use ARM and go lower. Ondemand PostgreSQL, Single Node, 2vCPU, 1GB MEM, 10GB Storage is $12.83 at AWS (db.t4g.micro at us-east-2)
    - babayega2 5y ago
      My experience is that on AWS there are hidden costs. Paying for traffic, and other stuff.
fabian2k 5y ago
That pricing model seems rather inherently tricky to me, and also quite expensive. At $1.50 per 10 million rows read this can get very expensive the moment you do a full table scan on any non-trivial table. And while this example is a trivial case where you only need minimal database knowledge to ensure that no full table scan is necessary, many real world cases are much more complex. It also seems very expensive compared to just renting DBs by instance, if you put any real load onto this. I can see this being attractive if your use case only queries single rows by key, but it's essentially a big minefield for any query more complex than that. A database with a rather opaque query planner doesn't seem like a good fit for this kind of pricing.
- foepys 5y ago
  I agree with this. You are also only one bug in the query planner away from going bankrupt. Imagine Planetscale upgrading to a version which contains a small edge-case bug and now you owe them tens of thousands because of it.
  - samlambert 5y ago
    If we caused a bug that did that we would refund the customer of course.
XCSme 5y ago
I was actually considering PlanetScale, but them saying "Every time a query retrieves a row from the database, it is counted as a row read." when it's actually all the scanned rows, sounds intentionally confusing. "Retrieving" sounds like it should only be counted rows returned by a query.
sushsjsuauahab 5y ago
Ssh into an ec2 instance, install mysql, and you'll never pay more than $7.50 a month!
- speedgoose 5y ago
  You need to spend a lot more on AWS if you want good performances.
  - sushsjsuauahab 5y ago
    But I'm talking about the contrapositive, where if you don't need good performance, you don't need to spend 5k.
radu_floricica 5y ago
I'm just leaving this here: https://www.hetzner.com/dedicated-rootserver/ax161/configurator https://www.hetzner.com/dedicated-rootserver/ax161/configura... Draw that nice red slide all the way to the right. No, it's not storage. Yeah, it's actually affordable. Yeah, that was a sexual sound you just made. You do have to be prepared to know some basic sysadmin, or pay somebody to do it for you. My newest server has about 60 cores and half a tera of ram. Surprisingly, it's not uber sharp - I went with high core count so individual queries actually got slower for about 20%. But that load... you can't even tell if the cpu load gauge is working. I can't wait to fill it up :D Maybe this black friday season I'll get it to 10%.
- flippant 5y ago
  What are you using this for?
  - radu_floricica 5y ago
    www.couriermanager.com - SaaS for courier companies, basically. The new server is the common pool instance - I have others for dedicated clients.
- amenod 5y ago
  ...and Hetzner just started offering their services in the US a few days ago. (EDIT: not affiliated) If you do something stupid with your code at least you won't go bankrupt, only your service will be slower.
  - boomlinde 5y ago
    I used their VPS service before and I didn't have to pay for overage, but now that it's part of their cloud offering, overage is charged.
  - Nextgrid 5y ago
    With the performance of these servers you have a huge margin for stupidity before you even notice any slowdowns.
ihusasmiiu 5y ago
Let me understand please. These people are selling a commercial product and their team has no idea whatsoever of what an index is? And this is news?
- fabian2k 5y ago
  It sounds a bit more like they were confused by the automatic index creation for foreign keys they expected to be there. So they probably knew they'd need an index, just assumed this was implicit in the foreign key.
arpa 5y ago
Well this was an embarrasing read.
nrmitchi 5y ago
As the author touches on, the main problem here isn't learning about indexes. It's about "infinity scaling" working too well for people who do not understand the consequences. In no sane version of the world should "not adding a db index" lead to getting a 50x bill at the end of the month without knowing. I am a strong believer that services that are based on "scale infinitly" really need hard budget controls, and slower-scaling (unless explicitly overidden/allowed, of course). If I accidently push very non-performant code, I kind of expect my service to get less performant, quickly realize the problem, and fix it. I don't expect a service to seemingly-magically detect my poor code, increase my bill by a couple orders-of-magnitude, and only alert me hours (if not days) later.
- bufferoverflow 5y ago
  If you create an inefficient process, you should be responsible for the consequences. Why would you expect some third party to take the responsibility? If you create a horrible internal combustion engine, your gas station should not bear the costs.
  - thih9 5y ago
    In theory yes; in practice it’s very easy to push inefficient code to production by accident, as shown in the article.
  - zerd 5y ago
    If you create an inefficient internal combustion engine, you'd know because you have to go to the gas station every 5 miles. In this case it would be like someone was filling up the the gas without you knowing, and then a few weeks later you get the bill, and then you realize that your engine is inefficient.
- anglinb 5y ago
  Haha yep, I was like wait I'm used to getting feedback from the system telling me I messed up and this I barley noticed. PlanetScale has Query Statistics that are really useful for spotting slow queries but don't expose the "rows read" so you can't really tie this view back to billing. I think they're aware of this though and might expose that information.
crorella 5y ago
It amazes me that things so basic and fundamental like understanding the way indexes work are often overlooked or not leveraged
revskill 5y ago
Thanks. At least i'll never use PlatnetScale. A good service should have config for me to alert/prevent these kinds of money wasting cases. Imagine how many wasted $$$ they earned based on common knowledge that they should prevent for customers instead.
racl101 5y ago
Yeah, It cost me two bad months of high RDS fees to learn about indexes. $900 in total. Then a bro showed me one night about the magic of indexes. 5 minutes worth of advice saved me hundreds of dollars per month in the future and all he asked in return was for some beer and chicken wings. Now that is a good bro. I'm happy to say I've paid it forward myself.
Aeolun 5y ago
What a crazy way to do billing though. At larger scales (more rows, more customers, more queries) the costs become absolutely insane.
hodgesrm 5y ago
My company runs a cloud service for ClickHouse. We've spent a lot of time thinking about pricing. In the end we arrived at (VMs + allocated storage) * management uplift + support fee. It's not a newfangled serverless pricing model, but it's something I can reason about as a multi-decade developer of database apps. I feel comfortable that our users--mostly devs--feel the same way. We work to help people optimize the compute and storage down to the lowest levels that meet their SLAs. The most important property of the model is that costs are capped. One of the things that I hear a lot from users of products like BigQuery is that they get nailed on consumption costs that they can't relate in a meaningful way to application behavior. There's a lot of innovation around SaaS pricing for data services but I'm still not convinced that the more abstract models really help users. We ourselves get nailed by "weird shit" expenses like use cases that hammer Zookeeper in bad ways across availability zones. We eat them because we don't think users should need to understand internals to figure out costs. The best SaaS services abstract away operational details and have a simple billing model that doesn't break when something unexpected happens on your apps. Would love to hear alternative view points. It's not an easy problem.
j3th9n 5y ago
Are StackOverflow topics now eligible for HN as soon as you mention the savings? Or is mentioning some numbers about the users enough? Or did I just click on an advertisement? So many questions.
bborud 5y ago
One shouldn't assume people know anything (even the most basic thing) about databases just because they say they do.

236 points by anglinb 5y ago

NicoJuicy 5y ago

0,15 $ per query... The world has gone insane.

cperciva 5y ago
It's not quite so crazy when you phrase it as $0.15 for reading a million items from the database.
- williamdclt 5y ago
  No that's still crazy. Scanning a million items isn't a big workload.
  - friedman23 5y ago
    I agree, it might be distributed in some way which is driving up the cost?
    - Nextgrid 5y ago
      The greed is distributed across the cloud provider's C-suite for sure. To be fair to them they refunded the bill afterwards according to the article but IMO we should not be accepting this kind of pricing models as normal. A million items isn't a big deal, distributed or not. If anything, if your distributed architecture makes reading a million items more costly than a single machine doing it then it's time to go back to the drawing board.
    - williamdclt 5y ago
      They mention that PlanetScale uses Vitess, so indeed it might be distributed
  - cperciva 5y ago
    I'm not saying that it's not crazy. Just that it's less crazy.
    - williamdclt 5y ago
      Fair enough!
- Spooky23 5y ago
  25 years ago i built and ran a biggish database system that supported a reservation system. Even given the limitations of the time (RDBMS cost, 9GB disks, Sun kit, etc), our cost of good sold for that type of workload was exponentially less. (At scale) Today, I could probably run that company off my MacBook Pro and have room to spare. That said, the rationale for choosing this technology is cute: “After seeing a ton of the best GitHub engineers end up at PlanetScale and seeing the process GitHub went through to issue simple migrations, we chose to use their service.” If you use the same methodology to choose a database that the public uses to choose between Bud Light, Miller Lite, and Coors Light, expect a suboptimal outcome.
  - Nextgrid 5y ago
    I have seen so much over-engineered startup shit costing thousands in AWS fees that could run faster on my laptop.
    - Spooky23 5y ago
      Lol funny that you say that. I’m not a developer by trade (mostly an email/excel/PowerPoint jockey these days), but am a local sme on a few things, one of which was absolutely critical for a very key project. I was asked to mock-up a prototype of a core process that produced correct outputs. Dusted off my old toolbox and mocked it up in a combo of python and bash. Probably a total of 900-1000 lines of “code”. The mock-up, running on some little vm, ended up outperforming the production solution for quite some time! :)
- akudha 5y ago
  One of my employers was using BigQuery. I was so scared that I might accidentally run queries and get a big bill, even though our tables weren’t that big. It is funny to look back, but getting huge bills without even realizing that we’re doing something wrong is very real possibility. Cloud vendors happily make their pricing opaque as it benefits them. I’d avoid even the best product in the industry, if their pricing is opaque. Or if there is a “Contact Us” button when there needn’t be.
- XCSme 5y ago
  Still crazy, charging $0.15 for probably less than a few ms of computing power.
  - cperciva 5y ago
    If those million items need to be read from different servers -- as they might well, in a distributed database -- it's definitely not just a few ms of computing power. For reference, reading a million items of up to 1 kB each costs $0.125 with on-demand dynamodb.
    - rustc 5y ago
      > For reference, reading a million items of up to 1 kB each costs $0.125 with on-demand dynamodb. Is that the same counting method as PlanetScale's "row read"? That is, `select title from posts order by title limit 10` on a table with 10 million rows and no index on `title` would cost $1.25 per query?
      - cperciva 5y ago
        DynamoDB doesn't have SQL queries; but yes, if you're performing an operation which reads 10 million items from ddb it will be absurdly expensive. It will also take an absurdly long time; by default ddb is limited to 40k read request units (= 80k eventually consistent reads of up to 1 kB) per second. Being so slow would probably make users realize they're doing something wrong.
        deleted 5y ago
        [deleted]
- NicoJuicy 5y ago
  20 of those queries give me a VPS at Hetzner for a month. Or 1 query = 3 gb / snapshot storage for a month.

max_hammer 5y ago

I hate this pricing model My company in boarded `fivetran` to source data from different tools. Budget got exhausted in sourcing `iterable` data

smsm42 5y ago

TLDR: author forgot to create indexes in cloud-based MySQL database and paid too much for the queries which were run as full-table scans. Interestingly enough, some DBs (like Cassandra) would refuse scan-type queries unless specifically asked to. I wonder if cloud-based DBs which charge per row inspected could have such mode... Though of course it's their incentive not to.

mekster 5y ago

Why do people get on stuff some known people use blindly? That is such a bad habit like everyone getting on git and getting burned and now it's irreversible with all the existing ecosystem. How hard is it to just spin up a beefy cloud instance and run a MySQL of your own with whatever backup strategy you got and do things the way it is than getting bitten by using stuff you're not even familiar with.

deleted 5y ago

[deleted]

smoldesu 5y ago

Huh, learning about this "Superwall" product constitutes as my horror-story-of-the-day. It's paywalling as a service, just what the industry needed. Thankfully it appears to be quarantined to iOS right now, but God does it feel like we're headed right back into Stallman's predictions about how SAASS will ruin the landscape of commercial technology.

jakemor 5y ago
Two sides to every coin. Ethical developers need ways to make low prices work, which is impossible without good testing suites.
- smoldesu 5y ago
  These days, if you're developing something for profit it's pretty hard to see your software as ethical. You're either trying to empower your user or trying to monetize them, the two will always fight one another and snuff the other out unless you, the developer, take a stand.I fully understand the market for proprietary software, but trying to define some ethical middle ground is just blatant lip service, nothing else.
Nextgrid 5y ago
It's better than ads though.
- andybak 5y ago
  It's better than targetted ads built on intrusive tracking that also enables several other abusive business practices. It's not better than "good old fashioned ads". Heck I'd even be ok with targeted ads if it could be done without the rest of the "destroy civil society" that seems to come along for the ride.
  - Nextgrid 5y ago
    Even if you solve the privacy problem, there's still a problem with advertising which is that it's inherently at odds with the user's interests. An advertising-funded product will always prioritize engagement - they want you to "engage" with the product even if it means degrading the experience intentionally such as making a process manual or take more steps than necessary (so that you are exposed to more ads). The "destroy civil society" problem you mention is a direct consequence of the pursuit of engagement. In contrast, with a paid product, the company's interests are directly aligned with yours and they have no incentive to intentionally degrade the experience or get in your way any more than necessary. They don't care about how much you "engage" with the product as long as the bill gets paid (if anything, the less you engage the better as it uses less server resources).

rabuse 5y ago

Gotta love cloud pricing. This is why I colocate.

bigbillheck 5y ago

I'm not a DB expert, but "750k users in a month." doesn't sound like a quantity that you'd need to use some kind of fancy special tooling for.

mkl95 5y ago
In a world where Juicero raised $120m, selling overengineered solutions for simple problems is not necessarily a bad idea.
- Nextgrid 5y ago
  The problem isn't selling (there's plenty of dubious or badly-priced products being around), it's that someone thought buying said product was a good idea.
dahdum 5y ago
It's far from big data even if they grow 10x in the next year, but if you are unfamiliar with database migrations and branches I can see the appeal of the product.
tdrdt 5y ago
I was thinking the same. It sound like this could be done on a private virtual server for less than $100/month.

xupybd 5y ago

No foreign keys to make migrations easier. That doesn't sound like the best trade off to me. Having the database constrained as much as possible makes maintenance so much easier. Many bugs don't escape into production as they're caught by the database constraints. Those that do get out do less damage to the data. I know scale comes with trade offs but that seems extreme to me.

derekperkins 5y ago
I'm a Vitess maintainer and I feel the same way. I don't plan to use any of the Online DDL because you'll have to pry my foreign keys out of my cold, dead hands. I understand the reasoning and limitations, but like you, the trade-off isn't worth it to me.
- anglinb 5y ago
  I'm so curious, so you maintain Vitess but don't use it personally?
  - derekperkins 5y ago
    I do use it in production and have for years, just not the online schema changes. It's fantastic and FKs are supported in a single shard, which we use heavily.
    - romero-jk 5y ago
      If you take out online schema changes and sharding, what's the use case for vitess?
      - derekperkins 5y ago
        Architected correctly, there's minimal need for cross-shard foreign keys. A common use case is sharding by tenant/customer id, which means that all records for a single customer live on a single shard. That lets you have all the FKs that you want, and any operations for that customer happen on a single shard, which gives you maximum speed and transactional guarantees.

Grimm1 5y ago

I feel like indexes are a pretty fundamental type of DB knowledge. In fact I'd say it's table stakes knowledge you should have if you're working with them. Further more, knowing that ForeignKeys typically apply an index to that column is also in my head basic knowledge. I'm sorry you got burnt, and congrats on learning a lesson, but you could have gotten the same knowledge by ever googling MySql ForeignKeys and saved yourself a headache. In fact it's like a big bullet point near the top of the docs page. "MySQL requires indexes on foreign keys and referenced keys so that foreign key checks can be fast and not require a table scan. In the referencing table, there must be an index where the foreign key columns are listed as the first columns in the same order. Such an index is created on the referencing table automatically if it does not exist. This index might be silently dropped later if you create another index that can be used to enforce the foreign key constraint. index_name, if given, is used as described previously." I'm not entirely sure why buzz around "developer learns basic knowledge" has this on the front page.

Nextgrid 5y ago
> I'm not entirely sure why buzz around "developer learns basic knowledge" has this on the front page. The problem is that in the old days, not knowing about indexes left you with an underperforming system or downtime. But in The Cloud™ it leaves you with an unreasonably huge bill and that somehow as an industry we're accepting this as normal.
- aspenmayer 5y ago
  Using money to solve business problems is good business sense, but only if that’s the best way to spend that money. I agree with you that the status quo is normal, but nonsensical.
- Grimm1 5y ago
  Which really is a head scratcher. You'd figure especially as a startup seeing a 5k oopsie isn't really as acceptable. Mistakes do happen and I don't mean any shade to this particular person (they'll never make this mistake again) but as an industry the aggregate consequence of this is you have a lot of waste and stupid choices that then have to be cleaned up when more knowledgeable (read highly paid) people are introduced later on. They'll have to clean up the mess which causes real business consequences that, and I've personally seen this, will directly impact bottom line and have no quick or easy solution to wiggle out of. Maybe it's acceptable for products like this because the balance between good engineering and company health probably aren't as cut and clear but stuff like this always makes me sad because it's such low hanging fruit, it doesn't require any real effort, just basic curiosity around your job.

AnotherGoodName 5y ago

I've seen things you people wouldn't believe. Millions burnt on consultants and licensing Oracle. I watched C series startups throwing it all away in a move to NoSQL. All those Amazon RDS fees will be lost in time.

vincentpants 5y ago
Like tears in the rain
- phoenixdblack 5y ago
  These two might just be the best comments i have ever seen on HN
  - redisman 5y ago
    Time to die
- xiphias2 5y ago
  That last metaphore was added by the actor himself...the director asked him to have some human showing in him, and now it's history.
  - utopcell 5y ago
    Actually, the whole part was added by the actor, not just the last metaphor.
    - xiphias2 5y ago
      I think we're both right :) https://en.wikipedia.org/wiki/Tears_in_rain_monologue?wprov=sfla1 https://en.wikipedia.org/wiki/Tears_in_rain_monologue?wprov=...
deleted 5y ago
[deleted]

samlambert 5y ago

This is definitely a lesson in the importance of indexes in general. We are well aware of the potential pitfalls with our current pricing. I’m happy to say we are nearly done modeling different metering rates for the product which would mean significantly lower bills for our users and avoid issues like this. It’s core to our mission that our product’s pricing is accessible and friendly to small teams. Part of being in beta was us wanting to figure out the best pricing based on usage patterns. That work is nearly done. As the post mentions we’ve credited back the amount.

anglinb 5y ago
Thanks Sam! As mentioned in the post, the PlantScale team was quick to credit our account for the overages and help us figure out what was going on. I'm personally super bullish on PlanetScale! With any new product there will be tradeoffs and rough edges but the positives, like easy migrations and database branches have definitely outweighed any difficulties.
- brasetvik 5y ago
  Kudos for being open about your mistakes. Could you share a little bit about what your thought process was in general when picking a database technology? You call out "easy migrations and database branches" outweighing other quirks, so some pros and cons weighing must've happened :) Is it easy, for example, to test things in your dev environment with realistic amounts of data, and to get an understanding of how the queries will execute, etc? These seem somewhat basic, and would've probably caught this (also kinda basic, sorry :) problem early. (As in discovering "why is this query that should be a few ms with an index lookup taking so long?" early on)
  - anglinb 5y ago
    Thanks! The decision making process was pretty unsophisticated tbh. Basically I spent the last couple years working at GitHub as a security engineer and had been pretty comfortable with MySQL so wanted to stick with that. I had heard from our database team how annoying migrations were and I had previously locked a postgres database multiple times in production trying to deploy a migration, so MySQL + safe migrations + some of the best engineers I've worked with pouring all their time into PlanetScale, made a ton of sense. So basically a combination of proven underlying tech + believing in the team. The migration workflow is really cool, basically when we create a PR we branch our production database and apply any migrations that are included in that PR and then that branch is used in our per-PR review environment. (Just Heroku Review Apps), then when we merge the PR, we also merge the deploy request in PlanetScale. Database branching is a super powerful concept once you've leaned into it. We don't really do any sort of load testing in a dev environment. We have one customer who is also a co-founder of our company so we just deploy whatever changes we're unsure about (after automated testing) to his application and see what happens. If anything looks off in Grafana we'll make take a look but it's usually "good enough" or "totally broken", very rarely do we take time to make something 25% better if it already works. The time to fix vs speed of shipping features tradeoff doesn't make sense for us. In this specific case, the query was taking place in a background job so 10ms - 500ms didn't really matter to us so we didn't really measure the timing, if we had we may have noticed it was slow but kinda a testament to PlanetScale that we didn't even notice ;)

dreyfan 5y ago

Don’t use DB providers that charge for rows/data scanned. Use Amazon RDS or Google Cloud SQL or just install it yourself on a VM. Pay for CPU, memory, and storage instead.

ldoughty 5y ago
Rows returned model works really well for certain data loads (where all data customers use is customer-keyed).... This model also scales DOWN really well .. while still providing good scalable availability... That said, I DO agree with the sentiment of paying for a set performance level (clu, memory, storage), to provide predictable pricing.. obviously these guys were bit by the scaling capability. I do a lot of pet projects, and I find DynamoDB works really well because my pet projects cost $0 most months... And I don't have to worry about servers, maintenance, or what not... I'm happy to do that at work, but I don't want that for my friends & fun projects... And I've not seen a decent DB managed RDS for <$5/month
- boulos 5y ago
  Disclosure: I used to work on Google Cloud. This is why BigQuery offers both models and lets you control the caps [1]. Buying fixed compute is effectively buying a throughput cap. Hard Quotas provide a similar function, but aren't a useful budgeting tool if you can't set them yourself. "Serverless" without limits is basically "infinite throughput, infinite budget" (though App Engine had quotas since day 1 and then budgets once charging was added). The default quotas give you some of that budget / throughput capping, but again if you can't lower them they might not help you. Either way, BQ won't drop ingestion or storage though because almost nobody wants their data deleted. As a provider, implementing strict budgets is impossible without having a fairly complex policy "if over $X/second stop all activity, oh except let me still do admin work, like adding indexes? Over $Y/second delete everything". I think having user adjustable quotas and throughput caps per "dimension" makes more sense but it puts the burden on the user and no provider offers good enough user control over quota. tl;dr: true budgets are hard to do, but every provider should strive to offer better quota/throughput controls. [1] https://cloud.google.com/bigquery/pricing https://cloud.google.com/bigquery/pricing [2] https://cloud.google.com/bigquery/docs/reservations-workload-management https://cloud.google.com/bigquery/docs/reservations-workload...

foreigner 5y ago

The real answer here is cost limiting. I don't want my cloud provider to keep working at the cost of an order of magnitude higher bill than I was expecting because of a bug in my code. I want to be able to set a billing limit and have them degrade or stop there service if I exceed the limit. AFAIK AWS doesn't have that. They do have the ability to send me alerts if my bill is unexpectedly high, but they still keep working until I go bankrupt. It's possible to use those alerts to implement your own "broke man's switch", but they don't have it built in.

babayega2 5y ago
That's why we use DigitalOcean a lot in Africa. You know upfront how much you will spend.
- miyuru 5y ago
  You can calculate how much RDS is gonna cost you per month beforehand. In fact, it is slightly cheaper at AWS. Ondemand PostgreSQL, Single Node, 1vCPU, 1GB MEM, 10GB Storage is $15 at DO Ondemand PostgreSQL, Single Node, 2vCPU, 1GB MEM, 10GB Storage is $14.29 at AWS (db.t3.micro at us-east-2) if reserved for 1yr no upfront Reserved PostgreSQL, Single Node, 2vCPU, 1GB MEM, 10GB Storage is $10.57 at AWS (db.t3.micro at us-east-2) Or you can use ARM and go lower. Ondemand PostgreSQL, Single Node, 2vCPU, 1GB MEM, 10GB Storage is $12.83 at AWS (db.t4g.micro at us-east-2)
  - babayega2 5y ago
    My experience is that on AWS there are hidden costs. Paying for traffic, and other stuff.

fabian2k 5y ago

That pricing model seems rather inherently tricky to me, and also quite expensive. At $1.50 per 10 million rows read this can get very expensive the moment you do a full table scan on any non-trivial table. And while this example is a trivial case where you only need minimal database knowledge to ensure that no full table scan is necessary, many real world cases are much more complex. It also seems very expensive compared to just renting DBs by instance, if you put any real load onto this. I can see this being attractive if your use case only queries single rows by key, but it's essentially a big minefield for any query more complex than that. A database with a rather opaque query planner doesn't seem like a good fit for this kind of pricing.

foepys 5y ago
I agree with this. You are also only one bug in the query planner away from going bankrupt. Imagine Planetscale upgrading to a version which contains a small edge-case bug and now you owe them tens of thousands because of it.
- samlambert 5y ago
  If we caused a bug that did that we would refund the customer of course.

XCSme 5y ago

I was actually considering PlanetScale, but them saying "Every time a query retrieves a row from the database, it is counted as a row read." when it's actually all the scanned rows, sounds intentionally confusing. "Retrieving" sounds like it should only be counted rows returned by a query.

sushsjsuauahab 5y ago

Ssh into an ec2 instance, install mysql, and you'll never pay more than $7.50 a month!

speedgoose 5y ago
You need to spend a lot more on AWS if you want good performances.
- sushsjsuauahab 5y ago
  But I'm talking about the contrapositive, where if you don't need good performance, you don't need to spend 5k.

radu_floricica 5y ago

I'm just leaving this here: https://www.hetzner.com/dedicated-rootserver/ax161/configurator https://www.hetzner.com/dedicated-rootserver/ax161/configura... Draw that nice red slide all the way to the right. No, it's not storage. Yeah, it's actually affordable. Yeah, that was a sexual sound you just made. You do have to be prepared to know some basic sysadmin, or pay somebody to do it for you. My newest server has about 60 cores and half a tera of ram. Surprisingly, it's not uber sharp - I went with high core count so individual queries actually got slower for about 20%. But that load... you can't even tell if the cpu load gauge is working. I can't wait to fill it up :D Maybe this black friday season I'll get it to 10%.

flippant 5y ago
What are you using this for?
- radu_floricica 5y ago
  www.couriermanager.com - SaaS for courier companies, basically. The new server is the common pool instance - I have others for dedicated clients.
amenod 5y ago
...and Hetzner just started offering their services in the US a few days ago. (EDIT: not affiliated) If you do something stupid with your code at least you won't go bankrupt, only your service will be slower.
- boomlinde 5y ago
  I used their VPS service before and I didn't have to pay for overage, but now that it's part of their cloud offering, overage is charged.
- Nextgrid 5y ago
  With the performance of these servers you have a huge margin for stupidity before you even notice any slowdowns.

ihusasmiiu 5y ago

Let me understand please. These people are selling a commercial product and their team has no idea whatsoever of what an index is? And this is news?

fabian2k 5y ago
It sounds a bit more like they were confused by the automatic index creation for foreign keys they expected to be there. So they probably knew they'd need an index, just assumed this was implicit in the foreign key.

arpa 5y ago

Well this was an embarrasing read.

nrmitchi 5y ago

As the author touches on, the main problem here isn't learning about indexes. It's about "infinity scaling" working too well for people who do not understand the consequences. In no sane version of the world should "not adding a db index" lead to getting a 50x bill at the end of the month without knowing. I am a strong believer that services that are based on "scale infinitly" really need hard budget controls, and slower-scaling (unless explicitly overidden/allowed, of course). If I accidently push very non-performant code, I kind of expect my service to get less performant, quickly realize the problem, and fix it. I don't expect a service to seemingly-magically detect my poor code, increase my bill by a couple orders-of-magnitude, and only alert me hours (if not days) later.

bufferoverflow 5y ago
If you create an inefficient process, you should be responsible for the consequences. Why would you expect some third party to take the responsibility? If you create a horrible internal combustion engine, your gas station should not bear the costs.
- thih9 5y ago
  In theory yes; in practice it’s very easy to push inefficient code to production by accident, as shown in the article.
- zerd 5y ago
  If you create an inefficient internal combustion engine, you'd know because you have to go to the gas station every 5 miles. In this case it would be like someone was filling up the the gas without you knowing, and then a few weeks later you get the bill, and then you realize that your engine is inefficient.
anglinb 5y ago
Haha yep, I was like wait I'm used to getting feedback from the system telling me I messed up and this I barley noticed. PlanetScale has Query Statistics that are really useful for spotting slow queries but don't expose the "rows read" so you can't really tie this view back to billing. I think they're aware of this though and might expose that information.

crorella 5y ago

It amazes me that things so basic and fundamental like understanding the way indexes work are often overlooked or not leveraged

revskill 5y ago

Thanks. At least i'll never use PlatnetScale. A good service should have config for me to alert/prevent these kinds of money wasting cases. Imagine how many wasted $$$ they earned based on common knowledge that they should prevent for customers instead.

racl101 5y ago

Yeah, It cost me two bad months of high RDS fees to learn about indexes. $900 in total. Then a bro showed me one night about the magic of indexes. 5 minutes worth of advice saved me hundreds of dollars per month in the future and all he asked in return was for some beer and chicken wings. Now that is a good bro. I'm happy to say I've paid it forward myself.

Aeolun 5y ago

What a crazy way to do billing though. At larger scales (more rows, more customers, more queries) the costs become absolutely insane.

hodgesrm 5y ago

My company runs a cloud service for ClickHouse. We've spent a lot of time thinking about pricing. In the end we arrived at (VMs + allocated storage) * management uplift + support fee. It's not a newfangled serverless pricing model, but it's something I can reason about as a multi-decade developer of database apps. I feel comfortable that our users--mostly devs--feel the same way. We work to help people optimize the compute and storage down to the lowest levels that meet their SLAs. The most important property of the model is that costs are capped. One of the things that I hear a lot from users of products like BigQuery is that they get nailed on consumption costs that they can't relate in a meaningful way to application behavior. There's a lot of innovation around SaaS pricing for data services but I'm still not convinced that the more abstract models really help users. We ourselves get nailed by "weird shit" expenses like use cases that hammer Zookeeper in bad ways across availability zones. We eat them because we don't think users should need to understand internals to figure out costs. The best SaaS services abstract away operational details and have a simple billing model that doesn't break when something unexpected happens on your apps. Would love to hear alternative view points. It's not an easy problem.

j3th9n 5y ago

Are StackOverflow topics now eligible for HN as soon as you mention the savings? Or is mentioning some numbers about the users enough? Or did I just click on an advertisement? So many questions.

bborud 5y ago

One shouldn't assume people know anything (even the most basic thing) about databases just because they say they do.