11 ms·
Startup Mistakes: Choice of Datastore
- onion2k 9y agoJust pick something and build an MVP with it. Then get on with the hard part of finding paying customers. You can fix the bad tech decisions later. Without customers it won't matter which database you used before your startup failed.
- StavrosK 9y agoI agree, but it's better if later you only have to fix ten bad decisions instead of twenty.
- mamcx 9y agoIs even worse to fix 1, but big. DataStore is 1 that is too big to pick wrong.
- izolate 9y agoExactly. This article typifies premature optimisation. If MongoDB lets you prototype faster, then use it until you have enough paying customers to warrant a refactor.
- StavrosK 9y agoI think you're greatly underestimating both the effort of changing a datastore from under a live application and the free time developers in a startup have.
- pandama 9y agoOh, wait. So you should actually think about what you're trying to build and what would work best for it? I just read this hilarious blog post that said the opposite. This is all too hard, I'll just use Postgres(R).
- StavrosK 9y agoBy "Said the oppposite" do you mean this part?: > Think before you pick a database. If you insist on not thinking, pick PostgreSQL. Trust me.
- deleted 9y ago[deleted]
- mannykannot 9y agoKnuth's aphorism on premature optimization is perhaps the most misunderstood point in software development. It applies to code-level algorithmic optimization, not architectural decisions, where a bad choice can lead down a very long dead-end. You can recover if you recognize this in time, but you are better of if you do not have to. Obligatory rewriting is not what you want to be doing just as competitors take notice of your initial success.
- notacoward 9y ago> If MongoDB lets you prototype faster Does it? Even a prototype involves some iteration, which can be derailed by the kind of missing-schema problems that the OP mentions. Throw in a few operational problems, and your prototype isn't really any faster than something with better features and reliability. What's better for that first few lines of code on your laptop might not even get you to a prototype.
- mamcx 9y agoBIG ERROR. Like, MONUMENTAL. That something (if chose very wrong) will totally derail you progress and will cost a lot of fix it later. Is incredible. Nobody remember that the most cost effective way to fix a problem is in the early stages? And, yes, the best overall primary datastore, like 90% of the cases, is a RDBMS. Very few actually need to use something else.
- tanglefuzz 9y agoWhat's the use of fixing something in a cost effective way if your startup is going to fail anyway? You need to validate and test first. Get customers ASAP. If that means writing shitty code and using a shitty database or not even a database at all, than that's great. You can always convert to a better system afterwards. I do agree that the longer you wait, the harder it gets to rewrite code and convert to different systems, but the most important thing for any startup is to get their idea validated as soon as possible.
- StavrosK 9y agoThis is just the balance fallacy. Relational and document databases aren't equally likely to be the type you need. Most of the times you have relational data, so you want relational databases. Pick the datastore that's more likely to fit your data from the start.
- geophile 9y agoYes and no. Yes, choose carefully at the beginning, and use relational. More specifically, use Postgres. But choose the right things to worry about at the beginning. So worry about something that can accommodate changing requirements, i.e., a relational database. Don't worry about scalability. You will be very, very lucky to ever have that problem. Worry about it then.
- StavrosK 9y agoAlso, if you find that some data is better suited for some other datastore, just move that data off. For example, I start with Postgres but almost always end up storing session data in Redis, for a bit more speed.
- hashkb 9y ago...as long as that something is Postgres.
- jrobn 9y agoI would suggest that something be PostgreSQL. With JSONB and and all the other goodies in version 10 they are too numerous to list there is no reason not to use it for 90% of apps. It WILL scale until you need something more specialized. Postgres is far from a boring relational database. Don’t make the mistake of building a mvp on rickety stilts and then swap them out later for proper columns while trying to run.
- hvidgaard 9y agoPostgre will be good for 99.99% of all workloads. And when you need something else, you should have enough revenue to migrate the parts that make sense to a different datastore.
- BozeWolf 9y agoAgree! Most of us techies think tech is more important than sales. We also loathe bad design decisions. But your customers (in most cases) don't care! But I always wonder exactly "what" is easier with mongo or another NoSQL db. Just get a relation db like PostgreSQL, MySQL or SQLite for that matter. If your site starts to get performance problems go to the pub, celebrate a bit of success and then optimise!
- maephisto 9y agoYup, people stuck on optimising (or overthinking which datastore to use) before actually having a business are missing the point.
- BozeWolf 9y agoOr they miss the point when overthinking the whole code for the program/website. Get it done. Start finding customers. The worst code I worked with was at travel agency startup and they are REALLY successful now. It was REALLY REALLY bad and hard to maintain. The owner of the business just said: well, bugs happen. But when a customer (not too many of course) encounters one, they probably call the helpdesk. Sounds stupid, worked great, business wise. 4 years later, I am sure they still have problems with the crappy code, but the business is fine!
- maephisto 9y agoAgree, the mantra of success is: GET SHIT DONE! Customers do not care what's your code coverage or how maintainable is your code. Or if it's Mongo or Postgres. They want features, they want it now, and if you can't deliver that because you are too busy refactoring that's bad.
- mannykannot 9y agoPeople starting a business need to be able to consider more than one thing at a time.
- emodendroket 9y agoI don't disagree, but you'd probably rather pick "something" tried, true, and appropriate for most use cases, and that's RDBMS in this case.
- jijji 9y agoyou pick "something" based on the requirements of the application or whatever solution you're building. if your requirements mention that you need to have multiple people accessing the database at the same time, RDBMS environments like MySQL, Postgres, Oracle would be appropriate. If it is a single application instance with no multi-user access, SQLite might be appropriate. However, if you're choosing SQLite, FileMaker, FoxPro, MS Access, etc and your requirements specify that you needed multi-user access, then you've made a large mistake.
- elvinyung 9y agoI feel like MongoDB/NoSQL is a horse that's been beaten so much in the last few years that no one is actually making that choice nowadays. Hasn't everyone already learned to stick with Postgres?
- StavrosK 9y agoUnfortunately, no. Some friends of mine picked another nonrel database (not Mongo) for their new company, which prompted this article. I still have to give this advice quite often.
- ohstopitu 9y agoI think that there are various reasons to pick DBs. Blindly picking a SQL DB (mysql/postgres etc.) is quite expensive from the get-go (a production ready mysql/postgres would cost ~30$/m). Mongo costs ~$10/m (MongoDB Atlas), Google's Datastore is Pay as you Go (so your initial cost is close to $0 till you get paying customers), AWS's DynamoDB is similarly priced as well. Sure, sadly all those noSQL solutions get really expensive as your usage goes up to normal non-webscale proportions, but at that point you have the $ to invest in a SQL solution. The above was mentioned with bootstrapped startups/services in mind. Not your usual million funded valley companies.
- StavrosK 9y agoHmm, how are you deploying things that a database costs $30/m? I just put it on the same server as the application worker and split it off if I need more performance later.
- ohstopitu 9y agomaybe I'm doing this wrong, but if I use RDS / Cloud SQL / Compose etc - a normal base cluster + another failover cluster + hourly/daily backups (based on your release schedule) all add up to around 30 - 50 based on various cloud providers.
- redwood 9y agoSo he used a database 7 years ago that was about one year old at the time and had a bad experience with it and now continues to judge it to this day without tracking the evolution since, including schemas in the current release candidate.
- StavrosK 9y agoNo, the article isn't about Mongo. Quite the opposite.
- williamstein 9y agoTL;DR: "just use Postgres"
- frik 9y agoTL;DR: "just use an MySQL or Postgres or SQLite"
- StavrosK 9y agoDon't use SQLite for production, for all the love I have for it, the client libraries are usually locking accesses and don't work properly with concurrent reads/writes.
- codazoda 9y agoIf your site is light on writes and always will be, I think SQLite is a good choice. Especially if it reduces the complexity of the system up front. I use it for a lot of personal projects that never gain more than a few thousand impressions a month. SQLite is also not too difficult to switch to a more advanced SQL in the future.
- StavrosK 9y agoI agree, I run a production (very write-light) site on SQLite and it has been great, but the site is pretty much almost static.
- alkonaut 9y agoIt's designed for single client, and works very well in production for single client. Mobile apps, desktop apps, and anywhere you can serialize access it works great. Let's say "don't use it for production for a multiuser server app" then I agree. But that isn't really a supported scenario at all.
- emodendroket 9y agoBut in all those scenarios the downsides of just writing JSON to disk or something are smaller too.
- jlebrech 9y agoIf your datastore has no schema your application must be the schema, you have to version your models and have those migrate when old data is accessed.
- gcb0 9y ago> startup mistakes > .io domain oh my.
- no_one_ever 9y agoJust because the post belongs to an .io domain doesn't mean you are allowed to automatically negate what the author may be trying to say.
- dordoka 9y agoWhat's the problem with a .io domain? Honest question.
- codazoda 9y agoThere was a recent failure that affected everyone using the TLD and a popular blog post explaining it and suggesting that you use something more reliable. Some of the comments on the article pointed out that the same company runs .org and a handful of other reliable TLD's.
- dordoka 9y agoThanks very much
- cocktailpeanuts 9y agoOP forgot the biggest mistake for choice of datastore: Blockchain.
- tabeth 9y agoTo the people saying "just do Postgres", what would you say to a startup that wants to create offline first apps? CouchDB, for example is way better at that than Postgres. 1) Use CouchDB and Postgres? 2) Somehow implement revisions and Postgres? 3) Use Postgres anyway and scrap the offline first?
- StavrosK 9y agoIf you want to create offline-first apps, I would use Postgres as the main datastore and use Couch to sync data between client and server. You'd have to decide which of your data would live where (or if you wanted to use Couch as a way to transfer data from the server to the client). Couch is a very good datastore for that use case, though, so I would definitely use it in some capacity for your purpose.
- tabeth 9y agoCan you expand on this? Do you mean you'd use Postgres as your main store and periodically "flush" data from CouchDB to Postgres (probably as JSON)? I don't see how that would work reliably. On the client would you source CouchDB or Postgres? Presumably you'd access CouchDB directly, but then why even use Postgres (for that subset of data, anyway).
- StavrosK 9y agoIt really depends on what your data looks like. If it's just game settings and state, put them in Couch and that's it, and keep user data like payments and activity there. If it's data you're going to want to run analytics on and sync to the client, you're probably going to have to store it in both places, I think.
- wonderous 9y ago@StavrosK: (aka the author) - what is the TLDR of the article? Ask since there appear to be a number of users including myself that appear to not get the intent of your article. — Meta-comment: Feel like if the poster is self-identifying as the author when posting the link, it’s verified via say email/domain, an HN username has been ID’d in the past as the author, etc. — it should be automatically obvious in post, comments, etc.
- StavrosK 9y agoIt's what I explicitly call out as a TLDR in the article itself: "Think before you pick a database. If you insist on not thinking, pick PostgreSQL. Trust me." Basically, "Postgres is a better default".
- wonderous 9y agoThere is no TLDR, summary, etc. clearly marked as such; by one estimate it takes 140 seconds or 468 words to get to the start of the sentence you quoted above; which is strange given your point is to stick to best practices unless there’s a strong reason not to do so. Strongly recommend moving that info to the top of your post.
- codazoda 9y agoThis article is a bit combative but I generally agree with the idea. I use both MySQL and MongoDB in my daily work on a classifieds site that does a few hundred million pageviews a month. Both are pretty solid performers. The article is correct that with Mongo you just move the schema into the code (new versions not withstanding). I think it's nicer to have the schema on the database side but it's really just user preference. We typically end up creating a schema class and defining it up-front anyway. There is also a small subset of cases where not having a schema at all is actually a benefit. Starting with a popular SQL engine is a really good tried and tested method though.
- beagle3 9y agoIt's more than user preference in the long run; When your schema is encoded in the database (with triggers, constraints, foreign keys, the whole shebang), you can be sure that whichever way you access the database, it is still consistent. When your schema is "in the code", even if you completely abstract it into a library, it means that the quick-one-off Ruby/Perl/C#/Python script will not have the integrity checks, and may corrupt your DB.
- mooreed 9y ago1) Read the image captions - they made my day. 2) Pick Postgres unless you have a compelling reason not to. 2b) Use the time gained by not over-engineering upfront, to focus on users and business logic. 3) Optimize/Evolve away from Postgres later as needed. 4) Profit.
- jihadjihad 9y agoThe reason MongoDB is simultaneously lauded and derided is because there is a bimodal distribution of people using it: those who build databases for a living, and those who query them. There is something to be said for being able to rapidly prototype ideas, especially if your primary skillset is jockeying JSON in the context of web/app development. However, deciding whether or not NoSQL is the right fit for you or your project/business depends on how much of your time you will be spending getting down and dirty with the database.
- maephisto 9y agoyes, yes, yes.
- bmh_ca 9y agoFWIW, my personal anecdote: I chose Google App Engine for a large enterprise application almost a decade ago and it's been absolutely, undeniably one of the best choices we made. The application is locked into Google, but that hasn't proven a problem yet and can be designed around if need be.
- StavrosK 9y agoMy personal anecdote is the exact opposite of yours. I picked GAE for one of my personal projects years ago and it has been terrible. You can't do certain kinds of lookups unless you build the indexes first, there are no delete cascades, no intra-table constraints, etc. I suspect the difference between us is that you spent the time working around these problems, whereas I expected it to just work.
- lowbloodsugar 9y agoWow. This fell off the front page in the time it took for me to drive to work. Guess the NoSQL crowd ain't got time for that. While I'm generally sympathetic to your post, there are some things that are red flags. >If you have ten services accessing the same database and sharing data between themselves Whoever access the database schema owns it. If you have ten systems accessing your database then ten teams own it. And if ten teams own it then nobody owns it. Nobody can change it. Seen this at a successful start-up that got big, but then couldn't rev order management, because every team had their finger in the pie, we couldn't do a schema update without breaking everyone, and of course one team was "under tremendous pressure to hit a major milestone and we just can't do that now" for over a year. Now I would turn this around into a win for RDBMS by suggesting the use of functions or stored procedures: with an RDBMS we can construct an API, and then we can version those APIs. And then the team that owns the database can do what they like. That said, we can do the same with NoSQL databases by not allowing other teams to access them. The team that owns the NoSQL database is required to maintain an API for it. I've only ever had nightmares with other teams coding against my schema. GraphQL worries me in that respect and I'd love to hear how people here have fared with long lasting GraphQL, in the real world. >Django, for example, makes migrations trivial, as you just change your application-level classes and the database gets migrated automatically. I've had automated migration systems grind to a halt and leave the DB fucked too. >Priscilla used a graph database when her data was relational. Her husband, furious, filed for divorce. There are no schema that are "relational" but not "a graph". However there are plenty of schema where a graph database is a natural fit but that require either one-table-per-node-type or building a graph model on top of your RDBMS (e.g. an Entity-Attribute schema). Oy. There are also many schema where there is only one entity type, but every join is against itself, and we're looking to join all the way out to the clique. In this case would you suggest an RDBMS, and then put the iteration in the application? You suggested earlier that making up for the inadequacies of the datastore in the application is a bad idea. I've got a graph application that uses NoSQL and it was the right call. An RDBMS would have allowed us to write something that worked for simple cases, but that would have brought the system to its knees based on some customer usage. The solution for the RDBMS would be the same as what we had to do for NoSQL. But up to that moment, the NoSQL allowed us to iterate far faster than an RDBMS. >For example, if you later need to compile a list of all the brands of all the products on your store, an RDBMS can easily do that by reading the “brands” table Only if you built a brands table. You can't argue that we can't predict the future, so use an RDBS because its easy to change, but then make arguments that require that the builder accurately predicted the future and built the schema with that foresight. Sure, we could go and pull a brands table out of the existing tables but thats work, and it might be work on a live database that brings it down. A graph database would be just as likely to have 10 brand nodes since the overhead of creating the first such node is far lower than creating an entire table and updating the schema. >Relational databases excel at easily providing answers to questions that weren’t predicted at the time when the data model was designed. Or a NoSQL database with spark. "But spark is something new to learn" And this is the biggest flaw in your argument. You're pro RDBMS because you know SQL and how to run RDBMS, create the schema, and write the queries. It is incredibly easy to get started with MongoDB. That right there is why it is popular. Not because its good. But when you say "Just get something started and use an RDBMS", you're actually saying "I know you know javascript, but I need you to learn Modula-2 for this part of the system". Fundamentally different syntax and strict types (or "schema"). >For all its unparallelizability, ACID is pretty damn nice. Until someone holds transactions open across network calls and kills throughput. I've seen that issue lose a company a multi-million dollar contract because of contention on a single row. Or until someone chooses the wrong isolation level ("But I used a transaction!") and two transactions happily decrement non-atomically. This shit be hard, and part of the "hard" is not knowing you're doing it wrong. >You’ll have plenty of time figuring out what to use when you know your exact usage patterns, if the business manages to not die until then. So start with something quick and easy then. You've already managed to describe two scenarios where an RDBMS blew up in practice (multiple teams hitting a DB causing schema lock; its a network, not a flat hierarchy). If you have an experienced SQL team, by all means go with RDBMS. But lets not pretend they are a panacea. Honestly, I'd use a graph database pretty much all the time - if I could only trust them. But its the quality, reliability and longevity that I have a problem with, not with the nature of how the database organizes data.
- jarym 9y agoI think the real mistake is one of: 1. I'm gonna use NoSQL because I've heard its really cool. 2. I'm not going to use SQL because its hard for me to write queries and change stuff on the fly. If either of those are true (gotta be honest with yourself when deciding) then STOP. You have to think about the best tool for the job overall - sometimes that will be NoSQL, sometimes it will be an RDBMS. If you can't decide which of the two, then Postgres with its JSON support is IMO the best starting point.