8 ms·
NoSQL No More
- nemothekid 13y ago>Many people seem surprised that we, a tech-savvy startup, would be moving to an "old" technology. Who is this "many?"
- whazzmaster 13y agoWhen I read stuff like this I try reeeeal hard to not immediately dismiss it as @hipsterhacker stereotypes, but weird groupthink like this shocks me. Keeping data that is by it's nature relational in a relational database is, to my mind, obvious. That it isn't for startups that are building their entire business on data foundations (because it's OLD!) is genuinely mind-boggling to me. I guess I'm the one that's old now.
- spamizbad 13y agoThere's a sizable contingent (but still a tiny minority) of developers who will outright dismiss anything SQL and typically view databases as these things you kinda pick on a whim based on whatever is easiest to shoehorn your MVP into. Fancy things like a powerful query language, robust data types, transactions, ACID-compliance etc? YAGNI. Although I don't share this view, I cant say its always wrong. If you're building simple MVPs to find product-market fit for ideas why bother? If there's a 90% chance what you're building will never evolve to need those features, why invest in them? But if your "product" ends up being a 10% survivor you better have a plan to move to something more powerful before you compound too much technical debt from your NoSQL database. Once you start scaling your business and have to face competition, NoSQL becomes productivity tarpit. Queries that take 20 minutes to write and optimize in the SQL realm can turn into day-long exercises in NoSQL. The reality is most developers aren't cranking out disposable software used to test the market for new ideas. We're working on things that already have a place, we just need to make them better. For us, the best option right now is SQL + maybe some specialized databases for certain purposes (Columnar for analytics, graph for relationship analysis, full-text for text, etc)
- bowlofpetunias 13y agoProbably the same people that call themselves "tech-savvy" and make such fundamental errors in matching their tools with their needs because "fashion". They're about as tech-savvy as the people that think voting machines are progress.
- DrJokepu 13y agoThis is why PostgreSQL is awesome. You can freely mix relational with non-relational (object) storage depending on your needs.
- fidotron 13y agoThis is only if you ignore why people moved to NoSQL in the first place. PgSQL is great for a lot of things, but I would argue that if you're using it you're betting on your overall product/service having some other killer advantage than data processing. The competent wing of the NoSQL crowd are using it in strange ways that enable new classes of product and service that cannot be achieved with PgSQL.
- woah 13y agoPlease elaborate?
- jeffdavis 13y agoCan you give examples? I assume you're talking about hadoop-related tech, but you left it vague.
- fidotron 13y agoA lot of the Hadoop users are in the right ballpark. Not necessarily that stack, but that approach to things, and the problems they are attacking, especially graph based data. My broader point is that if your project fits into pgSQL then you need another unique selling point and the data functions are just an implementation detail of some other aspect of your offering, whereas for many of the people not using that kind of thing their analytics and data systems are their selling point. (That's a backhanded compliment to pgSQL, in that it's easy enough to get right on small systems that any competent developer should be able to manage it, thus reducing the market value though). There is, of course, the blurred line of crazy MySQL deployments, many of which are barely relational.
- jeffdavis 13y ago
- azth 13y agoIs the page down for anyone else?
- farkerhaiku 13y agoyes. if only they had gone to a web scale mongodb backed blog engine.
- apta 13y agoSame here
- danbruc 13y agoThe NoSQL data model - please read as document oriented - is fundamentally flawed because it forces you into denormalization without good reasons like performance optimization but with all the disadvantages. The only good reason not to use a relational database is that they suck at modeling complex entities and their relationships. Shattering nice entities into small relations to get them normalized, adding join tables to model many to many relationships and all that fun. Everything that is not roughly tree-like is just painful. More than once I finally decided to just stuff the really complex pieces into a blob - XML or what ever worked best - and just deal with it at the application layer. Not pretty, not fun, but still less painful than modeling and dealing with it in the relational model.
- dhimes 13y agoI'm with you here. I've found that even phones have enough juice to handle most manipulation, and the website appears more responsive because the server isn't bogged down. Haven't scaled it yet, though...
- delluminatus 13y agoI wouldn't say the model is fundamentally flawed. Looking at the roots of NoSQL (which I believe was first popularized by Google's BigTable), the original intent of a NoSQL database is to allow redundant storage of a lot of data on a DFS. They're meant for situations where you have data that doesn't fit well into a relational database model, and you need to run BIG queries (think PageRank) over the entire data in a fault-tolerant way. For instance thinking about PageRank, how many discrete data elements are in a single web page? This is "higher-order" data that in a relational database might have hundreds or thousands or tens of thousands of columns per row. Of course there are also in-memory transactional databases called NoSQL databases which are meant to be replacements for the traditional relational data store, but those are a very different beast from the original NoSQL data model.
- danbruc 13y agoThat was bad and sensationalistic wording from my side. I should have said more clearly that I meant this in the context of modeling common business domains, not for specialized tasks that don't fit a relation model efficiently.
- visarga 13y agoI'm sorry but all I need can be handled by simple key/value stores, and I prefer the simplicity of JSON data structures, without predefined fields.
- SchizoDuckie 13y agoThat is all perfect until your project has matured into 2, 3, or 5 years old and you need to do patches and migrations. Or how about you have to transfer your project to another developer? I bet he/she'll be happy not having a spec of what's in where, when!
- rogerbinns 13y agoThose can just as easily be screwed up in relational databases. For example getting the schema wrong, or only updating it by accretion (adding new tables). A developer being below average (as half of all developers are), using the wrong tool for the job, not understanding things etc will make a mess. A claim that developers can screw things up is uninteresting (and true). A claim that there is no scenario in which a "NoSQL" database is the better solution is a lot more interesting, and requires more than "some developers could screw it up" to substantiate.
- SchizoDuckie 13y agoHear hear! Now can somebody convince the W3 and Mozilla that there's nothing wrong with implementing SQLite next to IndexedDB so that I can move forward without having to write an IndexedDB adapter for my clearly relational TV Show -> Season -> Episode data please? https://github.com/SchizoDuckie/DuckieTV/blob/angular/js/CRUD.entities.js https://github.com/SchizoDuckie/DuckieTV/blob/angular/js/CRU... Foreign Relationships on foreign keys are not difficult Many to many Relationships are not difficult SQL is not difficult Joining is not difficult Grouping is not difficult Now try to do that in IndexedDB / NoSQL, and suddenly you're in a world of hurt. It can work, but can it perform? Maybe. With time and patches. So wham, let's throw the option of implementing the probably most well tested piece of software (in the universe probably, see http://sqlite.org/testing.html http://sqlite.org/testing.html ) out of the window.
- frik 13y agoSo true. It's still sad that because of Mozilla and Microsoft we have no SQL API in all HTML5 browsers :( http://en.wikipedia.org/wiki/WebSQL http://en.wikipedia.org/wiki/WebSQL IndexedDB is fine for storing JSON objects, etc. but a relational database with SQL query syntax, indexes, etc. more powerful and means less code to write. With IndexedDB one has to reinvent the wheel to just get basic query features. WebSQL is not deprecation, the W3C Working Group Note actually says: 'This specification is no longer in active maintenance and the Web Applications Working Group does not intend to maintain it further'. WebSQL is only available in Webkit based Browsers (Safari, Chrome) which means most mobile browsers. As SQLite is in public domain, no company would "loose their face" if they choose to use it. They could fork off SQLite and change the SQL query syntax (parser) to whatever the W3C finds suitable. https://www.sqlite.org https://www.sqlite.org Mozilla Firefox and FirefoxOS both already ship SQLite for years and can be accessed by its internal JavaScript API. And several Microsoft products already use it anyway (e.g. Forza Xbox games). Microsoft has of course also various other SQL database libraries like MS Access JetRed (http://en.wikipedia.org/wiki/Extensible_Storage_Engine http://en.wikipedia.org/wiki/Extensible_Storage_Engine ), MS Outlook JetBlue (http://en.wikipedia.org/wiki/Extensible_Storage_Engine http://en.wikipedia.org/wiki/Extensible_Storage_Engine ) and MS SQL Server Express (http://en.wikipedia.org/wiki/SQL_Server_Express http://en.wikipedia.org/wiki/SQL_Server_Express ) the SQL backend originally forked off for WinFS for Longhorn (Vista beta). It would be trivial for Microsoft to choose one of its many SQL engines and add it to IE 12. The same goes for Mozilla (just expose the API of SQLite). For some reason Oracle and Mozilla pushed IndexedDB. Oracle has conflicting interests, as it owns OracleDB(SQL), MySQL (SQL) and BerkeleyDB (NoSQL and now also SQL support, based on SQLite). Oracle is an official "sponsor" of SQLite development and even ships it as part of it's BerkeleyDB package: http://www.oracle.com/technetwork/database/database-technologies/berkeleydb/overview/index.html http://www.oracle.com/technetwork/database/database-technolo... and https://www.sqlite.org/index.html#consortium_members https://www.sqlite.org/index.html#consortium_members Mozilla tried to explain why they prefer IndexedDB over WebSQL: http://hacks.mozilla.org/2010/06/beyond-html5-database-apis-and-the-road-to-indexeddb/ http://hacks.mozilla.org/2010/06/beyond-html5-database-apis-... and by an Mozilla dev https://web.archive.org/web/20130723044210/http://blog.vlad1.com/2009/04/06/html5-web-storage-and-sql https://web.archive.org/web/20130723044210/http://blog.vlad1... (I generally like Firefox and FirefoxOS but not implementing WebSQL is one of the worst actions that of Mozilla org, IMHO). One can speculate that a less powerful HTML5 API translates in the long run to more SQL server licenses for Oracle and Microsoft. If the web app devs cannot do the processing & storage on the client side, one has to do it on the server side. Anyway, I hope that we get an SQL API for HTML 5.x that also Mozilla and/or Microsoft implements. As of now WebSQL works fine in Webkit based browser which includes Safari, Chrome, Opera and includes also 95% of all smart phones.
- moron4hire 13y agoTypically, the best you can hope for out of your data is that it is a directed graph. Relational databases exist in a sweet spot of efficiency, widespread availability, and fairly decent ability to represent directed graphs. But you need to know the shape of your graph first. The document-store model tends towards giving up the graph entirely and just going with a tree. Us programmers love tree structures. They make things nice and tidy and can be traversed in finite time. But I've never worked on a project that could easily be modeled as a tree. Unfortunately, this was often realized after I had been working on the project for a year. You know how phenomenally fucked you are having a tree-like data structure when what you really need is a directed graph? You're basically all the fuckeds. Directed graphs generalize trees, so trying to shoehorn a tree where a directed graph is needed is basically bringing a Fischer Price tool set with you to Habitat for Humanity. Thus, from experience, I never start with a tree now. I always start with a directed graph. If I learn that the data is a tree after a year and not actually a directed graph, oh well, look at all this flexibility we didn't need.
- mrsteveman1 13y agoWhy refer to the database that didn't fit this use case as "the particular NoSQL database that we selected", while specifically mentioning the one that DID fit by name? I know Postgres is great, no convincing necessary :)
- astine 13y agoProbably avoiding a backlash from partisans of the that db complaining about the bad press and from partisans of other NoSQL dbs complaining that the points in the article don't apply to them.
- pokstad 13y agoI love CouchDB because it works directly with web browsers without middleware (and supports awesome replication), but I wouldn't mind structured records like in SQL databases. I end up enforcing schemas using update validation functions anyway. As long as it inputs and outputs in JSON over HTTP, I'd be happy.
- jasallen 13y agoTL;DR; of my comment: NoSQL is a good optimization sometimes. Don't prematurely optimize. If you don't know what you need, you need an RDBMS. NoSQL / Document databases got so cool so fast, and were usable with little to no actual know-how that people just lost their minds and used them by default. That was always the wrong decision. RDBMS are the swiss army knives. They do "all the things". But power, responsibility, etc. I use various types of NoSQL models for various special purposes, they are great, and should be used, for those purposes. You almost always need an RDBMS. If it's not your gold record (because you need quick writes and can lock a whole document at a time), then you need to replicate to RDMBS for better ad hoc reporting. OTH, if RDBMS is your gold, you may want to replicate to NoSQL for fast lookups. I'm currently working through a situation where the latter is required. I have hundreds of thousands of records and the need to solve a classic backpack problem (with hundreds of thousands of potential items to go in the pack, each of which can have n instances)... so I've replicated a few billion pre-solved solutions to Azure Table Storage. It solves a particular problem, but its not my WHOLE application's data store.
- iSnow 13y agoI think the author is conflating the document-oriented variant of NoSQL with the whole family of databases. I cannot criticize him for leaving Couch|Mongo if their data is rich in relations, but I would like to read his thoughts on graph databases for instance. Maybe their data is too large for Neo4J, but for querying n:m relationships, this model can offer advantages.
- platinumdragon 13y agoI agree with the overall premise that the technology you choose needs to match your problem, however it's unfortunate that the author seemed to think that NoSQL is only about document-based databases, and didn't do any research into something like a graph database. It would have been interesting to see if that would've met his needs better than taking his ball and running home.