10 ms·

> I think the most interesting question though is would they be able to get MVP and initial customers that set off this if they were moving (slightly) slower du

> I think the most interesting question though is would they be able to get MVP and initial customers that set off this if they were moving (slightly) slower due to SQL and slight overhead that comes with? I've used Postgres and Mongo pretty extensively, and for any reasonably seasoned developer, the startup overhead of an SQL system is a myth. There may upfront cost to learning how an RDMS and SQL work in the first place, but once you're familiar with them, they'll be faster than Mongo on any new project. The schemaless concept of a document database seems to be the major selling factor in velocity of movement, but once you've got a good handle on a migration framework in the vein of ActiveRecord or other popular software, that's negated completely. It also really doesn't take long before schemaless starts to cause big problems for you in terms of data consistency -- it's not just the big players that get bitten by this. The simplified query language is another one. SQL is a little bit obtuse, but it's not that bad once you have a handle on it, and a lot of people are familiar with it. Once you add in an ORM layer, the lazy-style access of a framework like Sequel or SQLAlchemy makes the developer experience quite a bit better than any Mongo APIs that I've seen. Also, after you get beyond trivial usage, SQL's flexibility so wildly outstrips Mongo's query documents that it's not even worth talking about. Postgres on the other hand ships with a great management CLI, a very powerful REPL (psql), and features like data types/constraints/transactions that guarantee you correctness with zero effort on your part. I can only speak for myself, but I'd take Postgres to the hackathon any day of the week.

martinald 9y ago
I totally agree with you, and started writing something about how understanding a good ORM takes nearly all the headache away. I think the thing people do find slow is a lot of 'documents within documents' in SQL. It turns out this is usually a bad development pattern long term but it is super fast being able to just add docs inside docs with no configuration. It feels very slow writing foreign keys, navigation props and schemas for this in SQL vs JSON, where you can just dump your object in and you're done. Basically; I think with noSQL you get some very short term gain for a lot of long term pain, and you're right, ORMs and other tooling solves this mostly. I myself fell for this trap, and while it was a nightmare it actually matured me more as a professional more than anything I've ever done recently. Regardless of crazy hype, I don't think I'll ever fall for a solution so easily without evaluating it properly. I think I assumed the "crowd" had done the tech due diligence on this stuff and it definitely wasn't the case.
- macintux 9y ago
  I agree that Postgres trumps Mongo for most use cases, but if ORM is the answer, you might be asking the wrong question. I love that Fowler discusses[1] how to avoid ORMs and offers only two possibilities: > Either you use the relational model in memory, or you don't use it in the database. He clearly has a blind spot. The correct answer, at least in some cases, is don't use objects in the first place. [1]: https://www.martinfowler.com/bliki/OrmHate.html https://www.martinfowler.com/bliki/OrmHate.html
  - anentropic 9y ago
    I've been quite skeptical of the anti-ORM sentiments, since I always found the ORM handy, not a problem etc. Two things recently started to change my mind One is, after profiling some views in our Django app I found that a significant portion of the time in list views (i.e. returning many objects) was spent instantiating ORM model objects. Not on fetching them from the db, but turning them into objects whose imminent destiny is to get turned back into dicts for serializing to JSON. So we switched to values() queries which return plain dicts, but then the ORM is just a query engine. The second was, in a large app, finding that many of the tables start to require a degree of care to only perform 'safe' queries, i.e. ones supported by appropriate indexes. The integration of the ORM into the framework provides many conveniences, but this tends to involve a lack of awareness/control over exactly which queries get made. Most of the patterns that are 'idiomatic' to the framework also tend to involve a lot of ORM querying logic pushed down in the view layer... while our need to perform only a restricted set of 'safe' queries pushed us in the other direction, towards a repository layer with carefully curated query code. I'm seriously considering exploring ORM-less development in future
    - wvenable 9y ago
      There's no single technology that's perfect for all situations. An ORM is really useful for a lot of very common tasks. Most apps include a lot of reading/writing of single rows and basic list queries for which ORMs are a superior way of working. But for anything involving complex querying, I typically just go straight to SQL. Most ORMs provide plenty of ways to handling different levels of query need from full ORM queries, to queries into entity objects, to queries into flat dictionaries. I don't think ORM-less development is the answer to the wisdom that an ORM isn't a complete and sufficient interface to the database. That's just throwing the baby out with the bathwater.
bulldoa 9y ago
Would you recommend some resource for learning to setup postgres? As a new grad I often times find myself frustrated with setting up postgres or other rdbms so I usually opt for easier systems like amazon RDS, firebase or mongo. Before docker became popular, I basically have to do something like [this](https://github.com/docker-library/postgres/blob/master/9.6/Dockerfile https://github.com/docker-library/postgres/blob/master/9.6/D...), which I don't think I have mastered till this day.
- dahdum 9y ago
  Docker works fine, but I prefer to run a vagrant machine with ansible to setup postgres + everything else for development. In production I just use RDS. This is the ansible role I use: https://github.com/ANXS/postgresql https://github.com/ANXS/postgresql
  - bulldoa 9y ago
    do you guys have replication set up to your dev such that dev will have somewhat current info? If so how do you guys handle that replication?
    - dahdum 9y ago
      Since we take nightly backups of the RDS instances and store them in S3, I just wrote some ansible commands to download/import into the local db.
- brandur 9y ago
  > As a new grad I often times find myself frustrated with setting up postgres or other rdbms so I usually opt for easier systems like amazon RDS, firebase or mongo. Yeah, I know exactly what you mean. Everything in Postgres is supremely well documented, but there's just so much documentation, that getting the basics down is pretty hard. I definitely went through a period of building muscle that was a tad frustrating to say the least. Unfortunately, I don't know a particularly succinct resource that I recommend right now, but I will suggest taking a look at official docs. Maybe the section on managing databases in particular [1]. In general, you should just get good at using createdb, dropdb, and psql (which is mostly just generic SQL). If you're on a Mac, installation and setup might looks as easy as: brew install postgresql brew services start postgresql createdb my-test-db psql my-test-db dropdb my-test-db (There's obviously far more there than just that, but at a basic level, most of the commands are reasonably easy to use.) [1] https://www.postgresql.org/docs/9.6/static/managing-databases.html https://www.postgresql.org/docs/9.6/static/managing-database...
rmrfrmrf 9y ago
> the startup overhead of an SQL system is a myth. It's also a myth that devs choose NoSQL over SQL because "SQL is too hard."
- tracker1 9y ago
  Can't speak for anyone else... for me, it was the shape of the data I was interacting with was best represented as nested objects... think a classifieds site... primary record is the ad/entry, but different types of products will have differing ancillary data. That was my use case in first using MongoDB. Why, because otherwise there were up to 30+ joins in order to bring up all the data around any given classified entry. Reshaping the data into a document store, with a handful of indexes made a lot of sense. Nearly effortless replication was also a nice feature, and needed for some extra endurance and scale. I'm pretty sad that RethinkDB didn't get farther, as their management interface and approach was a lot better than MongoDB. To me it's about what does the data look like, what are the performance needs, and how much do you want to spend, cross-train or hire out expertise. Not to mention the lock-in options. There are lots of options, and it depends on the project and use. I've worked in enough applications where at least parts of the application needed to be moved off of SQL short of spending FAR more on licensing and/or servers that were insanely expensive for the need the value wasn't good.
- kafkaesq 9y ago
  It's also a myth that devs choose NoSQL over SQL because "SQL is too hard." What I see more of is situations where the developers "know" SQL but still just don't want to sit down and figure out their data models. It's like, the temptation to reach for the nearest short-cut -- whether that means over-reliance on ORMs (or hodgepodge 'data access layers'); or continually munging stuff at the application layer for nearly every operation; or... Mongo -- is always just too great.
  - bryanrasmussen 9y ago
    figuring out the data model as you build can be beneficial, you realize what you need as you work with the thing and can add it on the fly, at a certain point of course you need to do a cleanup. Getting people to accept the cleanup requirement is the difficult part.
lewisl9029 9y ago
> I've used Postgres and Mongo pretty extensively, and for any reasonably seasoned developer, the startup overhead of an SQL system is a myth. There may upfront cost to learning how an RDMS and SQL work in the first place, but once you're familiar with them, they'll be faster than Mongo on any new project. > The schemaless concept of a document database seems to be the major selling factor in velocity of movement, but once you've got a good handle on a migration framework in the vein of ActiveRecord or other popular software, that's negated completely. It also really doesn't take long before schemaless starts to cause big problems for you in terms of data consistency -- it's not just the big players that get bitten by this. I don't disagree with the overall sentiment, but this post is overstating things in a few places... Schemas add friction in development. You can certainly minimize this friction with experience and tooling, but for a completely new project, no amount of experience or ORM magic is going to "completely negate" that additional friction vs a schemaless system where you can change the shapes of your data and the relationships between them without any restrictions whatsoever. For rapid prototyping of new projects, it could be entirely reasonable to trade off long term data-consistency benefits of having a well-defined schema in return for faster iteration times. But taking that schemaless prototype to production directly is decidedly unwise, so the final step in a prototyping process like this should involve a refactoring of your data model to introduce a schema/migrations system before that point. It would be fine to argue that that final refactoring step to make the schemaless prototype production-ready could completely negate the faster initial iteration times afforded by starting out with a schemaless DB (I too would generally agree with this assertion, but we must also remember not all prototypes will go to production, as prototypes are meant to prove out an idea, and the idea itself won't always work out). But to claim on a brand new project (where the data model is ill defined and prone to sweeping changes), one can iterate _faster_ while maintaining a schema and migrations system than on a schemaless system (given similar levels of proficiency with both systems) strikes me as hyperbolic.
- jdmichal 9y ago
  > For rapid prototyping of new projects, it could be entirely reasonable to trade off long term data-consistency benefits of having a well-defined schema in return for faster iteration times. But taking that schemaless prototype to production directly is decidedly unwise, so the final step in a prototyping process like this should involve a refactoring of your data model to introduce a schema/migrations system before that point. I don't get it. If you're not in production anyway, you're not working with real data. And if you're not working with real data, you don't have to migrate anything. The entire problem just straight-up disappears. Just blow away your database and recreate it with the new schema. Modify your test data insertion scripts to match and load them.
antoaravinth 9y ago
>I've used Postgres and Mongo pretty extensively, and for any reasonably seasoned developer, the startup overhead of an SQL system is a myth. There may upfront cost to learning how an RDMS and SQL work in the first place, but once you're familiar with them, they'll be faster than Mongo on any new project. Exactly, I was very new to the RDBMS world and always I have preferred to go for MongoDB. Once in my side project, I thought of learning RDBMS and used Postgresql. Once I learnt how transactions, joins works then RDBMS was totally an easy topic. I have used Jooq instead of ORM, which again helped me to learn the queries and underlying system of Postgresql!
api 9y ago
The issue is absolutely not the overhead of schemas or managing them. The issue is the overhead of systems administration when clustering. Clustering is a requirement to support high availability, and Postgres clustering is a real bear to set up. Mongo clustering is easy. Postgres and most other SQL databases hail from the era when things ran on a "box." They're designed that way. Clustering is a bolt-on. It's that more than queries or schemas that drive NoSQL. It would have been better to fork Postgres and create a cleaned-up easy to cluster version minus the obnoxious arcana, but that's no fun. It's more fun to reinvent the wheel. Edit: We migrated from Postgres to RethinkDB for this reason and are mostly happy with it. We miss the data integrity guards of SQL but getting clustering without a full time DBA and Raft consensus fail-over without intervention is a worthy trade off. We still use Postgres for back-end warehousing and analytics but those are not live systems so they can live on a "box."

martinald 9y ago

I totally agree with you, and started writing something about how understanding a good ORM takes nearly all the headache away. I think the thing people do find slow is a lot of 'documents within documents' in SQL. It turns out this is usually a bad development pattern long term but it is super fast being able to just add docs inside docs with no configuration. It feels very slow writing foreign keys, navigation props and schemas for this in SQL vs JSON, where you can just dump your object in and you're done. Basically; I think with noSQL you get some very short term gain for a lot of long term pain, and you're right, ORMs and other tooling solves this mostly. I myself fell for this trap, and while it was a nightmare it actually matured me more as a professional more than anything I've ever done recently. Regardless of crazy hype, I don't think I'll ever fall for a solution so easily without evaluating it properly. I think I assumed the "crowd" had done the tech due diligence on this stuff and it definitely wasn't the case.

macintux 9y ago
I agree that Postgres trumps Mongo for most use cases, but if ORM is the answer, you might be asking the wrong question. I love that Fowler discusses[1] how to avoid ORMs and offers only two possibilities: > Either you use the relational model in memory, or you don't use it in the database. He clearly has a blind spot. The correct answer, at least in some cases, is don't use objects in the first place. [1]: https://www.martinfowler.com/bliki/OrmHate.html https://www.martinfowler.com/bliki/OrmHate.html
- anentropic 9y ago
  I've been quite skeptical of the anti-ORM sentiments, since I always found the ORM handy, not a problem etc. Two things recently started to change my mind One is, after profiling some views in our Django app I found that a significant portion of the time in list views (i.e. returning many objects) was spent instantiating ORM model objects. Not on fetching them from the db, but turning them into objects whose imminent destiny is to get turned back into dicts for serializing to JSON. So we switched to values() queries which return plain dicts, but then the ORM is just a query engine. The second was, in a large app, finding that many of the tables start to require a degree of care to only perform 'safe' queries, i.e. ones supported by appropriate indexes. The integration of the ORM into the framework provides many conveniences, but this tends to involve a lack of awareness/control over exactly which queries get made. Most of the patterns that are 'idiomatic' to the framework also tend to involve a lot of ORM querying logic pushed down in the view layer... while our need to perform only a restricted set of 'safe' queries pushed us in the other direction, towards a repository layer with carefully curated query code. I'm seriously considering exploring ORM-less development in future
  - wvenable 9y ago
    There's no single technology that's perfect for all situations. An ORM is really useful for a lot of very common tasks. Most apps include a lot of reading/writing of single rows and basic list queries for which ORMs are a superior way of working. But for anything involving complex querying, I typically just go straight to SQL. Most ORMs provide plenty of ways to handling different levels of query need from full ORM queries, to queries into entity objects, to queries into flat dictionaries. I don't think ORM-less development is the answer to the wisdom that an ORM isn't a complete and sufficient interface to the database. That's just throwing the baby out with the bathwater.

bulldoa 9y ago

Would you recommend some resource for learning to setup postgres? As a new grad I often times find myself frustrated with setting up postgres or other rdbms so I usually opt for easier systems like amazon RDS, firebase or mongo. Before docker became popular, I basically have to do something like [this](https://github.com/docker-library/postgres/blob/master/9.6/Dockerfile https://github.com/docker-library/postgres/blob/master/9.6/D...), which I don't think I have mastered till this day.

dahdum 9y ago
Docker works fine, but I prefer to run a vagrant machine with ansible to setup postgres + everything else for development. In production I just use RDS. This is the ansible role I use: https://github.com/ANXS/postgresql https://github.com/ANXS/postgresql
- bulldoa 9y ago
  do you guys have replication set up to your dev such that dev will have somewhat current info? If so how do you guys handle that replication?
  - dahdum 9y ago
    Since we take nightly backups of the RDS instances and store them in S3, I just wrote some ansible commands to download/import into the local db.
brandur 9y ago
> As a new grad I often times find myself frustrated with setting up postgres or other rdbms so I usually opt for easier systems like amazon RDS, firebase or mongo. Yeah, I know exactly what you mean. Everything in Postgres is supremely well documented, but there's just so much documentation, that getting the basics down is pretty hard. I definitely went through a period of building muscle that was a tad frustrating to say the least. Unfortunately, I don't know a particularly succinct resource that I recommend right now, but I will suggest taking a look at official docs. Maybe the section on managing databases in particular [1]. In general, you should just get good at using createdb, dropdb, and psql (which is mostly just generic SQL). If you're on a Mac, installation and setup might looks as easy as: brew install postgresql brew services start postgresql createdb my-test-db psql my-test-db dropdb my-test-db (There's obviously far more there than just that, but at a basic level, most of the commands are reasonably easy to use.) [1] https://www.postgresql.org/docs/9.6/static/managing-databases.html https://www.postgresql.org/docs/9.6/static/managing-database...

rmrfrmrf 9y ago

> the startup overhead of an SQL system is a myth. It's also a myth that devs choose NoSQL over SQL because "SQL is too hard."

tracker1 9y ago
Can't speak for anyone else... for me, it was the shape of the data I was interacting with was best represented as nested objects... think a classifieds site... primary record is the ad/entry, but different types of products will have differing ancillary data. That was my use case in first using MongoDB. Why, because otherwise there were up to 30+ joins in order to bring up all the data around any given classified entry. Reshaping the data into a document store, with a handful of indexes made a lot of sense. Nearly effortless replication was also a nice feature, and needed for some extra endurance and scale. I'm pretty sad that RethinkDB didn't get farther, as their management interface and approach was a lot better than MongoDB. To me it's about what does the data look like, what are the performance needs, and how much do you want to spend, cross-train or hire out expertise. Not to mention the lock-in options. There are lots of options, and it depends on the project and use. I've worked in enough applications where at least parts of the application needed to be moved off of SQL short of spending FAR more on licensing and/or servers that were insanely expensive for the need the value wasn't good.
kafkaesq 9y ago
It's also a myth that devs choose NoSQL over SQL because "SQL is too hard." What I see more of is situations where the developers "know" SQL but still just don't want to sit down and figure out their data models. It's like, the temptation to reach for the nearest short-cut -- whether that means over-reliance on ORMs (or hodgepodge 'data access layers'); or continually munging stuff at the application layer for nearly every operation; or... Mongo -- is always just too great.
- bryanrasmussen 9y ago
  figuring out the data model as you build can be beneficial, you realize what you need as you work with the thing and can add it on the fly, at a certain point of course you need to do a cleanup. Getting people to accept the cleanup requirement is the difficult part.

lewisl9029 9y ago

> I've used Postgres and Mongo pretty extensively, and for any reasonably seasoned developer, the startup overhead of an SQL system is a myth. There may upfront cost to learning how an RDMS and SQL work in the first place, but once you're familiar with them, they'll be faster than Mongo on any new project. > The schemaless concept of a document database seems to be the major selling factor in velocity of movement, but once you've got a good handle on a migration framework in the vein of ActiveRecord or other popular software, that's negated completely. It also really doesn't take long before schemaless starts to cause big problems for you in terms of data consistency -- it's not just the big players that get bitten by this. I don't disagree with the overall sentiment, but this post is overstating things in a few places... Schemas add friction in development. You can certainly minimize this friction with experience and tooling, but for a completely new project, no amount of experience or ORM magic is going to "completely negate" that additional friction vs a schemaless system where you can change the shapes of your data and the relationships between them without any restrictions whatsoever. For rapid prototyping of new projects, it could be entirely reasonable to trade off long term data-consistency benefits of having a well-defined schema in return for faster iteration times. But taking that schemaless prototype to production directly is decidedly unwise, so the final step in a prototyping process like this should involve a refactoring of your data model to introduce a schema/migrations system before that point. It would be fine to argue that that final refactoring step to make the schemaless prototype production-ready could completely negate the faster initial iteration times afforded by starting out with a schemaless DB (I too would generally agree with this assertion, but we must also remember not all prototypes will go to production, as prototypes are meant to prove out an idea, and the idea itself won't always work out). But to claim on a brand new project (where the data model is ill defined and prone to sweeping changes), one can iterate _faster_ while maintaining a schema and migrations system than on a schemaless system (given similar levels of proficiency with both systems) strikes me as hyperbolic.

jdmichal 9y ago
> For rapid prototyping of new projects, it could be entirely reasonable to trade off long term data-consistency benefits of having a well-defined schema in return for faster iteration times. But taking that schemaless prototype to production directly is decidedly unwise, so the final step in a prototyping process like this should involve a refactoring of your data model to introduce a schema/migrations system before that point. I don't get it. If you're not in production anyway, you're not working with real data. And if you're not working with real data, you don't have to migrate anything. The entire problem just straight-up disappears. Just blow away your database and recreate it with the new schema. Modify your test data insertion scripts to match and load them.

antoaravinth 9y ago

api 9y ago

The issue is absolutely not the overhead of schemas or managing them. The issue is the overhead of systems administration when clustering. Clustering is a requirement to support high availability, and Postgres clustering is a real bear to set up. Mongo clustering is easy. Postgres and most other SQL databases hail from the era when things ran on a "box." They're designed that way. Clustering is a bolt-on. It's that more than queries or schemas that drive NoSQL. It would have been better to fork Postgres and create a cleaned-up easy to cluster version minus the obnoxious arcana, but that's no fun. It's more fun to reinvent the wheel. Edit: We migrated from Postgres to RethinkDB for this reason and are mostly happy with it. We miss the data integrity guards of SQL but getting clustering without a full time DBA and Raft consensus fail-over without intervention is a worthy trade off. We still use Postgres for back-end warehousing and analytics but those are not live systems so they can live on a "box."