6 ms·

I still don't understand database systems, how they work internally, and what problem they really solve. It seems that new paradigms like R and RAM-only key va

by jokoon 10y ago

I still don't understand database systems, how they work internally, and what problem they really solve. It seems that new paradigms like R and RAM-only key values system are just simpler, faster and cheaper in programmer time. Loading everything in RAM and do a search seems like a huge saving of time and just works for most cases. Usually if you have more complex needs, you need to adapt your solution, and databases don't seem like a silver bullet. Database queries seem like a solution to the problem of storing data on disk when RAM was too expensive. So today it's still used as some sort of standard, but when you can have 16GB of ram, I think you better teach yourself what sort of algorithm and data structures a database use to be faster, use the ones you like and need and solve your problem case by case. The example of how reddit store its data is pretty demonstrative that ultimately, you should not let a database system do all the work. Databases are just a file format to me, but the way it tries to work for you at a lower level will respond to basic cases, but when you increase complexity it's not relevant anymore. Especially today when you have big data and machine learning, everyone should just learn to understand data manipulation. Not saying to teach yourself C all over again, but having a decent idea of the math of what indexing is really about. Forcing yourself to use a database because the company always used it isn't appealing to me. It's like one of those things when a programmer had an idea which is based on a constraint, everyone starts using it, several products are made, but nobody really remember the original idea of the inventor of that paradigm.

andybak 10y ago
Time to wheel out pseudo-Greenspun again: "Any sufficiently complicated NoSQL program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of a decent RDMS." Or maybe: "Those who do not learn history are doomed to repeat it"
- threeseed 10y ago
  I guess companies like Google, Facebook, Twitter, Apple, Microsoft, LinkedIn, eBay who run their entire core business on NoSQL databases don't know what they are doing.
  - cwyers 10y ago
    Facebook, Twitter, Google and LinkedIn all use MySQL. A lot of it. Yes, they use other datastores, but claiming they run their entire core business on NoSQL is patently wrong.
    - jokoon 10y ago
      > but claiming they run their entire core business on NoSQL is patently wrong. Those companies are very large and thus they use many different technologies. > Yes, they use other datastores I guess that's what he meant by nosql. Overall he is saying nosql are for very specific, very large scale needs.
      - cwyers 10y ago
        The key word there is entire. They do not use exclusively NoSQL storage technologies. They use a mix of NoSQL and SQL. There are tradeoffs to each, and if you have as many really, really great engineers as such places employ, it can make sense to deploy whatever database technology best suits each narrow storage problem in order to meet absurdly high levels of usage. For 95% of us, who have neither the resources or the needs of a business that large, using a relational database for everything is probably the right choice.
  - andybak 10y ago
    The post I was responding to seemed to be arguing that relational databases were redundant and that was the point I was reacting to. I certainly didn't intend to suggest they were the only game in town.
coldtea 10y ago
>I still don't understand database systems, how they work internally, and what problem they really solve. The use a formal, mathematical abstraction (relational algebra), even if somewhat crapily implemented by particular RDBMS, to facilitate saving data, relationships between data, and queries, while at the same time assuring certain properties (e.g. ACID) and doing so in a platform/programming language neutral format and with a consistent and industry standard query interface. NoSQLs, key-stores and the like are not newer developments -- they predate databases. They were found to be a bad fit for what we wanted in enterprise use, and RDBS caught on. For uses with huge data (Google and such scale) a denormalised approach might be more practical for performance reasons, in which case developers and ad-hoc programs get to re-implement all the functionality and assurances of a RDBMS in an ad-hoc way on top of rawer stores. (And all this could be alleviated with a properly optimised RDBMS for such purposes the still respects relational algebra). In a conventional enterprise setting on the other hand, a DB trumps NoSQL etc solutions any day of the week. >Database queries seem like a solution to the problem of storing data on disk when RAM was too expensive. So today it's still used as some sort of standard, but when you can have 16GB of ram, I think you better teach yourself what sort of algorithm and data structures a database use to be faster, use the ones you like and need and solve your problem case by case. The whole idea is to free the data from being tied to a particular language, data structures and algorithms. Back in 2000-2007 it was all about XML, and how we should store data in XML format and get them back with XQUERY, XPATH and the like. A lot of people bought into the hype and the resulting products. Then it was about JSON -- and we now have JSON stores. In 10 years, it will be something else, again ad-hoc. Meanwhile SQL has worked for the past 3+ decades, and is based on a solid mathematical abstraction (relational algebra). >It's like one of those things when a programmer had an idea which is based on a constraint, everyone starts using it, several products are made, but nobody really remember the original idea of the inventor of that paradigm. You'd be surprised. Or rather you're exactly right -- few "really remember the original idea of the inventor of that paradigm", and that's why we're moving in circles with ad-hoc technologies re-implemented 30+ years after they were discarded. Or why people jumped enthusiastically to Mongo to return crying back to PostgreSQL. Read a little around here for example: http://www.dbdebunk.com/2015/07/the-sql-and-nosql-effects-will-they.html http://www.dbdebunk.com/2015/07/the-sql-and-nosql-effects-wi... http://www.allanalytics.com/author.asp?section_id=2386&doc_id=280912 http://www.allanalytics.com/author.asp?section_id=2386&doc_i... http://www.dbdebunk.com/2015/11/moving-in-circles-sql-for-nosql.html http://www.dbdebunk.com/2015/11/moving-in-circles-sql-for-no...
wiz21 10y ago
GB of RAM is fine until your database is one terabyte. Also, RDBMS do constraint enforcement for you and that's absolutely critical when you work with any sufficiently big database (I work on a 100GB DB with a team of 6 people, changing here and there and I can't tell you how crucial the constraints are to maintain the data quality in face of bugs, misjudgment, etc.). And no, it's not for lack of testing. When you work on millions of super complex records, you always miss some corner cases (unless you're NASA and have a huge budget to test thoroughly)

by jokoon 10y ago

andybak 10y ago

Time to wheel out pseudo-Greenspun again: "Any sufficiently complicated NoSQL program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of a decent RDMS." Or maybe: "Those who do not learn history are doomed to repeat it"

threeseed 10y ago
I guess companies like Google, Facebook, Twitter, Apple, Microsoft, LinkedIn, eBay who run their entire core business on NoSQL databases don't know what they are doing.
- cwyers 10y ago
  Facebook, Twitter, Google and LinkedIn all use MySQL. A lot of it. Yes, they use other datastores, but claiming they run their entire core business on NoSQL is patently wrong.
  - jokoon 10y ago
    > but claiming they run their entire core business on NoSQL is patently wrong. Those companies are very large and thus they use many different technologies. > Yes, they use other datastores I guess that's what he meant by nosql. Overall he is saying nosql are for very specific, very large scale needs.
    - cwyers 10y ago
      The key word there is entire. They do not use exclusively NoSQL storage technologies. They use a mix of NoSQL and SQL. There are tradeoffs to each, and if you have as many really, really great engineers as such places employ, it can make sense to deploy whatever database technology best suits each narrow storage problem in order to meet absurdly high levels of usage. For 95% of us, who have neither the resources or the needs of a business that large, using a relational database for everything is probably the right choice.
- andybak 10y ago
  The post I was responding to seemed to be arguing that relational databases were redundant and that was the point I was reacting to. I certainly didn't intend to suggest they were the only game in town.

coldtea 10y ago

>I still don't understand database systems, how they work internally, and what problem they really solve. The use a formal, mathematical abstraction (relational algebra), even if somewhat crapily implemented by particular RDBMS, to facilitate saving data, relationships between data, and queries, while at the same time assuring certain properties (e.g. ACID) and doing so in a platform/programming language neutral format and with a consistent and industry standard query interface. NoSQLs, key-stores and the like are not newer developments -- they predate databases. They were found to be a bad fit for what we wanted in enterprise use, and RDBS caught on. For uses with huge data (Google and such scale) a denormalised approach might be more practical for performance reasons, in which case developers and ad-hoc programs get to re-implement all the functionality and assurances of a RDBMS in an ad-hoc way on top of rawer stores. (And all this could be alleviated with a properly optimised RDBMS for such purposes the still respects relational algebra). In a conventional enterprise setting on the other hand, a DB trumps NoSQL etc solutions any day of the week. >Database queries seem like a solution to the problem of storing data on disk when RAM was too expensive. So today it's still used as some sort of standard, but when you can have 16GB of ram, I think you better teach yourself what sort of algorithm and data structures a database use to be faster, use the ones you like and need and solve your problem case by case. The whole idea is to free the data from being tied to a particular language, data structures and algorithms. Back in 2000-2007 it was all about XML, and how we should store data in XML format and get them back with XQUERY, XPATH and the like. A lot of people bought into the hype and the resulting products. Then it was about JSON -- and we now have JSON stores. In 10 years, it will be something else, again ad-hoc. Meanwhile SQL has worked for the past 3+ decades, and is based on a solid mathematical abstraction (relational algebra). >It's like one of those things when a programmer had an idea which is based on a constraint, everyone starts using it, several products are made, but nobody really remember the original idea of the inventor of that paradigm. You'd be surprised. Or rather you're exactly right -- few "really remember the original idea of the inventor of that paradigm", and that's why we're moving in circles with ad-hoc technologies re-implemented 30+ years after they were discarded. Or why people jumped enthusiastically to Mongo to return crying back to PostgreSQL. Read a little around here for example: http://www.dbdebunk.com/2015/07/the-sql-and-nosql-effects-will-they.html http://www.dbdebunk.com/2015/07/the-sql-and-nosql-effects-wi... http://www.allanalytics.com/author.asp?section_id=2386&doc_id=280912 http://www.allanalytics.com/author.asp?section_id=2386&doc_i... http://www.dbdebunk.com/2015/11/moving-in-circles-sql-for-nosql.html http://www.dbdebunk.com/2015/11/moving-in-circles-sql-for-no...

wiz21 10y ago

GB of RAM is fine until your database is one terabyte. Also, RDBMS do constraint enforcement for you and that's absolutely critical when you work with any sufficiently big database (I work on a 100GB DB with a team of 6 people, changing here and there and I can't tell you how crucial the constraints are to maintain the data quality in face of bugs, misjudgment, etc.). And no, it's not for lack of testing. When you work on millions of super complex records, you always miss some corner cases (unless you're NASA and have a huge budget to test thoroughly)