3 ms·

No it is not stupid. RDBMS are old thinking, and have not inovated in decades. They made sense when procedural programming was all the rage, now they just don

by ardit33 19y ago

No it is not stupid. RDBMS are old thinking, and have not inovated in decades. They made sense when procedural programming was all the rage, now they just don't anymore. My ideal database would just act like a big hash table. Where I can store an object and retrieve it at will, without having to learn any other language. (sql in this case), or use complicated ORMs, (Hibernate, or Django models). etc.. So, for some reason or the other, there hasn't been much inovation in this field. Maybe b/c the big players in databases like it as it is, and there is no incentive to inovate. Just b/c you drank your koolaid a while ago, doesn't mean everybody else has. In the programmer's poing of view, right now databases are a necessary evil, but they could be much better.

edw519 19y ago
Today's action items: - Give me a list of good (lifetime sales > $10,000) customers on the west coast (CA, OR, WA) who bought any product on the defective list during busy season (10/06, 11/06, 12/06) and haven't placed an order since our last email blast. We'll find out why. - Give me a list of phone numbers (using our Caller ID) of people who called in the last 4 days, whose call wasn't answered and had never called us before. We'll call them back. - Give me a list of our slowest moving (10%) products that haven't been ordered in the last 4 months that are sitting in prime space (Location beginning with 1,2, or 3) in the warehouse. We'll move them to pallets to make room for new stuff. - Give me a list of all products returned from Territory 7 in the last 3 months with a Problem Code related to fit or size. We want to make sure their description is correct on the website. I could go on (and on and on)... Do any of these with your "big hash table" in less than 1 second and earn a seat at the table with the big boys. Just because something is old doesn't mean it's obsolete. Maybe because it works so well. And just because someone understands this doesn't mean they "drank kool-aid". Nothing wrong with a little debate. But please come up with more creative methaphors.
- dfranke 19y ago
  Thank you! I've bookmarked this reply, so that the next 20 times I see ardit33's argument made I can just link to it rather than pounding out my own long-winded reply or just beating my head against my desk until it doesn't hurt any more. (The reply that I would have written if you didn't respond first would probably have emphasized the amount of code required rather than performance.)
  - edw519 19y ago
    lol I could have made any number of technical arguments. We'd be pissing at each other all day. Instead, I cited a few things that concern those who pay the bills. As hackers engaging in good technical debate here, it's too easy to lose sight of those whom we serve.
    - dfranke 19y ago
      It's frustrating how many smart people agree with ardit33's argument, though. I've even heard the same thing from spez. When you're mostly writing CRUD queries, you can get away with just treating the DB as a hash table, but it still means you're probably doing a lot of things inefficiently. The database needs of the company I work for now are just run-of-the-mill web app stuff, but still it's just another day at the office when I write an SQL query that could easily translate to 100 LOC worth of hash table operations. Here's a real example from our forum code that I've used as a job interview question: each user is member of some set of groups (groups in the UNIX sense), and each forum requires membership in all of some set of groups in order to be allowed to post. Given a userid, return the set forums to which that user is allowed to post. You can do it with a single, concise SQL statement containing nested existential quantifiers, and the query plan it generates is very fast.
      - mojuba 19y ago
        Let's say I have the entire structure in memory, interlinked with pointers. The solution in pseudocode: allowed_forums = [] // empty set for each group in user.groups allowed_forums |= group.forums // union That's all. I can probably write a compiler that translates this to SQL if necessary. Nice thing is that when I write this piece I don't care about how exactly data is stored and accessed. Now let's compare it with your SQL solution. You see, mine so simple, it's not even an interview question :)
        dfranke 19y ago
        1. Your solution is incorrect. I said you need to be a member of all the groups associated with a forum, not one of them. How would your code handle a public forum (one with no groups associated with it)? 2. Suppose you have one million users. Then you're sucking up an awful lot of RAM just for this tiny piece of functionality. 3. Your pseudocode makes a lot of convenient assumptions about what functionality is already provided in standard libraries and in the accessor methods for user.groups and group.forums. Let's see it written in a real language. 4. You say you don't care about how data is stored and accessed, but given the right set of reasonable assumptions, I can conclude that the running time of your algorithm is O(n^3). The beauty of relational databases is that you usually really don't have to care. The query optimizer is pretty good at figuring it out for you. EDIT: nostrademons, I saw that peer reply before you deleted it. That was just spookily similar :-)
- ardit33 19y ago
  Have you heard about filters? That's what they exactly do, narrow your choices according to specific circumstances. You get the list of object that fits the exact pattern of a filter (or multiple filters), and you are done (ok. maybe in java you have to cast the object). But that's it. Tables and SQL are old concepts which haven't evolved thru time. If SQL was so great, why in the heck there is so many ORM frameworks out there? Is it b/c people are stupid, or maybe b/c there is actually a need to just have a Object Oriented Database.
  - nostrademons 19y ago
    Yeah, list comprehensions are isomorphic to query languages. There's at least one database (Mnesia) and three database abstraction libraries (HaskellDB, SchemeQL, SQLAlchemy) that exploit this. That does not mean that databases are useless. Your [(customer.name, customer.phone) for customer in customers for sale in sales if sale.customer_id = customer.customer_id and sale.product_id in defective_product_list and sale.date.month in (10, 11, 12) and customer.lifetime_sales > 10000 and customer.state in ('CA', 'WA', 'OR')] isn't going to help much if you have thousands of customers and millions of sales.
    - edw519 19y ago
      The examples I cited are real. (When you're in the trenches, you don't need to make them up.) The database has 1,692,438 Customers and 7,823,356 Sales Orders. All 4 questions had subsecond response. But not using a single SQL statement as you suggest. Using a combination of resources: indexes, stored procedures, common functions, and other code, applied exactly as needed. And a RDBMS. Remember, this is a discussion of relational databases, not SQL. 2 different issues.
    - dfranke 19y ago
      HaskellDB would be so awesome if it actually worked. Every time I've tried to use it, though, the backend has choked on some piece of the generated SQL.
      - nostrademons 19y ago
- michaelneale 19y ago
  Also RDBMS' are based on some pretty solid mathematics, unlike almost all other areas of software (the other famous one is the lambda calculus). Generally where mathematics and software intersect, you end up with some powerful stable systems that stand the test of time.
  - michaelneale 19y ago
    Having said that, I have successfully done a lot of things by effectively bypassing the RDBMS. I just never felt that great about it.
- palish 19y ago
  Well.. I believe scaling out will solve the slowness problem. Map/reduce is really powerful.
- deleted 19y ago
  [deleted]

by ardit33 19y ago

edw519 19y ago

Today's action items: - Give me a list of good (lifetime sales > $10,000) customers on the west coast (CA, OR, WA) who bought any product on the defective list during busy season (10/06, 11/06, 12/06) and haven't placed an order since our last email blast. We'll find out why. - Give me a list of phone numbers (using our Caller ID) of people who called in the last 4 days, whose call wasn't answered and had never called us before. We'll call them back. - Give me a list of our slowest moving (10%) products that haven't been ordered in the last 4 months that are sitting in prime space (Location beginning with 1,2, or 3) in the warehouse. We'll move them to pallets to make room for new stuff. - Give me a list of all products returned from Territory 7 in the last 3 months with a Problem Code related to fit or size. We want to make sure their description is correct on the website. I could go on (and on and on)... Do any of these with your "big hash table" in less than 1 second and earn a seat at the table with the big boys. Just because something is old doesn't mean it's obsolete. Maybe because it works so well. And just because someone understands this doesn't mean they "drank kool-aid". Nothing wrong with a little debate. But please come up with more creative methaphors.

dfranke 19y ago
Thank you! I've bookmarked this reply, so that the next 20 times I see ardit33's argument made I can just link to it rather than pounding out my own long-winded reply or just beating my head against my desk until it doesn't hurt any more. (The reply that I would have written if you didn't respond first would probably have emphasized the amount of code required rather than performance.)
- edw519 19y ago
  lol I could have made any number of technical arguments. We'd be pissing at each other all day. Instead, I cited a few things that concern those who pay the bills. As hackers engaging in good technical debate here, it's too easy to lose sight of those whom we serve.
  - dfranke 19y ago
    It's frustrating how many smart people agree with ardit33's argument, though. I've even heard the same thing from spez. When you're mostly writing CRUD queries, you can get away with just treating the DB as a hash table, but it still means you're probably doing a lot of things inefficiently. The database needs of the company I work for now are just run-of-the-mill web app stuff, but still it's just another day at the office when I write an SQL query that could easily translate to 100 LOC worth of hash table operations. Here's a real example from our forum code that I've used as a job interview question: each user is member of some set of groups (groups in the UNIX sense), and each forum requires membership in all of some set of groups in order to be allowed to post. Given a userid, return the set forums to which that user is allowed to post. You can do it with a single, concise SQL statement containing nested existential quantifiers, and the query plan it generates is very fast.
    - mojuba 19y ago
      Let's say I have the entire structure in memory, interlinked with pointers. The solution in pseudocode: allowed_forums = [] // empty set for each group in user.groups allowed_forums |= group.forums // union That's all. I can probably write a compiler that translates this to SQL if necessary. Nice thing is that when I write this piece I don't care about how exactly data is stored and accessed. Now let's compare it with your SQL solution. You see, mine so simple, it's not even an interview question :)
      - dfranke 19y ago
        1. Your solution is incorrect. I said you need to be a member of all the groups associated with a forum, not one of them. How would your code handle a public forum (one with no groups associated with it)? 2. Suppose you have one million users. Then you're sucking up an awful lot of RAM just for this tiny piece of functionality. 3. Your pseudocode makes a lot of convenient assumptions about what functionality is already provided in standard libraries and in the accessor methods for user.groups and group.forums. Let's see it written in a real language. 4. You say you don't care about how data is stored and accessed, but given the right set of reasonable assumptions, I can conclude that the running time of your algorithm is O(n^3). The beauty of relational databases is that you usually really don't have to care. The query optimizer is pretty good at figuring it out for you. EDIT: nostrademons, I saw that peer reply before you deleted it. That was just spookily similar :-)
ardit33 19y ago
Have you heard about filters? That's what they exactly do, narrow your choices according to specific circumstances. You get the list of object that fits the exact pattern of a filter (or multiple filters), and you are done (ok. maybe in java you have to cast the object). But that's it. Tables and SQL are old concepts which haven't evolved thru time. If SQL was so great, why in the heck there is so many ORM frameworks out there? Is it b/c people are stupid, or maybe b/c there is actually a need to just have a Object Oriented Database.
- nostrademons 19y ago
  Yeah, list comprehensions are isomorphic to query languages. There's at least one database (Mnesia) and three database abstraction libraries (HaskellDB, SchemeQL, SQLAlchemy) that exploit this. That does not mean that databases are useless. Your [(customer.name, customer.phone) for customer in customers for sale in sales if sale.customer_id = customer.customer_id and sale.product_id in defective_product_list and sale.date.month in (10, 11, 12) and customer.lifetime_sales > 10000 and customer.state in ('CA', 'WA', 'OR')] isn't going to help much if you have thousands of customers and millions of sales.
  - edw519 19y ago
    The examples I cited are real. (When you're in the trenches, you don't need to make them up.) The database has 1,692,438 Customers and 7,823,356 Sales Orders. All 4 questions had subsecond response. But not using a single SQL statement as you suggest. Using a combination of resources: indexes, stored procedures, common functions, and other code, applied exactly as needed. And a RDBMS. Remember, this is a discussion of relational databases, not SQL. 2 different issues.
  - dfranke 19y ago
    HaskellDB would be so awesome if it actually worked. Every time I've tried to use it, though, the backend has choked on some piece of the generated SQL.
    - nostrademons 19y ago
michaelneale 19y ago
Also RDBMS' are based on some pretty solid mathematics, unlike almost all other areas of software (the other famous one is the lambda calculus). Generally where mathematics and software intersect, you end up with some powerful stable systems that stand the test of time.
- michaelneale 19y ago
  Having said that, I have successfully done a lot of things by effectively bypassing the RDBMS. I just never felt that great about it.
palish 19y ago
Well.. I believe scaling out will solve the slowness problem. Map/reduce is really powerful.
deleted 19y ago
[deleted]