7 ms·

I really disagree with this. The more features you use from your DBMS vendor, the more you shoot yourself in the following departments: - scalability. Logic an

I really disagree with this. The more features you use from your DBMS vendor, the more you shoot yourself in the following departments: - scalability. Logic and processing in the server is bad as it means you can only scale up and not scale out. Scale up is damn expensive. When you need that 64 core 96Gb machine to run it all on with 6 replicas will see what I mean. - complexity. The DBMS is a black box which sucks up tight coupling almost instantly. Coupling is bad for maintenance and scalability. SQL always ends up with heavy coupling. - lock in. Those features only work with postgres so you're stuck with it forever or induce a lot of cost to move away when a better solution comes along. - schema. Maintaining schema and dependencies is hell on all but a small system. These facts come from 20 years of using databases. Stuff that will save you: Use your DBMS as a simple flat store akin to a document store. Use a CQRS based architecture and encapsulate ALL of your logic in code. If you have to go at least partially relational for god's sake use an ORM that supports several databases well. Make a provider independent app level backup/restore facility such as the one with JIRA or TeamCity. NEVER hire a DBA - they exist only to create walled gardens to protect their existence. My current gig has cocked up on all areas and it's costing them over 40% of their turnover keeping it alive. Happy databasing :)

platonichvn 15y ago
I agree with you but I often question when people list "lock in" as an issue. How often do you see projects where they decide to move from one db to another? Even in cases where an ORM is used, making the migration process painless in theory, I doubt many projects take the risk and exercise their freedom of not being locked in.
- joevandyk 15y ago
  I've only seen people move to postgresql (generally from mysql). I've never heard of people moving away from postgresql. Granted, that's a small sample size.
  - pvh 15y ago
    Etsy did. I don't think it was a good decision and I think the reasons they did had more to do with resetting from some bad architecture decisions, but they're the example I know of. http://codeascraft.etsy.com/2011/09/28/scaling-etsy-what-went-wrong-what-went-right/ http://codeascraft.etsy.com/2011/09/28/scaling-etsy-what-wen... I think the big lesson here would be: don't have separate DBA and Engineering teams. On the other hand, I could name quite a few people who went the other way, like Urban Dictionary who run on our http://postgres.heroku.com http://postgres.heroku.com service, too. (I don't normally name customers, but he recorded a video saying he liked us so I think it's fair game. http://success.heroku.com/urbandictionary http://success.heroku.com/urbandictionary )
- gouranga 15y ago
  If you look at most "corporate" IT platforms, they tend to have favourite database engines that they need to use. If you have DBMS portability, then it gets one foot in the door straight away. Also contrary to the norm here, some applications last a long time. The one I work with daily is actually 20 years old this year. In that time a lot has changed and in the next 20 years a lot will as well. It makes sense for long-lived applications to plan well ahead and consider paradigm and architecture changes. This platform started in C++ on OS/2 with Oracle running on VAX/VMS at the back end. It went to Java/J2EE with an Oracle back end in the early 00's. It's now SQL Server and .Net.
  - gouranga 15y ago
    I'd like to know the reason that was downvoted...
  - cageface 15y ago
    You have to admit that a 20 year old app with over a PB of data and millions of lines of code is not exactly typical. For most apps the benefits of a structured database outweigh the costs.
  - neeleshs 15y ago
    Interesting to know what abstractions were found useful when migrating from VAX/VMS to Java/J2EE to .Net.
    - gouranga 15y ago
      Hibernate and nhibernate.
- deleted 15y ago
  [deleted]
joevandyk 15y ago
I've always found doing logic and processing in the database to be much faster and much more efficient than doing it outside. The closer you can process data to the data store, the faster it goes. Using materialized views can get you a long ways. There's also no reason why you can't use multiple smaller databases. Use dblink. http://www.postgresql.org/docs/9.1/static/contrib-dblink-connect.html http://www.postgresql.org/docs/9.1/static/contrib-dblink-con... I've never had a problem managing "schema and dependencies" on larger systems. Not sure what you are referring to there.
- gouranga 15y ago
  It goes much faster I agree. However you are very short sighted. That black box doesn't go on forever. It will work fine for the first 50 users, seems ok for the next 100, 500 we're doing ok. 1000 BOOM! Then what do you do? I'd rather have a slightly higher latency and be able to scale to 100,000 users. Materialized views are not needed with CQRS architecture. ALL tables (or sets) are materialized views already. No joins are ever made. dblink is ugly and if you ask me a problem, not a solution (latency, network io, memory). I'm not sure what you classify as "larger" but I'm talking tens of Tb, 1500 tables and over 2 million LOC in the application.
  - joevandyk 15y ago
    Instagram did just fine with postgresql with 27 million users. Many logical shards spread over a smaller amount of physical ones. http://www.databasesoup.com/2012/04/sharding-postgres-with-instagram.html http://www.databasesoup.com/2012/04/sharding-postgres-with-i...
    - gouranga 15y ago
      Instagram isn't very complicated to be honest so I doubt that's a good scalability example.
      - joevandyk 15y ago
        Skype's plan to support 1 billion users w/ postgres a few years ago? http://highscalability.com/skype-plans-postgresql-scale-1-billion-users http://highscalability.com/skype-plans-postgresql-scale-1-bi... and http://www.slideshare.net/adorepump/small-overview-of-skype-database-tools http://www.slideshare.net/adorepump/small-overview-of-skype-... Billions of records, 10,000 transactions per second. This was a couple of years ago. Dunno if Skype is complicated enough for you.
tensor 15y ago
Throwing out decades of work on storing data just because it's a hard problem won't help you solve the problem. All you are advocating is that instead of using an existing well tested solution, you should write your own and deal with all the assorted costs of redoing that work, and probably poorly. Similarly, using an ORM doesn't help you. All that does is hide the details from you. It still uses SQL underneath, but limits what you can do with it. Let's talk schemas. You have one whether your database understands it or not. Trying to pretend you don't have one doesn't make managing your schema any easier. It doesn't make building indicies on fields any easier. These are hard problem and pretending they don't exist by rolling your own implementation or hiding them under an ORM won't make them go away.
- gouranga 15y ago
  Decades of work is pretty wrong to be honest. They approach the wrong end of the problem. They never consider efferent coupling between database objects which is the absolute killer in anything but simple projects regardless of what is promoted or discussed. SQL makes that an order of magnitude worse. It's also not about it being a hard problem (which it incidentally is but that's another story). It's about controlling the scale of the problem as time goes on and sufficiently abstracting it without adding complexity. That's not hiding it - that's making it a non issue. Permanently. I am only advocating using a minimal subset of the functionality to aid migration in the future between engines, platforms, versions and storage paradigms. An ORM (in our case NHibernate) is a clean abstraction over the top which allows us to be completely mobile between vendors and versions. There is no NIH in it. The world changes pretty quick. Those who adapt quickly stay around.
  - josephcooney 15y ago
    A lot of 'enterprise' products take this route so they can be deployed on top of your RDBMS of choice (Oracle, DB2). They usually get pretty terrible performance out of their RDBMS as a result.
    - gouranga 15y ago
IsTom 15y ago
KV stores are painful to use. Only huge applications that are DB-bound have the problem of scaling database and they usually have a lot of money. Also you don't need your replicas to be as strong as your master db server.
- gouranga 15y ago
  I agree. I'm not advocating KV stores.
lwat 15y ago
I disagree with everything you say here. I seriously doubt you've been using databases for 20 years. Which databases have you used? How many hours experience do you have with each of them? Just because some people do it wrong doesn't mean you should throw out the baby with the bath water! In every stack I've used the database has been the most robust and amazing piece of technology and it's been an absolute dream come true for every project I've been involved in. (SQL Server / PostgreSQL - 10,000 hours experience)
- gouranga 15y ago
  It is robust and amazing. I agree but if you push it to the limits it will bite you on the arse. Probably about the same as you on just SQL Server and half as much again on Oracle...
  - lwat 15y ago
    If you push ANYTHING to the limits it will cause issues. I don't see how that's an argument against properly utilizing an RDBMS.
sciurus 15y ago
"The DBMS is a black box" The DBA you refuse to hire would likely disagree.
- gouranga 15y ago
  Which is why we don't like them :) Databases are for putting stuff in when you don't immediately need it. The DBA mindset pushes that the DB is the hammer for all nails.
  - tensor 15y ago
    Your ORM is a black box. One you don't seem to understand on top. Also, ironically, you are pushing your anti-RDBMS agenda as a solution for all problems complete with information devoid attacks such as "The world changes pretty quick. Those who adapt quickly stay around."Among others. You clearly are not in willing to reflect on the actual problems being presented to you. Instead you are clinging desperately to some sort of ideology that either you yourself came up with or someone sold you on. Look at the basic problems. You are not solving them or making them go away by "abstracting them away." The reason is that these particular problems can't be abstracted away. For example, you raise joins into the code that your programmers write instead of the RDMBS level. But you are still writing joins, they are just for loops on top of your ORM solution. Now you incur the cost of writing them manually each time and testing them, also dealing with ACID which almost surely you are not doing correctly. Or if you are, you are spending huge amounts of money to make it right. Even worse, your solution can't possible be as fast as the RDBMS solution since you need to go through all sorts of heavy abstraction. You are not seeing the actual problems at all and not surprisingly demonizing the type of people who do understand them: DBAs (assuming we are talking about expensive professional DBAs).
    - gouranga 15y ago
      Did I say I was anti-RDBMS? Not at all. I'm just careful. I still use RDBMS for everything - just differently and I avoid a fair amount of the tempting features. I'm not an advocate of NoSQL. The ideology came out of necessity and 20 years of watching people fall down the same holes. Joins - loops on top of ORM? Doing ACID myself? Do you understand what a non-toy ORM does? Do you understand the separation between the data model and the domain model? Sounds like you don't.
moe 15y ago
Scale up is damn expensive. When you need that 64 core 96Gb machine to run it all on with 6 replicas will see what I mean. Damn expensive, eh? As it happens I bought a couple boxes that size just recently. Except with 256G RAM instead of 96G. They cost $8500 per shot. That's not even 2 months of salary for a single developer. Don't underestimate Moore's Law.
- gouranga 15y ago
  That bit is cheap. How much does the storage subsystem cost? You know the whole stack of fully redundant iSCSI/FC kit behind it?
  - moe 15y ago
    How much does the storage subsystem cost? 15x 146G SAS spindles are included with the price. You know the whole stack of fully redundant iSCSI/FC kit behind it? Why, yes? We do run a FC fabric and various DAS boxes for various purposes, but not on the DB servers. I'm not sure what you're arguing about here, the figures (labor vs hardware cost) are not even in the same ballpark. As a rule of thumb: If a piece of gear can save your team 1 month of work then it usually pays for itself.
    - gouranga 15y ago
      Where do you but that from - I'm genuinely interested.
      - moe 15y ago
        Our Colo-Partner acquires and assembles the parts for us (mostly SuperMicro). In the US I've heard good things about ThinkMate. E.g. this is close to our configuration but with a different chassis; http://www.thinkmate.com/System/RAX_QS5-4410 http://www.thinkmate.com/System/RAX_QS5-4410
      - andyzweb 15y ago
        http://teddziuba.com/2010/12/the-3-basic-tools-of-systems-engineering.html http://teddziuba.com/2010/12/the-3-basic-tools-of-systems-en... http://blogs.msdn.com/b/elee/archive/2009/03/11/source-code-is-a-liability-not-an-asset.aspx http://blogs.msdn.com/b/elee/archive/2009/03/11/source-code-... In general throw money at the problem first, resort to code last
bunderbunder 15y ago
-scalability: I/O is often much more expensive than logic. With a well-normalized, well-indexed database, pushing logic off to the server often gives the query optimizer a chance to radically reduce how much data is retrieved. That can translate to less load on the disk controller, memory, controller, CPU, and network, and by extension more scalability. Of course, it might not work out that way, depending on the specific case. As usual, metrics rule all and slavishly following what should be rules of thumb at best is the worst kind of premature optimization. - Compelxity, lock in: The DBMS is like any other software component: If you're actually using it to its capabilities, then it's trivially obvious that it should be difficult to replace, because there are a lot of capabilities you have to find substitutes for. True, you could avoid being coupled to your dependencies by choosing to reinvent wheels whenever possible instead. That would obviously reduce coupling. But bloating your codebase is generally not the best way to make your code more maintainable. - schema: Pretty much any programming task is hell for people who aren't skilled at it. If you find that you're not very good at using the standard tools of a carpenter, that does not mean that the tools are fundamentally misdesigned. Much more likely, it means that you should be hiring someone else to do your carpentry for you. It's worth pointing out here that programmers shouldn't need to touch the schema in a well-designed system: They should be touching views and stored procedures that abstract away the implementation details.
ams6110 15y ago
If you are paying for your RDBMS (e.g. Oracle, SQL Server) and not using its features you are throwing your money away. Even if you are using a "free" database, if you're just tossing aside things like data types, permissions, referential integrity and check constraints, natural declarative handling of sets, and writing all that in your client application what happens when you want to put another front end on your data? You have to do all that again. Where's the hell then? What happens when someone has to do a SQL update manually? Referential integrity can save you from shooting yourself in the foot there. SQL injection? Much less likely if you're using stored procedures with parameters as your client interface. You have lock-in all over the place regardless. Languages. Frameworks. ORMs. Operating systems. All are decisions that are painful to change when something "better" comes along. You're better off making those decisions and then leveraging them for all you can get. RDBMSs have been around longer than many languages and most modern frameworks, and are still a very good, probably the best, general purpose solution for data storage and management.
- gouranga 15y ago
  We're paying for the support and insurance against failure, not the features. You don't do manual SQL updates. Why would you if you have an ORM? You update the domain and it does all of that stuff for you. I only care about the domain which is the stuff that makes the money - not writing SQL. ORMs don't suffer from SQL injection as they use properly parameterised prepared statements. Regarding lock in you are correct, but the RDBMS is the most politicised and expensive component which is why yo uneed the flexibility. I am in no way saying "RDBMSs are shit - NoSQL forever!".
Duff 15y ago
Huh? Your worried about complexity and lock in, so you want devs to replicate tried and true database features? I can understand wanting to avoid Oracle lockin.... But Postgres?
- gouranga 15y ago
  There is no replication of features at all (backup/restore is fundamentally different - it exports the domain to disk, not the schema). Well we could shift a deployment to Amazon RDS (Oracle or MySQL) quite happily in a couple of hours without having to rewrite everything because we used all the postgresql features...
sbov 15y ago
You seem to be coming a very specific viewpoint. Not all things you build will require scaling out. Not all systems will become as complex as you seem to be envisioning. Not all systems will be around in 20 years, much less require the types of changes in 20 years that necessitate changing the datastore. I view it as an optimization problem. Based upon what you're building, the aforementioned issues are more or less likely to occur. Optimize for what is likely. If I'm building an official site for a popular product, I'd bet on all the above being true. Some of what I build, even if 100% of people who could use the product would use it, will run just fine with a single database server that costs in the area of $8k. Planning for scale out is a complete waste of time here. Most of what I build starts out simple. Most of what I build starts out with 0 users. Will it be popular, will it last, will it need to scale? We have no clue. For these, we keep it simple: modularize our code based upon features, use our datastores to their fullest within features, and keep in mind how we might scale things out (which may ultimately be incorrect) - if we have to. Usually we don't have to. I believe it to be a fair balance in this scenario. Also, given your constraints, I don't quite understand how you could find a future datastore that is a better solution. How can you exploit better solutions and avoid lockin at the same time? If a better solution comes along in the future, it seems like you would be using features that would lock your datastore to that newer, better solution. However, you cannot do this because you need to avoid lockin.
- gouranga 15y ago
  I agree. For a fresh start (greenfield) it's probably fine. Unfortunately for 90%+ of IT products, it's not a greenfield. That's where the problem comes.
fallous 15y ago
Don't confuse the "RDBMS uber alles!" for all storage is equivalent to "RDBMS is always a fail." Certainly over the last decade or so there was a mistaken belief that all data should be in an RDBMS, even though they were flattened or required no relational algebra, but that doesn't discount the benefit of a tool that can handle such relations when necessary. Redis solves a certain set of problems for me, Memcached as well. ElasticSearch solves another set, and RDBMS yet another. Square pegs for square holes, round pegs for round holes.
it 15y ago
On the other hand, SQL lets you write concise and efficient queries that run without lots of round-trips between the db and whatever machine you're accessing it from. ORMs force you to write less expressive code that is less efficient, hardly a winning combination.
- gouranga 15y ago
  To be honest, if you have a good ORM, it's better at writing SQL than you are. Compare to compilers being better at writing machine code...
kfool 15y ago
- schema. Maintaining schema and dependencies is hell on all but a small system When you have the tools you have the power.
jacques_chester 15y ago
Being "locked in" to an opensource database doesn't strike me as some kind of unbearable death trap.

platonichvn 15y ago

I agree with you but I often question when people list "lock in" as an issue. How often do you see projects where they decide to move from one db to another? Even in cases where an ORM is used, making the migration process painless in theory, I doubt many projects take the risk and exercise their freedom of not being locked in.

joevandyk 15y ago
I've only seen people move to postgresql (generally from mysql). I've never heard of people moving away from postgresql. Granted, that's a small sample size.
- pvh 15y ago
  Etsy did. I don't think it was a good decision and I think the reasons they did had more to do with resetting from some bad architecture decisions, but they're the example I know of. http://codeascraft.etsy.com/2011/09/28/scaling-etsy-what-went-wrong-what-went-right/ http://codeascraft.etsy.com/2011/09/28/scaling-etsy-what-wen... I think the big lesson here would be: don't have separate DBA and Engineering teams. On the other hand, I could name quite a few people who went the other way, like Urban Dictionary who run on our http://postgres.heroku.com http://postgres.heroku.com service, too. (I don't normally name customers, but he recorded a video saying he liked us so I think it's fair game. http://success.heroku.com/urbandictionary http://success.heroku.com/urbandictionary )
gouranga 15y ago
If you look at most "corporate" IT platforms, they tend to have favourite database engines that they need to use. If you have DBMS portability, then it gets one foot in the door straight away. Also contrary to the norm here, some applications last a long time. The one I work with daily is actually 20 years old this year. In that time a lot has changed and in the next 20 years a lot will as well. It makes sense for long-lived applications to plan well ahead and consider paradigm and architecture changes. This platform started in C++ on OS/2 with Oracle running on VAX/VMS at the back end. It went to Java/J2EE with an Oracle back end in the early 00's. It's now SQL Server and .Net.
- gouranga 15y ago
  I'd like to know the reason that was downvoted...
- cageface 15y ago
  You have to admit that a 20 year old app with over a PB of data and millions of lines of code is not exactly typical. For most apps the benefits of a structured database outweigh the costs.
- neeleshs 15y ago
  Interesting to know what abstractions were found useful when migrating from VAX/VMS to Java/J2EE to .Net.
  - gouranga 15y ago
    Hibernate and nhibernate.
deleted 15y ago
[deleted]

joevandyk 15y ago

I've always found doing logic and processing in the database to be much faster and much more efficient than doing it outside. The closer you can process data to the data store, the faster it goes. Using materialized views can get you a long ways. There's also no reason why you can't use multiple smaller databases. Use dblink. http://www.postgresql.org/docs/9.1/static/contrib-dblink-connect.html http://www.postgresql.org/docs/9.1/static/contrib-dblink-con... I've never had a problem managing "schema and dependencies" on larger systems. Not sure what you are referring to there.

gouranga 15y ago
It goes much faster I agree. However you are very short sighted. That black box doesn't go on forever. It will work fine for the first 50 users, seems ok for the next 100, 500 we're doing ok. 1000 BOOM! Then what do you do? I'd rather have a slightly higher latency and be able to scale to 100,000 users. Materialized views are not needed with CQRS architecture. ALL tables (or sets) are materialized views already. No joins are ever made. dblink is ugly and if you ask me a problem, not a solution (latency, network io, memory). I'm not sure what you classify as "larger" but I'm talking tens of Tb, 1500 tables and over 2 million LOC in the application.
- joevandyk 15y ago
  Instagram did just fine with postgresql with 27 million users. Many logical shards spread over a smaller amount of physical ones. http://www.databasesoup.com/2012/04/sharding-postgres-with-instagram.html http://www.databasesoup.com/2012/04/sharding-postgres-with-i...
  - gouranga 15y ago
    Instagram isn't very complicated to be honest so I doubt that's a good scalability example.
    - joevandyk 15y ago
      Skype's plan to support 1 billion users w/ postgres a few years ago? http://highscalability.com/skype-plans-postgresql-scale-1-billion-users http://highscalability.com/skype-plans-postgresql-scale-1-bi... and http://www.slideshare.net/adorepump/small-overview-of-skype-database-tools http://www.slideshare.net/adorepump/small-overview-of-skype-... Billions of records, 10,000 transactions per second. This was a couple of years ago. Dunno if Skype is complicated enough for you.

tensor 15y ago

Throwing out decades of work on storing data just because it's a hard problem won't help you solve the problem. All you are advocating is that instead of using an existing well tested solution, you should write your own and deal with all the assorted costs of redoing that work, and probably poorly. Similarly, using an ORM doesn't help you. All that does is hide the details from you. It still uses SQL underneath, but limits what you can do with it. Let's talk schemas. You have one whether your database understands it or not. Trying to pretend you don't have one doesn't make managing your schema any easier. It doesn't make building indicies on fields any easier. These are hard problem and pretending they don't exist by rolling your own implementation or hiding them under an ORM won't make them go away.

gouranga 15y ago
Decades of work is pretty wrong to be honest. They approach the wrong end of the problem. They never consider efferent coupling between database objects which is the absolute killer in anything but simple projects regardless of what is promoted or discussed. SQL makes that an order of magnitude worse. It's also not about it being a hard problem (which it incidentally is but that's another story). It's about controlling the scale of the problem as time goes on and sufficiently abstracting it without adding complexity. That's not hiding it - that's making it a non issue. Permanently. I am only advocating using a minimal subset of the functionality to aid migration in the future between engines, platforms, versions and storage paradigms. An ORM (in our case NHibernate) is a clean abstraction over the top which allows us to be completely mobile between vendors and versions. There is no NIH in it. The world changes pretty quick. Those who adapt quickly stay around.
- josephcooney 15y ago
  A lot of 'enterprise' products take this route so they can be deployed on top of your RDBMS of choice (Oracle, DB2). They usually get pretty terrible performance out of their RDBMS as a result.
  - gouranga 15y ago

IsTom 15y ago

KV stores are painful to use. Only huge applications that are DB-bound have the problem of scaling database and they usually have a lot of money. Also you don't need your replicas to be as strong as your master db server.

gouranga 15y ago
I agree. I'm not advocating KV stores.

lwat 15y ago

I disagree with everything you say here. I seriously doubt you've been using databases for 20 years. Which databases have you used? How many hours experience do you have with each of them? Just because some people do it wrong doesn't mean you should throw out the baby with the bath water! In every stack I've used the database has been the most robust and amazing piece of technology and it's been an absolute dream come true for every project I've been involved in. (SQL Server / PostgreSQL - 10,000 hours experience)

gouranga 15y ago
It is robust and amazing. I agree but if you push it to the limits it will bite you on the arse. Probably about the same as you on just SQL Server and half as much again on Oracle...
- lwat 15y ago
  If you push ANYTHING to the limits it will cause issues. I don't see how that's an argument against properly utilizing an RDBMS.

sciurus 15y ago

"The DBMS is a black box" The DBA you refuse to hire would likely disagree.

gouranga 15y ago
Which is why we don't like them :) Databases are for putting stuff in when you don't immediately need it. The DBA mindset pushes that the DB is the hammer for all nails.
- tensor 15y ago
  Your ORM is a black box. One you don't seem to understand on top. Also, ironically, you are pushing your anti-RDBMS agenda as a solution for all problems complete with information devoid attacks such as "The world changes pretty quick. Those who adapt quickly stay around."Among others. You clearly are not in willing to reflect on the actual problems being presented to you. Instead you are clinging desperately to some sort of ideology that either you yourself came up with or someone sold you on. Look at the basic problems. You are not solving them or making them go away by "abstracting them away." The reason is that these particular problems can't be abstracted away. For example, you raise joins into the code that your programmers write instead of the RDMBS level. But you are still writing joins, they are just for loops on top of your ORM solution. Now you incur the cost of writing them manually each time and testing them, also dealing with ACID which almost surely you are not doing correctly. Or if you are, you are spending huge amounts of money to make it right. Even worse, your solution can't possible be as fast as the RDBMS solution since you need to go through all sorts of heavy abstraction. You are not seeing the actual problems at all and not surprisingly demonizing the type of people who do understand them: DBAs (assuming we are talking about expensive professional DBAs).
  - gouranga 15y ago
    Did I say I was anti-RDBMS? Not at all. I'm just careful. I still use RDBMS for everything - just differently and I avoid a fair amount of the tempting features. I'm not an advocate of NoSQL. The ideology came out of necessity and 20 years of watching people fall down the same holes. Joins - loops on top of ORM? Doing ACID myself? Do you understand what a non-toy ORM does? Do you understand the separation between the data model and the domain model? Sounds like you don't.

moe 15y ago

Scale up is damn expensive. When you need that 64 core 96Gb machine to run it all on with 6 replicas will see what I mean. Damn expensive, eh? As it happens I bought a couple boxes that size just recently. Except with 256G RAM instead of 96G. They cost $8500 per shot. That's not even 2 months of salary for a single developer. Don't underestimate Moore's Law.

gouranga 15y ago
That bit is cheap. How much does the storage subsystem cost? You know the whole stack of fully redundant iSCSI/FC kit behind it?
- moe 15y ago
  How much does the storage subsystem cost? 15x 146G SAS spindles are included with the price. You know the whole stack of fully redundant iSCSI/FC kit behind it? Why, yes? We do run a FC fabric and various DAS boxes for various purposes, but not on the DB servers. I'm not sure what you're arguing about here, the figures (labor vs hardware cost) are not even in the same ballpark. As a rule of thumb: If a piece of gear can save your team 1 month of work then it usually pays for itself.
  - gouranga 15y ago
    Where do you but that from - I'm genuinely interested.
    - moe 15y ago
      Our Colo-Partner acquires and assembles the parts for us (mostly SuperMicro). In the US I've heard good things about ThinkMate. E.g. this is close to our configuration but with a different chassis; http://www.thinkmate.com/System/RAX_QS5-4410 http://www.thinkmate.com/System/RAX_QS5-4410
    - andyzweb 15y ago
      http://teddziuba.com/2010/12/the-3-basic-tools-of-systems-engineering.html http://teddziuba.com/2010/12/the-3-basic-tools-of-systems-en... http://blogs.msdn.com/b/elee/archive/2009/03/11/source-code-is-a-liability-not-an-asset.aspx http://blogs.msdn.com/b/elee/archive/2009/03/11/source-code-... In general throw money at the problem first, resort to code last

bunderbunder 15y ago

-scalability: I/O is often much more expensive than logic. With a well-normalized, well-indexed database, pushing logic off to the server often gives the query optimizer a chance to radically reduce how much data is retrieved. That can translate to less load on the disk controller, memory, controller, CPU, and network, and by extension more scalability. Of course, it might not work out that way, depending on the specific case. As usual, metrics rule all and slavishly following what should be rules of thumb at best is the worst kind of premature optimization. - Compelxity, lock in: The DBMS is like any other software component: If you're actually using it to its capabilities, then it's trivially obvious that it should be difficult to replace, because there are a lot of capabilities you have to find substitutes for. True, you could avoid being coupled to your dependencies by choosing to reinvent wheels whenever possible instead. That would obviously reduce coupling. But bloating your codebase is generally not the best way to make your code more maintainable. - schema: Pretty much any programming task is hell for people who aren't skilled at it. If you find that you're not very good at using the standard tools of a carpenter, that does not mean that the tools are fundamentally misdesigned. Much more likely, it means that you should be hiring someone else to do your carpentry for you. It's worth pointing out here that programmers shouldn't need to touch the schema in a well-designed system: They should be touching views and stored procedures that abstract away the implementation details.

ams6110 15y ago

If you are paying for your RDBMS (e.g. Oracle, SQL Server) and not using its features you are throwing your money away. Even if you are using a "free" database, if you're just tossing aside things like data types, permissions, referential integrity and check constraints, natural declarative handling of sets, and writing all that in your client application what happens when you want to put another front end on your data? You have to do all that again. Where's the hell then? What happens when someone has to do a SQL update manually? Referential integrity can save you from shooting yourself in the foot there. SQL injection? Much less likely if you're using stored procedures with parameters as your client interface. You have lock-in all over the place regardless. Languages. Frameworks. ORMs. Operating systems. All are decisions that are painful to change when something "better" comes along. You're better off making those decisions and then leveraging them for all you can get. RDBMSs have been around longer than many languages and most modern frameworks, and are still a very good, probably the best, general purpose solution for data storage and management.

gouranga 15y ago
We're paying for the support and insurance against failure, not the features. You don't do manual SQL updates. Why would you if you have an ORM? You update the domain and it does all of that stuff for you. I only care about the domain which is the stuff that makes the money - not writing SQL. ORMs don't suffer from SQL injection as they use properly parameterised prepared statements. Regarding lock in you are correct, but the RDBMS is the most politicised and expensive component which is why yo uneed the flexibility. I am in no way saying "RDBMSs are shit - NoSQL forever!".

Duff 15y ago

Huh? Your worried about complexity and lock in, so you want devs to replicate tried and true database features? I can understand wanting to avoid Oracle lockin.... But Postgres?

gouranga 15y ago
There is no replication of features at all (backup/restore is fundamentally different - it exports the domain to disk, not the schema). Well we could shift a deployment to Amazon RDS (Oracle or MySQL) quite happily in a couple of hours without having to rewrite everything because we used all the postgresql features...

sbov 15y ago

You seem to be coming a very specific viewpoint. Not all things you build will require scaling out. Not all systems will become as complex as you seem to be envisioning. Not all systems will be around in 20 years, much less require the types of changes in 20 years that necessitate changing the datastore. I view it as an optimization problem. Based upon what you're building, the aforementioned issues are more or less likely to occur. Optimize for what is likely. If I'm building an official site for a popular product, I'd bet on all the above being true. Some of what I build, even if 100% of people who could use the product would use it, will run just fine with a single database server that costs in the area of $8k. Planning for scale out is a complete waste of time here. Most of what I build starts out simple. Most of what I build starts out with 0 users. Will it be popular, will it last, will it need to scale? We have no clue. For these, we keep it simple: modularize our code based upon features, use our datastores to their fullest within features, and keep in mind how we might scale things out (which may ultimately be incorrect) - if we have to. Usually we don't have to. I believe it to be a fair balance in this scenario. Also, given your constraints, I don't quite understand how you could find a future datastore that is a better solution. How can you exploit better solutions and avoid lockin at the same time? If a better solution comes along in the future, it seems like you would be using features that would lock your datastore to that newer, better solution. However, you cannot do this because you need to avoid lockin.

gouranga 15y ago
I agree. For a fresh start (greenfield) it's probably fine. Unfortunately for 90%+ of IT products, it's not a greenfield. That's where the problem comes.

fallous 15y ago

Don't confuse the "RDBMS uber alles!" for all storage is equivalent to "RDBMS is always a fail." Certainly over the last decade or so there was a mistaken belief that all data should be in an RDBMS, even though they were flattened or required no relational algebra, but that doesn't discount the benefit of a tool that can handle such relations when necessary. Redis solves a certain set of problems for me, Memcached as well. ElasticSearch solves another set, and RDBMS yet another. Square pegs for square holes, round pegs for round holes.

it 15y ago

On the other hand, SQL lets you write concise and efficient queries that run without lots of round-trips between the db and whatever machine you're accessing it from. ORMs force you to write less expressive code that is less efficient, hardly a winning combination.

gouranga 15y ago
To be honest, if you have a good ORM, it's better at writing SQL than you are. Compare to compilers being better at writing machine code...

kfool 15y ago

- schema. Maintaining schema and dependencies is hell on all but a small system When you have the tools you have the power.

jacques_chester 15y ago

Being "locked in" to an opensource database doesn't strike me as some kind of unbearable death trap.