5 ms·
From a software engineering perspective, this is entirely backwards. Databases offer a great way of looking at your data. However they have very poor stories
by btilly 13y ago
From a software engineering perspective, this is entirely backwards.
Databases offer a great way of looking at your data.
However they have very poor stories around version control, rollout, rollback, separating metadata from data, and creating multiple parallel environments for production, qa, and every dev engineer. Yes, I know that database vendors claim otherwise, but compared to what is standard for most programming environments these days, it is a major limitation.
But it doesn't stop there. If you are lucky enough to get a busy site, you will find that the database is the inevitable bottleneck that can't simply be scaled by putting machines in parallel. The more work you push down to that, the sooner you'll hit that bottleneck, and the harder it is to fix once you do. When you've experienced, as I have, the joys of periodic outages because the database choked at a million dynamic pages/day, you'll value moving logic out of the database, the ability to put caching in front of it, etc.
Now to all of this the database vendors have a trump card, what happens if multiple applications access the database? If your business logic is there, then you're fine, otherwise you're reimplementing that logic multiple times! The thing is that in the 15 years that I've seen people making that argument, I've not encountered a lot of organizations opening up a business critical transactional database for random applications. And in practice it is easier and saner to put an RPC layer on top of of your existing app, in which case you get the business logic that way.
I guess I should somewhere note that I spent several years of my life with my primary job being to write fancy SQL. I was also the go to person on the software development side for "get us to scale the database better". I do not criticize databases out of a lack of knowledge of how to use them. I criticize them based on a large amount of experience with how to make the most effective use of their capabilities.
- asdf1234 13y agoThere's nothing terribly complicated about recursive queries and common table expressions, and for the use cases in which they're commonly used (hierarchical data being one of those) they perform significantly better than all of the popular SQL alternatives in almost every case.
- btilly 13y agoI was referring to putting business logic in your database, and not any particular query technique.
- jeffdavis 13y agoSometimes it just saves you huge amounts of time and code to do something in the database though. For instance, if you need to join against a remote data source, you can: 1. Write a bunch of code to pull the data from both sources and join it in the application. You have to pick a join algorithm at the beginning, and if your data changes you may need to rewrite it later. 2. Add a remote table in the database and just do the join there. The database can use statistics to pick a join algorithm for you. I'm sure there are cases where you'd pick #1. But in many cases, the database offers a feature that allows you to implement a solution and call it "done" in minutes or hours where an application-based solution would drag on for days or weeks and carry a larger maintenance burden.
- btilly 13y agoPlease go to https://news.ycombinator.com/item?id=5934136 https://news.ycombinator.com/item?id=5934136 and look at the final paragraph. Yes, I'm well aware of what databases can do for me, and have made a good living getting them to jump through hoops to do their thing. However part of knowing how to use a tool well is knowing its limitations. And from what I've learned, a web-based CRUD application is usually better off with business logic in the web layer, and not in the database. Also I have absolutely no idea why you brought up joining against remote data sources. However I've had to do it against multiple databases over the years. And in my experience with Access, Sybase, Oracle, PostgreSQL and MySQL, I found that for all but the simplest cases, #1 was preferable. For each of those databases I can name a case where I started with #2 with clear issues, and wound up with #1 working well. Why? Because if you understand query performance, you can make #1 work well. And databases tend to do a horrible job of understanding what's going on with remote data sources and picking good query plans for them. Which can cause #2 to suck in hard to fix ways. (Plus with PostgreSQL specifically there were security issues that made the DBA unwilling to open up one database remotely to the other. He was willing to let my application access both, but didn't want to allow other applications to be able to do it.) That said, most people who need access to remote data are in a simple circumstance. And I know a lot more about query performance than your average software developer. So if #2 can work for you, you should go that route.
- jeffdavis 13y ago"will find that the database is the inevitable bottleneck that can't simply be scaled by putting machines in parallel" In context, the alternatives available are something like: 1. Issue one recursive query 2. Issue many non-recursive queries 3. Use some kind of denormalized structure that is tuned to the one specific recursive query in question, but might perform badly on other queries. #3 can work if you are focusing on the optimization of a single query and can pick a structure tuned to that one query. You can even parallelize it, because you can know the partitioning scheme that will work best for that one query. But outside of those constraints, or if the query profile changes over time, it does not work well. You end up needing to duplicate the data into many different structures to accommodate many different queries. And I don't see how #2 has any advantage over #1 performance wise. Add caching layers or other performance hacks as needed.
- btilly 13y agoFor further context, I'm responding to someone who is advocating putting your business layer in your database. I'm not against using tools that exist to make accessing your database more efficient - including recursive queries. I'm against making your database responsible for more of your application than it needs to be.
- joevandyk 13y agoWhy can't you version control database code/schema? I can do it fine using Rails migrations. I've also been looking at http://sqitch.org/ http://sqitch.org/ as well. Why can't you create parallel instances for each dev/environment? (If licensing costs are a problem, maybe look at a free one?)
- btilly 13y agoWhy can't you version control database code/schema? I can do it fine using Rails migrations. I've also been looking at http://sqitch.org/ http://sqitch.org/ as well. If you're willing to use a third party tool to manage the database, and ONLY use that tool, then you can solve version control/migration. I've used and written such tools myself. However once you have software development and database administration separated, the DBAs generally need access to the database in a way that does not go through your tool / software development cycle / etc. And once you have random things in your database that your tool knows nothing about, you're on your way to maintenance headaches. Furthermore third party tools like this generally want your site to be offline during database migrations. But if you've got a production site going 24x7, then you really want to figure out how to roll out upgrades without downtime. This is much, much easier if you can upgrade database and website independently of each other, with the database going first. That important detail tends to be poorly handled in my experience. (Though things change and I'd love to hear if that had improved.) Furthermore you have the headache of identifying the difference between data tables and metadata tables. That is, some tables contain data that gets modified on the fly during normal use. (eg user information) Other tables contain data that is really application configuration. (eg allowed statuses for something) Why can't you create parallel instances for each dev/environment? (If licensing costs are a problem, maybe look at a free one?) You can. The difficulty lies in keeping them synchronized with what is in your repository. And also in providing ways to copy user information to the private instances so that devs can reproduce production problems. (Doubly so when your production problem is a performance issue that requires a copy of production data to recreate.) These problems are all solvable. I've solved them. But the general story that databases provide on these issues does not match the simplicity and flexibility of having databases focus on providing a common data structure, and the application being stored elsewhere in your favorite source control system (eg git).
- jacques_chester 13y agoIn general, "ordinary" software has these advantages for the simple reason that it's ephemeral. The software starts with a clean universe each time you kill it and relaunch. That makes lots of things much, much easier. Databases don't get to do that. The data is persistant between instances, between sessions and between copies. It never goes away and the database must always consider all other considerations secondary. If you want to change the object model of a Ruby program, you change the source and relaunch. If you want to change the schema of a database, you will need to provide the database instructions on what you want to do with all the existing data that is now inconsistent. If you look at the hoops people jump through in always-on software, as opposed to "restart with #e33cf4" software, it's similarly gnarly. Relational databases are kissing cousins with heavy-duty type systems. They make you do homework and eat your vegetables to forestall classes of error lots of people have never encountered and so discount to zero. Then it's decided that they aren't agile and fwoosh, out they go.
- btilly 13y agoAbsolutely. Which is why I find it works well to have code flow from dev to production, and databases flow the other way. With all changes starting as well-tested patches that are then applied in production and propagated elsewhere.