3 ms·

Crazy idea: Write your data-handling code in it's own tight little module. Use good abstractions so that the rest of your app doesn't give a crap HOW it's happ

Crazy idea: Write your data-handling code in it's own tight little module. Use good abstractions so that the rest of your app doesn't give a crap HOW it's happening. When it's time to update to millions of clustered servers running bigTable, you rewrite the module, and you're done. Software Engineering saves the day!

alexgartrell 17y ago
3 upvotes and 2 downvotes without a single response. As a junior member of Hacker News, I demand an education when I'm downvoted! :)
- akeefer 17y ago
  I wasn't one of the ones who upvoted or downvoted you, but just because you asked . . . while abstractions around your data-handling code are always a good idea, on some level they always leak. (For example, see the classic post http://blogs.tedneward.com/2006/06/26/The+Vietnam+Of+Computer+Science.aspx http://blogs.tedneward.com/2006/06/26/The+Vietnam+Of+Compute... describing how ORM, one of the most common approaches to encapsulating data-handling code, tends to fail). Most of your code is data-handling code on some fundamental level, so it's pretty inevitable that it's going to matter to your application how that data is stored and what sorts of operations you can do on it, and that's especially true if you care about performance (which you generally always do after some level). So even if you have a nice abstraction of your query layer, the kinds of questions you can ask the database efficiently depend heavily on the underlying storage mechanism, and so your application logic has to be built so it only asks the right kinds of questions. Even abstracting away the differences between, say, Oracle and MS SQLServer and MySQL is difficult enough, because of the different capabilities and performance characteristics. When you do that, you basically end up coding to the lowest common denominator, which can often limit what your applications does. Trying to come up with an abstraction that can encapsulate the difference between a row- versus column-oriented database, or between a relational database and some other kind of storage, is pretty much a losing proposition: they're too fundamentally different in terms of what kind of data you can store, how you can store it, how you can query it, what kinds of transactional guarantees you get, and what operations are fast and which ones are slow. So you really do kind of have to take your best shot at it, choose an approach, and if you choose wrong and have to change, it's just going to hurt. A lot. Good encapsulation and abstraction will ease some of the pain, but it's more like drinking whiskey before your leg gets sawed off than it is like general anesthesia.
tybris 17y ago
Well, you're right in the sense that Newton was right about gravity. It works perfectly down here on earth, but breaks down out in the universe. In a distributed system you'll want to take into account caching, replication, atomicity, security, etc. In other words, that's either going to be a hell of a complex API or an API that gives you too little control to build a high-performance system. There's no telling what works best, every computer system is a set of trade-offs and it always depends on the application.

alexgartrell 17y ago

3 upvotes and 2 downvotes without a single response. As a junior member of Hacker News, I demand an education when I'm downvoted! :)

akeefer 17y ago
I wasn't one of the ones who upvoted or downvoted you, but just because you asked . . . while abstractions around your data-handling code are always a good idea, on some level they always leak. (For example, see the classic post http://blogs.tedneward.com/2006/06/26/The+Vietnam+Of+Computer+Science.aspx http://blogs.tedneward.com/2006/06/26/The+Vietnam+Of+Compute... describing how ORM, one of the most common approaches to encapsulating data-handling code, tends to fail). Most of your code is data-handling code on some fundamental level, so it's pretty inevitable that it's going to matter to your application how that data is stored and what sorts of operations you can do on it, and that's especially true if you care about performance (which you generally always do after some level). So even if you have a nice abstraction of your query layer, the kinds of questions you can ask the database efficiently depend heavily on the underlying storage mechanism, and so your application logic has to be built so it only asks the right kinds of questions. Even abstracting away the differences between, say, Oracle and MS SQLServer and MySQL is difficult enough, because of the different capabilities and performance characteristics. When you do that, you basically end up coding to the lowest common denominator, which can often limit what your applications does. Trying to come up with an abstraction that can encapsulate the difference between a row- versus column-oriented database, or between a relational database and some other kind of storage, is pretty much a losing proposition: they're too fundamentally different in terms of what kind of data you can store, how you can store it, how you can query it, what kinds of transactional guarantees you get, and what operations are fast and which ones are slow. So you really do kind of have to take your best shot at it, choose an approach, and if you choose wrong and have to change, it's just going to hurt. A lot. Good encapsulation and abstraction will ease some of the pain, but it's more like drinking whiskey before your leg gets sawed off than it is like general anesthesia.

tybris 17y ago

Well, you're right in the sense that Newton was right about gravity. It works perfectly down here on earth, but breaks down out in the universe. In a distributed system you'll want to take into account caching, replication, atomicity, security, etc. In other words, that's either going to be a hell of a complex API or an API that gives you too little control to build a high-performance system. There's no telling what works best, every computer system is a set of trade-offs and it always depends on the application.