4 ms·

He's right, in some ways. Long term, SSDs are going to have a massive impact on DBMS technologies. Disk latency is the dominant performance inhibitor for a lo

He's right, in some ways. Long term, SSDs are going to have a massive impact on DBMS technologies. Disk latency is the dominant performance inhibitor for a lot of DBMS applications. The area I'm working in, RDF triple stores, involves large joins and relatively small amounts of total data transfer, resulting in a lot of disk seeks per unit data transferred. In this area, SSDs will be a huge leap from disk techs. He's missed the point by targeting column stores, however. Column stores are primarily aimed at data warehouses, which (generally) have the following characteristics: - Extremely large amounts of data stored. - Queries tend to involve very large, unpredictable scans (aggregations, etc). - Since queries are relatively unpredictable, there's a limit to the effectiveness of caching. While SSDs can have blistering transfer rates and are very suitable for large scans, they're still vastly more expensive per gig than conventional disks. The kinds of queries being done are good for disks: they involve long continuous reads, so disk's primary disadvantage of slow seeks is not so relevant here. Column stores are aimed at this market by aiding faster effective transfer of data, since you're only reading data from the columns you're interested in off the disk. There's other claimed advantages like better compression ratios, too. Data warehouses are one of the primary growth areas for the DBMS market at the moment. There's a bunch of small players, who all have a chance to grow because the major heavyweights like Oracle and MS don't have overly strong products in the field - although that'll no doubt change with MS's acquisition of DATAllegro, one of the aforementioned small players. I'm sure Stonebraker and co will be keeping an eye on SSDs, but they're safe for some time to come yet. edit: formatting

jsrn 18y ago
He's missed the point by targeting column stores, however. Column stores are primarily aimed at data warehouses, which (generally) have the following characteristics: [...] He is not missing the point. Quote from the article: "My conclusion is, on the whole, no. The column store, when used to support existing petabyte OLAP systems may be worth the grief, but for transactional systems, at which the TRM is aiming and from which column stores would extract, not so much." OLAP = Online Analytical Processing = Data Warehousing, i.e. he primarily questions the use of column stores for transaction processing (OLTP) and recognizes the usefulness of column stores for large data warehouses.
- AlisdairO 18y ago
  Then he's criticising the applicability of column stores to a mode of use for which they're not designed. While there have been suggestions that correctly implemented column stores might provide acceptable OLTP performance, I don't see many people advocating them as a replacement for Oracle, DB2, or MSSQL. That seems to me to be missing the point, rather. "My interest is this: given the Next Revolution, do either a TRM or column store database have a purpose?" His post minimises the significance of data warehousing when it's a significant growth area, particularly for new players in the market. Column stores are applicable to DWs much smaller than petabyte scale - and indeed, I haven't seen any figures that suggest that Vertica has been scaled up that far yet. He's also ignored the impact of physical storage layer design on memory organisation and CPU cache performance. The use of SSDs is not a panacea that renders physical storage layout irrelevant. edit: quotes, and the realisation that 'minimalises' is not, in fact, a word.
  - jsrn 18y ago
    Then he's criticising the applicability of column stores to a mode of use for which they're not designed. ah, ok. That makes sense, agreed. Here is question to you - not directly concerning the article - If we talk about all those new (or not so new) hash databases (a.k.a. "key/value stores" like Amazon Simple db, Google's equivalent which they make available with Google App Engine, Berkeley DB etc.) and if we talk about OLTP: I have been thinking lateley that those stores that often require the programmer to denormalize and trade relational features for performance could soon become largely irrelevant SSDs, and soon (OLTP databases are often not that big in my experience [compared to OLAP databases], many should fit into an SSD today). In contrast to column stores those key/value stores are explicitely marketed for transaction processing (if a RDBMS doesn't scale enough). As I understand it, most of the time performance and scaling problems with relational databases result from the database being disk bound - a problem that should largely vanish with SSDs. Do you agree?
    - frig 18y ago
      Yeah, that guy's writing is muddled and obscure, but I think your points are closer to (what I think is) his intended point: Column-oriented dbs are essentially a "heroic engineering" way to optimize a datastore for a particular workload -- continuous-read-heavy-batch-processing -- by engineering around the performance peculiarities of traditional hard drives. Clearly this is a sensible strategy for optimizing a datastore for certain workloads: at the moment that strategy delivers material performance improvements in the scenarios it's designed for (material enough that if the performance of your system on those workloads is economically important to you, it's worth the cost to build or buy a system that'd significantly speed things up). What I think he's saying is that the rise of ssd may make this engineering effort essentially useless outside of a handful of niches (essentially, the niches reduce to: data volumes too large to economically fit into ssd within the foreseeable future): - in a storage medium with heavy seek times, the engineering effort to implement a column store (instead of just using an off-the-shelf rdbms) can pay off in a somewhat broad range of usages - in ssd-ish storage media, the engineering effort isn't going to be worth the benefit, most of the time, compared to just using a stock rdbms The mention of the TRM fits into this picture of his intended claim. If you're not familiar the TRM is a mystery shrouded in an enigma: a bunch of big, credible names in the database world claimed have invented a radical new way of implementing the backend of a relational database that would've offered radically better performance characteristics (essentially it made joins so unbelievably 'cheap' that it was no longer necessary to denormalize for performance; supposedly the more-normalized you went the better TRM would perform). The issue with the TRM (transrelational model) is that: - the supposed core concept is patented, but doesn't explain en toto how it'd work (for obvious reason) - there's a ton of secrecy and ndas and so on surrounding anyone and everyone who got a good glimpse of the full picture -- the core inventors seem extremely protective of their ip, to the point pretty much nothing material has leaked about it's supposed workings - the company that was supposedly doing the first commercial implementation folded, ostensibly for non-technical reasons but again it's so secretive no one really knows what happened So it's a big mystery. There's basically a couple schools of thought on it: - it does actually work, but a comedy of errors / business climate / personality conflicts / whatever have prevented it from either being commercially implemented or from having a fuller picture of its workings disclosed publicly. Things have been quiet since then due to the protectiveness and penchant for secrecy on part of the principals. - it looked good on paper, but in doing the actual implementation some unavoidable complication turned up that prevented it from obtaining the needed performance (either at the time -- 2005ish -- or forever). The big names associated with it have kept quiet since then partly out of embarrassment (publicly endorsing a flop, kinda like hawking endorsing a free energy machine) and partly again out of concern for the principals' protectiveness - it was some kind of hypey thing that blew up in their faces; essentially a belief they could attract funding and customers by virtue of their reputations and claims of a revolutionary approach, combined with a belief that they could do an awesome-enough imlpementation of a non-revolutionary datastore approach -- basically do a best-practices, clean-room build of the current state of the art -- that no one would be the wiser I tend to think the second option is the likeliest story. All of that is a long windup for a very quick pitch: Assuming the TRM wasn't just bunk it would then be one of two things: - heroic engineering to bring revolutionary performance gains to systems built around disk-based storage - some kind of heroic datastore engineering that'd work better with a more ram-like storage system (eg ssds) That's a bit of a non-answer answer, but the connection to his main line of reasoning is something like: - if it's the former, then it's another example of heroic engineering made irrelevant by ssds, as even a 'traditional' rdbms can be made similarly performant on an ssd without all that effort; this is my take on the author's opinion - if it's the latter, then maybe there's something to be gleaned from it; I don't think this is what the author thinks So, yeah: from reading the rest of his blog (posts are wordy, but there's not that many of them) I think he's got the following idea: - implementing a traditional rdbms is hard, but mainly b/c of all the work you have to do to make it not perform like a dog under - ssds radically shake up the assumed performance contours of your persistent storage, enough so that a lot of the specifics designing an rdbms for performance might change - additionally, this guy has the impression this implementation might be "easy"; that is, a lot less work to do compared to writing a traditional rdbms from scratch There you have it, I think.
ableal 18y ago
In a previous post, the author suggested that SSDs might make effective storing fewer bytes in more tables: http://drcoddwasright.blogspot.com/2009/03/part-2-was-to-be-study-in-my-building.html http://drcoddwasright.blogspot.com/2009/03/part-2-was-to-be-...: "So, if we need 1,000 rows from the 0NF table Foo, we could get them as 10 rows from table Able and 100 rows from table Baker, where Able and Baker are a full decomposition of Foo. It is the full decomposition of tables that make SSD databases useful."
- AlisdairO 18y ago
  Absolutely - SSDs make irrelevant a lot of the advantages that a denormalised storage model provides: denormalisation means fewer seeks but more data stored/read, which is often a good tradeoff for disks, but not for SSDs. It should be noted that column-oriented architectures offer the ability to compress away a lot of the excess information (they're very friendly to RLE-based compression, for example, and vertica integrates awareness of the characteristics of that compression into the query optimiser). Fundamentally, though, in a data warehouse you're still going to be talking about pretty vast quantities of data, and disks are going to hold an advantage in price/performance for much longer than they will in the OLTP sector. I don't mean to suggest that the author has no idea what he's talking about, more that he's set up a bit of a straw man.

jsrn 18y ago

He's missed the point by targeting column stores, however. Column stores are primarily aimed at data warehouses, which (generally) have the following characteristics: [...] He is not missing the point. Quote from the article: "My conclusion is, on the whole, no. The column store, when used to support existing petabyte OLAP systems may be worth the grief, but for transactional systems, at which the TRM is aiming and from which column stores would extract, not so much." OLAP = Online Analytical Processing = Data Warehousing, i.e. he primarily questions the use of column stores for transaction processing (OLTP) and recognizes the usefulness of column stores for large data warehouses.

AlisdairO 18y ago
Then he's criticising the applicability of column stores to a mode of use for which they're not designed. While there have been suggestions that correctly implemented column stores might provide acceptable OLTP performance, I don't see many people advocating them as a replacement for Oracle, DB2, or MSSQL. That seems to me to be missing the point, rather. "My interest is this: given the Next Revolution, do either a TRM or column store database have a purpose?" His post minimises the significance of data warehousing when it's a significant growth area, particularly for new players in the market. Column stores are applicable to DWs much smaller than petabyte scale - and indeed, I haven't seen any figures that suggest that Vertica has been scaled up that far yet. He's also ignored the impact of physical storage layer design on memory organisation and CPU cache performance. The use of SSDs is not a panacea that renders physical storage layout irrelevant. edit: quotes, and the realisation that 'minimalises' is not, in fact, a word.
- jsrn 18y ago
  Then he's criticising the applicability of column stores to a mode of use for which they're not designed. ah, ok. That makes sense, agreed. Here is question to you - not directly concerning the article - If we talk about all those new (or not so new) hash databases (a.k.a. "key/value stores" like Amazon Simple db, Google's equivalent which they make available with Google App Engine, Berkeley DB etc.) and if we talk about OLTP: I have been thinking lateley that those stores that often require the programmer to denormalize and trade relational features for performance could soon become largely irrelevant SSDs, and soon (OLTP databases are often not that big in my experience [compared to OLAP databases], many should fit into an SSD today). In contrast to column stores those key/value stores are explicitely marketed for transaction processing (if a RDBMS doesn't scale enough). As I understand it, most of the time performance and scaling problems with relational databases result from the database being disk bound - a problem that should largely vanish with SSDs. Do you agree?
  - frig 18y ago
    Yeah, that guy's writing is muddled and obscure, but I think your points are closer to (what I think is) his intended point: Column-oriented dbs are essentially a "heroic engineering" way to optimize a datastore for a particular workload -- continuous-read-heavy-batch-processing -- by engineering around the performance peculiarities of traditional hard drives. Clearly this is a sensible strategy for optimizing a datastore for certain workloads: at the moment that strategy delivers material performance improvements in the scenarios it's designed for (material enough that if the performance of your system on those workloads is economically important to you, it's worth the cost to build or buy a system that'd significantly speed things up). What I think he's saying is that the rise of ssd may make this engineering effort essentially useless outside of a handful of niches (essentially, the niches reduce to: data volumes too large to economically fit into ssd within the foreseeable future): - in a storage medium with heavy seek times, the engineering effort to implement a column store (instead of just using an off-the-shelf rdbms) can pay off in a somewhat broad range of usages - in ssd-ish storage media, the engineering effort isn't going to be worth the benefit, most of the time, compared to just using a stock rdbms The mention of the TRM fits into this picture of his intended claim. If you're not familiar the TRM is a mystery shrouded in an enigma: a bunch of big, credible names in the database world claimed have invented a radical new way of implementing the backend of a relational database that would've offered radically better performance characteristics (essentially it made joins so unbelievably 'cheap' that it was no longer necessary to denormalize for performance; supposedly the more-normalized you went the better TRM would perform). The issue with the TRM (transrelational model) is that: - the supposed core concept is patented, but doesn't explain en toto how it'd work (for obvious reason) - there's a ton of secrecy and ndas and so on surrounding anyone and everyone who got a good glimpse of the full picture -- the core inventors seem extremely protective of their ip, to the point pretty much nothing material has leaked about it's supposed workings - the company that was supposedly doing the first commercial implementation folded, ostensibly for non-technical reasons but again it's so secretive no one really knows what happened So it's a big mystery. There's basically a couple schools of thought on it: - it does actually work, but a comedy of errors / business climate / personality conflicts / whatever have prevented it from either being commercially implemented or from having a fuller picture of its workings disclosed publicly. Things have been quiet since then due to the protectiveness and penchant for secrecy on part of the principals. - it looked good on paper, but in doing the actual implementation some unavoidable complication turned up that prevented it from obtaining the needed performance (either at the time -- 2005ish -- or forever). The big names associated with it have kept quiet since then partly out of embarrassment (publicly endorsing a flop, kinda like hawking endorsing a free energy machine) and partly again out of concern for the principals' protectiveness - it was some kind of hypey thing that blew up in their faces; essentially a belief they could attract funding and customers by virtue of their reputations and claims of a revolutionary approach, combined with a belief that they could do an awesome-enough imlpementation of a non-revolutionary datastore approach -- basically do a best-practices, clean-room build of the current state of the art -- that no one would be the wiser I tend to think the second option is the likeliest story. All of that is a long windup for a very quick pitch: Assuming the TRM wasn't just bunk it would then be one of two things: - heroic engineering to bring revolutionary performance gains to systems built around disk-based storage - some kind of heroic datastore engineering that'd work better with a more ram-like storage system (eg ssds) That's a bit of a non-answer answer, but the connection to his main line of reasoning is something like: - if it's the former, then it's another example of heroic engineering made irrelevant by ssds, as even a 'traditional' rdbms can be made similarly performant on an ssd without all that effort; this is my take on the author's opinion - if it's the latter, then maybe there's something to be gleaned from it; I don't think this is what the author thinks So, yeah: from reading the rest of his blog (posts are wordy, but there's not that many of them) I think he's got the following idea: - implementing a traditional rdbms is hard, but mainly b/c of all the work you have to do to make it not perform like a dog under - ssds radically shake up the assumed performance contours of your persistent storage, enough so that a lot of the specifics designing an rdbms for performance might change - additionally, this guy has the impression this implementation might be "easy"; that is, a lot less work to do compared to writing a traditional rdbms from scratch There you have it, I think.

ableal 18y ago

In a previous post, the author suggested that SSDs might make effective storing fewer bytes in more tables: http://drcoddwasright.blogspot.com/2009/03/part-2-was-to-be-study-in-my-building.html http://drcoddwasright.blogspot.com/2009/03/part-2-was-to-be-...: "So, if we need 1,000 rows from the 0NF table Foo, we could get them as 10 rows from table Able and 100 rows from table Baker, where Able and Baker are a full decomposition of Foo. It is the full decomposition of tables that make SSD databases useful."

AlisdairO 18y ago
Absolutely - SSDs make irrelevant a lot of the advantages that a denormalised storage model provides: denormalisation means fewer seeks but more data stored/read, which is often a good tradeoff for disks, but not for SSDs. It should be noted that column-oriented architectures offer the ability to compress away a lot of the excess information (they're very friendly to RLE-based compression, for example, and vertica integrates awareness of the characteristics of that compression into the query optimiser). Fundamentally, though, in a data warehouse you're still going to be talking about pretty vast quantities of data, and disks are going to hold an advantage in price/performance for much longer than they will in the OLTP sector. I don't mean to suggest that the author has no idea what he's talking about, more that he's set up a bit of a straw man.