5 ms·

Sounded crackpot-ish to me. Anyone know wtf he was talking about?

by fizx 18y ago

briansmith 18y ago
It sounds reasonable to me. Basically he's saying that effective caching is usually better than organizing data in columns instead of rows. Then he said that organizing data in columns instead of rows is not a new idea.
ableal 18y ago
Far from 'crackpot-ish'. The author belongs to the hardcore DB-side hacker party. He's putting forward the idea that SSDs (solid state disks), by upsetting the memory hierarchy applecart, may have deep effects on our work with storage. (apologies for picking a provocative quote for the title, but it seemed to me that the front page was lacking its recommended daily dose of hacking.)
- jrockway 18y ago
  This article was about religion, not hacking.
- frig 18y ago
  You seem informed. You know if there've been any peeps on transrelational stuff since 04-05?
- AlisdairO 18y ago
  He's right, in some ways. Long term, SSDs are going to have a massive impact on DBMS technologies. Disk latency is the dominant performance inhibitor for a lot of DBMS applications. The area I'm working in, RDF triple stores, involves large joins and relatively small amounts of total data transfer, resulting in a lot of disk seeks per unit data transferred. In this area, SSDs will be a huge leap from disk techs. He's missed the point by targeting column stores, however. Column stores are primarily aimed at data warehouses, which (generally) have the following characteristics: - Extremely large amounts of data stored. - Queries tend to involve very large, unpredictable scans (aggregations, etc). - Since queries are relatively unpredictable, there's a limit to the effectiveness of caching. While SSDs can have blistering transfer rates and are very suitable for large scans, they're still vastly more expensive per gig than conventional disks. The kinds of queries being done are good for disks: they involve long continuous reads, so disk's primary disadvantage of slow seeks is not so relevant here. Column stores are aimed at this market by aiding faster effective transfer of data, since you're only reading data from the columns you're interested in off the disk. There's other claimed advantages like better compression ratios, too. Data warehouses are one of the primary growth areas for the DBMS market at the moment. There's a bunch of small players, who all have a chance to grow because the major heavyweights like Oracle and MS don't have overly strong products in the field - although that'll no doubt change with MS's acquisition of DATAllegro, one of the aforementioned small players. I'm sure Stonebraker and co will be keeping an eye on SSDs, but they're safe for some time to come yet. edit: formatting
  - jsrn 18y ago
    He's missed the point by targeting column stores, however. Column stores are primarily aimed at data warehouses, which (generally) have the following characteristics: [...] He is not missing the point. Quote from the article: "My conclusion is, on the whole, no. The column store, when used to support existing petabyte OLAP systems may be worth the grief, but for transactional systems, at which the TRM is aiming and from which column stores would extract, not so much." OLAP = Online Analytical Processing = Data Warehousing, i.e. he primarily questions the use of column stores for transaction processing (OLTP) and recognizes the usefulness of column stores for large data warehouses.
    - AlisdairO 18y ago
      Then he's criticising the applicability of column stores to a mode of use for which they're not designed. While there have been suggestions that correctly implemented column stores might provide acceptable OLTP performance, I don't see many people advocating them as a replacement for Oracle, DB2, or MSSQL. That seems to me to be missing the point, rather. "My interest is this: given the Next Revolution, do either a TRM or column store database have a purpose?" His post minimises the significance of data warehousing when it's a significant growth area, particularly for new players in the market. Column stores are applicable to DWs much smaller than petabyte scale - and indeed, I haven't seen any figures that suggest that Vertica has been scaled up that far yet. He's also ignored the impact of physical storage layer design on memory organisation and CPU cache performance. The use of SSDs is not a panacea that renders physical storage layout irrelevant. edit: quotes, and the realisation that 'minimalises' is not, in fact, a word.
      - jsrn 18y ago
        Then he's criticising the applicability of column stores to a mode of use for which they're not designed. ah, ok. That makes sense, agreed. Here is question to you - not directly concerning the article - If we talk about all those new (or not so new) hash databases (a.k.a. "key/value stores" like Amazon Simple db, Google's equivalent which they make available with Google App Engine, Berkeley DB etc.) and if we talk about OLTP: I have been thinking lateley that those stores that often require the programmer to denormalize and trade relational features for performance could soon become largely irrelevant SSDs, and soon (OLTP databases are often not that big in my experience [compared to OLAP databases], many should fit into an SSD today). In contrast to column stores those key/value stores are explicitely marketed for transaction processing (if a RDBMS doesn't scale enough). As I understand it, most of the time performance and scaling problems with relational databases result from the database being disk bound - a problem that should largely vanish with SSDs. Do you agree?

Sounded crackpot-ish to me. Anyone know wtf he was talking about?

by fizx 18y ago

Sounded crackpot-ish to me. Anyone know wtf he was talking about?

It sounds reasonable to me. Basically he's saying that effective caching is usually better than organizing data in columns instead of rows. Then he said that organizing data in columns instead of rows is not a new idea.

ableal 18y ago

Far from 'crackpot-ish'. The author belongs to the hardcore DB-side hacker party. He's putting forward the idea that SSDs (solid state disks), by upsetting the memory hierarchy applecart, may have deep effects on our work with storage. (apologies for picking a provocative quote for the title, but it seemed to me that the front page was lacking its recommended daily dose of hacking.)

jrockway 18y ago
This article was about religion, not hacking.
frig 18y ago
You seem informed. You know if there've been any peeps on transrelational stuff since 04-05?
AlisdairO 18y ago
He's right, in some ways. Long term, SSDs are going to have a massive impact on DBMS technologies. Disk latency is the dominant performance inhibitor for a lot of DBMS applications. The area I'm working in, RDF triple stores, involves large joins and relatively small amounts of total data transfer, resulting in a lot of disk seeks per unit data transferred. In this area, SSDs will be a huge leap from disk techs. He's missed the point by targeting column stores, however. Column stores are primarily aimed at data warehouses, which (generally) have the following characteristics: - Extremely large amounts of data stored. - Queries tend to involve very large, unpredictable scans (aggregations, etc). - Since queries are relatively unpredictable, there's a limit to the effectiveness of caching. While SSDs can have blistering transfer rates and are very suitable for large scans, they're still vastly more expensive per gig than conventional disks. The kinds of queries being done are good for disks: they involve long continuous reads, so disk's primary disadvantage of slow seeks is not so relevant here. Column stores are aimed at this market by aiding faster effective transfer of data, since you're only reading data from the columns you're interested in off the disk. There's other claimed advantages like better compression ratios, too. Data warehouses are one of the primary growth areas for the DBMS market at the moment. There's a bunch of small players, who all have a chance to grow because the major heavyweights like Oracle and MS don't have overly strong products in the field - although that'll no doubt change with MS's acquisition of DATAllegro, one of the aforementioned small players. I'm sure Stonebraker and co will be keeping an eye on SSDs, but they're safe for some time to come yet. edit: formatting
- jsrn 18y ago
  He's missed the point by targeting column stores, however. Column stores are primarily aimed at data warehouses, which (generally) have the following characteristics: [...] He is not missing the point. Quote from the article: "My conclusion is, on the whole, no. The column store, when used to support existing petabyte OLAP systems may be worth the grief, but for transactional systems, at which the TRM is aiming and from which column stores would extract, not so much." OLAP = Online Analytical Processing = Data Warehousing, i.e. he primarily questions the use of column stores for transaction processing (OLTP) and recognizes the usefulness of column stores for large data warehouses.
  - AlisdairO 18y ago
    Then he's criticising the applicability of column stores to a mode of use for which they're not designed. While there have been suggestions that correctly implemented column stores might provide acceptable OLTP performance, I don't see many people advocating them as a replacement for Oracle, DB2, or MSSQL. That seems to me to be missing the point, rather. "My interest is this: given the Next Revolution, do either a TRM or column store database have a purpose?" His post minimises the significance of data warehousing when it's a significant growth area, particularly for new players in the market. Column stores are applicable to DWs much smaller than petabyte scale - and indeed, I haven't seen any figures that suggest that Vertica has been scaled up that far yet. He's also ignored the impact of physical storage layer design on memory organisation and CPU cache performance. The use of SSDs is not a panacea that renders physical storage layout irrelevant. edit: quotes, and the realisation that 'minimalises' is not, in fact, a word.
    - jsrn 18y ago
      Then he's criticising the applicability of column stores to a mode of use for which they're not designed. ah, ok. That makes sense, agreed. Here is question to you - not directly concerning the article - If we talk about all those new (or not so new) hash databases (a.k.a. "key/value stores" like Amazon Simple db, Google's equivalent which they make available with Google App Engine, Berkeley DB etc.) and if we talk about OLTP: I have been thinking lateley that those stores that often require the programmer to denormalize and trade relational features for performance could soon become largely irrelevant SSDs, and soon (OLTP databases are often not that big in my experience [compared to OLAP databases], many should fit into an SSD today). In contrast to column stores those key/value stores are explicitely marketed for transaction processing (if a RDBMS doesn't scale enough). As I understand it, most of the time performance and scaling problems with relational databases result from the database being disk bound - a problem that should largely vanish with SSDs. Do you agree?