6 ms·

This has the potential of really disrupting the enterprise data warehouse sector. All the MPP vendors today (HP Vertica, EMC Greenplum, Teradata) have exhorbita

This has the potential of really disrupting the enterprise data warehouse sector. All the MPP vendors today (HP Vertica, EMC Greenplum, Teradata) have exhorbitant pricing and ridiculous licensing. With their pricing - 1000 $ per TB per year, I would be really worried if I were Teradata (Not so much if I were IBM).

huggyface 14y ago
This is essentially ParAccel as a service. So it is one of those data warehouse vendors. As to the pricing, as always it isn't so straightforward -- it's $999 per TB a year if you pay $3,000 upfront, and then another $1000 for a single, very weak database server.
jasondc 14y ago
A lot of large enterprises won't be comfortable hosting their data outside of their own data centers. The killer application is making a portable, on premises version of this functionality without the high price.
- monstrado 14y ago
  Cloudera is doing just that with the recent announcement / open sourcing of Impala. Based on Amazon's description of their hosted product, the technology is very similar. Impala is still in beta, and columnar storage (trevni/avro) is right around the corner...with that, you can do petabyte scale queries for a very low cost. https://github.com/cloudera/impala https://github.com/cloudera/impala
  - flanger 14y ago
    Platfora is doing some interesting work with interactive, in-memory BI for Hadoop. They essentially do away with the traditional DW/ETL model and create ephemeral in-memory 'lenses' for querying and visualization.
  - capkutay 14y ago
    Impala is married to Hadoop. What if your data infrastructure isn't built on hbase and too complex/large to integrate it easily? Would impala still serve that purpose?
    - TallGuyShort 14y ago
      If your data is too difficult to integrate into HDFS (doesn't have to be HBase) using existing Hadoop tools, I suspect you're going to have to do some work to use that data on any platform.
    - monstrado 14y ago
      Impala doesn't require HBase to operate, it can use raw HDFS. Simple example, if you had a few terabytes of TSV files, you could easily copy the raw data into HDFS and then create a simple schema around it. All queries on this data would be in parallel across all the nodes in the cluster, this is partly due to the distributed nature of HDFS.
- redidas 14y ago
  Another barrier will be the integration with their current "business intelligence" solutions. Microstrategy and Jaspersoft support is a good start, but what about Microsoft, Oracle and SAP offerings? Switching to Amazon would involve rewriting your ETL process, and retooling your reporting software, and converting all your existing, currently-used reports. A huge expense in data warehousing projects isn't the hardware - it's the consultants, the time, and the people to support the thing. I'm sure this is a great solution for companies looking to start a data warehouse, or maybe companies looking to revamp their reporting environment completely... but other than that it'd be a hard sell...
  - facorreia 14y ago
    I bet they're getting ready for the upcoming "big data" explosion and not so much eyeing migration from already deployed data warehouses.
- optimusclimb 14y ago
  I've heard this argument time and time again in the context of various solutions/technologies. I still today feel that for 95% of the companies that "feel this way", it's simply the result of foolish paranoia among older, upper management. The type of thing that separates a "Fast Company" from stodgy and likely to be disrupted companies.
  - TallGuyShort 14y ago
    For some industries that's true - for others, being "Fast" isn't the priority.
  - scottmey 14y ago
    Totally agree, and they'll just be left in the dust while paying bucket loads for services that we can now acquire for greater deals less. At the end of the day, Amazon should have no problems finding clients interested in this service.
  - numbsafari 14y ago
    Some of those industries have legal and or regulatory reasons for not hosting all their data on AWS. In addition, AWS isn't always the economic boon it's made out to be. Those fees can really add up once you start moving enough data around. I'm a big fan of AWS. But, like any other tool, it's not meant for every job.
  - kyrra 14y ago
    what are the regulations on things like health records, personal information, etc... stuff that has tight restrictions on how the data is handled. Can these types of data be stored on Amazon or similar services and still be in compliance of data protection laws?
    - JunkDNA 14y ago
      For health records the regulations are a bit of a confusing mess when it comes to cloud storage. Basically, it boils down to "whatever your organization's legal team says". In theory, if data is encrypted in transit, encrypted at rest, and access is limited/logged, then it should meet US HIPAA requirements. However, that may not be enough to satisfy a particularly conservative legal department. There are also nuances about who holds the encryption keys, how are they managed, etc... Notably, Amazon won't actually stick their neck out and certify AWS as HIPAA compliant through a business associates agreement (interestingly Microsoft will for Azure: http://www.windowsazure.com/en-us/support/trust-center/compliance/ http://www.windowsazure.com/en-us/support/trust-center/compl...). I've been told by consultants that Amazon has so much business it's just not worth their time to bother with the headache of setting up such agreements.
- bbromhead 14y ago
  Most enterprises already outsource a lot of their IT, data warehousing especially whether its to a big player like IBM or boutique consultancies. For most applications does it really matter whether the data sits in your data center or Amazons? Nope... cause the organisation your company contracted to manage already has full access to all your secrets. So really Amazon is just another IT outsourcer except you don't need a long drawn out sales process.
- mitultiwari 14y ago
  There are lots of startups hosting their services on AWS. Their data is already on AWS. Amazon's Redshift makes perfect sense for them. Many startups like Qubole have been already working on providing such cloud based solutions for data analysis.
pinarsezer 14y ago
Rumor has it that they picked ParAccel to provide the underlying tech. I'm curious about how that compares to Vertica and Teradata.
- monstrado 14y ago
  Looks that way, https://twitter.com/ParAccel/status/273866697289650176 https://twitter.com/ParAccel/status/273866697289650176
- coldskull 14y ago
  thats correct. i have heard that from a first hand source
jl6 14y ago
Cost per TB for storage is a tiny proportion of the total costs of a data warehouse.

huggyface 14y ago

This is essentially ParAccel as a service. So it is one of those data warehouse vendors. As to the pricing, as always it isn't so straightforward -- it's $999 per TB a year if you pay $3,000 upfront, and then another $1000 for a single, very weak database server.

jasondc 14y ago

A lot of large enterprises won't be comfortable hosting their data outside of their own data centers. The killer application is making a portable, on premises version of this functionality without the high price.

monstrado 14y ago
Cloudera is doing just that with the recent announcement / open sourcing of Impala. Based on Amazon's description of their hosted product, the technology is very similar. Impala is still in beta, and columnar storage (trevni/avro) is right around the corner...with that, you can do petabyte scale queries for a very low cost. https://github.com/cloudera/impala https://github.com/cloudera/impala
- flanger 14y ago
  Platfora is doing some interesting work with interactive, in-memory BI for Hadoop. They essentially do away with the traditional DW/ETL model and create ephemeral in-memory 'lenses' for querying and visualization.
- capkutay 14y ago
  Impala is married to Hadoop. What if your data infrastructure isn't built on hbase and too complex/large to integrate it easily? Would impala still serve that purpose?
  - TallGuyShort 14y ago
    If your data is too difficult to integrate into HDFS (doesn't have to be HBase) using existing Hadoop tools, I suspect you're going to have to do some work to use that data on any platform.
  - monstrado 14y ago
    Impala doesn't require HBase to operate, it can use raw HDFS. Simple example, if you had a few terabytes of TSV files, you could easily copy the raw data into HDFS and then create a simple schema around it. All queries on this data would be in parallel across all the nodes in the cluster, this is partly due to the distributed nature of HDFS.
redidas 14y ago
Another barrier will be the integration with their current "business intelligence" solutions. Microstrategy and Jaspersoft support is a good start, but what about Microsoft, Oracle and SAP offerings? Switching to Amazon would involve rewriting your ETL process, and retooling your reporting software, and converting all your existing, currently-used reports. A huge expense in data warehousing projects isn't the hardware - it's the consultants, the time, and the people to support the thing. I'm sure this is a great solution for companies looking to start a data warehouse, or maybe companies looking to revamp their reporting environment completely... but other than that it'd be a hard sell...
- facorreia 14y ago
  I bet they're getting ready for the upcoming "big data" explosion and not so much eyeing migration from already deployed data warehouses.
optimusclimb 14y ago
I've heard this argument time and time again in the context of various solutions/technologies. I still today feel that for 95% of the companies that "feel this way", it's simply the result of foolish paranoia among older, upper management. The type of thing that separates a "Fast Company" from stodgy and likely to be disrupted companies.
- TallGuyShort 14y ago
  For some industries that's true - for others, being "Fast" isn't the priority.
- scottmey 14y ago
  Totally agree, and they'll just be left in the dust while paying bucket loads for services that we can now acquire for greater deals less. At the end of the day, Amazon should have no problems finding clients interested in this service.
- numbsafari 14y ago
  Some of those industries have legal and or regulatory reasons for not hosting all their data on AWS. In addition, AWS isn't always the economic boon it's made out to be. Those fees can really add up once you start moving enough data around. I'm a big fan of AWS. But, like any other tool, it's not meant for every job.
- kyrra 14y ago
  what are the regulations on things like health records, personal information, etc... stuff that has tight restrictions on how the data is handled. Can these types of data be stored on Amazon or similar services and still be in compliance of data protection laws?
  - JunkDNA 14y ago
    For health records the regulations are a bit of a confusing mess when it comes to cloud storage. Basically, it boils down to "whatever your organization's legal team says". In theory, if data is encrypted in transit, encrypted at rest, and access is limited/logged, then it should meet US HIPAA requirements. However, that may not be enough to satisfy a particularly conservative legal department. There are also nuances about who holds the encryption keys, how are they managed, etc... Notably, Amazon won't actually stick their neck out and certify AWS as HIPAA compliant through a business associates agreement (interestingly Microsoft will for Azure: http://www.windowsazure.com/en-us/support/trust-center/compliance/ http://www.windowsazure.com/en-us/support/trust-center/compl...). I've been told by consultants that Amazon has so much business it's just not worth their time to bother with the headache of setting up such agreements.
bbromhead 14y ago
Most enterprises already outsource a lot of their IT, data warehousing especially whether its to a big player like IBM or boutique consultancies. For most applications does it really matter whether the data sits in your data center or Amazons? Nope... cause the organisation your company contracted to manage already has full access to all your secrets. So really Amazon is just another IT outsourcer except you don't need a long drawn out sales process.
mitultiwari 14y ago
There are lots of startups hosting their services on AWS. Their data is already on AWS. Amazon's Redshift makes perfect sense for them. Many startups like Qubole have been already working on providing such cloud based solutions for data analysis.

pinarsezer 14y ago

Rumor has it that they picked ParAccel to provide the underlying tech. I'm curious about how that compares to Vertica and Teradata.

monstrado 14y ago
Looks that way, https://twitter.com/ParAccel/status/273866697289650176 https://twitter.com/ParAccel/status/273866697289650176
coldskull 14y ago
thats correct. i have heard that from a first hand source

jl6 14y ago

Cost per TB for storage is a tiny proportion of the total costs of a data warehouse.