10 ms·

Having provisioned these sorts of "big data" systems in the past, it's now about how much you have today, it's how much you'll have over a growth period. The ad

Having provisioned these sorts of "big data" systems in the past, it's now about how much you have today, it's how much you'll have over a growth period. The advantages of a scale-out system like cassandra, riak (RIP), memsql, big-table, cockroach, etc., are that they can grow with you from 3TB, to 9TB, to 81TB (as an example, if you're on some exponential growth curve with your data). It's not that you can't do it with Postgres and MySQL via sharding, it's that you don't have to if your access patterns match one of the models that one of the above databases fits well within. So, that person above may not need it today, but they may need it sooner rather than later.

gshulegaard 9y ago
This is a really insightful comment. It's not that you can't get Postgres to handle any volume...it's how much work is required on your part.
noncoml 9y ago
Designing for the future is a guaranteed project failure. If one is starting a new project, hence contemplating what DB to use, starting with an ACID db is a safe bet in most cases(unless of course they are already starting with a huge amount of data). By the time the outgrow the ACID database they will have a better idea of what exactly they need and more importantly they will have the resources to make the switch. BTW: riak is hopefully not dead yet, let's see what bet365 can do with it..
- Thaxll 9y ago
  No it's not, say that have 2TB of logs daily, good luck designing that with a regular RDMS without sharding / clustering / re-balancing capabilities.
  - noncoml 9y ago
    In which case they would not be designing for the future. Did you miss this: "(unless of course they are already starting with a huge amount of data)"?
  - jacques_chester 9y ago
    I expect either of Citus or Greenplum can deal with that kind of volume. Citus acts as an extension, Greenplum is descended from an earlier fork (now merging its back to mainline as part of overall development). Citus specifically cite analytics and write throughput as their strengths. I believe them. They do seem to charge for shard rebalancing and some other features. Greenplum comes with an inbuilt data science toolkit (MADlib) and has a long history of dealing with intensive workloads. I don't know if there is a commercial version with extra features. I don't think so, but I haven't looked closely. Greenplum also gave birth to HAWQ, which uses the Greenplum distributed/parallel query planner to plan queries over Hadoop. Disclosure: I work for Pivotal, we do most Greenplum development.
    - user5994461 9y ago
      And citus will be a nightmare to setup and maintain, for what elasticsearch could have done easier and better out of the box. Use the right tool for the job.
      - jacques_chester 9y ago
        I was pointing out that 2TB/day is not a very high write volume for modern OSS RDBMSes, especially distributed ones. As for elasticsearch, I have found it to be temperamental. The "out of the box" experience was usually unhappy. For logging I've seen people move to other tools (splunk, stackdriver) more than once.
        user5994461 9y ago
        I have found elasticsearch to be one the easiest database I ever had to setup and maintain. Some tuning to do when you have TB on multiple boxes but nothing hardcore. 2 TB/day is fairly easy... as long as you have a bit of experience AND one of the database that can scale horizontally AND the company buys the amount of hardware that is required. StackDriver/Sumologic/logentries are cloud solutions, so obviously one doesn't have to deal with the database ever.
- wjossey 9y ago
  I don't think we disagree, but I want to be cautious about saying "whatever we start with is always good enough." I'm co-founder of a company, doing all of the technical work, and I too am using a very vanilla stack: Postgres, Elasticsearch, Redis, Memcache, Rails. However, having worked on massive scale-out projects before, I recognize where my pain points are going to be with my tech choices. [1] I'm storing calendar data in postgres. This data is primarily key/value based, needs to be persistent, and on a per user basis potentially is tens of thousands of records. This is a medium term problem, and I'll likely move this data out of postgres and into a scale out solution sooner rather than later. I need relatively fast lookup times, but this is primarily a "high write, low read" scenario, that Cassandra is well suited for. [2] I'm storing a bunch of text from blog posts, news articles, etc., in Postgres. I really dont' need the data to be in postgres, it's just a handy place to store it to synchronize with Algolia & Elasticsearch, where the true work happens. However, I dont' want to hold onto TBs & TBs of news articles in a postgres system. As that system scales out, it'll be unnecessarily expensive, as I don't need all the bells and whistles I'm getting from Postgres, for what is effectively a form of cold storage. This is a long term problem, and I likely won't tackle it for well over a year. So, you're right, I'm not contemplating what DB to use, but I'm definitely aware of why I won't just leveraging the same database in perpetuity. Being focused on delivering customer value is vital, and that's my intent; however, scalability and cost structure do matter over the long term, so an understanding of where one will go is good as well.
  - noncoml 9y ago
    Agreed
  - simonw 9y ago
    Have you considered Citus as your scale-out solution? https://www.citusdata.com https://www.citusdata.com - maybe you can have your PostgreSQL and eat it too.
    - wjossey 9y ago
      Citus seems really cool (this is not a critique of the product), but probably not something I’d use. They seem to be very close in feature set to bigquery, and if I have to choose between the two bigquery wins for me. There are definitely situations where citus is a better solution, but I think the scenarios where I could need a scale out analytics platform, I’d choose BQ. Still, very supportive of what they are doing at Citus. It’s great.
- threeseed 9y ago
  > Designing for the future is a guaranteed project failure. Based on what ? There is no evidence or logic behind this. If you know that you are producing say 1GB a day of data that you must store then it is sensible and prudent to select a technology that can support your needs a year or two from now. And this idea that you always have resources to make a switch later on is rarely the case. The more common scenario is that a team is established to implement something, it moves into production and then the team either disbands or moves onto something else. Why do you think so many people try and plan for the future ? In most enterprise companies at least it would be considered a failure to have a project that in a year's time required another business case just to replace the database that the team should've appropriately selected in the first place.
  - noncoml 9y ago
    Knowing that you will hit the limit of the database in X years is different. Designing for the future is a horrible choice because where do you stop? Eg. Why are you sure that Cassandra, or whatever you choose, will be good enough for the future? Maybe you will need a Spanner type database or something even more big/consistent/feature-full.
    - deleted 9y ago
      [deleted]
- davedx 9y ago
  > Designing for the future is a guaranteed project failure. Yes, I understand this. I would not want to get bogged down on infrastructure when trying to deliver early versions of a product. At the same time, I think database choice is one of the more important tech stack decisions you should make early on. Cassandra also sounds like a good bet for the type of data I'm interested in storing (ML model inputs, so lots of key -> numerical value data). But my experience is mostly around RDBMS and some NoSQL. Hence the question. Thanks for the answer!
  - noncoml 9y ago
    > ML model inputs, so lots of key -> numerical value data How big are the numerical value data? AFAIK Cassandra does like large amount of data per value. https://docs.datastax.com/en/cql/3.3/cql/cql_reference/refLimits.html https://docs.datastax.com/en/cql/3.3/cql/cql_reference/refLi... Also what do you expect the total amount of data to be? BTW, I just recommended to start with PostgreSQL because it just works and will have your back no matter what ways you like to use retrieve you data. However if you are absolutely sure you don’t require secondary indexes, joins, etc, and just need a KV database, Cassandra will work fine too.

gshulegaard 9y ago

This is a really insightful comment. It's not that you can't get Postgres to handle any volume...it's how much work is required on your part.

noncoml 9y ago

Designing for the future is a guaranteed project failure. If one is starting a new project, hence contemplating what DB to use, starting with an ACID db is a safe bet in most cases(unless of course they are already starting with a huge amount of data). By the time the outgrow the ACID database they will have a better idea of what exactly they need and more importantly they will have the resources to make the switch. BTW: riak is hopefully not dead yet, let's see what bet365 can do with it..

Thaxll 9y ago
No it's not, say that have 2TB of logs daily, good luck designing that with a regular RDMS without sharding / clustering / re-balancing capabilities.
- noncoml 9y ago
  In which case they would not be designing for the future. Did you miss this: "(unless of course they are already starting with a huge amount of data)"?
- jacques_chester 9y ago
  I expect either of Citus or Greenplum can deal with that kind of volume. Citus acts as an extension, Greenplum is descended from an earlier fork (now merging its back to mainline as part of overall development). Citus specifically cite analytics and write throughput as their strengths. I believe them. They do seem to charge for shard rebalancing and some other features. Greenplum comes with an inbuilt data science toolkit (MADlib) and has a long history of dealing with intensive workloads. I don't know if there is a commercial version with extra features. I don't think so, but I haven't looked closely. Greenplum also gave birth to HAWQ, which uses the Greenplum distributed/parallel query planner to plan queries over Hadoop. Disclosure: I work for Pivotal, we do most Greenplum development.
  - user5994461 9y ago
    And citus will be a nightmare to setup and maintain, for what elasticsearch could have done easier and better out of the box. Use the right tool for the job.
    - jacques_chester 9y ago
      I was pointing out that 2TB/day is not a very high write volume for modern OSS RDBMSes, especially distributed ones. As for elasticsearch, I have found it to be temperamental. The "out of the box" experience was usually unhappy. For logging I've seen people move to other tools (splunk, stackdriver) more than once.
      - user5994461 9y ago
        I have found elasticsearch to be one the easiest database I ever had to setup and maintain. Some tuning to do when you have TB on multiple boxes but nothing hardcore. 2 TB/day is fairly easy... as long as you have a bit of experience AND one of the database that can scale horizontally AND the company buys the amount of hardware that is required. StackDriver/Sumologic/logentries are cloud solutions, so obviously one doesn't have to deal with the database ever.
wjossey 9y ago
I don't think we disagree, but I want to be cautious about saying "whatever we start with is always good enough." I'm co-founder of a company, doing all of the technical work, and I too am using a very vanilla stack: Postgres, Elasticsearch, Redis, Memcache, Rails. However, having worked on massive scale-out projects before, I recognize where my pain points are going to be with my tech choices. [1] I'm storing calendar data in postgres. This data is primarily key/value based, needs to be persistent, and on a per user basis potentially is tens of thousands of records. This is a medium term problem, and I'll likely move this data out of postgres and into a scale out solution sooner rather than later. I need relatively fast lookup times, but this is primarily a "high write, low read" scenario, that Cassandra is well suited for. [2] I'm storing a bunch of text from blog posts, news articles, etc., in Postgres. I really dont' need the data to be in postgres, it's just a handy place to store it to synchronize with Algolia & Elasticsearch, where the true work happens. However, I dont' want to hold onto TBs & TBs of news articles in a postgres system. As that system scales out, it'll be unnecessarily expensive, as I don't need all the bells and whistles I'm getting from Postgres, for what is effectively a form of cold storage. This is a long term problem, and I likely won't tackle it for well over a year. So, you're right, I'm not contemplating what DB to use, but I'm definitely aware of why I won't just leveraging the same database in perpetuity. Being focused on delivering customer value is vital, and that's my intent; however, scalability and cost structure do matter over the long term, so an understanding of where one will go is good as well.
- noncoml 9y ago
  Agreed
- simonw 9y ago
  Have you considered Citus as your scale-out solution? https://www.citusdata.com https://www.citusdata.com - maybe you can have your PostgreSQL and eat it too.
  - wjossey 9y ago
    Citus seems really cool (this is not a critique of the product), but probably not something I’d use. They seem to be very close in feature set to bigquery, and if I have to choose between the two bigquery wins for me. There are definitely situations where citus is a better solution, but I think the scenarios where I could need a scale out analytics platform, I’d choose BQ. Still, very supportive of what they are doing at Citus. It’s great.
threeseed 9y ago
> Designing for the future is a guaranteed project failure. Based on what ? There is no evidence or logic behind this. If you know that you are producing say 1GB a day of data that you must store then it is sensible and prudent to select a technology that can support your needs a year or two from now. And this idea that you always have resources to make a switch later on is rarely the case. The more common scenario is that a team is established to implement something, it moves into production and then the team either disbands or moves onto something else. Why do you think so many people try and plan for the future ? In most enterprise companies at least it would be considered a failure to have a project that in a year's time required another business case just to replace the database that the team should've appropriately selected in the first place.
- noncoml 9y ago
  Knowing that you will hit the limit of the database in X years is different. Designing for the future is a horrible choice because where do you stop? Eg. Why are you sure that Cassandra, or whatever you choose, will be good enough for the future? Maybe you will need a Spanner type database or something even more big/consistent/feature-full.
  - deleted 9y ago
    [deleted]
davedx 9y ago
> Designing for the future is a guaranteed project failure. Yes, I understand this. I would not want to get bogged down on infrastructure when trying to deliver early versions of a product. At the same time, I think database choice is one of the more important tech stack decisions you should make early on. Cassandra also sounds like a good bet for the type of data I'm interested in storing (ML model inputs, so lots of key -> numerical value data). But my experience is mostly around RDBMS and some NoSQL. Hence the question. Thanks for the answer!
- noncoml 9y ago
  > ML model inputs, so lots of key -> numerical value data How big are the numerical value data? AFAIK Cassandra does like large amount of data per value. https://docs.datastax.com/en/cql/3.3/cql/cql_reference/refLimits.html https://docs.datastax.com/en/cql/3.3/cql/cql_reference/refLi... Also what do you expect the total amount of data to be? BTW, I just recommended to start with PostgreSQL because it just works and will have your back no matter what ways you like to use retrieve you data. However if you are absolutely sure you don’t require secondary indexes, joins, etc, and just need a KV database, Cassandra will work fine too.