4 ms·

Many people ARE using petabyte scale databases. They don't start out that way, but they will be if they're successful, and if they're not they probably shouldn

by jjirsa 9y ago

Many people ARE using petabyte scale databases. They don't start out that way, but they will be if they're successful, and if they're not they probably shouldn't be running their own databases anyway (go RDS it up or something).

deleted 9y ago
[deleted]
eberkund 9y ago
Unless the amount of data you are collecting is increasing at a vastly unsustainable rate, you will be making megabucks and be able to afford a team of engineers to undertake a major redesign for version 2. But picking technologies based on scale at the onset is going to make getting to that point all the more unlikely.
happymellon 9y ago
What? No they aren't, I work with satellite imagery processing which is quite large in it raw data format, and after a decade we are not dealing with petabytes of active data, hundreds of gigs for a full earth coverage. Before that I have held positions in finance, dealing with realtime transaction processing. We did not work in petabytes. If you are working in petabytes you are storing crap in your production database, and 99% of that data is wasted.
- GrayShade 9y ago
  > What? No they aren't, I work with satellite imagery processing which is quite large in it raw data format, and after a decade we are not dealing with petabytes of active data, hundreds of gigs for a full earth coverage. This get large rather fast when processing time series from multiple sensors. But I agree with you otherwise, and it's not like you're storing satellite products in a database. Now I'm curious what NSA is using :).
  - ddorian43 9y ago
    https://accumulo.apache.org/ https://accumulo.apache.org/ See (History): https://en.wikipedia.org/wiki/Apache_Accumulo https://en.wikipedia.org/wiki/Apache_Accumulo
    - GrayShade 9y ago
      Thanks, I wasn't aware of it.
  - EvilGrin 9y ago
    You could just check their Github? :) https://github.com/nationalsecurityagency https://github.com/nationalsecurityagency
- altstar 9y ago
  Yeah right. We are using nosql in manufacturing. One second data for thousands of instruments create a lot of data very quickly. And that is per plant. Having a lot of plants at various sites in the world adds up.
  - jjirsa 9y ago
    Exactly this. Whenever you collect millions of things, even at low resolution, it adds up quickly Collecting high resolution and filtering down to low resolution for lower importance is still difficult and can require petabytes for real world use cases

by jjirsa 9y ago

deleted 9y ago

[deleted]

eberkund 9y ago

Unless the amount of data you are collecting is increasing at a vastly unsustainable rate, you will be making megabucks and be able to afford a team of engineers to undertake a major redesign for version 2. But picking technologies based on scale at the onset is going to make getting to that point all the more unlikely.

happymellon 9y ago

What? No they aren't, I work with satellite imagery processing which is quite large in it raw data format, and after a decade we are not dealing with petabytes of active data, hundreds of gigs for a full earth coverage. Before that I have held positions in finance, dealing with realtime transaction processing. We did not work in petabytes. If you are working in petabytes you are storing crap in your production database, and 99% of that data is wasted.

GrayShade 9y ago
> What? No they aren't, I work with satellite imagery processing which is quite large in it raw data format, and after a decade we are not dealing with petabytes of active data, hundreds of gigs for a full earth coverage. This get large rather fast when processing time series from multiple sensors. But I agree with you otherwise, and it's not like you're storing satellite products in a database. Now I'm curious what NSA is using :).
- ddorian43 9y ago
  https://accumulo.apache.org/ https://accumulo.apache.org/ See (History): https://en.wikipedia.org/wiki/Apache_Accumulo https://en.wikipedia.org/wiki/Apache_Accumulo
  - GrayShade 9y ago
    Thanks, I wasn't aware of it.
- EvilGrin 9y ago
  You could just check their Github? :) https://github.com/nationalsecurityagency https://github.com/nationalsecurityagency
altstar 9y ago
Yeah right. We are using nosql in manufacturing. One second data for thousands of instruments create a lot of data very quickly. And that is per plant. Having a lot of plants at various sites in the world adds up.
- jjirsa 9y ago
  Exactly this. Whenever you collect millions of things, even at low resolution, it adds up quickly Collecting high resolution and filtering down to low resolution for lower importance is still difficult and can require petabytes for real world use cases