4 ms·

> spatial data at rates of petabytes per day I'm having trouble thinking up examples - can you provide some?

by rcpt 12y ago

jandrewrogers 12y ago
These tend to be the aggregate flow of pervasive sensor networks i.e. millions of machines generating this data autonomously. Collected personal location data alone is 6-10 PB/day globally, most of which is currently discarded because few people can handle it. Some are just incredibly high bandwidth platforms, which are more common in industrial and sensor settings. There are more than a couple operational environments today generating billions of records every second that must be stored (not summarized). Some of the sexier petabyte-per-day sources include mobile phone telemetry, vehicle telemetry, and countless remote-sensing networks and platforms (of which there are far more than most people imagine). A petabyte per day is not that much data per entity, there are a lot of entities, and you can support this data rate in a single rack of commodity hardware. A more typical generic IoT application at a large company is 1-100 TB/day, but that is quickly creeping upward as it becomes more cost effective to deploy sensors and store/analyze the data. And more often than not, they need to store weeks or months of that data so the overall data set size is still quite large.
- x0x0 12y ago
  How many companies have million sized sensor networks? I guess few. And even at eg 400k records per second with an average record size of 1k, you'll generate ~ 30 TB per day. I just can't imagine there are many situations with more data incoming. Can you share company names or more concrete examples? Even ups only delivers 17M packages / day. edit: also, which industries customarily have thousand vertex polygons? Thanks for the interesting blog post.
  - pas 12y ago
    I've a not really relevant answer, but Netflix has more than 1.2 billion time series ( http://techblog.netflix.com/2014/12/introducing-atlas-netflixs-primary.html http://techblog.netflix.com/2014/12/introducing-atlas-netfli... ), which they update about every 5 sec. Though I don't know how much they store for how long. The interesting time series are heavy machinery telemetry feeds (for example a Rolls Royce jet engine has its own communications subsystem to phone home to a RR engineering station via satellite, which however, likely makes it low-bandwidth).
- pjc50 12y ago
  Collected personal location data alone is 6-10 PB/day globally, most of which is currently discarded because few people can handle it Knowing where everyone is at all times; the ultimate enabler for weaponised harrasment.
spitfire 12y ago
Ship or aircraft tracking data. Or if your the NSA, tracking mobile phones.
rch 12y ago
Think about the scale of agriculture in e.g. India, with network-capable farming equipment. http://www.trimble.com/agriculture/ http://www.trimble.com/agriculture/

> spatial data at rates of petabytes per day I'm having trouble thinking up examples - can you provide some?

by rcpt 12y ago

> spatial data at rates of petabytes per day I'm having trouble thinking up examples - can you provide some?

These tend to be the aggregate flow of pervasive sensor networks i.e. millions of machines generating this data autonomously. Collected personal location data alone is 6-10 PB/day globally, most of which is currently discarded because few people can handle it. Some are just incredibly high bandwidth platforms, which are more common in industrial and sensor settings. There are more than a couple operational environments today generating billions of records every second that must be stored (not summarized). Some of the sexier petabyte-per-day sources include mobile phone telemetry, vehicle telemetry, and countless remote-sensing networks and platforms (of which there are far more than most people imagine). A petabyte per day is not that much data per entity, there are a lot of entities, and you can support this data rate in a single rack of commodity hardware. A more typical generic IoT application at a large company is 1-100 TB/day, but that is quickly creeping upward as it becomes more cost effective to deploy sensors and store/analyze the data. And more often than not, they need to store weeks or months of that data so the overall data set size is still quite large.

x0x0 12y ago
How many companies have million sized sensor networks? I guess few. And even at eg 400k records per second with an average record size of 1k, you'll generate ~ 30 TB per day. I just can't imagine there are many situations with more data incoming. Can you share company names or more concrete examples? Even ups only delivers 17M packages / day. edit: also, which industries customarily have thousand vertex polygons? Thanks for the interesting blog post.
- pas 12y ago
  I've a not really relevant answer, but Netflix has more than 1.2 billion time series ( http://techblog.netflix.com/2014/12/introducing-atlas-netflixs-primary.html http://techblog.netflix.com/2014/12/introducing-atlas-netfli... ), which they update about every 5 sec. Though I don't know how much they store for how long. The interesting time series are heavy machinery telemetry feeds (for example a Rolls Royce jet engine has its own communications subsystem to phone home to a RR engineering station via satellite, which however, likely makes it low-bandwidth).
pjc50 12y ago
Collected personal location data alone is 6-10 PB/day globally, most of which is currently discarded because few people can handle it Knowing where everyone is at all times; the ultimate enabler for weaponised harrasment.

spitfire 12y ago

Ship or aircraft tracking data. Or if your the NSA, tracking mobile phones.

rch 12y ago

Think about the scale of agriculture in e.g. India, with network-capable farming equipment. http://www.trimble.com/agriculture/ http://www.trimble.com/agriculture/