4 ms·

Meet the Firehose Seven Thousand Times Bigger Than Twitter's

47 points by coffee 16y ago

nostrademons 16y ago
1000 tweets/second seems quite low. IIRC, when I was still doing financial software (2007), the markets threw off about 50 GB/day, and a given trade was about 80 bytes. That's perhaps 600 million trades per day, or an order of magnitude higher than Twitter. And at the time, stock market volume had quadrupled over the last two years or so, so it's probably much higher than that. I wonder how much, say, FaceBook's newsfeed volume is. Or how much data Walmart or Fedex or Safeway process from their point-of-sale systems.
- mark_h 16y ago
  ...and even that is going to look insignificant when this comes online: http://www.skatelescope.org/ http://www.skatelescope.org/ Expected to produce around an exabyte a day.
- wallflower 16y ago
  > Or how much data Walmart or Fedex or Safeway process from their point-of-sale systems. Walmart is in its own class. There was a rumor a while back that they were trying to correlate their video feeds (of the POS/checkout) with their transaction database. That would mean matching video of customers with their transactions. All of them.
  - eru 16y ago
    What would you want to get out of that data? (I can see that it's useful, I just want to hear what you can imagine.)
    - wallflower 16y ago
      In general, an isolated video is not interesting. It becomes more interesting and potentially scary once you have a digital, searchable, analyzable history of video customer transactions. Reduce leakage/theft: If a cashier has been already flagged and the customer matches up in the network of cashier's friends (Facebook?), possibly in conjunction with another theft deterrent system, be able for managers to watch in real time a potential leakage event (where they don't scan certain items). Hyper-targeted marketing: (New) wedding ring detected. Commence deluge of in-kind marketing partnerships with Home Depot, maybe even Crate & Barrel. Customer over the past three months has been showing signs of possible pregnancy relative to their baseline body mass index. Somehow, non creepily, market to them via 3rd party mailing lists who had no idea how you learned she was expecting or more subtly by changing the default landing homepage of walmart.com to reflect more future mother when her cookie is detected. Kids. If the kids seem hyperactive in the overhead view, email coupons for toys that appeal to ADD-type kids. Over the last year of transactions, customer's head has been exhibiting signs of male pattern baldness. Send them targeted coupons for hats to see if they think its something they need to cover up. RFID: If RFID ever really enters the picture, via Walmart customer RFID cards - then you can get into real dystopian scenarios of following someone through the entire store.
deleted 16y ago
[deleted]

47 points by coffee 16y ago

1000 tweets/second seems quite low. IIRC, when I was still doing financial software (2007), the markets threw off about 50 GB/day, and a given trade was about 80 bytes. That's perhaps 600 million trades per day, or an order of magnitude higher than Twitter. And at the time, stock market volume had quadrupled over the last two years or so, so it's probably much higher than that. I wonder how much, say, FaceBook's newsfeed volume is. Or how much data Walmart or Fedex or Safeway process from their point-of-sale systems.

mark_h 16y ago
...and even that is going to look insignificant when this comes online: http://www.skatelescope.org/ http://www.skatelescope.org/ Expected to produce around an exabyte a day.
wallflower 16y ago
> Or how much data Walmart or Fedex or Safeway process from their point-of-sale systems. Walmart is in its own class. There was a rumor a while back that they were trying to correlate their video feeds (of the POS/checkout) with their transaction database. That would mean matching video of customers with their transactions. All of them.
- eru 16y ago
  What would you want to get out of that data? (I can see that it's useful, I just want to hear what you can imagine.)
  - wallflower 16y ago
    In general, an isolated video is not interesting. It becomes more interesting and potentially scary once you have a digital, searchable, analyzable history of video customer transactions. Reduce leakage/theft: If a cashier has been already flagged and the customer matches up in the network of cashier's friends (Facebook?), possibly in conjunction with another theft deterrent system, be able for managers to watch in real time a potential leakage event (where they don't scan certain items). Hyper-targeted marketing: (New) wedding ring detected. Commence deluge of in-kind marketing partnerships with Home Depot, maybe even Crate & Barrel. Customer over the past three months has been showing signs of possible pregnancy relative to their baseline body mass index. Somehow, non creepily, market to them via 3rd party mailing lists who had no idea how you learned she was expecting or more subtly by changing the default landing homepage of walmart.com to reflect more future mother when her cookie is detected. Kids. If the kids seem hyperactive in the overhead view, email coupons for toys that appeal to ADD-type kids. Over the last year of transactions, customer's head has been exhibiting signs of male pattern baldness. Send them targeted coupons for hats to see if they think its something they need to cover up. RFID: If RFID ever really enters the picture, via Walmart customer RFID cards - then you can get into real dystopian scenarios of following someone through the entire store.

deleted 16y ago

[deleted]