3 ms·

Absolutely, the vast majority (95%+) of logs are never read by a human. Therefore, processing it is enormously wasteful. A good architecture will write once and

Absolutely, the vast majority (95%+) of logs are never read by a human. Therefore, processing it is enormously wasteful. A good architecture will write once and not touch anything until it is needed. I spent years working on system handling 50+PB/day of logs. No database or ELK can handle that, and even if it did it would be prohibitively expensive.

Keyframe 6y ago
Where did you work? CERN?
- jeffbee 6y ago
  It's adorable when people think scientific computing has the same scale as a Google or Microsoft.
  - FridgeSeal 6y ago
    Sorry what was that? https://www.spie.org/news/photonics-focus/mayjun-2020/square-kilometer-array-big-data?SSO=1 https://www.spie.org/news/photonics-focus/mayjun-2020/square...
    - jeffbee 6y ago
      Ignoring the fantasy b.s. in the second half of the article, the stuff at the top is exactly what I mean. A mighty 400 GB/s: i.e. much less than the > 50 PB/day of logs the other person mentioned; 1600 hours of SD video per second: i.e. about 1-2 million concurrent HD streams, or much less than the amount actually served by YouTube. IBM Summit "world's most powerful supercomputer": < 5000 nodes, i.e. much below the median cell size described in the 2015 Borg paper. Summit is a respectable computer but it would get lost in a corner of a FAANG datacenter.
      - lukeschlather 6y ago
        CERN is a correct example. The LHC reportedly generates 1PB per second: https://home.cern/news/news/computing/cern-data-centre-passes-200-petabyte-milestone https://home.cern/news/news/computing/cern-data-centre-passe...
        jeffbee 6y ago
        If you define “generates” to mean “discards” then yes.
        welterde 6y ago
        It still gets processed though and only all non-interesting events get discarded.. Otherwise the tape alone to store it on would exceed their total operating budget in a day, so they have to be a bit clever about it.
        tutfbhuf 6y ago
        I think that even Google can not save 1PB/s in 2020.
      - pstrateman 6y ago
        400GB/s is about 35 PB/day not quite as big a difference
      - tedisaacs 6y ago
        it’s adorable how proud you are to have worked at FAANG and how angry you get at the idea some other organisation handles equivalent scale
        andialo 6y ago
        touche
      - welterde 6y ago
        The numbers are not fantasy at all - this will be a huge radio telescope - one square kilometer of pure collecting area and thousands of receiving antennas (For reference: Arecibo has around 0.073 km^2). We are talking data input to the correlator on the terabit/s scale. And technology-demonstration with ASKAP are well under way. ALMA is working quite well by now as well (> 600 Gb/s with just an 50 antenna array).

Keyframe 6y ago

Where did you work? CERN?

jeffbee 6y ago
It's adorable when people think scientific computing has the same scale as a Google or Microsoft.
- FridgeSeal 6y ago
  Sorry what was that? https://www.spie.org/news/photonics-focus/mayjun-2020/square-kilometer-array-big-data?SSO=1 https://www.spie.org/news/photonics-focus/mayjun-2020/square...
  - jeffbee 6y ago
    Ignoring the fantasy b.s. in the second half of the article, the stuff at the top is exactly what I mean. A mighty 400 GB/s: i.e. much less than the > 50 PB/day of logs the other person mentioned; 1600 hours of SD video per second: i.e. about 1-2 million concurrent HD streams, or much less than the amount actually served by YouTube. IBM Summit "world's most powerful supercomputer": < 5000 nodes, i.e. much below the median cell size described in the 2015 Borg paper. Summit is a respectable computer but it would get lost in a corner of a FAANG datacenter.
    - lukeschlather 6y ago
      CERN is a correct example. The LHC reportedly generates 1PB per second: https://home.cern/news/news/computing/cern-data-centre-passes-200-petabyte-milestone https://home.cern/news/news/computing/cern-data-centre-passe...
      - jeffbee 6y ago
        If you define “generates” to mean “discards” then yes.
        welterde 6y ago
        It still gets processed though and only all non-interesting events get discarded.. Otherwise the tape alone to store it on would exceed their total operating budget in a day, so they have to be a bit clever about it.
        tutfbhuf 6y ago
        I think that even Google can not save 1PB/s in 2020.
    - pstrateman 6y ago
      400GB/s is about 35 PB/day not quite as big a difference
    - tedisaacs 6y ago
      it’s adorable how proud you are to have worked at FAANG and how angry you get at the idea some other organisation handles equivalent scale
      - andialo 6y ago
        touche
    - welterde 6y ago
      The numbers are not fantasy at all - this will be a huge radio telescope - one square kilometer of pure collecting area and thousands of receiving antennas (For reference: Arecibo has around 0.073 km^2). We are talking data input to the correlator on the terabit/s scale. And technology-demonstration with ASKAP are well under way. ALMA is working quite well by now as well (> 600 Gb/s with just an 50 antenna array).