4 ms·
Strangely, or perhaps not, the most interesting part of this article to me was this: "Using the Digital Backend instrument on the GBRT, Dr. Gajjar and the List
by tgtweak 9y ago
Strangely, or perhaps not, the most interesting part of this article to me was this:
"Using the Digital Backend instrument on the GBRT, Dr. Gajjar and the Listen team observed FRB 121102 for five hours. From this, they accumulating 400 terabytes of data"
That has to be one hell of an all flash array to take in that much data at a steady state of 22.75 GiB/s
- theoh 9y agoPaper describing the instrument: "The Breakthrough Listen Search for Intelligent Life: A Wideband Data Recorder System for the Robert C. Byrd Green Bank Telescope" https://arxiv.org/abs/1707.06024 https://arxiv.org/abs/1707.06024
- rikkus 9y agoRelevant part: "Each storage server contains 36 hard drives, each with 6 TB capacity. These are configured as three 11-disk RAID 6 arrays plus 3 global hot swap spares."
- tgtweak 9y agoImpressive if you think in raid 6 you have double a single drive's write speed (not counting cache). They'd need 60+ of those arrays to ingest without back pressure.
- userbinator 9y agoA single spinning disk is already capable of >200MB/s sequential write: shttp://www.storagereview.com/wd_black_6tb_hdd_review http://www.storagereview.com/wd_black_6tb_hdd_review 100 of those in parallel will suffice to achieve >20GB/s, with roughly the same capacity of a few hundred TB.
- pbhjpbhj 9y agoIs use of 100 parallel disks at all standard? There has to be some overhead in processing, can you really just stripe across more disks and directly multiply the throughput? How much is that costing to run?
- sigi45 9y agoThey probably filter and compress it, put it through a fiber glas system to storage systems of universities and so on. Those store systems are huge and yeah are build to do exactly this.
- xenophonf 9y agoI've seen petabyte-scale SANs for bioinformatics HPCs. Other sciences (esp. the high energy physics community) have similarly scaled storage and compute requirements.
- sathackr 9y ago7200 disks in 1 "array" for 30PB of storage. This cluster is described as their "playground" https://cds.cern.ch/record/2015206/files/CephScaleTestMarch2015.pdf https://cds.cern.ch/record/2015206/files/CephScaleTestMarch2... https://indico.cern.ch/event/594977/contributions/2407197/attachments/1396464/2128985/RedHatDay-Ceph.pdf https://indico.cern.ch/event/594977/contributions/2407197/at...
- polemic 9y agoFun fact: the Square Kilometre Array (SKA telescope) will gather roughly 3 TB/s when it's up an running.
- ourmandave 9y agoWould it be a reverse DoS attack when the North Koreans try to steal the data and 3 TB/s crashes their Pentium II "mainframe"?
- draugadrotten 9y ago“There is no greater danger than underestimating your opponent.” ― Lao Tzu
- jordache 9y agowhat do they mean by accumulated data? Are they mapping the raw signal to some other format that necessitates this huge file size? Is it recording the signal at multiple wavelengths?