7 ms·
The Australian Square Kilometre Array Pathfinder hits the big-data highway
- privong 10y agoPlease note, this is for a Square Kilometer Array pathfinder (the Australian Square Kilometre Array Pathfinder, ASKAP), not the Square Kilometer Array. Construction hasn't even begun on the latter, AFAIK.
- ismail 10y agoReferences: https://en.wikipedia.org/wiki/Square_Kilometre_Array https://en.wikipedia.org/wiki/Square_Kilometre_Array https://en.wikipedia.org/wiki/Australian_Square_Kilometre_Array_Pathfinder https://en.wikipedia.org/wiki/Australian_Square_Kilometre_Ar...
- deleted 10y ago[deleted]
- nrki 10y agoActually, it generates 5.2TB/sec. "Its antennas are now churning out 5.2 terabytes of data per second"
- jankey 10y agoEither case, I doubt it is "15% of the internets current data rate" as they claim. E.g. the newish submarine cable between the US and Japan has 60 Tb capacity. And that's just one cable.
- virtuallynathan 10y agoA large chunk of that capacity will be for private use, not for the capital-I Internet. 280Tbps is about right for interdomain Internet traffic, it lines up with Cisco's predictions/measurements: http://www.cisco.com/c/en/us/solutions/collateral/service-provider/visual-networking-index-vni/vni-hyperconnectivity-wp.html http://www.cisco.com/c/en/us/solutions/collateral/service-pr...
- wyager 10y ago> A large chunk of that capacity will be for private use, What private use requires that much bandwidth?
- discodave 10y agoInter region traffic of large cloud providers.
- sandworm101 10y agoId call the inner movements of data within google/facebook private networks part of the internet. When i do a google search, or watch a youtube vid, i know i am generating more traffic than the http requests and answer from the edge of google's network. The inner workings of the largest private networks often service the internet, if not by name, and therefore should be included in its total bandwidth. But good luck measuring it.
- billiam 10y agoNah, it is mostly traffic to manage the computing activity of large enterprises, not directly related to what we consider the Internet.
- discodave 10y agoWhile it may be that a good chunk of private traffic is in the service of internet services, I think calling it 'the internet' is wrong technically and in spirit. The spirit and technical definition is that fundamentally the internet is a connection between private networks. Initially it was some universities, ARPA and so on. Now it's Google, Facebook, Verizon, Amazon, AT&T and so on. What happens on these networks is private network traffic. If some of that traffic travels long distances it doesn't make it less private. Why count inter-region traffic of cloud providers, why not count the inter-AZ traffic of AWS regions? Why not count the inter-rack traffic and so on. The difference between the internet and private networks is not about distance travelled it's about ownership. For instance if traffic goes between Google and AWS at a peering location in CA, then that is internet traffic even if the data doesn't go very far.
- neolefty 10y agoTo clarify definitions, data rate will always be less than or equal to capacity. In practice, average rate will be much less than peak capacity.
- deleted 10y ago[deleted]
- Frqy3 10y agoAlso, that is the raw output from the antenna array. This is then analysed before storing, compressing it down to a much more managable few GB/s.
- plugger 10y agoIt's analysed and stored in Melbourne, Victoria. The SKA Pathfinder is in Western Australia outside of Geraldton. Hence the data transfer statistics as the two sites are roughly 4,000km from one another.
- yread 10y ago5.2TB/s not Tb/s. BIG difference!
- dunkelsten 10y agoFor Americans, that's 2.00773TB/s per Square Mile.
- sn6uv 10y agoI think you mean 13.468 TB/s per square mile.
- semi-extrinsic 10y agoAre those Metric terabytes (1000^4), or in freedom units (1024^4)?
- snerbles 10y agoShould be tebibytes (TiB) for the latter, though binary prefixes don't have much adoption outside of the software world. http://www.physics.nist.gov/cuu/Units/binary.html http://www.physics.nist.gov/cuu/Units/binary.html
- wlesieutre 10y agoOr inside of the software world, for that matter.
- semi-extrinsic 10y agoHuh? 1024^4 is how Windows defines a terabyte, accordingly that's what most people think a terabyte is. OS X was the first OS to switch to 1000^4 with Snow Leopard (2009). Different Linux system tools use either standard.
- snerbles 10y agoReferring the abbreviation of "TiB" versus "TB" for 1024^4.
- de_Selby 10y agoAnd 6.353 x 10^16 nibbles per furlong^2 per fortnight
- valarauca1 10y agoStilling waiting for when we put a radio telescope at L3 and L2 giving mankind a 14Tm baseline interferometry. The current 12Mm interferometry is nice, but earth is so small.
- privong 10y agoWe have something along those lines, albeit not quite as long-baseline. Spektr-R is a radio satellite on a elliptical orbit that gives baselines up to 350,000 km (0.35 Gm), linked to Earth-based facilities. https://en.wikipedia.org/wiki/Spektr-R https://en.wikipedia.org/wiki/Spektr-R
- zeristor 10y agoRadio scintillation of the Interstellar Medium becomes an issue, I daresay the radio equivalent of "active optics" could help. I did mention this at an outreach talk with Astroblack morphologies[0] and Tim O'Brien[1] who pointed out that ISM scintillation was on his slides I did think up 100s of radio telescopes in a bird cage orbit at the distance of Jupiter, impractical of course but Space VLBI does reward thinking big. Isn't it worth having telescopes well out the ecliptic plane, I seem to recollect VLBI is about filling in the UV plane. It was a long time ago, and it was just a second year physics project. [0] - http://www.artscatalyst.org/astro-black-morphologies-flow-motion http://www.artscatalyst.org/astro-black-morphologies-flow-mo... [1] - http://www.jb.man.ac.uk/~tob/ http://www.jb.man.ac.uk/~tob/
- dTal 10y agoNote that the actual collecting area will be 4072 square meters [1]. 12 out of 36 antennas are currently working. That's roughly 31 Kbps per square millimeter. Or 1 bit per second per 33 square microns. [1] http://www.atnf.csiro.au/projects/askap/specs.html http://www.atnf.csiro.au/projects/askap/specs.html
- rb2k_ 10y agoI wonder how much of that can be compressed for transfer and storage.
- ashleysmithgpu 10y agoThe csiro logo they have looks very familiar: https://regmedia.co.uk/2012/10/08/cisco.jpg https://regmedia.co.uk/2012/10/08/cisco.jpg https://www.csiro.au/~/media/Web-team/Images/CSIRO_Logo/logo.png https://www.csiro.au/~/media/Web-team/Images/CSIRO_Logo/logo...
- tangue 10y agoCisco went to court... and lost. Here's an article where I 've learned that there's a stylized version of the Golden Gate in Cisco's logo and that CSIRO's logo is the shape of Australia. http://www.itnews.com.au/news/csiro-beats-cisco-in-fight-over-logo-402571 http://www.itnews.com.au/news/csiro-beats-cisco-in-fight-ove...
- sandworm101 10y agoHonestly, i never linked the cisco logo to the goldengate. It looked like meters of an audio output or something else electronic, an o-scope. The proportions seem off, probably because i never see the bridge from the side. Im betting they want to play down the link to the city name. "Frisco" is a week trademark because of its common use. Cisco is akin to the "Syfy" channel. You might win on the letters, but not the spoken sounds ... As oppossed to the famous canadian case of "MikeRoeSoft.com" where sound trumped letters.
- TeMPOraL 10y agoYou win on letters, you lose on i18n. "Syfy" in Polish is plural of "syf", which means disorder, dirt, something ugly, or (probably also the origin of the word) syphilis. Obviously, this made telling some friends about some shows I watched rather awkward. ;).
- sandworm101 10y agoAnd that is why i read hn.
- nickspacek 10y ago
- sctb 10y agoWe've reverted the title from the submitted “New Australian Square Kilometer Array Generates 5.2Tb/s”. One of the reasons the guidelines ask us to prefer original titles is because it's surprisingly difficult to generate new ones that remain accurate.
- satysin 10y agoHow do you even begin to deal with that much data per second? What do they do with it? Storing more than a few minutes of data isn't practical is it? Do they process it in realtime?
- maxander 10y agoFrom the article: > Once out of the telescope, the data is going through a new, almost automatic data-processing system we’ve developed. Of course, they don't go into detail (instead making a very odd analogy to a bread machine), but I think the implication is that the data is processed immediately before being recorded. A similar approach is used at the LHC. A better analogy might be; when you take a picture on your phone, it comes out of the camera chip in a form equivalent to a bitmap file. But those are big and a pain to deal with, so its converted to jpeg or something else reasonable before being stored, perhaps losing a bit of irrelevant noise in the process.
- ChuckMcM 10y agoIn parallel :-) Divide that number by the number of telescopes in operation (in this case 12) and you're fetching 433 GB/sec on a telescope. Imagine that is I/Q data coming from the antenna. Now you're only looking for neutral hydrogen which has a much narrower spectrum, so you process that data with a band pass decimating filter, that takes you down to a few GB/sec. Then take the redshift into account and process everything that is in your 'target range' (a red shift of .26) plus or minus a bit and now your down to perhaps megabytes per second per telescope.
- autocorr 10y agoThe example of being able to reduce the data rate by only configuring the system to look in a narrow frequency range in the spectra is perceptive! Although unfortunately you cannot divide the data rate down by the number of antennas by processing each antenna in parallel. Interferometers such as this must correlate the signals between each unique pair of antennas and in approximate real time (1 sec to 1 min range). The data rate comes from sending the signal of each individual antenna by fiber or waveguide to a special FPGA/ASIC super computer called a correlator that computes the correlated signal between each unique pair of antennas. So it all goes into the hopper at once. Now what you're saying can actually be done, and is done on very long baseline facilities like the VLBA or Event Horizon Telescope (EHN). As an example, the EHN uses hydrogen atomic clocks to essentially time stamp the signal from a single antenna into a 50 TB disk pack, where each of the antennas are off by themselves from Hawaii, Spain, to the South Pole. They then bring all the disk packs to MIT where they then do the correlation from the time stamped data streams. This process is actually very complex however and it requires more total processing in the end. The higher computation comes from having to repeatedly process the data searching for subtle time offsets and antenna position offsets to find the correlation (this is how they use interferometers to measure those 1 cm/year continental drifts).
- NamTaf 10y agoFun fact: the CSIRO is the only Australian organisation to have a 2nd level domain name, lacking the .com and just using .au. This is because they were the first organisation to use a domain name in Australia and got in before the regulation specified it to be .{com|gov|edu|etc}.au
- zer0t3ch 10y agoThat is a surprisingly fun fact.
- nl 10y agoFor those interested in the architecture used to process this, take a look at http://www.slideshare.net/SparkSummit/spark-at-nasajplchris-mattmann http://www.slideshare.net/SparkSummit/spark-at-nasajplchris-..., especially slide 29