8 ms·
How CERN serves 1EB of data via FUSE [video]
- penguin_booze 2y agoDumb question: is it right to think that the the experiments' results are reproducible? If so, what's the value in keeping results from distant past, given the data is generated at this enormous rate?
- jonathan-re 2y agoWell generally yes, but that isn’t how it works there. Since the things they want to measure in their experiments are so atomically small, sensor noise becomes a huge problem. So it’s not enough to find the sensor readings for NewMysteriousParticleX to be sure that it actually exists, it could just have been noise. So you have to run the experiment again and again until your datapoint is statistically significant enough that you are sure, it wasn’t just noise. A couple of years ago there was this case where they almost found a new particle, the significance was pretty close to the threshold - the problem was that this particle was not expected and would have shaken the foundations of particle physics. Some weeks later the particle has vanished back into the abyss of noise.
- maybeben 2y agoi mean, they also have one of the largest ceph deployments. anything is scalable with no budget.
- pas 2y agoslide 22 states that the cost is 1 CHF/TB/month (on 10+2 erasure coded disks), though it would be interesting to do a breakdown of costs (development, hardware, maintenance, datacenter, servicing, management, etc..)
- pclmulqdq 2y ago1 CHF/TB/month is a bit expensive for storage at that scale, so it would definitely be interesting to see what they're spending the money on and what they are (and aren't) counting in that price.
- hackernewds 2y agoNo budget often tags along with no accountability
- rob_c 2y agoTape backup, accessibility, networking, availability... At 1CHF/TB that's a lot better than my local university still charging >100x that for such services internally
- pclmulqdq 2y agoEconomies of scale in storage are significant. Also, I don't know why you put up with your university charging 100x that when you can store things on AWS for $5-10/TB/month (or less). That comes with all the guarantees (or more) of durability and availability you get from the university.
- lnauta 2y agoDon't forget bandwidth costs at this scale are significant to say the least.
- pclmulqdq 2y agoI assume most of that exabyte is stored in 2-3 datacenters, in which case the bandwidth cost is actually relatively small. Downloading it would cost a fortune (or take an eternity), but if it stays in the datacenter (or stays in AWS), bandwidth is cheap.
- hi-v-rocknroll 2y agoThey probably consume Panasas, IBM, DDN, and BeeGFS gear and licensing too.
- rob_c 2y agoGood god no. Nowhere near anything so crass. CEPH and EOS all the way
- adev_ 2y agoNop. Most internal data is spread between Ceph and home-made distributed storage system named EOS (https://indico.cern.ch/event/138478/contributions/149912/attachments/116849/165874/EOS-ALICE.pdf https://indico.cern.ch/event/138478/contributions/149912/att...) running over commodity hardware. The only commerical-backed storage system is the long term storage tape system. Still it has an home-made overlay API over it to interface with the rest of the systems.
- dark-star 2y agoAll their important "administrative" stuff (think Active Directory/LDAP user database, mail and other essential services) run on proprietary storage systems from a commercial vendor (not IBM though), with enterprise support and all. At least that was the case a few years ago when we last talked to one of the heads of IT at CERN, but I guess it hasn't drastically changed
- vfclists 2y ago[flagged]
- Enginerrrd 2y agoA lot of the benefit has come from learning expertise in applications. Tons of the data science tools have roots in CERN. Tons of interesting statistical methods, tons of experience R&D with superconductors and all manners of sensors. Tons of math/ computation techniques / modeling, etc would not be here without for CERN. It would be sort of silly to expect that any of their actual discoveries or tests of the SM would have any actual application, but the ancillary benefits are there.
- hackernewds 2y agoWhich tons? And why would it be silly? If actual new particles or physical phenomena were found the applications would be trillions
- adgjlsfhk1 2y agoa particle that requires 30 km particle accelerator to produce isn't going to have that many applications on earth
- coherentpony 2y agohttps://en.wikipedia.org/wiki/CERN#Scientific_achievements https://en.wikipedia.org/wiki/CERN#Scientific_achievements Here's a couple, in case you don't want to read the page: - CERN pioneered the introduction of TCP/IP for its intranet, beginning in 1984 - CERN has developed a number of policies and official documents that enable and promote open science - The CERN Science Gateway, opened in October 2023,[179] is CERN's latest facility for science outreach and education I purposefully picked items that weren't directly particle physics related.
- scottapotamas 2y agoJust tacking some detail onto "promote open science". CERN was/is a large early user and supporter of the open source KiCAD electronics CAD tooling. The downstream impact of improved accessibility to solid ECAD tooling has been a large contributing factor to the growing ecosystem of open electronics. A lot of really impressive test and measurement equipment to support their research is developed in the open (see https://ohwr.org/project https://ohwr.org/project). People on HN are probably most likely to have heard of the White Rabbit timing project, but there's fantastic voltmeter designs, a lot of FPGA projects for carriers, gateware, fun RF designs.
- renewiltord 2y agoMost of the magic is in https://eos-web.web.cern.ch/eos-web/ https://eos-web.web.cern.ch/eos-web/ apparently
- hackernewds 2y ago[flagged]
- synicalx 2y ago1EB with only 30k users, thats a wild TB-per-user ratio. My frame of reference; the largest storage platform I've ever worked on was a combined ~60PB (give or take) and that had hundreds of millions of users.
- hackernewds 2y agoThat's the scale of the universe, compared to data generated by humans
- shric 2y agoMy frame of reference; the largest storage platform I've ever worked on was a combined ~tens of EB (give or take) and that had over a billion users.
- chipdart 2y agoMost humans don't handle sensor and simulation data for a living, though. CERN just so happens to employ thousands who do that for a living.
- lmihaig 2y agoWhen experiments are running the sensors generate about 1PB of data per second. They have to do multiple (I think four?) layers of filtering, including hardware level to get to actual manageable numbers.
- elashri 2y agoIt depends on which experiment. We call it trigger system. And it varies according to each experiment requirements and physics of interest. For example LHCb is doing now full trigger system on a software side (No hardware FPGA triggering) and mainly utilizing GPUs for that. That would be hare to achieve with the harsher conditions and requirements of CMS and ATLAS. But yes at LHCb we discard about 97% of the data generated during collisions. Disclaimer: I work on LHCb trigger system
- a-dub 2y agodoes modern fuse still context switch too much or does it now use io_uring or similar?
- Dwedit 2y agoLast I read about FUSE, adding a 128KB read-ahead buffer drastically reduced context switching.
- mappu 2y agoFUSE over io_uring is still WIP: https://lwn.net/Articles/988186/ https://lwn.net/Articles/988186/ FUSE Passthrough landed in kernel 6.9, which also reduces context switching in some cases: https://www.phoronix.com/news/Linux-6.9-FUSE-Passthrough https://www.phoronix.com/news/Linux-6.9-FUSE-Passthrough . The benchmarks in this article are pretty damning for regular FUSE.
- a-dub 2y agoyeah but still not great for metadata operations, no? i remember it was really not great for large sets of search paths because it defeated the kernel's built-in metadata caches with excessive context switching?
- Dwedit 2y agoFUSE Passthrough is only useful for filesystems that wrap an existing filesystem, such as union mounts. Otherwise, you don't have an open file to hand over.
- qwertox 2y agoIIRC I had issues with inotify when I was editing files on a remote machine via SSHFS, when these files were being used inside a Docker container. inotify inside the container did not trigger the notifications, whereas it did, when editing a file with an editor directly on that host. I think this was related to FUSE, that Docker just didn't get notified.
- z3t4 2y agoThe inotify signals might work if you add -v a whole directory
- udev4096 2y agoThis is fascinating. How are they managing or even taking backup for this gigantic storage?
- ephimetheus 2y agoFor experiment data, there is a layer on top of all of this that distributes datasets across the computing grid. That system has a way to handle replicate at the dataset level.
- rob_c 2y agoTape and off-site replicas at globally distributed data centres for science. Of the 1EB a huge amount of that is probably in automated recall and replication with "users" running staged processing of the data at different sites ultimately with data being reduced to "manageable" GB-TB level for scientists to do science
- fnands 2y agoYup, lots of tape for stuff in cold storage, and then some subset of that on disk spread out over several sites. It's kinda interesting to watch anything by Alberto Pace, the head of storage at CERN to get an understanding of the challenges and constraints: https://www.youtube.com/watch?v=ym2am-FumXQ https://www.youtube.com/watch?v=ym2am-FumXQ I was basically on the helpdesk for the system for a few years so had to spend a fair amount of time helping people replicate data from one place to another, or from tape onto disk.
- JBorrow 2y agoThey use a distributed data management tool called RUCIO: https://rucio.cern.ch https://rucio.cern.ch to distribute data on the grid.
- tantalor 2y agoWhat is Rucio? Rucio enables centralized management of large volumes of data backed by many heterogeneous storage backends. Data is physically distributed over a large number of storage servers, potentially each relying on different storage technologies (SSD/Disk/Tape/Object storage) and, frequently, managed by different teams of system administrators. Rucio builds on top of this heterogeneous infrastructure and provides an interface which allows users to interact with the storage backends in a unified way. The smallest operational unit in Rucio is a file. Rucio enables users to upload, download, and declaratively manage groups of such files. https://rucio.cern.ch/documentation/started/what_is_rucio https://rucio.cern.ch/documentation/started/what_is_rucio
- niemandhier 2y agoPeople here keep claiming “Anything is possible with unlimited budget”. Cerns budget is 1.4 billion Euro, 50 million Euro for all IT infrastructure. https://cds.cern.ch/record/2888205/files/English.pdf#page18 https://cds.cern.ch/record/2888205/files/English.pdf#page18 It’s not the money, it’s the people. Update: Added source.
- lokimedes 2y agoAlso, the in-kind contributions from hundreds of institutes around the world. Much can, and has, been said about physicist code, but CERN is the center of a massive community of “pre-dropout” geniuses. I can’t count the number of former students that later joined Google and the likes. Many are frequenting HN.
- adev_ 2y agoCERN was a good example of how much can be done with how little when you have the right people. For a long time, the entire Linux distribution (Scientific Linux) used for ~15K collaborators, the infra and the grid computing was managed by a team of around 4-5 people. The teams managing the network access (LanDB), the distributed computing system, the scientific framework (ROOT) and the storage are also small, dedicated skilled teams. And the result speaks for itself. Unfortunately, most of that went to shit quite recently when they replaced the previous head of IT by a Microsoft fanboy/girl coming from outside of the scientific environment. The first thing he/she did was to force Microsoft bloatware everywhere to replace existing working OSS solutions.
- wuming2 2y ago> Unfortunately, most of that went to shit quite recently when they replaced the previous head of IT by a Microsoft fanboy(girl?) coming from outside of the scientific environment. Painful to read so I did a short check. From a news post I don’t want to link here, but easily found searching “CERN, the famous scientific lab where the web was born, tells us why it's ditching Microsoft and helping others do the same”, direction taken in 2019 seemed quite the opposite. I am not sure how current head of IT at CERN, Enrica Porcari, fits in to the story. Insider info will be appreciated.
- InDubioProRubio 2y agoThe things you can build when everyone is a rockstar :D
- jgalt212 2y agoI'm convinced CERN could greatly benefit from "middle out".
- vfclists 2y agoRe: https://news.ycombinator.com/item?id=41716523 https://news.ycombinator.com/item?id=41716523, > over the years what discoveries have been made at CERN that have had practical social and economic benefits to humanity as a whole? Some responders to the question believe I was criticizing a supposed wastefulness of the research. Not knowing the benefits of the discoveries in high energy physics, ie the stuff the accelerators are actually built to discover, doesn't mean I was criticizing it. Responses referenced the contributions the development of the infrastructure supporting the basic research itself have made, which is fine, but not the benefits of high energy physics discoveries. So to rephrase the question - What are the practical social and economic benefits to society that the discoveries in high-energy particle physics at institutions like CERN have made over the years? This is not just in relation to CERN, but world wide, such those experiments which create pools of water deep underground to study cosmic rays etc.
- deelowe 2y agoYou're probably getting replies like that because it's a bit of an odd question. Academic research isn't really done to achieve a particular purpose or goal. The piratical benefit literally is academic.
- dylan604 2y agoIt's also one of the first questions from people that very much are criticizing, so even if it was an sincere question it will be lumped together. Not recognizing/addressing this when posing the question does nothing to prevent it from the lumping.
- superhuzza 2y agoThe piratical benefit may be particle cannons? Yarrgh!
- coherentpony 2y ago> Responses referenced the contributions the development of the infrastructure supporting the basic research itself have made, which is fine, but not the benefits of high energy physics discoveries. I was one of those responders. There were two very deliberate reasons I specifically avoided talking about particle physics: 1. I interpreted the tone of the original question to be extremely highly cynical of any scientific contribution particle physics has made, so I instead went for 'consequential' things. More excitement around education, outreach, and other adjacent aspects that are beneficial to humans. I did this to potentially avoid, "How has discovering a new Boson made my rent cheaper?" types of arguments that are only ever made in bad faith, but have been made to me a disheartening number of times in my career; and 2. I am scientist and I have collaborators and colleagues at CERN, but I'm not a particle physicist and so I didn't feel adequately qualified to highlight them. I was expecting someone with more expertise would jump in and simply do a better job than I ever could. If I interpreted the tone of your question incorrectly, please understand that it wasn't an intentional sleight on you, and simply an artefact of a) plain text being an incredibly poor medium for communicating nuance; b) a defensive measure that I have had the displeasure of dealing with in the past. And if you were genuinely curious, that's wonderful, and I'm sorry that I didn't offer you more grace in my response.
- julienchastang 2y agoSomewhat off topic, but CERN has a fantastic science museum attached to it that I had the privilege of visiting last summer. There is of course Tim Berners-Lee's NeXT workstation, but also so much more. It is also the only science museum I've visited that addresses topics in cyberinfrastructure such as serving out massive amounts of data. (I personally get interested when I see old StorageTek tapes lol.) The more traditional science displays are also great. Check it out if you are ever in the Geneva area. It is an easy bus ride to get out there.
- zrules 2y agoDon’t forget to visit the gift shop too. They don’t have an online store so it’s the only place to get CERN ‘gears’. You can easily overspend there for gifts your friends and family will appreciate (if they know and like about CERN and its missions).
- QuietWatchtower 2y agoWhat's funny is that I just visited the museum a few months ago, and am coincidentally wearing a CERN hat I got there while reading the post and comments. I also highly recommend checking out the museum!
- deleted 2y ago[deleted]
- olantwin 2y agoThere are also free tours basically every day, without pre-booking. The itineraries vary, but usually one of the old accelerators (synchrocyclotron) and the ATLAS visitor centre are shown.