3 ms·

I wrote about it here: https://aravindh.net/post/sysadmin/ https://aravindh.net/post/sysadmin/ > What form do these jobs have? Mostly batch jobs written as b

I wrote about it here: https://aravindh.net/post/sysadmin/ https://aravindh.net/post/sysadmin/ > What form do these jobs have? Mostly batch jobs written as bash scripts. Occasionally, some users run singularity containers. But, all through SLURM. > How do you manage workloads? SLURM > How do you manage resources? As a sysadmin, my inventory is via Ansible. All activity on servers happen via Ansible only. > Do users have quota for compute and storage? Yes, Users typically can run 72 cores, 512 GiB of mem at a time. Rest is queued until resources are released. Disk quota is only for home directories - 400GiB(enforced by ZFS refquota). > Do you use GPUs? If so, how do you deal with malfunction? No, weirdly our workloads(genetics and genomics) don’t fit the GPUs very well, as they are sparse matrix walks with wide precision floats. But, we plan to try for some other stuff soon. > Is this a distributed processing? No. Jobs run one node at a time. > Are the machines heterogeneous? Yes, inteL based servers all the way from Haswell to Cascade lake. > What do you use for that cluster? SLURM > What if a job requires dependencies? Taken care of by SLURM. > Do you create a compute environment on the fly or do all jobs have the same dependencies and these don't change much? I guess you mean the software libraries and tools that jobs use? If so, our central software repo is NFS mounted on all compute nodes. Users can install things they need if admin priv is not needed. If not either Singularity or email to me. > How do you do data governance? This is the painful and human oriented task. We lock down data transport to outside world, educate the users about data policies and then spend a lot of time looking at data flows with hope. > Is the data read only? Do you have an API to fetch the data from the job code? Sorry, I could not understand this question. Do you mean metadata about A job?

aspyct 6y ago
Great article, thanks for sharing :)
- anp 6y ago
  I used to work in an environment not dissimilar from what's described above. The uni has some hardware details here: https://in.nau.edu/hpc/details/ https://in.nau.edu/hpc/details/. Many of my answers would be similar, although the presence of physics, astronomy, and a few other areas motivates GPUs these days apparently. I was doing genomics workloads in the 900-1200gb RAM range with 30+ cores. There's a pretty interesting NSF-wide project for managing clusters in a more commoditized way as part of https://www.xsede.org https://www.xsede.org. You might describe it as a "private heterogeneous almost-cloud" in its goals? That might be saying a bit much. Density, latency, storage throughput, etc. were in favor of DIY (plus the pricey professionals to run it) rather than cloud offerings. When I was there (2016) I did some basic math for being able to use a cloud provider for some of our lighter workloads when the local cluster was loaded down. Astronomical without a contract, which is quite the thing to set up, etc. Worth noting that while they do alright, NAU (first link) is hardly a top-tier university with bleeding edge technical requirements.
siscia 6y ago
I work in a kinda similar environment even if bigger. To manage runtime dependencies we use cvmfs. A read only distributed filesystems, kinda like a very specific NFS for software distribution. If that is never been a problem, cvmfs it is quite stable and very used in our field.
joeraut 6y ago
Fascinating blog post of yours, thanks for sharing it!
mbreese 6y ago
Yeah, cloud dev-ops and HPC are so similar and still oh-so different (I run a smaller HPC cluster for genomics). When reading these questions, someone with an HPC background would probably already know the answers (or just accept SLURM as an answer for everything). With respect to the original questions -- I've investigated running a full SLURM cluster on the cloud and it just never seems worth the effort. For larger clusters than mine, maybe, but then when you start to hit the levels you're talking about, I just don't see the point in moving to the cloud. It would be hard to hit that sweet spot where the costs would make sense. Amazon even has a series on scaling a SLURM cluster with AWS EC2 provisioning, but it just seemed like more work than was justified for us.
- Jugurtha 6y ago
  See reply above[^0]. I'm interested in Jupyter cell level job granularity for "MLOps". [^0]: https://news.ycombinator.com/item?id=23129417 https://news.ycombinator.com/item?id=23129417
aliceryhl 6y ago
The article says you're in Aarhus, so I assume you work at Aarhus university? I'm a student from Denmark myself, and I'm curious, which departments mainly need the compute power?
Jugurtha 6y ago
Thank you for the reply and the post. We're running JupyterLab for some students, and when several of them are training models, they consume GPU memory. Now, we thought of several ways of solving this. We have a branch that uses Kubernetes but this is not the one that is deployed. We know about SLURM and this is something we will support for a coarse granularity. i.e: notebook level jobs. This is a common scenario and we'll need it for our AppBook (we allow users to turn a notebook into an application with one click and we automatically generate the form fields for the features, and the API endpoint for the model). However, I'm interested in finer granularity computation; as in: I want a job for the cell. This involves looking into Jupyter kernels to circumvent the front-end<-->kernel disconnection (we'll have to do that for another use case in low internet bandwidth scenario). >Sorry, I could not understand this question. Do you mean metadata about A job? How do your users upload, and manage data access and is the data mutable and versioned. What version of the code ran on which version of the data, and who changed that data, etc. How do users access the data from a Jupyter notebook? Is the data in different object storages? Do you proxy the requests and handle access, things like that.

aspyct 6y ago

Great article, thanks for sharing :)

anp 6y ago
I used to work in an environment not dissimilar from what's described above. The uni has some hardware details here: https://in.nau.edu/hpc/details/ https://in.nau.edu/hpc/details/. Many of my answers would be similar, although the presence of physics, astronomy, and a few other areas motivates GPUs these days apparently. I was doing genomics workloads in the 900-1200gb RAM range with 30+ cores. There's a pretty interesting NSF-wide project for managing clusters in a more commoditized way as part of https://www.xsede.org https://www.xsede.org. You might describe it as a "private heterogeneous almost-cloud" in its goals? That might be saying a bit much. Density, latency, storage throughput, etc. were in favor of DIY (plus the pricey professionals to run it) rather than cloud offerings. When I was there (2016) I did some basic math for being able to use a cloud provider for some of our lighter workloads when the local cluster was loaded down. Astronomical without a contract, which is quite the thing to set up, etc. Worth noting that while they do alright, NAU (first link) is hardly a top-tier university with bleeding edge technical requirements.

siscia 6y ago

I work in a kinda similar environment even if bigger. To manage runtime dependencies we use cvmfs. A read only distributed filesystems, kinda like a very specific NFS for software distribution. If that is never been a problem, cvmfs it is quite stable and very used in our field.

joeraut 6y ago

Fascinating blog post of yours, thanks for sharing it!

mbreese 6y ago

Yeah, cloud dev-ops and HPC are so similar and still oh-so different (I run a smaller HPC cluster for genomics). When reading these questions, someone with an HPC background would probably already know the answers (or just accept SLURM as an answer for everything). With respect to the original questions -- I've investigated running a full SLURM cluster on the cloud and it just never seems worth the effort. For larger clusters than mine, maybe, but then when you start to hit the levels you're talking about, I just don't see the point in moving to the cloud. It would be hard to hit that sweet spot where the costs would make sense. Amazon even has a series on scaling a SLURM cluster with AWS EC2 provisioning, but it just seemed like more work than was justified for us.

Jugurtha 6y ago
See reply above[^0]. I'm interested in Jupyter cell level job granularity for "MLOps". [^0]: https://news.ycombinator.com/item?id=23129417 https://news.ycombinator.com/item?id=23129417

aliceryhl 6y ago

The article says you're in Aarhus, so I assume you work at Aarhus university? I'm a student from Denmark myself, and I'm curious, which departments mainly need the compute power?

Jugurtha 6y ago

Thank you for the reply and the post. We're running JupyterLab for some students, and when several of them are training models, they consume GPU memory. Now, we thought of several ways of solving this. We have a branch that uses Kubernetes but this is not the one that is deployed. We know about SLURM and this is something we will support for a coarse granularity. i.e: notebook level jobs. This is a common scenario and we'll need it for our AppBook (we allow users to turn a notebook into an application with one click and we automatically generate the form fields for the features, and the API endpoint for the model). However, I'm interested in finer granularity computation; as in: I want a job for the cell. This involves looking into Jupyter kernels to circumvent the front-end<-->kernel disconnection (we'll have to do that for another use case in low internet bandwidth scenario). >Sorry, I could not understand this question. Do you mean metadata about A job? How do your users upload, and manage data access and is the data mutable and versioned. What version of the code ran on which version of the data, and who changed that data, etc. How do users access the data from a Jupyter notebook? Is the data in different object storages? Do you proxy the requests and handle access, things like that.