3 ms·
ETL and batch processing have little to do with each other. Batch processing is a term from HPC systems, which are typically multi-user, and have to share comp
by crabbone 4y ago
ETL and batch processing have little to do with each other.
Batch processing is a term from HPC systems, which are typically multi-user, and have to share computing resources by means of a workload manager (eg. Slurm, PBS, UGE etc.) Workload managers function by defining queues, queues have associated resources (eg. number of CPUs, particular network hardware, accelerators, s.a. GPUs and so on). Users interact with workload managers by scheduling execution of their programs on particular queues (because they want the resources associated with those queues).
The interaction between the user and the workload manager is described as "submission of a batch job", where this is contrasted with running a program interactively (s.t. the user can simply start the program and observe its behavior right away). I believe, these are the batch jobs the article is talking about. These kinds of jobs are very common in HPC, and I'd imagine in ML performed on HPC resources.
- prpl 4y agoBatch processing predates HPC. It goes back to mainframes - it’s just that HPC had a common lineage there. Like you said, it’s an unattended job that runs. That’s what it’s getting at- setting up routines for unattended execution in batch. BTW Chris Ré did stuff with UW Madison /HTCondor and definitely knows what batch computing is