3 ms·

ETL is a great use case for LLMs, and probably half of what they’re actually referring to with the vague phrase “batch processing” It makes a lot of sense to

ETL is a great use case for LLMs, and probably half of what they’re actually referring to with the vague phrase “batch processing” It makes a lot of sense to use an LLM to understand data then generate code to do mass extract, but the article is really oblique and it’s hard to tell what’s they’re doing

crabbone 4y ago
ETL and batch processing have little to do with each other. Batch processing is a term from HPC systems, which are typically multi-user, and have to share computing resources by means of a workload manager (eg. Slurm, PBS, UGE etc.) Workload managers function by defining queues, queues have associated resources (eg. number of CPUs, particular network hardware, accelerators, s.a. GPUs and so on). Users interact with workload managers by scheduling execution of their programs on particular queues (because they want the resources associated with those queues). The interaction between the user and the workload manager is described as "submission of a batch job", where this is contrasted with running a program interactively (s.t. the user can simply start the program and observe its behavior right away). I believe, these are the batch jobs the article is talking about. These kinds of jobs are very common in HPC, and I'd imagine in ML performed on HPC resources.
- prpl 4y ago
  Batch processing predates HPC. It goes back to mainframes - it’s just that HPC had a common lineage there. Like you said, it’s an unattended job that runs. That’s what it’s getting at- setting up routines for unattended execution in batch. BTW Chris Ré did stuff with UW Madison /HTCondor and definitely knows what batch computing is
whoevercares 4y ago
ChatGPT can already do the whole ETL thing given an intent or profile, detect data quality issues. It just hasn’t fit into batch processing yet. But on the other hands, do you really need batch processing to understand data? Wouldn’t a sample work for a whole lot of cases already?

crabbone 4y ago

ETL and batch processing have little to do with each other. Batch processing is a term from HPC systems, which are typically multi-user, and have to share computing resources by means of a workload manager (eg. Slurm, PBS, UGE etc.) Workload managers function by defining queues, queues have associated resources (eg. number of CPUs, particular network hardware, accelerators, s.a. GPUs and so on). Users interact with workload managers by scheduling execution of their programs on particular queues (because they want the resources associated with those queues). The interaction between the user and the workload manager is described as "submission of a batch job", where this is contrasted with running a program interactively (s.t. the user can simply start the program and observe its behavior right away). I believe, these are the batch jobs the article is talking about. These kinds of jobs are very common in HPC, and I'd imagine in ML performed on HPC resources.

prpl 4y ago
Batch processing predates HPC. It goes back to mainframes - it’s just that HPC had a common lineage there. Like you said, it’s an unattended job that runs. That’s what it’s getting at- setting up routines for unattended execution in batch. BTW Chris Ré did stuff with UW Madison /HTCondor and definitely knows what batch computing is

whoevercares 4y ago

ChatGPT can already do the whole ETL thing given an intent or profile, detect data quality issues. It just hasn’t fit into batch processing yet. But on the other hands, do you really need batch processing to understand data? Wouldn’t a sample work for a whole lot of cases already?