It's worth noting that the reason this works well is that Datasets is backed by Arrow, which allows streaming remote data. More info in the base library docs: h

It's worth noting that the reason this works well is that Datasets is backed by Arrow, which allows streaming remote data. More info in the base library docs: https://huggingface.co/docs/datasets/stream https://huggingface.co/docs/datasets/stream