3 ms·

I think it’s quite rare that a dataset is too large to be processed on a single volume. EBS volumes max out at 64Tb

hobs 2y ago
It's really the only reason to use Spark in the first place, because you're doing non-local level processing - when I was throwing 100TB at the problem it made more sense than most of the data science tasks I see it used for.
- mnsc 2y ago
  This magnitude of data fascinates me. Can you elaborate on how that much data came to be and what kind of processing was needed for all that data? And if not, maybe point to some podcast, blog that goes into the nitty gritty of those types of real big data challenges.
  - leprechaun1066 2y ago
    Financial market data (especially FX data) can have thousands of ticks per second. It's not unheard of for data engineers in that space to sometimes handle 1TB per day.
  - hobs 2y ago
    Click/traffic data for a top 100 website. We weren't doing a ton, but basic recommendation processing, search improvement, pattern matching in user behavior, etc We normally still would only need to process say, the last 10 days of user data to get decent recommendations, but occasionally it would make sense for processes running over the entire dataset. Also this isn't that large when you consider binary artifacts (say, healthcare imaging) being stored in a database, which pretty sure that's what a lot of electronic healthcare record systems do. A random company I bumped into has a 40TB OLTP database to this effect.
threeseed 2y ago
If you're using EBS for shuffle/source data you've already behind. If you can afford it you should be using the host NVME drives or FSX and relying on Spark to handle outages. The difference in performance will be orders of magnitude different. And in this case you won't have the ability to store 64TB. The average max is 2TB.
anonymousDan 2y ago
It's not just disk capacity though. Disk bandwidth, CPU and memory capacity/bandwidth etc can also become bottlenecks on a single machine.

hobs 2y ago

It's really the only reason to use Spark in the first place, because you're doing non-local level processing - when I was throwing 100TB at the problem it made more sense than most of the data science tasks I see it used for.

mnsc 2y ago
This magnitude of data fascinates me. Can you elaborate on how that much data came to be and what kind of processing was needed for all that data? And if not, maybe point to some podcast, blog that goes into the nitty gritty of those types of real big data challenges.
- leprechaun1066 2y ago
  Financial market data (especially FX data) can have thousands of ticks per second. It's not unheard of for data engineers in that space to sometimes handle 1TB per day.
- hobs 2y ago
  Click/traffic data for a top 100 website. We weren't doing a ton, but basic recommendation processing, search improvement, pattern matching in user behavior, etc We normally still would only need to process say, the last 10 days of user data to get decent recommendations, but occasionally it would make sense for processes running over the entire dataset. Also this isn't that large when you consider binary artifacts (say, healthcare imaging) being stored in a database, which pretty sure that's what a lot of electronic healthcare record systems do. A random company I bumped into has a 40TB OLTP database to this effect.

threeseed 2y ago

If you're using EBS for shuffle/source data you've already behind. If you can afford it you should be using the host NVME drives or FSX and relying on Spark to handle outages. The difference in performance will be orders of magnitude different. And in this case you won't have the ability to store 64TB. The average max is 2TB.

anonymousDan 2y ago

It's not just disk capacity though. Disk bandwidth, CPU and memory capacity/bandwidth etc can also become bottlenecks on a single machine.