6 ms·
Lessons Learned from Scaling to Multi-Terabyte Datasets
- sponaugle 2y agoWhen I first read the title I had in my head that it said multi-petabyte. When I think Multi-terabyte, I think 'yea, that is mostly in RAM'. 4TBs of RAM is sort of the new lower norm in DB machines, and GCP and AWS have high RAM machines at 12TB and 24TB. Not that these are not good ideas about scaling to multi-terabytes.. it is just interesting how much data has grown in size.
- v2thegreat 2y agoHi! I'm the author. I've seen this comment several times and thought I'd address it here. I should've clarified that when I say multi-terabyte, I meant 100s TBs of data. It's my first time writing anything, so I'm still getting the hang of things. Damn, that would've been a more clickbait title, too. Oh well.
- sponaugle 2y agoAh yea, no worries. It is always good to consider ways to optimize, either for TB or PB!
- fifilura 2y agoSince you are already running on AWS, I wonder if you would consider AWS serverless EMR an option? https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/emr-serverless.html https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGu... I guess the pricing will be cheaper than renting entire machines since you are doing something like micro-spot allocations for map or reduce jobs. We have done a lot of analytical and also batched workflows with AWS Athena. This is really cheap and also, once you get a hang of it, you get surprised how much you can achieve with only SQL. I am hoping serverless EMR would be roughly equivalent, just with another language...
- therobots927 2y agoI’m not familiar with AWS Athena. But I am very familiar with being restricted to SQL for data processing and the sky really is the limit as long as you’re working with tabular data.
- fifilura 2y agoAthena is presto/trino rebranded. But you dont really pay for CPU and it can basically scale to 100 CPUs for free. It is not really a database but an SQL interface for map/reduce jobs. But it can handle any size of data and is pretty efficient when it is partitioned. And it has zero setup cost.
- therobots927 2y agoAh yes then I am very familiar with Athena- like tools. At my old job I used pyspark and I must say I did enjoy how easy it was to make custom python UDFs in combination with SQL so I miss that in a pure SQL environment. But for a lot of applications SQL gets the job done.
- fifilura 2y agoYeah this is why i am hoping that EMR serverless would be the next step. You get the freedom of not having to set up the resources. It is cheap because you are basically eating the breadcrumbs of available cpu capacity "oh i have 10 idle cpus here in Italy to do one part of you map job" etc. But you are not locked in to SQL. (Athena has UDFs but they are not very user friendly).
- sammysidhu 2y agohttps://github.com/Eventual-Inc/Daft https://github.com/Eventual-Inc/Daft Is also great at these types of workloads since it’s both distributed and vectorized!
- hdhshdhshdjd 2y ago[dead]
- atemerev 2y agoA “multi-terabyte dataset” is something that fits into a single machine, and can be loaded into Clickhouse in a few minutes.
- JBorrow 2y agoWell, that is only true (and helpful) if it is a dataset that benefits from being in a tabular format. Consider, for instance, particle-based simulation data (loading that into any database is a waste of time...), or a set of images that you need to perform non-standard individualized processing on (again, a waste of time). Stuffing stuff into postgres or clickhouse is great for your typical 'data science' workflows where data is consistent and the problem is 'oh no our transaction volume has increased 100x'. But in other (some would claim more interesting) cases, using database systems is unhelpful.
- v2thegreat 2y agoHi! I'm the author. You're right! The context that I was talking about was actually in the GIS space! Depending on what you're trying to do, it's not uncommon to have a single dataset that's 50TB large (think: 100-500m resolution global raster with daily data for 30 years) And that's not even considering any memory overhead when performing operations on top of that data. That type of stuff wouldn't fit into Postgres (we tried once, a long time ago), and usually, you might be trying to use multiple datasets simultaneously, so that data adds up pretty quickly! It's my first post (and writing) ever, so I have room for improvement. Thanks for taking the time to read!
- afpx 2y agoProbably ignorant question, but how do I avoid the issues of cross join say across several tables (or equivalents)? It seems like I always need more than a many TBs of memory. Do you spinup a U-18TB1 112xlarge or something to the equivalent as needed? Or, are you running it all day? I've never used Clickhouse, but I'd dealt with a spark system that was written like 10 years ago. I'm wondering if it's something easily migrated to Clickhouse?
- 2y ago
- jakozaur 2y agoServers grow much bigger. 256GB of RAM is pretty much a standard rack server on a major cloud. On AWS, ultra memory with 24TB of RAM is one API call away from you :-). The article took note of that, but tooling on one node puzzles me. So, multi TB can still be a single node with DuckDB. It's also rather a small ClickHouse cluster. It sounds easier to use DuckDB than the proposed tools. In many use cases, if it's not 24/7, you can use a data lake (e.g. Iceberg) and query it if needed. Databricks seems a way to go, since the author uses Spark.
- mulmen 2y agoWhat do you mean by “not 24/7”? Why would a data lake go offline?
- pantsforbirds 2y agoIt's pretty common to run analytics workloads on a fixed schedule. I've setup anlaytics workflows that ingest data every day/week/quarter and generate a report once a day. With workflows like that something like DuckDB can be amazing
- jakozaur 2y agoIt is very common that the majority of work happens during business hours. In some setups it's easy. It's better to get beefy instance during that time and avoid having it running all the time.
- dimaor 2y agonot sure, but maybe it's possible only to update the model in a specific time? are there other uses to the data apart from learning and validation?
- creshal 2y agoProbably in the sense of "you're only running the really heavy analytics queries once a day/week/month", and thus don't constantly need the 24TB instances.
- mritchie712 2y ago
- deleted 2y ago[deleted]