9 ms·
Amazon's exabyte-scale migration from Apache Spark to Ray on EC2
- Narhem 2y agoAbsolutely insane work. So much data you’d think they would come up with a custom solution instead of using the “newest available toolkit” but I understand how much of a mess dealing with that much data is.
- thedood 2y agoHi Narhem - one of the devs that worked on the migration here. The data volume, and subsequent compute power required to process it, is actually one of the things that led us to Ray (or Ray Core specifically) since it had the distributed computing primitives (tasks and actors) that we needed to build out our envisioned solution with very few compromises. One thing we DIDN'T want to do was just throw another one-liner SQL statement running on a new data processing framework at the problem, since that leads us back to the problems we had with Spark - not enough low-level control for such an important problem. In short, after evaluating our options, Ray seemed to strike the best balance between the one efficiency extreme of, say, building out custom "compaction-optimized" hardware/clusters, and the other maintainability extreme of just letting the latest managed cloud service run a 1-liner SQL statement for us without ever looking under the hood. Regardless, I expect both our existing solution and the distributed compute frameworks leveraged to deliver it to continue to evolve over time.
- mannyv 2y agoCrazy that the project took almost 4 years end-to-end, and it's still ongoing. I had no idea anything at AWS had that long of an attention span. It's funny and telling that in the end, it's all backed by CSVs in s3. Long live CSV!
- jerrygenser 2y agoThey reference parquet files, not sure if it's only CSV or CSV even figures in that heavily other than the first iteration before migrating to spark
- thedood 2y agoHi mannyv - one of the devs that worked on the migration here. It has been a pretty long project - approached with caution due to the criticality of keeping our BI datasets healthy - but the preliminary results produced year-over-year kept looking promising enough to keep after it. =) Also, we mostly have Parquet data cataloged in S3 today, but delimited text is indeed ubiquitous and surprisingly sticky, so we continue to maintain some very large datasets natively in this format. However, while the table's data producer may prefer to write delimited text, they are almost always converted to Parquet during the compaction process to produce a read-optimized table variant downstream.
- gregw2 2y agoAre you all shifting over to storing as iceberg-enriched parquet yet and letting it (within, say Athena) manage compaction or thinking about it, or is it not worth it since this new Ray+Parquet thing is working for you?
- thedood 2y agoAs alluded to in the blog post, Ray+Parquet+Iceberg is the next frontier we'd like to make our compactor and similar procedures available on in open source so that the community can start bringing similar benefits for their Iceberg workloads. Stay tuned. =)
- sdfsdfadf 2y ago[flagged]
- wenc 2y agoMost people are moving away from CSV for big datasets, except in exceptional cases involving linear reads (append only ETL). CSV has one big upside which is human readability. But so many downsides: poor random access, no typing, no compression, complex parser needing to handle exceptions.
- 100pctremote 2y agoMost people don't directly query or otherwise operate on raw CSV, though. Large source datasets in CSV format still reign in many enterprises, but these are typically read into a dataframe, manipulated and stored as Parquet and the like, then operated upon by DuckDB, Polars, etc., or modeled (E.g. DBT) and pushed to an OLAP target.
- wenc 2y agoThere are folks who still directly query CSV formats in a data lake using a query engine like Athena or Spark or Redshift Spectrum — which ends up being much slower and consuming more resources than is necessary due to full table scans. CSV is only good for append only. But so is Parquet and if you can write Parquet from the get go, you save on storage as well has have a directly queryable column store from the start. CSV still exists because of legacy data generating processes and dearth of Parquet familiarity among many software engineers. CSV is simple to generate and easy to troubleshoot without specialized tools (compared to Parquet which requires tools like Visidata). But you pay for it elsewhere.
- whoevercares 2y agoIs it really AWS? I don’t recall any service called BDT
- p0rkbelly 2y agoThe second paragraph discusses that BDT is an internal team at Amazon Retail. They used AWS and Ray to do this.
- qwerp 2y agoour plans are measured in centuries
- quadrature 2y agoAnyone know enough about ray to comment on what the exact performance unlock was ?. They mention that it gave them enough control over the distribution of work so that they could avoid unnecessary reads/write. That seems like a good win but I would assume that doing compaction in python would be quite slow.
- thedood 2y agoSome of the initial differentiators are described at the bottom of our design doc at https://github.com/ray-project/deltacat/blob/main/deltacat/compute/compactor/TheFlashCompactorDesign.pdf https://github.com/ray-project/deltacat/blob/main/deltacat/c.... But yes, controlling file I/O was also an important part of this since it allowed us to (1) run more targeted downloads/reads of only the Parquet row groups and columns participating in compaction and (2) track dirty/clean files to skip unnecessary re-writes of "clean" files that weren't altered by compaction. Also, just better leveraging catalog metadata (e.g., primary key indexes if available) to filter out more files in the initial scan, and to copy clean files into the compacted variant by reference (when supported by the underlying catalog format). The trick with doing compaction in Python was to ensure that the most performance-sensitive code was delegated to more optimal C++ (e.g, Ray and Arrow) and Rust (e.g., Daft) code paths. If we did all of our per-record processing ops in pure Python, compaction would indeed be much slower.
- quadrature 2y agoThanks a lot for the explanation. Sounds a lot like how Pyspark allows for declarative definitions of computation that is actually executed in Java.
- ZeroCool2u 2y agoThis is one of the first times I've heard of people using Daft in the wild. Would you be able to elaborate on where Daft came in handy? Edit: Nvm, I kept reading! Thanks for the interesting post!
- whoiscroberts 2y agoRay user here, what language actors are they using? Ray support Python Java and cpp actors…
- thedood 2y agoWe wrote the compactor in Python but, as noted in my previous response to quadrature, most of the performance sensitive code is written in C++ and Rust (but still invoked from Python).
- esafak 2y agoAre we talking about big data ETL here? I did not know Ray was suited for it.
- thedood 2y agoThis is a specialized ETL use-case - similar to taking a single SQL query and creating a dedicated distributed application tailored to run only that query. The lower-level primitives in Ray Core (tasks and actors) are general purpose enough to make building this type of application possible, but you'll be hard pressed to quickly (i.e., with less than 1 day of effort) make any arbitrary SQL query or dataframe operation run with better efficiency or scale on Ray than on dedicated data processing frameworks like Spark. IMO, the main value add of frameworks like Spark lies more in unlocking "good enough" efficiency and scale for almost any ETL job relatively quickly and easily, even if it may not run your ETL job optimally.
- dekhn 2y agoSpeaking as a distributed computing nerd, Ray is definitely one of the more interesting and exciting frameworks I've seen in a while. It's one of those systems where reading the manual, I can see that I'm not going to have to learn anything new, because the mental model resembles so many distributed systems I've worked with before (I dunno about anybody else, but tensorflow is an example of a distributed system that forced me to forget basically everything I knew before I could be even remotely productive in it). Unclear if it's in the best interests of anyscale to promote Ray as a general purpose cluster productivity tool, even if it's good at that more general use case.
- robertnishihara 2y agoI'm glad you find it exciting! Our intention from the start was for Ray to be general purpose. And the core Ray APIs are quite general (basically just scheduling a Python function somewhere in a cluster or instantiating a Python class as a process somewhere in the cluster). We had AI use cases in mind from the start, since we were grad students in AI. But the generality has really been important since AI workloads encompass a huge variety of computational patterns (allreduce style communication patterns on GPUs for training, embarrassingly parallel data processing workloads on spot instances, and so on).
- frankjr 2y ago> From the typical Amazon EC2 customer’s perspective, this translates to saving over $120MM/year on Amazon EC2 on-demand R5 instance charges. Does the sales team know about this? /jk
- taeric 2y agoThis assumes they aren't reinvesting the money back to other tangible improvements in the services, though?
- deleted 2y ago[deleted]
- andrewxdiamond 2y agoAmazon sales and business teams are constantly focused on reducing costs for customers and they celebrate this internally too. I have seen dozens of big ticket “we saved this customer $xxxK/year” posts on slack and other internal venues, the customer obsession is real.
- immibis 2y agoEven the reduced cost is still ten times what it costs on a traditional provider, though, right?
- sofixa 2y agoWildly depends. How many people would it take to set up, deploy, maintain, secure, back up something like S3, RDS, SQS similarly redundantly based on top of just raw compute? For some organisations, it will make sense to internalise and DIY everything. For many others, AWS/equivalent enables them to break down internal silos (instead of having to file a ticket to get storage team to provision storage for your new fancy RabbitMQ cluster, multiplied by 10 for networking, security AV bullshit, etc.) and just consume everything as a service. It's much faster, and in some cases even maybe cheaper. Cf. Dropbox that started on S3, and a decade afterwards, migrated to self-managed datacenters.
- 100pctremote 2y agoRather nuts. New challenge: build datacenters quickly enough to support the new platform.
- whalesalad 2y agoVideo from the author deep diving this. https://www.youtube.com/watch?v=h7svj_oAY14 https://www.youtube.com/watch?v=h7svj_oAY14
- zitterbewegung 2y agoI was in a workshop that taught me Ray. It was interesting to know that the people who started Spark were also involved in making Ray. This is not badmouthing either project just an observation and if you architected one task you would be good at attacking the same problem better .
- fs111 2y agoWell spark was really a showcase Project for mesos when it was created. Now everyone knows a lot more
- bigcat12345678 2y agoSpark was never a showcase for mesos Mesos was a derivative idea from some sporadic idea from 2 level scheduling from inside Google based on mapreduce Mesos was wrong from day one, they thought they have the right idea, but was really caught up by a Small group of Google engineers who happened to be from academia. These engineers were very good at having theoretically nice idea In the time mesos was invented, Google had its own mesos, which is a similar project by learning wrong lesson from Borg. That thing is Omega. Eventually everyone decided that Borg was right all along, thus the kubernetes
- fs111 2y agoI have no idea what any of the google tech has to do with anything here. Quoting from the original spark paper: > Spark is built on top of Mesos [16, 15], a “cluster operat- > ing system” that lets multiple parallel applications share > a cluster in a fine-grained manner and provides an API > for applications to launch tasks on a cluster https://people.csail.mit.edu/matei/papers/2010/hotcloud_spark.pdf https://people.csail.mit.edu/matei/papers/2010/hotcloud_spar... Note how Matei Zaharia - the inventor of spark - is also on the mesos paper: https://people.eecs.berkeley.edu/~alig/papers/mesos.pdf https://people.eecs.berkeley.edu/~alig/papers/mesos.pdf
- dekhn 2y ago
- igmor 2y agoCan you share any data on how big of a cluster is running Ray jobs?
- thedood 2y agoFrom the blog post, the largest individual Ray cluster that was observed running a production compaction job in Q1 had 26,846 vCPUs and ~210TiB of RAM. This is roughly equivalent to a Ray cluster composed of 839 r5.8xlarge EC2 nodes (w/ 32 vCPUs and 256GiB RAM per node).
- layoric 2y agoFor those interested, this would be at a cost of: - ~$1691/hour on demand - ~$1065/hour reserved - ~$521/hour spot Not including any related data transfer costs.
- LarsDu88 2y agoWow that is not NEARLY as expensive as I would have imagined considering the scale of the data involved.
- artemisart 2y agoThat's also $41k / 26k / 13k per day or $1.2M / 767k / 375k per month!
- LarsDu88 2y agoI mean, for all of Amazon's business intelligence, running on servers operated by Amazon... tis a mere pittance.
- robertnishihara 2y agoI'm one of the creators of Ray. A few thoughts :) 1. This is truly impressive work from AWS. Patrick Ames began speaking about this a couple years ago, though at this point the blog post is probably the best reference. https://www.youtube.com/watch?v=h7svj_oAY14 https://www.youtube.com/watch?v=h7svj_oAY14 2. This is not a "typical" Ray use case. I'm not aware of any other exabyte scale data processing workloads. Our bread and butter is ML workloads: training, inference, and unstructured data processing. 3. We have a data processing library called Ray Data for ingesting and processing data, often done in conjunction with training and inference. However, I believe in this particular use case, the heavy lifting is largely done with Ray's core APIs (tasks & actors), which are lower level and more flexible, which makes sense for highly custom use cases. Most Ray users use the Ray libraries (train, data, serve), but power users often use the Ray core APIs. 4. Since people often ask about data processing with Ray and Spark, Spark use cases tend to be more geared toward structured data and CPU processing. If you are joining a bunch of tables together or running SQL queries, Spark is going to be way better. If you're working with unstructured data (images, text, video, audio, etc), need mixed CPU & GPU compute, are doing deep learning and running inference, etc, then Ray is going to be much better.
- justsocrateasin 2y agoI'm just learning about this tool now and had a brief question if you have the time: The paper mentions support for zero-copy intranode object sharing which links to serialization in the Ray docs - https://docs.ray.io/en/latest/ray-core/objects/serialization.html#serialization https://docs.ray.io/en/latest/ray-core/objects/serialization... I'm really curious how this is performant - I recently tried building a pipeline that leveraged substantial multiprocessing in Python, and found that my process was bottlenecked by the serialization/deserialization that occurs during Python multiprocessing. Would love any reading or explanation you can provide as to how this doesn't also bottleneck a process in Ray, since it seems that data transferred between workers and nodes will need to serialized and deserialized. Thanks in advance! Really cool tool, hopefully I'll be able to use it sooner rather than later.
- robertnishihara 2y ago
- parhamn 2y agoIm curious, how do data scientists use these massive datasets, especially the old stuff. Is it more of a compliance and need/should-save type thing or is the data actually useful? Im baffled by these numbers having never used a large BI tool, and am genuinely curious how the data is actually used operationally. As a layman, I imagine lots of it loses relevancy very quickly, e.g Amazon sales data from 5 years ago is marginally useful to determining future trends and analyzing new consumer behavior regimes?
- gregw2 2y agoIf you have seasonal demand patterns, you generally need three years history to do good predictive analytics. I do tend to agree data from five years ago is rarely relevant BUT our business is still using for some BI purposes data from the fiscal year before COVID as a comparison baseline for certain analytics/business processes which have been slow to reach pre-COVID levels of performance. So that means we are now using data 6 years old, comparing this year to that pre-COVID year for certain analytics!
- physicsguy 2y agoYeah 100%. I worked in wind energy for a while and the DS team would be pulling as much data as they could get to establish a baseline for normality due to seasonal trends in the wind. This also varied enormously around the world - for e.g. the UK is fairly windy all year, but India typically gets 2/3 of it's generated wind energy in the monsoon season which is about 3 months.
- smfjaw 2y agoI work in finance and it's great having big historical datasets, even if the figures are far lower in previous years it's good to see system 'shocks' and these can be used at a different magnitude/scaled for future forecasting
- refset 2y agoYep backtesting is critical step in the deployment of new trading algorithms. Related, I rather enjoyed reading this other thread from June: "Ask HN: Is KDB a sane choice for a datalake in 2024?" https://news.ycombinator.com/item?id=40625800 https://news.ycombinator.com/item?id=40625800
- OutOfHere 2y agoCan you help us understand how others can use and derive value from Ray DeltaCAT? What would be the specific use cases?
- thedood 2y agoSome of DeltaCAT's goals and use cases have been discussed in this 2022 talk: https://youtu.be/M3pZDp1zock?t=4676 https://youtu.be/M3pZDp1zock?t=4676 Today, our immediate next goal for DeltaCAT is to ensure that the compactor, and similar procedures for Ray, can run on Apache Iceberg. So, if you're an Iceberg user relying on procedures like Spark's "rewrite_data_files" and/or "rewrite_positional_delete_files" to compact your datasets today, then DeltaCAT will let you easily run similar compaction procedures on Ray to realize similar efficiency/scale improvements (even if it winds up delegating some of the work to other projects like PyIceberg, Daft, etc. along the way). Going forward, we'd like DeltaCAT to also provide better general-purpose abstractions (e.g., reading/writing/altering large datasets) to simplify writing Ray apps in Python that work across (1) different catalog formats like Iceberg, Hudi, and Delta and (2) different distributed data processing frameworks like Ray Data, Daft, Modin, etc. From the perspective of an internal DeltaCAT developer, another goal is to just reduce the maintainability burden and dev hours required to write something like a compactor that works across multiple catalogs (i.e., by ensuring that all interfaces used by such a procedure can be readily implemented for multiple catalog formats like Iceberg, Hudi, Delta, etc.).
- jgalt212 2y agoSlightly flip, but it's interesting that no one believes in or brags about cost savings via statistical sampling techniques these days.
- dekhn 2y agowell, I can save money by eating only lentils, but I prefer a richer diet. As do BI folks in a highly profitable company.
- rubenvanwyk 2y agoWinning comment.
- jgalt212 2y agoIt's a terribly inelegant and inefficient solution that no one should be "proud" of. The only time you need N=all is for the general ledger.
- disgruntledphd2 2y ago> The only time you need N=all is for the general ledger. If you're predicting for each user, you need all of the data. And generally you probably wouldn't want to sample too much for BI as it could lead to people making wrong decisions. But yeah, in general sampling rocks and is super effective.
- whoevercares 2y agoI wonder if similar performance can be achieved with Spark accelerator like https://github.com/apache/datafusion-comet https://github.com/apache/datafusion-comet. Of course it didn’t exist 4 years ago, but would it cheaper to build?
- spott 2y agoIt sounds like a lot of the value here was in a more appropriate level of abstraction, not in the speed of the compute itself. Ray allowed them to optimize elements that spark didn’t, and that was what improved performance, not that spark itself was slow.
- jaychia 2y agoI work on Daft and we’ve been collaborating with the team at Amazon to make this happen for about a year now! We love Ray, and are excited about the awesome ecosystem of useful + scalable tools that run on it for model training and serving. We hope that Daft can complement the rest of the Ray ecosystem to enable large scale ETL/analytics to also run on your existing Ray clusters. If you have an existing Ray cluster setup, you absolutely should have access to best-in-class ETL/analytics without having to run a separate Spark cluster. Also, on the nerdier side of things - the primitives that Ray provides gives us a real opportunity to build a solid non-JVM based, vectorized distributed query engine. We’re already seeing extremely good performance improvements here vs Spark, and are really excited about some of the upcoming work to get even better performance and memory stability. This collaboration with Amazon really battle-tested our framework :) happy to answer any questions if folks have them.
- thedood 2y agoGood to see you here! It's been great working with Daft to further improve data processing on Ray, and the early results of incorporating Daft into the compactor have been very impressive. Also agree with the overall sentiment here that Ray clusters should be able to run best-in-class ETL without requiring a separate cluster maintained by another framework (Spark or otherwise). This also creates an opportunity to avoid many inefficient, high-latency cross-cluster data exchange ops often run out of necessity today (e.g., through an intermediate cloud storage layer like S3).
- hiyer 2y agoWe chose Ray over Spark in my previous company mostly because we were a Python shop and Ray is Python-native (though it's implemented in C++ I believe). It worked very well for us even for real-time queries - though we were obviously nowhere near the scale that AWS is at.
- jameskraus 2y agoI wonder if there are any good primers to these technologies. Maybe a DDIA-like book or some lectures?
- OJFord 2y agoNeed a phonebook just for all the Apache ones.
- uptownfunk 2y agoAre they that bored over there at amazon.
- alberth 2y agoHas anyone found a good ELI5 site for all the different AI toolsets. Eg Ray, Databricks, Notebooks etc
- PROMISE_237 2y ago[dead]
- PeterCorless 2y agoI remember when everyone shifted from Apache Hadoop to Apache Spark. This seems like a possibly similar sea change. I am not sure if many other users will embrace Ray over Spark, but this is a sign that people are looking to either improve Spark on some fundamental levels, or are going to reach out to new technologies to resolve their problems. Cool stuff.
- e28eta 2y agoI wonder what they’re doing to combat the growth rate of their data. A 13x speed up, or 82% cost reduction is great, but doesn’t seem significant enough compared to the growth of the business and (my assumption) demand for adding new data sources and added data to existing sources. Like, if the current latency is ~60 minutes for 90% of updates, will it ever be better than that? Won’t it just slowly degrade until the next multi-year migration? PS: this article was infuriating to read on iPad - it kept jumping back to the top of the page and couldn’t figure out why
- thedood 2y agoAs noted in the bottom of the design doc at https://github.com/ray-project/deltacat/blob/main/deltacat/compute/compactor/TheFlashCompactorDesign.pdf https://github.com/ray-project/deltacat/blob/main/deltacat/c..., we also improved the runtime efficiency of compaction from O(nlogn) to O(n). However, a lot of this also comes down to making intentional data engineering decisions to control how physical data is laid out (and retained) across files to keep reads/writes as localized as possible. For example, we found that grouping records according to the date they were last updated to be very helpful, as outlined in our 2022 Ray Summit talk: https://youtu.be/u1XqELIRabI?t=1589 https://youtu.be/u1XqELIRabI?t=1589.
- e28eta 2y agoThat’s a nice durable improvement! Thanks
- LarsDu88 2y agoAs someone who used Ray for ML, this is impressive as hell. I did not even realize it was that viable a substitute for straight data processing tasks.
- maxnevermind 2y agoWhy did it take multiple years to do that I wonder? After a new compaction framework is ironed out it should not be that difficult to onboard/spread it across to all the tables in especially considering they had a parallel setup in Spark still, so they can afford hiccups in a new setup.