2 ms·
Spark probably powers 90% of FANG batch data pipelines. Theres still lots of hive and Hadoop clusters out there that are just being maintained for reporting unt
by faizshah 3y ago
Spark probably powers 90% of FANG batch data pipelines. Theres still lots of hive and Hadoop clusters out there that are just being maintained for reporting until they get funding to swap to some fancy DBT setup.
Spark is pretty much ubiquitous and the default solution for batch processing right now (especially pyspark). There’s also a lot of users of AWS Glue with spark.