5 ms·

Aside from Apache Spark, what's replaced it and does it still face the same speed of access limitations compared to just zipping through giant CSVs with awk or

Aside from Apache Spark, what's replaced it and does it still face the same speed of access limitations compared to just zipping through giant CSVs with awk or whatever streaming APIs you write with your own preferred language?

bpodgursky 5y ago
I use Spark for a number of jobs for language-specific features still but I think within 2 years all custom code will be trivially invoked as native UDFs in SQL data warehouses (ie Snowflake, which has essentially solved big-data performance as a going concern). I just write SQL in Snowflake and it replaces 95% of what I would otherwise have done in custom MapReduce or Spark code.
- mohanmcgeek 5y ago
  But I can write SQL in Spark just fine. Can't I?
  - bpodgursky 5y ago
    You can write some SQL in Spark, but 1) Why would you want to maintain your own Spark infrastructure? Spark on Kube is a huge improvement over YARN but you still have to deal with OOMEs, filled disks, Kube upgrades, pushing custom images to container registries, etc etc etc. 2) Snowflake is probably 10-50x as performant as Spark for data manipulation. I don't know what kind of unholy demonic incantations Snowflake is doing on the backend to support their SQL performance, but it's really freaking fast. There's just no other way to cut it. I've spent 5-10 years eking every ounce of performance I can get out of a Hadoop/Spark cluster. I'm not trying to be unreasonable about this. I would love for OSS to be competitive; it's great for the world, and it would be great for my skill set and earning potential. But it's not a contest, and if you think standalone Spark is going to be a viable competitor in a couple years, you are deluding yourself. Make informed choices about your career and investment.
    - rxin 5y ago
      You should try Databricks, especially the new Photon engine powering Spark. In general more performant than Snowflake in SQL and a lot more flexible. (There are some cases in which Databricks would be slower but the perf is improving rapidly.)
      - Fiahil 5y ago
        Databricks has an extremely bad API. So, sure, your Spark jobs might be a little bit faster some times, but why would you use it if you can't even read logs of running jobs?
      - soulbadguy 5y ago
        Very surprised by this. Do you have a reference ?
        fnord123 5y ago
        Fyi, rxin is co-founder of databricks.
        soulbadguy 5y ago
        That explains it
      - deleted 5y ago
        [deleted]
      - StreamBright 5y ago
        The biggest selling point of Snowflake for most of the customers is that they do not need to maintain the infrastructure.
      - fritkot 5y ago
        Of course you would say that it's more performant and flexible ...TCP-DS was just a PR ploy
      - 5y ago
- deepstack 5y ago
  SQL will always be faster than Hadoop and MapReduce. The main reason to use those other slower services is developer are not use to SQL or declarative programming, and insist on having the code in Procedural way.
  - lmm 5y ago
    That's completely backwards. Mapreduce-like approaches are how SQL datastores are implemented underneath; the absolute best case for SQL is to equal hand-tuned mapreduce-like performance, and often it will be slower (you're at the mercy of your query planner to pick the right indices, do joins in the right order, etc.). The main reason people use SQL is because they find it easier to express a query that way (which is completely legitimate - if your query planner is good enough most of the time, you've got better things to be doing than hand-tuning your query execution).
    - 988747 5y ago
      No, that does not seem correct. SQL Datastores are not "map-reduce underneath", they have optimized datastructures for efficient querying (i.e. indices). Map-reduce is equivalent to those cases in SQL database where you have full table scan in your query plan - basically brute-forcing your way through the dataset.
      - lmm 5y ago
        You can (and often should) have indices in a map-reduce situation as well - you just build them in an explicit, visible way. But in most of the relevant use cases you're doing some kind of aggregation over the whole table, so indices don't help any.
        tremon 5y ago
        And if your primary use-case is column-wise aggregation over the whole table, in SQL you'd use a (compressed) column store rather than a row store as your table storage method.
- markus_zhang 5y ago
  What I really dislike modern cloud DWH such as Snowflake is that it hides a lot of things from me. Since I'm not a CTO who worries about not delivering, but a junior DE who actually wants to learn things, I really prefer that things were done in the old ways where we had to manage our own infrastructure and our own code for ETL. These kinds of things can not be learned "just for fun" because one has to work in a real environment.
  - manigandham 5y ago
    What do you mean? Of course you can still learn them "just for fun" if you want. There are plenty of columnar data warehouses (memsql, greenplum, vertica, clickhouse, etc) and data processing frameworks (spark, flink, etc) that you can look at, implement and run yourself. It's all using the same principles underneath.
    - markus_zhang 5y ago
      What I'm saying is that you can surely scratch the basics from personal use, but it's completely different from real usage and such can only be trained on job. Now that those jobs are fewer as everyone goes on cloud.
  - tharne 5y ago
    > What I really dislike modern cloud DWH such as Snowflake is that it hides a lot of things from me. This is by design. The less skill required to use a tool, the less some CTO has to pay the person using it.
kohlerm 5y ago
Apache Flink is an alternative with "real" (not micro batching) streaming support.
- easytiger 5y ago
  I spent a while putting a POC of Flink for a use case together at client behest a couple of years ago. Really struggled with it. Seemed slow as heck too. I'd much rather write a few hundred extra lines of my own code to do stuff like that (ingest -> process -> output. Modern software fashion trends seems to create as much work as it tries to save; as extensive and magical as a lot of the features are.
StreamBright 5y ago
There are two kind of approaches that I see: - skip streaming entirely and have near real time solutions using just storage + a query engine - have streaming using message queues and lambda architecture In both cases the goal is that your freshest data shows up on a dashboard.

bpodgursky 5y ago

I use Spark for a number of jobs for language-specific features still but I think within 2 years all custom code will be trivially invoked as native UDFs in SQL data warehouses (ie Snowflake, which has essentially solved big-data performance as a going concern). I just write SQL in Snowflake and it replaces 95% of what I would otherwise have done in custom MapReduce or Spark code.

mohanmcgeek 5y ago
But I can write SQL in Spark just fine. Can't I?
- bpodgursky 5y ago
  You can write some SQL in Spark, but 1) Why would you want to maintain your own Spark infrastructure? Spark on Kube is a huge improvement over YARN but you still have to deal with OOMEs, filled disks, Kube upgrades, pushing custom images to container registries, etc etc etc. 2) Snowflake is probably 10-50x as performant as Spark for data manipulation. I don't know what kind of unholy demonic incantations Snowflake is doing on the backend to support their SQL performance, but it's really freaking fast. There's just no other way to cut it. I've spent 5-10 years eking every ounce of performance I can get out of a Hadoop/Spark cluster. I'm not trying to be unreasonable about this. I would love for OSS to be competitive; it's great for the world, and it would be great for my skill set and earning potential. But it's not a contest, and if you think standalone Spark is going to be a viable competitor in a couple years, you are deluding yourself. Make informed choices about your career and investment.
  - rxin 5y ago
    You should try Databricks, especially the new Photon engine powering Spark. In general more performant than Snowflake in SQL and a lot more flexible. (There are some cases in which Databricks would be slower but the perf is improving rapidly.)
    - Fiahil 5y ago
      Databricks has an extremely bad API. So, sure, your Spark jobs might be a little bit faster some times, but why would you use it if you can't even read logs of running jobs?
    - soulbadguy 5y ago
      Very surprised by this. Do you have a reference ?
      - fnord123 5y ago
        Fyi, rxin is co-founder of databricks.
        soulbadguy 5y ago
        That explains it
    - deleted 5y ago
      [deleted]
    - StreamBright 5y ago
      The biggest selling point of Snowflake for most of the customers is that they do not need to maintain the infrastructure.
    - fritkot 5y ago
      Of course you would say that it's more performant and flexible ...TCP-DS was just a PR ploy
    - 5y ago
deepstack 5y ago
SQL will always be faster than Hadoop and MapReduce. The main reason to use those other slower services is developer are not use to SQL or declarative programming, and insist on having the code in Procedural way.
- lmm 5y ago
  That's completely backwards. Mapreduce-like approaches are how SQL datastores are implemented underneath; the absolute best case for SQL is to equal hand-tuned mapreduce-like performance, and often it will be slower (you're at the mercy of your query planner to pick the right indices, do joins in the right order, etc.). The main reason people use SQL is because they find it easier to express a query that way (which is completely legitimate - if your query planner is good enough most of the time, you've got better things to be doing than hand-tuning your query execution).
  - 988747 5y ago
    No, that does not seem correct. SQL Datastores are not "map-reduce underneath", they have optimized datastructures for efficient querying (i.e. indices). Map-reduce is equivalent to those cases in SQL database where you have full table scan in your query plan - basically brute-forcing your way through the dataset.
    - lmm 5y ago
      You can (and often should) have indices in a map-reduce situation as well - you just build them in an explicit, visible way. But in most of the relevant use cases you're doing some kind of aggregation over the whole table, so indices don't help any.
      - tremon 5y ago
        And if your primary use-case is column-wise aggregation over the whole table, in SQL you'd use a (compressed) column store rather than a row store as your table storage method.
markus_zhang 5y ago
What I really dislike modern cloud DWH such as Snowflake is that it hides a lot of things from me. Since I'm not a CTO who worries about not delivering, but a junior DE who actually wants to learn things, I really prefer that things were done in the old ways where we had to manage our own infrastructure and our own code for ETL. These kinds of things can not be learned "just for fun" because one has to work in a real environment.
- manigandham 5y ago
  What do you mean? Of course you can still learn them "just for fun" if you want. There are plenty of columnar data warehouses (memsql, greenplum, vertica, clickhouse, etc) and data processing frameworks (spark, flink, etc) that you can look at, implement and run yourself. It's all using the same principles underneath.
  - markus_zhang 5y ago
    What I'm saying is that you can surely scratch the basics from personal use, but it's completely different from real usage and such can only be trained on job. Now that those jobs are fewer as everyone goes on cloud.
- tharne 5y ago
  > What I really dislike modern cloud DWH such as Snowflake is that it hides a lot of things from me. This is by design. The less skill required to use a tool, the less some CTO has to pay the person using it.

kohlerm 5y ago

Apache Flink is an alternative with "real" (not micro batching) streaming support.

easytiger 5y ago
I spent a while putting a POC of Flink for a use case together at client behest a couple of years ago. Really struggled with it. Seemed slow as heck too. I'd much rather write a few hundred extra lines of my own code to do stuff like that (ingest -> process -> output. Modern software fashion trends seems to create as much work as it tries to save; as extensive and magical as a lot of the features are.

StreamBright 5y ago

There are two kind of approaches that I see: - skip streaming entirely and have near real time solutions using just storage + a query engine - have streaming using message queues and lambda architecture In both cases the goal is that your freshest data shows up on a dashboard.