3 ms·

Since you are already running on AWS, I wonder if you would consider AWS serverless EMR an option? https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGu

Since you are already running on AWS, I wonder if you would consider AWS serverless EMR an option? https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/emr-serverless.html https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGu... I guess the pricing will be cheaper than renting entire machines since you are doing something like micro-spot allocations for map or reduce jobs. We have done a lot of analytical and also batched workflows with AWS Athena. This is really cheap and also, once you get a hang of it, you get surprised how much you can achieve with only SQL. I am hoping serverless EMR would be roughly equivalent, just with another language...

therobots927 2y ago
I’m not familiar with AWS Athena. But I am very familiar with being restricted to SQL for data processing and the sky really is the limit as long as you’re working with tabular data.
- fifilura 2y ago
  Athena is presto/trino rebranded. But you dont really pay for CPU and it can basically scale to 100 CPUs for free. It is not really a database but an SQL interface for map/reduce jobs. But it can handle any size of data and is pretty efficient when it is partitioned. And it has zero setup cost.
  - therobots927 2y ago
    Ah yes then I am very familiar with Athena- like tools. At my old job I used pyspark and I must say I did enjoy how easy it was to make custom python UDFs in combination with SQL so I miss that in a pure SQL environment. But for a lot of applications SQL gets the job done.
    - fifilura 2y ago
      Yeah this is why i am hoping that EMR serverless would be the next step. You get the freedom of not having to set up the resources. It is cheap because you are basically eating the breadcrumbs of available cpu capacity "oh i have 10 idle cpus here in Italy to do one part of you map job" etc. But you are not locked in to SQL. (Athena has UDFs but they are not very user friendly).

therobots927 2y ago

I’m not familiar with AWS Athena. But I am very familiar with being restricted to SQL for data processing and the sky really is the limit as long as you’re working with tabular data.

fifilura 2y ago
Athena is presto/trino rebranded. But you dont really pay for CPU and it can basically scale to 100 CPUs for free. It is not really a database but an SQL interface for map/reduce jobs. But it can handle any size of data and is pretty efficient when it is partitioned. And it has zero setup cost.
- therobots927 2y ago
  Ah yes then I am very familiar with Athena- like tools. At my old job I used pyspark and I must say I did enjoy how easy it was to make custom python UDFs in combination with SQL so I miss that in a pure SQL environment. But for a lot of applications SQL gets the job done.
  - fifilura 2y ago
    Yeah this is why i am hoping that EMR serverless would be the next step. You get the freedom of not having to set up the resources. It is cheap because you are basically eating the breadcrumbs of available cpu capacity "oh i have 10 idle cpus here in Italy to do one part of you map job" etc. But you are not locked in to SQL. (Athena has UDFs but they are not very user friendly).