3 ms·

2021? More like 2010. Hadoop is getting deprecated rapidly and more companies split their write and read workloads. Separated storage and compute is also popula

2021? More like 2010. Hadoop is getting deprecated rapidly and more companies split their write and read workloads. Separated storage and compute is also popular. Scala is not used that much, I think it is not worth the time investment. More and more companies go for Kotlin instead of Java when these want to tap into the Java ecosystem.

adilkhash 6y ago
Hadoop is still widely used in enterprises (especially in banks), if you have experience working with Hadoop ecosystems it is a big plus anyway.
- StreamBright 6y ago
  Yes, that is the status quo. There is also some trends: https://trends.google.com/trends/explore?date=today%205-y&geo=US&q=%2Fm%2F0fdjtq,%2Fm%2F0ndhxqz https://trends.google.com/trends/explore?date=today%205-y&ge...
smattiso 6y ago
Are you working in this field? I am looking for a consultant to setup a modern data processing pipeline for a data driven hardware product I am building.
- StreamBright 6y ago
  Yes, I am moving companies to their next data pipeline, that is my specialty. I have added my email to my profile, you can reach out to me.
pjmlp 6y ago
Kotlin's future is tied with Android, on the JVM it will be another Scala in 5 years time. If JetBrains gets lucky, they might manage to create a cross-platform Kotlin eco-system as they are trying hard to push, as means to sell InteliJ licenses. Lets see if it doesn't end like like Typesafe.
kentm 6y ago
What does your typical stack look like?
- StreamBright 6y ago
  Depends. Just some random mixture of stacks: PrestoDB, S3, Airflow, Luigi, Dremio, Athena, Hive LLAP, EMC Isilon, Kafka. My favorite so far is S3 + PrestoDB with either ORC or Parquet files. It is a solid DWH solution for most enterprises on the cloud. (Cloud or not is a different discussion). It works for small scale (50TB) to really high scale (50PB). There are some (very few) gotchas and moving parts as opposed to Hadoop + co. You can combine it with Kafka for streaming data and you got yourself a pretty solid data solution.

adilkhash 6y ago

Hadoop is still widely used in enterprises (especially in banks), if you have experience working with Hadoop ecosystems it is a big plus anyway.

StreamBright 6y ago
Yes, that is the status quo. There is also some trends: https://trends.google.com/trends/explore?date=today%205-y&geo=US&q=%2Fm%2F0fdjtq,%2Fm%2F0ndhxqz https://trends.google.com/trends/explore?date=today%205-y&ge...

smattiso 6y ago

Are you working in this field? I am looking for a consultant to setup a modern data processing pipeline for a data driven hardware product I am building.

StreamBright 6y ago
Yes, I am moving companies to their next data pipeline, that is my specialty. I have added my email to my profile, you can reach out to me.

pjmlp 6y ago

Kotlin's future is tied with Android, on the JVM it will be another Scala in 5 years time. If JetBrains gets lucky, they might manage to create a cross-platform Kotlin eco-system as they are trying hard to push, as means to sell InteliJ licenses. Lets see if it doesn't end like like Typesafe.

kentm 6y ago

What does your typical stack look like?

StreamBright 6y ago
Depends. Just some random mixture of stacks: PrestoDB, S3, Airflow, Luigi, Dremio, Athena, Hive LLAP, EMC Isilon, Kafka. My favorite so far is S3 + PrestoDB with either ORC or Parquet files. It is a solid DWH solution for most enterprises on the cloud. (Cloud or not is a different discussion). It works for small scale (50TB) to really high scale (50PB). There are some (very few) gotchas and moving parts as opposed to Hadoop + co. You can combine it with Kafka for streaming data and you got yourself a pretty solid data solution.