4 ms·
Counterpoint: You need to do some processing, like for example create a summary report of several gigs of really big json docs every few seconds. This is a pro
by dimitar 5y ago
Counterpoint:
You need to do some processing, like for example create a summary report of several gigs of really big json docs every few seconds. This is a problem that can really be sped up by using more cores.
Which option will you choose:
1.) Install configure and maintain a stream processing tool/framework and add dependencies to your code? Oh and don't forget to add service discovery, special filesystems and install 7 different runtimes and all the dependencies.
2.) Run Kubernetes + Kafka and several python or node.js microservices (each with many instances, because node.js is practically single-threaded) - one to chop up the data and put it on a queue, another that reads it from the queue and actually processes it then placing the results on the queue and another one reading it from there and responding to the ui.
3.) Create a function that does all the processing per document. Run it using pmap (parallel map, starts off threads in the background) or reducers in Clojure or your favourite framework. It uses all the cores on your machine.
I know I'll chose the more Boring technology in most cases, as most cases are pretty boring and don't need all that advanced services, orchestrators and architectures.
- nanis 5y ago+1 ... You do (3) until someone hires a "data engineer" straight out of a bootcamp and tells everyone above you that the team is not following "modern practices" and using archaic approaches. Then, you get pulled into regular meetings to explain the time line for migrating to a proper structure. Now, there will come a time where you'll need to automate this. If/when that time comes, if you already have decent cloud competency, then it is far easier to tie everything using S3, SNS, SQS, Lambda (or other equivalent) than going the Kubernetes/Kafka route.
- wwweston 5y ago> Now, there will come a time where you'll need to automate this. Unless you buy hiring-from-a-bootcamp-to-be-told-your-practices-are-archaic-and-pulled-into-regular-meetings as a service, of course.
- robotastronaut 5y agoI think there's another side to this coin: very small software teams moving quickly in markets where truly senior devs are scarce. It makes far more engineering and business sense to invest in cloud infrastructure than it is to build and control all of those systems in-house. I can hire a middle-of-the-road developer and trust they've (at the very least) heard of the AWS or GCP tools/services we're using, but if I wrote my own systems in clojure/elixir/whatever (even though that's what I'd prefer to do), then there's nowhere near the likelihood that the new engineer will know what to do, and it'll take months to train them up to even a basic level of competency. You can make all sorts of "it's better in the long run" kind of arguments, but those don't help when the C suite says "yeah sure, maybe that's the best approach, but we need to get this done right now." That's where clicking a few buttons to spin up a load balancer in front of a handful of serverless handlers becomes rather nice.
- marcus_holmes 5y agoUntil you come to look for your next job, and suddenly find that your experience with Boring technology is worth nothing, because everyone is running their should-be-a-monolith, serves-maybe-10-customers-an-hour application as a set of microservices in Kubernetes.
- Nextgrid 5y agoDepends if you're developing to be a developer or developing to solve business problems. If you're doing the latter, your clients hire you to get their problems solved and couldn't care less about the tech behind it (but will be pleased to be paying $100/month for a server instead of $1000/month for various AWS services).
- marcus_holmes 5y agoThis totally should be the case. But it isn't.
- reese_john 5y agoI don't disagree, but if you are fully invested in some cloud vendor, there are maybe easier and cheaper options than basically rolling your own mapreduce. In AWS for instance, you are one API call away from running a Spark job in Glue or EMR, that effectively does all that for you. Plus you get logging, monitoring, concurrency limiting, retries. Or, even easier, you could just dump the big json docs in S3 and use Athena to directly query them with SQL.
- jmchuster 5y agoFor many companies (1) and (2) are the boring choices, as the infra is already in place, well-supported, and there are existing patterns to follow for how it's already being used by other teams in your company.
- cjfd 5y agoAnd every single refactoring that leads ever so slightly towards (3) will be an obvious improvement in performance and reliablity...
- jmchuster 5y agoRight, if you have the resources and the ability, then writing bespoke tailored solutions will yield better results. You only get problems when your custom solution isn't as good as a hardened solution, or you can't spend the time to make it so.
- dopylitty 5y agoQuerying big JSON files in S3 is basically the entire reason S3 Select exists. Zero infrastructure needed in that case.