4 ms·

You know: write a program like in the old days. I would perform each request in a single process: read in the metadata (mainly structure) and then either proce

by gumby 2y ago

You know: write a program like in the old days. I would perform each request in a single process: read in the metadata (mainly structure) and then either process each tab sequentially or more likely map the whole thing into memory and spawn a thread for each tab, then write the whole thing out in order. No need for the overhead of microservices: locating, invoking, transferring data, and synchronizing responses, much less dealing with all the pain of lost connections, abnormal termination and so on. The largest excel sheet I've worked on is only about 500 MB and (does a quick search of my local filesystem) almost all are less than one MB. So in the (rare) worst case the transmission doesn't justify spreading it around; in the common case there's no benefit.

handwarmers 2y ago
So what happens when this hypothetical machine of yours, that has enough NIC bandwidth to process the scale of data that needs to be streamed in both directions, enough CPU power to handle millions of concurrent requests in a process or a thread of their own, and enough ram/ fast enough disks to to map and swap all the files that are being converted, goes down? In 2022, Google Workspace apparently had ~ 3 billion users (8 million of which paying) https://developers.googleblog.com/en/year-in-review-12-awesome-ways-for-developers-to-learn-build-and-grow-with-google-workspace/#:~:text=2022%20was%20also%20a%20year,than%205%20billion%20app%20installs https://developers.googleblog.com/en/year-in-review-12-aweso... . Not every solution needs microservices. But also, we have problems today that we did not have solutions for "in the old days".
- cess11 2y ago
  Ten million per day seems unlikely to translate to millions of concurrent requests. This kind of task is typically suited to be made into a single thing, you don't want partial conversions hanging around in 'microservices' if something goes wrong. As for scaling, I'd likely put this in a process definition and run it on the BEAM if I were to make such a product. That way millions of requests per hour can hit my cluster and those that fail somehow will just get cleaned up and the transaction rolled back, the clients get 'sorry, try again' or 'sorry, we're working on fixing it', and the rest happily chug along.
  - handwarmers 2y ago
    Apache Beam?
    - cess11 2y ago
      No, the BEAM: https://en.wikipedia.org/wiki/BEAM_(Erlang_virtual_machine) https://en.wikipedia.org/wiki/BEAM_(Erlang_virtual_machine)
- gumby 2y ago
  You don't have to run them all on a single device! But you hardly need a bunch of microservices, or any, really, to do any one tranlation.
- swiftcoder 2y ago
  You keep confusing horizontal scaling with microservices. The two are basically unrelated. You have to horizontally scale regardless of whether you are running regular services or microservices - the goal of micro services is just to increase the granularity of horizontal scaling (or, more often, to solve organisational issues around feature/code ownership)

by gumby 2y ago

handwarmers 2y ago

So what happens when this hypothetical machine of yours, that has enough NIC bandwidth to process the scale of data that needs to be streamed in both directions, enough CPU power to handle millions of concurrent requests in a process or a thread of their own, and enough ram/ fast enough disks to to map and swap all the files that are being converted, goes down? In 2022, Google Workspace apparently had ~ 3 billion users (8 million of which paying) https://developers.googleblog.com/en/year-in-review-12-awesome-ways-for-developers-to-learn-build-and-grow-with-google-workspace/#:~:text=2022%20was%20also%20a%20year,than%205%20billion%20app%20installs https://developers.googleblog.com/en/year-in-review-12-aweso... . Not every solution needs microservices. But also, we have problems today that we did not have solutions for "in the old days".

cess11 2y ago
Ten million per day seems unlikely to translate to millions of concurrent requests. This kind of task is typically suited to be made into a single thing, you don't want partial conversions hanging around in 'microservices' if something goes wrong. As for scaling, I'd likely put this in a process definition and run it on the BEAM if I were to make such a product. That way millions of requests per hour can hit my cluster and those that fail somehow will just get cleaned up and the transaction rolled back, the clients get 'sorry, try again' or 'sorry, we're working on fixing it', and the rest happily chug along.
- handwarmers 2y ago
  Apache Beam?
  - cess11 2y ago
    No, the BEAM: https://en.wikipedia.org/wiki/BEAM_(Erlang_virtual_machine) https://en.wikipedia.org/wiki/BEAM_(Erlang_virtual_machine)
gumby 2y ago
You don't have to run them all on a single device! But you hardly need a bunch of microservices, or any, really, to do any one tranlation.
swiftcoder 2y ago
You keep confusing horizontal scaling with microservices. The two are basically unrelated. You have to horizontally scale regardless of whether you are running regular services or microservices - the goal of micro services is just to increase the granularity of horizontal scaling (or, more often, to solve organisational issues around feature/code ownership)