4 ms·

What would your solution for 10 million such requests within a day be?

by handwarmers 2y ago

grugagag 2y ago
Couldn’t the conversion run locally?
- handwarmers 2y ago
  My assumption is that the comment I was responding to addressed Google Docs specifically (which afaik is web only?). If this is the case, then your options to run locally would be for the conversion to run in your browser, or for you to have some Google agent run on your computer that can handle these requests instead of the Google servers themselves, neither of which is a scalable option in this case (due to browser differences / expecting people to be able to install software locally to their devices)?
  - jraph 2y ago
    > due to browser differences Surely a file conversion should not be affected too much by browser differences, it should be a pure function, pure calculation not requiring too many APIs and which doesn't have much to do with rendering.
gumby 2y ago
You know: write a program like in the old days. I would perform each request in a single process: read in the metadata (mainly structure) and then either process each tab sequentially or more likely map the whole thing into memory and spawn a thread for each tab, then write the whole thing out in order. No need for the overhead of microservices: locating, invoking, transferring data, and synchronizing responses, much less dealing with all the pain of lost connections, abnormal termination and so on. The largest excel sheet I've worked on is only about 500 MB and (does a quick search of my local filesystem) almost all are less than one MB. So in the (rare) worst case the transmission doesn't justify spreading it around; in the common case there's no benefit.
- handwarmers 2y ago
  So what happens when this hypothetical machine of yours, that has enough NIC bandwidth to process the scale of data that needs to be streamed in both directions, enough CPU power to handle millions of concurrent requests in a process or a thread of their own, and enough ram/ fast enough disks to to map and swap all the files that are being converted, goes down? In 2022, Google Workspace apparently had ~ 3 billion users (8 million of which paying) https://developers.googleblog.com/en/year-in-review-12-awesome-ways-for-developers-to-learn-build-and-grow-with-google-workspace/#:~:text=2022%20was%20also%20a%20year,than%205%20billion%20app%20installs https://developers.googleblog.com/en/year-in-review-12-aweso... . Not every solution needs microservices. But also, we have problems today that we did not have solutions for "in the old days".
  - cess11 2y ago
    Ten million per day seems unlikely to translate to millions of concurrent requests. This kind of task is typically suited to be made into a single thing, you don't want partial conversions hanging around in 'microservices' if something goes wrong. As for scaling, I'd likely put this in a process definition and run it on the BEAM if I were to make such a product. That way millions of requests per hour can hit my cluster and those that fail somehow will just get cleaned up and the transaction rolled back, the clients get 'sorry, try again' or 'sorry, we're working on fixing it', and the rest happily chug along.
    - handwarmers 2y ago
      Apache Beam?
      - cess11 2y ago
        No, the BEAM: https://en.wikipedia.org/wiki/BEAM_(Erlang_virtual_machine) https://en.wikipedia.org/wiki/BEAM_(Erlang_virtual_machine)
  - gumby 2y ago
    You don't have to run them all on a single device! But you hardly need a bunch of microservices, or any, really, to do any one tranlation.
hmeh 2y ago
You may be conflating microservices and horizontal scaling. You don't need to have multiple (disparate) microservices to scale. Microservices have absolutely nothing to do with scaling. That's a myth started by people who never understood the actual point of microservices, which was partitioning and continuity of developer productivity.
camgunz 2y ago
Maybe this'll come off as snarky, but I would build Excel! It would spread 10 million requests across the 10 million users, be totally immune to network outages, and my users could rest assured I wasn't thumbing through their data.

What would your solution for 10 million such requests within a day be?

grugagag 2y ago

Couldn’t the conversion run locally?

handwarmers 2y ago
My assumption is that the comment I was responding to addressed Google Docs specifically (which afaik is web only?). If this is the case, then your options to run locally would be for the conversion to run in your browser, or for you to have some Google agent run on your computer that can handle these requests instead of the Google servers themselves, neither of which is a scalable option in this case (due to browser differences / expecting people to be able to install software locally to their devices)?
- jraph 2y ago
  > due to browser differences Surely a file conversion should not be affected too much by browser differences, it should be a pure function, pure calculation not requiring too many APIs and which doesn't have much to do with rendering.

gumby 2y ago

You know: write a program like in the old days. I would perform each request in a single process: read in the metadata (mainly structure) and then either process each tab sequentially or more likely map the whole thing into memory and spawn a thread for each tab, then write the whole thing out in order. No need for the overhead of microservices: locating, invoking, transferring data, and synchronizing responses, much less dealing with all the pain of lost connections, abnormal termination and so on. The largest excel sheet I've worked on is only about 500 MB and (does a quick search of my local filesystem) almost all are less than one MB. So in the (rare) worst case the transmission doesn't justify spreading it around; in the common case there's no benefit.

handwarmers 2y ago
So what happens when this hypothetical machine of yours, that has enough NIC bandwidth to process the scale of data that needs to be streamed in both directions, enough CPU power to handle millions of concurrent requests in a process or a thread of their own, and enough ram/ fast enough disks to to map and swap all the files that are being converted, goes down? In 2022, Google Workspace apparently had ~ 3 billion users (8 million of which paying) https://developers.googleblog.com/en/year-in-review-12-awesome-ways-for-developers-to-learn-build-and-grow-with-google-workspace/#:~:text=2022%20was%20also%20a%20year,than%205%20billion%20app%20installs https://developers.googleblog.com/en/year-in-review-12-aweso... . Not every solution needs microservices. But also, we have problems today that we did not have solutions for "in the old days".
- cess11 2y ago
  Ten million per day seems unlikely to translate to millions of concurrent requests. This kind of task is typically suited to be made into a single thing, you don't want partial conversions hanging around in 'microservices' if something goes wrong. As for scaling, I'd likely put this in a process definition and run it on the BEAM if I were to make such a product. That way millions of requests per hour can hit my cluster and those that fail somehow will just get cleaned up and the transaction rolled back, the clients get 'sorry, try again' or 'sorry, we're working on fixing it', and the rest happily chug along.
  - handwarmers 2y ago
    Apache Beam?
    - cess11 2y ago
      No, the BEAM: https://en.wikipedia.org/wiki/BEAM_(Erlang_virtual_machine) https://en.wikipedia.org/wiki/BEAM_(Erlang_virtual_machine)
- gumby 2y ago
  You don't have to run them all on a single device! But you hardly need a bunch of microservices, or any, really, to do any one tranlation.

hmeh 2y ago

You may be conflating microservices and horizontal scaling. You don't need to have multiple (disparate) microservices to scale. Microservices have absolutely nothing to do with scaling. That's a myth started by people who never understood the actual point of microservices, which was partitioning and continuity of developer productivity.

camgunz 2y ago

Maybe this'll come off as snarky, but I would build Excel! It would spread 10 million requests across the 10 million users, be totally immune to network outages, and my users could rest assured I wasn't thumbing through their data.