3 ms·
Our work is incredibly i/o-bound, so we would probably not see much of a performance difference no matter what language we chose. Additionally, our presentation
by greggyb 4y ago
Our work is incredibly i/o-bound, so we would probably not see much of a performance difference no matter what language we chose. Additionally, our presentation layer is served directly from an in-memory, columnstore database and is divorced from our back-end by a batch ETL process.
To give a high-level overview of what we are doing, we call a series of APIs which can provide current-state metadata about a SaaS platform. We poll those APIs on a schedule and diff the results over time, providing to our customers a view of this change over time. So our work is to schedule these API calls, wait for the results, persist the results to storage, and then identify the deltas. Once this is done, we load into our presentation layer for customers to interact with.
We run an independent process per customer and use OS permissions to isolate these from one another, and plan to containerize these processes for further isolation. Customers each get a local ZFS dataset and a separate blob storage container. Isolation is a big deal for us, as the permissions we require to call these APIs for our customers end up being a de-facto global admin (this is a limitation of the SaaS vendor, not us). The dataset-per-customer model allows us to lose a single (and backed up, so not really single) encryption keyfile to forget everything sensitive from that customer.
This last paragraph only emphasizes that the language is not the most significant consideration for our work. We could probably orchestrate the API calls as effectively in shell pipelines. The diffing process is a bit hairier, and my current focus.