3 ms·
I work in a very data heavy HPC space and you are glossing over many things. * to get performant access to a storage cluster is non-trivial, there are many dif
by __turbobrew__ 5y ago
I work in a very data heavy HPC space and you are glossing over many things.
* to get performant access to a storage cluster is non-trivial, there are many different variables in place which must be correctly tuned to get good performance. Network topology, high quality NICs and switches, a tuned Linux kernel, client side caching settings, network packet sizes, file system block sizes, erasure coding settings, etc.
* your solution mentions nothing of backups, offsite failovers, or disaster recovery plans.
* your solution mentions nothing of a physical datacenter: fire suppression, battery backups, hvac, power supplies, backup generators, server racks, sound isolation, workspaces for hardware maintenance, network cable routing.
* If you have multiple geolocations you need to have dark fiber or ip transit between locations with multiple ISPs to have high speed connections between sites without downtime.
In addition to the raw costs you have to factor in the lead time of building a qualified infrastructure team, building out the requirements, provisioning hardware and datacenter space, setting everything up, and then tuning everything. With infinite money this is probably still a 2 year lead time at minimum.
I do agree that running multi petabyte workloads in AWS is probably not optimal, but when you are a startup in the growth stage it is probably better worth your time throwing VC money at AWS and building out your product. Eventually, the business should probably migrate to self managed infrastructure once the right product fit has been found and the business is looking to streamline.