3 ms·

I think your instincts are correct, I use scaleway for clusters of mediocrity feeding fat clients and to feed cloudflare for vanity sites.. Needing the fastest

I think your instincts are correct, I use scaleway for clusters of mediocrity feeding fat clients and to feed cloudflare for vanity sites.. Needing the fastest cpu, great networks and accepting backup lock-in are examples of laziness in design. Sourcing commodity materials and already having a few nodes in every role is my preferred bet. Being able to upgrade the node performance by 10 times if scaling plans fail wont hurt either..

sqeaky 9y ago
I agree with you , I do disagree with one tiny part. I agree with your entirely on any sort of lock-in, lock-in is bad and constrains future options. Known cost up front for unknown cost in the future should generally be seen as a business risk. The tiny place where I disagree is that some tasks might take unbounded CPU/GPU time and network bandwidth. Your task clearly does not and I agree your workload looks like a common workload. When working for the Air Force Weather Wing, it seemed clear that our decision to use the super computers we used was a budgetary decision and that is what bounded the size of the data we fed into our systems. Because we generated forecasts we had tight time constraints to meet, a 72 hour forecast that takes 100 hours to make is useless. Weather modeling and forecasting is still advancing and even the latest models are not stable simulations (small fluctuations in inputs can cause large fluctuations in output) and this makes the most nuanced data possible desirable. While most commercial systems used 25km polygons covering the earth, ours used 17km polygons when I left. This meant that we had more precise sims and could accurately model further in the future, perhaps by several days. Even faster CPUs and networking might lead to a drop to 8-11km polygons allowing even better predictions. Ideally the meteorologists want to skip the rounding to regional polygons and use the parcels or air model that local forecasters use for the entire world. Presuming Moore's law holds up and weather modeling is amenable to newer GPUs I do not see weather modeling backing away from needing the fastest hardware money can buy in the foreseeable future.

sqeaky 9y ago

I agree with you , I do disagree with one tiny part. I agree with your entirely on any sort of lock-in, lock-in is bad and constrains future options. Known cost up front for unknown cost in the future should generally be seen as a business risk. The tiny place where I disagree is that some tasks might take unbounded CPU/GPU time and network bandwidth. Your task clearly does not and I agree your workload looks like a common workload. When working for the Air Force Weather Wing, it seemed clear that our decision to use the super computers we used was a budgetary decision and that is what bounded the size of the data we fed into our systems. Because we generated forecasts we had tight time constraints to meet, a 72 hour forecast that takes 100 hours to make is useless. Weather modeling and forecasting is still advancing and even the latest models are not stable simulations (small fluctuations in inputs can cause large fluctuations in output) and this makes the most nuanced data possible desirable. While most commercial systems used 25km polygons covering the earth, ours used 17km polygons when I left. This meant that we had more precise sims and could accurately model further in the future, perhaps by several days. Even faster CPUs and networking might lead to a drop to 8-11km polygons allowing even better predictions. Ideally the meteorologists want to skip the rounding to regional polygons and use the parcels or air model that local forecasters use for the entire world. Presuming Moore's law holds up and weather modeling is amenable to newer GPUs I do not see weather modeling backing away from needing the fastest hardware money can buy in the foreseeable future.