3 ms·

You're doing whatnow?? This may come off as arrogant, but it sounds like something that could be rewritten in Fortran to run on an old laptop with similar exec

You're doing whatnow?? This may come off as arrogant, but it sounds like something that could be rewritten in Fortran to run on an old laptop with similar execution time.

derefr 5y ago
I think they mean that they may need thousands of HPC cores for a few hundred milliseconds, and then zero cores right after; and (for some reason) absolutely cannot just do the work serially instead of in parallel.
- semi-extrinsic 5y ago
  Yeah, I also interpret it that way, but I cannot understand what kind of system requires such a scheme. Even if the state space is non-ergodic, you should be able to get by with running your sampling on a dozen larger systems in some sense. Or just run on a workstation for a few months to precompute and tabulate the result for all possible inputs.
  - ddoolin 5y ago
    I realize this was a week ago, but the parent comment is basically the case. The "some reason" is the expected time from request->response (which includes aggregation/interpretation of results) is 1-2 seconds with a sample size of ~50,000 for a pretty involved model that needs to be run on demand/in "real time" with no good, predictable time window. We can definitely get by running the sampling on a dozen larger systems, but those systems still cost more than (one thousand Lambda executions * a few hundred times a week max). Pre-computing is an interesting idea, but would certainly be an epic amount of work. The input set is also quite large (compiled motorsports data based on current state of an event). I'm also totally willing to admit I'm probably in over my head in designing such a system, but I did try a few different ways of architecting this and tweaking the parallelization in the source and this was the best cross of time/money I could get! It's definitely been a learning process for me; although I've got some years experience with scaling out, the system and requirements were much different than I've been exposed to. I'm sure this first take will be changed or at least refined once it's been in use for awhile.

You're doing whatnow?? This may come off as arrogant, but it sounds like something that could be rewritten in Fortran to run on an old laptop with similar execution time.

derefr 5y ago

I think they mean that they may need thousands of HPC cores for a few hundred milliseconds, and then zero cores right after; and (for some reason) absolutely cannot just do the work serially instead of in parallel.

semi-extrinsic 5y ago
Yeah, I also interpret it that way, but I cannot understand what kind of system requires such a scheme. Even if the state space is non-ergodic, you should be able to get by with running your sampling on a dozen larger systems in some sense. Or just run on a workstation for a few months to precompute and tabulate the result for all possible inputs.
- ddoolin 5y ago
  I realize this was a week ago, but the parent comment is basically the case. The "some reason" is the expected time from request->response (which includes aggregation/interpretation of results) is 1-2 seconds with a sample size of ~50,000 for a pretty involved model that needs to be run on demand/in "real time" with no good, predictable time window. We can definitely get by running the sampling on a dozen larger systems, but those systems still cost more than (one thousand Lambda executions * a few hundred times a week max). Pre-computing is an interesting idea, but would certainly be an epic amount of work. The input set is also quite large (compiled motorsports data based on current state of an event). I'm also totally willing to admit I'm probably in over my head in designing such a system, but I did try a few different ways of architecting this and tweaking the parallelization in the source and this was the best cross of time/money I could get! It's definitely been a learning process for me; although I've got some years experience with scaling out, the system and requirements were much different than I've been exposed to. I'm sure this first take will be changed or at least refined once it's been in use for awhile.