4 ms·

I know a lot of places tend to use things like AWS spot instances for their CI runners but they obviously provide inconsistent performance. As others have noted

I know a lot of places tend to use things like AWS spot instances for their CI runners but they obviously provide inconsistent performance. As others have noted you can always measure performance through other metrics like certain counters, not just the absolute runtime duration. I can recommend getting your company a dedicated instance from OVH or Hetzner since they are dirt-cheap compared to cloud offerings. Setup some simple runner containers with properly constrained CPU and memory that would be similar to production environment, hook them up to your Gitlab or Github and you are good to go. You don't really need high availability for development things like CI runners.

Waterluvian 3y ago
I’ve wondered about something like that. Is it possible to get the number of cpu operations some test takes? I get with modern CPUs it won’t be as exact as when I was counting 8080 operations, but it would surely tell you something. Or at higher level abstractions, things like how many Python byte codes.
- c0balt 3y ago
  Yesn't, you can by, e.g., using qemu[0]. This however has the obvious downside of not accounting for pipelining and cache as well as the subset of instructions encoded in microcode vs. hardware. All of those are very important but CPU (microarchitecture, cache) specific. [0]: For example: https://www.qemu.org/docs/master/devel/replay.html#instruction-counting https://www.qemu.org/docs/master/devel/replay.html#instructi... more can be found searching the docs.
- choroid 3y ago
  Iai^1 uses CacheGrind^2 to count instructions [1] - https://github.com/bheisler/iai https://github.com/bheisler/iai [2] - https://valgrind.org/docs/manual/cg-manual.html https://valgrind.org/docs/manual/cg-manual.html
- tomsmeding 3y ago
  On Intel CPUs yes you can, using perf. See https://www.brendangregg.com/perf.html https://www.brendangregg.com/perf.html ; getting started: perf stat command
boulos 3y ago
> but they obviously provide inconsistent performance No. Unless you're using a "burstable" (overcommitted) instance family, this shouldn't be the case. The performance will be consistent, but you may get preempted before you finish your work. I would keep those concepts separate.
- phamilton 3y ago
  That's true, however a lot of spot usage ends up being heavily diversified across instance types in order to avoid momentary supply issues and optimize cost. Across different families, CPU performance can vary by a decent amount.
  - boulos 3y ago
    Sure, but you should still keep the concepts disjoint. "I took whatever I could get" isn't the same as "spot instances are inconsistent".
Aurornis 3y ago
CI/CD runners are a great example of something that doesn’t require a dedicated 24/7 server. Spot instances are great for this purpose unless you’re looking for ultra-stable benchmarking performance. If you do need precise performance measurements for customer experience, it’s usually better to run the code on the same hardware you’d be using in production anyway. For client workloads this generally involves getting the actual hardware your customers would be using: Laptops, Macs, and other things you’re not getting from cloud providers. Also, the performance differences of cloud instances aren’t so significant that it warrants going out of your way to set up something dedicated. If you need more performance, just get larger spot instances.

Waterluvian 3y ago

I’ve wondered about something like that. Is it possible to get the number of cpu operations some test takes? I get with modern CPUs it won’t be as exact as when I was counting 8080 operations, but it would surely tell you something. Or at higher level abstractions, things like how many Python byte codes.

c0balt 3y ago
Yesn't, you can by, e.g., using qemu[0]. This however has the obvious downside of not accounting for pipelining and cache as well as the subset of instructions encoded in microcode vs. hardware. All of those are very important but CPU (microarchitecture, cache) specific. [0]: For example: https://www.qemu.org/docs/master/devel/replay.html#instruction-counting https://www.qemu.org/docs/master/devel/replay.html#instructi... more can be found searching the docs.
choroid 3y ago
Iai^1 uses CacheGrind^2 to count instructions [1] - https://github.com/bheisler/iai https://github.com/bheisler/iai [2] - https://valgrind.org/docs/manual/cg-manual.html https://valgrind.org/docs/manual/cg-manual.html
tomsmeding 3y ago
On Intel CPUs yes you can, using perf. See https://www.brendangregg.com/perf.html https://www.brendangregg.com/perf.html ; getting started: perf stat command

boulos 3y ago

> but they obviously provide inconsistent performance No. Unless you're using a "burstable" (overcommitted) instance family, this shouldn't be the case. The performance will be consistent, but you may get preempted before you finish your work. I would keep those concepts separate.

phamilton 3y ago
That's true, however a lot of spot usage ends up being heavily diversified across instance types in order to avoid momentary supply issues and optimize cost. Across different families, CPU performance can vary by a decent amount.
- boulos 3y ago
  Sure, but you should still keep the concepts disjoint. "I took whatever I could get" isn't the same as "spot instances are inconsistent".

Aurornis 3y ago

CI/CD runners are a great example of something that doesn’t require a dedicated 24/7 server. Spot instances are great for this purpose unless you’re looking for ultra-stable benchmarking performance. If you do need precise performance measurements for customer experience, it’s usually better to run the code on the same hardware you’d be using in production anyway. For client workloads this generally involves getting the actual hardware your customers would be using: Laptops, Macs, and other things you’re not getting from cloud providers. Also, the performance differences of cloud instances aren’t so significant that it warrants going out of your way to set up something dedicated. If you need more performance, just get larger spot instances.