4 ms·

That's solvable with modern cloud offerings - Provision spot instances for a few minutes and shut them down afterwards. Let the cloud provider deal with demand

That's solvable with modern cloud offerings - Provision spot instances for a few minutes and shut them down afterwards. Let the cloud provider deal with demand balancing. I think the real issue is that developers waiting for PRs to go green are taking a coffee break between tasks, not sitting idly getting annoyed. If that's the case you're cutting into rest time and won't get much value out of optimizing this.

IshKebab 1y ago
Both companies I've worked in recently have been too paranoid about IP to use the cloud for CI. Anyway I don't see how that solves any of the issues except maybe cost to some degree (but maybe not; cloud is expensive).
- fragmede 1y ago
  Sorta. For CI/CD you can use spot instances and spin them down outside of business hours, so they can end up being cheaper than buying many really beefy machines and amortizing them over the standard depreciation schedule.
  - IshKebab 1y ago
    Yeah though not for silicon verification because we always run more tests overnight. They get basically 100% utilisation.
- simonw 1y ago
  Were they running CI on their own physical servers under a desk or in a basement somewhere, or renting their own racks in a data center just for CI?
  - mike_hearn 1y ago
    There are non-IP reasons to go outside the big clouds for CI. Most places I worked over the years had dedicated hardware for at least some CI jobs because otherwise it's too hard to get repeatable performance numbers. At some point you have an outage in production caused by a new build passing tests but having much lower performance, or performance is a feature of the software being sold, and so people decide they need to track perf with repeatable load tests.
  - IshKebab 1y ago
    Data center.
- jiggawatts 1y ago
  That’s paranoid to the point of lunacy. Azure for example has “confidential compute” that encrypts even the memory contents of the VM such that even their own engineers can’t access the contents. As long as you don’t back up the disks and use HTTPS for pulls, I don’t see a realistic business risk. If a cloud like Azure or AWS got caught stealing competitor code they’d be sued and immediately lose a huge chunk of their customers. It makes zero business sense to do so. PS: Microsoft employees have made public comments saying that they refuse to even look at some open source repository to avoid any risk of accidentally “contaminating” their own code with something that has an incompatible license.
  - kccqzy 1y ago
    I don't know about Azure's implementation of confidential compute but GCP's version basically essentially relies on AMD SEV-SVP. Historically there have been vulnerabilities that undermine the confidentiality guarantee.
    - jiggawatts 1y ago
      Mandatory XKCD: https://xkcd.com/538/ https://xkcd.com/538/ Nobody's code is that secret, especially not from a vendor like Microsoft. Unless all development is done with air-gapped machines, realistic development environments are simultaneously exposed to all of the following "leakage risks" because they're using third-party software, almost certainly including a wide range of software from Microsoft: - Package managers, including compromised or malicious packages. Microsoft owns both NuGet and NPM! - IDEs and their plugins, the latter especially can be a security risk. What developer doesn't use Microsoft VS Code these days? - CLI and local build tools. - SCM tools such as GitHub Enterprise (Microsoft again!) - The CI/CD tooling including third-party tools. - The operating system itself. Microsoft Windows is still a very popular platform, especially in enterprise environments. - The OS management tools, anti-virus, monitoring, etc... And on and on. Unless you live in a total bubble world with USB sticks used to ferry your dependencies into your windowless facility underground, your code is "exposed" to third parties all of the time. Worrying about possible vulnerabilities in encrypted VMs in a secure cloud facility is missing the real problem that your developers are probably using their home gaming PC for work because it's 10x faster than the garbage you gave them. Yes, this happens. All the time. You just don't know because you made the perfect the enemy of the good.

IshKebab 1y ago

Both companies I've worked in recently have been too paranoid about IP to use the cloud for CI. Anyway I don't see how that solves any of the issues except maybe cost to some degree (but maybe not; cloud is expensive).

fragmede 1y ago
Sorta. For CI/CD you can use spot instances and spin them down outside of business hours, so they can end up being cheaper than buying many really beefy machines and amortizing them over the standard depreciation schedule.
- IshKebab 1y ago
  Yeah though not for silicon verification because we always run more tests overnight. They get basically 100% utilisation.
simonw 1y ago
Were they running CI on their own physical servers under a desk or in a basement somewhere, or renting their own racks in a data center just for CI?
- mike_hearn 1y ago
  There are non-IP reasons to go outside the big clouds for CI. Most places I worked over the years had dedicated hardware for at least some CI jobs because otherwise it's too hard to get repeatable performance numbers. At some point you have an outage in production caused by a new build passing tests but having much lower performance, or performance is a feature of the software being sold, and so people decide they need to track perf with repeatable load tests.
- IshKebab 1y ago
  Data center.
jiggawatts 1y ago
That’s paranoid to the point of lunacy. Azure for example has “confidential compute” that encrypts even the memory contents of the VM such that even their own engineers can’t access the contents. As long as you don’t back up the disks and use HTTPS for pulls, I don’t see a realistic business risk. If a cloud like Azure or AWS got caught stealing competitor code they’d be sued and immediately lose a huge chunk of their customers. It makes zero business sense to do so. PS: Microsoft employees have made public comments saying that they refuse to even look at some open source repository to avoid any risk of accidentally “contaminating” their own code with something that has an incompatible license.
- kccqzy 1y ago
  I don't know about Azure's implementation of confidential compute but GCP's version basically essentially relies on AMD SEV-SVP. Historically there have been vulnerabilities that undermine the confidentiality guarantee.
  - jiggawatts 1y ago
    Mandatory XKCD: https://xkcd.com/538/ https://xkcd.com/538/ Nobody's code is that secret, especially not from a vendor like Microsoft. Unless all development is done with air-gapped machines, realistic development environments are simultaneously exposed to all of the following "leakage risks" because they're using third-party software, almost certainly including a wide range of software from Microsoft: - Package managers, including compromised or malicious packages. Microsoft owns both NuGet and NPM! - IDEs and their plugins, the latter especially can be a security risk. What developer doesn't use Microsoft VS Code these days? - CLI and local build tools. - SCM tools such as GitHub Enterprise (Microsoft again!) - The CI/CD tooling including third-party tools. - The operating system itself. Microsoft Windows is still a very popular platform, especially in enterprise environments. - The OS management tools, anti-virus, monitoring, etc... And on and on. Unless you live in a total bubble world with USB sticks used to ferry your dependencies into your windowless facility underground, your code is "exposed" to third parties all of the time. Worrying about possible vulnerabilities in encrypted VMs in a secure cloud facility is missing the real problem that your developers are probably using their home gaming PC for work because it's 10x faster than the garbage you gave them. Yes, this happens. All the time. You just don't know because you made the perfect the enemy of the good.