5 ms·

I've never really understood "minutes" as a unit of build work. What kind of server are we talking about? What CPU? How much RAM? How fast is the storage acces

I've never really understood "minutes" as a unit of build work. What kind of server are we talking about? What CPU? How much RAM? How fast is the storage access? Is my instance virtualized? And if so, do I have dedicated resources? I have a build that takes around 70 minutes on an 8-core i9 with 32 GB of RAM and M.2 SSDs. What does that translate into for Gitlab "minutes"?

sushshshsh 6y ago
Probably a lifetime :) Just kidding. Either they went the cheap route and stuck it on some price efficient EC2 instances, or they went the vogue-but-expensive route of lambdas for "rapid processing and ease of development"
MichaelBurge 6y ago
Presumably they're the same machines as before, so anyone affected already knows the translation.
JoshTriplett 6y ago
What would be the most immediately understandable way to present that? Suppose they added new, faster servers in the future; what unit would make the most sense to offer that won't change in the future?
- ivalm 6y ago
  I mean like other public cloud providers it would make sense to have instances and per minute pricing for an instance. If there is only one instance type that's fine.
  - JoshTriplett 6y ago
    What about if there's one instance type, but sometimes that instance gets upgraded so that the same things take less time? Is there a unit that would make more sense than "minutes", and be stable over time? For instance, "time to compile XYZ well-known project"?
    - ivalm 6y ago
      Time to compile is a hard to gauge metric. I’d rather just be transparent that the instance type has changed. It’s presumably in a big standardized DC. They don’t have a continuum of instance configurations, they probably upgrade rarely and systematically. If they are mid upgrade just have 2 instance types available then sunset the older one. Since the upgraded instance is a new instance type it can have new (or same) pricing. In addition, they could publish benchmarks for each instance type if they want. It is literally what we see with cloud providers having v1/v2/v3 names for some instance types.
- WindyLakeReturn 6y ago
  Probably could just include a * that points to the infrastructure it runs on with a link to the update history of that infrastructure. It only becomes an issue when the infrastructure isn't standardized, but that is just exposing the underlying issue where the same build won't take equal time due to a difference in infrastructure.
- nrclark 6y ago
  A generic "build credit" term could be better, maybe - with some details about what one credit gets you. Maybe one build-credit is normalized to "One minute on an 'n1-standard-1' with 4GB of RAM and 40GB of storage." Under a system like that, users could maybe choose between a couple of different worker types. Or if there's only ever the one type, periodically the 'n1-standard-1' could be swapped out for whatever is the latest-greatest for the same price.
mumblerino 6y ago
The point of a service like this is that you should not concern yourself with dirty hardware details. They give you a certain power for a number of minutes. It seems reasonable. If they upgrade their servers, they can fit more minutes per machine, or give it away by speeding up runs for free. If you want to think about “minutes on XYZ machine”, then build your own CI on AWS and pay exactly what you want.
deastman 6y ago
We define Pipeline minutes as the execution time for your pipelines. You bring up an interesting point, though. So today, for our Linux Runners on GitLab.com, those Runners are currently offered only on one machine type, Google Compute n1-standard-1 instances with 3.75GB of RAM. Our current Windows Runners on GitLab.com are Google Compute n1-standard-2 instances with 2 vCPUs and 7.5GB RAM. In the future, for Linux and Windows Runners, we will offer more GCP machine types. For our soon to launch macOS Build Cloud beta, we are planning to start with one virtual machine size and then possibly offer different machine configurations at GA. And yes - the virtual machine used for your build on GitLab.com are dedicated only to your pipeline job and immediately deleted on job completion. Finally, the only way to know how long your current build job will take on a GCP n1-standard-1 compared to the 8-core machine is to run the job and compare the results. I assume that your 8-core machine is probably a physical box, so you should of course, get better performance than a 1-2 vCPU VM. A few reference links: https://docs.gitlab.com/ee/user/gitlab_com/#shared-runners https://docs.gitlab.com/ee/user/gitlab_com/#shared-runners https://gitlab.com/groups/gitlab-org/-/epics/1830 https://gitlab.com/groups/gitlab-org/-/epics/1830 https://gitlab.com/groups/gitlab-org/-/epics/2426 https://gitlab.com/groups/gitlab-org/-/epics/2426 Darren Eastman: Product Manager GitLab Runner
- MayeulC 6y ago
  > In the future, for Linux and Windows Runners, we will offer more GCP machine types. I can see free plan users starting to see their CI jobs randomly timeout if they have access to those machines. I get that execution time isn't a bad metric, all things considered. But I would have expected actual CPU `time` (1), maybe mixed with memory usage.
  - jrochkind1 6y ago
    AWS or GCS charge for VMs in actual clock (wall) time though, right? Tthat your load may spend more or less time waiting on IO instead of actually using the CPU... I would not expect to effect your charge. Which is the main difference between wall time and actual CPU time, right?
muspimerol 6y ago
All you have to do is look, they list the shared runner specs: https://docs.gitlab.com/ee/user/gitlab_com/index.html#shared-runners https://docs.gitlab.com/ee/user/gitlab_com/index.html#shared...
judge2020 6y ago
In case anyone is wondering about GitHub (not GitLab), they're planning to add multiple runner sizes[0]. Unfortunately it was moved from Q4 2020 to 'future' so there's no expected release time. 0: https://github.com/github/roadmap/issues/95 https://github.com/github/roadmap/issues/95
ta95846893 6y ago
Wouldn't Watt hours (or Joules) be a unit that more accurately represents the amount of work being done, and the cost of running the physical server?
hinkley 6y ago
That probably dovetails with the free minutes. If you want to benchmark it before making a commitment, you can.
jrockway 6y ago
That would probably be 280 minutes of "build time" to do your build in 70 minutes. My math is: they are going to call your 8 core CPU 16 vCPUs, because it's something cloud providers can charge money for but Hyperthreading/SMT doesn't really speed up builds that much. It does do something, but 2 threads scheduled on the same core is not going to be 2x faster than one thread being scheduled on that core. (It sure will use 2x the RAM, though, while it sits there waiting to execute instructions. They will charge you for that too!) Then, because they are running on a 64 or 128 core CPU with the same TDP as your consumer chip, things are going to be really clocked down -- your desktop may boost to 5GHz during the build (assuming you got a good cooler and overrode the time limits for boosting; who doesn't, though?), this thing will be running at 2.5GHz. So you will need twice as many actual CPU cores to get the same performance. I am being very pessimistic with these numbers, but I am continually amazed at how slow computers in the cloud are compared to my desktop. And when you're being charged by the minute, there is no incentive to make the computers faster, of course -- the business incentive is to make them slower! Buyer beware. (To be fair, they are getting a lot better Wh/build out of their system than you are. If you were paying for the electricity and cooling and got paid no matter how slow the build was, you'd make the same decision.)

sushshshsh 6y ago

Probably a lifetime :) Just kidding. Either they went the cheap route and stuck it on some price efficient EC2 instances, or they went the vogue-but-expensive route of lambdas for "rapid processing and ease of development"

MichaelBurge 6y ago

Presumably they're the same machines as before, so anyone affected already knows the translation.

JoshTriplett 6y ago

What would be the most immediately understandable way to present that? Suppose they added new, faster servers in the future; what unit would make the most sense to offer that won't change in the future?

ivalm 6y ago
I mean like other public cloud providers it would make sense to have instances and per minute pricing for an instance. If there is only one instance type that's fine.
- JoshTriplett 6y ago
  What about if there's one instance type, but sometimes that instance gets upgraded so that the same things take less time? Is there a unit that would make more sense than "minutes", and be stable over time? For instance, "time to compile XYZ well-known project"?
  - ivalm 6y ago
    Time to compile is a hard to gauge metric. I’d rather just be transparent that the instance type has changed. It’s presumably in a big standardized DC. They don’t have a continuum of instance configurations, they probably upgrade rarely and systematically. If they are mid upgrade just have 2 instance types available then sunset the older one. Since the upgraded instance is a new instance type it can have new (or same) pricing. In addition, they could publish benchmarks for each instance type if they want. It is literally what we see with cloud providers having v1/v2/v3 names for some instance types.
WindyLakeReturn 6y ago
Probably could just include a * that points to the infrastructure it runs on with a link to the update history of that infrastructure. It only becomes an issue when the infrastructure isn't standardized, but that is just exposing the underlying issue where the same build won't take equal time due to a difference in infrastructure.
nrclark 6y ago
A generic "build credit" term could be better, maybe - with some details about what one credit gets you. Maybe one build-credit is normalized to "One minute on an 'n1-standard-1' with 4GB of RAM and 40GB of storage." Under a system like that, users could maybe choose between a couple of different worker types. Or if there's only ever the one type, periodically the 'n1-standard-1' could be swapped out for whatever is the latest-greatest for the same price.

mumblerino 6y ago

The point of a service like this is that you should not concern yourself with dirty hardware details. They give you a certain power for a number of minutes. It seems reasonable. If they upgrade their servers, they can fit more minutes per machine, or give it away by speeding up runs for free. If you want to think about “minutes on XYZ machine”, then build your own CI on AWS and pay exactly what you want.

deastman 6y ago

We define Pipeline minutes as the execution time for your pipelines. You bring up an interesting point, though. So today, for our Linux Runners on GitLab.com, those Runners are currently offered only on one machine type, Google Compute n1-standard-1 instances with 3.75GB of RAM. Our current Windows Runners on GitLab.com are Google Compute n1-standard-2 instances with 2 vCPUs and 7.5GB RAM. In the future, for Linux and Windows Runners, we will offer more GCP machine types. For our soon to launch macOS Build Cloud beta, we are planning to start with one virtual machine size and then possibly offer different machine configurations at GA. And yes - the virtual machine used for your build on GitLab.com are dedicated only to your pipeline job and immediately deleted on job completion. Finally, the only way to know how long your current build job will take on a GCP n1-standard-1 compared to the 8-core machine is to run the job and compare the results. I assume that your 8-core machine is probably a physical box, so you should of course, get better performance than a 1-2 vCPU VM. A few reference links: https://docs.gitlab.com/ee/user/gitlab_com/#shared-runners https://docs.gitlab.com/ee/user/gitlab_com/#shared-runners https://gitlab.com/groups/gitlab-org/-/epics/1830 https://gitlab.com/groups/gitlab-org/-/epics/1830 https://gitlab.com/groups/gitlab-org/-/epics/2426 https://gitlab.com/groups/gitlab-org/-/epics/2426 Darren Eastman: Product Manager GitLab Runner

MayeulC 6y ago
> In the future, for Linux and Windows Runners, we will offer more GCP machine types. I can see free plan users starting to see their CI jobs randomly timeout if they have access to those machines. I get that execution time isn't a bad metric, all things considered. But I would have expected actual CPU `time` (1), maybe mixed with memory usage.
- jrochkind1 6y ago
  AWS or GCS charge for VMs in actual clock (wall) time though, right? Tthat your load may spend more or less time waiting on IO instead of actually using the CPU... I would not expect to effect your charge. Which is the main difference between wall time and actual CPU time, right?

muspimerol 6y ago

All you have to do is look, they list the shared runner specs: https://docs.gitlab.com/ee/user/gitlab_com/index.html#shared-runners https://docs.gitlab.com/ee/user/gitlab_com/index.html#shared...

judge2020 6y ago

In case anyone is wondering about GitHub (not GitLab), they're planning to add multiple runner sizes[0]. Unfortunately it was moved from Q4 2020 to 'future' so there's no expected release time. 0: https://github.com/github/roadmap/issues/95 https://github.com/github/roadmap/issues/95

ta95846893 6y ago

Wouldn't Watt hours (or Joules) be a unit that more accurately represents the amount of work being done, and the cost of running the physical server?

hinkley 6y ago

That probably dovetails with the free minutes. If you want to benchmark it before making a commitment, you can.

jrockway 6y ago

That would probably be 280 minutes of "build time" to do your build in 70 minutes. My math is: they are going to call your 8 core CPU 16 vCPUs, because it's something cloud providers can charge money for but Hyperthreading/SMT doesn't really speed up builds that much. It does do something, but 2 threads scheduled on the same core is not going to be 2x faster than one thread being scheduled on that core. (It sure will use 2x the RAM, though, while it sits there waiting to execute instructions. They will charge you for that too!) Then, because they are running on a 64 or 128 core CPU with the same TDP as your consumer chip, things are going to be really clocked down -- your desktop may boost to 5GHz during the build (assuming you got a good cooler and overrode the time limits for boosting; who doesn't, though?), this thing will be running at 2.5GHz. So you will need twice as many actual CPU cores to get the same performance. I am being very pessimistic with these numbers, but I am continually amazed at how slow computers in the cloud are compared to my desktop. And when you're being charged by the minute, there is no incentive to make the computers faster, of course -- the business incentive is to make them slower! Buyer beware. (To be fair, they are getting a lot better Wh/build out of their system than you are. If you were paying for the electricity and cooling and got paid no matter how slow the build was, you'd make the same decision.)