4 ms·

I also did this with GCP preemtiple instances and worked great for a while until I found out one random day that networking issues may also occur in addition to

I also did this with GCP preemtiple instances and worked great for a while until I found out one random day that networking issues may also occur in addition to shutting off your instances within 24 hours. On sandbox clusters thou it’s been very smooth for over half a year. Highly recommend.

transect 7y ago
We use preemptibles for our CI fleet and it's great. We can run a hundred instances at full tilt boogie for 8 hours a day and the nodepool downscales to zero while we sleep. It's a no brainer if your controller (and use case) can handle preemption gracefully.
- __float 7y ago
  What CI software do you use? I played around with spot instances and Jenkins, and it was quite a poor experience.
  - transect 7y ago
    We use Jenkins to invoke Tekton pipelines (https://github.com/tektoncd/pipeline https://github.com/tektoncd/pipeline) with a wrapper we wrote. The pipeline runs, outputs junit to a bucket and we pull it back and give it to jenkins. Was a bit of a lift to get working out of the gate but it's been mostly smooth (and flexible and cheap) since then.
boulos 7y ago
Disclosure: I work for Google Cloud (and on preemptible VMs). There isn’t anything about preemptible to “cause networking issues”. We may have had a general networking outage, if that’s what you experienced, but we don’t additionally make networking worse for preemptible. You’re likely to get shot, but we don’t adjust throttles or anything.
- johnjungles 7y ago
  Thanks for letting me know. Overall, we use them for high availability and non mission critical tasks anyways but now that you say this, it gives me more confidence in recommending it. It was strange, it only affected one of the two GKE cluster (both had preemptive nodes). I tried debugging and found that it wasn’t able to even ping any external ips (which is why I thought maybe preemtible nodes also might have networking issues). There weren’t any notices about general networking outages within the last +\- 48 hours (I subscribe to the RSS GCP outages via slack). I ended up manually rebuilding the nodes (non preemtible nodes temp) and it went away. As soon as healthchecks were failing, I created a debug container and kubectl exec -it bashed in and saw that it wasn’t able to connect to the internet. It could ping/telnet internal IP databases just fine. It does warn me that it’s in beta so, again maybe you’re working through some things but thought you should know in case other people have experienced this.
  - dirtydroog 7y ago
    Did your instances have external IPs? We had connectivity issues when using NAT due to it dropping SYN packets. We just gave everything that needed to talk to external IPs their own external IPs.
    - boulos 7y ago
      Glad you found a workaround, but please file a support ticket for this. If you don’t have support: (a) consider it and/or (b) send me an email with your info (contact in profile) and I’ll file a bug.
      - dirtydroog 7y ago
        We actually did have a dedicated Googler who oversaw our move from AWS to GCP. I asked him about it but never got a reply. Anyway, yes. It caused us problems. Attempts to connect to machines outside of GCP would fail repeatedly due to the NA dropping the SYN. This would mean our connect() call would timeout repeatedly. There was an ICMP response I could see with tcpdump: ICMP host aaa.bbb.ccc.ddd unreachable - admin prohibited filter, length 68

transect 7y ago

We use preemptibles for our CI fleet and it's great. We can run a hundred instances at full tilt boogie for 8 hours a day and the nodepool downscales to zero while we sleep. It's a no brainer if your controller (and use case) can handle preemption gracefully.

__float 7y ago
What CI software do you use? I played around with spot instances and Jenkins, and it was quite a poor experience.
- transect 7y ago
  We use Jenkins to invoke Tekton pipelines (https://github.com/tektoncd/pipeline https://github.com/tektoncd/pipeline) with a wrapper we wrote. The pipeline runs, outputs junit to a bucket and we pull it back and give it to jenkins. Was a bit of a lift to get working out of the gate but it's been mostly smooth (and flexible and cheap) since then.

boulos 7y ago

Disclosure: I work for Google Cloud (and on preemptible VMs). There isn’t anything about preemptible to “cause networking issues”. We may have had a general networking outage, if that’s what you experienced, but we don’t additionally make networking worse for preemptible. You’re likely to get shot, but we don’t adjust throttles or anything.

johnjungles 7y ago
Thanks for letting me know. Overall, we use them for high availability and non mission critical tasks anyways but now that you say this, it gives me more confidence in recommending it. It was strange, it only affected one of the two GKE cluster (both had preemptive nodes). I tried debugging and found that it wasn’t able to even ping any external ips (which is why I thought maybe preemtible nodes also might have networking issues). There weren’t any notices about general networking outages within the last +\- 48 hours (I subscribe to the RSS GCP outages via slack). I ended up manually rebuilding the nodes (non preemtible nodes temp) and it went away. As soon as healthchecks were failing, I created a debug container and kubectl exec -it bashed in and saw that it wasn’t able to connect to the internet. It could ping/telnet internal IP databases just fine. It does warn me that it’s in beta so, again maybe you’re working through some things but thought you should know in case other people have experienced this.
- dirtydroog 7y ago
  Did your instances have external IPs? We had connectivity issues when using NAT due to it dropping SYN packets. We just gave everything that needed to talk to external IPs their own external IPs.
  - boulos 7y ago
    Glad you found a workaround, but please file a support ticket for this. If you don’t have support: (a) consider it and/or (b) send me an email with your info (contact in profile) and I’ll file a bug.
    - dirtydroog 7y ago
      We actually did have a dedicated Googler who oversaw our move from AWS to GCP. I asked him about it but never got a reply. Anyway, yes. It caused us problems. Attempts to connect to machines outside of GCP would fail repeatedly due to the NA dropping the SYN. This would mean our connect() call would timeout repeatedly. There was an ICMP response I could see with tcpdump: ICMP host aaa.bbb.ccc.ddd unreachable - admin prohibited filter, length 68