18 ms·
We use Kubernetes and spot instances to reduce EC2 billing up to 80%
- johnjungles 7y agoI also did this with GCP preemtiple instances and worked great for a while until I found out one random day that networking issues may also occur in addition to shutting off your instances within 24 hours. On sandbox clusters thou it’s been very smooth for over half a year. Highly recommend.
- transect 7y agoWe use preemptibles for our CI fleet and it's great. We can run a hundred instances at full tilt boogie for 8 hours a day and the nodepool downscales to zero while we sleep. It's a no brainer if your controller (and use case) can handle preemption gracefully.
- __float 7y agoWhat CI software do you use? I played around with spot instances and Jenkins, and it was quite a poor experience.
- transect 7y agoWe use Jenkins to invoke Tekton pipelines (https://github.com/tektoncd/pipeline https://github.com/tektoncd/pipeline) with a wrapper we wrote. The pipeline runs, outputs junit to a bucket and we pull it back and give it to jenkins. Was a bit of a lift to get working out of the gate but it's been mostly smooth (and flexible and cheap) since then.
- boulos 7y agoDisclosure: I work for Google Cloud (and on preemptible VMs). There isn’t anything about preemptible to “cause networking issues”. We may have had a general networking outage, if that’s what you experienced, but we don’t additionally make networking worse for preemptible. You’re likely to get shot, but we don’t adjust throttles or anything.
- johnjungles 7y agoThanks for letting me know. Overall, we use them for high availability and non mission critical tasks anyways but now that you say this, it gives me more confidence in recommending it. It was strange, it only affected one of the two GKE cluster (both had preemptive nodes). I tried debugging and found that it wasn’t able to even ping any external ips (which is why I thought maybe preemtible nodes also might have networking issues). There weren’t any notices about general networking outages within the last +\- 48 hours (I subscribe to the RSS GCP outages via slack). I ended up manually rebuilding the nodes (non preemtible nodes temp) and it went away. As soon as healthchecks were failing, I created a debug container and kubectl exec -it bashed in and saw that it wasn’t able to connect to the internet. It could ping/telnet internal IP databases just fine. It does warn me that it’s in beta so, again maybe you’re working through some things but thought you should know in case other people have experienced this.
- dirtydroog 7y agoDid your instances have external IPs? We had connectivity issues when using NAT due to it dropping SYN packets. We just gave everything that needed to talk to external IPs their own external IPs.
- boulos 7y agoGlad you found a workaround, but please file a support ticket for this. If you don’t have support: (a) consider it and/or (b) send me an email with your info (contact in profile) and I’ll file a bug.
- dirtydroog 7y agoWe actually did have a dedicated Googler who oversaw our move from AWS to GCP. I asked him about it but never got a reply. Anyway, yes. It caused us problems. Attempts to connect to machines outside of GCP would fail repeatedly due to the NA dropping the SYN. This would mean our connect() call would timeout repeatedly. There was an ICMP response I could see with tcpdump: ICMP host aaa.bbb.ccc.ddd unreachable - admin prohibited filter, length 68
- PudgePacket 7y agoThis is quite a neat strategy, leveraging elastic compute costs and kubernetes "self-healing". I'm surprised I haven't heard more about this kind of technique before. I fully acknowledge this will only work in certain scenarios and for certain workloads, eg not ideal for long running/cache/database style services.
- toomuchtodo 7y agoThis is run of the mill for job schedulers (Nomad, Mesos/Marathon, Globus), it’s just more accessible with k8s, containers, and VM spot pricing than historically. As you mention, definitely don’t do this where persistence is paramount (cache that is expensive to backfill upon recovery, database, etc) but it’s just fine for transient workloads or workloads you can rapidly and safely preempt and resume.
- 3fe9a03ccd14ca5 7y agoIf your instance is stateless, and your app can easily self-heal, there’s lots of computing paradigms you can explore. Serverless/function is also an option. Of course, at the end of the day rarely is something ever truly stateless.
- deleted 7y ago[deleted]
- halbritt 7y agoEh, I'm currently running an old version of DCOS which does a pretty terrible job of scaling down. Much more than just a scheduler is required to make this work well.
- ec109685 7y agoYeah, your fleet can become fragmented.
- deleted 7y ago[deleted]
- tra3 7y agoI’m still trying to decide on an orechestrator for my deployments. Any reason swarm wouldn’t work instead of k8s?
- sandGorgon 7y agoWe are moving from swarm to k3s.
- tuananh 7y agoif i were you, i would watch k3s closely. it's super promising; i may not want to dpeloy anything production critical there.
- tra3 7y agoWhy is that? I thought it was a verified distribution?
- tuananh 7y agoit's certified yes. but for me, community is a crucial point and i think it's not big enough yet. maybe that's just me.
- sandGorgon 7y agoI think it has one of the best communities. And especially the Rancher team are extremely responsive on twitter
- sk0g 7y agoDocker is pretty much abandoning Swarm though, so wouldn't recommend that at least. https://www.bretfisher.com/is-swarm-dead-answered-by-a-docker-captain/ https://www.bretfisher.com/is-swarm-dead-answered-by-a-docke...
- tra3 7y ago
- threeseed 7y agoI hope people don't go and take this advice and just run everything on Spot as that is a mistake. It is very common for AWS to completely run out of entire classes of instance types e.g. all of R5 or all of M5. And when that happens your cluster will die. What you want to do is split your cluster into minimum two node groups e.g. Core and Task: Core: On-Demand for all of your critical and management apps e.g. monitoring. Task: Spot for your random, ephemeral jobs that aren't a big deal if it needs to be re-ran. So for a Spark cluster for example you would pin your driver to the Core nodes and let the executors run on the Task nodes.
- tuananh 7y agoauthor here: this is why i suggest to use mix of reserved instances and spot instances. worse case scenario, everything will be on the small number of reserved + on-demand instances. there are other strategies to avoid this as well using multiple type of instance, of different sizes, on different availability zones. because price spike is different for each combinations using bigger size instance than usual because bigger = more stable / less likely to get evicted
- ryeguy 7y agoWhy not use spot fleets to solve this? You can specify multiple acceptable instance types.
- tuananh 7y agobecause at the time when we did this (2016-2017), kops did not support spot fleet
- halbritt 7y agoI'm also puzzled why people wouldn't just use spot fleet with on-demand as backup.
- tuananh 7y agobecause at the time when we did this (2016-2017), kops did not support spot fleet. worst case scenario, you get mix of reserved + ondemand instances.
- khc 7y agoThis is pretty common, databricks for example uses regular instances for the driver and spot instances for workers by default.
- juliansimioni 7y agoOn the one hand, building fault tolerant infrastructure that can, as a side effect, work painlessly on spot instances is great. On the other hand, you can purchase reserved instances and get ~60% cost savings with zero engineering work. Its worth thinking long and hard about whether the cost of engineering time is worth that next 20%. There's also a lot of useful ground in between "critical state, must never be lost (like a database)" and "can handle being terminated with 2 minutes notice". A service that can be re-created if necessary but takes 10 minutes to start up is really scary if run on spot instances, but can still be pretty useful.
- NikolaeVarius 7y agoNote, AWS savings plans make this even easier https://aws.amazon.com/savingsplans/ https://aws.amazon.com/savingsplans/
- ckdarby 7y ago^ Came here to say the above. Majority of the time the Saving Plan is actually what you want. Unless you have legacy app you know must be around for 3 years and have zero efforts to try to refactor.
- 40acres 7y agoAny idea of when this product was introduced?
- NikolaeVarius 7y agoEnd of 2019 https://aws.amazon.com/blogs/aws-cost-management/reinvent-round-up-savings-plans-cost-categories-and-more/ https://aws.amazon.com/blogs/aws-cost-management/reinvent-ro...
- 40acres 7y agoSo is this blog post redundant?
- tuananh 7y agoauthor here: please note that this is what we did in 2016-2017 where kops(what we use to provision) did not support spot fleet yet. also, this works out so well for our use case because we were using .net framework at the time so the cost saving was huge. a lot has changes since then. Also, this strategy is not limited to AWS, the similiar type of instances are also available on Azure, GCP, etc...
- saym 7y ago.NET core has been a godsend in making .NET an interoperable option for cloud architectures. I'm continually impressed with the open source embrace from Microsoft.
- prchaudhari_007 7y agoWe are using a similar strategy. We have 4 different node groups 2 with on demand instance types 2 with spot instance types It has been working really well for us. We are doing some further optimisations to reduce the costs and improve the performance of our stack. Also, we have jenkins scheduled jobs which kills the staging infra every evening and starts it up in morning. We also keep the staging completely down over the weekends.
- Thristle 7y agoBtw, spotinst (we, I work there) released Ocean in 2018/2019 which is the K8S equivalent of our EC2 solution (elaatigroup) and Eco which is our AWS reserved recommendation product. I won't start a whole marketing speak but I'll just say that spotinst moved away from just a "cost saving solution" and are now a more rounded cloud managememt solution (ease of use, cost monitoring and insights).
- tuananh 7y agoHi, The whole idea was from spotinst blog. Thanks a lot! I just glue all the opensource projects together with some changes here and there. If the idea didn't work, I will def consider using Spotinst. However, every cost saving is important for our startup back then. we were a small shop in Southeast Asia where senior engineer merely cost $1000 a month. I was thinking maybe I can save the cut from Spotinst too :)
- Thristle 7y agoNow that is the cut-throat cost saving strategy we like to see! :)
- jijji 7y agoI've found that rewriting an application from C# to something quicker like PHP or Go negates the whole need to do this kind of stuff because the C# application is 10x faster written in PHP or 20x faster written in Go. I love it when you change one line of a shared lib in C# and it takes an hour for the application to recompile.
- aliswe 7y agoAn hour? If you use NuGet you should be able to keep your compile times down. In my own projects I have maybe 10s compile times.
- seabrookmx 7y ago10x faster in PHP you say? I'd love to see such a benchmark!
- hinkley 7y agoGP seems to be placing a high premium on developer productivity. Given salaries in the valley these days, that’s probably not wrong. PHP, though...
- seabrookmx 7y ago> the C# application is 10x faster written in PHP My interpretation of this is that the parent commenter thinks the application runtime is faster in PHP than C#.. Modern (dotnet core) C# often beats the pants off go in webserver workloads (Golang likely has the edge in compute and certainly in anything that values startup time). I really don't see a universe where idiomatic PHP beats out idiomatic C# though. > I love it when you change one line of a shared lib in C# and it takes an hour for the application to recompile Here they switch gears to developer productivity. My employer has an extremely large (LoC in the millions) C# codebase that does a from-scratch compile (including C++ dependencies) in ~30minutes. On developer environments, this is incremental and while not as fast as we'd like, it only takes a handful of seconds per iteration if you're actively developing on it. If you aren't familiar with C# tooling or really value a REPL, other stacks _will_ likely be more productive. But like you're saying, at that point why PHP? Why not reach for Ruby (is that still hip?) or node or something?
- geuis 7y agoSigh. I had a very long and detailed reply typed out on my phone about the trevails of dealing with kubernetes in the last 2 weeks. Then Safari decides to reload the page and it all got lost. I’m literally emotionally drained after unsuccessfully working with k8s after 2+ weeks. It’s incredibly over complicated and documentation is all over the place. I had a large write up of my experiences but those are lost and I don’t have the energy to retype all of that. I simply wanted to utilize k8s to help provide some auto scaling and redundancies for a 10 year old service I run. After 2 weeks of deep diving on this topic and getting essentially nowhere, even with the help of a friend that does this for his day job and him waving his hands not being able to help, I’m reluctantly done. The technology is just not ready. It’s too complicated. The documentation isn’t sufficient. Sure you can document every nut and bolt, but if you can’t create simple patterns for people to follow you lose. There’s too much change going on between versions. At my last 2 companies, they each had a team of 2-10 people working on implementing kubernetes. After over a year at each company, no significant progress had been made on k8s. Sure some stuff was migrated over but no significant services were running with it.
- stewartm 7y agoYou are definitely right that it is a fast moving target and hence can be frustrating to work with at the moment, particularly if you are trying to get it running on-prem. It is still relatively early days, and there is plenty of distillation to come, before an easy predictable set of patterns emerge. Not wanting you to go to the pain of trying to recreate your original post, but of interest, what kinda of things were the primary areas of pain from your work?
- cbushko 7y agoI think this depends on where you run kubernetes and how you set it up. We moved from AWS + ECS to GCP + kubernetes. We use and use terraform to set it all up so that consistency in tooling/patterns helps. We had one full time senior person doing the move (me) and 6 other coming in an out of the project through the year as their capacity allowed. One of my clusters is running 5000+ containers with ease. Not huge by other company standards but big enough.
- z3t4 7y agoWhat kind of services are you guys running that require you to scale up and down? Why not get one or two dedicated server(s) and run everything on them!? The post had no numbers, but I'm pretty sure you would come off even cheaper if you use dedicated servers, even managed.
- tpxl 7y agoI'm running a workload that takes about 3 minutes to run per request (compute heavy MVP), which means that a big surge of users once a day at peak would require a LOT of dedicated servers to serve in time. My plan is to use dedicated servers for most of the load and some elastic capacity at peak loads if necessary.
- ramraj07 7y agoI have a similar use case in mind, TBF I plan to just keep a small EC2 instance up for this need, and assemble a nice PC at home to catch up with the queue from my home for heavy workloads, the cost is so much cheaper and I get a PC as well! Worst case I spin up one more worker with more specs if the queue gets long. Sounds less effort than doing all this scaling work for a MVP and counterproductive when I'd rather spend more time on my actual logic
- tpxl 7y agoThat's my initial plan as well, until the requirements get too high for my PC :)
- collyw 7y agoAnd how much developer time did it cost? We have done the same - our bills went down, but not by as much as 80% I think closer to 50%. But it took a fair bit of developer time, and we now have a lot of Kubernetes related problems to deal with. I guess those will smooth out over time, but I don't think anyone ever factors in this stuff when they claim great savings. Developer time ain't cheap. On a plus note, running multiple small boxes via Kubernetes does give you a more high availability system. If one instance goes down, there will still be another one available, so it's not all negative.
- satanspastaroll 7y agoI think most people over-estimate the need for availability anyway. It's tempting to build the very best you can, when in reality a business really needs a good-enough solution. A whole ton of software can be down every weekend if needed. A great in-between is to simply have a backup server ready to go in a few minutes time. Super simple compared to orchestrated container system. Of course for client projects that spec out a certain number of 9's must be done just so, but can also be billed accordingly.
- merb 7y agowhat were the challenges or what challenges do you have? currently my company runs on GKE and auto scheduling, even from "spot" instances (called preemptible on gke) is a breeze. we basically only run jobs on the spot instances that don't need to run instantenious, so it's really cheap for us.
- collyw 7y agoLarge uploads and downloads filling up memory and crashing pods. When we had one large machine it didn't have this problem.
- ramraj07 7y agoGoes back to the fact that stack overflow itself runs on a single beefed up machine for all their traffic (with a backup machine of course). What this company does that needs so many instances? And they use the same tech too (.NET). Instead of thinking about that people always think about over engineering for "scale" to overcompensate bad code.
- coolfreights 7y agohttps://coolfreights.com/wp-json/oembed/1.0/embed?url=https://coolfreights.com/ https://coolfreights.com/wp-json/oembed/1.0/embed?url=https:... https://coolfreights.com/wp-json/oembed/1.0/embed?url=https://coolfreights.com/&format=xml https://coolfreights.com/wp-json/oembed/1.0/embed?url=https:... https://coolfreights.com/ https://coolfreights.com/ https://www.googletagmanager.com/ns.html?id=GTM-MKCPSDV https://www.googletagmanager.com/ns.html?id=GTM-MKCPSDV https://coolfreights.com/#content https://coolfreights.com/#content https://coolfreights.com/ https://coolfreights.com/ https://coolfreights.com/refrigerated-truck-dubai-abu-dhabi/ https://coolfreights.com/refrigerated-truck-dubai-abu-dhabi/ https://coolfreights.com/freezer-truck-for-rent-in-dubai/ https://coolfreights.com/freezer-truck-for-rent-in-dubai/ https://coolfreights.com/chiller-truck-for-rent-in-dubai/ https://coolfreights.com/chiller-truck-for-rent-in-dubai/ https://coolfreights.com/food-truck-rental/ https://coolfreights.com/food-truck-rental/ https://coolfreights.com/reefer-truck-rental/ https://coolfreights.com/reefer-truck-rental/ https://coolfreights.com/refrigerator-vans-in-dubai-high-quality/ https://coolfreights.com/refrigerator-vans-in-dubai-high-qua... https://coolfreights.com/freezer-van-rental-transport-in-dubai-abu-dhabi-sharjah-ajman/ https://coolfreights.com/freezer-van-rental-transport-in-dub... https://coolfreights.com/chiller-vans-dubai-abu-dhabi/ https://coolfreights.com/chiller-vans-dubai-abu-dhabi/ https://coolfreights.com/high-roof-van-for-rent-dubai/ https://coolfreights.com/high-roof-van-for-rent-dubai/ https://coolfreights.com/blog/ https://coolfreights.com/blog/ https://coolfreights.com/about-us/ https://coolfreights.com/about-us/ https://coolfreights.com/sevices/ https://coolfreights.com/sevices/ https://coolfreights.com/delivery-product-gallery/ https://coolfreights.com/delivery-product-gallery/ https://coolfreights.com/team/ https://coolfreights.com/team/ https://coolfreights.com/coolfreights-certified-from-dubai-uae-business/ https://coolfreights.com/coolfreights-certified-from-dubai-u... https://coolfreights.com/listings/ https://coolfreights.com/listings/ https://coolfreights.com/contacts/ https://coolfreights.com/contacts/ https://coolfreights.com/ https://coolfreights.com/ https://coolfreights.com/terms-and-conditions/ https://coolfreights.com/terms-and-conditions/ https://coolfreights.com/sitemap/ https://coolfreights.com/sitemap/ https://coolfreights.com/faqs/ https://coolfreights.com/faqs/ https://coolfreights.com/helps/ https://coolfreights.com/helps/ https://coolfreights.com/hire-chiller-van-for-valentine-flowers-delivery-in-dubai/ https://coolfreights.com/hire-chiller-van-for-valentine-flow... https://coolfreights.com/12-helpful-tips-for-doing-toyota-hiace-van/ https://coolfreights.com/12-helpful-tips-for-doing-toyota-hi... https://coolfreights.com/commercial-toyota-hiace-chiller-van-for-rent-in-dubai/ https://coolfreights.com/commercial-toyota-hiace-chiller-van... https://www.facebook.com/coolfreights/ https://www.facebook.com/coolfreights/ http://twitter.com/Cool_Freights http://twitter.com/Cool_Freights https://www.linkedin.com/in/coolfreights https://www.linkedin.com/in/coolfreights https://www.youtube.com/channel/UC5bQgcogTK-0I25JB42TcYA https://www.youtube.com/channel/UC5bQgcogTK-0I25JB42TcYA https://www.pinterest.com/coolfreights/ https://www.pinterest.com/coolfreights/ https://www.instagram.com/cool_freights/ https://www.instagram.com/cool_freights/ https://coolfreights.com/ https://coolfreights.com/ https://coolfreights.com/terms-and-conditions/ https://coolfreights.com/terms-and-conditions/ https://coolfreights.com/sitemap/ https://coolfreights.com/sitemap/ https://coolfreights.com/faqs/ https://coolfreights.com/faqs/ https://coolfreights.com/helps/ https://coolfreights.com/helps/ https://coolfreights.com/ https://coolfreights.com/ https://coolfreights.com/wp-content/uploads/2017/01/cropped-coolfreights-logo-site-s-24.png https://coolfreights.com/wp-content/uploads/2017/01/cropped-... tel:971508979457 mailto:info@coolfreights.com https://coolfreights.com/ https://coolfreights.com/ https://coolfreights.com/refrigerated-truck-dubai-abu-dhabi/ https://coolfreights.com/refrigerated-truck-dubai-abu-dhabi/ https://coolfreights.com/freezer-truck-for-rent-in-dubai/ https://coolfreights.com/freezer-truck-for-rent-in-dubai/ https://coolfreights.com/chiller-truck-for-rent-in-dubai/ https://coolfreights.com/chiller-truck-for-rent-in-dubai/ https://coolfreights.com/food-truck-rental/ https://coolfreights.com/food-truck-rental/ https://coolfreights.com/reefer-truck-rental/ https://coolfreights.com/reefer-truck-rental/ https://coolfreights.com/refrigerator-vans-in-dubai-high-quality/ https://coolfreights.com/refrigerator-vans-in-dubai-high-qua... https://coolfreights.com/freezer-van-rental-transport-in-dubai-abu-dhabi-sharjah-ajman/ https://coolfreights.com/freezer-van-rental-transport-in-dub... https://coolfreights.com/chiller-vans-dubai-abu-dhabi/ https://coolfreights.com/chiller-vans-dubai-abu-dhabi/ https://coolfreights.com/high-roof-van-for-rent-dubai/ https://coolfreights.com/high-roof-van-for-rent-dubai/ https://coolfreights.com/blog/ https://coolfreights.com/blog/ https://coolfreights.com/about-us/ https://coolfreights.com/about-us/ https://coolfreights.com/sevices/ https://coolfreights.com/sevices/ https://coolfreights.com/delivery-product-gallery/ https://coolfreights.com/delivery-product-gallery/ https://coolfreights.com/team/ https://coolfreights.com/team/ https://coolfreights.com/coolfreights-certified-from-dubai-uae-business/ https://coolfreights.com/coolfreights-certified-from-dubai-u... https://coolfreights.com/listings/ https://coolfreights.com/listings/ https://coolfreights.com/contacts/ https://coolfreights.com/contacts/ https://coolfreights.com/refrigerated-truck-dubai-abu-dhabi/ https://coolfreights.com/refrigerated-truck-dubai-abu-dhabi/ https://coolfreights.com/wp-content/uploads/2019/07/buildings-1839726_960_720.jpg https://coolfreights.com/wp-content/uploads/2019/07/building... https://coolfreights.com/chiller-vans-dubai-abu-dhabi/ https://coolfreights.com/chiller-vans-dubai-abu-dhabi/ https://coolfreights.com/wp-content/uploads/2019/07/dubai-1915297_960_720.jpg https://coolfreights.com/wp-content/uploads/2019/07/dubai-19... https://coolfreights.com/hire-chiller-van-for-valentine-flowers-delivery-in-dubai/ https://coolfreights.com/hire-chiller-van-for-valentine-flow... https://coolfreights.com/wp-content/uploads/2020/02/valintine-1.jpg https://coolfreights.com/wp-content/uploads/2020/02/valintin... https://coolfreights.com/hire-chiller-van-for-valentine-flowers-delivery-in-dubai/ https://coolfreights.com/hire-chiller-van-for-valentine-flow... https://coolfreights.com/hire-chiller-van-for-valentine-flowers-delivery-in-dubai/ https://coolfreights.com/hire-chiller-van-for-valentine-flow... https://coolfreights.com/12-helpful-tips-for-doing-toyota-hiace-van/ https://coolfreights.com/12-helpful-tips-for-doing-toyota-hi... https://coolfreights.com/wp-content/uploads/2019/12/DSC0330-798x466-1-400x234.jpg https://coolfreights.com/wp-content/uploads/2019/12/DSC0330-... https://coolfreights.com/12-helpful-tips-for-doing-toyota-hiace-van/ https://coolfreights.com/12-helpful-tips-for-doing-toyota-hi... https://coolfreights.com/12-helpful-tips-for-doing-toyota-hiace-van/ https://coolfreights.com/12-helpful-tips-for-doing-toyota-hi... https://coolfreights.com/hire-chiller-van-for-valentine-flowers-delivery-in-dubai/ https://coolfreights.com/hire-chiller-van-for-valentine-flow... https://coolfreights.com/12-helpful-tips-for-doing-toyota-hiace-van/ https://coolfreights.com/12-helpful-tips-for-doing-toyota-hi... https://coolfreights.com/commercial-toyota-hiace-chiller-van-for-rent-in-dubai/ https://coolfreights.com/commercial-toyota-hiace-chiller-van... https://coolfreights.com/ https://coolfreights.com/ https://coolfreights.com/ https://coolfreights.com/ https://coolfreights.com/hire-rentals-freezer-truck-vans-with-30-off-in-dubai/ https://coolfreights.com/hire-rentals-freezer-truck-vans-wit... https://coolfreights.com/wp-content/uploads/2019/03/hjgjh-400x267.jpg https://coolfreights.com/wp-content/uploads/2019/03/hjgjh-40... https://coolfreights.com/hire-rentals-freezer-truck-vans-with-30-off-in-dubai/ https://coolfreights.com/hire-rentals-freezer-truck-vans-wit... https://coolfreights.com/hire-rentals-freezer-truck-vans-with-30-off-in-dubai/ https://coolfreights.com/hire-rentals-freezer-truck-vans-wit... https://coolfreights.com/hire-rentals-freezer-truck-vans-with-30-off-in-dubai/ https://coolfreights.com/hire-rentals-freezer-truck-vans-wit... https://coolfreights.com/news-of-refrigerated-truck-dubai/ https://coolfreights.com/news-of-refrigerated-truck-dubai/ https://coolfreights.com/hire-rentals-freezer-truck-vans-with-30-off-in-dubai/#respond https://coolfreights.com/hire-rentals-freezer-truck-vans-wit... https://coolfreights.com/hire-rentals-chiller-van-truck-with-30-off/ https://coolfreights.com/hire-rentals-chiller-van-truck-with... https://coolfreights.com/wp-content/uploads/2019/03/Chiller-truck-rental-induabi-400x267.jpg https://coolfreights.com/wp-content/uploads/2019/03/Chiller-... https://coolfreights.com/hire-rentals-chiller-van-truck-with-30-off/ https://coolfreights.com/hire-rentals-chiller-van-truck-with... https://coolfreights.com/hire-rentals-chiller-van-truck-with-30-off/ https://coolfreights.com/hire-rentals-chiller-van-truck-with... https://coolfreights.com/hire-rentals-chiller-van-truck-with-30-off/ https://coolfreights.com/hire-rentals-chiller-van-truck-with... https://coolfreights.com/news-of-refrigerated-truck-dubai/ https://coolfreights.com/news-of-refrigerated-truck-dubai/ https://coolfreights.com/hire-rentals-chiller-van-truck-with-30-off/#respond https://coolfreights.com/hire-rentals-chiller-van-truck-with... https://coolfreights.com/hire-rentals-trucks-van-with-30-off/ https://coolfreights.com/hire-rentals-trucks-van-with-30-off... https://coolfreights.com/wp-content/uploads/2019/03/refrigerator-truck-rental--400x267.jpg https://coolfreights.com/wp-content/uploads/2019/03/refriger... https://coolfreights.com/hire-rentals-trucks-van-with-30-off/ https://coolfreights.com/hire-rentals-trucks-van-with-30-off... https://coolfreights.com/hire-rentals-trucks-van-with-30-off/ https://coolfreights.com/hire-rentals-trucks-van-with-30-off... https://coolfreights.com/hire-rentals-trucks-van-with-30-off/ https://coolfreights.com/hire-rentals-trucks-van-with-30-off... https://coolfreights.com/news-of-refrigerated-truck-dubai/ https://coolfreights.com/news-of-refrigerated-truck-dubai/ https://coolfreights.com/hire-rentals-trucks-van-with-30-off/#respond https://coolfreights.com/hire-rentals-trucks-van-with-30-off... https://coolfreights.com/refrigerated-truck-dubai-abu-dhabi/ https://coolfreights.com/refrigerated-truck-dubai-abu-dhabi/ https://coolfreights.com/refrigerated-truck-dubai-abu-dhabi/ https://coolfreights.com/refrigerated-truck-dubai-abu-dhabi/ https://coolfreights.com/refrigerated-truck-dubai-abu-dhabi/ https://coolfreights.com/refrigerated-truck-dubai-abu-dhabi/ https://coolfreights.com/refrigerator-vans-in-dubai-high-quality/ https://coolfreights.com/refrigerator-vans-in-dubai-high-qua... https://coolfreights.com/refrigerator-vans-in-dubai-high-quality/ https://coolfreights.com/refrigerator-vans-in-dubai-high-qua... https://coolfreights.com/refrigerator-vans-in-dubai-high-quality/ https://coolfreights.com/refrigerator-vans-in-dubai-high-qua... https://coolfreights.com/chiller-truck-for-rent-in-dubai/ https://coolfreights.com/chiller-truck-for-rent-in-dubai/ https://coolfreights.com/chiller-truck-for-rent-in-dubai/ https://coolfreights.com/chiller-truck-for-rent-in-dubai/ https://coolfreights.com/chiller-truck-for-rent-in-dubai/ https://coolfreights.com/chiller-truck-for-rent-in-dubai/ https://coolfreights.com/chiller-vans-dubai-abu-dhabi/ https://coolfreights.com/chiller-vans-dubai-abu-dhabi/ https://coolfreights.com/chiller-vans-dubai-abu-dhabi/ https://coolfreights.com/chiller-vans-dubai-abu-dhabi/ https://coolfreights.com/chiller-vans-dubai-abu-dhabi/ https://coolfreights.com/chiller-vans-dubai-abu-dhabi/ https://coolfreights.com/refrigerated-truck-dubai-abu-dhabi/ https://coolfreights.com/refrigerated-truck-dubai-abu-dhabi/ https://coolfreights.com/refrigerator-vans-in-dubai-high-quality/ https://coolfreights.com/refrigerator-vans-in-dubai-high-qua... https://coolfreights.com/refrigerated-truck-dubai-abu-dhabi/ https://coolfreights.com/refrigerated-truck-dubai-abu-dhabi/ https://coolfreights.com/chiller-truck-for-rent-in-dubai/ https://coolfreights.com/chiller-truck-for-rent-in-dubai/ https://coolfreights.com/chiller-vans-dubai-abu-dhabi/ https://coolfreights.com/chiller-vans-dubai-abu-dhabi/ https://coolfreights.com/freezer-van-rental-transport-in-dubai-abu-dhabi-sharjah-ajman/ https://coolfreights.com/freezer-van-rental-transport-in-dub... https://coolfreights.com/refrigerated-truck-dubai-abu-dhabi/ https://coolfreights.com/refrigerated-truck-dubai-abu-dhabi/ https://coolfreights.com/chiller-truck-for-rent-in-dubai/ https://coolfreights.com/chiller-truck-for-rent-in-dubai/ https://coolfreights.com/refrigerated-truck-dubai-abu-dhabi/ https://coolfreights.com/refrigerated-truck-dubai-abu-dhabi/ https://coolfreights.com/refrigerated-truck-dubai-abu-dhabi/ https://coolfreights.com/refrigerated-truck-dubai-abu-dhabi/ data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7 https://coolfreights.com/wp-content/uploads/2019/10/chiller-Truck-Rental.png https://coolfreights.com/wp-content/uploads/2019/10/chiller-... https://coolfreights.com/chiller-truck-for-rent-in-dubai/ https://coolfreights.com/chiller-truck-for-rent-in-dubai/ data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7 https://coolfreights.com/wp-content/uploads/2019/10/chiller-van-rental.png https://coolfreights.com/wp-content/uploads/2019/10/chiller-... https://coolfreights.com/chiller-vans-dubai-abu-dhabi/ https://coolfreights.com/chiller-vans-dubai-abu-dhabi/ data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7 https://coolfreights.com/wp-content/uploads/2019/10/freezer-truck-rental.png https://coolfreights.com/wp-content/uploads/2019/10/freezer-... https://coolfreights.com/freezer-truck-for-rent-in-dubai/ https://coolfreights.com/freezer-truck-for-rent-in-dubai/ data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7 https://coolfreights.com/wp-content/uploads/2019/10/Freezer-van-for-rent.png https://coolfreights.com/wp-content/uploads/2019/10/Freezer-... https://coolfreights.com/freezer-van-rental-transport-in-dubai-abu-dhabi-sharjah-ajman/ https://coolfreights.com/freezer-van-rental-transport-in-dub... data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7 https://coolfreights.com/wp-content/uploads/2019/10/refrigerated-truck-rental.png https://coolfreights.com/wp-content/uploads/2019/10/refriger... https://coolfreights.com/refrigerated-truck-dubai-abu-dhabi/ https://coolfreights.com/refrigerated-truck-dubai-abu-dhabi/ data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7 https://coolfreights.com/wp-content/uploads/2019/10/Refrigerated-Van-for-rental.png https://coolfreights.com/wp-content/uploads/2019/10/Refriger... https://coolfreights.com/refrigerator-vans-in-dubai-high-quality/ https://coolfreights.com/refrigerator-vans-in-dubai-high-qua... data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7 https://coolfreights.com/wp-content/uploads/2019/05/refrigerated-and-chiller-transport-company-1.png https://coolfreights.com/wp-content/uploads/2019/05/refriger... https://coolfreights.com/sevices/ https://coolfreights.com/sevices/ https://coolfreights.com/sevices/ https://coolfreights.com/sevices/ https://coolfreights.com/chiller-vans-dubai-abu-dhabi/https://coolfreights.com/chiller-vans-dubai-abu-dhabi/ https://coolfreights.com/chiller-vans-dubai-abu-dhabi/https:... https://coolfreights.com/sevices/ https://coolfreights.com/sevices/ https://coolfreights.com/chiller-vans-dubai-abu-dhabi/ https://coolfreights.com/chiller-vans-dubai-abu-dhabi/ https://coolfreights.com/sevices/ https://coolfreights.com/sevices/ data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7 https://coolfreights.com/wp-content/uploads/2019/01/img_bb-scaled.jpg https://coolfreights.com/wp-content/uploads/2019/01/img_bb-s... data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7 https://coolfreights.com/wp-content/uploads/2019/01/img_bb.png https://coolfreights.com/wp-content/uploads/2019/01/img_bb.p... data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7 https://coolfreights.com/wp-content/uploads/2019/01/Chiller-51.jpg https://coolfreights.com/wp-content/uploads/2019/01/Chiller-... data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7 https://coolfreights.com/wp-content/uploads/2019/01/chiller-truck-rental-dubai-refrigerated-truck-dubai-freezer-truck-high.png https://coolfreights.com/wp-content/uploads/2019/01/chiller-... data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7 https://coolfreights.com/wp-content/uploads/2017/02/chiller-truck-for-hire-dubai-freezer-truck-rental-dubai.jpg https://coolfreights.com/wp-content/uploads/2017/02/chiller-... data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7
- coolfreights 7y agohttps://coolfreights.com/wp-content/uploads/2019/01/partner-02-150x118.png https://coolfreights.com/wp-content/uploads/2019/01/partner-... data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7 https://coolfreights.com/wp-content/uploads/2019/01/partner-05-150x118.png https://coolfreights.com/wp-content/uploads/2019/01/partner-... data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7 https://coolfreights.com/wp-content/uploads/2019/01/partner-06-1-150x118.png https://coolfreights.com/wp-content/uploads/2019/01/partner-... data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7 https://coolfreights.com/wp-content/uploads/2019/01/partner-07-150x118.png https://coolfreights.com/wp-content/uploads/2019/01/partner-... data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7 https://coolfreights.com/wp-content/uploads/2019/01/partner-08-150x118.png https://coolfreights.com/wp-content/uploads/2019/01/partner-... data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7 https://coolfreights.com/wp-content/uploads/2019/01/partner-09-150x118.png https://coolfreights.com/wp-content/uploads/2019/01/partner-... data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7 https://coolfreights.com/wp-content/uploads/2019/01/partner-10-150x118.png https://coolfreights.com/wp-content/uploads/2019/01/partner-... data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7 https://coolfreights.com/wp-content/uploads/2019/01/partner-11-150x118.png https://coolfreights.com/wp-content/uploads/2019/01/partner-... data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7 https://coolfreights.com/wp-content/uploads/2019/01/partner-12-150x118.png https://coolfreights.com/wp-content/uploads/2019/01/partner-... https://coolfreights.com/hire-chiller-van-for-valentine-flowers-delivery-in-dubai/ https://coolfreights.com/hire-chiller-van-for-valentine-flow... https://coolfreights.com/12-helpful-tips-for-doing-toyota-hiace-van/ https://coolfreights.com/12-helpful-tips-for-doing-toyota-hi... https://coolfreights.com/commercial-toyota-hiace-chiller-van-for-rent-in-dubai/ https://coolfreights.com/commercial-toyota-hiace-chiller-van... https://coolfreights.com/30-best-frozen-foods-in-dubai/ https://coolfreights.com/30-best-frozen-foods-in-dubai/ https://www.facebook.com/coolfreights/ https://www.facebook.com/coolfreights/ http://twitter.com/Cool_Freights http://twitter.com/Cool_Freights https://www.linkedin.com/in/coolfreights https://www.linkedin.com/in/coolfreights https://www.youtube.com/channel/UC5bQgcogTK-0I25JB42TcYA https://www.youtube.com/channel/UC5bQgcogTK-0I25JB42TcYA https://www.pinterest.com/coolfreights/ https://www.pinterest.com/coolfreights/ https://www.instagram.com/cool_freights/ https://www.instagram.com/cool_freights/ https://www.facebook.com/coolfreights/ https://www.facebook.com/coolfreights/ http://twitter.com/Cool_Freights http://twitter.com/Cool_Freights https://www.linkedin.com/in/coolfreights https://www.linkedin.com/in/coolfreights https://www.youtube.com/channel/UC5bQgcogTK-0I25JB42TcYA https://www.youtube.com/channel/UC5bQgcogTK-0I25JB42TcYA https://www.pinterest.com/coolfreights/ https://www.pinterest.com/coolfreights/ https://www.instagram.com/cool_freights/ https://www.instagram.com/cool_freights/ https://coolfreights.com/ https://coolfreights.com/ https://coolfreights.com/blog/ https://coolfreights.com/blog/ https://coolfreights.com/about-us/ https://coolfreights.com/about-us/ https://coolfreights.com/listings/ https://coolfreights.com/listings/ https://coolfreights.com/contacts/ https://coolfreights.com/contacts/ https://qaisarabbas.com/ https://qaisarabbas.com/ https://coolfreights.com/ https://coolfreights.com/ https://coolfreights.com/wp-content/plugins/gtranslate/flags/16/en-us.png https://coolfreights.com/wp-content/plugins/gtranslate/flags... https://coolfreights.com/ https://coolfreights.com/ https://coolfreights.com/ https://coolfreights.com/ https://coolfreights.com/ https://coolfreights.com/ https://coolfreights.com/wp-content/uploads/2019/02/3cea7fdf-5a95-49ed-a475-89d6fd54f155.jpg https://coolfreights.com/wp-content/uploads/2019/02/3cea7fdf... tel://542215676835/ javascript:void(0); Hidden links: https://coolfreights.com/#mob-menu https://coolfreights.com/#mob-menu https://coolfreights.com/#mob-menu2 https://coolfreights.com/#mob-menu2 https://coolfreights.com/ https://coolfreights.com/ https://coolfreights.com/wp-content/uploads/2019/10/chiller-Truck-Rental.png https://coolfreights.com/wp-content/uploads/2019/10/chiller-... https://coolfreights.com/chiller-truck-for-rent-in-dubai/ https://coolfreights.com/chiller-truck-for-rent-in-dubai/ https://coolfreights.com/wp-content/uploads/2019/10/chiller-van-rental.png https://coolfreights.com/wp-content/uploads/2019/10/chiller-... https://coolfreights.com/chiller-vans-dubai-abu-dhabi/ https://coolfreights.com/chiller-vans-dubai-abu-dhabi/ https://coolfreights.com/wp-content/uploads/2019/10/freezer-truck-rental.png https://coolfreights.com/wp-content/uploads/2019/10/freezer-... https://coolfreights.com/freezer-truck-for-rent-in-dubai/ https://coolfreights.com/freezer-truck-for-rent-in-dubai/ https://coolfreights.com/wp-content/uploads/2019/10/Freezer-van-for-rent.png https://coolfreights.com/wp-content/uploads/2019/10/Freezer-... https://coolfreights.com/freezer-van-rental-transport-in-dubai-abu-dhabi-sharjah-ajman/ https://coolfreights.com/freezer-van-rental-transport-in-dub... https://coolfreights.com/wp-content/uploads/2019/10/refrigerated-truck-rental.png https://coolfreights.com/wp-content/uploads/2019/10/refriger... https://coolfreights.com/refrigerated-truck-dubai-abu-dhabi/ https://coolfreights.com/refrigerated-truck-dubai-abu-dhabi/ https://coolfreights.com/wp-content/uploads/2019/10/Refrigerated-Van-for-rental.png https://coolfreights.com/wp-content/uploads/2019/10/Refriger... https://coolfreights.com/refrigerator-vans-in-dubai-high-quality/ https://coolfreights.com/refrigerator-vans-in-dubai-high-qua... https://coolfreights.com/#page https://coolfreights.com/#page
- sgt 7y agoHow much would it cost to host this on bare metal and co-lo servers, I wonder. Probably orders of magnitude less, but only if your ops costs are very low. If you developers have a DevOps culture, it's doable.
- tuananh 7y agoour clients (airlines) would very much prefer we use AWS over the smaller, lesser known offering.
- aiisjustanif 7y agoVery interesting seeing as Airlines don’t move fast tech wise and usually have DCs.
- Jnr 7y agoWhy are people so obsessed by AWS? It is one of the most expensive hosting solutions that tries hard to lock you into their ecosystem. I somewhat understand why enterprises want to use it, but why are small startups using it so much and then complaining about the cost? Nowadays when we have high speed internet, and a lot of things are containerized, it is so simple to change hosting partners. Just pick one that doesn't cost an arm and a leg and move to a different one if it didn't fit very well. I have used linux containers for 10 years now and changed hosting a few times, each time reducing costs even more. Yes, it is a bit of manual labour, but if you have someone with sysadmin/devops skills, it is easily doable.
- Phr0ztByte 7y agoI completely agree. Most startup's can do with cheaper or even bare-metal hosting for years before having to use highly-scalable solutions like AWS.
- tuananh 7y agoIt's more complicated than that. Sometimes, the decision is not just in the engineers' hands. eg - our clients (airlines) would very much prefer we use AWS over the smaller, lesser known offering.
- thu2111 7y agoWhat justification do the airlines give for placing demands on your own suppliers?
- tuananh 7y agoi'm not familiar with that part. also sale people can make all kind of promises to close the deal.
- tyingq 7y agoIt's pretty common for Fortune 500 type companies to ask for all sorts of intrusive terms they perceive as a benefit to them. Indemnity, long net payment terms, penalties for missed SLA, etc. They could be specifying AWS in this case to reduce latency and/or avoid the internet as a dependency. Maybe the client side is on AWS already, or they already have high bandwidth / dual path via AWS direct connect, and don't wish to do that for a different provider. The reward being that once you're in, they are too big and slow to ever move away from your product.
- tkyjonathan 7y agoI reduce my RDS bill by 66% by keeping it on EC2 and using Percona Server.
- rcarmo 7y agoIf you want to play with an equivalent (barebones, spot instance) K3s Azure setup I use, the template is available here: https://github.com/rcarmo/azure-k3s-cluster https://github.com/rcarmo/azure-k3s-cluster This is NOT For production use (that’s what the managed AKS service is for), but I like to tinker with the internals, so I keep an instance running permanently with a few toy services.
- tuananh 7y agoThanks. this would be useful to spin up a k3s cluster for testing.
- varelaz 7y agoWell I don't think that using spot instances is a very good thing to consider there. 1) What if all of them go down would it make application unusable? I recall one or two weeks on AWS when all spots were down because of capacity problems. 2) You cannot rely on requests to complete on spot instances they can go up and down when they need. For background jobs it's fine, you can retry, but for UX? And if you need to choose this approach to save your costs instead of migration to bare metal, which is usually ~10 times less than AWS then something is wrong with your architecture and you are locked.
- tuananh 7y ago> save your costs instead of migration to bare metal, like i mentioned in other comments. due to our use cases, bare metal is out of question since the clients (airlines) would very much prefer we use AWS over the smaller, lesser known offering.
- varelaz 7y agoWe had recent migration from AWS to Hetzner. We used combined clusters with ondemand + spot instances. After migration to Hetzner we doubled capacity and still saved around half. AWS is very good for flexible work loads, but if load and process is constant and well defined, bare metal hosting much more effective. And there are a lot of well known bare metal hosting providers on the market.
- tuananh 7y agoif workload is constant, nothing beats baremetal in term of cost.
- f0rmatfunction 7y agoFor 2 - horizontal scaling would address this. If there are 2+ copies of the service running behind a load balancer, either can go down/be recreated as an on demand instance if the spot capacity is revoked. AWS (and I’m guessing the other cloud platforms too) has a container service that will dynamically add containers to the load balancer for you
- rasikjain 7y agoGood to see .NET Core here in article and the Microsoft trying to embrace into more opensource solutions. Due to the interoperability, This makes the application to be deployed at lower cost on containers.
- jimnotgym 7y ago> Now, the biggest sunk cost are obviously RDS and EC2. I don't understand how something you pay for every month is being considered a sunk cost? Am I missing an up front charge, or does the writer not understand what it means?
- kim0 7y agoHello fellow hackers, I had worked on some terraform automation for running eks on spot instances. If you find that interesting, let me know. I can help run that in production if needed.
- thrown777 7y agoWow! This article does a surprisingly good job of exploring how insanely incompetent the kubernetes crowd truly is. "We don’t have any real ops guy as you think these days. Whenever we need something setup, we just have to page someone from India team to create instances for us" If you literally can't turn on the computer you are deploying your software to maybe you shouldn't be writing software. Not everyone needs to be a "software engineer". I hear there's lots of money in investment banking or being a lawyer or tons of other interesting occupations. I'm ready for the next dotcom bust.
- dejot 7y agoI got this far: "Now before I tell this story, I will admit first hand that the actual number is lower than 80%."
- dejot 7y agoI got this far: "Now before I tell this story, I will admit first hand that the actual number is lower than 80%." clickbait
- zdawg 7y agoLove this thread. If you are using K8s and want to reduce both the time you spend managing compute infra and the associated cloud costs (whether for AWS, Azure or GCP) a Spot Instance or Preemtible VM, DIY approach is certainly possible, but will require a lot of setup work. Imagine handling multiple autoscaling groups for multiple Spot Instance types - an absolute necessity to diversify interruption risk, dealing with slow autoscaling or classic autoscaling that only considers cpu/mem and not actual pod requirements or no easy way to create a buffer of spare nodes for high priority workloads that suddenly need capacity, or identifying over-provisioning of machine sizes (based on incorrect pod requirements)which greatly exceed the actual needs of your pods. As an alternative, you can try Spotinst's Ocean product (yes, I work there) for K8s and ECS where not only is your infra management simplified, but you can easily reduce your cloud compute cost by 80%.
- AtlasBarfed 7y ago"We don’t have any real ops guy as you think these days. Whenever we need something setup, we just have to page someone from India team to create instances for us and then proceed to set them up ourselves." Whiskey Tango Foxtrot. That is insane. This is a vain attempt to constrain costs?