10 ms·
UC Berkeley launches SkyPilot to help navigate soaring cloud costs
- bigcat12345678 4y agoHave seen this project a while ago, it's a radically exceptional idea. Could be quite interesting.
- bashtoni 4y agoFrom reading this launch post, I'm not convinced this is going to save too much money. The project automatically selects the cheapest cloud to run a job, and does it there - which sounds sensible. In reality though, these jobs presumably need large volumes of input data. If your input data is in cloud A, and you run a job in cloud B, typically any cost saving from running in cloud B will be more than offset by the egress cost to get the data out of cloud A. This project is therefore only useful for scenarios where you need to do large amounts of compute on relatively small volumes of data. Is that really a common scenario?
- vladf 4y agoThey have a project which addresses this concern as well: https://skyplane.org/en/latest/benchmark.html https://skyplane.org/en/latest/benchmark.html
- parasj 4y agoI'm one of the creators of Skyplane. Skyplane can migrate large datasets between cloud regions at 10s of Gbps while compressing data to reduce egress fees. Happy to chime in! https://github.com/skyplane-project/skyplane https://github.com/skyplane-project/skyplane
- ensemblehq 4y agoCongrats on the launch! I had a similar idea once a few years back but failed to materialize it. You might want to consider other cloud providers like Sushi Cloud to get costs even lower. Happy to do an intro if it seems interesting.
- bushbaba 4y agoOr to leverage cheaper compute/energy when it’s available. https://www.crusoecloud.com/features/ https://www.crusoecloud.com/features/
- helsinkiandrew 4y agoAnd isn’t the biggest issue with running potentially large jobs in the cloud the cut off when it’s cheaper to use your own hardware. After a few months or dozens of runs of your large model in the cloud you may have reached the point where purchasing would have been cheaper. Something that could look at your code, data and budget and say upto X runs use cloud A, for more than Y runs it would be cheaper to buy/lease these GPUs etc. would be interesting.
- YetAnotherNick 4y agoI think it’s common to train 100s of models on the same data for experiments. Then you would only need to copy data once to all the cloud storage and run experiments as you wish. Also most cloud provider don’t charge for ingress so you could move the data from something like R2 to cloud as many times you want..
- covi 4y ago+1. We've heard from some heavy users that Cloudfare R2 is saving them $$$ on egress costs: https://www.cloudflare.com/products/r2/ https://www.cloudflare.com/products/r2/ As outlined in the position paper (linked by another commenter) we believe such tailwinds are increasingly helping foster the "Sky" and making workloads moving between clouds much easier.
- covi 4y agoI'm one of the creators of SkyPilot. Thanks for the thoughtful questions and let me try to take a stab: SkyPilot is not just for multi clouds. It's useful for all of these scenarios: - using a single region of one cloud - using multiple regions of one cloud - using multiple clouds Data transfer between zones/regions within a cloud is much cheaper than across clouds. We see many users falling in the "one cloud" category and they frequently read 10s of TBs of data across regions to do ML training. Finally, saving money is one of several key problems we aim to solve, and there are quite a few ways to save other than lots-of-compute-on-small-data. Other reasons why you may want to use a system like SkyPilot include (1) improving resource availability (big pain point for GPUs/TPUs) (2) use one interface and know that your jobs can migrate across regions or clouds More rationale in the intro blog post: https://medium.com/@zongheng_yang/skypilot-ml-and-data-science-on-any-cloud-with-massive-cost-savings-244189cc7c0f https://medium.com/@zongheng_yang/skypilot-ml-and-data-scien...
- walrus01 4y agoAt some point people will rediscover that "buying" whole sets of equipment through a lease financing company on a lease-to-own plan with $1 end payment, and colocating it the traditional way can often be significantly cheaper than paying endless "cloud" costs. naysayers will say, but the cloud allows you abstract away your salary costs of engineers! look at all the people you aren't hiring! I say: If your needs are sufficiently complex you already have a number of well paid six figure salary people on staff to admin your cloud-based software architecture/setup... Hire people who know how to build infrastructure down to the bare metal.
- marricks 4y agoIt's funny that that is exactly how disruption works. Use tons of VC money to flood a market with a cheaper product, become dominant, then jack up prices hoping people forgot the alternatives. It doesn't have to be jacking up prices for exploitation either, just sometimes the costs were always there dictating the price and, eventually, you have to foot the bill for those costs.
- manv1 4y agoI've configured Cisco routers, F5s, LOMs, PDCs, NAS boxes, built servers from parts. And I'd never do it again. That shit has literally no value, and it's quite difficult to configure, manage, and maintain. And no matter how good your process, you (or your team member) inevitably will forget to go into the BIOS and turn off power saving...or forget to turn off (or on) proxy arp. Or you'll reboot a box and it won't come back. All costs can be negotiated with up-front commitments. We are small, but we pay .01/GB for egress bandwidth. And we got that price from akamai, fastly, and AWS by asking. I don't remember how much real colo places charge anymore, but it was a whole lot more than that. Enterprise SSDs, redundant hardware, etc cost money. Does your startup qualify for a lease? And really, why spend money on something that has no value to your business? Does having 5 data centers add any real value or competitive advantage? Does having that F5 expert on staff make your end-users happier? If you answer yes, then go for it. But really, it's a total waste of time and money. You might as well make your own pencils and paper for the amount of value it delivers.
- 4y ago
- random3 4y agoIt's fascinating to see the descendants of the original AMP Lab (https://amplab.cs.berkeley.edu/ https://amplab.cs.berkeley.edu/) are still generating value long after it ended. For reference, Spark and Mesos are two other projects that came out of the same effort. This blog post from the authors https://medium.com/@zongheng_yang/skypilot-ml-and-data-science-on-any-cloud-with-massive-cost-savings-244189cc7c0f https://medium.com/@zongheng_yang/skypilot-ml-and-data-scien... I believe autostop (simply stopping idle clusters) alone can masively (X times) costs for many organizations.
- covi 4y ago(SkyPilot dev) Can't agree more on autostop / autodown. Subjective to measure but save our own costs massively.
- d4rkp4ttern 4y agoI believe the Ray suite of ML + distributed computing tools is also from AMP descendants: https://docs.ray.io/ https://docs.ray.io/
- ww520 4y agoI feel that the old SETI@home project can use a comeback. Most home machines have powerful GPU these days. The ML problems are embarrassingly parallel and can be distributed to the home machines. Just need to work out the economy for everyone involved.
- spullara 4y agohttps://vast.ai https://vast.ai?
- YetAnotherNick 4y agoSingle ML job can’t be distributed very well. More often than not they hit network limit, even on single zone 1 GB/s network speed that we normally get. Most of the distributed workloads use something like NVLink.
- ww520 4y agoThat's true. Most ML algorithms have the iterate-until-converge pattern. How about some tasks like hyper parameter tuning or trying out different algorithms against the same data set? Those can be run in parallel. ML research should aim to produce more parallel algorithms.
- andbberger 4y agostate of the art models don't fit on a single gpu anymore, not for a while
- Frost1x 4y agoIs that more for training or inference use? I realize this isn't always entirely decoupled in certain online learning approaches. I don't work in ML, am certainly not an expert, and am genuinely curious where this space is at now in terms of hardware requirements for SOTA methodologies these days, especially inference phase HW requirements for just running stuff that's out there.
- 4y ago
- ritwikgupta 4y agoWe’re using SkyPilot and Skyplane at Berkeley and the Department of Defense to scale our damage assessment from satellite imagery machine learning models [0] to run for all of Ukraine, all the time! We’re able to detect instances of war crimes rapidly and use them to help plan the reconstruction of Ukraine. [0] https://xview2.org/ https://xview2.org/
- m2fkxy 4y agodo you use SAR?
- ritwikgupta 4y agoWe do not currently use SAR! High-resolution SAR is currently very hard to find. Vendors like Capella Space are pretty pricey for HADR operations, though they do provide some data for deeply discounted/free prices for major disasters. Beyond cost, SAR is just a deeply different form of imaging. The underlying physics that define a SAR image are so different from EO that many computer vision methods, in my opinion, will need to be re-invented to accommodate it. In fact, the next good bit of my Ph.D. is dedicated to accomplishing just that. Urban scenes look very chaotic in SAR due to multi-path and layover effects. Defining damage in that environment is more aggregate than individual buildings. We're trying to figure out a good way to define what damage even means on SAR. In fact, one of the biggest accomplishments of our xView2 work was being the first comprehensive effort to define what damage means, for any natural disaster, from EO satellite imagery!
- m2fkxy 4y agoThanks for the writeup! Agree on SAR being vastly different to interpret, and I can imagine the priciness. I also understand that the capacity/demand is another complicating factor now over Ukraine. As you explained SAR phenomenology is not straight forward at all (pun intended). If I could suggest an option to explore a SAR-based damage proxy indicator, it would be interferometric coherence. This suggests however that you get your hands on InSAR data, both timely (to avoid decorrelation, which can happen over the span of a few days) and at good resolution to get relevant information at building-level. As you can imagine this is not easy nor cheap to get. I am unaware if Capella is InSAR-capable; COSMO-SkyMed and TerraSAR/TanDEM/PAZ are the usual suspects when it comes to SAR constellations with InSAR capability and good frozen orbits. ICEYE I believe is trying to get there, but things are evolving rapidly. Sentinel-1 is a wonderful mission with solid InSAR capability, but I am not sure C-band delivers the resolution you need. Maybe NISAR is worth keeping an eye on in the future? Fascinating work that matters, good luck with it!
- teleforce 4y agoPrevious HN post on UC Berkeley position paper on the future of cloud computing: https://news.ycombinator.com/item?id=31449735 https://news.ycombinator.com/item?id=31449735
- Helmut10001 4y agoI wonder why UC Berkeley doesn't build a proper HPC, they have a Data School and should provide this service for free to their faculties. We have "free" HPC resources at TU Dresden (Germany) (meaning: faculties do not need to pay for using HPC resources and they are not calculated in project budgets). I once applied for a job at University of Virginia, and they didn't have a HPC - everything was bought from AWS. When students accidently left stuff running, the professor had to beg Amazon to reimburse the fees. This was the main reason I was hesitant taking the offer. I even calculated building my own "Cloud" with a Proxmox cluster at home, so that I could teach students the basic stuff.
- fakename 4y agohttps://research-it.berkeley.edu/services-projects/high-performance-computing-savio https://research-it.berkeley.edu/services-projects/high-perf...
- untilted 4y agoA lot of Berkeley researchers use the NERSC facilities at the nearby Berkeley National Lab: https://www.nersc.gov/ https://www.nersc.gov/
- Helmut10001 4y agoInteresting, thanks!
- Fomite 4y agoAt several universities I've been at, HPC groups have been utterly unprepared (and disinterested in becoming prepared) to handle PII or any sort of health or confidential data.
- reacharavindh 4y agoAs we are talking anecdotes. Universities I’ve been at that researched sensitive data like human genetics and some commercially sensitive data have been excellent at data security, and provided a centralised HPC cluster at a marginal cost than it would have been at AWS..
- semitones 4y agoIn my experience, if you want _the_ best price-performance ratio for ML jobs, your best bet is CoreWeave. They have great GPU availability and it's a lot easier to scale up/down very quickly than on the popular clouds. I would really be surprised if anyone else out there is offering a better price point for these types of workloads..
- praveenhm 4y agocheckout lambda labs, they have very competitive pricing on GPU, they are almost half the price of CoreWeave at the last check.
- covi 4y agoIndeed! Having lower-cost GPU clouds in the "Sky" is on our immediate roadmap: https://github.com/skypilot-org/skypilot/blob/master/ROADMAP.md https://github.com/skypilot-org/skypilot/blob/master/ROADMAP... In fact, as we speak we're working with folks at Lambda Labs to add support for their cloud. If other providers are interested, we'd be happy to chat. (SkyPilot dev here)
- mattnewton 4y agoLambdalabs is fantastic, but they are so fantastic and popular they often seem to run out of gpus available to rent for me :D
- bushbaba 4y agoCommon issue for anyone running at scale. The crypto winter should hopefully address this.
- bushbaba 4y agoThis is great for list price workloads. But once you have large usage (N+ MM/year) you’d likely get better savings by doing a multi-year usage contract.
- donnygreenberg 4y agoI've been using SkyPilot for 3 months within an ML platform project and it is exactly as awesome as it sounds. The overall experience for launching and managing compute is thoughtful and ergonomic in a way that we've been conditioned not to expect from cloud compute (let alone GPU compute). It was a real shock to start using it, and now every time I need to jump back into the weeds of a cloud provider's APIs I'm reminded of how nice I have in SkyPilot land. I think you'd have a hard time finding a better cloud launcher anywhere. I couldn't.
- mlboss 4y agoDoes it support spot instances ?
- mdaniel 4y agothey'd be crazy not to, unless the ML pipeline didn't support retries: https://skypilot.readthedocs.io/en/latest/examples/spot-jobs.html https://skypilot.readthedocs.io/en/latest/examples/spot-jobs...
- jsd_dmatrix 4y agoUnderneath, Skypilot uses a Ray quite a bit. You can read is paper here: https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s02-stoica.pdf https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s...
- vtuulos 4y agoThis is great! If you are interested in running ML workloads across clouds, Netflix's Metaflow will officially announce support for all clouds tomorrow: https://outerbounds.com/blog/metaflow-on-all-major-clouds/ https://outerbounds.com/blog/metaflow-on-all-major-clouds/ Quite a coincidence :)
- KronisLV 4y agoAren't AWS, Azure and GCP all comparatively expensive? They are good for certain workloads and it's good that this project exists, but if you wanted cheap cloud resources, you'd probably need to look at one of the smaller cloud vendors: Hetzner, Scaleway, DigitalOcean, Vultr, Contabo and others. Of course, if you need GPUs, things can get limiting.
- pid-1 4y agoFor infrequent, unpredictable workloads using something like spot instances on AWS and scaling as needed (including to zero) will likely be cheaper. I think Hetzner and other budget cloud providers are excellent choices for use cases that require always on, reasonably predictable workloads like webservers.
- vidarh 4y agoThe thing is they are so much cheaper, especially for egress, that it takes really extreme spikes or batch jobs for it to be cheaper to use AWS vs. leaving an excess number of server running permanently on somewhere like Hetzner. Very few sites have variable enough traffic that scaling up and down with usage saves enough to even get close. There certainly are genuine cases where AWS will come out ahead, but they're rare. E.g. if you suddenly need several hundred instances for on average a few days a month, it's probably worth it. Very few people do that, and paying for the capacity to be able to offer that is part of the reason for why AWS is so expensive. Peoples belief that AWS is cheap is another reason. Ironically, the ability to run a hybrid setup that scales up in AWS when you genuinely need rapid extra capacity changes the maths even further in favour of dedicated servers from places like Hetzner (as does the fact most places like Hetzner now have their own cloud offerings) because it means you can go closer to the wire on your dedicated servers.
- freediver 4y agoThey usually are, unless you get interruptible instances. You can compare pricing for most cloud providers here: https://cloudoptimizer.io https://cloudoptimizer.io (my hobby project)
- 4y ago
- laurencerowe 4y agoHaving worked on large compute intensive projects in both academia and industry I am somewhat sceptical about multi-cloud tools. Normally you do your compute where the data is since egress is so expensive. And it just doesn't seem worth developing against a cloud agnostic API and limiting myself to the lowest common denominator. If you have a large workload though they may be helpful as a negotiating tactic to play one cloud off against another.
- covi 4y agoOne big reason people use multiple regions or clouds: higher resource availability. Allocating scarce resources can be very hard on the cloud. These are things like high-end GPUs (both on-demand and spot; the latter being much harder to get) and also beefy CPU-based instances. We've seen 10s of hours of waiting times or longer. The natural solution is to have the flexibility to allocate resources in multiple regions (and ultimately, clouds) to increase the total pool size.
- laurencerowe 4y agoI have seen very large jobs that needed to be split up across multiple regions but doing so required replicating large amounts of data. That comes at a cost too so it requires planning to work out which regions you want to use. Once that is done it is fairly easy to set up spot clusters in each region to process the jobs in the queue.
- chse_cake 4y agoIt lost me when the specification was in YAML. I hate YAML for ML.
- mdaniel 4y agoThen lucky you: continue to use JSON since it's a subset of YAML
- covi 4y agoSkyPilot devs here, happy to answer any questions. GitHub repo (Apache 2 license): https://github.com/skypilot-org/skypilot https://github.com/skypilot-org/skypilot Getting started is easy: $ pip install "skypilot[aws,gcp,azure]" # Pick your clouds $ sky check $ sky launch
- lewiso1 4y agoLooks great! Do you anticipate any pushback from cloud providers, who might one day decide to restrict access to stop people getting a better price elsewhere?
- covi 4y agoWe don't anticipate this will happen for a few reasons. When the usage of SkyPilot (or a SkyPilot-like "intercloud broker" system) is small, it probably doesn't warrant the dominant clouds' attention. When the usage gets bigger, I'm not sure how providers can restrict access anyway (curious if there are precedents). There are quite a few large multicloud platforms like Snowflake or Databricks heavily utilizing AWS/GCP/Azure already. (Granted, these platforms are not meta-cloud, in the sense of moving their customer workloads transparently across clouds.) Ultimately we see such a system to grow the pie for the whole cloud market. The incumbents' relative shares may drop, but their absolute volume will grow.
- master_crab 4y agoI’m curious how this affects network and data storage costs. Maintaining data storage and private fiber to all the clouds has its own costs.
- freediver 4y agoConisder adding CoreWeave for GPUs. The A100 there is about half the cost of V100 on large clouds.
- mkhalil 4y agoI remember when the "cloud" hype was first beginning to make a splash. A pillar selling point was "savings". "Reduce your costs", they said. "Don't worry about those expensive hardware costs", or management. "Only pay for what you need", they said. Until everyone was hooked and the tooling was everywhere. Even just putting the pricing for hardware/services aside, the bandwidth charges alone are quite amazing. Most have it at $80 per TB per month. That's $80 a month for a 3mbit connection. [3mbps * 3600 seconds an hour * 24 hours * 30 days ~= 1TB]. Even if you pretended it was 6mbps for double the redundancy, that's still $80 for a 6mbit connection. I understand there's more to it than just transfer amount/speed, but this is in a place where bandwidth is supposed to be cheaper, as it's "purchased" in wholesale, yet a 1GBit upload connection could transfer 330TB in a month, which would cost over $23,000 at Microsoft/Google/Amazon. I am not saying the cloud doesn't make sense. I love me some Firebase. CDN's are game changers. And GPU's per hour is the only way some scientists/researchers can afford to do work [although it does contribute to the NVIDIA's focusing on manufacturing high price/low volume]. That said, it's not for everyone. A cheap VPS can do a lot. Specifically for a university or an institution with a lot of cheap/free bandwidth, on-site would not only be cheaper, but also have lower latency for its campus users and more control wise. Also, if done right, there's a possibility of teaching the students relevant real-world solutions using the knowledge gained/within the university.
- master_crab 4y agoMost people don’t bother refactoring their application and leave whatever IIS garbage they have running on windows 20xx in a cloud instance (to say nothing about the windows licensing cost). Also universities don’t have the same cost models as companies. Their labs have less money - except those institutions with large endowments - but they all have access to near limitless “slave” labor in the form of grad students and post docs.
- bombcar 4y agoWhen I was involved with univs the labs were bursting at the seams with hardware, because companies abounded who loved nothing more than dumping product on us at cheap or free, especially if we'd get students using it.
- julienchastang 4y agoFor those of you in US academia, checkout the NSF Jetstream2 cloud [0] available through ACCESS grants [1]. They have various flavors of VMs including regular, GPU, and large instances. [0] https://jetstream-cloud.org/ https://jetstream-cloud.org/ [1] https://access-ci.org/ https://access-ci.org/
- josh_carterPDX 4y agoI remember when Cloudability first did this to manage AWS costs. They ended up getting scooped up by Apptio, but what they built was pretty awesome. It was especially useful for big teams who often spun up a large EC2 instance for testing, but forgot to take it out of service.
- kamikaze675 4y agoThere are several 'web3' projects in this space, vast.ai, already mentioned, is focused on renting out GPU's from servers hosted pretty much anywhere. For low cost server hosting and egress bandwidth, Threefold (https://www.threefold.io/build/ https://www.threefold.io/build/) and Akash are two new startup web3 ecosystems looking to radically bring down the costs of cloud compute, storage, and bandwidth with GPU support on both of their roadmaps. Still very early, but at 80-95% raw cloud cost savings even beating out Hetzner/DO, its an interesting niche of token incentivized physical infrastructure networks (called TIPIN for short).