10 ms·
Show HN: Cloud GPUs for Deep Learning – At 1/3 the Cost of AWS/GCP
- seefashion 6y agolooks great
- bayesianasian 6y agoIs there an option to request for more persistent storage? From the website, I inferred that each 8x GPU machine comes with 2TB SSD storage? Would I be able to request for another 2TB without requesting for more GPU instances? Also, I am assuming that requesting more GPU instances allows me to access more persistent storage?
- ilmoi 6y agoI’m a self taught ML engineer, and when I was starting on my ML journey I was incredibly frustrated by cloud services like AWS/GCP. a)they were super expensive, b)it took me longer to setup a working GPU instance than to learn to build my first model! So I built https://gpu.land/ https://gpu.land/. It’s a simple service that only does one thing: rents out Tesla V100s in the cloud. Why is it awesome? - It’s dirt-cheap. You get a Tesla V100 for $0.99/hr, which is 1/3 the cost of AWS/GCP/Azure/[insert big cloud name]. - It’s dead simple. It takes 2mins from registration to a launched instance. Instances come pre-installed with everything you need for Deep Learning, including a 1-click Jupyter server. - It sports a retro, MS-DOS-like look. Because why not:) The most common question I get is - how is this so cheap? The answer is because AWS/GCP are charging you a huge markup and I’m not. In fact I’m charging just enough to break even, and built this project really to give back to community (and to learn some of the tech in the process). HN special: email me a few lines about yourself and what you’re working on and get $10 in free credit. I’m at hi@gpu.land. Otherwise I’m around for any questions!
- taf2 6y agoBecause a lot of the data could be sensitive do you sign baa’s for HIPAA?
- ilmoi 6y agoSorry I missed this. Yes, more than happy to. Email me at hi@gpu.land.
- chovybizzass 6y agowas going to mine but you say not to.
- ilmoi 6y agothanks for reading the FAQ! Most people don't seem to bother:)
- Sanzig 6y agoIs general CUDA development permitted, or can I only use this for deep learning? There's a simulation problem that I encounter often at work which should parallelize very well, and I was considering getting my feet wet with CUDA by writing a solver for it.
- ilmoi 6y agoCUDA develpment is totally fine. The service was designed with DL in mind, but that shouldn't stop CUDA dev in any way.
- ogiberstein 6y agoWow, sounds really useful! Will check it out
- liuliu 6y agoThank you! This looks indeed cheap. Can you share more stories on data persistence / checkpointing? If I have a job that requires 8 V100 with 3 days, what types of reliability am I looking at?
- ilmoi 6y agoExactly the same as you would be had you been running an EC2 instance at AWS. The machines are hosted, maintained and managed in exactly the same way in a whitelabel DC.
- sacheendra 6y agoThis I highly unlikely. Hyperscalers like AWS have custom power delivery, rack organization, redundant networking, etc. which make their instances reliable. Long running jobs often use checkpoints which require high speed networking and storage, which I don't see an option for. Eg, I cam get EC2 instances with 100gbps networking. Great job starting the service! But, I think you have ways to go before reaching Hyperscalers level reliability.
- shrubble 6y agoIt's highly likely that the reliability of this service will be as good or better than AWS. I have servers hosted in a similar quality datacenter with literally 9 years of uptime. It would be 10 but the customer shut down his services... By the way, unless something has changed, you can't get more than 10gbps of throughput of a single tcp stream in those 100gbps setups...
- nixgeek 6y agoWhy do you believe an individual stream is limited to 10Gbps?
- shrubble 6y agoThey have stated as much in the documentation. See various speeds and feeds depending on instance size and type, and whether inside or outside the VPC at e.g. here: https://d1.awsstatic.com/events/reinvent/2019/REPEAT_2_Deep-dive_into_100G_networking_&_Elastic_Fabric_Adapter_on_Amazon_EC2_CMP334-R2.pdf https://d1.awsstatic.com/events/reinvent/2019/REPEAT_2_Deep-...
- ashish01 6y agoNice! I really like the top up feature. Really gives me peace of mind that I will not blow up my budget accidentally. I have been using https://datacrunch.io/ https://datacrunch.io/ which has almost same feature set.
- etaioinshrdlu 6y agoCan you use GeForce cards and then make us pinky swear to only use it for blockchain processing applications? This is what LeaderGPU does (https://www.leadergpu.com/#chose-best https://www.leadergpu.com/#chose-best), and they had even lower pricing. Although lately they have had demand outstrip supply. Hetzner used to have GTX 1080 instances for about $100 a month, no longer though, and I'm lucky to be grandfathered in to about 8 of them. I told myself a couple years ago that compute will only get cheaper over time. In reality, compute has gotten MORE expensive over time! I cannot match or scale the compute price I locked in a couple years ago with Hetzner. Some of that is the crypto market raising GPU prices, but it is also NVIDIA's licensing making their cheaper cards unavailable in cloud servers...
- ghoomketu 6y agoFor somebody who was 0 knowledge of ML but has always been wary of the huge costs involved this sure looks like a great offering! But I still don't know what it would cost to get something useful out of it. Can you (or anybody who knows about ML) tell me a very very ballpark amount of what costs it incurs to train a model like www.remove.bg that automatically removes background from photos? I'm not trying to build a clone but curios what sort ot financial investment it takes to make such things.
- perennate 6y agoUnless you're using unsupervised learning or can find a good dataset, most of the cost will be in labeling the data. I'm not too familiar with background removal, there may be some self-supervised/contrastive learning approaches, but generally they don't work as well as supervised learning. Even for a week of training, the compute cost is only $200. Edit: maybe you can get some decent results just with COCO segmentation labels: https://towardsdatascience.com/background-removal-with-deep-learning-c4f2104b3157 https://towardsdatascience.com/background-removal-with-deep-...
- ghoomketu 6y agoDoes it make sense to add random backgrounds to already transparent pngs and then use it to train the models? P.S. Really useful link btw. Thank you. P.P.S. One more question if you don't mind. After I've trained the model what kind of hardware does it require to use that model to convert a png image to transparent? I'm guessing that requirements are different when you already have a trained model? Sorry for all the n00b questions. I'm not trying to create a background removal site btw, just trying to grasp what kind of costs it takes to create and run something like it.
- typon 6y agoHey ilmoi, Do you support multi-node training? Particularly, can I reserve for example a 64-GPU instance via 8 nodes and perform distributed training?
- Proven 6y agoNot much on high speed networking in the answer, which sounds suspicious (it's the most important part)
- ilmoi 6y agoYou can! All machines within a single account are inside a private VLAN, which means they all talk to each other (out of the box, no setup required), but nobody else on gpu.land sees them. See the "Are instances within my account connected?" item in the FAQ - https://gpu.land/faq https://gpu.land/faq If you're going to need 64 GPUs I'll have to increase your account limit (currently 16 GPUs per account). Email me at hi@gpu.land
- teruakohatu 6y agoIs there any persistent storage options? I looked through the faq and comparisons and couldn't find any mention of it.
- ilmoi 6y agoWhen you stop the instances (instead of killing it), you persist data. It costs $0.02/gb/mo, so $4/mo for a 200gb harddrive. I will add to FAQ, thanks for pointing out.
- teruakohatu 6y agoThat sounds very reasonable thank you. I will send you an email soon before I give it a try.
- mraza007 6y agoHey I loved the product you made. As a student who is always concerned about the spending on cloud services as they can be pricey but I’m just curious I have used google Colab in the past and it charges $10/Month so I wanted to know how is it different when compared to google colab I would love to hear your thoughts on this
- ilmoi 6y agoHey! So the way I would describe your progression as a student (at least from my experience): - Traditional ML -> probably can run on your laptop - Simple DL -> colab is great. Cheapest there is. - SOTA DL -> you're probably looking at training times well into 10s of hours / days, so you'll need something that can last longer than 12h of colab time. Plus at this point you're probably sophisticated enough that you want to setup your instance once and start/stop it rather than going through setup with colab every time. That's where https://gpu.land/ https://gpu.land/ fits it. So tl;dr; - absolutely use colab first (it's cheapest), but when you outgrow it, consider gpu.land.
- mraza007 6y agoperfect and it totally makes sense thanks for clarifying
- benohear 6y agoThis is all very cool, and FWIW I also love the retro design. Sorry if I missed it, but one thing I couldn't find on the website is what legal entity is behind the service. It would be important for me and the organisations I work with to know who we are trusting with our data. It might even be a legal obligation to have that info on the site depending on which country you are located in (Germany for sure, not sure about others).
- ilmoi 6y agoHey - for sure. Because I'm the only team member, the entity behind the business is registered as a "sole trader". If you email me on hi@gpu.land I'm happy to share the details.
- akrymski 6y agoCongrats on the launch! Cheaper GPUs are always welcome ;-)
- samsammurphy 6y agoCooooool!
- neil1 6y agoThis is a pretty cool project
- 37ef_ced3 6y agoAVX-512 neural net inference on inexpensive, CPU-only cloud compute instances: https://NN-512.com https://NN-512.com An AVX-512 Skylake-X cloud compute instance costs $10 per CPU-core per month at Vultr (https://www.vultr.com/products/cloud-compute/ https://www.vultr.com/products/cloud-compute/), and you can do about 18 DenseNet121 inferences per CPU-core per second (in series, not batched) using tools like NN-512 GPU cloud compute is almost unbelievably expensive. Even Linode charges $1000 per month, or $1.50 per hour (look at the GPU plans: https://www.linode.com/pricing/#row--compute https://www.linode.com/pricing/#row--compute). It's really hard to keep that GPU saturated, which is what you need to do to get your money's worth As AVX-512 becomes better supported by Intel and AMD chips, it becomes more attractive as an alternative to expensive GPU instances for workloads with small amounts of inference mixed with other computation
- rckoepke 6y agoWhat does this have to do with https://gpu.land/ https://gpu.land/ ? I've been very impressed with NN-512 in your past postings but I'm failing to see the direct salience w.r.t the top level post. Perhaps readers would benefit from an apples-to-apples comparison of V100 to some CPU in a training per dollar metric? Preferably using something like MLPerf. You do mention inference but I think most people looking at https://gpu.land/ https://gpu.land/ will be far more interested in training rather than inference. I think the most direct competitor to this ShowHN would be https://vast.ai/ https://vast.ai/
- ilmoi 6y agoYou are correct that vast.ai is pretty close. The biggest difference is probably security / guaranteed uptime. With vast you're getting what it says on the tin - a machine from a marketplace. Could come from anyone / anywhere. No idea what else is running on it. Ours are hosted in a professional DC, managed and secured as they should be. If anyone's curious, there's a detailed comparison page with other platforms here - https://gpu.land/versus https://gpu.land/versus
- perennate 6y agoI think Linode/Paperspace/AWS/GCP/etc. are closer to "direct competitors". I would be much more comfortable trusting a single entity that owns its servers than a GPU rental marketplace like vast.ai.
- DIVx0 6y agoCould this be used for bespoke cloud gaming? I was considering setting up a parsec host with paperspace GPU. I want more control over the game client than geforce now or stadia provides and a better GPU and availability than Shadow.tech currently offers. *edit: I see that only Linux hosts are available at the moment so that kills my use case but I'll keep the question up for grins and giggles
- ilmoi 6y agoNot really. Firstly it will be way too expensive for you (cloud gamins is like $10/mo, this is $10/10h). Secondly the GPUs installed (Tesla V100s) are not designed for gaming - rather for ML & scientific computing. You should check out some of the services people mention in https://www.reddit.com/r/cloudygamer/ https://www.reddit.com/r/cloudygamer/
- dindresto 6y agoHow does the Tesla V100 compare to the Tesla P100, which Scaleway offers at a similar price of €1/hour? https://www.scaleway.com/en/gpu-instances/ https://www.scaleway.com/en/gpu-instances/
- ilmoi 6y agoTesla P100 is the previous generation. P100 > V100 > A100. You can see detailed benchmarking here - http://ai-benchmark.com/ranking_deeplearning.html http://ai-benchmark.com/ranking_deeplearning.html
- DougN7 6y agoI know nothing about crypto mining, but why wouldn’t this be allowed? How would you stop it?
- imhoguy 6y agoWoudn't pay off really, check the FAQ.
- ilmoi 6y agoyou'll be making around $6/d, while losing $24h paying for the service.
- ska 6y agosee jsnell's point which is worth considering and mitigating if your (CC) processor isn't able to catch.
- jsnell 6y agoMining cryptocurrencies is a good way to launder stolen credit cards numbers to cash. It is scalable (unlike most digital goods) and automatable (unlike most physical goods). It being unprofitable will not dissuade anyone from this use case: they are not paying with their own money to start with. So it is best to forbid it in the TOS and just autokill the miners, rather than wait for the chargebacks. Especially if the compute is being sold at cost, since there is no buffer of profits to balance out the abuse.
- ilmoi 6y agoWow thanks for posting this! I defo haven't thought about this use case. So interesting. So we actually explicitly prohibit mining in the T&Cs, it's just that in the FAQ I wanted to be a bit more human and dissuade people. Also, there's quite a few protections built-in at the network layer to make sure mining isn't possible. Ports, ips, dns, even DPI. I learnt a lot about the early days of bitcoin and mining protocols when I was building gpu.land:) But again, thanks for sharing this. Really good to know.
- wrongdonf 6y agoI love a dutch programmer
- ilmoi 6y agohah I am European indeed but not Dutch:) But I have spent a year living in the Netherlands.
- artem_mazur 6y agoLooks interesting! Cool!!
- ffpip 6y agoFYI, the site shows nothing until the scripts from stripe.com load. I know that that these breakages might not happen to all users, but it shouldn't be completely broken until the browser connects to Stripe. https://images2.imgbox.com/b2/42/n87NuBRT_o.png https://images2.imgbox.com/b2/42/n87NuBRT_o.png
- ilmoi 6y agoDo you think it's the scripts from stripe that are blocking the view? I would guess it's auth0. This was my first front-end project so any feedback 100% welcome.
- ffpip 6y agoYes. When stripe is blocked, the page is blank, but when the js from stripe.com is allowed, the page loads.
- jandrese 6y agoFWIW when I went to the site I had to whitelist scripts from js.stripe.com to get anything more than a black screen.
- maremmano 6y agoI came for the GPU, I stayed for the vintage CSS
- stuartbman 6y agoThis is a very cool project, but can I ask if there's a particular CSS package you used to get this site appearance?
- ilmoi 6y agoThanks! I used Tailwind CSS and built the theme from scratch, but I drew inspiration from: - https://nostalgic-css.github.io/NES.css/ https://nostalgic-css.github.io/NES.css/ - https://jdan.github.io/98.css/ https://jdan.github.io/98.css/
- usmannk 6y agoThis is seriously cool. I want to mention that on AWS you can reliably get V100's for $0.918/hr (I do this all the time) by using spot instances. The spot price hasn't changed by even 0.01 for at least a year, so you're extremely unlikely to get an unexpected shutdown. Also, how did you manage to get V100s? What was that like?
- ilmoi 6y agoYep you're correct that V100s are available as spot instances. If you're willing to write some extra code for saving down weights you could easily go with them. Actually, I'm renting V100s. Got lucky to know the right person at the right time:)
- usmannk 6y agoAh gotcha, did you have to move them to your DC and have them racked?
- tombh 6y agoThese seems very similarly priced to AWS Spot Instances. But you're guaranteeing uptime?
- perennate 6y agoIn the FAQ it says: > Is my instance guaranteed? > Yes, unless you run out of credit. To prevent that from happening be sure to setup automatic top ups.
- perennate 6y agoThe website design put me off but the FAQ makes it clear that it is quite professional. They have information about physical security, GDPR compliance, and VLAN. Wish there was some information about storage, e.g. if it's local disk or distributed block storage, and how many replicas. Very nice that unused credit is refundable.
- ilmoi 6y agoGreat point re storage, I'll be sure to add to the FAQ. To answer your question it's SSD block storage.
- mastermojo 6y agoVery cool. Is there a way to persist disks at a cost and attach/unattach Voltas for training? EDIT: found in the FAQ: Compute: $0.99/hr / 1x Tesla V100 (running instance only) Storage: $0.02/GB/month (running and stopped instances)
- ilmoi 6y agoYep - most people stop instances and come back whenever they're ready to train again. No charges for GPUs while the instance is stopped. If you have a say 200GB harddrive you're only paying $4/month for storage.
- paul_milovanov 6y agothis is great, except apparently there's no support for containers? is that on the roadmap? thanks!
- ilmoi 6y agoNot externally (in the same way that vast.ai does it), but you could always boot the instance -> ssh in -> manually run nvidia-docker. Added to feature requests!
- paul_milovanov 6y agothanks!
- iujjkfjdkkdkf 6y agoSorry if I missed it in the FAQ, do you have finite capacity or do you have an agreement with the data center where you can bring on more gpus with demand. My worry would be to wake up one day and find that all GPUs are in use. And (this may sound naive) I work with other companies' data, I have a responsibility to them to keep it safe, do you have any concerns with this being used for "professional" applications or are you targeting research / hobby? To be clear, I am dealing with things that my clients are comfortable with me working with on mainstream cloud providers, not state secrets or data with legislative requirements.
- ilmoi 6y agoGreat questions. 1) There's a finite capacity in the high 10s of GPUs. Currently the service is utilized at <10% capacity, so there's a lot of room to grow before we run out. If that was to happen (or even if we get to 50%), I could go and request more and (hopefully) expand the capacity quickly. 2)When building gpu.land I specifically wanted to make it safe for users to upload / store sensitive data. Of course the service is more geared towards hobbyists / researchers, just because that's an easier market to reach for a solo dev - but there is nothing at the technical level that sacrifices data privacy. For example, data on instances is encrypted both at rest and in transit within the DC + you and only you have SSH keys to your instance and nobody else can access it. Check out our security & privacy section in our FAQ - https://gpu.land/faq https://gpu.land/faq
- iujjkfjdkkdkf 6y agoOne more comment / suggestion: I watched the video and saw there are lots of pre-configured environments. I see this a lot, but to be honest this is not to helpful for me because I'm not doing development in your environment, I'm writing and checking the code somewhere else and then training on the gpus. For the same reason, I'm only getting value out of the gpus when I am right ready to train. So I would be much happier if I could push a docker, or maybe even a conda environment spec, along with my code, attach data storage, and run e.g. train.py (or more likely a shell script that calls it) to completion and then immediately release the GPUs. Everything else is just the overhead of as quickly as possible trying to get the environment right, run my script, and then shut down the instance as soon as it's done. It would be awesome to have this kind of functionality, or is there a way to do that I missed?
- ilmoi 6y agoBoth are valid points. And no, there's currently no programmatic access to machines - but that's high on the roadmap. Regarding environments, it's funny, when I was starting out I would kill for a pre-configured instance because I wanted to focus on modeling and didn't care if package X was version x.x.y or x.x.z. But as you grow as an ML engineer and develop your own toolkit these things start to matter. So when creating a machine on gpu.land you have the choice of going pre-configured or just having a clean Ubuntu image. The former is meant for newcommers while the latter for pros. That was my thingking!
- iujjkfjdkkdkf 6y agoThanks a lot for answering my questions. I am definitely going to give this a try. Right now I see this as a great option for an always on dev instance that I can work in all day with GPU access and not worry about breaking the bank. I saw someone ask on reddit about separating storage from the GPU instance, so that one could do data transfer and other setup without reserving a gpu. I want to echo how important this is. Another case is where I might want to use only one GPU and then scale up to 8 for training, or i might want to have N GPU instances attached to the same storage to run jobs in parallel. There are lots of other examples, but overall it would add much flexibility. On the other hand, it might encourage people to be more "peaky" in their use, which could be a challenge for you. From what I understand, you are much better off if I want 1 gpu for 70 hrs vs, 70 gpus for an hour, in which case I understand how you might want to encourage steady use.
- tehsauce 6y agoFor anyone looking for cheap gpus (quite a bit cheaper than this even), I highly recommend the marketplace on vast.ai I’ve spent hundreds of dollars there and will spend many more.
- ilmoi 6y agoYou are right in that it's cheaper, but not that much cheaper if you look at renting v100s. Actually, on the website there's a whole section that compares us to them - https://gpu.land/versus https://gpu.land/versus
- lostmsu 6y agoI wonder if this project will stay sustainable when V100 prices in other clouds drop below $1/h. This is not unthinkable considering A100 becoming more widespread.
- bitobserver 6y ago> At 1/3 the Cost of AWS/GCP For on-demand this is true, but spot GPU instances can have competitive pricing. At this moment, a p3.2xlarge instance (1 Tesla v100) is 92 cents/hr on the AWS spot market in us-east-1. The p3.8xlarge (4 GPU) is $3.62/hr.
- 2Gkashmiri 6y agoOfftopic but I've been considering starting a "render farm" business. I can fork out some capital, take a loan and such but the important thing is, rent and electricity is almost free. So, wpuld it work?
- Reubend 6y agoHey there, just wanted to share that for me (using Opera browser with ad block) scrolling is broken on your landing page.