5 ms·
My annoyance was managing state. I’d have to spend hours installing tools, downloading data, updating code, then when I want to go to bed I have to package it u
by itake 3y ago
My annoyance was managing state. I’d have to spend hours installing tools, downloading data, updating code, then when I want to go to bed I have to package it up and store as much as I can on s3 before shutting off the $$ server.
- izuchukwu 3y agoYou should check out https://brev.dev https://brev.dev. You can rent GPUs, pause instances, use your own AWS/GCP accounts to make use of your credits, and the CLI lets you use your GPU as if it’s on your local machine.
- throwaway6734 3y agoDocker?
- akiselev 3y agoYou can try skypilot: https://skypilot.readthedocs.io/en/latest/ https://skypilot.readthedocs.io/en/latest/ It handles storage, setup, etc for machine learning work loads across several providers - which helps a lot if you need one of the instances that rarely have capacity like 8x A100 pods.
- orloffm 3y agoAren't Vortex AI Workbooks on Google Cloud exactly the tool to solve this?
- sneak 3y agoThis is what containers solve. Don't waste time manually installing things. Store state in a database via the app on a different host.
- ElectricalUnion 3y agoI used to think that the 300MB containers I shipped to prd were reasonably unoptimized. Those AMD ROCm containers are like 14GiB compressed.
- kunwon1 3y agoSpeaking as someone who has encountered similar difficulties, this response has strong 'Draw the rest of the owl' vibes
- sneak 3y agoSpeaking as someone who has solved these difficulties hundreds of times, "draw the rest of the owl" doesn't tell you the specific things to google to get detailed examples and tutorials on how millions of others have sidestepped these repeated issues.
- fennecfoxy 3y ago[dead]
- machinawhite 3y agoWell then you're wasting time managing your containers? Have you ever used k8s, it's a full time job lol
- zztop44 3y agoI don’t think anyone is suggesting k8s for running an ai model as a side project on a single machine.
- machinawhite 3y ago> a side project on a single machine Sorry I don't see how will containers help here
- itake 3y agoYep... you spend hours messing around with docker containers and debugging all the weird build errors. I am less familiar with storing data in a db (for ml hosting concerns), but I'd imagine it would add overhead (as opposed to accessing files on disk). You also have to deal with hosting a db and configuring the schema.
- sneak 3y agoYou "spend hours messing around" with everything you don't know or understand at first. One could say the same about writing the software itself. At its core Dockerfiles are just shell scripts with worse syntax, so it's not really that much more to learn. Once you get it done once, you don't have to screw around with it anymore, and you have it on any box you want in seconds. In either case you have to spend hours screwing around with your environment. If those hours result in a Dockerfile, then it's the last time. If they don't, then it's each time you want it on a new host (which as was correctly pointed out a pain in the ass). Storing data in a database vs in files on disk is like application development 101 and is pretty much a required skill period. It's required that you learn how to do this because almost all applications revolve around storing some kind of state and, as was noted, you can't reasonably expect it to persist on the app server without additional ops headaches. Many people will host dbs for you without you having to think about it. Schema is only required if you use a structured db (which is advisable) but it doesn't take that long.
- ElectricalUnion 3y ago> Storing data in a database vs in files on disk is like application development 101 It's OK until you're dealing with, say, 130GiB of tensors, on what is effectively a binary blob that needs to be mostly in VRAM somehow. I really don't want to read 130GiB of blobs from a database all the time.
- novok 3y agoIMO tensors & other large binary blobs are fairly edge-casey. You might as well treat them like video files and video file servers also don't store large videos in databases either, and most devs don't have large binary blob management experience.
- ciberado 3y agoI've played a lot with Stable Diffusion using AWS spot instances, mostly because it is the platform with which I'm more familiar. The Terraform script[0] should be easy to adapt to any other project of this kind. Let me know if you are interested, and maybe we can find time to work on it together :). [0] https://github.com/ciberado/stable-diffusion-webui-terraform-spot https://github.com/ciberado/stable-diffusion-webui-terraform...
- fnordpiglet 3y agoaws s3 sync + image snapshot
- itake 3y agoall that costs time :-/ When you're finished working, you got to wait for the syncs to finish (or I guess run them all the time?). For home computer setups, you can simply walk away when you need a break.
- fnordpiglet 3y agoNo, you write a script that runs the sync and shut down the instance. When for instance is stopped you don’t pay for it. Resuming it is a simple api call. You don’t even really need to do the sync, it’s just to be certain if the instance volume is lost you have a backup. The shutdown / stop on an instance is like closing the lid on your laptop. When you start it again it resumes where it was left off. In the mean time the instance doesn’t occupy a VM. A caveat is you can’t really do this with spot instances. You would need to do a sync and rebuild on start. But, again, scriptable easily.
- pdntspa 3y agoDont forget you can play computer games on it too!
- mackman 3y agoIf you shut down an ec2 instance you only pay for the volume not the compute. Just shut it down and start it up the next day.