7 ms·
Workers AI: Serverless GPU-powered inference
- jgrahamc 3y agoTL;DR: GPUs all over the Cloudflare global network; working closely with Microsoft, Meta, Hugging Face, Databricks, NVIDIA; new Cloudflare-native vector database; inference embedded in Cloudflare Workers; native support for WebGPU. Live demo: https://ai.cloudflare.com/ https://ai.cloudflare.com/
- ushakov 3y agoDo you actually run the inference in the worker? Or is it like what Fermyon does where they basically host the models for you and you get a SDK that is automatically connected to the function?
- celso 3y agoUnlike the first version of Constellation, Workers AI runs inference directly on GPUs that we are (quickly) installing in our global network.
- ushakov 3y agoBut the code isn't running on the worker? It runs somewhere else on a GPU cluster?
- jgrahamc 3y agoIt's a little like how Cloudflare Workers runs. You don't know which CPU it runs on, all you know is it's a CPU close to your end user. Same goes for this. We are rolling out GPUs everywhere across the globe and so Workers AI will just use a nearby GPU. Probably in the same machine as your workers, or maybe the same data center, or whatever other smart routing decision we make. What we are not doing is running a massive GPU cluster somewhere. This is all distributed and that's the power of owning your own network.
- pseg134 3y agoSince they don’t seem to be able to give a simple answer: the inference does not run in the worker. It connects to external GPUs.
- eastdakota 3y agoI think the confusion is what is meant by "in the Worker." From a hardware perspective, the GPU may be in the same machine as the CPU that's powering the Worker. Or they may be across different machines in our network. We are not routing requests to some third party. And we will try to run the inference task as close as possible to who/whatever requested it. The whole idea of "serverless" is you shouldn't have to worry about what machine where runs whatever unless you're on the team building the scheduling and routing logic at Cloudflare.
- thegagne 3y agoI think his question is more about does the worker directly access the GPU and thus require js tooling to handle the GPU somehow (no), or does it make subrequests to a separate GPU service not running the worker runtime (yes).
- aprxi 3y agoIs CodeLlama somewhere on the roadmap?
- foggedb0nk 3y agoAny chance you're looking for technical product folks to work on this? I actually worked on a very similar deployment internally at Livepeer (focus was on live video enhancements but also generalized edge compute)!
- rita3ko 3y agowe always are! email is rita at cloudflare dot com :)
- foggedb0nk 3y agothanks!
- tebbers 3y agoHey John, great work on this! Just a headsup, small typo on that page under R2: "Build mutli-cloud training architectures with free egress."
- jgrahamc 3y agoThanks. Getting it fixed.
- claytonjy 3y agoI see plans for more models via HF partnership, but can I or will I be able to run a custom fine-tuned version of a supported model?
- celso 3y agoOn top of our hosted and supported catalog of models, and the deploy to CF partnerships like the HF one, you will also be able to bring your own custom model at some point in time.
- claytonjy 3y agoAwesome. What about compiled model support? Running most of the listed models without compilation only makes sense for hobby projects.
- rtcode_io 3y agoWe were building AI https://efn.kr/#ai https://efn.kr/#ai into https://RTCode.io https://RTCode.io and ... Cloudflare drops this! Sweet! Now, we have BYOAI. Our whole offering runs on their network https://RTEdge.net https://RTEdge.net Our playground lets one live-code user workers that deploy to Cloudflare Worker for Platforms! - https://sw.rt.ht/?io https://sw.rt.ht/?io (in-browser) - https://sw.rt.ht/ https://sw.rt.ht/ (region-Earth)
- deleted 3y ago[deleted]
- ignoramous 3y agoAll well and good, but why is the webpage loading this obfuscated javascript file? https://archive.is/htQgN https://archive.is/htQgN
- cetinsert 3y agoCEO here, simply because we want to protect the core components that differentiate our services. Similar to, say https://www.photopea.com/ https://www.photopea.com/ and many others we can find if we look behind the scenes. Once we raise funding, and establish strong market presence, we will revisit this decision, and dedicate developer resources to sharing our in-house tech with the world more openly. This will take a full position. If you liked what you see, and want to work with us, send us an email at work@elefunc.com and we will get in touch once we have open positions!
- mmahemoff 3y agoLooks interesting, but as feedback, the above-the-fold message on your homepage is buzzword salad, not really clear what you're offering a potential customer here. "Elefunc is building the real-time web, to empower thought-speed creativity." By way of comparison, I know exactly what I'm getting as a developer on Replit's homepage: "Make something great. Build software collaboratively with the power of AI, on any device, without spending a second on setup"
- 3y ago
- siwakotisaurav 3y agoThe biggest one missing is stable diffusion.
- rita3ko 3y agostay tuned!
- deleted 3y ago[deleted]
- foggedb0nk 3y agoAny chance you're looking for technical product folks to work on this? I actually lead a very similar deployment internally at Livepeer (focus was on live video enhancements but also generalized edge compute)!
- jgrahamc 3y agoYou can always email jgc@cloudflare.com and I'll route the resume to the right people.
- foggedb0nk 3y agothanks!
- thyrox 3y agoThis would be a game changer if they had something for image generation as well. Oh well, maybe it's coming soon as the page says it's just a small preview. The best part for me personally is it's available on all plans and pricing looks good too. OT but if cloudflare fixes their false positives on showing random captchas when I'm trying to browse the net on VPN they will surely be one of my favorite companies.
- celso 3y agoImage generation is coming.
- arbitrarian 3y agoIt will be interesting to see if they can undercut OpenAI themselves on the cost for running Whisper in the cloud.
- bobjmiles 3y agohttps://blog.salad.com/whisper-large-v2-benchmark/ https://blog.salad.com/whisper-large-v2-benchmark/
- danskeren 3y agoAre OpenAI still limiting Whisper to 50 requests per minute (and gpt-3.5-turbo at 3/min)? If so then they don't need to undercut OpenAI but just provide unlimited requests.. it's near impossible to provide user-facing AI solutions to customers due to these limits, and it's a very bad user experience (and security risk) to force users to provide their own OpenAI API keys. https://replicate.com/ https://replicate.com/ is much better with an average 10 requests per second but would still very much prefer unlimited (where they can monitor high volume users to see if it's legitimate), or add new pricing model where e.g. 0-10k/rpm is at $0.001/sec, 10-100k/rpm is at $0.010/sec, 100k+/rpm is at $0.100/sec (pricing would of course need to be fine-tuned, just a quick example).
- bredren 3y agoI tried serverless for whisper on an existing competitive service. It had a cold boot and run on a 8 word STT time of 45 seconds and warm never got past 15 seconds. This does not work for STT, where it has to be much faster turnaround. Can anyone give feedback on if whisper and any of its model sizes can work well on serverless? Do most AI serverless solutions suffer from significant cold boot delays? The cheapest persistent GPU cloud instance I saw on G was ~$160 a month. Is that roughly the kind of money people need to be prepared to spend to have a model ready to go at all times as a service to another product?
- FL33TW00D 3y agoWhisper large is only 1.5B params, why not run it client side with something like https://github.com/FL33TW00D/whisper-turbo https://github.com/FL33TW00D/whisper-turbo (Disclaimer: I am the author)
- bredren 3y agoSeems like webgpu is not supported by mobile safari yet. And possibly has coverage in only 65% of the desktop browser market. [1] Does that roughly conform to how you understand the current penetration of this browser api? Presuming coverage for a given user, I don’t have a good answer for why to consider remote. It seems like it would be worth testing compatibility for webgpu and attempting to run on the client if possible, but then have a remote instance available otherwise. Does that make sense to you? Can you tell me another reason why someone would want a remote instance of whisper given 20x realtime potential at client in your project? [1] https://caniuse.com/webgpu https://caniuse.com/webgpu
- FL33TW00D 3y agoIndeed WebGPU is basically only supported on Chromium based browsers. This means that the primary usecase for whisper-turbo and my upcoming libraries is Electron/Tauri apps. For users that don't have WebGPU supported for whatever reason, we will still hit OAI/other server deployment. In the ideal case there should be a 90% cost reduction and same or improved UX. Someone will still want a remote instance today as there is still engineering to be done. I need more aggressive quantization, better developer experience and more features in order to get people off of the OAI API.
- mistrial9 3y ago[flagged]
- deleted 3y ago[deleted]
- xfalcox 3y agoEmbedding cost and model choice makes this a very compelling choice. I'm working on leveraging embeddings in https://github.com/discourse/discourse-ai https://github.com/discourse/discourse-ai where it powers offering related topics, semantic search, tag and category recommendations among other things. A cheap offering like this can make it a lot more reasonable for self-hosters.
- 0xDEF 3y agoHow well-proteced is the "edge" in edge computing? I can see Cloudflare has edge locations in many countries. Can entities with physical access to Cloudflare's edge machines get access to sensitive user data?
- celso 3y agohttps://blog.cloudflare.com/securing-memory-at-epyc-scale/ https://blog.cloudflare.com/securing-memory-at-epyc-scale/ https://blog.cloudflare.com/speeding-up-linux-disk-encryption/ https://blog.cloudflare.com/speeding-up-linux-disk-encryptio...
- r1ch 3y agoWith Cloudflare's default settings, a malicious entity can intercept any Cloudflare <-> Backend connections invisibly to the end user since the SSL certificates aren't validated. The end user also can be victim to plain old HTTP MITM on Cloudflare's upstream networks, as happened in 2016: https://news.ycombinator.com/item?id=12091900 https://news.ycombinator.com/item?id=12091900 It's hard to take Cloudflare's commitment to security seriously when they still ship such terrible default settings.
- NicoJuicy 3y agoWhat do you mean? You can install certificates by cloudflare and then the only one that can connect to your server is from cloudflare. No one can intercept it then. If you're talking about flexible SSL. Sure, you can use it purely as a https proxy for your SEO score of your blog. But securing it is not much effort. If it's just for a static blog, I'm not sure what you would though.
- r1ch 3y agoIf you have a valid HTTPS certificate for example.com and then add example.com to Cloudflare, your overall security decreases because the path from the CF datacenter to your origin is now vulnerable to MITM - the default SSL setting is "Full" which doesn't check certificate validity. To the less experienced sysadmin everything looks like it's working fine and users also don't notice any difference, which is why it's a terrible default. Sure you _can_ configure Cloudflare securely, but it should be secure out of the box. But that adds friction when the origin doesn't have a valid SSL certificate which probably hurts someone's KPIs.
- winddude 3y agokinda cool, but rather limited if you can't use custom models.
- amayne 3y agoThis is very cool. I'm still trying to understand the pricing. What is a "neuron" in this context? A token? A character? "Neurons are a way to measure AI output that always scales down to zero (if you get no usage, you will be charged for 0 neurons). To give you a sense of what you can accomplish with a thousand neurons, you can: generate 130 LLM responses, 830 image classifications, or 1,250 embeddings." 130 LLM responses of that length? 1,250 embeddings what size of text?
- eastdakota 3y agoIt's effectively a unit of time benchmarked to what we can accomplish in that time as of Sept 27, 2023 (launch). The challenge here is that because we're abstracting away the underlying hardware it's not the same as renting a VM for a period of time. We also don't want to create perverse incentives that keep us from making the underlying system faster. It's similar to how AWS standardized EC2 to a standard compute unit. Over time, as we continue to add faster and faster hardware and better optimize models we expect the cost of a neuron will trend down but the amount of AI inference work that you can do with a neuron will remain relatively constant.
- KRAKRISMOTT 3y agoPlease rename it, or at least make sure it corresponds to actual neural operations. It's terribly confusing for practitioners.
- coolspot 3y agoSounds like 1 Neuron ~= X FLOPS
- esafak 3y agoThen call it something like Neural Time Unit (NTU) or Computational Time Unit (CTU) because neurons make people think of neural networks. As in, you pay for the size of your model.
- choppaface 3y agoReally amazing stuff to see this launch with Hugginface! Hope to see it expand beyond text too. “neuron” is a cute name but there’s too much conceptual overlap with floating point ops, layers, model parameters etc which are time independent. Should just call them inference credits or something. When some large model runs on multiple GPUs it’s even more confusing what neurons / dollars per second might be.
- latchkey 3y agoInteresting how the same exact blog url was used previously in April 2021. https://news.ycombinator.com/item?id=26795517 https://news.ycombinator.com/item?id=26795517
- eastdakota 3y agoThat was our early alpha cooperation with NVIDIA. We've learned a lot since then. Not to mention, the AI ecosystem has grown up a bunch. But you are correct: this is something we've been planning for for a loooooooong time.
- latchkey 3y agoI'd love to see a blog post about the knowledge delta and growth progress. That's more interesting to me than the actual announcements.
- willquack 3y agoI've never played with Cloudflare workers, but I thought they were implemented as JavaScript runtimes that form an edge computing network Are the models run in JavaScript/WebAssembly behind the scenes?
- celso 3y agoYou can use Javascript or Wasm to interface with the AI binding, think of it of as the SDK, but the inference task itself runs natively on top of a ML runtime and the models are loaded into GPUs.
- ryandetzel 3y agoI want to love workers but I've never had great luck with anymore more than a basic crud app. Even getting an external db to connect proved to be more work than it should have, and their docs are all outdated and all over the place, often contradicting themselves.
- throwaway77384 3y agoThe docs are soooooo bad. This is the same story with any exciting / new product. Honestly, I'm beginning to think that we need some kind of documentation-first style development. Like TDD, but DDD.... I have integrated Facebook APIs, Instagram (well, same thing), Google APIs, Stripe APIs, Mailchimp APIs, etc. etc. And the only thing common among all of them? The documentation is _terrible_..., like, _terrible_. I also run a few products online and I spend so, so much time trying to get the documentation right. It's incredibly boring and tedious, but I really feel that if you want to set yourself apart from the big players, make good documentation. It can't be that hard.
- kristopolous 3y agogood documentation is the key to php's success along with a bunch of other things that get dismissed as "inferior technology". The worst documentation are the jargon filled abstract vibes-based ones where the authors basically typed it with one hand on the keyboard. It's like "ok, you're amazing. Now how do I resolve this error and what are your command line flags?"
- celso 3y agoWhat do you think is missing in the docs?
- jonplackett 3y agoVery cool and also very simple as I’d expect from Cloudflare. But I have a question - why not make inference as easy as the translation? Why do I have to run that in a worker rather than just as a simple API call? That would be much simpler. Is there a technical reason or is it that people would want to have logic before making the call to llama?
- rita3ko 3y agoyou can do both! all of our models are supported either via workers / pages binding (which makes it really easy to host the rest of the logic), and via REST API docs: https://developers.cloudflare.com/workers-ai/get-started/rest-api/ https://developers.cloudflare.com/workers-ai/get-started/res... (llama specific example here too under curl: https://developers.cloudflare.com/workers-ai/models/llm/ https://developers.cloudflare.com/workers-ai/models/llm/ )
- jonplackett 3y agoAwesome! Thanks for that. Cloudflare smashing it on simplicity as per usual.
- seejayseesjays 3y agoIt would be super cool to see SD running on this! hyped to play around with llama since I don't have access to a good GPU.
- UnlockedSecrets 3y agoI am having quite a few issues getting the reference API code at https://developers.cloudflare.com/workers-ai/models/llm/ https://developers.cloudflare.com/workers-ai/models/llm/ to work? {'errors': [{'code': 'invalid_union', 'unionErrors': [{'issues': [{'code': 'invalid_type', 'expected': 'object', 'received': 'string', 'path': ['body'], 'message': 'Expected object, received string'}], 'name': 'ZodError'}, {'issues': [{'code': 'invalid_type', 'expected': 'object', 'received': 'string', 'path': ['body'], 'message': 'Expected object, received string'}], 'name': 'ZodError'}], 'path': ['body'], 'message': 'Invalid input'}], 'success': False, 'result': {}}
- pdwittig 3y agoSorry for the trouble. The docs has since been updated - you should try again. Also happy to help if you run into any other issues - pwittig at cloudflare dot com.
- NicoJuicy 3y agoAt this cost, no one will be running embeddings elsewhere :o https://twitter.com/eastdakota/status/1707056412575023352?t=zkWdK8OCZJGaF1IIo6jAjA&s=19 https://twitter.com/eastdakota/status/1707056412575023352?t=...
- showerst 3y agoI tried to spin up a free plan and run the whisper demo on a new worker and it immediately just gives me an: Error 1102 Worker exceeded resource limits Did I mess up the config or is it just not intended to be tried out without being on a paid plan already?
- pdwittig 3y agoIt's 100% designed to let you try it out for free, so something else must be going on. Feel free to message me at pwittig at cloudflare dot com, and I'm happy to help debug. Also, we're still figuring some things out, but current limits are here: https://developers.cloudflare.com/workers-ai/platform/limits/ https://developers.cloudflare.com/workers-ai/platform/limits...
- deleted 3y ago[deleted]
- edunteman 3y agoGiven this is a hosted API rather than arbitrary hosting, why choose the word "serverless"? Do you plan to offer arbitrary hosting in the future? (bias: am Banana CEO)
- AmericanOP 3y agoHow does Workers AI compare to Replicate.com?
- pdwittig 3y agoThere are a lot of similarities, but here are a few differences: It's region-less, and runs your inference task on the Cloudflare network, near your end users. Though that's not entirely true yet - we'll be in 100 sites by EOY '23, and nearly everywhere by EOY '24. It was built to work alongside our new vector database, Vectorize, out of the box. It's accessible to all developers, regardless of where you deploy (via API), but we wanted to offer a seamless option for developers already building on Cloudflare - Workers, Pages, etc.
- baobabKoodaa 3y agoThis could become something useful in the future, but right now it appears to be toy models only. I'm assuming Cloudflare will add useful models eventually, but the cold start times are going to be horrible on those. I'm struggling to think of useful applications for this. Maybe one day.
- celso 3y agoWhich models would you find useful?
- baobabKoodaa 3y agoDepends on the task, but generally, LLaMA 2 finetunes of at least 13B params, 4-bit quantized
- celso 3y agodid you see we have lamma2-7b int8?
- baobabKoodaa 3y agoCan you provide an example of a task where the 7b model is useful?
- mountainriver 3y agoWhat’s the cold boot time? Isn’t that the most important part?
- ryanrussell 3y ago@jgrahamc Very curious if you can elaborate on Vectorize. More than edge GPU's, entering the Vector DB marketplace and a CF proprietary integration is interesting (and a bit scary) to build on. - Will Vectorize ever get OSS'd? - If you want to migrate either direction from some other Vector DB(Milvus, Weaviate, Qdrant, Pinecone, etc), what should you expect in terms of level of effort and features? - What inherent advantages(latency? features?) would you get exclusively from Vectorize?
- deanCommie 3y agoAWS too -> https://aws.amazon.com/blogs/aws/amazon-bedrock-is-now-generally-available-build-and-scale-generative-ai-applications-with-foundation-models/ https://aws.amazon.com/blogs/aws/amazon-bedrock-is-now-gener... > (Coming Soon) The Llama 2 13B and 70B parameter models by Meta will soon be available via Amazon Bedrock’s fully managed API for inference and fine-tuning.
- capableweb 3y agoUnless I missed something, it seems they didn't share which models by parameter count they're gonna be offering, nor anything at all about cost. Also, this strikes me as really weird phrasing: > Llama 2 now available for global usage on Cloudflare’s serverless platform, providing privacy-first, local inference to all "privacy-first & local inference" would be to run it locally, on your own hardware, isn't that exactly what should be referred by when using "local"? Or has the definitions of words completely gone out the window as of late?
- bhhaskin 3y ago100% marketing speak.
- warkdarrior 3y agoAt least you get privacy from Meta.
- capableweb 3y agoConsidering this is a collaboration, don't you think the agreement between them has something about feeding data back to Meta?
- NicoJuicy 3y agoI just think it's loading the models and revenue sharing. It would be similar as their partnership with Huggin Face, I suppose. What data would be useful to capture? Outside of a human feedback loop, I don't think there is an actual use-case for it. Note: not certain
- capableweb 3y agoWithout anyone of us actually knowing the details of the collaboration, all we can do is guess, I guess. > What data would be useful to capture? Pipe the prompts straight to Meta and I'm sure they'll be able to extract a ton of useful data.
- iAkashPaul 3y agoThought they were gonna go CPU based inference with llama.cpp/ggml, this project should definitely be made into a programming book
- NicoJuicy 3y agoI think they quickly reached the limits of CPU inference and that's why they decided to scale up with GPU's. Because of how they operate, they can't just release in a single DC like other providers. It's a partial "all or nothing" scenario.
- deleted 3y ago[deleted]
- jonplackett 3y agoCan you use this for _anything_ you like? Business uses es etc? Not sure what the terms are for Llama. I thought they were restricted in some way?
- ninjin 3y agoLLaMa 2 is sadly not open source and not open science, despite what Facebook keeps on claiming: https://blog.opensource.org/metas-llama-2-license-is-not-open-source https://blog.opensource.org/metas-llama-2-license-is-not-ope... Thus you will have to read the license and judge whether it is compatible with your business or use cases. One should give them credit for making it available, which is a lot better than plenty of others. However, actually open models are starting to appear, so perhaps we will soon see Facebook and the likes making theirs open as well? Who knows.
- deleted 3y ago[deleted]
- mohitgangrade 3y agoThis sounds really cool. I already love cloudflare because of how easy they make it to compete with bigger companies for indie devs like myself. Their pricing for their products always seems so much more affordable than something like AWS or GCP. I'm using R2 for storage for a client project, and from my calculations we would have to pay almost 3 times more if we hosted the files with AWS on S3. I really hope they keep adding all the latest open-source AI models to their platform. If their pricing will be as cheap as they state in this blog post, then I would rather use this service than install models on my computer. To get a good inference speed for open source models on my PC right now, I have to let usage spike to up to 100%...