19 ms·

Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

891 points by taubek 3y ago

salil999 3y ago
https://archive.ph/STnXb https://archive.ph/STnXb
coltongaodgd 3y ago
[dead]
r13a 3y ago
Could anyone explain how this can be constructed as a private solution? I'm not familiar with Azure platform. Is the inference processed on private instance ? I can't imagine how it could be feasible given the hardware required to run gpt3.5/4. So the best case scenario is: 1. A web ui runs on a private instances. So any user input (chat or files) are only seen by these instances 2. Any chat historisation or RAG is also done on these instances too. 3. Embeddings compuation may possibly be done on the private instance 4. The embeddings are then sent to the Microsoft GPU farm for inference. So at one point my data has to leave my private network. The problem is that the data can easily be retro-engineered from the embeddings. How can this be presented as a private LLM ?
albert_e 3y ago
is there away to run this on AWS instead. we were looking to explore Llama2 for internal use
- froggychairs 3y ago
  OpenAI models are exclusively Azure only. Llama2 should have an AWS option I believe?
- axpy906 3y ago
  Use SageMaker: https://www.philschmid.de/sagemaker-llama-llm https://www.philschmid.de/sagemaker-llama-llm
- villgax 3y ago
  Have your engineers set this up internally https://huggingface.co/spaces/huggingface-projects/llama-2-7b-chat https://huggingface.co/spaces/huggingface-projects/llama-2-7...
  - speedgoose 3y ago
    You can’t really replace ChatGPT 4 with llama2 7B.
    - villgax 3y ago
      Yeah right try getting the same answer after two months
- klysm 3y ago
  Msft spent a lot of money to ensure that was not an option w chatgpt
  - RockyMcNuts 3y ago
    https://about.fb.com/news/2023/07/llama-2/ https://about.fb.com/news/2023/07/llama-2/ https://huggingface.co/blog/hugging-face-endpoints-on-azure https://huggingface.co/blog/hugging-face-endpoints-on-azure You can of course run Llama2 in Azure, but you can't host OpenAI models in AWS
- braydenm 3y ago
  Amazon Bedrock makes Claude 2 available, as well as some other models.
- gdiamos 3y ago
  We can run llama 2 on an AWS vm if you have enough GPUs: https://lamini.ai/ https://lamini.ai/ Install in 10 minutes. Make sure you have enough GPU memory to fit your llama model if you want good perf
coldblues 3y ago
Pretty sure Azure has a moderation endpoint enabled by default that makes using the OpenAI API an awful experience.
Y_Y 3y ago
So the public access one isn't private and secure?
- froggychairs 3y ago
  I believe it’s implying the free ChatGPT collects data and this one doesn’t.
- nwoli 3y ago
  I thought sama said they don’t use data going through the api for training. Guess we can’t trust that statement
  - jumploops 3y ago
    That is correct, they do not use the data going through the API for training, but they do use the data from the web and mobile interfaces (unless you explicitly turn it off).
    - quickthrower2 3y ago
      “We don’t water down your beer”. Oh nice! “But that is lager”
- candiddevmike 3y ago
  > However, ChatGPT risks exposing confidential intellectual property. I don't remember seeing this disclaimer on the ChatGPT website, gee maybe OpenAI should add this so folks stop using it.
  - cmarschner 3y ago
    If you use ChatGPT through the app or website they can use the data for training, unless you turn it off. https://help.openai.com/en/articles/5722486-how-your-data-is-used-to-improve-model-performance https://help.openai.com/en/articles/5722486-how-your-data-is...
littlestymaar 3y ago
“private and secure” from the company that let contractor listen to your private Teams conversation for data labeling purpose, and monitor your activity on your own computer with their OS…
- tharwan 3y ago
  Huh. I missed this one. Got a link?
  - dijital 3y ago
    At a guess it's this story: https://www.vice.com/en/article/xweqbq/microsoft-contractors-listen-to-skype-calls https://www.vice.com/en/article/xweqbq/microsoft-contractors...
    - littlestymaar 3y ago
      Ah yes it was Skype and not Teams, my bad.
- svaha1728 3y ago
  Move fast and break things, including basic security. Why anyone trusts Azure that all these prompts won't eventually be leaked is beyond me. No one goes broke trusting Azure, but I'd love it if someone was held responsible. https://www.schneier.com/blog/archives/2023/08/microsoft-signing-key-stolen-by-chinese.html https://www.schneier.com/blog/archives/2023/08/microsoft-sig...
refulgentis 3y ago
Crappy clone of ChatGPT frontend, half missing, half direct copy. Implied and overly vast claims of insecurity + lack of privacy, that are narrowly true, i.e. for _Chat_GPT. Really surprised to see this aggressive of language 1) written down 2) on Github. I'd be pretty pissed if I was OpenAI, regardless of the $10B.
- jeremyjh 3y ago
  I have no doubt OpenAI is on board. This is just bringing more paid users to their platform because it still uses their API.
- sebzim4500 3y ago
  I think OpenAI is entirely on board with the idea that OpenAI sells to consumers and Azure/Microsoft sells the same product to enterprise. That's how it's been working for months, and if OpenAI objected they would have done something about it.
croes 3y ago
Yeah sure, I totally trust you after the Storm-0558 desaster
TuringNYC 3y ago
Curious if anyone has done a side-by-side analysis of this offering vs just running LLaMA? I'm currently running a side-by-side comparison/evaluation of MSFT GPT via Cognitive Services vs LLaMA[7B/13B/70B] and intrigued by the possibility of a truly air-gapped offering not limited by external computer power (nor by metered fees racking up.) Any reads on comparisons would be nice to see. (yes, I realize we'll eventually run into the same scaling issues w/r/t GPUs)
- FrenchDevRemote 3y ago
  chatgpt is obviously a LOT better, llama doesn't even understand some prompts and since LLMs aren't even that good to begin with, it's obvious you want the SOTA to do anything useful unless maybe you're finetuning
  - baobabKoodaa 3y ago
    > and since LLMs aren't even that good to begin with, it's obvious you want the SOTA to do anything useful unless maybe you're finetuning This is overkill. First of all, ChatGPT isn't even the SOTA, so if you "want SOTA to do anything useful", then this ChatGPT offering would be as useless as LLaMA according to you. Second, there are many individual tasks where even those subpar LLaMA models are useful - even without finetuning.
    - FrenchDevRemote 3y ago
      it's the SOTA for chat(prove me wrong), and you can always use the API directly even for simple tasks they're less reliable and needs more prompt engineering
      - baobabKoodaa 3y ago
        > it's the SOTA for chat(prove me wrong) GPT-4 beats ChatGPT on all benchmarks. You can easily google these.
villgax 3y ago
Yeah right for the three letter agencies to have a backdoor, hard pass on something that cannot be deterministic with a seed
PaulWaldman 3y ago
Since the only users who would likely care about this derive far more value than the $20/month of OpenAI's direct offering. Why doesn't OpenAI market this service, but with chat history, for something like $200/month?
- semitones 3y ago
  That's a laughable price for an enterprise subscription. And the reason is, it's enough for OpenAI to "say" that they're "not going to use your data" - you need a cloud deployment where you can control network boundaries to _prove_ that your data isn't going anywhere it isn't supposed to.
  - agildehaus 3y ago
    Unless you're physically controlling the network boundaries, how are you proving that on any cloud service?
- unnouinceput 3y ago
  OenAI IS Microsoft. Don't get tangled in the web of creating different entities when they are all part of the same pyramid. Also GitHub IS Microsoft too!!
  - nixgeek 3y ago
    GitHub was acquired by Microsoft, and they are no longer legally separate entities. Microsoft is an investor in OpenAI, but does not own it, and they are legally separate companies. OpenAI is not Microsoft and it is factually incorrect to claim that OpenAI is Microsoft. [1] https://blogs.microsoft.com/blog/2023/01/23/microsoftandopenaiextendpartnership/ https://blogs.microsoft.com/blog/2023/01/23/microsoftandopen...
    - Scoundreller 3y ago
      But saying they're just an investor isn't quite doing the arrangement the justice it deserves. There seems to be a lot of strings attached to that investment. It's not just a straight trade of dollars for shares, but many further contractual obligations.
robbomacrae 3y ago
This is potentially a huge deal. Companies are concerned using ChatGPT might violate data privacy policies if someone puts in user data or invalidate trade secrets protections if someone uploads sections of code. I suspect many companies have been waiting for an enterprise version.
- deleted 3y ago
  [deleted]
- judge2020 3y ago
  I imagine most companies serious about this created their own wrappers around the API or contracted it out, likely using private Azure GPUs.
  - Normal_gaussian 3y ago
    Most companies are either not tech companies, or do not have the knowledge to manage such a project within reasonable cost bounds.
    - jmathai 3y ago
      Most companies are trying to figure out exactly what generative AI is and how to use it in their business. Given how new this is - I doubt any large company has done much besides ban the public ChatGPT. So this is probably very relevant for them.
- tbrownaw 3y ago
  This is a web UI that talks to a (separate) Azure OpenAI resource that you can deploy into your subscription as a SaaS instance.
  - hackernewds 3y ago
    So how is it any different
bouke 3y ago
How is this different from the other OpenAI GUI? Why another one by Microsoft? https://github.com/microsoft/sample-app-aoai-chatGPT https://github.com/microsoft/sample-app-aoai-chatGPT.
- FrenchDevRemote 3y ago
  i'm pretty sure it's a part of it
- pjmlp 3y ago
  I bet there are plenty of OKR/KPIs now tied to AI at Microsoft.
- wodenokoto 3y ago
  There's at least two more. There's also https://github.com/Azure-Samples/azure-search-openai-demo https://github.com/Azure-Samples/azure-search-openai-demo And you can deploy a chat bot from within the Azure playground which runs on another codebase.
- colonwqbang 3y ago
  Bigger companies are cautious about using GPT-style products due to data security concerns. But most big companies trust Microsoft more or less blindly. Now that Microsoft has an official "enterprise" version out, the floodgates are open. They stand to make a killing.
- pamelafox 3y ago
  This is an internal ChatGPT, whereas that sample is ChatGPT constrained to internal search results (using RAG approach). Source: I help maintain the RAG samples.
paxys 3y ago
Would it be too much to mention somewhere in the README what this repo actually contains? Just docs? Deployment files? Some application (which does..something)? The model itself?
- Xenoamorphous 3y ago
  The repo contains the UI code, not the model or anything else around ChatGPT, it just uses Azure’s ChatGPT API which doesn’t share data with OpenAI.
  - paxys 3y ago
    So basically – what you really need to do to run Azure ChatGPT is go and click some buttons in the Azure portal. This repo is a sample UI that you could possibly use to talk to that instance, but really you will probably always build your own or embed it directly into your products. So calling the repo "azurechatgpt" is misleading. It should really be "sample-chatgpt-api-frontend" or something of that sort.
    - saliagato 3y ago
      Yes exactly
    - laurels-marts 3y ago
      Correct. If offers a front-end scaffolding for your enterprise ChatGPT app. Uses Next/NextAuth/Tailwind etc. for deployment on Azure App Service that hooks into Azure Cosmos DB and Azure OpenAI (the actual model).
  - deleted 3y ago
    [deleted]
  - wodenokoto 3y ago
    Isn’t there also some sort of backend stuff in there? How else would it keep track of history and accept documents. I don’t know enough typescript to understand where the front end stops and the backend begins I this code
gdiamos 3y ago
Can you fine tune it?
- jensen2k 3y ago
  Yes! You can.
  - gdiamos 3y ago
    Is it the same api as the public OpenAI
  - saliagato 3y ago
    How?
xeckr 3y ago
No better than the API.
Ecstatify 3y ago
Our company is pushing everyone to use a similar offering. Most of the company is doing low value work … still using excels even though we have a custom ERP. Now seeing people who couldn’t write a coherent email before write 3 page emails. The illusion of being productive by doing more work even though it has zero impact on the bottom line. It’s insane how inefficient organisations are. No doubt we’ll have some KPI soon about using the tool.
- simmerup 3y ago
  If anything it's less productive because people have to parse all that nonsense. I was gobsmacked to hear a friend say that their work guidance is to use ChatGPT to write letters to external clients for example. I know for sure I'd be insulted if someone sent me paragraphs of text to read created from a sentence long prompt. I'd rather have the prompt, my time is valuable as well.
  - mritchie712 3y ago
    ahhhh, but they're pasting the 3 page email into ChatGPT ("summarize this"). The future is here.
    - ilyt 3y ago
      Wouldn't be surprised if that was next Outlook feature. Cue someone making some horrible error because some crucial information didn't survive ChatGPT->ChatGPT round-trip
      - mritchie712 3y ago
        it's already here... https://blogs.microsoft.com/blog/2023/03/16/introducing-microsoft-365-copilot-your-copilot-for-work/ https://blogs.microsoft.com/blog/2023/03/16/introducing-micr...
      - ddmma 3y ago
        Actually this was in an Azure hacktoon some time ago https://devpost.com/software/amabot https://devpost.com/software/amabot
Xenoamorphous 3y ago
Darn I just spent a week or so working on a ChatGPT clone that used Azure ChatGPT API due to the privacy aspect. Wasted effort I guess.
- saliagato 3y ago
  This is exactly the same
  - ddmma 3y ago
    Welcome to the club :)
    - jeremycarter 3y ago
      I'm also in this club but we wrote it months ago.
jrm4 3y ago
"Private and secure" From Microsoft? Ha.
gdiamos 3y ago
we wrote a blog post about why companies do this here: https://www.lamini.ai/blog/specialize-llms-to-private-data-databricks https://www.lamini.ai/blog/specialize-llms-to-private-data-d... Here are a few: Data privacy Ownership of IP Control over ops The table in the blog lists the top 10 reasons why companies do this based on about 50 customer interviews.
ajhai 3y ago
A lot of companies are already using projects like chatbot-ui with Azure's OpenAI for similar local deployments. Given this is as close to local ChatGPT as any other project can get, this is a huge deal for all those enterprises looking to maintain control over their data. Shameless plug: Given the sensitivity of the data involved, we believe most companies prefer locally installed solutions to cloud based ones at least in the initial days. To this end, we just open sourced LLMStack (https://github.com/TryPromptly/LLMStack https://github.com/TryPromptly/LLMStack) that we have been working on for a few months now. LLMStack is a platform to build LLM Apps and chatbots by chaining multiple LLMs and connect to user's data. A quick demo at https://www.youtube.com/watch?v=-JeSavSy7GI https://www.youtube.com/watch?v=-JeSavSy7GI. Still early days for the project and there are still a few kinks to iron out but we are very excited for it.
- toomuchtodo 3y ago
  Can you plug this together with tools like api2ai to create natural language defined workflow automations that interact with external APIs?
  - ajhai 3y ago
    There is a generic HTTP API processor that can be used to call APIs as part of the app flow which should help invoke tools. Currently working on improving documentation so it is easy to get started with the project. We also have some features planned around function calling that should make it easy to natively integrate tools into the app flows.
  - cosbgn 3y ago
    You can use unfetch.com to make API calls via LLMs and build automations. (I'm building it)
    - scrum-treats 3y ago
      Is it possible to not use Google with unfetch.com?
extr 3y ago
One thing I still don't understand is what _is_ the ChatGPT front end exactly? I've used other "conversational" implementations built with the API and they never work quite as well, it's obvious that you run out of context after a few conversation turns. Is ChatGPT doing some embedding lookup inside the conversation thread to make the context feel infinite? I've noticed anecdotally it definitely isn't infinite, but it's pretty good at remembering details from much earlier. Are they using other 1st party tricks to help it as well?
- MaxLeiter 3y ago
  It uses a sliding context windows. Older tokens are dropped as new ones stream in
  - extr 3y ago
    I don't believe that's the whole story. Other conversational implementations use sliding context windows and it's very noticable as context drops off. Whereas ChatGPT seems to retain the "gist" of the conversation much longer.
    - lsaferite 3y ago
      I mean, I explicitly have the LLM summarize content that's about to fall out of the window as a form of pre-emptive token compression. I'd expect maybe they do something similar.
      - kuchenbecker 3y ago
        I feel like we're describing short vs long term memory.
        tsunamifury 3y ago
        That’s exactly what it is. It’s just it turns out you need very good generalized or focused simple reasoning to do accurate compression or else the abstraction and movement to long term memory doesn’t include the most important content. Or worse distracting details. I’ve been working on short and long term memory windows at allofus.ai for about 6 months now and it’s way more complex than I had originally thought it would be. Even if you can magically extend the content window, the added data confuses and waters down the reasoning of the LLM. You must do layered abstraction and compression with goal based memory for it to continue to reason without distraction of irrelevant data. It’s an amazing realization, almost like a proof that memory is a kind of layered reasoning compression system. Intelligence of any kind can’t understand everything forever. It must cull the irrelevant details, process the remains and reason on a vector that arises from them.
RockyMcNuts 3y ago
so...how can we make this support plugins like Code Interpreter, Wolfram, Zapier or Workato, and whatnot?
leerob 3y ago
This is awesome to see, feels heavily inspired (in a good way) by the version we made at Vercel[1]. Same tech stack: Next.js, NextAuth, Tailwind, Shadcn UI, Vercel AI SDK, etc. I'd expect this trend of managed ChatGPT clones to continue. You can own the stack end to end, and even swap out OpenAI for a different LLM (or your own model trained on internal company data) fairly easily. [1]: https://vercel.com/templates/next.js/nextjs-ai-chatbot https://vercel.com/templates/next.js/nextjs-ai-chatbot
- vladiliescu 3y ago
  Really, they use Vercel AI SDK?
  - leerob 3y ago
    Well, they did. The repo appears to have been deleted.
mark_l_watson 3y ago
This seems like such an obvious thing to do. I see the use of general purpose LLMs like ChatGPT, but smaller fine tuned models will probably end up being more useful for deployed applications in most companies. Off topic, but I was experimenting with LLongMA-2-7b-16K today, running it very inexpensively in the cloud, and given about 12K of context text it really performed well. This is an easy model to deploy. 7B parameter models can be useful.
- stavros 3y ago
  Is there an easy way to play with these models, as someone who hasn't deployed them? I can download/compile llama.cpp, but I don't know which models to get/where to put them/how to run them, so if someone knows about some automated downloader along with some list of "best models", that would be very helpful.
  - robertnishihara 3y ago
    If you want to try out the Llama-2 models (7B, 13B, 70B), you can get started very easily with Anyscale Endpoints (~2 min). https://app.endpoints.anyscale.com/ https://app.endpoints.anyscale.com/
  - tayo42 3y ago
    For llama, the 4bit quantized ones, small models like the 7b one. The ggml format. That will run on your local cpu. Google those terms too. you can look on hugging face for the actual model to download then load it and send prompts to it
    - stavros 3y ago
      Thanks, maybe it's as easy as downloading the ggml and running it with Llama.cpp. I'll try that, thanks!
      - tayo42 3y ago
        there is also a python wrapper that has a web ui built in for llama.cpp, if it wasnt easy enough already
Havoc 3y ago
How does this work in terms of utilization? The isolation presumably means buying gpu capacity and only using a %?
- asabla 3y ago
  Basically you get N tokens/second (or if it was minute, can check tomorrow if you're really interested) per deployment. So if you would outgrow on deployment, just deploy another one (with the associated costs of course). One deployment = a deployed model which you can query On top of that, depending on the model you're using, you also see a cost increment for each 1000 request you make.
  - Havoc 3y ago
    Ah right so it is somewhat shared. Not like your own gpu type situation
    - weird-eye-issue 3y ago
      Why did you make that assumption?
justinlloyd 3y ago
Interesting release, though still lacking a few features I've had to resort building myself such as code summary, code base architecture summary, and conversation history summary. ChatGPT (the web UI) now has the ability to execute code, and make function callbacks, but I prefer running that code locally, especially if I am debugging. This latter part, conversation history summary, is something that ChatGPT web UI does reasonably well, giving it a long history, but a sentiment extraction and salient detail extraction before summarizing is immensely useful for remembering details in the distant past. I've been building on top of the GPT4 model and tinkering with multi-model (gpt4 + davinci) usage too, though I am finding with the MoE that Davinci isn't as important. Fine tuning has been helpful for specific code bases too. If I had the time I'd like to play with an MoE of Llama2, as a compare and contrast, but that ain't gonna happen anytime soon.
alpinemeadow 3y ago
We have this at IKEA for a while now. Not impressed, but funny to read the hallucinations.
- BoorishBears 3y ago
  I'd expect a company like IKEA to have the expertise to create interfaces specific to their workflows so hallucinations aren't an issue. Imo if you're making an open ended chat interface for a business, you're doing it wrong.
- riwsky 3y ago
  Have you considered instruction-tuning it with text, instead of just pictures?
padolsey 3y ago
I'm confused. If this is just a front-end for the OpenAI API then how does it remove the data privacy concern? Your data still ends up with Azure/OpenAI, right? It doesn't stay localized to your instance; it's not your GPU running the transformations. You have no way of knowing whether your data is being used to train models. If customer data is sensitive, I'm pretty sure running a 70B llama (or similar) on a bunch of A100s is the only way?
- dbish 3y ago
  Azure is hosting and operating the service themselves rather then for OpenAI, with all the security requirements that come with that. I assume this comes with different data and access restrictions as well and ability to run in secured instances (and nothing sent to OpenAI the company). Most companies use cloud already for their data, processing, etc. and aren’t running anything major locally, let alone ML models, this is putting trust in the cloud they already use.
  - nmstoker 3y ago
    Yes, this was my understanding.
  - padolsey 3y ago
    Ah that's fair. But it is my impression that the bulk of privacy/confidentiality concerns (e.g. law/health/..) would require "end to end" data safety. Not sure if I'm making sense. I guess microsoft is somehow more trustworthy than openai themselves... EDIT: what you say about existing cloud customers being able to extend their trust to this new thing makes sense, thanks.
    - PoignardAzur 3y ago
      Right. If I was an European company worried about, say, industrial espionage, this wouldn't be nearly enough to reassure me.
  - deleted 3y ago
    [deleted]
EGreg 3y ago
We just have to trust them and take their word for it? Or what? https://azure.microsoft.com/en-us/explore/trusted-cloud/privacy https://azure.microsoft.com/en-us/explore/trusted-cloud/priv... https://azure.microsoft.com/en-us/blog/3-reasons-why-azure-s-infrastructure-is-secure/ https://azure.microsoft.com/en-us/blog/3-reasons-why-azure-s... I guess I would trust them, since they're big and they make these promises and other big companies use them.
aantix 3y ago
I don't understand - chat with a file? I want to chat and ask about an entire body of knowledge - wiki pages, git commit diffs/messages, jira tasks.
PoignardAzur 3y ago
> However, ChatGPT risks exposing confidential intellectual property. One option is to block corporate access to ChatGPT, but people always find workarounds Pretty bold thing to say to your potential clients. "You can always tell your employees not to use our product, but they won't listen to you."
- pwarner 3y ago
  It's almost like employees might have their own computers?
H8crilA 3y ago
What's the practical difference between this and OpenAI API? All I can see is the same product but offered by a larger organization. I.e. they're more likely to get the security details right, and you can potentially win more in a lawsuit should things go bad.
- jeffschofield 3y ago
  A few months ago my team moved to Azure for capacity reasons. We were constantly dealing with 429 errors and couldn't get in touch with Open AI, while Azure offered more instances. Eventually got more from Open AI so we load balance both. The only difference is the 3.5 turbo model on Azure is outdated.
  - RockyMcNuts 3y ago
    you can ask for gpt-4, it took a while due to capacity constraints but we got it
- ebiester 3y ago
  Compliance and customer trust. Azure can sign a BAA, for example. If you are Building LLM capability on top of your SaaS, your customers want assurances about their data.
singingfish 3y ago
I was looking through our server logs the other day and spotted the openai bot going through our stuff ... however a decent bit of our content is now augmented by GPT ...
atlgator 3y ago
Is this a full, standalone deployment including GPT-3 (or whatever version) or just a secured frontend that sends data to GPT hosted outside the enterprise zone? Edit: Uses Azure OpenAI as the backend
jmorgan 3y ago
This appears to be a web frontend with authentication for Azure's OpenAI API, which is a great choice if you can't use Chat GPT or its API at work. If you're looking to try the "open" models like Llama 2 (or it's uncensored version Llama 2 Uncensored), check out https://github.com/jmorganca/ollama https://github.com/jmorganca/ollama or some of the lower level runners like llama.cpp (which powers the aforementioned project I'm working on) or Candle, the new project by hugging face. What's are folks' take on this vs Llama 2, which was recently released by Facebook Research? While I haven't tested it extensively, 70B model is supposed to rival Chat GPT 3.5 in most areas, and there are now some new fine-tuned versions that excel at specific tasks like coding (the 'codeup' model) or the new Wizard Math (https://github.com/nlpxucan/WizardLM https://github.com/nlpxucan/WizardLM) which claims to outperform ChatGPT 3.5 on grade school math problems.
- ttul 3y ago
  Llama 2 might by some measures be close to GPT 3.5, but it’s nowhere near GPT 4, nor Anthropic Claude 2 or Cohere’s model. The closed source players have the best researchers - they are being paid millions a year with tons of upside - and it’s hard to keep pace with that. My sense is that the foundation model companies have an edge for now and will probably stay a few steps ahead of the open source realm simply for economic reasons. Over the long run, open source will eventually overtake. Chances are this will happen once the researchers who are making magic happen get their liquidity and can start working for free again out in the open.
  - robertnishihara 3y ago
    > Llama 2 might by some measures be close to GPT 3.5, but it’s nowhere near GPT 4 I think you're right about this, and benchmarks we've run at Anyscale support this conclusion [1]. The caveat there (which I think will be a big boon for open models) is that techniques like fine-tuning makes a HUGE difference and can bridge the quality gap between Llama-2 and GPT-4 for many (but not all) problems. [1] https://www.anyscale.com/blog/fine-tuning-llama-2-a-comprehensive-case-study-for-tailoring-models-to-unique-applications https://www.anyscale.com/blog/fine-tuning-llama-2-a-comprehe...
gerdesj 3y ago
Well, your AChatGPT connection might be private but your Windows machine will leak like a sieve. It is embarrassing how needy the blasted things are about signing in via Azure/Microsoft, instead of a local or AD account. Even worse is the naff "choose how insidious you would like us to be" series of questions. How would you like your ads? Targeted or non targeted? How about not at all? Nope. In this day and age, exactly how private does anyone expect their comms/thoughts/files/data to be? I recall reading a recent MS EULA and it seems I have to say three Hail Marys every third Tuesday for using Arch Linux on my PCs. I could install Edge, and did but I don't like the nasty homepage - a bit right wing ... - why on earth is a browser pushing "news"? Its a browser. To be fair I had to dump all the homepage crap that Firefox pushed when I finally dumped anything to do with Chrome. Please don't use the words private and secure when you have your fingers crossed behind your back.
- xist 3y ago
  The article is referring to enterprise usage - and you're quoting all the consumer level attributes (aka cheap/sometimes subsidized version). At the enterprise level where this is intended to be ran, things are much diffrerent. If you're not aware of the differences or use cases, perhaps you're not the target audience who should be using or configuring it.
  - gerdesj 3y ago
    Why don't we give the willy waving a miss? Win 10 and 11 are steering you to cloud first, out of the box. That's fine if you like it, but I don't and quite a lot of my customers don't. The real problem is about data sovereignty. I'm a Brit and ... MS isn't.
    - xist 3y ago
      The article is about use in an enterprise. An enterprise runs professional/enterprise/ltsc versions which do NOT steer you to the cloud - what data sovereignty concerns have you seen in those editions of windows/server? They've gone through a lot of pains to ensure those concerns are taken care of for enterprises/governments so i'm curious the ones you think they missed. You can make the argument for their consumer editions sure, but that's a different product with different features, different price point for different users.
tananaev 3y ago
This is not ChatGPT. It's just a front end for Azure OpenAI APIs. Not sure why they're so blatantly use the trademark. They will probably have to rename it soon.
- mkinsella 3y ago
  Microsoft is a major investor in OpenAI. Guaranteed they worked with OpenAI on this and have partnership to use the trademarks.
- castrodd 3y ago
  Microsoft owns OpenAI so I doubt that they will be asked to rename this.
  - mythz 3y ago
    They'll only own 49% of shares.
- white_dragon88 3y ago
  [dead]
ChatGTP 3y ago
Upvoted, cannot wait for this, yes yes yes. The companies have been waiting for this.
37112741 3y ago
Nothing in the repo details how this addresses privacy concerns of running inference on someone else's LLM. To be isolated from other users of the service is not the same thing as having a private inference engine. > Private: Built-in guarantees around the privacy of your data and fully isolated from those operated by OpenAI. Do tell.
- RockyMcNuts 3y ago
  So where do you draw the line? No cloud instances, no cloud SQL like Snowflake, no Teams or Office 365, no S3/blob storage? Run everything on-prem like 10 years ago? It's only going to get more impossible. All that VC money going in at 100x revenue needs a return and they aren't going leave money on the table with full-featured open-source or CentOS type alternatives. All those data engineering startups, database providers with 'open-source' + cloud hosting, the 'open-source' is going to be just 'open' enough to claim there is some fallback for someone else to pick up the mantle using the community version, if the cloud version gets enshittified beyond reason. You're not going to even be able to run the full-featured software version on-prem because the economics of cloud are so much better. Unless you are writing and compiling your own code you are going to be out of luck if your privacy standard is that high. That war has been lost. And Web3 sure ain't gonna save you either.
  - 37112741 3y ago
    They should clearly spell out what is and is not "private". As it is we simply have a blurp about some undefined guarantees. And some comments here in thread saying "this is as close as you're going to get to local GPT" are deeply wrong. But then there is easy VC money (just like with ADs ..) and certain "clever" geeks throw social responsibility out the window as usual and are pushing all sorts of deeply invasive applications ("let our proxy for Microsoft hoover your inbox!") based on these undefined "Privacy guarantees". If we accept this just we accepted the very flawed solutions we were given by corporation regarding social networking and ads, we are going to be stuck with it, suffer the consequences, and there will be no incentive to develop alternatives that actually address issues and work. Homomorphic Encryption works. It just doesn't work very efficiently right now but that is an intellectual problem that can be solved if we push for actual privacy for this critical technology as it will be fully enmeshed in all parts of our lives. "Think of the children" if that helps.
Racing0461 3y ago
[dead]
polski-g 3y ago
This is not private. It's still hosted on Microsoft's cloud.
10g1k 3y ago
I did a unit of AI at university, and the front of the textbook contained a quote by some ye olde AI theorist, something like: "I'm not concerned that artificial intelligence will take over the world. I'm concerned that human intelligence has yet to do so."
crooked-v 3y ago
The big question: If this is truly secure and private, can people use it to generate things related to porn or violence?
- deleted 3y ago
  [deleted]
hieu229 3y ago
This is a neat project from Microsoft I've been building https://gasbyai.com https://gasbyai.com, a beautiful chat UI that support self-hosted, with ChatGPT plugins, extract content from pdf/url. GasbyAI supports Azure, OpenAI, and custom API endpoints in case you want to run with your own models
jhoechtl 3y ago
Can any one shed light on what "local" means? Local on my own private machine or local in my Azure Tenant?
- jzombie 3y ago
  I am pretty sure it means run the UI locally and access Azure-hosted ChatGPT. The environment vars seem to indicate that as well.
- wodenokoto 3y ago
  Assuming you are referring to this section: https://github.com/microsoft/azurechatgpt/blob/main/docs/3-run-locally.md https://github.com/microsoft/azurechatgpt/blob/main/docs/3-r... It means you run the front end (the chat-gui) and the backend code from the repo. This code connects to cosmo-db for uploading documents used for "chat with you pdf" and connects to an OpenAI instance on Azure for the chat inferrence.
osigurdson 3y ago
I'm a little confused by how the relationship works between OpenAI and Microsoft. It is possible for anyone to register for an OpenAI account and use their APIs. Within Azure the same thing is much more difficult as it is necessary to be a "real" business in order to use it. I maintain an open source OpenAI library and would like to add support for Azure but can't because of this restriction. Why can't I just use my regular Azure account?
- devjab 3y ago
  Microsoft owns enough of OpenAI that their endgame goal of putting GPT like features into Azure and Office365 for enterprise customers is what we’re likely to see happen. OpenAI will likely target private consumers while Microsoft focuses on enterprise. I can use my own organisation as an example. We’re an investment bank that does green energy within the EU. We would absolutely use GPT if it was legal, but it isn’t, and it likely never will be considering their finance model is partly to steal as much data as they can. Even if it’s not so polite to say that. This is where Microsoft comes into the picture. In non-tech enterprise you’re buying Microsoft products because everyone wants windows, outlook and office. We can wish it wasn’t like that, but where is the realistic alternative? I’m not anti Microsoft by the way, in all my decades in the enterprise business they’ve easily been the best and most consistent business partner for any IT. When Amazon saw how much money there was on the operations side of EU enterprise they quickly caught up, but Amazon doesn’t sell a Office365 product. So anyway, once you have Office365, you’re also likely to use Teams as your communications platform (which is why there is an anti-trust case against it), Sharepoint as your document platform, and, well, Azure as your cloud platform. Except you might use AWS because Amazon is also great. In some ways they are even more compliant with EU legislation than Microsoft. But if Microsoft can throw GPT products into Azure the same way they put Teams and Sharepoint into Office365… well, then where is their competition? And having GPT features within Office365 will only further their advantage on the office platform. I mean, there are companies which won’t use Outlook, but there won’t be when ChatGPT writes your e-mails. So this isn’t necessarily for you. It’s just part of Microsoft’s over all strategy for total IT domination in Enterprise. I mean, we’re going into RPA (robot process automation) a journey I went through in another Enterprise organisation a few years back. Back then you had to consider what go buy, would it be BluePrism, UIPath, automation anywhere, something else? Today there is no competition to Microsoft’s PowerAutomate if you’re already a Microsoft customer. It’s literally $500 a month vs $50k a month… I mean… that’s the future for GPT on Azure. It’s probably necessary too. Their prices have made a lot of organisations look outside of Azure. Toward places like Hetzner or even self-hosting, but if Azure comes with GPT… well then.
roymj88 3y ago
It was really good when the access was enabled via OpenAI, but ever since its moved to Azure subscription, getting preview access is stalled. Wouldn't be a big deal for others, but for smalltime devs like me it becomes a big challenge.. Hope OpenAI provides a developer env or so where we can try things out..
cpill 3y ago
IS it possible for someone to give us the lower bound on the cost of running a 70B model in the cloud? How much memory does Llamba-2 take? What would it cost to fine tune it?
didibus 3y ago
When will LLMs be good enough to write the code for a competitive or better LLM to themselves?
- nsvd 3y ago
  If this happens, we certainly won't be the first to know.
sagarpatil 3y ago
Azure API is definitely faster than OpenAI and they also seem to provide access to 32k generously compared to OAI.
Roark66 3y ago
Private and secure? I thought the main issue with privacy and security of (not at all)OpenAI models is that by using their products you agree for them to retain all the data you send and receive from the models forever for whatever they choose to use it for. Or is this just a thing for free use? If you pay, do you get a Ts&Cs that don't contain any wording like this? Still, even if there was no specific "we own everything" statement there could be pretty much standard statement of "we'll retain data as required for the delivery and improvement of the service" which is essentially the same thing. So, any company that allows it's employees to use chatgpt for work stuff (writing emails with company secrets etc) is definitely not engaging in "secure and private" use. Unless there is very clear data ownership, for example, customer owns the data going in and going out. I can't see how it can be any different. The problem (not at all)OpenAI has in delivery such service is that in contrast to open source models I'm told there is a lot of "secret sauce" around the model(not just the model itself). Specifically input/output processing, result scoring and so on.
- homero 3y ago
  I'm pretty sure the point of this version is not to export data hence the name
- kiratp 3y ago
  > Starting on March 1, 2023, we are making two changes to our data usage and retention policies: > OpenAI will not use data submitted by customers via our API to train or improve our models, unless you explicitly decide to share your data with us for this purpose. You can opt-in to share data. > Any data sent through the API will be retained for abuse and misuse monitoring purposes for a maximum of 30 days, after which it will be deleted (unless otherwise required by law). https://openai.com/policies/api-data-usage-policies https://openai.com/policies/api-data-usage-policies
  - actionfromafar 3y ago
    Unless required by law… I wonder what law.
mteigers 3y ago
Anyone have any thoughts as to ballpark costs to run this? My napkin math on the cosmo-db requirements is failing me (largely because I do not know Azure at all). I'm wondering as a hobbyist / tinkerer if a solution like this is "affordable" (I know it's all relative)
jonplackett 3y ago
Anyone know what the cost is for Azure VS OpenAI?
- djtriptych 3y ago
  I believe the prices are identical: https://azure.microsoft.com/en-us/pricing/details/cognitive-services/openai-service/ https://azure.microsoft.com/en-us/pricing/details/cognitive-... https://openai.com/pricing https://openai.com/pricing disclaimer/source: I work at Microsoft on Azure/OpenAI
longnguyen 3y ago
Interesting. One of my most requested feature for my small native apps[0][1] was to support Azure OpenAI service. Apparently, many organizations have their own Azure OpenAI deployment and won’t let their employees use the public OpenAI service. My understanding is that Azure makes sure all network traffic is isolated to their network so they have more controls over how their organization use ChatGPT. I created a super simple step-by-step guide on how to obtain an Azure OpenAI endpoint & key here: https://pdfpals.com/help/how-to-generate-azure-openai-api-key https://pdfpals.com/help/how-to-generate-azure-openai-api-ke... Hope it would be useful to someone just getting started with Azure. [0]: https://boltai.com https://boltai.com [1]: https://pdfpals.com https://pdfpals.com
CodeCompost 3y ago
I've tried it out. Right now it seems more of a proof-of-concept than a real-world application. Having said that, the concepts and ideas in there are definitely reusable.
warthog 3y ago
Did this just kill a lot of AI startups that were targeting enterprises?
tcarambat1010 3y ago
Im not surprised Azure would add something like this to the stack. We build AnythingLLM (https://github.com/Mintplex-Labs/anything-llm https://github.com/Mintplex-Labs/anything-llm) back in June due to some enterprise customers wanting something isolated they could run on premises with Azure OpenAI support + any vector DB they want. With Azure's move to try to internalize any enterprise integration for AI it makes sense to make a chatbot wrapper because its a no-moat move. I think a lot of the "moat" if one can exist in the "chat with your docs" vertical is just integrations into flows and data sources SMB/Enterprises are already using. For businesses, in my experience, the on-prem thing has been the first decision point - without question. Azure wrapper could be nice to have for those who cannot use chatGPT on the work comp but have access to this instead. I wonder what kind of hypervisor view it gives to Azure admins for those who use it - it any. Multi-tenant instances was the second highest demand from SMB/Enterprise customers for AnythingLLM.
b33j0r 3y ago
Annnd it’s a 404. Less than a day later. The last article I see linking to it was published this morning. Not sure what happened here, but “404’s at just-announced permalinks” seems to be on the rise lately. Don’t turn me into a late-onset pedant. Fine. URIs are permanent forever! For all resources! ;)
- duluca 3y ago
  It's disappointing. I wonder why they got cold feet. This is one of the reasons why I try to fork projects that I really like. But I didn't get around to this one until it was already made private.
byteknight 3y ago
Literally 404
byteknight 3y ago
And they removed it :) [0] You're welcome. [0] https://github.com/microsoft/azurechatgpt https://github.com/microsoft/azurechatgpt [1] https://web.archive.org/web/20230814080150/https://github.com/microsoft/azurechatgpt https://web.archive.org/web/20230814080150/https://github.co...
- borissk 3y ago
  I can imagine how the conversation went with the enterprise customers: "Where does this send the data our employees enter?" "Same place as if they used the free ChatGPT chat bot..."
  - wodenokoto 3y ago
    No it doesn't. It sends it to an LLM hosted inside the Enterprises own Azure Subscription.
- omarhaneef 3y ago
  Ugh. Any clue as to why?
  - giulio8 3y ago
    I suspect they want to redirect to https://github.com/microsoft/chat-copilot https://github.com/microsoft/chat-copilot with FluentUI webapp and C# webapi... And the backend stores from qdrant to chroma ... Sequential Planner...
- aa-morgan 3y ago
  If anyone needs access to the code, you just need to /forks on the web.archive link above and download from there. i.e. https://web.archive.org/web/20230814150922/https://github.com/microsoft/azurechatgpt/forks https://web.archive.org/web/20230814150922/https://github.co... (the cache ID updates when you change the URL)
- latebird22 3y ago
  Does anybody know a fork with the last commit (9116afe)?
sagarpatil 3y ago
Now returns a 404. Interesting.
jenka93 3y ago
Looks like they removed it. Wondering why…
tuxguy 3y ago
The linked github repo was active yesterday and is now returning a 404. anybody know why ?
graybeardhacker 3y ago
Link is 404 now. Anyone fork it before it went 404?
- WalterSobchak 3y ago
  quite a few. here's one: https://github.com/imartinez2/azurechatgpt https://github.com/imartinez2/azurechatgpt

891 points by taubek 3y ago

salil999 3y ago

https://archive.ph/STnXb https://archive.ph/STnXb

coltongaodgd 3y ago

[dead]

r13a 3y ago

Could anyone explain how this can be constructed as a private solution? I'm not familiar with Azure platform. Is the inference processed on private instance ? I can't imagine how it could be feasible given the hardware required to run gpt3.5/4. So the best case scenario is: 1. A web ui runs on a private instances. So any user input (chat or files) are only seen by these instances 2. Any chat historisation or RAG is also done on these instances too. 3. Embeddings compuation may possibly be done on the private instance 4. The embeddings are then sent to the Microsoft GPU farm for inference. So at one point my data has to leave my private network. The problem is that the data can easily be retro-engineered from the embeddings. How can this be presented as a private LLM ?

albert_e 3y ago

is there away to run this on AWS instead. we were looking to explore Llama2 for internal use

froggychairs 3y ago
OpenAI models are exclusively Azure only. Llama2 should have an AWS option I believe?
axpy906 3y ago
Use SageMaker: https://www.philschmid.de/sagemaker-llama-llm https://www.philschmid.de/sagemaker-llama-llm
villgax 3y ago
Have your engineers set this up internally https://huggingface.co/spaces/huggingface-projects/llama-2-7b-chat https://huggingface.co/spaces/huggingface-projects/llama-2-7...
- speedgoose 3y ago
  You can’t really replace ChatGPT 4 with llama2 7B.
  - villgax 3y ago
    Yeah right try getting the same answer after two months
klysm 3y ago
Msft spent a lot of money to ensure that was not an option w chatgpt
- RockyMcNuts 3y ago
  https://about.fb.com/news/2023/07/llama-2/ https://about.fb.com/news/2023/07/llama-2/ https://huggingface.co/blog/hugging-face-endpoints-on-azure https://huggingface.co/blog/hugging-face-endpoints-on-azure You can of course run Llama2 in Azure, but you can't host OpenAI models in AWS
braydenm 3y ago
Amazon Bedrock makes Claude 2 available, as well as some other models.
gdiamos 3y ago
We can run llama 2 on an AWS vm if you have enough GPUs: https://lamini.ai/ https://lamini.ai/ Install in 10 minutes. Make sure you have enough GPU memory to fit your llama model if you want good perf

coldblues 3y ago

Pretty sure Azure has a moderation endpoint enabled by default that makes using the OpenAI API an awful experience.

Y_Y 3y ago

So the public access one isn't private and secure?

froggychairs 3y ago
I believe it’s implying the free ChatGPT collects data and this one doesn’t.
nwoli 3y ago
I thought sama said they don’t use data going through the api for training. Guess we can’t trust that statement
- jumploops 3y ago
  That is correct, they do not use the data going through the API for training, but they do use the data from the web and mobile interfaces (unless you explicitly turn it off).
  - quickthrower2 3y ago
    “We don’t water down your beer”. Oh nice! “But that is lager”
candiddevmike 3y ago
> However, ChatGPT risks exposing confidential intellectual property. I don't remember seeing this disclaimer on the ChatGPT website, gee maybe OpenAI should add this so folks stop using it.
- cmarschner 3y ago
  If you use ChatGPT through the app or website they can use the data for training, unless you turn it off. https://help.openai.com/en/articles/5722486-how-your-data-is-used-to-improve-model-performance https://help.openai.com/en/articles/5722486-how-your-data-is...

littlestymaar 3y ago

“private and secure” from the company that let contractor listen to your private Teams conversation for data labeling purpose, and monitor your activity on your own computer with their OS…

tharwan 3y ago
Huh. I missed this one. Got a link?
- dijital 3y ago
  At a guess it's this story: https://www.vice.com/en/article/xweqbq/microsoft-contractors-listen-to-skype-calls https://www.vice.com/en/article/xweqbq/microsoft-contractors...
  - littlestymaar 3y ago
    Ah yes it was Skype and not Teams, my bad.
svaha1728 3y ago
Move fast and break things, including basic security. Why anyone trusts Azure that all these prompts won't eventually be leaked is beyond me. No one goes broke trusting Azure, but I'd love it if someone was held responsible. https://www.schneier.com/blog/archives/2023/08/microsoft-signing-key-stolen-by-chinese.html https://www.schneier.com/blog/archives/2023/08/microsoft-sig...

refulgentis 3y ago

Crappy clone of ChatGPT frontend, half missing, half direct copy. Implied and overly vast claims of insecurity + lack of privacy, that are narrowly true, i.e. for _Chat_GPT. Really surprised to see this aggressive of language 1) written down 2) on Github. I'd be pretty pissed if I was OpenAI, regardless of the $10B.

jeremyjh 3y ago
I have no doubt OpenAI is on board. This is just bringing more paid users to their platform because it still uses their API.
sebzim4500 3y ago
I think OpenAI is entirely on board with the idea that OpenAI sells to consumers and Azure/Microsoft sells the same product to enterprise. That's how it's been working for months, and if OpenAI objected they would have done something about it.

croes 3y ago

Yeah sure, I totally trust you after the Storm-0558 desaster

TuringNYC 3y ago

Curious if anyone has done a side-by-side analysis of this offering vs just running LLaMA? I'm currently running a side-by-side comparison/evaluation of MSFT GPT via Cognitive Services vs LLaMA[7B/13B/70B] and intrigued by the possibility of a truly air-gapped offering not limited by external computer power (nor by metered fees racking up.) Any reads on comparisons would be nice to see. (yes, I realize we'll eventually run into the same scaling issues w/r/t GPUs)

FrenchDevRemote 3y ago
chatgpt is obviously a LOT better, llama doesn't even understand some prompts and since LLMs aren't even that good to begin with, it's obvious you want the SOTA to do anything useful unless maybe you're finetuning
- baobabKoodaa 3y ago
  > and since LLMs aren't even that good to begin with, it's obvious you want the SOTA to do anything useful unless maybe you're finetuning This is overkill. First of all, ChatGPT isn't even the SOTA, so if you "want SOTA to do anything useful", then this ChatGPT offering would be as useless as LLaMA according to you. Second, there are many individual tasks where even those subpar LLaMA models are useful - even without finetuning.
  - FrenchDevRemote 3y ago
    it's the SOTA for chat(prove me wrong), and you can always use the API directly even for simple tasks they're less reliable and needs more prompt engineering
    - baobabKoodaa 3y ago
      > it's the SOTA for chat(prove me wrong) GPT-4 beats ChatGPT on all benchmarks. You can easily google these.

villgax 3y ago

Yeah right for the three letter agencies to have a backdoor, hard pass on something that cannot be deterministic with a seed

PaulWaldman 3y ago

Since the only users who would likely care about this derive far more value than the $20/month of OpenAI's direct offering. Why doesn't OpenAI market this service, but with chat history, for something like $200/month?

semitones 3y ago
That's a laughable price for an enterprise subscription. And the reason is, it's enough for OpenAI to "say" that they're "not going to use your data" - you need a cloud deployment where you can control network boundaries to _prove_ that your data isn't going anywhere it isn't supposed to.
- agildehaus 3y ago
  Unless you're physically controlling the network boundaries, how are you proving that on any cloud service?
unnouinceput 3y ago
OenAI IS Microsoft. Don't get tangled in the web of creating different entities when they are all part of the same pyramid. Also GitHub IS Microsoft too!!
- nixgeek 3y ago
  GitHub was acquired by Microsoft, and they are no longer legally separate entities. Microsoft is an investor in OpenAI, but does not own it, and they are legally separate companies. OpenAI is not Microsoft and it is factually incorrect to claim that OpenAI is Microsoft. [1] https://blogs.microsoft.com/blog/2023/01/23/microsoftandopenaiextendpartnership/ https://blogs.microsoft.com/blog/2023/01/23/microsoftandopen...
  - Scoundreller 3y ago
    But saying they're just an investor isn't quite doing the arrangement the justice it deserves. There seems to be a lot of strings attached to that investment. It's not just a straight trade of dollars for shares, but many further contractual obligations.

robbomacrae 3y ago

This is potentially a huge deal. Companies are concerned using ChatGPT might violate data privacy policies if someone puts in user data or invalidate trade secrets protections if someone uploads sections of code. I suspect many companies have been waiting for an enterprise version.

deleted 3y ago
[deleted]
judge2020 3y ago
I imagine most companies serious about this created their own wrappers around the API or contracted it out, likely using private Azure GPUs.
- Normal_gaussian 3y ago
  Most companies are either not tech companies, or do not have the knowledge to manage such a project within reasonable cost bounds.
  - jmathai 3y ago
    Most companies are trying to figure out exactly what generative AI is and how to use it in their business. Given how new this is - I doubt any large company has done much besides ban the public ChatGPT. So this is probably very relevant for them.
tbrownaw 3y ago
This is a web UI that talks to a (separate) Azure OpenAI resource that you can deploy into your subscription as a SaaS instance.
- hackernewds 3y ago
  So how is it any different

bouke 3y ago

How is this different from the other OpenAI GUI? Why another one by Microsoft? https://github.com/microsoft/sample-app-aoai-chatGPT https://github.com/microsoft/sample-app-aoai-chatGPT.

FrenchDevRemote 3y ago
i'm pretty sure it's a part of it
pjmlp 3y ago
I bet there are plenty of OKR/KPIs now tied to AI at Microsoft.
wodenokoto 3y ago
There's at least two more. There's also https://github.com/Azure-Samples/azure-search-openai-demo https://github.com/Azure-Samples/azure-search-openai-demo And you can deploy a chat bot from within the Azure playground which runs on another codebase.
colonwqbang 3y ago
Bigger companies are cautious about using GPT-style products due to data security concerns. But most big companies trust Microsoft more or less blindly. Now that Microsoft has an official "enterprise" version out, the floodgates are open. They stand to make a killing.
pamelafox 3y ago
This is an internal ChatGPT, whereas that sample is ChatGPT constrained to internal search results (using RAG approach). Source: I help maintain the RAG samples.

paxys 3y ago

Would it be too much to mention somewhere in the README what this repo actually contains? Just docs? Deployment files? Some application (which does..something)? The model itself?

Xenoamorphous 3y ago
The repo contains the UI code, not the model or anything else around ChatGPT, it just uses Azure’s ChatGPT API which doesn’t share data with OpenAI.
- paxys 3y ago
  So basically – what you really need to do to run Azure ChatGPT is go and click some buttons in the Azure portal. This repo is a sample UI that you could possibly use to talk to that instance, but really you will probably always build your own or embed it directly into your products. So calling the repo "azurechatgpt" is misleading. It should really be "sample-chatgpt-api-frontend" or something of that sort.
  - saliagato 3y ago
    Yes exactly
  - laurels-marts 3y ago
    Correct. If offers a front-end scaffolding for your enterprise ChatGPT app. Uses Next/NextAuth/Tailwind etc. for deployment on Azure App Service that hooks into Azure Cosmos DB and Azure OpenAI (the actual model).
- deleted 3y ago
  [deleted]
- wodenokoto 3y ago
  Isn’t there also some sort of backend stuff in there? How else would it keep track of history and accept documents. I don’t know enough typescript to understand where the front end stops and the backend begins I this code

gdiamos 3y ago

Can you fine tune it?

jensen2k 3y ago
Yes! You can.
- gdiamos 3y ago
  Is it the same api as the public OpenAI
- saliagato 3y ago
  How?

xeckr 3y ago

No better than the API.

Ecstatify 3y ago

Our company is pushing everyone to use a similar offering. Most of the company is doing low value work … still using excels even though we have a custom ERP. Now seeing people who couldn’t write a coherent email before write 3 page emails. The illusion of being productive by doing more work even though it has zero impact on the bottom line. It’s insane how inefficient organisations are. No doubt we’ll have some KPI soon about using the tool.

simmerup 3y ago
If anything it's less productive because people have to parse all that nonsense. I was gobsmacked to hear a friend say that their work guidance is to use ChatGPT to write letters to external clients for example. I know for sure I'd be insulted if someone sent me paragraphs of text to read created from a sentence long prompt. I'd rather have the prompt, my time is valuable as well.
- mritchie712 3y ago
  ahhhh, but they're pasting the 3 page email into ChatGPT ("summarize this"). The future is here.
  - ilyt 3y ago
    Wouldn't be surprised if that was next Outlook feature. Cue someone making some horrible error because some crucial information didn't survive ChatGPT->ChatGPT round-trip
    - mritchie712 3y ago
      it's already here... https://blogs.microsoft.com/blog/2023/03/16/introducing-microsoft-365-copilot-your-copilot-for-work/ https://blogs.microsoft.com/blog/2023/03/16/introducing-micr...
    - ddmma 3y ago
      Actually this was in an Azure hacktoon some time ago https://devpost.com/software/amabot https://devpost.com/software/amabot

Xenoamorphous 3y ago

Darn I just spent a week or so working on a ChatGPT clone that used Azure ChatGPT API due to the privacy aspect. Wasted effort I guess.

saliagato 3y ago
This is exactly the same
- ddmma 3y ago
  Welcome to the club :)
  - jeremycarter 3y ago
    I'm also in this club but we wrote it months ago.

jrm4 3y ago

"Private and secure" From Microsoft? Ha.

gdiamos 3y ago

we wrote a blog post about why companies do this here: https://www.lamini.ai/blog/specialize-llms-to-private-data-databricks https://www.lamini.ai/blog/specialize-llms-to-private-data-d... Here are a few: Data privacy Ownership of IP Control over ops The table in the blog lists the top 10 reasons why companies do this based on about 50 customer interviews.

ajhai 3y ago

A lot of companies are already using projects like chatbot-ui with Azure's OpenAI for similar local deployments. Given this is as close to local ChatGPT as any other project can get, this is a huge deal for all those enterprises looking to maintain control over their data. Shameless plug: Given the sensitivity of the data involved, we believe most companies prefer locally installed solutions to cloud based ones at least in the initial days. To this end, we just open sourced LLMStack (https://github.com/TryPromptly/LLMStack https://github.com/TryPromptly/LLMStack) that we have been working on for a few months now. LLMStack is a platform to build LLM Apps and chatbots by chaining multiple LLMs and connect to user's data. A quick demo at https://www.youtube.com/watch?v=-JeSavSy7GI https://www.youtube.com/watch?v=-JeSavSy7GI. Still early days for the project and there are still a few kinks to iron out but we are very excited for it.

toomuchtodo 3y ago
Can you plug this together with tools like api2ai to create natural language defined workflow automations that interact with external APIs?
- ajhai 3y ago
  There is a generic HTTP API processor that can be used to call APIs as part of the app flow which should help invoke tools. Currently working on improving documentation so it is easy to get started with the project. We also have some features planned around function calling that should make it easy to natively integrate tools into the app flows.
- cosbgn 3y ago
  You can use unfetch.com to make API calls via LLMs and build automations. (I'm building it)
  - scrum-treats 3y ago
    Is it possible to not use Google with unfetch.com?

extr 3y ago

One thing I still don't understand is what _is_ the ChatGPT front end exactly? I've used other "conversational" implementations built with the API and they never work quite as well, it's obvious that you run out of context after a few conversation turns. Is ChatGPT doing some embedding lookup inside the conversation thread to make the context feel infinite? I've noticed anecdotally it definitely isn't infinite, but it's pretty good at remembering details from much earlier. Are they using other 1st party tricks to help it as well?

MaxLeiter 3y ago
It uses a sliding context windows. Older tokens are dropped as new ones stream in
- extr 3y ago
  I don't believe that's the whole story. Other conversational implementations use sliding context windows and it's very noticable as context drops off. Whereas ChatGPT seems to retain the "gist" of the conversation much longer.
  - lsaferite 3y ago
    I mean, I explicitly have the LLM summarize content that's about to fall out of the window as a form of pre-emptive token compression. I'd expect maybe they do something similar.
    - kuchenbecker 3y ago
      I feel like we're describing short vs long term memory.
      - tsunamifury 3y ago
        That’s exactly what it is. It’s just it turns out you need very good generalized or focused simple reasoning to do accurate compression or else the abstraction and movement to long term memory doesn’t include the most important content. Or worse distracting details. I’ve been working on short and long term memory windows at allofus.ai for about 6 months now and it’s way more complex than I had originally thought it would be. Even if you can magically extend the content window, the added data confuses and waters down the reasoning of the LLM. You must do layered abstraction and compression with goal based memory for it to continue to reason without distraction of irrelevant data. It’s an amazing realization, almost like a proof that memory is a kind of layered reasoning compression system. Intelligence of any kind can’t understand everything forever. It must cull the irrelevant details, process the remains and reason on a vector that arises from them.

RockyMcNuts 3y ago

so...how can we make this support plugins like Code Interpreter, Wolfram, Zapier or Workato, and whatnot?

leerob 3y ago

This is awesome to see, feels heavily inspired (in a good way) by the version we made at Vercel[1]. Same tech stack: Next.js, NextAuth, Tailwind, Shadcn UI, Vercel AI SDK, etc. I'd expect this trend of managed ChatGPT clones to continue. You can own the stack end to end, and even swap out OpenAI for a different LLM (or your own model trained on internal company data) fairly easily. [1]: https://vercel.com/templates/next.js/nextjs-ai-chatbot https://vercel.com/templates/next.js/nextjs-ai-chatbot

vladiliescu 3y ago
Really, they use Vercel AI SDK?
- leerob 3y ago
  Well, they did. The repo appears to have been deleted.

mark_l_watson 3y ago

This seems like such an obvious thing to do. I see the use of general purpose LLMs like ChatGPT, but smaller fine tuned models will probably end up being more useful for deployed applications in most companies. Off topic, but I was experimenting with LLongMA-2-7b-16K today, running it very inexpensively in the cloud, and given about 12K of context text it really performed well. This is an easy model to deploy. 7B parameter models can be useful.

stavros 3y ago
Is there an easy way to play with these models, as someone who hasn't deployed them? I can download/compile llama.cpp, but I don't know which models to get/where to put them/how to run them, so if someone knows about some automated downloader along with some list of "best models", that would be very helpful.
- robertnishihara 3y ago
  If you want to try out the Llama-2 models (7B, 13B, 70B), you can get started very easily with Anyscale Endpoints (~2 min). https://app.endpoints.anyscale.com/ https://app.endpoints.anyscale.com/
- tayo42 3y ago
  For llama, the 4bit quantized ones, small models like the 7b one. The ggml format. That will run on your local cpu. Google those terms too. you can look on hugging face for the actual model to download then load it and send prompts to it
  - stavros 3y ago
    Thanks, maybe it's as easy as downloading the ggml and running it with Llama.cpp. I'll try that, thanks!
    - tayo42 3y ago
      there is also a python wrapper that has a web ui built in for llama.cpp, if it wasnt easy enough already

Havoc 3y ago

How does this work in terms of utilization? The isolation presumably means buying gpu capacity and only using a %?

asabla 3y ago
Basically you get N tokens/second (or if it was minute, can check tomorrow if you're really interested) per deployment. So if you would outgrow on deployment, just deploy another one (with the associated costs of course). One deployment = a deployed model which you can query On top of that, depending on the model you're using, you also see a cost increment for each 1000 request you make.
- Havoc 3y ago
  Ah right so it is somewhat shared. Not like your own gpu type situation
  - weird-eye-issue 3y ago
    Why did you make that assumption?

justinlloyd 3y ago

Interesting release, though still lacking a few features I've had to resort building myself such as code summary, code base architecture summary, and conversation history summary. ChatGPT (the web UI) now has the ability to execute code, and make function callbacks, but I prefer running that code locally, especially if I am debugging. This latter part, conversation history summary, is something that ChatGPT web UI does reasonably well, giving it a long history, but a sentiment extraction and salient detail extraction before summarizing is immensely useful for remembering details in the distant past. I've been building on top of the GPT4 model and tinkering with multi-model (gpt4 + davinci) usage too, though I am finding with the MoE that Davinci isn't as important. Fine tuning has been helpful for specific code bases too. If I had the time I'd like to play with an MoE of Llama2, as a compare and contrast, but that ain't gonna happen anytime soon.

alpinemeadow 3y ago

We have this at IKEA for a while now. Not impressed, but funny to read the hallucinations.

BoorishBears 3y ago
I'd expect a company like IKEA to have the expertise to create interfaces specific to their workflows so hallucinations aren't an issue. Imo if you're making an open ended chat interface for a business, you're doing it wrong.
riwsky 3y ago
Have you considered instruction-tuning it with text, instead of just pictures?

padolsey 3y ago

I'm confused. If this is just a front-end for the OpenAI API then how does it remove the data privacy concern? Your data still ends up with Azure/OpenAI, right? It doesn't stay localized to your instance; it's not your GPU running the transformations. You have no way of knowing whether your data is being used to train models. If customer data is sensitive, I'm pretty sure running a 70B llama (or similar) on a bunch of A100s is the only way?

dbish 3y ago
Azure is hosting and operating the service themselves rather then for OpenAI, with all the security requirements that come with that. I assume this comes with different data and access restrictions as well and ability to run in secured instances (and nothing sent to OpenAI the company). Most companies use cloud already for their data, processing, etc. and aren’t running anything major locally, let alone ML models, this is putting trust in the cloud they already use.
- nmstoker 3y ago
  Yes, this was my understanding.
- padolsey 3y ago
  Ah that's fair. But it is my impression that the bulk of privacy/confidentiality concerns (e.g. law/health/..) would require "end to end" data safety. Not sure if I'm making sense. I guess microsoft is somehow more trustworthy than openai themselves... EDIT: what you say about existing cloud customers being able to extend their trust to this new thing makes sense, thanks.
  - PoignardAzur 3y ago
    Right. If I was an European company worried about, say, industrial espionage, this wouldn't be nearly enough to reassure me.
- deleted 3y ago
  [deleted]

EGreg 3y ago

We just have to trust them and take their word for it? Or what? https://azure.microsoft.com/en-us/explore/trusted-cloud/privacy https://azure.microsoft.com/en-us/explore/trusted-cloud/priv... https://azure.microsoft.com/en-us/blog/3-reasons-why-azure-s-infrastructure-is-secure/ https://azure.microsoft.com/en-us/blog/3-reasons-why-azure-s... I guess I would trust them, since they're big and they make these promises and other big companies use them.

aantix 3y ago

I don't understand - chat with a file? I want to chat and ask about an entire body of knowledge - wiki pages, git commit diffs/messages, jira tasks.

PoignardAzur 3y ago

> However, ChatGPT risks exposing confidential intellectual property. One option is to block corporate access to ChatGPT, but people always find workarounds Pretty bold thing to say to your potential clients. "You can always tell your employees not to use our product, but they won't listen to you."

pwarner 3y ago
It's almost like employees might have their own computers?

H8crilA 3y ago

What's the practical difference between this and OpenAI API? All I can see is the same product but offered by a larger organization. I.e. they're more likely to get the security details right, and you can potentially win more in a lawsuit should things go bad.

jeffschofield 3y ago
A few months ago my team moved to Azure for capacity reasons. We were constantly dealing with 429 errors and couldn't get in touch with Open AI, while Azure offered more instances. Eventually got more from Open AI so we load balance both. The only difference is the 3.5 turbo model on Azure is outdated.
- RockyMcNuts 3y ago
  you can ask for gpt-4, it took a while due to capacity constraints but we got it
ebiester 3y ago
Compliance and customer trust. Azure can sign a BAA, for example. If you are Building LLM capability on top of your SaaS, your customers want assurances about their data.

singingfish 3y ago

I was looking through our server logs the other day and spotted the openai bot going through our stuff ... however a decent bit of our content is now augmented by GPT ...

atlgator 3y ago

Is this a full, standalone deployment including GPT-3 (or whatever version) or just a secured frontend that sends data to GPT hosted outside the enterprise zone? Edit: Uses Azure OpenAI as the backend

jmorgan 3y ago

This appears to be a web frontend with authentication for Azure's OpenAI API, which is a great choice if you can't use Chat GPT or its API at work. If you're looking to try the "open" models like Llama 2 (or it's uncensored version Llama 2 Uncensored), check out https://github.com/jmorganca/ollama https://github.com/jmorganca/ollama or some of the lower level runners like llama.cpp (which powers the aforementioned project I'm working on) or Candle, the new project by hugging face. What's are folks' take on this vs Llama 2, which was recently released by Facebook Research? While I haven't tested it extensively, 70B model is supposed to rival Chat GPT 3.5 in most areas, and there are now some new fine-tuned versions that excel at specific tasks like coding (the 'codeup' model) or the new Wizard Math (https://github.com/nlpxucan/WizardLM https://github.com/nlpxucan/WizardLM) which claims to outperform ChatGPT 3.5 on grade school math problems.

ttul 3y ago
Llama 2 might by some measures be close to GPT 3.5, but it’s nowhere near GPT 4, nor Anthropic Claude 2 or Cohere’s model. The closed source players have the best researchers - they are being paid millions a year with tons of upside - and it’s hard to keep pace with that. My sense is that the foundation model companies have an edge for now and will probably stay a few steps ahead of the open source realm simply for economic reasons. Over the long run, open source will eventually overtake. Chances are this will happen once the researchers who are making magic happen get their liquidity and can start working for free again out in the open.
- robertnishihara 3y ago
  > Llama 2 might by some measures be close to GPT 3.5, but it’s nowhere near GPT 4 I think you're right about this, and benchmarks we've run at Anyscale support this conclusion [1]. The caveat there (which I think will be a big boon for open models) is that techniques like fine-tuning makes a HUGE difference and can bridge the quality gap between Llama-2 and GPT-4 for many (but not all) problems. [1] https://www.anyscale.com/blog/fine-tuning-llama-2-a-comprehensive-case-study-for-tailoring-models-to-unique-applications https://www.anyscale.com/blog/fine-tuning-llama-2-a-comprehe...

gerdesj 3y ago

Well, your AChatGPT connection might be private but your Windows machine will leak like a sieve. It is embarrassing how needy the blasted things are about signing in via Azure/Microsoft, instead of a local or AD account. Even worse is the naff "choose how insidious you would like us to be" series of questions. How would you like your ads? Targeted or non targeted? How about not at all? Nope. In this day and age, exactly how private does anyone expect their comms/thoughts/files/data to be? I recall reading a recent MS EULA and it seems I have to say three Hail Marys every third Tuesday for using Arch Linux on my PCs. I could install Edge, and did but I don't like the nasty homepage - a bit right wing ... - why on earth is a browser pushing "news"? Its a browser. To be fair I had to dump all the homepage crap that Firefox pushed when I finally dumped anything to do with Chrome. Please don't use the words private and secure when you have your fingers crossed behind your back.

xist 3y ago
The article is referring to enterprise usage - and you're quoting all the consumer level attributes (aka cheap/sometimes subsidized version). At the enterprise level where this is intended to be ran, things are much diffrerent. If you're not aware of the differences or use cases, perhaps you're not the target audience who should be using or configuring it.
- gerdesj 3y ago
  Why don't we give the willy waving a miss? Win 10 and 11 are steering you to cloud first, out of the box. That's fine if you like it, but I don't and quite a lot of my customers don't. The real problem is about data sovereignty. I'm a Brit and ... MS isn't.
  - xist 3y ago
    The article is about use in an enterprise. An enterprise runs professional/enterprise/ltsc versions which do NOT steer you to the cloud - what data sovereignty concerns have you seen in those editions of windows/server? They've gone through a lot of pains to ensure those concerns are taken care of for enterprises/governments so i'm curious the ones you think they missed. You can make the argument for their consumer editions sure, but that's a different product with different features, different price point for different users.

tananaev 3y ago

This is not ChatGPT. It's just a front end for Azure OpenAI APIs. Not sure why they're so blatantly use the trademark. They will probably have to rename it soon.

mkinsella 3y ago
Microsoft is a major investor in OpenAI. Guaranteed they worked with OpenAI on this and have partnership to use the trademarks.
castrodd 3y ago
Microsoft owns OpenAI so I doubt that they will be asked to rename this.
- mythz 3y ago
  They'll only own 49% of shares.
white_dragon88 3y ago
[dead]

ChatGTP 3y ago

Upvoted, cannot wait for this, yes yes yes. The companies have been waiting for this.

37112741 3y ago

Nothing in the repo details how this addresses privacy concerns of running inference on someone else's LLM. To be isolated from other users of the service is not the same thing as having a private inference engine. > Private: Built-in guarantees around the privacy of your data and fully isolated from those operated by OpenAI. Do tell.

RockyMcNuts 3y ago
So where do you draw the line? No cloud instances, no cloud SQL like Snowflake, no Teams or Office 365, no S3/blob storage? Run everything on-prem like 10 years ago? It's only going to get more impossible. All that VC money going in at 100x revenue needs a return and they aren't going leave money on the table with full-featured open-source or CentOS type alternatives. All those data engineering startups, database providers with 'open-source' + cloud hosting, the 'open-source' is going to be just 'open' enough to claim there is some fallback for someone else to pick up the mantle using the community version, if the cloud version gets enshittified beyond reason. You're not going to even be able to run the full-featured software version on-prem because the economics of cloud are so much better. Unless you are writing and compiling your own code you are going to be out of luck if your privacy standard is that high. That war has been lost. And Web3 sure ain't gonna save you either.
- 37112741 3y ago
  They should clearly spell out what is and is not "private". As it is we simply have a blurp about some undefined guarantees. And some comments here in thread saying "this is as close as you're going to get to local GPT" are deeply wrong. But then there is easy VC money (just like with ADs ..) and certain "clever" geeks throw social responsibility out the window as usual and are pushing all sorts of deeply invasive applications ("let our proxy for Microsoft hoover your inbox!") based on these undefined "Privacy guarantees". If we accept this just we accepted the very flawed solutions we were given by corporation regarding social networking and ads, we are going to be stuck with it, suffer the consequences, and there will be no incentive to develop alternatives that actually address issues and work. Homomorphic Encryption works. It just doesn't work very efficiently right now but that is an intellectual problem that can be solved if we push for actual privacy for this critical technology as it will be fully enmeshed in all parts of our lives. "Think of the children" if that helps.

Racing0461 3y ago

[dead]

polski-g 3y ago

This is not private. It's still hosted on Microsoft's cloud.

10g1k 3y ago

I did a unit of AI at university, and the front of the textbook contained a quote by some ye olde AI theorist, something like: "I'm not concerned that artificial intelligence will take over the world. I'm concerned that human intelligence has yet to do so."

crooked-v 3y ago

The big question: If this is truly secure and private, can people use it to generate things related to porn or violence?

deleted 3y ago
[deleted]

hieu229 3y ago

This is a neat project from Microsoft I've been building https://gasbyai.com https://gasbyai.com, a beautiful chat UI that support self-hosted, with ChatGPT plugins, extract content from pdf/url. GasbyAI supports Azure, OpenAI, and custom API endpoints in case you want to run with your own models

jhoechtl 3y ago

Can any one shed light on what "local" means? Local on my own private machine or local in my Azure Tenant?

jzombie 3y ago
I am pretty sure it means run the UI locally and access Azure-hosted ChatGPT. The environment vars seem to indicate that as well.
wodenokoto 3y ago
Assuming you are referring to this section: https://github.com/microsoft/azurechatgpt/blob/main/docs/3-run-locally.md https://github.com/microsoft/azurechatgpt/blob/main/docs/3-r... It means you run the front end (the chat-gui) and the backend code from the repo. This code connects to cosmo-db for uploading documents used for "chat with you pdf" and connects to an OpenAI instance on Azure for the chat inferrence.

osigurdson 3y ago

I'm a little confused by how the relationship works between OpenAI and Microsoft. It is possible for anyone to register for an OpenAI account and use their APIs. Within Azure the same thing is much more difficult as it is necessary to be a "real" business in order to use it. I maintain an open source OpenAI library and would like to add support for Azure but can't because of this restriction. Why can't I just use my regular Azure account?

devjab 3y ago
Microsoft owns enough of OpenAI that their endgame goal of putting GPT like features into Azure and Office365 for enterprise customers is what we’re likely to see happen. OpenAI will likely target private consumers while Microsoft focuses on enterprise. I can use my own organisation as an example. We’re an investment bank that does green energy within the EU. We would absolutely use GPT if it was legal, but it isn’t, and it likely never will be considering their finance model is partly to steal as much data as they can. Even if it’s not so polite to say that. This is where Microsoft comes into the picture. In non-tech enterprise you’re buying Microsoft products because everyone wants windows, outlook and office. We can wish it wasn’t like that, but where is the realistic alternative? I’m not anti Microsoft by the way, in all my decades in the enterprise business they’ve easily been the best and most consistent business partner for any IT. When Amazon saw how much money there was on the operations side of EU enterprise they quickly caught up, but Amazon doesn’t sell a Office365 product. So anyway, once you have Office365, you’re also likely to use Teams as your communications platform (which is why there is an anti-trust case against it), Sharepoint as your document platform, and, well, Azure as your cloud platform. Except you might use AWS because Amazon is also great. In some ways they are even more compliant with EU legislation than Microsoft. But if Microsoft can throw GPT products into Azure the same way they put Teams and Sharepoint into Office365… well, then where is their competition? And having GPT features within Office365 will only further their advantage on the office platform. I mean, there are companies which won’t use Outlook, but there won’t be when ChatGPT writes your e-mails. So this isn’t necessarily for you. It’s just part of Microsoft’s over all strategy for total IT domination in Enterprise. I mean, we’re going into RPA (robot process automation) a journey I went through in another Enterprise organisation a few years back. Back then you had to consider what go buy, would it be BluePrism, UIPath, automation anywhere, something else? Today there is no competition to Microsoft’s PowerAutomate if you’re already a Microsoft customer. It’s literally $500 a month vs $50k a month… I mean… that’s the future for GPT on Azure. It’s probably necessary too. Their prices have made a lot of organisations look outside of Azure. Toward places like Hetzner or even self-hosting, but if Azure comes with GPT… well then.

roymj88 3y ago

It was really good when the access was enabled via OpenAI, but ever since its moved to Azure subscription, getting preview access is stalled. Wouldn't be a big deal for others, but for smalltime devs like me it becomes a big challenge.. Hope OpenAI provides a developer env or so where we can try things out..

cpill 3y ago

IS it possible for someone to give us the lower bound on the cost of running a 70B model in the cloud? How much memory does Llamba-2 take? What would it cost to fine tune it?

didibus 3y ago

When will LLMs be good enough to write the code for a competitive or better LLM to themselves?

nsvd 3y ago
If this happens, we certainly won't be the first to know.

sagarpatil 3y ago

Azure API is definitely faster than OpenAI and they also seem to provide access to 32k generously compared to OAI.

Roark66 3y ago

Private and secure? I thought the main issue with privacy and security of (not at all)OpenAI models is that by using their products you agree for them to retain all the data you send and receive from the models forever for whatever they choose to use it for. Or is this just a thing for free use? If you pay, do you get a Ts&Cs that don't contain any wording like this? Still, even if there was no specific "we own everything" statement there could be pretty much standard statement of "we'll retain data as required for the delivery and improvement of the service" which is essentially the same thing. So, any company that allows it's employees to use chatgpt for work stuff (writing emails with company secrets etc) is definitely not engaging in "secure and private" use. Unless there is very clear data ownership, for example, customer owns the data going in and going out. I can't see how it can be any different. The problem (not at all)OpenAI has in delivery such service is that in contrast to open source models I'm told there is a lot of "secret sauce" around the model(not just the model itself). Specifically input/output processing, result scoring and so on.

homero 3y ago
I'm pretty sure the point of this version is not to export data hence the name
kiratp 3y ago
> Starting on March 1, 2023, we are making two changes to our data usage and retention policies: > OpenAI will not use data submitted by customers via our API to train or improve our models, unless you explicitly decide to share your data with us for this purpose. You can opt-in to share data. > Any data sent through the API will be retained for abuse and misuse monitoring purposes for a maximum of 30 days, after which it will be deleted (unless otherwise required by law). https://openai.com/policies/api-data-usage-policies https://openai.com/policies/api-data-usage-policies
- actionfromafar 3y ago
  Unless required by law… I wonder what law.

mteigers 3y ago

Anyone have any thoughts as to ballpark costs to run this? My napkin math on the cosmo-db requirements is failing me (largely because I do not know Azure at all). I'm wondering as a hobbyist / tinkerer if a solution like this is "affordable" (I know it's all relative)

jonplackett 3y ago

Anyone know what the cost is for Azure VS OpenAI?

djtriptych 3y ago
I believe the prices are identical: https://azure.microsoft.com/en-us/pricing/details/cognitive-services/openai-service/ https://azure.microsoft.com/en-us/pricing/details/cognitive-... https://openai.com/pricing https://openai.com/pricing disclaimer/source: I work at Microsoft on Azure/OpenAI

longnguyen 3y ago

Interesting. One of my most requested feature for my small native apps[0][1] was to support Azure OpenAI service. Apparently, many organizations have their own Azure OpenAI deployment and won’t let their employees use the public OpenAI service. My understanding is that Azure makes sure all network traffic is isolated to their network so they have more controls over how their organization use ChatGPT. I created a super simple step-by-step guide on how to obtain an Azure OpenAI endpoint & key here: https://pdfpals.com/help/how-to-generate-azure-openai-api-key https://pdfpals.com/help/how-to-generate-azure-openai-api-ke... Hope it would be useful to someone just getting started with Azure. [0]: https://boltai.com https://boltai.com [1]: https://pdfpals.com https://pdfpals.com

CodeCompost 3y ago

I've tried it out. Right now it seems more of a proof-of-concept than a real-world application. Having said that, the concepts and ideas in there are definitely reusable.

warthog 3y ago

Did this just kill a lot of AI startups that were targeting enterprises?

tcarambat1010 3y ago

Im not surprised Azure would add something like this to the stack. We build AnythingLLM (https://github.com/Mintplex-Labs/anything-llm https://github.com/Mintplex-Labs/anything-llm) back in June due to some enterprise customers wanting something isolated they could run on premises with Azure OpenAI support + any vector DB they want. With Azure's move to try to internalize any enterprise integration for AI it makes sense to make a chatbot wrapper because its a no-moat move. I think a lot of the "moat" if one can exist in the "chat with your docs" vertical is just integrations into flows and data sources SMB/Enterprises are already using. For businesses, in my experience, the on-prem thing has been the first decision point - without question. Azure wrapper could be nice to have for those who cannot use chatGPT on the work comp but have access to this instead. I wonder what kind of hypervisor view it gives to Azure admins for those who use it - it any. Multi-tenant instances was the second highest demand from SMB/Enterprise customers for AnythingLLM.

b33j0r 3y ago

Annnd it’s a 404. Less than a day later. The last article I see linking to it was published this morning. Not sure what happened here, but “404’s at just-announced permalinks” seems to be on the rise lately. Don’t turn me into a late-onset pedant. Fine. URIs are permanent forever! For all resources! ;)

duluca 3y ago
It's disappointing. I wonder why they got cold feet. This is one of the reasons why I try to fork projects that I really like. But I didn't get around to this one until it was already made private.

byteknight 3y ago

Literally 404

byteknight 3y ago

And they removed it :) [0] You're welcome. [0] https://github.com/microsoft/azurechatgpt https://github.com/microsoft/azurechatgpt [1] https://web.archive.org/web/20230814080150/https://github.com/microsoft/azurechatgpt https://web.archive.org/web/20230814080150/https://github.co...

borissk 3y ago
I can imagine how the conversation went with the enterprise customers: "Where does this send the data our employees enter?" "Same place as if they used the free ChatGPT chat bot..."
- wodenokoto 3y ago
  No it doesn't. It sends it to an LLM hosted inside the Enterprises own Azure Subscription.
omarhaneef 3y ago
Ugh. Any clue as to why?
- giulio8 3y ago
  I suspect they want to redirect to https://github.com/microsoft/chat-copilot https://github.com/microsoft/chat-copilot with FluentUI webapp and C# webapi... And the backend stores from qdrant to chroma ... Sequential Planner...
aa-morgan 3y ago
If anyone needs access to the code, you just need to /forks on the web.archive link above and download from there. i.e. https://web.archive.org/web/20230814150922/https://github.com/microsoft/azurechatgpt/forks https://web.archive.org/web/20230814150922/https://github.co... (the cache ID updates when you change the URL)
latebird22 3y ago
Does anybody know a fork with the last commit (9116afe)?

sagarpatil 3y ago

Now returns a 404. Interesting.

jenka93 3y ago

Looks like they removed it. Wondering why…

tuxguy 3y ago

The linked github repo was active yesterday and is now returning a 404. anybody know why ?

graybeardhacker 3y ago

Link is 404 now. Anyone fork it before it went 404?

WalterSobchak 3y ago
quite a few. here's one: https://github.com/imartinez2/azurechatgpt https://github.com/imartinez2/azurechatgpt