7 ms·

Windows AI Studio Preview

204 points by Jayakumark 3y ago

Const-me 3y ago
> will run only on NVIDIA GPUs for the preview I wonder why Microsoft helps nVidia, instead of using their own technology? Here’s an example: https://github.com/Const-me/Cgml https://github.com/Const-me/Cgml
- capableweb 3y ago
  Companies of the size of Microsoft usually function more like a country with various companies within, rather than one company with a unified view and unified goals. So probably, the group wasn't even aware of that technology because it's far away by either professional connection, or by personal/relationship connections, or they knew about it but had another goal than "maximize use of own stuff" and made the call that the tradeoffs wasn't worth it.
  - Const-me 3y ago
    I don’t work for Microsoft, and I don’t care whether people are using Microsoft’s technologies. As a consumer, I’m not happy with nVidia’s monopoly in the field. I find it disappointing that Microsoft chose to play along, instead of disrupting the landscape. I think the only reason for that monopoly is the mental inertia of everyone involved. The complexity of these AI models is contained within the data in the models (gigabytes of numbers in these tensors), the GPU-running code is rather simple, most of that code is basic BLAS stuff. Unlike traditional GPGPU applications (FEM, numerical simulations, fluid dynamics), the compute kernels used in AI are easily portable across GPU APIs. Maybe when I have some free time, I should port my library to Linux + Vulkan, just to prove the point.
- lostmsu 3y ago
  Have you considered ILGPU? https://ilgpu.net/ https://ilgpu.net/
  - Const-me 3y ago
    No, I have not considered ILGPU, it’s the first time I see that thing. Very interesting, but I’m not sure I would have used it for that project. In my experience, using any non-native GPU API is asking for troubles. On Windows, the native ones are D3D 11 and 12, on Linux and Android it’s often Vulkan, and on MacOS it’s either Metal or that newer thing they have built specifically for AI. It seems ILGPU only has backends for CUDA, OpenCL, and CPU SIMD. I have general suspicion towards custom compilers. Writing a good compiler is hard, for GPUs even harder due to the weird execution model and insufficient documentation from GPU vendors. HLSL compiler is supported by Microsoft, and compute shaders are used by many millions of gamers every day. Similarly, CUDA is supported by nVidia, usually pretty stable, my only issue with CUDA is vendor lock-in. I have an impression people are often unhappy with the quality and hardware compatibility of less popular GPU APIs like ROCm and OpenCL. BTW, I asked a friend to test my program on their low-and laptop. The laptop has some Intel core i3 with integrated GPU. The performance wasn’t great at about 1 token/second (single-channel memory), but at least my code worked. I’m not sure it would have worked on that computer if the backend was based on OpenCL.
- pjmlp 3y ago
  Their own technology is DirectML, not raw DirectX 11. https://blogs.windows.com/windowsdeveloper/2023/11/15/elevating-the-developer-experience-on-windows-with-new-ai-tools-and-productivity-tools/ https://blogs.windows.com/windowsdeveloper/2023/11/15/elevat... https://blogs.windows.com/windowsdeveloper/2023/12/14/directml-accelerating-ai-on-windows-now-with-npus/ https://blogs.windows.com/windowsdeveloper/2023/12/14/direct...
DeathArrow 3y ago
Microsoft also has a nice AI/ML framework called ML.NET.
franzb 3y ago
Why is this on GitHub?
- Mashimo 3y ago
  Why would it not be?
  - SSLy 3y ago
    Because there's no real source included therein.
    - Mashimo 3y ago
      True. But it's easy to publish something on github, and you get an bug tracker. So .. I guess they do it because it's easy to do so?
      - capableweb 3y ago
        When even Microsoft uses their own platform in a non-intended way, you know it's an official Microsoft project.
mngdtt 3y ago
Installed this, and it didn't let me do anything. Despite the fact the readme says stuff runs locally, first it asked me to link to my github account, and then all models required me to ask someone (I think meta?) for permission on github or use a huggingface token or whatever. So I uninstalled it and now my wsl prompt starts with (base) and I don't know how to disable it and all my python scripts are broken because they can't find all the libraries I've installed from pip throughout the years. 0/10 would not recommend.
- ZunarJ5 3y ago
  You need to turn off the default conda environment. Try: conda config --set auto_activate_base false
  - mngdtt 3y ago
    Thanks !!!
    - ZunarJ5 3y ago
      :)
- dazzlefruit 3y ago
  (base) sounds like Anaconda. Try typing conda deactivate?
a1o 3y ago
Curious it uses Ubuntu 18.04 with 20.04 and 22.04 already released and 24.04 just around the corner.
ourmandave 3y ago
RAG Project Coming soon! =(
davikr 3y ago
Anyone knows when Microsoft will release a local OCR model officially? I haven't seen anyone talking about this, but the one they're shipping with Snipping Tool (OneOcr) is top-tier and beats everything out there like Tesseract, easyOCR etc. The model is technically in everyone's Windows installs, but we don't have the C++ projected WinRT headers to use the Microsoft.Windows.Vision library.
- capableweb 3y ago
  Seems they've released a bunch already? Image-to-text models, filtered by "Microsoft": https://huggingface.co/models?pipeline_tag=image-to-text&sort=trending&search=microsoft https://huggingface.co/models?pipeline_tag=image-to-text&sor...
  - davikr 3y ago
    Wow, very nice! Thank you, hadn't come across them in my search. It looks like it doesn't recognize more than one line at once, but combined with another model or algorithm to detect text bounding boxes it'd be handy.
    - CharlieDigital 3y ago
      Not sure if it's what's used in the Azure Document Intelligence service, but my experience with it is pretty good. The one somewhat unique offering in Azure is the Document Layout model which gives you back the OCR with titles, headers, paragraphs, and tables all labeled. This is really, really good for RAG since it's often useful to stuff the nearest header into the chunk of text when generating an embedding (much, much better results this way).
Jayakumark 3y ago
Adding link to VScode extension: https://marketplace.visualstudio.com/items?itemName=ms-windows-ai-studio.windows-ai-studio&ssr=false#overview https://marketplace.visualstudio.com/items?itemName=ms-windo...
blackoil 3y ago
Windows AI Studio is in real a Linux AI Studio as it needs WSL to run. Little funny
- troupo 3y ago
  Obligatory "is this the year of Linux in desktop" joke
  - pjmlp 3y ago
    It is kind of ironic that the Year of Linux on the Desktop is all based on having Linux VMs across all major mainstream desktop OSes, even on ChromeOS, Crostini runs inside something similar to WSL.
- severino 3y ago
  But can you get this to run in Linux? From the article I can see it's a VSCode extension or something like that but the name kind of implies it's Windows only.
  - diffeomorphism 3y ago
    > https://github.com/microsoft/windows-ai-studio/blob/main/QA.md https://github.com/microsoft/windows-ai-studio/blob/main/QA.... For now, but seems to be planned in the future.
    - severino 3y ago
      Yep, didn't read that. In the meantime, kudos to Canonical (and other parties) who helped Microsoft with WSL. Now they can also use Linux to produce Windows-only software! :D
      - baz00 3y ago
        The effort always stops when the commercial goal is achieved.
andruby 3y ago
Apple has good uniform hardware to enable this, but they are a product company and an “AI studio” would not fit their usual definition of a product. I do hope they are considering going in that direction though.
- TheRoque 3y ago
  That's the weird thing. They seem to have the best hardware for the price, for individuals to develop and use local LLMs, but so far they have been pretty quiet on all this.
  - diffeomorphism 3y ago
    Highly depends on the price. Currently 1600€ gets you a laptop with 8gb shared memory, which surely is a nice device for other use cases but for developing local LLMs that seems awfully limiting. With upgraded ram (for just 230€ for each 8GB) and storage (just over 1000€ for 2tb; a samsung 990 pro is like 170€) that might be another story, but "check your specs before downloading this app" seems very un-appley. Also, no cuda support etc. Maybe if they make it exclusive to the mac studio?
    - kolinko 3y ago
      Cuda would be pointless for Apple, because cuda is for Nvidia only, no? As for the price - when you get to 32/64/96GB levels of RAM it’s the cheapest setup on the market that can get you this much VRAM. At least that’s what it was half a year ago when I checked the last time
      - diffeomorphism 3y ago
        > Cuda would be pointless for Apple, because cuda is for Nvidia only, no? That is the point. Cuda is widely used in particular for training as opposed to tuning and is just flat out not available. So you buy your nice $6000 machine and it just does not work for its intended purpose. > it’s the cheapest setup on the market that can get you this much VRAM. shared VRAM is not everything.

204 points by Jayakumark 3y ago

> will run only on NVIDIA GPUs for the preview I wonder why Microsoft helps nVidia, instead of using their own technology? Here’s an example: https://github.com/Const-me/Cgml https://github.com/Const-me/Cgml

capableweb 3y ago
Companies of the size of Microsoft usually function more like a country with various companies within, rather than one company with a unified view and unified goals. So probably, the group wasn't even aware of that technology because it's far away by either professional connection, or by personal/relationship connections, or they knew about it but had another goal than "maximize use of own stuff" and made the call that the tradeoffs wasn't worth it.
- Const-me 3y ago
  I don’t work for Microsoft, and I don’t care whether people are using Microsoft’s technologies. As a consumer, I’m not happy with nVidia’s monopoly in the field. I find it disappointing that Microsoft chose to play along, instead of disrupting the landscape. I think the only reason for that monopoly is the mental inertia of everyone involved. The complexity of these AI models is contained within the data in the models (gigabytes of numbers in these tensors), the GPU-running code is rather simple, most of that code is basic BLAS stuff. Unlike traditional GPGPU applications (FEM, numerical simulations, fluid dynamics), the compute kernels used in AI are easily portable across GPU APIs. Maybe when I have some free time, I should port my library to Linux + Vulkan, just to prove the point.
lostmsu 3y ago
Have you considered ILGPU? https://ilgpu.net/ https://ilgpu.net/
- Const-me 3y ago
  No, I have not considered ILGPU, it’s the first time I see that thing. Very interesting, but I’m not sure I would have used it for that project. In my experience, using any non-native GPU API is asking for troubles. On Windows, the native ones are D3D 11 and 12, on Linux and Android it’s often Vulkan, and on MacOS it’s either Metal or that newer thing they have built specifically for AI. It seems ILGPU only has backends for CUDA, OpenCL, and CPU SIMD. I have general suspicion towards custom compilers. Writing a good compiler is hard, for GPUs even harder due to the weird execution model and insufficient documentation from GPU vendors. HLSL compiler is supported by Microsoft, and compute shaders are used by many millions of gamers every day. Similarly, CUDA is supported by nVidia, usually pretty stable, my only issue with CUDA is vendor lock-in. I have an impression people are often unhappy with the quality and hardware compatibility of less popular GPU APIs like ROCm and OpenCL. BTW, I asked a friend to test my program on their low-and laptop. The laptop has some Intel core i3 with integrated GPU. The performance wasn’t great at about 1 token/second (single-channel memory), but at least my code worked. I’m not sure it would have worked on that computer if the backend was based on OpenCL.
pjmlp 3y ago
Their own technology is DirectML, not raw DirectX 11. https://blogs.windows.com/windowsdeveloper/2023/11/15/elevating-the-developer-experience-on-windows-with-new-ai-tools-and-productivity-tools/ https://blogs.windows.com/windowsdeveloper/2023/11/15/elevat... https://blogs.windows.com/windowsdeveloper/2023/12/14/directml-accelerating-ai-on-windows-now-with-npus/ https://blogs.windows.com/windowsdeveloper/2023/12/14/direct...

DeathArrow 3y ago

Microsoft also has a nice AI/ML framework called ML.NET.

franzb 3y ago

Why is this on GitHub?

Mashimo 3y ago
Why would it not be?
- SSLy 3y ago
  Because there's no real source included therein.
  - Mashimo 3y ago
    True. But it's easy to publish something on github, and you get an bug tracker. So .. I guess they do it because it's easy to do so?
    - capableweb 3y ago
      When even Microsoft uses their own platform in a non-intended way, you know it's an official Microsoft project.

mngdtt 3y ago

Installed this, and it didn't let me do anything. Despite the fact the readme says stuff runs locally, first it asked me to link to my github account, and then all models required me to ask someone (I think meta?) for permission on github or use a huggingface token or whatever. So I uninstalled it and now my wsl prompt starts with (base) and I don't know how to disable it and all my python scripts are broken because they can't find all the libraries I've installed from pip throughout the years. 0/10 would not recommend.

ZunarJ5 3y ago
You need to turn off the default conda environment. Try: conda config --set auto_activate_base false
- mngdtt 3y ago
  Thanks !!!
  - ZunarJ5 3y ago
    :)
dazzlefruit 3y ago
(base) sounds like Anaconda. Try typing conda deactivate?

a1o 3y ago

Curious it uses Ubuntu 18.04 with 20.04 and 22.04 already released and 24.04 just around the corner.

ourmandave 3y ago

RAG Project Coming soon! =(

davikr 3y ago

Anyone knows when Microsoft will release a local OCR model officially? I haven't seen anyone talking about this, but the one they're shipping with Snipping Tool (OneOcr) is top-tier and beats everything out there like Tesseract, easyOCR etc. The model is technically in everyone's Windows installs, but we don't have the C++ projected WinRT headers to use the Microsoft.Windows.Vision library.

capableweb 3y ago
Seems they've released a bunch already? Image-to-text models, filtered by "Microsoft": https://huggingface.co/models?pipeline_tag=image-to-text&sort=trending&search=microsoft https://huggingface.co/models?pipeline_tag=image-to-text&sor...
- davikr 3y ago
  Wow, very nice! Thank you, hadn't come across them in my search. It looks like it doesn't recognize more than one line at once, but combined with another model or algorithm to detect text bounding boxes it'd be handy.
  - CharlieDigital 3y ago
    Not sure if it's what's used in the Azure Document Intelligence service, but my experience with it is pretty good. The one somewhat unique offering in Azure is the Document Layout model which gives you back the OCR with titles, headers, paragraphs, and tables all labeled. This is really, really good for RAG since it's often useful to stuff the nearest header into the chunk of text when generating an embedding (much, much better results this way).

Jayakumark 3y ago

Adding link to VScode extension: https://marketplace.visualstudio.com/items?itemName=ms-windows-ai-studio.windows-ai-studio&ssr=false#overview https://marketplace.visualstudio.com/items?itemName=ms-windo...

blackoil 3y ago

Windows AI Studio is in real a Linux AI Studio as it needs WSL to run. Little funny

troupo 3y ago
Obligatory "is this the year of Linux in desktop" joke
- pjmlp 3y ago
  It is kind of ironic that the Year of Linux on the Desktop is all based on having Linux VMs across all major mainstream desktop OSes, even on ChromeOS, Crostini runs inside something similar to WSL.
severino 3y ago
But can you get this to run in Linux? From the article I can see it's a VSCode extension or something like that but the name kind of implies it's Windows only.
- diffeomorphism 3y ago
  > https://github.com/microsoft/windows-ai-studio/blob/main/QA.md https://github.com/microsoft/windows-ai-studio/blob/main/QA.... For now, but seems to be planned in the future.
  - severino 3y ago
    Yep, didn't read that. In the meantime, kudos to Canonical (and other parties) who helped Microsoft with WSL. Now they can also use Linux to produce Windows-only software! :D
    - baz00 3y ago
      The effort always stops when the commercial goal is achieved.

andruby 3y ago

Apple has good uniform hardware to enable this, but they are a product company and an “AI studio” would not fit their usual definition of a product. I do hope they are considering going in that direction though.

TheRoque 3y ago
That's the weird thing. They seem to have the best hardware for the price, for individuals to develop and use local LLMs, but so far they have been pretty quiet on all this.
- diffeomorphism 3y ago
  Highly depends on the price. Currently 1600€ gets you a laptop with 8gb shared memory, which surely is a nice device for other use cases but for developing local LLMs that seems awfully limiting. With upgraded ram (for just 230€ for each 8GB) and storage (just over 1000€ for 2tb; a samsung 990 pro is like 170€) that might be another story, but "check your specs before downloading this app" seems very un-appley. Also, no cuda support etc. Maybe if they make it exclusive to the mac studio?
  - kolinko 3y ago
    Cuda would be pointless for Apple, because cuda is for Nvidia only, no? As for the price - when you get to 32/64/96GB levels of RAM it’s the cheapest setup on the market that can get you this much VRAM. At least that’s what it was half a year ago when I checked the last time
    - diffeomorphism 3y ago
      > Cuda would be pointless for Apple, because cuda is for Nvidia only, no? That is the point. Cuda is widely used in particular for training as opposed to tuning and is just flat out not available. So you buy your nice $6000 machine and it just does not work for its intended purpose. > it’s the cheapest setup on the market that can get you this much VRAM. shared VRAM is not everything.