6 ms·

20x Faster Background Removal in the Browser Using ONNX Runtime with WebGPU

165 points by buss_jan 2y ago

wruza 2y ago
Interesting, there’s also node version in /packages.
- wruza 2y ago
  Tried it, and it's absolutely half-baked. Doesn't accept its own config typed param, messes up with own internal urls, cannot run from non-project dir. Although the segmentation quality is much better than that of `rembg`, the interface to it is just unamazing. Update: nope, it's sharper, but fails at different images at the same rate. gist: https://gist.github.com/sou-long/5c7cfee57f5399918c9072552afe2ec8 https://gist.github.com/sou-long/5c7cfee57f5399918c9072552af... (adapted from a real project, just for reference)
andrewstuart 2y ago
Worth noting that background removal is built in to Preview on Macos.
- dagmx 2y ago
  It’s also built into Safari and Photos on all their platforms and available as an API that can be called by any app https://developer.apple.com/wwdc23/10176 https://developer.apple.com/wwdc23/10176
  - oefrha 2y ago
    Huh, I've been copying background-removed subjects out of Preview and didn't realize there's a VisionKit API. Looks like it's quite easy to use too, I put together a quick and dirty script in a couple minutes and it worked wonderfully: import AppKit import VisionKit @main struct Script { static func main() async { let image = NSImage(contentsOfFile: "input.heic")! let view = ImageAnalysisOverlayView() let analyzer = ImageAnalyzer() let configuration = ImageAnalyzer.Configuration(.visualLookUp) let analysis = try! await analyzer.analyze(image, orientation: .up, configuration: configuration) view.analysis = analysis let subjects = await view.subjects for (index, subject) in subjects.enumerated() { let subjectImage = try! await subject.image let pngData = NSBitmapImageRep(data: subjectImage.tiffRepresentation!)!.representation( using: .png, properties: [:]) try! pngData?.write(to: URL(fileURLWithPath: "subject-\(index).png")) print("subject-\(index).png") } } }
- Abishek_Muthian 2y ago
  Was searching for an equivalent for Linux, came across rembg. https://github.com/danielgatis/rembg https://github.com/danielgatis/rembg
tlarkworthy 2y ago
Onnx is cool, the other option is tensorflow js which I have found quite nice as a usable matrix lib for JS with shockingly good perf.would love to know how well they compare
- dleeftink 2y ago
  Also shout out to Taichi and GPU.js for alternatives in this space. I've also had success with Hamster.js, that 'parallelizes' computations using Web workers instead of the GPU (who knows, in the future the two might be combined?).
  - salamo 2y ago
    They are probably two different use cases. Parallelizing with web workers could be faster for algorithms that do a lot of branching (minimax comes to mind) but if you can vectorize (matmuls for example) then GPU probably dominates.
    - dleeftink 2y ago
      It would be cool to implement some of these in either library to see how they stack up. In the Hamster.js case, I am envisioning each worker having access to a seperate GPU on your local machine..and having results come in asynchronously on the main thread. Massive in-browser simulations with access to existing JS visualisation packages would make real-time prototyping more feasible.
  - lukan 2y ago
    "who knows, in the future the two might be combined?" You can combine both today alreay and I experiment with it. The problem is still the high latency of the GPU. It takes ages, to get an answer and the timing is not consistent. That makes all scheduling for me a nightmare, when dividing jobs between the CPU and GPU. It would probably require a new hardware architectur, to make use of that in a sane way, so that GPU and CPU are more closely connected. (there are some designs aiming for this, as far as I know) edit: you probably meant hamsterS.js
38 2y ago
[flagged]
- deleted 2y ago
  [deleted]
tommek4077 2y ago
If I run it in a browser on my client, why going to a website in the first place?
- jazzyjackson 2y ago
  to resolve a short url to a piece of software i guess
  - PUSH_AX 2y ago
    This novel concept should have a catchy name…
    - jazzyjackson 2y ago
      why should i save software to disk when i can run it anytime i want in the browser? :p
- deleted 2y ago
  [deleted]
DaiPlusPlus 2y ago
Background Removal can be thought of as Foreground Segmentation, inverted. That is no trivial feat; my undergraduate thesis was on segmentation, but using only “mechanical” approaches, no NNs, etc), hence my appreciation! But here’s something I don’t understand: (And someone please correct me if I’m wrong!) - now I do understand that NNs are to software what FPGAs are to hardware, and the ability to pick any node and mess with it (delete, clone, more connections, less connections, link weights, swap-out the activation functions, etc) means they’re perfect for evolutionary-algorithms that mutate, spawn, and cull these NNs until they solve some problem (e.g. playing Super Mario on a NES (props to Tom7) or in this case, photo background segmentation. …now, assuming the analogy to FPGAs still holds, with NNs being an incredibly inefficient way to encode and execute steps in a data-processing pipeline (but very efficient at evolving that pipeline) - doesn’t it then mean that whatever process is encoded in the NN, it should both be possible to represent in some more efficient representation (I.e. computer program code, even if it’s highly parallelised) and that “compiling” it down is essential for performance? And if so, then why are models/systems like this being kept in NN form? (I look forward to revisiting this post a decade from now and musing at my current misconceptions)
- johndough 2y ago
  Neural networks are not trained with evolutionary algorithms because they are very slow, especially for the millions or billions of parameters that NNs have. Instead, stochastic gradient descent is used for training, which is much more efficient.
- sitkack 2y ago
  There is some work to convert NNs to decision trees. https://towardsdatascience.com/neural-networks-as-decision-trees-89cd9fdcdf6a https://towardsdatascience.com/neural-networks-as-decision-t... https://arxiv.org/abs/2210.05189 https://arxiv.org/abs/2210.05189 I haven't reviewed any of it, I only know of it tangentially. https://www.semanticscholar.org/paper/Converting-A-Trained-Neural-Network-To-a-Decision-Boz/fb6172737873a69bd8d0117c88301121b5cabfa3 https://www.semanticscholar.org/paper/Converting-A-Trained-N... Distilling a Neural Network Into a Soft Decision Tree https://arxiv.org/abs/1711.09784 https://arxiv.org/abs/1711.09784 GradTree: Learning Axis-Aligned Decision Trees with Gradient Descent https://arxiv.org/abs/2305.03515 https://arxiv.org/abs/2305.03515
jvdvegt 2y ago
MS teams does this already, right? (I assume they do, as it didn't work in Firefox until recently) Or do they do it server side?
- afro88 2y ago
  I'm pretty sure they do it client side. The latency on your video preview is non existent.
forgotusername6 2y ago
"Therefore, the first run of the network will take ~300 ms and consecutive runs will be ~100 ms" I only skimmed the article, but I don't think they mention the size of the image. 100ms is not that impressive when you consider that you need to be three times as fast for acceptable video frame rate.
- diggan 2y ago
  > I only skimmed the article, but I don't think they mention the size of the image. 100ms is not that impressive when you consider that you need to be three times as fast for acceptable video frame rate. You don't need three times as fast for acceptable video frame rates in a video editor, you need a system that allows you to cache "rendered" frames so when the user does an edit, it renders to this cache, then once done, the user can play it back in real-time. This is essentially how all video editors handle edits on clips/video today. Some effects/edits can be applied in real-time, but the more advanced one (I'd say background removal being one of them) works with this type of caching system.
asdsfasdfasdf 2y ago
[flagged]
pjmlp 2y ago
As long as one uses a Chrome distribution. WebGPU is at least one year away of becoming usable for cross browser deployment.
- diggan 2y ago
  > WebGPU is at least one year away of becoming usable for cross browser deployment. In Firefox it seems to be behind a feature flag and Safari seems to have it in it's "Technology Preview" (some sort of release candidate?), so seems closer that I at least though.
  - pjmlp 2y ago
    Firefox has had it as feature flag for at least one year now, Safari just announced the technology preview during WWDC updates. WebGL 2.0 took almost a decade to be fully supported, and still has issues on Safari, don't expect WebGPU to be any faster. Also note that Google is the culprit why WebGL Compute did not happen, WebGPU was going to sort all problems, and even though they use DirectX on Windows, apparently it was a big issue to use Metal Compute on Apple instead of OpenGL, and then they ended up improving Angle on top of Metal anyway. Web politics.
rubi1945 2y ago
[flagged]
deleted 2y ago
[deleted]
Naira_Nicol 2y ago
[flagged]
- adzm 2y ago
  this sounds like an LLM for sure.
  - Naira_Nicol 2y ago
    Yes, ONNX Runtime with WebGPU for removing backgrounds in web browsers is a required step forward in web-based image processing. It's a fast engine for running machine learning models trained with ONNX. And because WebGPU is designed for web-based graphics and compute tasks, its functionality is similar to that of native GPU programming. It's handel the complex image processing tasks like background removal to happen quickly and in real-time. This approach can deliver up to a 20x speed boost compared to traditional CPU methods. It attracts user interactions for sure. Developers convert machine learning models trained in tasks like semantic segmentation to ONNX format using ONNX. These models run efficiently in browsers with ONNX Runtime and accelerated WebGPU computations. This integration democratizes access to advanced image processing capabilities that were previously limited to native apps, and now they're available with standard web technologies.

165 points by buss_jan 2y ago

wruza 2y ago

Interesting, there’s also node version in /packages.

wruza 2y ago
Tried it, and it's absolutely half-baked. Doesn't accept its own config typed param, messes up with own internal urls, cannot run from non-project dir. Although the segmentation quality is much better than that of `rembg`, the interface to it is just unamazing. Update: nope, it's sharper, but fails at different images at the same rate. gist: https://gist.github.com/sou-long/5c7cfee57f5399918c9072552afe2ec8 https://gist.github.com/sou-long/5c7cfee57f5399918c9072552af... (adapted from a real project, just for reference)

andrewstuart 2y ago

Worth noting that background removal is built in to Preview on Macos.

dagmx 2y ago
It’s also built into Safari and Photos on all their platforms and available as an API that can be called by any app https://developer.apple.com/wwdc23/10176 https://developer.apple.com/wwdc23/10176
- oefrha 2y ago
  Huh, I've been copying background-removed subjects out of Preview and didn't realize there's a VisionKit API. Looks like it's quite easy to use too, I put together a quick and dirty script in a couple minutes and it worked wonderfully: import AppKit import VisionKit @main struct Script { static func main() async { let image = NSImage(contentsOfFile: "input.heic")! let view = ImageAnalysisOverlayView() let analyzer = ImageAnalyzer() let configuration = ImageAnalyzer.Configuration(.visualLookUp) let analysis = try! await analyzer.analyze(image, orientation: .up, configuration: configuration) view.analysis = analysis let subjects = await view.subjects for (index, subject) in subjects.enumerated() { let subjectImage = try! await subject.image let pngData = NSBitmapImageRep(data: subjectImage.tiffRepresentation!)!.representation( using: .png, properties: [:]) try! pngData?.write(to: URL(fileURLWithPath: "subject-\(index).png")) print("subject-\(index).png") } } }
Abishek_Muthian 2y ago
Was searching for an equivalent for Linux, came across rembg. https://github.com/danielgatis/rembg https://github.com/danielgatis/rembg

tlarkworthy 2y ago

Onnx is cool, the other option is tensorflow js which I have found quite nice as a usable matrix lib for JS with shockingly good perf.would love to know how well they compare

dleeftink 2y ago
Also shout out to Taichi and GPU.js for alternatives in this space. I've also had success with Hamster.js, that 'parallelizes' computations using Web workers instead of the GPU (who knows, in the future the two might be combined?).
- salamo 2y ago
  They are probably two different use cases. Parallelizing with web workers could be faster for algorithms that do a lot of branching (minimax comes to mind) but if you can vectorize (matmuls for example) then GPU probably dominates.
  - dleeftink 2y ago
    It would be cool to implement some of these in either library to see how they stack up. In the Hamster.js case, I am envisioning each worker having access to a seperate GPU on your local machine..and having results come in asynchronously on the main thread. Massive in-browser simulations with access to existing JS visualisation packages would make real-time prototyping more feasible.
- lukan 2y ago
  "who knows, in the future the two might be combined?" You can combine both today alreay and I experiment with it. The problem is still the high latency of the GPU. It takes ages, to get an answer and the timing is not consistent. That makes all scheduling for me a nightmare, when dividing jobs between the CPU and GPU. It would probably require a new hardware architectur, to make use of that in a sane way, so that GPU and CPU are more closely connected. (there are some designs aiming for this, as far as I know) edit: you probably meant hamsterS.js

38 2y ago

[flagged]

deleted 2y ago
[deleted]

tommek4077 2y ago

If I run it in a browser on my client, why going to a website in the first place?

jazzyjackson 2y ago
to resolve a short url to a piece of software i guess
- PUSH_AX 2y ago
  This novel concept should have a catchy name…
  - jazzyjackson 2y ago
    why should i save software to disk when i can run it anytime i want in the browser? :p
deleted 2y ago
[deleted]

DaiPlusPlus 2y ago

Background Removal can be thought of as Foreground Segmentation, inverted. That is no trivial feat; my undergraduate thesis was on segmentation, but using only “mechanical” approaches, no NNs, etc), hence my appreciation! But here’s something I don’t understand: (And someone please correct me if I’m wrong!) - now I do understand that NNs are to software what FPGAs are to hardware, and the ability to pick any node and mess with it (delete, clone, more connections, less connections, link weights, swap-out the activation functions, etc) means they’re perfect for evolutionary-algorithms that mutate, spawn, and cull these NNs until they solve some problem (e.g. playing Super Mario on a NES (props to Tom7) or in this case, photo background segmentation. …now, assuming the analogy to FPGAs still holds, with NNs being an incredibly inefficient way to encode and execute steps in a data-processing pipeline (but very efficient at evolving that pipeline) - doesn’t it then mean that whatever process is encoded in the NN, it should both be possible to represent in some more efficient representation (I.e. computer program code, even if it’s highly parallelised) and that “compiling” it down is essential for performance? And if so, then why are models/systems like this being kept in NN form? (I look forward to revisiting this post a decade from now and musing at my current misconceptions)

johndough 2y ago
Neural networks are not trained with evolutionary algorithms because they are very slow, especially for the millions or billions of parameters that NNs have. Instead, stochastic gradient descent is used for training, which is much more efficient.
sitkack 2y ago
There is some work to convert NNs to decision trees. https://towardsdatascience.com/neural-networks-as-decision-trees-89cd9fdcdf6a https://towardsdatascience.com/neural-networks-as-decision-t... https://arxiv.org/abs/2210.05189 https://arxiv.org/abs/2210.05189 I haven't reviewed any of it, I only know of it tangentially. https://www.semanticscholar.org/paper/Converting-A-Trained-Neural-Network-To-a-Decision-Boz/fb6172737873a69bd8d0117c88301121b5cabfa3 https://www.semanticscholar.org/paper/Converting-A-Trained-N... Distilling a Neural Network Into a Soft Decision Tree https://arxiv.org/abs/1711.09784 https://arxiv.org/abs/1711.09784 GradTree: Learning Axis-Aligned Decision Trees with Gradient Descent https://arxiv.org/abs/2305.03515 https://arxiv.org/abs/2305.03515

jvdvegt 2y ago

MS teams does this already, right? (I assume they do, as it didn't work in Firefox until recently) Or do they do it server side?

afro88 2y ago
I'm pretty sure they do it client side. The latency on your video preview is non existent.

forgotusername6 2y ago

"Therefore, the first run of the network will take ~300 ms and consecutive runs will be ~100 ms" I only skimmed the article, but I don't think they mention the size of the image. 100ms is not that impressive when you consider that you need to be three times as fast for acceptable video frame rate.

diggan 2y ago
> I only skimmed the article, but I don't think they mention the size of the image. 100ms is not that impressive when you consider that you need to be three times as fast for acceptable video frame rate. You don't need three times as fast for acceptable video frame rates in a video editor, you need a system that allows you to cache "rendered" frames so when the user does an edit, it renders to this cache, then once done, the user can play it back in real-time. This is essentially how all video editors handle edits on clips/video today. Some effects/edits can be applied in real-time, but the more advanced one (I'd say background removal being one of them) works with this type of caching system.

asdsfasdfasdf 2y ago

[flagged]

pjmlp 2y ago

As long as one uses a Chrome distribution. WebGPU is at least one year away of becoming usable for cross browser deployment.

diggan 2y ago
> WebGPU is at least one year away of becoming usable for cross browser deployment. In Firefox it seems to be behind a feature flag and Safari seems to have it in it's "Technology Preview" (some sort of release candidate?), so seems closer that I at least though.
- pjmlp 2y ago
  Firefox has had it as feature flag for at least one year now, Safari just announced the technology preview during WWDC updates. WebGL 2.0 took almost a decade to be fully supported, and still has issues on Safari, don't expect WebGPU to be any faster. Also note that Google is the culprit why WebGL Compute did not happen, WebGPU was going to sort all problems, and even though they use DirectX on Windows, apparently it was a big issue to use Metal Compute on Apple instead of OpenGL, and then they ended up improving Angle on top of Metal anyway. Web politics.

rubi1945 2y ago

[flagged]

deleted 2y ago

[deleted]

Naira_Nicol 2y ago

[flagged]

adzm 2y ago
this sounds like an LLM for sure.
- Naira_Nicol 2y ago
  Yes, ONNX Runtime with WebGPU for removing backgrounds in web browsers is a required step forward in web-based image processing. It's a fast engine for running machine learning models trained with ONNX. And because WebGPU is designed for web-based graphics and compute tasks, its functionality is similar to that of native GPU programming. It's handel the complex image processing tasks like background removal to happen quickly and in real-time. This approach can deliver up to a 20x speed boost compared to traditional CPU methods. It attracts user interactions for sure. Developers convert machine learning models trained in tasks like semantic segmentation to ONNX format using ONNX. These models run efficiently in browsers with ONNX Runtime and accelerated WebGPU computations. This integration democratizes access to advanced image processing capabilities that were previously limited to native apps, and now they're available with standard web technologies.