11 ms·

Segment Anything Model (SAM) can "cut out" any object in an image

725 points by crakenzak 4y ago

crakenzak 4y ago
This is going along with the new Segment Anything Model paper Meta AI just released: Paper: https://scontent-sea1-1.xx.fbcdn.net/v/t39.2365-6/10000000_6331779526880473_6748528980292947838_n.pdf?_nc_cat=102&ccb=1-7&_nc_sid=3c67a6&_nc_ohc=lnYqcLNTtLQAX8kMpyF&_nc_ht=scontent-sea1-1.xx&oh=00_AfDW5q4b2Po8QQgtD94Cyc18IkLCP-z5JYevagqZbriPkg&oe=6432ACEA https://scontent-sea1-1.xx.fbcdn.net/v/t39.2365-6/10000000_6... Announcement: https://ai.facebook.com/blog/segment-anything-foundation-model-image-segmentation/ https://ai.facebook.com/blog/segment-anything-foundation-mod... Code & Model Weights: https://github.com/facebookresearch/segment-anything https://github.com/facebookresearch/segment-anything
- lofaszvanitt 4y ago
  Why can't they give proper filenames to these research papers. This drives me nuts.
- ftxbro 4y ago
  if Tim Berners-Lee saw that paper link he would have never allowed the url to be invented
  - LoganDark 4y ago
    I'm so shocked by how almost every query parameter is required and there's even a freaking signature for validating the URL itself. -Emily
    - CyberDildonics 4y ago
      Who is Emily?
      - revolvingocelot 4y ago
        Someone whose signature validates the grandparent comment itself, I assume
      - LoganDark 4y ago
        That would be me. We have DID and add a signature when the person commenting isn't actually Logan. It's not actually intended to be content. -Emily
        CyberDildonics 4y ago
        Why not have two different user names? What if you get logged out and the other identity is the one that knows the password?
        LoganDark 4y ago
        > Why not have two different user names? We do on some platforms like Discord. > What if you get logged out and the other identity is the one that knows the password? Before we moved to a password manager we used the same password everywhere. Now, obviously, we have a password manager. -Emily
        CyberDildonics 4y ago
        Don't you run the risk of only one person knowing the password to the password manager?
        LoganDark 4y ago
        > Don't you run the risk of only one person knowing the password to the password manager? No, because systems don't necessarily work that way. For us, the boundaries between our members aren't totally uncrossable. Information has gotten through in the past when it would be especially important or needed. Though I guess it's funny that this topic comes up under a post called "segment anything". I guess our brain did that~ -Emily
    - TylerE 4y ago
      Signature is likely so the caching layer doesn't have to do all that validation, at least as an initial check.
      - egeozcan 4y ago
        If I'm remembering correctly, people were accessing the original versions of the images uploaded to instagram (removing filters, and even masks) using parameter engineering (We love putting engineering at the end of everything that requires more than 5 seconds of thought these days, so why not). That could be why.
- jauer 4y ago
  That paper link is a CDN URL that is dynamically generated to point to your closest POP when you load the abstract. It will be different for many people and will break eventually. Abstract: https://ai.facebook.com/research/publications/segment-anything/ https://ai.facebook.com/research/publications/segment-anythi...
- Loranubi 4y ago
  https://arxiv.org/abs/2304.02643 https://arxiv.org/abs/2304.02643
minimaxir 4y ago
You know an AI project is serious when it has its own domain name instead of a subdomain.
benatkin 4y ago
They seem to avoid using their own brand a lot. They have a zillion domain names and they register a new one and don't use the logo except in the favicon and footer. I've seen similar stuff including divesting OSS projects like PyTorch and GraphQL which Google wouldn't. To me that's tacit admission that the Facebook and Meta names are tarnished. And they are, by the content they showed users in Myanmar with the algorithmic feed, and by Cambridge Analytica. Maybe the whole "Meta" name is no different from the rebranding of Philip Morris.
- smoldesu 4y ago
  On the one hand, sure. Facebook's brand is about as hip as a bag of Werther's Originals. On the other hand, this is one of those things (like VR) that is a distinctly non-Facebook project. It makes no sense to position or market this as "Facebook" research. The Homepod isn't called the iPod Home for obvious reasons, so it stands to reason that Facebook execs realized selling someone a "Facebook Quest" sounds like a metaphor for ayahuasca. It's not entirely stupid to rebrand, especially considering how diverse (and undeniably advanced) they've become in fields like AI and VR.
  - oefnak 4y ago
    I actively avoid everything that has anything to do with Facebook, and I can't be the only one.
    - smoldesu 4y ago
      Yeah, me too. I also avoid everything Apple and Google makes, but I'm not going to pretend like the Alphabet rebranding is their attempt at hiding who they are.
      - xiphias2 4y ago
        Alphabet wasn't a rebranding: the founding billionaires got bored of Google, and wanted to take out a few billion dollars per year out of it to create new toys without sharing it with Google.
ren_engineer 4y ago
what do you think facebook's gameplan is here? Are they trying to commoditize AI by releasing this and Llama as a move against OpenAI, Microsoft, and Google? They had to have known the Llama weights would be leaked and now they are releasing this
- _the_inflator 4y ago
  I think Meta's gameplan is complex. Inspiration as well as adoption, not stepping on the toes of regulators prolly another intention. Have a look at PyTorch for example. Massively popular ML framework, with its lots of interesting projects running. If Meta frequently shares their "algorithms" they take the blame out of its usage. After all, who is to blame when everybody does "it" and you are very open about it. Use cases, talent visibility as well as attraction also plays a role. After all, Google was so fancied, due to its many open source projects. "Show, don't tell".
- jayd16 4y ago
  Well there's some patent offense and defense in making and releasing research papers. There's some recruiting aspects to it. Its also a way to commoditize your inverse if you assume this sort of stuff brings AR and the metaverse closer to reach.
- dragonwriter 4y ago
  I think cranking out open source projects like this raises Meta AI’s profile and helps them attract attention and people, and I don’t think selling AI qua AI is their business plan, selling services built on top is. And commoditized AI means that the AI vendors don’t get to rent-seek on people doing that, whereas narrowly controlled monopoly/oligopoly AI would mean that the AI vendors extract the value produced by downstream applications.
  - vagabund 4y ago
    I've always half-believed that the relatively open approach to industry research in ML was a result of the inherent compute-based barrier to entry for productizing a lot of the insights. Collaborating on improving the architectural SoTA gets the handful of well-capitalized incumbents further ahead more quickly, and solidifies their ML moat before new entrants can compete. Probably too cynical, but you can potentially view it as a weak form of collusion under the guise of open research.
dimatura 4y ago
The network architecture and scale don't seem to be a big departure from recent SOTA, but a pretty massive amount of labeled data went into it. And it seems to work pretty well! The browser demo is great. This will probably see a lot of use, especially considering the liberal licensing.
- bjacobt 4y ago
  I apologize if this is obvious, but are both the model and checkpoint (as referenced in getting started section in readme) Apache 2.0? Can it be used for commercial applications?
  - dimatura 4y ago
    As far as I can tell, it can. The code itself has a `LICENSE` file with the Apache license, and the readme says "The model is licensed under the Apache 2.0 license.". Strangely, the FAQ in the blog post doesn't address this question, which I expect will be frequent.
    - phkahler 4y ago
      Isn't Apache 2 a free software license without some of the GPLv3 things some don't like? I think a more BSD would be better, or LGPL. Either would be more business friendly.
      - MacsHeadroom 4y ago
        LGPL is not business friendly at all. It's among the least business friendly licenses there is. Apache 2.0 is slightly more business friendly than BSD. With some caveats, software licenses from most to least business friendly roughly go: Apache > BSD > MIT > MPL > LGPL > GPL > AGPL
        kyle-rb 4y ago
        LGPL is more business friendly than GPL; it's literally "lesser" GPL. You can use LGPL in commercial, closed-source projects as long as you keep the LGPL code in a separate dynamically linked library, e.g. a DLL, and provide a way for users to swap it out for their own patched DLL if they wish. (Plus some other license terms.) Also, you can always use LGPL code under the terms of the GPL, so there's no way LGPL is more restrictive than GPL.
cloudking 4y ago
Pretty cool, Runway has a similar green screening feature that can 1-click segment a subject from the background across an entire video: https://runwayml.com/ai-magic-tools/ https://runwayml.com/ai-magic-tools/
MrGando 4y ago
This is great, runs very fast in Chrome for me.
yeldarb 4y ago
Wow, this is pretty epic. I put it through its paces on a pretty wide variety of images that have tripped up recent zero-shot models[1] and am thoroughly impressed. We have a similar "Smart Polygon" tool[2] built into Roboflow but this is next level. Having the model running in the browser makes it so much more fun to use. Stoked it's open source; we're going to work on adding it to our annotation tool ASAP. [1] Some examples from Open Flamingo last week https://news.ycombinator.com/item?id=35348500 https://news.ycombinator.com/item?id=35348500 [2] https://blog.roboflow.com/automated-polygon-labeling-computer-vision/ https://blog.roboflow.com/automated-polygon-labeling-compute...
- teruakohatu 4y ago
  It would be nice if it could run entirely in the browser but the image embedding model is absolutely huge.
  - blandcoffee 4y ago
    Can you expand further on what you mean by this?
    - teruakohatu 4y ago
      The SAM model is small (4m params) , but requires image embedding to be computed from what is I think a 600m params model. Right now the demo uploads the image to get the embeddings, then runs the actual segmentation locally.
neom 4y ago
yikes. I went to film school in the early 2000s and spent hours and hours on levels/HDR based masking, I've used the adobe tools recently and they're good... this is... yikes...I wonder how people in their mid 20s today learning photoshop are going to deal with their graduating jobs.
- jacquesm 4y ago
  Not. This is homing in on the SF UI that Deckard used in Blade Runner. https://www.youtube.com/watch?v=hHwjceFcF2Q https://www.youtube.com/watch?v=hHwjceFcF2Q All it takes is a couple of tools glued together and you're getting there.
  - marstall 4y ago
    "gimme a hardcopy right there."
  - zestyping 4y ago
    The most unrealistic part of this sequence is that the printer works on the first try.
- ChatGTP 4y ago
  Don’t you get it, no one is supposed to have a job? Silicon Valley is going to buy us all the things we need ?
arduinomancer 4y ago
This exists as a feature on iOS You can long press on an image and it cuts out whatever thing it thinks you're pressing on They also use it in interesting ways, like making stuff in the photo slightly overlap the clock on the lockscreen Does anyone know if that works the same way as this?
- neom 4y ago
  This is: understand everything in the image as elements, subject or whatever.
- hbn 4y ago
  It would still be nice if iOS had some kind of interface like this where you can nudge it in the right direction if it's confusing something like a jacket and the background. iOS gives its best attempt which is usually pretty good, but if it didn't get it right you're basically SOL.
dang 4y ago
Related: Meta New Segmentation Model - https://news.ycombinator.com/item?id=35453625 https://news.ycombinator.com/item?id=35453625 - April 2023 (7 comments)
swframe2 4y ago
The best solution I've seen is https://github.com/xuebinqin/DIS https://github.com/xuebinqin/DIS. You should try the DIS example images at the SAM site. The main issue I have with DIS is that creating the labels of my own dataset is super expensive (I think it might be easier to generate the training data using stable diffusion rather than human labelling)
- pradyumnabang 4y ago
  Forgive my ignorance, but why is creating the labels super expensive? How much does it cost per image and what's your dataset size ?
  - swframe2 4y ago
    It is related to subpixel labelling. When a line/curve in the foreground is smaller than a pixel you end up having to edit the mask one pixel at a time. The authors of DIS are working on a new dataset and model which should work for my use case. BTW, I used DIS to create the labels of a batch of 20 images, I manually corrected the labels and used them to fine tune a new model. That worked well but still it took me several hours to edit labels. I tried using stable diffusion generated labels several weeks ago but I think with controlnet and other advances I should try again. (My dataset is about 100k images. I probably only need to label about 10k to fine tune DIS).
AdilZtn 4y ago
That's amazing! This model is a huge opportunity to create annotated data (with decent quality) for just a few dollars. People will iterate more quickly with this kind of foundation model.
justinator 4y ago
Demo is running slow - cutting out is an impressive ability - I'm to assume it also fills in the background? If so: that's next level. Maybe that Photoshop monthly subscription will be worth it (providing this sort of ability is going to be baked in with AdobeAI's version soon)
- exodust 4y ago
  It doesn't fill in the background and has nothing to do with Adobe. You could bring the cut-outs back into Photoshop. I tried that, but this SAM tool reduces the size of the cut-outs slightly, so the cut-out won't match original image dimensions.
  - justinator 4y ago
    >It doesn't fill in the background Thanks - like I wrote, the demo was DOA for me. >and has nothing to do with Adobe. I know. Masking and selecting is something you do in Adobe products and Adobe will be coming out with their own version of this (if I were a betting man)
syntheweave 4y ago
Finally, I'll be able to fill line art with flat colors without fussing around with thresholds and painting in boundaries. (It does have difficulty finding the smallest possible area, but it's a significant advance over most existing options since in my brief test, it can usually spot the entire silhouette of figures, which is where painting a boundary is most tedious).
reaperman 4y ago
Multiple block diagrams and the paper note that one of the inputs is supposed to be "text", but none of the example Jupyter notebooks or the live demo page show how to use those. I'm assuming just run the text into CLIP, take the resulting embedding, and throw it directly in as a prompt, which then gets re-encoded by the SAM prompt encoder? > "Prompt encoder. We consider two sets of prompts: sparse (points, boxes, text) and dense (masks). We represent points and boxes by positional encodings [95] summed with learned embeddings for each prompt type and free-form text with an off-the-shelf text encoder from CLIP [82]. Dense prompts (i.e., masks) are embedded using convolutions and summed element-wise with the image embedding." Edit: Found the answer myself: https://github.com/facebookresearch/segment-anything/issues/4 https://github.com/facebookresearch/segment-anything/issues/...
- deleted 4y ago
  [deleted]
richardw 4y ago
Surely this changes the security camera game? No more being fooled by clouds going overhead.
bobsmooth 4y ago
Seeing it run on a headset is the coolest part. Lots of applications for AR.
sashank_1509 4y ago
Extremely impressive system. Blows everything else (including CLIP from OpenAI) out of the water. We are inching closer to solving Computer Vision!
- wongarsu 4y ago
  It's really impressive, and better than anything I've seen, but is it really leagues better than whatever Photoshop is using? Of course being on github and permissively license is huge.
  - vanjajaja1 4y ago
    My question exactly, didn't photoshop already solve this like 5+ years ago?
    - dymk 4y ago
      Have you used Photoshop's magic wand tool in the last 5 years? No, it's nowhere close to this good.
      - Auracle 4y ago
        They have a newer “object selection” tool that’s similar.
      - Kaijo 4y ago
        The Magic Wand tool mostly just selects similarly colored pixels based on a simple algorithm. The Object Selection, Select Subject and Sky Replacement tools use AI detection and can be configured to run locally or on Adobe's GPUs. Having played with this demo, they seem in a similar league to me.
        drummojg 4y ago
        I'm just happy that a tool that PropriAdobe has sat on and crowed about for years is soon to be a click away for the masses and the hackers.
code51 4y ago
It’s interesting that (clearly visible) text parts that cannot be handled properly by most OCR approaches also get left out by SAM in auto-predictions.
jonplackett 4y ago
This would make a great input for ControlNet
RomanPushkin 4y ago
Stalin's dream https://en.wikipedia.org/wiki/Censorship_of_images_in_the_Soviet_Union https://en.wikipedia.org/wiki/Censorship_of_images_in_the_So...
geenew 4y ago
That’s it for me
nielsbot 4y ago
Sigh. Does not work in Safari on macOS (ARM). Works in Chrome though.
- subarctic 4y ago
  It seems to work in firefox on macOS (ARM) fwiw
fzliu 4y ago
Computer vision seems to be gravitating heavily towards self-attention. While the results here are impressive, I'm not quite convinced that vision encoders are the right way forward. I just can't wrap my head around how discretizing images, which are continuous in two dimensions, into patches is the most optimal way to do visual recognition. What's preventing us from taking something like convnext or a hybrid conv/attention model and hooking that up to a decoder stack? I fee like the results would be similar if not better. EDIT: Clarifying that encoder/decoder refers to the transformer stack, not an autoencoder.
- neodypsis 4y ago
  > What's preventing us from taking something like convnext or a hybrid conv/attention model and hooking that up to a decoder? I fee like the results would be similar if not better. You mean like in an U-Net architecture?
- skybrian 4y ago
  Google seems to be doing it all with transformers. It's not open source, though: > Here we highlight some results of ViT-22B. Note that in the paper we also explore several other problem domains, like video classification, depth estimation, and semantic segmentation. https://ai.googleblog.com/2023/03/scaling-vision-transformers-to-22.html https://ai.googleblog.com/2023/03/scaling-vision-transformer...
- intalentive 4y ago
  IMO optimal visual recognition should be sensorimotor-based and video-first. In the real world, action and perception are intertwined. Supervised training on static pixel arrays seems backward and primitive.
- sorenjan 4y ago
  MLP-Mixer uses only multi-layer perceptrons. It was released in 2021, but ReBotNet that was released this year uses mixer layers too. Still uses patches though, but they're mixing data between patches.
subarctic 4y ago
The demo is pretty cool but it looks like you can just select things and have it highlight them in blue - is there a way to remove objects from the image and have the background filled in behind them?
- dymk 4y ago
  You could probably content-aware fill the area that SAM identifies with another tool
  - exodust 4y ago
    Yeh I did that, but it's not a great experience. For some reason this tool makes a slightly smaller mask than is found in the original image. So when you copy the masked area back into Photoshop, it doesn't match. Almost there, but not quite.
syrusakbary 4y ago
This is awesome. If you try the demo they provide [0], the inference is handled purely in the client using a ONNX model that only weights around ~8Mb [1] [2]. Really impressive stuff! Congrats to the team that achieved it [0] https://segment-anything.com/demo https://segment-anything.com/demo [1] https://segment-anything.com/model/interactive_module_quantized_592547_2023_03_19_sam6_long_uncertain.onnx https://segment-anything.com/model/interactive_module_quanti... [2] https://segment-anything.com/model/interactive_module_quantized_592547_2023_03_20_sam6_long_all_masks_extra_data_with_ious.onnx https://segment-anything.com/model/interactive_module_quanti...
- kevmo314 4y ago
  It isn't purely client-side. The embeddings are generated on the server so your image is still sent to Meta for processing. https://github.com/facebookresearch/segment-anything/issues/16 https://github.com/facebookresearch/segment-anything/issues/...
  - trzy 4y ago
    The linked issue says nothing of the sort. Are you sure?
    - GaggiX 4y ago
      SAM uses a CLIP-H as an image encoder, a very large encoder that obviously does not fit in 8 MB. Also read the "Efficient & flexible model design" section on the page. Also in the FAQ: "How big is the model? The image encoder has 632M parameters. The prompt encoder and mask decoder have 4M parameters."
      - GaggiX 4y ago
        I cannot edit the comment anymore but I should have written MAE ViT-H and not CLIP-H (same size but CLIP models are trained in a different way).
        4y ago
georgelyon 4y ago
It seems like the output of this model is masks, but for cropping you really need to be able to pull partial color out of certain pixels (for example, pulling a translucent object out from of a colored background). I tried the demo, and it fails pretty miserably on a vase. Anyone know of a model that can do this well?
- teruakohatu 4y ago
  This is a segmentation model, it 'just' creates segmentation masks. You then take the mask and cut out whatever you need from the image itself. If you need to figure out how to remove a transparentency that is another whole problem, probably a generative problem (generate a new texture for the vase).
dekhn 4y ago
I have a small number of naive questions. I already have a fine-tuned tardigrade detection model that gives me tardigrade bounding boxes (data comes from labelled images on my microscope). I want to do masks as well. Right now my home server w/ an RTX 2080 is able to do mask prediction in about ~4 seconds (I'm running the sample script in "directory" mode) per image (640x480). I'd love to be able to get the first mask back in 0.1ms, so I can do 10FPS on the scope. Is there a practical way to speed things up (my guess would be buying the absolute fastest GPU I can afford)? I can run the object detection to get a reasonable seed location, is that what the paper means by prompting?
- bnqscrtm 4y ago
  A 4s run time for object segmentation at 640x480 sounds like it's not using the GPU at all. Something like that should run on a VGA image in at most a few hundred ms. For the second part of the question, a 2080 should get you close to 10FPS operation. For a ballpark estimate, using an off-the-shelf repo like Ultralytics's YOLOv5 lets you run object detection (not masking) at something like 100FPS. Masking should not add that much overhead. w.r.t. GPUs yes, these days more money equals more speed for GPU NN inference, though there are diminishing returns. A 3090 might get you the best bang for your buck these days while still having enough VRAM to run fancier models which may need more than the 12 GiB many other GPUs have. Finally, I haven't read the paper too carefully but I believe that by prompting they mean that you have the option of describing in human language what you want the model to select, rather than the model being "hardwired" to do this. In other words, you could prompt the model to "segment the red car only" and it would do it, rather than just having the model blindly segment every object in the image, and then relying on custom scripting to potentially post-process these segments.
  - GC_tris 4y ago
    Yeah, a 3090 should do well. If you want to try it on one reach out to me (email in profile). We rent those out in the cloud. Would allow you to confirm performance before buying one for local use.
  - dekhn 4y ago
rhacker 4y ago
Is it just me or is it really really really slow?
ericmathison 4y ago
This reminds me of https://cleanup.pictures/ https://cleanup.pictures/ which actually removes objects from the original.
bfung 4y ago
Def. cool demo. As a polished product & ease of use, still has ways to go. Apple’s cut&paste of subjects in photos, while lacking SAM’s generality, is masterfully executed - even my 70yr old mom can use it.
retox 4y ago
The TV/VHS example image does not do well.
an_aparallel 4y ago
Ahh - finally - we can now hyperlink real life. Perfect. HTRL (Hypertext Reality Language) <human></human> <ad></ad>
jacooper 4y ago
> The model is licensed under the Apache 2.0 license. Finally, an actually open model
- novaRom 4y ago
  and it will work on any smartphone because: The image encoder has 632M parameters. The prompt encoder and mask decoder have 4M parameters.
tgv 4y ago
Impressive, but not really perfect, is it? In sa_10016721, it misses the obvious, and in sa_10020386, it misses most of the train in the center, and a bunch of the parked cars (pretty random). In sa_10179757, it labels 3 out of 4 letters of the shipping company's name (?), and a handful of windows, and while perhaps it sees the ship as one piece, the people in the foreground are split in many parts.
fn-mote 4y ago
Kind of off topic, but: I've never seen such crappy issues filed in a repo [1]. I don't read issues for major repositories, so perhaps this is standard? There are a ton of one line "issues", no clear example, test case, attempt to debug, not even a pull request for the one that points out a typo in the README. Gross. It seems like none of these issues will never be read because they are going to drown in garbage. [1] https://github.com/facebookresearch/segment-anything/issues https://github.com/facebookresearch/segment-anything/issues
- MF-DOOM 4y ago
  Since anyone can open an issue, they’re bound to be low quality unless moderated.
  - kristopolous 4y ago
    some trackers handle this by asking pointed questions. "Steps to reproduce" "platforms affected" "expected behavior" etc. Microsoft knows this. Get on the ball, guys.
contingencies 4y ago
See also http://remove.bg http://remove.bg
vietanhdev 4y ago
[dead]

725 points by crakenzak 4y ago

This is going along with the new Segment Anything Model paper Meta AI just released: Paper: https://scontent-sea1-1.xx.fbcdn.net/v/t39.2365-6/10000000_6331779526880473_6748528980292947838_n.pdf?_nc_cat=102&ccb=1-7&_nc_sid=3c67a6&_nc_ohc=lnYqcLNTtLQAX8kMpyF&_nc_ht=scontent-sea1-1.xx&oh=00_AfDW5q4b2Po8QQgtD94Cyc18IkLCP-z5JYevagqZbriPkg&oe=6432ACEA https://scontent-sea1-1.xx.fbcdn.net/v/t39.2365-6/10000000_6... Announcement: https://ai.facebook.com/blog/segment-anything-foundation-model-image-segmentation/ https://ai.facebook.com/blog/segment-anything-foundation-mod... Code & Model Weights: https://github.com/facebookresearch/segment-anything https://github.com/facebookresearch/segment-anything

lofaszvanitt 4y ago
Why can't they give proper filenames to these research papers. This drives me nuts.
ftxbro 4y ago
if Tim Berners-Lee saw that paper link he would have never allowed the url to be invented
- LoganDark 4y ago
  I'm so shocked by how almost every query parameter is required and there's even a freaking signature for validating the URL itself. -Emily
  - CyberDildonics 4y ago
    Who is Emily?
    - revolvingocelot 4y ago
      Someone whose signature validates the grandparent comment itself, I assume
    - LoganDark 4y ago
      That would be me. We have DID and add a signature when the person commenting isn't actually Logan. It's not actually intended to be content. -Emily
      - CyberDildonics 4y ago
        Why not have two different user names? What if you get logged out and the other identity is the one that knows the password?
        LoganDark 4y ago
        > Why not have two different user names? We do on some platforms like Discord. > What if you get logged out and the other identity is the one that knows the password? Before we moved to a password manager we used the same password everywhere. Now, obviously, we have a password manager. -Emily
        CyberDildonics 4y ago
        Don't you run the risk of only one person knowing the password to the password manager?
        LoganDark 4y ago
        > Don't you run the risk of only one person knowing the password to the password manager? No, because systems don't necessarily work that way. For us, the boundaries between our members aren't totally uncrossable. Information has gotten through in the past when it would be especially important or needed. Though I guess it's funny that this topic comes up under a post called "segment anything". I guess our brain did that~ -Emily
  - TylerE 4y ago
    Signature is likely so the caching layer doesn't have to do all that validation, at least as an initial check.
    - egeozcan 4y ago
      If I'm remembering correctly, people were accessing the original versions of the images uploaded to instagram (removing filters, and even masks) using parameter engineering (We love putting engineering at the end of everything that requires more than 5 seconds of thought these days, so why not). That could be why.
jauer 4y ago
That paper link is a CDN URL that is dynamically generated to point to your closest POP when you load the abstract. It will be different for many people and will break eventually. Abstract: https://ai.facebook.com/research/publications/segment-anything/ https://ai.facebook.com/research/publications/segment-anythi...
Loranubi 4y ago
https://arxiv.org/abs/2304.02643 https://arxiv.org/abs/2304.02643

minimaxir 4y ago

You know an AI project is serious when it has its own domain name instead of a subdomain.

benatkin 4y ago

They seem to avoid using their own brand a lot. They have a zillion domain names and they register a new one and don't use the logo except in the favicon and footer. I've seen similar stuff including divesting OSS projects like PyTorch and GraphQL which Google wouldn't. To me that's tacit admission that the Facebook and Meta names are tarnished. And they are, by the content they showed users in Myanmar with the algorithmic feed, and by Cambridge Analytica. Maybe the whole "Meta" name is no different from the rebranding of Philip Morris.

smoldesu 4y ago
On the one hand, sure. Facebook's brand is about as hip as a bag of Werther's Originals. On the other hand, this is one of those things (like VR) that is a distinctly non-Facebook project. It makes no sense to position or market this as "Facebook" research. The Homepod isn't called the iPod Home for obvious reasons, so it stands to reason that Facebook execs realized selling someone a "Facebook Quest" sounds like a metaphor for ayahuasca. It's not entirely stupid to rebrand, especially considering how diverse (and undeniably advanced) they've become in fields like AI and VR.
- oefnak 4y ago
  I actively avoid everything that has anything to do with Facebook, and I can't be the only one.
  - smoldesu 4y ago
    Yeah, me too. I also avoid everything Apple and Google makes, but I'm not going to pretend like the Alphabet rebranding is their attempt at hiding who they are.
    - xiphias2 4y ago
      Alphabet wasn't a rebranding: the founding billionaires got bored of Google, and wanted to take out a few billion dollars per year out of it to create new toys without sharing it with Google.

ren_engineer 4y ago

what do you think facebook's gameplan is here? Are they trying to commoditize AI by releasing this and Llama as a move against OpenAI, Microsoft, and Google? They had to have known the Llama weights would be leaked and now they are releasing this

_the_inflator 4y ago
I think Meta's gameplan is complex. Inspiration as well as adoption, not stepping on the toes of regulators prolly another intention. Have a look at PyTorch for example. Massively popular ML framework, with its lots of interesting projects running. If Meta frequently shares their "algorithms" they take the blame out of its usage. After all, who is to blame when everybody does "it" and you are very open about it. Use cases, talent visibility as well as attraction also plays a role. After all, Google was so fancied, due to its many open source projects. "Show, don't tell".
jayd16 4y ago
Well there's some patent offense and defense in making and releasing research papers. There's some recruiting aspects to it. Its also a way to commoditize your inverse if you assume this sort of stuff brings AR and the metaverse closer to reach.
dragonwriter 4y ago
I think cranking out open source projects like this raises Meta AI’s profile and helps them attract attention and people, and I don’t think selling AI qua AI is their business plan, selling services built on top is. And commoditized AI means that the AI vendors don’t get to rent-seek on people doing that, whereas narrowly controlled monopoly/oligopoly AI would mean that the AI vendors extract the value produced by downstream applications.
- vagabund 4y ago
  I've always half-believed that the relatively open approach to industry research in ML was a result of the inherent compute-based barrier to entry for productizing a lot of the insights. Collaborating on improving the architectural SoTA gets the handful of well-capitalized incumbents further ahead more quickly, and solidifies their ML moat before new entrants can compete. Probably too cynical, but you can potentially view it as a weak form of collusion under the guise of open research.

dimatura 4y ago

The network architecture and scale don't seem to be a big departure from recent SOTA, but a pretty massive amount of labeled data went into it. And it seems to work pretty well! The browser demo is great. This will probably see a lot of use, especially considering the liberal licensing.

bjacobt 4y ago
I apologize if this is obvious, but are both the model and checkpoint (as referenced in getting started section in readme) Apache 2.0? Can it be used for commercial applications?
- dimatura 4y ago
  As far as I can tell, it can. The code itself has a `LICENSE` file with the Apache license, and the readme says "The model is licensed under the Apache 2.0 license.". Strangely, the FAQ in the blog post doesn't address this question, which I expect will be frequent.
  - phkahler 4y ago
    Isn't Apache 2 a free software license without some of the GPLv3 things some don't like? I think a more BSD would be better, or LGPL. Either would be more business friendly.
    - MacsHeadroom 4y ago
      LGPL is not business friendly at all. It's among the least business friendly licenses there is. Apache 2.0 is slightly more business friendly than BSD. With some caveats, software licenses from most to least business friendly roughly go: Apache > BSD > MIT > MPL > LGPL > GPL > AGPL
      - kyle-rb 4y ago
        LGPL is more business friendly than GPL; it's literally "lesser" GPL. You can use LGPL in commercial, closed-source projects as long as you keep the LGPL code in a separate dynamically linked library, e.g. a DLL, and provide a way for users to swap it out for their own patched DLL if they wish. (Plus some other license terms.) Also, you can always use LGPL code under the terms of the GPL, so there's no way LGPL is more restrictive than GPL.

cloudking 4y ago

Pretty cool, Runway has a similar green screening feature that can 1-click segment a subject from the background across an entire video: https://runwayml.com/ai-magic-tools/ https://runwayml.com/ai-magic-tools/

MrGando 4y ago

This is great, runs very fast in Chrome for me.

yeldarb 4y ago

Wow, this is pretty epic. I put it through its paces on a pretty wide variety of images that have tripped up recent zero-shot models[1] and am thoroughly impressed. We have a similar "Smart Polygon" tool[2] built into Roboflow but this is next level. Having the model running in the browser makes it so much more fun to use. Stoked it's open source; we're going to work on adding it to our annotation tool ASAP. [1] Some examples from Open Flamingo last week https://news.ycombinator.com/item?id=35348500 https://news.ycombinator.com/item?id=35348500 [2] https://blog.roboflow.com/automated-polygon-labeling-computer-vision/ https://blog.roboflow.com/automated-polygon-labeling-compute...

teruakohatu 4y ago
It would be nice if it could run entirely in the browser but the image embedding model is absolutely huge.
- blandcoffee 4y ago
  Can you expand further on what you mean by this?
  - teruakohatu 4y ago
    The SAM model is small (4m params) , but requires image embedding to be computed from what is I think a 600m params model. Right now the demo uploads the image to get the embeddings, then runs the actual segmentation locally.

neom 4y ago

yikes. I went to film school in the early 2000s and spent hours and hours on levels/HDR based masking, I've used the adobe tools recently and they're good... this is... yikes...I wonder how people in their mid 20s today learning photoshop are going to deal with their graduating jobs.

jacquesm 4y ago
Not. This is homing in on the SF UI that Deckard used in Blade Runner. https://www.youtube.com/watch?v=hHwjceFcF2Q https://www.youtube.com/watch?v=hHwjceFcF2Q All it takes is a couple of tools glued together and you're getting there.
- marstall 4y ago
  "gimme a hardcopy right there."
- zestyping 4y ago
  The most unrealistic part of this sequence is that the printer works on the first try.
ChatGTP 4y ago
Don’t you get it, no one is supposed to have a job? Silicon Valley is going to buy us all the things we need ?

arduinomancer 4y ago

This exists as a feature on iOS You can long press on an image and it cuts out whatever thing it thinks you're pressing on They also use it in interesting ways, like making stuff in the photo slightly overlap the clock on the lockscreen Does anyone know if that works the same way as this?

neom 4y ago
This is: understand everything in the image as elements, subject or whatever.
hbn 4y ago
It would still be nice if iOS had some kind of interface like this where you can nudge it in the right direction if it's confusing something like a jacket and the background. iOS gives its best attempt which is usually pretty good, but if it didn't get it right you're basically SOL.

dang 4y ago

Related: Meta New Segmentation Model - https://news.ycombinator.com/item?id=35453625 https://news.ycombinator.com/item?id=35453625 - April 2023 (7 comments)

swframe2 4y ago

The best solution I've seen is https://github.com/xuebinqin/DIS https://github.com/xuebinqin/DIS. You should try the DIS example images at the SAM site. The main issue I have with DIS is that creating the labels of my own dataset is super expensive (I think it might be easier to generate the training data using stable diffusion rather than human labelling)

pradyumnabang 4y ago
Forgive my ignorance, but why is creating the labels super expensive? How much does it cost per image and what's your dataset size ?
- swframe2 4y ago
  It is related to subpixel labelling. When a line/curve in the foreground is smaller than a pixel you end up having to edit the mask one pixel at a time. The authors of DIS are working on a new dataset and model which should work for my use case. BTW, I used DIS to create the labels of a batch of 20 images, I manually corrected the labels and used them to fine tune a new model. That worked well but still it took me several hours to edit labels. I tried using stable diffusion generated labels several weeks ago but I think with controlnet and other advances I should try again. (My dataset is about 100k images. I probably only need to label about 10k to fine tune DIS).

AdilZtn 4y ago

That's amazing! This model is a huge opportunity to create annotated data (with decent quality) for just a few dollars. People will iterate more quickly with this kind of foundation model.

justinator 4y ago

Demo is running slow - cutting out is an impressive ability - I'm to assume it also fills in the background? If so: that's next level. Maybe that Photoshop monthly subscription will be worth it (providing this sort of ability is going to be baked in with AdobeAI's version soon)

exodust 4y ago
It doesn't fill in the background and has nothing to do with Adobe. You could bring the cut-outs back into Photoshop. I tried that, but this SAM tool reduces the size of the cut-outs slightly, so the cut-out won't match original image dimensions.
- justinator 4y ago
  >It doesn't fill in the background Thanks - like I wrote, the demo was DOA for me. >and has nothing to do with Adobe. I know. Masking and selecting is something you do in Adobe products and Adobe will be coming out with their own version of this (if I were a betting man)

syntheweave 4y ago

Finally, I'll be able to fill line art with flat colors without fussing around with thresholds and painting in boundaries. (It does have difficulty finding the smallest possible area, but it's a significant advance over most existing options since in my brief test, it can usually spot the entire silhouette of figures, which is where painting a boundary is most tedious).

reaperman 4y ago

Multiple block diagrams and the paper note that one of the inputs is supposed to be "text", but none of the example Jupyter notebooks or the live demo page show how to use those. I'm assuming just run the text into CLIP, take the resulting embedding, and throw it directly in as a prompt, which then gets re-encoded by the SAM prompt encoder? > "Prompt encoder. We consider two sets of prompts: sparse (points, boxes, text) and dense (masks). We represent points and boxes by positional encodings [95] summed with learned embeddings for each prompt type and free-form text with an off-the-shelf text encoder from CLIP [82]. Dense prompts (i.e., masks) are embedded using convolutions and summed element-wise with the image embedding." Edit: Found the answer myself: https://github.com/facebookresearch/segment-anything/issues/4 https://github.com/facebookresearch/segment-anything/issues/...

deleted 4y ago
[deleted]

richardw 4y ago

Surely this changes the security camera game? No more being fooled by clouds going overhead.

bobsmooth 4y ago

Seeing it run on a headset is the coolest part. Lots of applications for AR.

sashank_1509 4y ago

Extremely impressive system. Blows everything else (including CLIP from OpenAI) out of the water. We are inching closer to solving Computer Vision!

wongarsu 4y ago
It's really impressive, and better than anything I've seen, but is it really leagues better than whatever Photoshop is using? Of course being on github and permissively license is huge.
- vanjajaja1 4y ago
  My question exactly, didn't photoshop already solve this like 5+ years ago?
  - dymk 4y ago
    Have you used Photoshop's magic wand tool in the last 5 years? No, it's nowhere close to this good.
    - Auracle 4y ago
      They have a newer “object selection” tool that’s similar.
    - Kaijo 4y ago
      The Magic Wand tool mostly just selects similarly colored pixels based on a simple algorithm. The Object Selection, Select Subject and Sky Replacement tools use AI detection and can be configured to run locally or on Adobe's GPUs. Having played with this demo, they seem in a similar league to me.
      - drummojg 4y ago
        I'm just happy that a tool that PropriAdobe has sat on and crowed about for years is soon to be a click away for the masses and the hackers.

code51 4y ago

It’s interesting that (clearly visible) text parts that cannot be handled properly by most OCR approaches also get left out by SAM in auto-predictions.

jonplackett 4y ago

This would make a great input for ControlNet

RomanPushkin 4y ago

Stalin's dream https://en.wikipedia.org/wiki/Censorship_of_images_in_the_Soviet_Union https://en.wikipedia.org/wiki/Censorship_of_images_in_the_So...

geenew 4y ago

That’s it for me

nielsbot 4y ago

Sigh. Does not work in Safari on macOS (ARM). Works in Chrome though.

subarctic 4y ago
It seems to work in firefox on macOS (ARM) fwiw

fzliu 4y ago

Computer vision seems to be gravitating heavily towards self-attention. While the results here are impressive, I'm not quite convinced that vision encoders are the right way forward. I just can't wrap my head around how discretizing images, which are continuous in two dimensions, into patches is the most optimal way to do visual recognition. What's preventing us from taking something like convnext or a hybrid conv/attention model and hooking that up to a decoder stack? I fee like the results would be similar if not better. EDIT: Clarifying that encoder/decoder refers to the transformer stack, not an autoencoder.

neodypsis 4y ago
> What's preventing us from taking something like convnext or a hybrid conv/attention model and hooking that up to a decoder? I fee like the results would be similar if not better. You mean like in an U-Net architecture?
skybrian 4y ago
Google seems to be doing it all with transformers. It's not open source, though: > Here we highlight some results of ViT-22B. Note that in the paper we also explore several other problem domains, like video classification, depth estimation, and semantic segmentation. https://ai.googleblog.com/2023/03/scaling-vision-transformers-to-22.html https://ai.googleblog.com/2023/03/scaling-vision-transformer...
intalentive 4y ago
IMO optimal visual recognition should be sensorimotor-based and video-first. In the real world, action and perception are intertwined. Supervised training on static pixel arrays seems backward and primitive.
sorenjan 4y ago
MLP-Mixer uses only multi-layer perceptrons. It was released in 2021, but ReBotNet that was released this year uses mixer layers too. Still uses patches though, but they're mixing data between patches.

subarctic 4y ago

The demo is pretty cool but it looks like you can just select things and have it highlight them in blue - is there a way to remove objects from the image and have the background filled in behind them?

dymk 4y ago
You could probably content-aware fill the area that SAM identifies with another tool
- exodust 4y ago
  Yeh I did that, but it's not a great experience. For some reason this tool makes a slightly smaller mask than is found in the original image. So when you copy the masked area back into Photoshop, it doesn't match. Almost there, but not quite.

syrusakbary 4y ago

This is awesome. If you try the demo they provide [0], the inference is handled purely in the client using a ONNX model that only weights around ~8Mb [1] [2]. Really impressive stuff! Congrats to the team that achieved it [0] https://segment-anything.com/demo https://segment-anything.com/demo [1] https://segment-anything.com/model/interactive_module_quantized_592547_2023_03_19_sam6_long_uncertain.onnx https://segment-anything.com/model/interactive_module_quanti... [2] https://segment-anything.com/model/interactive_module_quantized_592547_2023_03_20_sam6_long_all_masks_extra_data_with_ious.onnx https://segment-anything.com/model/interactive_module_quanti...

kevmo314 4y ago
It isn't purely client-side. The embeddings are generated on the server so your image is still sent to Meta for processing. https://github.com/facebookresearch/segment-anything/issues/16 https://github.com/facebookresearch/segment-anything/issues/...
- trzy 4y ago
  The linked issue says nothing of the sort. Are you sure?
  - GaggiX 4y ago
    SAM uses a CLIP-H as an image encoder, a very large encoder that obviously does not fit in 8 MB. Also read the "Efficient & flexible model design" section on the page. Also in the FAQ: "How big is the model? The image encoder has 632M parameters. The prompt encoder and mask decoder have 4M parameters."
    - GaggiX 4y ago
      I cannot edit the comment anymore but I should have written MAE ViT-H and not CLIP-H (same size but CLIP models are trained in a different way).
      - 4y ago

georgelyon 4y ago

It seems like the output of this model is masks, but for cropping you really need to be able to pull partial color out of certain pixels (for example, pulling a translucent object out from of a colored background). I tried the demo, and it fails pretty miserably on a vase. Anyone know of a model that can do this well?

teruakohatu 4y ago
This is a segmentation model, it 'just' creates segmentation masks. You then take the mask and cut out whatever you need from the image itself. If you need to figure out how to remove a transparentency that is another whole problem, probably a generative problem (generate a new texture for the vase).

dekhn 4y ago

I have a small number of naive questions. I already have a fine-tuned tardigrade detection model that gives me tardigrade bounding boxes (data comes from labelled images on my microscope). I want to do masks as well. Right now my home server w/ an RTX 2080 is able to do mask prediction in about ~4 seconds (I'm running the sample script in "directory" mode) per image (640x480). I'd love to be able to get the first mask back in 0.1ms, so I can do 10FPS on the scope. Is there a practical way to speed things up (my guess would be buying the absolute fastest GPU I can afford)? I can run the object detection to get a reasonable seed location, is that what the paper means by prompting?

bnqscrtm 4y ago
A 4s run time for object segmentation at 640x480 sounds like it's not using the GPU at all. Something like that should run on a VGA image in at most a few hundred ms. For the second part of the question, a 2080 should get you close to 10FPS operation. For a ballpark estimate, using an off-the-shelf repo like Ultralytics's YOLOv5 lets you run object detection (not masking) at something like 100FPS. Masking should not add that much overhead. w.r.t. GPUs yes, these days more money equals more speed for GPU NN inference, though there are diminishing returns. A 3090 might get you the best bang for your buck these days while still having enough VRAM to run fancier models which may need more than the 12 GiB many other GPUs have. Finally, I haven't read the paper too carefully but I believe that by prompting they mean that you have the option of describing in human language what you want the model to select, rather than the model being "hardwired" to do this. In other words, you could prompt the model to "segment the red car only" and it would do it, rather than just having the model blindly segment every object in the image, and then relying on custom scripting to potentially post-process these segments.
- GC_tris 4y ago
  Yeah, a 3090 should do well. If you want to try it on one reach out to me (email in profile). We rent those out in the cloud. Would allow you to confirm performance before buying one for local use.
- dekhn 4y ago

rhacker 4y ago

Is it just me or is it really really really slow?

ericmathison 4y ago

This reminds me of https://cleanup.pictures/ https://cleanup.pictures/ which actually removes objects from the original.

bfung 4y ago

Def. cool demo. As a polished product & ease of use, still has ways to go. Apple’s cut&paste of subjects in photos, while lacking SAM’s generality, is masterfully executed - even my 70yr old mom can use it.

retox 4y ago

The TV/VHS example image does not do well.

an_aparallel 4y ago

Ahh - finally - we can now hyperlink real life. Perfect. HTRL (Hypertext Reality Language) <human></human> <ad></ad>

jacooper 4y ago

> The model is licensed under the Apache 2.0 license. Finally, an actually open model

novaRom 4y ago
and it will work on any smartphone because: The image encoder has 632M parameters. The prompt encoder and mask decoder have 4M parameters.

tgv 4y ago

Impressive, but not really perfect, is it? In sa_10016721, it misses the obvious, and in sa_10020386, it misses most of the train in the center, and a bunch of the parked cars (pretty random). In sa_10179757, it labels 3 out of 4 letters of the shipping company's name (?), and a handful of windows, and while perhaps it sees the ship as one piece, the people in the foreground are split in many parts.

fn-mote 4y ago

Kind of off topic, but: I've never seen such crappy issues filed in a repo [1]. I don't read issues for major repositories, so perhaps this is standard? There are a ton of one line "issues", no clear example, test case, attempt to debug, not even a pull request for the one that points out a typo in the README. Gross. It seems like none of these issues will never be read because they are going to drown in garbage. [1] https://github.com/facebookresearch/segment-anything/issues https://github.com/facebookresearch/segment-anything/issues

MF-DOOM 4y ago
Since anyone can open an issue, they’re bound to be low quality unless moderated.
- kristopolous 4y ago
  some trackers handle this by asking pointed questions. "Steps to reproduce" "platforms affected" "expected behavior" etc. Microsoft knows this. Get on the ball, guys.

contingencies 4y ago

See also http://remove.bg http://remove.bg

vietanhdev 4y ago

[dead]