11 ms·

Building an Internet Scale Meme Search Engine

785 points by whoisburbansky 4y ago

komali2 4y ago
Would love to see that load balancer implementation, as I'm a scrub and this project fascinates me.
- ksdme9 4y ago
  nginx makes it really simple to setup a load balancer, it defaults to distributing requests equally between all upstream hosts but you can always assign weights to each of your server. https://docs.nginx.com/nginx/admin-guide/load-balancer/http-load-balancer/ https://docs.nginx.com/nginx/admin-guide/load-balancer/http-...
  - mandatory 4y ago
    Yep, this is exactly what I'm running on the raspberry pi LB. Nginx makes it super easy!
memeatlas 4y ago
Although not as elegant a solution as this I've also tried my hand as well at indexing and categorizing memes. I wanted to save a very specific type of meme though since there are, in my mind, 2 main categories of memes. The first category are what I call "story" memes, they are standalone and typically what you see being shared on Facebook. They usually have text and are able to tell a story on their own with no additional context and can be presented as a single post, story, etc, (think 4 panel comics). The second type are reaction memes. These are used to respond to people and usually convey a feeling towards a post or tweet. They can also be standalone so they should probably be considered a subset of the "story" memes. I've gravitated towards the reaction memes as I see more utility in them and can be used in a more universal way. My site if anyone is interested (its still a work in progress): https://www.memeatlas.com https://www.memeatlas.com
- suave_dude 4y ago
  What are you using as the back end to host your website?
- operator-name 4y ago
  These different approaches really compliment each other - most of the memes you've categorised are used in a variety of situations and therefore not suited to text searching. Meanwhile if you're looking for a specific meme that you've seen, text search is the way to go. Ideally there would be a best of both worlds where you could search memes by "characters" or "formats" in addition to text. As feedback, it would be nice to search all memes from the homepage. The search on https://www.memeatlas.com/meme-templates.html https://www.memeatlas.com/meme-templates.html also seems to be broken.
julianeon 4y ago
This makes me wonder what other cool things I can do with an old iPhone.
andai 4y ago
I have a "hackish but works for me" meme database: I use my Telegram "self chat" to send memes I like to myself, and I tag them with the kind of words I'm likely to search for when looking for them later. Works great for me. It's kind of like trying to come up with a good Google search phrase, based on how other people must have phrased something, but relying on knowledge of how you phrase things instead.
- solarkraft 4y ago
  I do this for some, but most of the time my use case is to look for something very obscure from a long time ago that I didn't regard as interesting when I first saw it (so didn't make the effort to manually categorize it).
Fabricio20 4y ago
Does the Vision API call back to apple servers in any fashion? Like how google on-device voice recognition APIs will call back to Google when you are online (unless you explicitly pass flags to force it in offline mode). If so, is there any risk in getting your account suspended or ip range banned somehow because of this, for example?
- mandatory 4y ago
  Nope, you can use it totally offline. No way of getting banned as far as I'm aware.
suave_dude 4y ago
I have a question what do you guys think is the best back end for a video search engine app?
nowahe 4y ago
Out of curiosity, how does your Image Similarity Search works ? Are you also using some feature of Apple's Vision framework, or running some ML model on your linode instance ?
- mandatory 4y ago
  The image similarity search is probably a blog post of its own. Short TL;DR: It runs off my home server running a large vector database (opendistro): https://opendistro.github.io/for-elasticsearch-docs/docs/knn/approximate-knn/ https://opendistro.github.io/for-elasticsearch-docs/docs/knn...
solarkraft 4y ago
That's a fun way to do OCR. Next up: Classifying memes by subjects and themes to build something like KnowYourMeme's gallery, but for every meme. Bonus: Index from a lot of sources to help track a meme's origin. This type of thing is on my long list of "can somebody else please do this already".
the_arun 4y ago
Nice hack. But doesn't Google/(any public search engine) image search do this for us already?
looki 4y ago
Very cool project! I'll try to remember it the next time I'm looking for a specific image. I noticed that repeated appearances of the search term are ranked higher, which isn't necessarily productive. Also, some kind of duplicate detection would be nice. Searching for "SpongeBob" yields many copies of the same images that mentions "SpongeBob" several times.
danbruder 4y ago
this is very clever. I wonder what other use cases could leverage this approach
joshu 4y ago
Heh, this finds a bunch of copies of a video I made. If you are going to cache them and repost them, you probably need to have a DMCA process.
permo-w 4y ago
I have absolutely no experience in this area and I'm curious: is there really no open-source text recognition software that's on-par with or close to Apple's (presumably proprietary) implementation? the article mentions Tesseract. is that the current best open-source option?
- satvikchoudhary 4y ago
  Certainly not. https://paperswithcode.com/task/optical-character-recognition https://paperswithcode.com/task/optical-character-recognitio...
paulmd 4y ago
Is there a pgsync equivalent for Oracle? Spent some time building replication from a source-of-truth to a search engine at a previous job. Wish we could have used postgres but the tools were dictated rather than letting the requirements drive the tooling.
ariehkovler 4y ago
This is mad and I love it.
petercooper 4y ago
You can do it on macOS as well, it has the same API for fast high quality OCR. I used it to create an OCR system to detect secrets or credentials in screencasts: https://github.com/peterc/videocr https://github.com/peterc/videocr
9dev 4y ago
This is the single bestest thing I have read in a long while. Absolute madness. Pure bliss.
Scaevolus 4y ago
Looks like you've solved the OCR problem, now to solve the duplication problem and use it as a ranking hint. :-)
schappim 4y ago
Pretty insane. If you don’t want to use iPhones, I made macOCR a while back. It uses the same vision APIs, with a very simple CLI interface. See: https://github.com/schappim/macOCR https://github.com/schappim/macOCR
MrGilbert 4y ago
This is remarkable. I'd love to see that combined with some kind of sentiment analysis like Microsoft offers, just to see if something useful comes out of it. Sometimes, I don’t know the exact words when looking for a meme, but once I see it, I know that’s the one.
- astrange 4y ago
  Unfortunately semantic analysis barely works at the best of times, but it especially doesn't work here. Computers… they're just not good at irony. IME CLIP embedding search can work strangely on memes as well, because it gets confused when images have words in them. Basically the same problem reported in the original CLIP paper where it thinks an apple and a piece of paper with "apple" written on it are the same thing.
- Cloudly 4y ago
  Semantic search using CLIP embeddings / other LLM embeddings could be an amazing addition too.
CobrastanJorji 4y ago
This title really undersells the absolute insanity of the described solution. This is a beautiful example of "if it's stupid, but it works, it's not stupid." The justification is very convincing. One thing I'm curious about: how did you build your corpus of meme images and videos?
- aemreunal 4y ago
  (Not the author) Maybe they leveraged https://knowyourmeme.com/ https://knowyourmeme.com/? But that can't possibly have all the random memes, could it?
  - code_duck 4y ago
    OP’s meme site lists where each image comes from. Looking through it I mainly see ifunny and 9gag.
  - solarkraft 4y ago
    Do you crawl telegram channels?
  - mandatory 4y ago
    Author here: KnowYourMeme is one of many sites that memes are continually ingested from (any site that has memes I try to ingest regularly) :)
    - counttheforks 4y ago
      What about copyright?
      - deleted 4y ago
        [deleted]
    - spiffytech 4y ago
      How do you ingest your social circle's in-group memes? Are they reliably posted to meme generator sites?
transitivebs 4y ago
My #1 recommendation for anyone thinking about the convoluted OCR solution: use a cheap OCR API and save yourself months of time / hassle / upkeep. Google's OCR API is a good place to start, but AWS has one too and dozens of others out there.
- mike_d 4y ago
  Without this "convoluted OCR solution" it never would have been built. Mandatory would have easily had to spend hundreds of thousands of dollars to OCR his meme collection alone, even without scraping other meme sites.
- elorant 4y ago
  If you have to process tens of millions of photos though the cost gradually becomes forbidding.
- langsoul-com 4y ago
  The author calculated the cost of 1 iPhone se was sub $50, which is 27k Google ocr images. Only makes sense for small scale. Unfortunate how there's no decent ocr library to self host, would be cheoaer than cloud costs.
- papito 4y ago
  On the contrary, this sort of creative thinking is what's needed instead of automatically reaching for that shiny Cloud Toy. It's easy to get a proof-of-concept working, sure, but at scale, you start torching through cash. Many places keep adding cloud services to their stacks until one day someone in the C-suite notices the AWS bill.
yakubin 4y ago
The photo under Upgrading the iPhone OCR Service Into An OCR Cluster. In the future, data centres are going to host racks of iPhones.
- sankha93 4y ago
  It is already here to be honest. I know BrowserStack and other mobile testing platforms (at Facebook and Amazon) do host real devices, both Android and iPhones, in server farms like this. Meta wrote a blog post about it: https://engineering.fb.com/2016/07/13/android/the-mobile-device-lab-at-the-prineville-data-center/ https://engineering.fb.com/2016/07/13/android/the-mobile-dev... At one of my previous workplaces, we discussed running the Z3 theorem prover on an iPhone cluster, because they run so much faster on A series processor than a desktop Intel machine.
- oefrha 4y ago
  Modern app click farms already have walls of iPhones.
- formerly_proven 4y ago
  Reminds me of imgix, who built their product on Apple libraries so ended up having racks of macs to run their service.
- leokennis 4y ago
  To be fair...insane performance in a tiny cool package...iPhones would make great servers if you could order them without the screen/camera etc. :-)
  - defrost 4y ago
    There's your startup right there - washing line racks of discarded iPhones, near bleeding edge, busted screens, still functional o/wise. Low entry cost, recycling, eco-friendly, . . . a deck that writes itself.
ksdme9 4y ago
Love the inventiveness. My question is about the image distribution costs. All the memes on the site seem to be coming straight off an object storage, all that bandwidth consumption has got to add up(?). Some sort of a CDN might help depending on the search patterns.
ekns 4y ago
Last time I looked into OCR stuff I came to a similar conclusion (though I didn't implement anything back thne). It would be really nice to have "open source" models that had similar performance, without having to deal with the iphone cluster hackery.
j0li0t 4y ago
[flagged]
JustARandomGuy 4y ago
Very inventive. Admittedly when I read the first few paragraphs, I was thinking “he’s got to have $40K of iPhones doing image processing” but you made a good point about being able to use iPhones with screen and other damage. What was your average price per iPhone, if you don’t mind disclosing?
Freak_NL 4y ago
> Better yet, I don’t even want to use them as phones, so even iPhones that are IMEI banned or are locked to unpopular networks are perfectly fine for my use. Fences worldwide will be overjoyed to hear of this novel application.
fuzzygroup 4y ago
This is utterly fantastic and you are to be commended for your Crazy Mad Scientist genius!
merpkz 4y ago
Nice project, I wanted to build meme search engine myself one day, but figured Tesseract will fail at most of the memes because of how stylized those have become. So I tuned down my meme source to only /r/bertstrips as those contain sane looking text and it's working quite alright - project has no frontend yet, I search from cli and click links. > Initial testing with the Postgres Full Text Search indexing functionality proved unusably slow at the scale of anything over a million images, even when allocated the appropriate hardware resources. I can guarantee you that correctly setup PostgreSQL text search will be faster than ES with much, much less hardware resources needed, it's just a matter of correctly creating tsvector column and creating GIN index on it (and ofc asking right queries so it's actually used). I can help you out setting postgres schema up and debugging queries if you are interested, for testing purposes at least.
- tmzt 4y ago
  I recently worked on a project using lnx.rs. Simple to setup and use and fast at the scale I was using it. Built on Tantivy with a custom fast fuzzy search feature. If you want to go beyond meme sites and possibly detect memes in the wild, common crawl might be something to start with.
- iamflimflam1 4y ago
  One issue I've had with postgres full text search is when you want to rank using ts_rank you end up with a full table scan.
csande17 4y ago
I wonder how the performance of Vision.framework on desktop Mac hardware compares to a cluster of phones. (The author mentions that it was "fairly slow", but it sounds like they were running an iOS app in the simulator and not a macOS app.)
- papito 4y ago
  Same thoughts. You can make that fly on a Mac Mini (provided it can be made to work close to the metal and not in an emulator)
- lathiat 4y ago
  I would expect an M1 native Mac app to work similarly well. Though the iPhone solution may win on price. The framework is supported on macOS (even tvOS apparently) https://developer.apple.com/documentation/vision https://developer.apple.com/documentation/vision
Thorentis 4y ago
IaaS - iPhone as a Service, coming soon to AWS.
oefrha 4y ago
> My preliminary speed tests were fairly slow on my Macbook. However, once I deployed the app to an actual iPhone the speed of OCR was extremely promising (possibly due to the Vision framework using the GPU). I was then able to perform extremely accurate OCR on thousands of images in no time at all, even on the budget iPhone models like the 2nd gen SE. I suppose that’s an old Intel MacBook? I’d be very surprised if the Vision framework performs better on a 2nd gen iPhone SE than even the first M1 MacBook Air.
- iamflimflam1 4y ago
  I think he was running it in the simulator - which won't perform anywhere near as fast on this type of thing.
  - oefrha 4y ago
    The simulator runs a native build though. Might not be optimized, not sure.
dirtyid 4y ago
Outrageous effort! So far japanese, mandarin returning results as well. Do you have list of sources where memes are ingested from? Would be nice to have some option to explore memes by category.
mateuszbuda 4y ago
Great project! With your DIY attitude, if you need to build your own infrastructure for web scraping, here is a tutorial for mobile proxy setup which might be helpful: https://scrapingfish.com/blog/byo-mobile-proxy-for-web-scraping https://scrapingfish.com/blog/byo-mobile-proxy-for-web-scrap...
lysecret 4y ago
Wait what this is absolutely brilliant. Actually insane it works so well using a stack of iPhones as an ocr server. My deepest respect.
philsnow 4y ago
I sat down literally last night and started sketching out the scratch-my-own-itch solution to more or less exactly this problem, because I too have meme-aphasia where I know there exists a meme that fits perfectly in a conversation, but I have about 5 seconds to find it before the moment passes. I'm so, so glad to see that I'm not the only person in the world with the same "problem". Well done, mandatory. edit: holy crap you even index videos, nice
baradhiren07 4y ago
https://findthatmeme.com/?search=take+my+money https://findthatmeme.com/?search=take+my+money
ipsum2 4y ago
This is amazing. Out of curiosity, why not try deep learning OCR software instead of Tesseract? PaddleOCR is popular.
- ducktective 4y ago
  >PaddleOCR I once tried installing it on a recent Ubuntu. After messing with dependency hell and pip downloading half of the internet, when I finally invoked the CLI, it complained _again_ about a missing runtime dependency. I called it quits. DL people are simply not interested in bundled static binaries.
kome 4y ago
you are a genius. also, the Search Engine works so well
1wsk 4y ago
Could you not extract the model and run it on a server? Its probably not as easy but i know it has been done with NeuralHash
kevmo314 4y ago
Could you expose your iPhone cluster as an OCR API? Seems like it would be competitive with the GCP API.
- urbandw311er 4y ago
  I wonder if there might be something in the Apple TOS prohibiting this.
  - OJFord 4y ago
    I think there is right? At least assuming it's the same as Macs where the minimum billing period is 1 day, because you're technically renting a physical Mac (and 1d is the minimum that qualifies as adhering to that).
    - urbandw311er 4y ago
      I was more referring to bundling up an OS framework library feature and exposing it as an Internet API.
Arbortheus 4y ago
This is great, I particularly like the part about using compute from old unwanted iPhones. Quite an inventive way to reuse/recycle otherwise obsolete hardware!
lormayna 4y ago
If you don't need advanced search features, you can use Sonic (https://github.com/valeriansaliou/sonic https://github.com/valeriansaliou/sonic). It's blazing fast and you can save lot of money on servers.
- remram 4y ago
  Someone mentioned lnx.rs in this thread, how does it compare? And how about all the other new Rust solutions like MeiliSearch, Toshi, Quickwit?
  - lormayna 4y ago
    I don't know, I am not an expert on this topic. But this could be a nice topic for a blog article that will hit the frontpage on HN :)
petesergeant 4y ago
I was hoping this would help me find the Database Iceberg meme that shows different levels of database insanity. It didn’t. Anyone have a link?
- cnity 4y ago
  This one? https://www.reddit.com/r/ProgrammerHumor/comments/wuu689/the_sql_iceberg/ https://www.reddit.com/r/ProgrammerHumor/comments/wuu689/the...
  - petesergeant 4y ago
    Yes, that’s it, thankyou. Been searching a few days for it
    - cnity 4y ago
      You're welcome. FWIW I literally put your wording into google image search to find it: > Database Iceberg meme that shows different levels of database insanity But possibly my google search results are tailored differently from yours.
spuz 4y ago
I'm curious how well the iPhone OCR actually works. How do you deal with errors? Is the error rate low enough that you can accept the output from the iPhone OCR as is or do you also run it through a cleaning process (e.g. spell check)?
- 9dev 4y ago
  For a single data point, it works exceptionally well for me. I routinely copy-paste from images or screenshots, and it rarely fails (mostly for handwriting or obscure fonts).
  - leokennis 4y ago
    I am not sure if the Photos app search also uses the same OCR. But sometimes I search for a word and iOS will find a photo that has that text in a 50x20 cluster of pixels somewhere far in the background...it's remarkably good.
marginalia_nu 4y ago
Now this is the sort of disgusting pile of jank I love to see.
tomw1808 4y ago
Absolutely amazing on the tech side!!! Now, after reading the article, I gave your search engine a try. I was looking for that futurama its a trap meme (pretty much pops up on any image search here https://www.google.com/search?q=futurama+its+a+trap https://www.google.com/search?q=futurama+its+a+trap) The problem is, the search engine you built is now very text-heavy, which seems to be usually very unconnected to the actual meme. So, searching for "its a trap" did not yield the results I was actually hoping for, but made total sense looking at how the search was implemented. Are you planning to implement an actual tagging of the content of some sorts? Maybe a clustering of similar objects (like iphone clusters similar peoples faces in the gallery) and then tag those clusters with keywords somehow?
- mandatory 4y ago
  Yes I definitely want to improve the search to be better. It is currently very text heavy and I (only recently) got image similarity indexing working. Hoping to leverage this to do something like you mentioned! I'd also like to figure out how to turn an image into a description of whats in it. My ML/tensorflow knowledge is very weak though, so I still have a lot to learn here.
yreg 4y ago
This is absolutely brilliant. I believe I will actually use it a lot if you keep the site up. Minor feedback for the blog post: It deserves a better meta description (for link previews). The first paragraph doesn't advertise how good the article is going to be.
sneak 4y ago
Don't you have to re-sign and re-deploy tour iOS app every 7 days to keep it running on the iPhones?
Liquidor 4y ago
Brilliant! :-) Maybe a dumb question, but could you use your data to train a new OCR model so you wouldn't have to rely on iOS? I don't know much about ML/AI so maybe not feasible.
nysv 4y ago
If only there was a way to filter out ifunny results, I absolutely detest that watermark.
francis-io 4y ago
Would be great if the images had a unique name so I could save them without having to rename them.
mseidl 4y ago
Do you use mongodb to make it web scale? You turn it on and it scales right up.
- zffr 4y ago
  https://www.youtube.com/watch?v=b2F-DItXtZs https://www.youtube.com/watch?v=b2F-DItXtZs
Tade0 4y ago
I wonder how it fares against deep-fried "E" or Opossum memes?
geek_at 4y ago
really amazing! I love the solution and the project in general
aabajian 4y ago
I had a friend in med school who wrote a very early note-taking app for the iPad. Turns out that there was no way to render PowerPoint files when the iPad first came out. He realized that the iOS/Mac OS "quick preview" function could be used to take screenshots of each PowerPoint slide. For a brief time, his was the only app that could display PowerPoints (albeit, they were just screenshots!). There's a lot of hidden utility in Apple libraries.
surume 4y ago
Thank you for building this! It's so much fun. I looked for memes that I saw years ago and found them in seconds. Excellent work!!
nisegami 4y ago
Is this the person on /r/hardwareswap who's been looking for semi-functional, even IMEI banned, iPhones?
2Gkashmiri 4y ago
Are you going to open source the "app" part of it ? I would love to replicate this setup for my own project.... I am thinking, load balanced, multi location redundant "iOS machines" with 3-4 phones in with power backup and internet dongle. We could use something like zerotier/tailscale to get internet access from outside your local network
CiceroCiceronis 4y ago
This is really brilliant to see, and I've been surprised for quite a long time that nothing similar exists. I think it's a real shame that few people with interest in memes have interest in building solutions like this that help us engage with them. People in the 21st century know a lot about the mistakes of the past century that led to much popular culture of the time being lost (especially terminally online people who've watched lots of Youtube documentaries about lost Dr. Who episodes and so on), so it surprises me how little we try and avoid those same mistakes with today's ephemeral pop culture in the form of memes. People like yourself who want to help make the internet's huge corpus of memes tractable are part of the solution in terms of meme archival and cultural memory. (There's a good meme metadiscussion group on Discord, "The Philosopher's Meme," which you might be interested in joining. People there would be very keen to discuss what you've made.)
- spiffytech 4y ago
  I've always been surprised that Reddit hasn't built meme search into its site search. Memes are a core part of the Reddit experience, yet it's difficult to find something I know I saw before.
- jychang 4y ago
  Not familiar with Discord, do you have a link to that group?
  - CiceroCiceronis 4y ago
    https://discord.com/invite/8MVFRMa https://discord.com/invite/8MVFRMa
gerdesj 4y ago
"It looked like it was time to bite the bullet and write an OCR iOS server in Swift." Quite a large bullet required, one with plenty of chewing left in it.
kgbcia 4y ago
That's genius. I realized the cost advantage of text to speech on an old android versus Google cloud
causality0 4y ago
I tried a few memes. The results were quite poor, and vastly inferior to just using Google. In the case of text searches I had to scroll through dozens of results before finding the original meme images.
iamflimflam1 4y ago
Love the hackiness of this - however, the vision framework is available on Desktop macs as well - https://developer.apple.com/documentation/vision https://developer.apple.com/documentation/vision and specifically: https://developer.apple.com/documentation/vision/vnrecognizetextrequest https://developer.apple.com/documentation/vision/vnrecognize...
- price456987 4y ago
  What's the cost of building and running a cluster of iPhones vs Mac Minis?
  - Bad_CRC 4y ago
    in the article $40 second hand, imei banned and broken screen iPhones are being used so...
    - vlunkr 4y ago
      There's a ton of compute power available in the form of unused phones.
- pronoiac 4y ago
  There's ocrit, a CLI utility using Apple's Vision framework for OCR: https://github.com/insidegui/ocrit https://github.com/insidegui/ocrit
- cassiogo 4y ago
  > My preliminary speed tests were fairly slow on my Macbook. However, once I deployed the app to an actual iPhone the speed of OCR was extremely promising (possibly due to the Vision framework using the GPU). I was then able to perform extremely accurate OCR on thousands of images in no time at all, even on the budget iPhone models like the 2nd gen SE. He does mention running it on a macbook
  - dblitt 4y ago
    I would assume he's using an intel macbook and wouldn't have the gpu acceleration (and subsequent Vision framework integration) of the m1
skizm 4y ago
Where did the original set of meme images, gifs, and videos come from?
coayer 4y ago
I love the iPhone cluster so much!

Would love to see that load balancer implementation, as I'm a scrub and this project fascinates me.

ksdme9 4y ago
nginx makes it really simple to setup a load balancer, it defaults to distributing requests equally between all upstream hosts but you can always assign weights to each of your server. https://docs.nginx.com/nginx/admin-guide/load-balancer/http-load-balancer/ https://docs.nginx.com/nginx/admin-guide/load-balancer/http-...
- mandatory 4y ago
  Yep, this is exactly what I'm running on the raspberry pi LB. Nginx makes it super easy!

Although not as elegant a solution as this I've also tried my hand as well at indexing and categorizing memes. I wanted to save a very specific type of meme though since there are, in my mind, 2 main categories of memes. The first category are what I call "story" memes, they are standalone and typically what you see being shared on Facebook. They usually have text and are able to tell a story on their own with no additional context and can be presented as a single post, story, etc, (think 4 panel comics). The second type are reaction memes. These are used to respond to people and usually convey a feeling towards a post or tweet. They can also be standalone so they should probably be considered a subset of the "story" memes. I've gravitated towards the reaction memes as I see more utility in them and can be used in a more universal way. My site if anyone is interested (its still a work in progress): https://www.memeatlas.com https://www.memeatlas.com

suave_dude 4y ago
What are you using as the back end to host your website?
operator-name 4y ago
These different approaches really compliment each other - most of the memes you've categorised are used in a variety of situations and therefore not suited to text searching. Meanwhile if you're looking for a specific meme that you've seen, text search is the way to go. Ideally there would be a best of both worlds where you could search memes by "characters" or "formats" in addition to text. As feedback, it would be nice to search all memes from the homepage. The search on https://www.memeatlas.com/meme-templates.html https://www.memeatlas.com/meme-templates.html also seems to be broken.

julianeon 4y ago

This makes me wonder what other cool things I can do with an old iPhone.

andai 4y ago

I have a "hackish but works for me" meme database: I use my Telegram "self chat" to send memes I like to myself, and I tag them with the kind of words I'm likely to search for when looking for them later. Works great for me. It's kind of like trying to come up with a good Google search phrase, based on how other people must have phrased something, but relying on knowledge of how you phrase things instead.

solarkraft 4y ago
I do this for some, but most of the time my use case is to look for something very obscure from a long time ago that I didn't regard as interesting when I first saw it (so didn't make the effort to manually categorize it).

Fabricio20 4y ago

Does the Vision API call back to apple servers in any fashion? Like how google on-device voice recognition APIs will call back to Google when you are online (unless you explicitly pass flags to force it in offline mode). If so, is there any risk in getting your account suspended or ip range banned somehow because of this, for example?

mandatory 4y ago
Nope, you can use it totally offline. No way of getting banned as far as I'm aware.

suave_dude 4y ago

I have a question what do you guys think is the best back end for a video search engine app?

nowahe 4y ago

Out of curiosity, how does your Image Similarity Search works ? Are you also using some feature of Apple's Vision framework, or running some ML model on your linode instance ?

mandatory 4y ago
The image similarity search is probably a blog post of its own. Short TL;DR: It runs off my home server running a large vector database (opendistro): https://opendistro.github.io/for-elasticsearch-docs/docs/knn/approximate-knn/ https://opendistro.github.io/for-elasticsearch-docs/docs/knn...

solarkraft 4y ago

That's a fun way to do OCR. Next up: Classifying memes by subjects and themes to build something like KnowYourMeme's gallery, but for every meme. Bonus: Index from a lot of sources to help track a meme's origin. This type of thing is on my long list of "can somebody else please do this already".

the_arun 4y ago

Nice hack. But doesn't Google/(any public search engine) image search do this for us already?

looki 4y ago

Very cool project! I'll try to remember it the next time I'm looking for a specific image. I noticed that repeated appearances of the search term are ranked higher, which isn't necessarily productive. Also, some kind of duplicate detection would be nice. Searching for "SpongeBob" yields many copies of the same images that mentions "SpongeBob" several times.

danbruder 4y ago

this is very clever. I wonder what other use cases could leverage this approach

joshu 4y ago

Heh, this finds a bunch of copies of a video I made. If you are going to cache them and repost them, you probably need to have a DMCA process.

permo-w 4y ago

I have absolutely no experience in this area and I'm curious: is there really no open-source text recognition software that's on-par with or close to Apple's (presumably proprietary) implementation? the article mentions Tesseract. is that the current best open-source option?

satvikchoudhary 4y ago
Certainly not. https://paperswithcode.com/task/optical-character-recognition https://paperswithcode.com/task/optical-character-recognitio...

paulmd 4y ago

Is there a pgsync equivalent for Oracle? Spent some time building replication from a source-of-truth to a search engine at a previous job. Wish we could have used postgres but the tools were dictated rather than letting the requirements drive the tooling.

ariehkovler 4y ago

This is mad and I love it.

petercooper 4y ago

You can do it on macOS as well, it has the same API for fast high quality OCR. I used it to create an OCR system to detect secrets or credentials in screencasts: https://github.com/peterc/videocr https://github.com/peterc/videocr

9dev 4y ago

This is the single bestest thing I have read in a long while. Absolute madness. Pure bliss.

Scaevolus 4y ago

Looks like you've solved the OCR problem, now to solve the duplication problem and use it as a ranking hint. :-)

schappim 4y ago

Pretty insane. If you don’t want to use iPhones, I made macOCR a while back. It uses the same vision APIs, with a very simple CLI interface. See: https://github.com/schappim/macOCR https://github.com/schappim/macOCR

MrGilbert 4y ago

This is remarkable. I'd love to see that combined with some kind of sentiment analysis like Microsoft offers, just to see if something useful comes out of it. Sometimes, I don’t know the exact words when looking for a meme, but once I see it, I know that’s the one.

astrange 4y ago
Unfortunately semantic analysis barely works at the best of times, but it especially doesn't work here. Computers… they're just not good at irony. IME CLIP embedding search can work strangely on memes as well, because it gets confused when images have words in them. Basically the same problem reported in the original CLIP paper where it thinks an apple and a piece of paper with "apple" written on it are the same thing.
Cloudly 4y ago
Semantic search using CLIP embeddings / other LLM embeddings could be an amazing addition too.

CobrastanJorji 4y ago

This title really undersells the absolute insanity of the described solution. This is a beautiful example of "if it's stupid, but it works, it's not stupid." The justification is very convincing. One thing I'm curious about: how did you build your corpus of meme images and videos?

aemreunal 4y ago
(Not the author) Maybe they leveraged https://knowyourmeme.com/ https://knowyourmeme.com/? But that can't possibly have all the random memes, could it?
- code_duck 4y ago
  OP’s meme site lists where each image comes from. Looking through it I mainly see ifunny and 9gag.
- solarkraft 4y ago
  Do you crawl telegram channels?
- mandatory 4y ago
  Author here: KnowYourMeme is one of many sites that memes are continually ingested from (any site that has memes I try to ingest regularly) :)
  - counttheforks 4y ago
    What about copyright?
    - deleted 4y ago
      [deleted]
  - spiffytech 4y ago
    How do you ingest your social circle's in-group memes? Are they reliably posted to meme generator sites?

transitivebs 4y ago

My #1 recommendation for anyone thinking about the convoluted OCR solution: use a cheap OCR API and save yourself months of time / hassle / upkeep. Google's OCR API is a good place to start, but AWS has one too and dozens of others out there.

mike_d 4y ago
Without this "convoluted OCR solution" it never would have been built. Mandatory would have easily had to spend hundreds of thousands of dollars to OCR his meme collection alone, even without scraping other meme sites.
elorant 4y ago
If you have to process tens of millions of photos though the cost gradually becomes forbidding.
langsoul-com 4y ago
The author calculated the cost of 1 iPhone se was sub $50, which is 27k Google ocr images. Only makes sense for small scale. Unfortunate how there's no decent ocr library to self host, would be cheoaer than cloud costs.
papito 4y ago
On the contrary, this sort of creative thinking is what's needed instead of automatically reaching for that shiny Cloud Toy. It's easy to get a proof-of-concept working, sure, but at scale, you start torching through cash. Many places keep adding cloud services to their stacks until one day someone in the C-suite notices the AWS bill.

yakubin 4y ago

The photo under Upgrading the iPhone OCR Service Into An OCR Cluster. In the future, data centres are going to host racks of iPhones.

sankha93 4y ago
It is already here to be honest. I know BrowserStack and other mobile testing platforms (at Facebook and Amazon) do host real devices, both Android and iPhones, in server farms like this. Meta wrote a blog post about it: https://engineering.fb.com/2016/07/13/android/the-mobile-device-lab-at-the-prineville-data-center/ https://engineering.fb.com/2016/07/13/android/the-mobile-dev... At one of my previous workplaces, we discussed running the Z3 theorem prover on an iPhone cluster, because they run so much faster on A series processor than a desktop Intel machine.
oefrha 4y ago
Modern app click farms already have walls of iPhones.
formerly_proven 4y ago
Reminds me of imgix, who built their product on Apple libraries so ended up having racks of macs to run their service.
leokennis 4y ago
To be fair...insane performance in a tiny cool package...iPhones would make great servers if you could order them without the screen/camera etc. :-)
- defrost 4y ago
  There's your startup right there - washing line racks of discarded iPhones, near bleeding edge, busted screens, still functional o/wise. Low entry cost, recycling, eco-friendly, . . . a deck that writes itself.

ksdme9 4y ago

Love the inventiveness. My question is about the image distribution costs. All the memes on the site seem to be coming straight off an object storage, all that bandwidth consumption has got to add up(?). Some sort of a CDN might help depending on the search patterns.

ekns 4y ago

Last time I looked into OCR stuff I came to a similar conclusion (though I didn't implement anything back thne). It would be really nice to have "open source" models that had similar performance, without having to deal with the iphone cluster hackery.

j0li0t 4y ago

[flagged]

JustARandomGuy 4y ago

Very inventive. Admittedly when I read the first few paragraphs, I was thinking “he’s got to have $40K of iPhones doing image processing” but you made a good point about being able to use iPhones with screen and other damage. What was your average price per iPhone, if you don’t mind disclosing?

Freak_NL 4y ago

> Better yet, I don’t even want to use them as phones, so even iPhones that are IMEI banned or are locked to unpopular networks are perfectly fine for my use. Fences worldwide will be overjoyed to hear of this novel application.

fuzzygroup 4y ago

This is utterly fantastic and you are to be commended for your Crazy Mad Scientist genius!

merpkz 4y ago

Nice project, I wanted to build meme search engine myself one day, but figured Tesseract will fail at most of the memes because of how stylized those have become. So I tuned down my meme source to only /r/bertstrips as those contain sane looking text and it's working quite alright - project has no frontend yet, I search from cli and click links. > Initial testing with the Postgres Full Text Search indexing functionality proved unusably slow at the scale of anything over a million images, even when allocated the appropriate hardware resources. I can guarantee you that correctly setup PostgreSQL text search will be faster than ES with much, much less hardware resources needed, it's just a matter of correctly creating tsvector column and creating GIN index on it (and ofc asking right queries so it's actually used). I can help you out setting postgres schema up and debugging queries if you are interested, for testing purposes at least.

tmzt 4y ago
I recently worked on a project using lnx.rs. Simple to setup and use and fast at the scale I was using it. Built on Tantivy with a custom fast fuzzy search feature. If you want to go beyond meme sites and possibly detect memes in the wild, common crawl might be something to start with.
iamflimflam1 4y ago
One issue I've had with postgres full text search is when you want to rank using ts_rank you end up with a full table scan.

csande17 4y ago

I wonder how the performance of Vision.framework on desktop Mac hardware compares to a cluster of phones. (The author mentions that it was "fairly slow", but it sounds like they were running an iOS app in the simulator and not a macOS app.)

papito 4y ago
Same thoughts. You can make that fly on a Mac Mini (provided it can be made to work close to the metal and not in an emulator)
lathiat 4y ago
I would expect an M1 native Mac app to work similarly well. Though the iPhone solution may win on price. The framework is supported on macOS (even tvOS apparently) https://developer.apple.com/documentation/vision https://developer.apple.com/documentation/vision

Thorentis 4y ago

IaaS - iPhone as a Service, coming soon to AWS.

oefrha 4y ago

> My preliminary speed tests were fairly slow on my Macbook. However, once I deployed the app to an actual iPhone the speed of OCR was extremely promising (possibly due to the Vision framework using the GPU). I was then able to perform extremely accurate OCR on thousands of images in no time at all, even on the budget iPhone models like the 2nd gen SE. I suppose that’s an old Intel MacBook? I’d be very surprised if the Vision framework performs better on a 2nd gen iPhone SE than even the first M1 MacBook Air.

iamflimflam1 4y ago
I think he was running it in the simulator - which won't perform anywhere near as fast on this type of thing.
- oefrha 4y ago
  The simulator runs a native build though. Might not be optimized, not sure.

dirtyid 4y ago

Outrageous effort! So far japanese, mandarin returning results as well. Do you have list of sources where memes are ingested from? Would be nice to have some option to explore memes by category.

mateuszbuda 4y ago

Great project! With your DIY attitude, if you need to build your own infrastructure for web scraping, here is a tutorial for mobile proxy setup which might be helpful: https://scrapingfish.com/blog/byo-mobile-proxy-for-web-scraping https://scrapingfish.com/blog/byo-mobile-proxy-for-web-scrap...

lysecret 4y ago

Wait what this is absolutely brilliant. Actually insane it works so well using a stack of iPhones as an ocr server. My deepest respect.

philsnow 4y ago

I sat down literally last night and started sketching out the scratch-my-own-itch solution to more or less exactly this problem, because I too have meme-aphasia where I know there exists a meme that fits perfectly in a conversation, but I have about 5 seconds to find it before the moment passes. I'm so, so glad to see that I'm not the only person in the world with the same "problem". Well done, mandatory. edit: holy crap you even index videos, nice

baradhiren07 4y ago

https://findthatmeme.com/?search=take+my+money https://findthatmeme.com/?search=take+my+money

ipsum2 4y ago

This is amazing. Out of curiosity, why not try deep learning OCR software instead of Tesseract? PaddleOCR is popular.

ducktective 4y ago
>PaddleOCR I once tried installing it on a recent Ubuntu. After messing with dependency hell and pip downloading half of the internet, when I finally invoked the CLI, it complained _again_ about a missing runtime dependency. I called it quits. DL people are simply not interested in bundled static binaries.

kome 4y ago

you are a genius. also, the Search Engine works so well

1wsk 4y ago

Could you not extract the model and run it on a server? Its probably not as easy but i know it has been done with NeuralHash

kevmo314 4y ago

Could you expose your iPhone cluster as an OCR API? Seems like it would be competitive with the GCP API.

urbandw311er 4y ago
I wonder if there might be something in the Apple TOS prohibiting this.
- OJFord 4y ago
  I think there is right? At least assuming it's the same as Macs where the minimum billing period is 1 day, because you're technically renting a physical Mac (and 1d is the minimum that qualifies as adhering to that).
  - urbandw311er 4y ago
    I was more referring to bundling up an OS framework library feature and exposing it as an Internet API.

Arbortheus 4y ago

This is great, I particularly like the part about using compute from old unwanted iPhones. Quite an inventive way to reuse/recycle otherwise obsolete hardware!

lormayna 4y ago

If you don't need advanced search features, you can use Sonic (https://github.com/valeriansaliou/sonic https://github.com/valeriansaliou/sonic). It's blazing fast and you can save lot of money on servers.

remram 4y ago
Someone mentioned lnx.rs in this thread, how does it compare? And how about all the other new Rust solutions like MeiliSearch, Toshi, Quickwit?
- lormayna 4y ago
  I don't know, I am not an expert on this topic. But this could be a nice topic for a blog article that will hit the frontpage on HN :)

petesergeant 4y ago

I was hoping this would help me find the Database Iceberg meme that shows different levels of database insanity. It didn’t. Anyone have a link?

cnity 4y ago
This one? https://www.reddit.com/r/ProgrammerHumor/comments/wuu689/the_sql_iceberg/ https://www.reddit.com/r/ProgrammerHumor/comments/wuu689/the...
- petesergeant 4y ago
  Yes, that’s it, thankyou. Been searching a few days for it
  - cnity 4y ago
    You're welcome. FWIW I literally put your wording into google image search to find it: > Database Iceberg meme that shows different levels of database insanity But possibly my google search results are tailored differently from yours.

spuz 4y ago

I'm curious how well the iPhone OCR actually works. How do you deal with errors? Is the error rate low enough that you can accept the output from the iPhone OCR as is or do you also run it through a cleaning process (e.g. spell check)?

9dev 4y ago
For a single data point, it works exceptionally well for me. I routinely copy-paste from images or screenshots, and it rarely fails (mostly for handwriting or obscure fonts).
- leokennis 4y ago
  I am not sure if the Photos app search also uses the same OCR. But sometimes I search for a word and iOS will find a photo that has that text in a 50x20 cluster of pixels somewhere far in the background...it's remarkably good.

marginalia_nu 4y ago

Now this is the sort of disgusting pile of jank I love to see.

tomw1808 4y ago

Absolutely amazing on the tech side!!! Now, after reading the article, I gave your search engine a try. I was looking for that futurama its a trap meme (pretty much pops up on any image search here https://www.google.com/search?q=futurama+its+a+trap https://www.google.com/search?q=futurama+its+a+trap) The problem is, the search engine you built is now very text-heavy, which seems to be usually very unconnected to the actual meme. So, searching for "its a trap" did not yield the results I was actually hoping for, but made total sense looking at how the search was implemented. Are you planning to implement an actual tagging of the content of some sorts? Maybe a clustering of similar objects (like iphone clusters similar peoples faces in the gallery) and then tag those clusters with keywords somehow?

mandatory 4y ago
Yes I definitely want to improve the search to be better. It is currently very text heavy and I (only recently) got image similarity indexing working. Hoping to leverage this to do something like you mentioned! I'd also like to figure out how to turn an image into a description of whats in it. My ML/tensorflow knowledge is very weak though, so I still have a lot to learn here.

yreg 4y ago

This is absolutely brilliant. I believe I will actually use it a lot if you keep the site up. Minor feedback for the blog post: It deserves a better meta description (for link previews). The first paragraph doesn't advertise how good the article is going to be.

sneak 4y ago

Don't you have to re-sign and re-deploy tour iOS app every 7 days to keep it running on the iPhones?

Liquidor 4y ago

Brilliant! :-) Maybe a dumb question, but could you use your data to train a new OCR model so you wouldn't have to rely on iOS? I don't know much about ML/AI so maybe not feasible.

nysv 4y ago

If only there was a way to filter out ifunny results, I absolutely detest that watermark.

francis-io 4y ago

Would be great if the images had a unique name so I could save them without having to rename them.

mseidl 4y ago

Do you use mongodb to make it web scale? You turn it on and it scales right up.

zffr 4y ago
https://www.youtube.com/watch?v=b2F-DItXtZs https://www.youtube.com/watch?v=b2F-DItXtZs

Tade0 4y ago

I wonder how it fares against deep-fried "E" or Opossum memes?

geek_at 4y ago

really amazing! I love the solution and the project in general

aabajian 4y ago

I had a friend in med school who wrote a very early note-taking app for the iPad. Turns out that there was no way to render PowerPoint files when the iPad first came out. He realized that the iOS/Mac OS "quick preview" function could be used to take screenshots of each PowerPoint slide. For a brief time, his was the only app that could display PowerPoints (albeit, they were just screenshots!). There's a lot of hidden utility in Apple libraries.

surume 4y ago

Thank you for building this! It's so much fun. I looked for memes that I saw years ago and found them in seconds. Excellent work!!

nisegami 4y ago

Is this the person on /r/hardwareswap who's been looking for semi-functional, even IMEI banned, iPhones?

2Gkashmiri 4y ago

Are you going to open source the "app" part of it ? I would love to replicate this setup for my own project.... I am thinking, load balanced, multi location redundant "iOS machines" with 3-4 phones in with power backup and internet dongle. We could use something like zerotier/tailscale to get internet access from outside your local network

CiceroCiceronis 4y ago

This is really brilliant to see, and I've been surprised for quite a long time that nothing similar exists. I think it's a real shame that few people with interest in memes have interest in building solutions like this that help us engage with them. People in the 21st century know a lot about the mistakes of the past century that led to much popular culture of the time being lost (especially terminally online people who've watched lots of Youtube documentaries about lost Dr. Who episodes and so on), so it surprises me how little we try and avoid those same mistakes with today's ephemeral pop culture in the form of memes. People like yourself who want to help make the internet's huge corpus of memes tractable are part of the solution in terms of meme archival and cultural memory. (There's a good meme metadiscussion group on Discord, "The Philosopher's Meme," which you might be interested in joining. People there would be very keen to discuss what you've made.)

spiffytech 4y ago
I've always been surprised that Reddit hasn't built meme search into its site search. Memes are a core part of the Reddit experience, yet it's difficult to find something I know I saw before.
jychang 4y ago
Not familiar with Discord, do you have a link to that group?
- CiceroCiceronis 4y ago
  https://discord.com/invite/8MVFRMa https://discord.com/invite/8MVFRMa

gerdesj 4y ago

"It looked like it was time to bite the bullet and write an OCR iOS server in Swift." Quite a large bullet required, one with plenty of chewing left in it.

kgbcia 4y ago

That's genius. I realized the cost advantage of text to speech on an old android versus Google cloud

causality0 4y ago

I tried a few memes. The results were quite poor, and vastly inferior to just using Google. In the case of text searches I had to scroll through dozens of results before finding the original meme images.

iamflimflam1 4y ago

Love the hackiness of this - however, the vision framework is available on Desktop macs as well - https://developer.apple.com/documentation/vision https://developer.apple.com/documentation/vision and specifically: https://developer.apple.com/documentation/vision/vnrecognizetextrequest https://developer.apple.com/documentation/vision/vnrecognize...

price456987 4y ago
What's the cost of building and running a cluster of iPhones vs Mac Minis?
- Bad_CRC 4y ago
  in the article $40 second hand, imei banned and broken screen iPhones are being used so...
  - vlunkr 4y ago
    There's a ton of compute power available in the form of unused phones.
pronoiac 4y ago
There's ocrit, a CLI utility using Apple's Vision framework for OCR: https://github.com/insidegui/ocrit https://github.com/insidegui/ocrit
cassiogo 4y ago
> My preliminary speed tests were fairly slow on my Macbook. However, once I deployed the app to an actual iPhone the speed of OCR was extremely promising (possibly due to the Vision framework using the GPU). I was then able to perform extremely accurate OCR on thousands of images in no time at all, even on the budget iPhone models like the 2nd gen SE. He does mention running it on a macbook
- dblitt 4y ago
  I would assume he's using an intel macbook and wouldn't have the gpu acceleration (and subsequent Vision framework integration) of the m1

skizm 4y ago

Where did the original set of meme images, gifs, and videos come from?

coayer 4y ago

I love the iPhone cluster so much!