3 ms·
Hey justhw, I launched something similar [1] a few months back. I’d love to talk with you about your stack. Here are the three biggest problems I ran into. 1.
by joemasilotti 6y ago
Hey justhw, I launched something similar [1] a few months back. I’d love to talk with you about your stack. Here are the three biggest problems I ran into.
1. Request timeouts
Even "fast" blogs sometimes have hiccups or rate-limit scrapers/crawlers. The more I scale the more often I see requests timing out. I've tried to mitigate this with retry buttons and trying the request again, but it doesn't always work.
2. Scaling Puppeteer
I saw from another of your comments that this is built on top of Puppeteer. From my experience, trying to scale to even 3 concurrent renders starts to fall apart when you have to spin up a new Chrome instance for each request.
I've looked into Browserless [2] but they charge per second of usage, and I'm not at the point where I want to have that big of an expense.
3. Getting a relevant image
I love that your tool adds an image to the thumbnail as well! I struggled with searching Unsplash using the text in the title of the article but they never came out that well.
I love it when people make these massively useful services stitching together a few simple tools. Congrats on the launch!
[1] https://mugshotbot.com https://mugshotbot.com
[2] https://www.browserless.io https://www.browserless.io
- justhw 6y agoHi joel, we can def chat. Email in profile.
- ingvul 6y agoWould love to hear some sort of summary though :)
- justhw 6y agoHey joe, I actually haven't had scaling issues yet since the requests are relatively few but I run it on a cpu optimized server so that might be helping out. As for getting a relevant image I use pixabay.com, their TOS and database is a lot. Tip when searching for image, filter your title using keyword extraction library [1] to simplify the query. Hope that answers some things.
- justhw 6y agoEdit: * their TOS is user friendly and they carry more images. [1] = https://pypi.org/project/rake-nltk/ https://pypi.org/project/rake-nltk/
- deleted 6y ago[deleted]