6 ms·

Fair warning: This is a blog post advertisement for ScrapingBee. The data is still interesting. The most interesting chart is one of the last: Proportion of Fa

Fair warning: This is a blog post advertisement for ScrapingBee. The data is still interesting. The most interesting chart is one of the last: Proportion of Failures over time. As expected, more recent product links are less likely to 404 or 5xx. Going back to 2014, almost 1/3 of the featured links give a 4xx or a 5xx response. That’s a lot! More surprising, links as recent as 2020 show a 1/4 failure rate. Those projects basically launched on PH, then shut down shortly afterward. Moreover, this analysis can’t actually account for products that have been shuttered but still have landing pages online. It’s ultra cheap to keep a placeholder “Sorry we’re closed” page online, so I imagine a lot of these projects are shutdown but counted as “success”. Subjectively, this matches what I’ve gathered from watching PH. Getting a PH featured product listing seems to be a badge of honor, but PH users aren’t really interested in using 99% of the products and the submitters aren’t actually interested in building them past proof of concept. Recently, the bulk of postings seem to be advertisements for paid information products or pay-to-join communities.

imilk 5y ago
> Fair warning: This is a blog post advertisement for ScrapingBee. The data is still interesting. Sure, but it's no different than any other blog post from a company. And framing it that way is quite disingenuous since the post pretty much only sticks to the topic and doesn't overtly promote their product.
daolf 5y ago
I find your warning a bit unfair as there are literally no CTA inside the blog content promoting our product and only 2 internal links toward other educational posts. But anyway, I thought about taking a random sample of pages who returns a "200". Let's say 150, and manually tagging them to find if they're "dead" or not. And then reuse the "dead or alive but a 200" ratio for all the pages but I was afraid that I'd need to tag much more than 150 pages to have a significant statistical result.
- runnerup 5y ago
  For what it's worth, you could watch how quickly the confidence intervals converge as you sample the data, to see if it's worth continuing or if the variance is too high and whether you'd have to check thousands of pages by hand: from scipy.stats import binomtest chance_of_dead_page = binomtest(landing_page_counter["dead"], landing_page_counter["total"]).proportion_ci(confidence_level=0.90) print(f'Chance of a dead but existing landing page (90% Confidence Interval):{chance_of_dead_page.low * 100:.2f}% to {chance_of_dead_page.high * 100:.2f}%')
- PragmaticPulp 5y ago
  > I find your warning a bit unfair as there are literally no CTA inside the blog content promoting our product It’s obviously blog content designed to promote your product, hosted on the company’s product website. I don’t see how the FYI is unfair. I added it because the content was valuable but HN can be finicky about blog posts from companies advertising their own products. Trying to get ahead of indignant dismissals.
  - halpert 5y ago
    Where do they promote the product? It looks like they’re using a random http library.
  - dewey 5y ago
    > It’s obviously blog content designed to promote your product, hosted on the company’s product website. I don’t see how the FYI is unfair. There's so many blog posts posted here that could fall under "content marketing" umbrella if you want to be strict. I feel like there's no problem with that if the content is valuable and people like/upvote it. After all this is a platform that is doing marketing for YC where YC companies are supposed to post their content too. That "warning" also stuck out to me as a bit unfair as I was even looking for how it hooks into ScrapingBee (as I was curious how these scraping-aaS platforms interface with custom code) and couldn't find anything.
  - 1123581321 5y ago
    Yours ended up coming across as the indignant dismissal. As a community member I didn’t appreciate the warning. From the second paragraph on your comment was an interesting contribution, though. I’m surprised that many of those PoC businesses have stayed online at all, but I guess romaine are easy to renew.
  - deleted 5y ago
    [deleted]
- time_to_smile 5y ago
  > I find your warning a bit unfair as there are literally no CTA inside the blog content promoting our product and only 2 internal links toward other educational posts. I've worked in or adjacent to the content marketing world long enough to know that a CTA is not necessary for the post to be marketing/advertising. One of the major goals of content marketing it to establish the authority of the brand. You are well aware that the raison d'etre of that post is to spread awareness of and establish the authority of ScrapingBee. It doesn't mean the post is not interesting, useful or valuable. But that post exists fundamentally for marketing/brand purposes. Parents warning is completely fair, especially since they immediately point out the value of the post.
  - deleted 5y ago
    [deleted]
  - frogpelt 5y ago
    What's the point of the warning though? If you're paying attention you know it's content marketing. If you're oblivious, the marketing probably isn't working. Either way, you probably don't need a warning.
    - pessimizer 5y ago
      I love warnings, because I don't have to pay as much attention.
nicoburns 5y ago
> Going back to 2014, almost 1/3 of the featured links give a 4xx or a 5xx response. That’s a lot! Is that a lot? I would have been less surprised if it were 1/3rd of links still live.
- brimble 5y ago
  I thought the same thing until I realized that dead domains are often snapped up by squatters/spammers (or just by other people who want that domain for actual reasons) so may not error when requested.
martin_a 5y ago
> More surprising, links as recent as 2020 show a 1/4 failure rate. Those projects basically launched on PH, then shut down shortly afterward. This would very well fit a "fail fast" attitude with testing MVPs, wouldn't it? At least that's what I would guess. Got a great start with PH but didn't move on from there, so the domain was not renewed...
- city41 5y ago
  Does a great start on PH mean very much? The chances your target audience is there for most products seems very low. I would love to see this data compared to all products/startups in general, but of course that's probably difficult to do.
  - granshaw 5y ago
    Bingo. IME Producthunt is lazy marketing for indie founders. Their main user base is other founders, wannabe founders and super tech literate power users who are itching to use “the next new beta thing”, who’ll be a tiny percentage of any userbase. Granted there are cases where that market IS aligned with your product, eg if you built a low cost site-builder or low cost social media publishing platform
zaarn 5y ago
Don't forget cases where a product's domain expired and has since be reused for something else entirely (or a product with similar goal but new vendor)
agentdrtran 5y ago
I would love a PH where you have to show off an actual product (instead of blogspam guides, design resources, etc, even though these can be useful)
polote 5y ago
> Fair warning: This is a blog post advertisement for ScrapingBee. Yeah we have all seen it is on scrapingBee, no need for a warning.
paxys 5y ago
The most surprising thing to me is not the failures but that the devs won't even pay a few bucks a year to keep the domains online. If I spent time and effort into building a product that went viral and got a bunch of users, I'd at least leave the front page of it up indefinitely as some sort of tombstone.
sasquatch69 5y ago
Fair warning: this response is an advertisement for ycombinator
itronitron 5y ago
That's a lot of incredible journeys that have reached their end.

imilk 5y ago

> Fair warning: This is a blog post advertisement for ScrapingBee. The data is still interesting. Sure, but it's no different than any other blog post from a company. And framing it that way is quite disingenuous since the post pretty much only sticks to the topic and doesn't overtly promote their product.

daolf 5y ago

I find your warning a bit unfair as there are literally no CTA inside the blog content promoting our product and only 2 internal links toward other educational posts. But anyway, I thought about taking a random sample of pages who returns a "200". Let's say 150, and manually tagging them to find if they're "dead" or not. And then reuse the "dead or alive but a 200" ratio for all the pages but I was afraid that I'd need to tag much more than 150 pages to have a significant statistical result.

runnerup 5y ago
For what it's worth, you could watch how quickly the confidence intervals converge as you sample the data, to see if it's worth continuing or if the variance is too high and whether you'd have to check thousands of pages by hand: from scipy.stats import binomtest chance_of_dead_page = binomtest(landing_page_counter["dead"], landing_page_counter["total"]).proportion_ci(confidence_level=0.90) print(f'Chance of a dead but existing landing page (90% Confidence Interval):{chance_of_dead_page.low * 100:.2f}% to {chance_of_dead_page.high * 100:.2f}%')
PragmaticPulp 5y ago
> I find your warning a bit unfair as there are literally no CTA inside the blog content promoting our product It’s obviously blog content designed to promote your product, hosted on the company’s product website. I don’t see how the FYI is unfair. I added it because the content was valuable but HN can be finicky about blog posts from companies advertising their own products. Trying to get ahead of indignant dismissals.
- halpert 5y ago
  Where do they promote the product? It looks like they’re using a random http library.
- dewey 5y ago
  > It’s obviously blog content designed to promote your product, hosted on the company’s product website. I don’t see how the FYI is unfair. There's so many blog posts posted here that could fall under "content marketing" umbrella if you want to be strict. I feel like there's no problem with that if the content is valuable and people like/upvote it. After all this is a platform that is doing marketing for YC where YC companies are supposed to post their content too. That "warning" also stuck out to me as a bit unfair as I was even looking for how it hooks into ScrapingBee (as I was curious how these scraping-aaS platforms interface with custom code) and couldn't find anything.
- 1123581321 5y ago
  Yours ended up coming across as the indignant dismissal. As a community member I didn’t appreciate the warning. From the second paragraph on your comment was an interesting contribution, though. I’m surprised that many of those PoC businesses have stayed online at all, but I guess romaine are easy to renew.
- deleted 5y ago
  [deleted]
time_to_smile 5y ago
> I find your warning a bit unfair as there are literally no CTA inside the blog content promoting our product and only 2 internal links toward other educational posts. I've worked in or adjacent to the content marketing world long enough to know that a CTA is not necessary for the post to be marketing/advertising. One of the major goals of content marketing it to establish the authority of the brand. You are well aware that the raison d'etre of that post is to spread awareness of and establish the authority of ScrapingBee. It doesn't mean the post is not interesting, useful or valuable. But that post exists fundamentally for marketing/brand purposes. Parents warning is completely fair, especially since they immediately point out the value of the post.
- deleted 5y ago
  [deleted]
- frogpelt 5y ago
  What's the point of the warning though? If you're paying attention you know it's content marketing. If you're oblivious, the marketing probably isn't working. Either way, you probably don't need a warning.
  - pessimizer 5y ago
    I love warnings, because I don't have to pay as much attention.

nicoburns 5y ago

> Going back to 2014, almost 1/3 of the featured links give a 4xx or a 5xx response. That’s a lot! Is that a lot? I would have been less surprised if it were 1/3rd of links still live.

brimble 5y ago
I thought the same thing until I realized that dead domains are often snapped up by squatters/spammers (or just by other people who want that domain for actual reasons) so may not error when requested.

martin_a 5y ago

> More surprising, links as recent as 2020 show a 1/4 failure rate. Those projects basically launched on PH, then shut down shortly afterward. This would very well fit a "fail fast" attitude with testing MVPs, wouldn't it? At least that's what I would guess. Got a great start with PH but didn't move on from there, so the domain was not renewed...

city41 5y ago
Does a great start on PH mean very much? The chances your target audience is there for most products seems very low. I would love to see this data compared to all products/startups in general, but of course that's probably difficult to do.
- granshaw 5y ago
  Bingo. IME Producthunt is lazy marketing for indie founders. Their main user base is other founders, wannabe founders and super tech literate power users who are itching to use “the next new beta thing”, who’ll be a tiny percentage of any userbase. Granted there are cases where that market IS aligned with your product, eg if you built a low cost site-builder or low cost social media publishing platform

zaarn 5y ago

Don't forget cases where a product's domain expired and has since be reused for something else entirely (or a product with similar goal but new vendor)

agentdrtran 5y ago

I would love a PH where you have to show off an actual product (instead of blogspam guides, design resources, etc, even though these can be useful)

polote 5y ago

> Fair warning: This is a blog post advertisement for ScrapingBee. Yeah we have all seen it is on scrapingBee, no need for a warning.

paxys 5y ago

The most surprising thing to me is not the failures but that the devs won't even pay a few bucks a year to keep the domains online. If I spent time and effort into building a product that went viral and got a bunch of users, I'd at least leave the front page of it up indefinitely as some sort of tombstone.

sasquatch69 5y ago

Fair warning: this response is an advertisement for ycombinator

itronitron 5y ago

That's a lot of incredible journeys that have reached their end.