4 ms·

Show HN: Make your AngularJS, EmberJS, or BackboneJS website crawlable by Google

13 points by chaddeshon 13y ago

http://www.BromBone.com I posted “Tell HN: My Web App has 13 Users” 3 months ago (https://news.ycombinator.com/item?id=5386966). Over time, it grew to 150 (free) users, but retention was pretty much zero. I was trying to cast a wide net and offer a generic “headless browser as a service”. It turned out I had make a service that was a little helpful to many people, but didn't really solve anyone's problem completely. I finally took the advise offered in that thread. I focused on a problem I understood, and solved it completely instead of just offering a helpful tool. Instead of a “hosted headless browser” BromBone is now a complete solution for people who want their javascript websites indexed by Google. BromBone.com - The javascript SEO problem has been solved. Google can't index javascript driven webpages. The accepted solutions have been: 1. Run your own PhantomJS server to render snapshots and serve the snapshots to Google (a real pain). 2. Accept that your pages won't be listed on Google. With BromBone you don't have to either. You just have to make a small change to your .htaccess file. BromBone crawls your websites, runs your javascript in a web browser, and saves the rendered page. When Google crawls your site, proxy the requests to BromBone, and we'll send the rendered snapshot to Google. Smile. Your javascript driven website is on Google.

chaddeshon 13y ago
Clickable: http://www.brombone.com http://www.brombone.com
tocomment 13y ago
That sounds really useful! Will google penalize for this though?
- chaddeshon 13y ago
  This is actually the method that Google recommends for these AJAX powered websites. https://developers.google.com/webmasters/ajax-crawling/ https://developers.google.com/webmasters/ajax-crawling/ I am just letting people out source the hard work. Since the rendered page will be proxied from BromBone, there will be no way to tell that the same it rendered the file instead of the normal webserver.
  - wikwocket 13y ago
    That's very useful info; I had no idea. I thought the Google crawlers were basically headless Chrome bots so I wouldn't have expected this. Do you mention this on your site? Seems like a good endorsement, "This is the correct way to get indexed, as per Google!"
    - chaddeshon 13y ago
      I have a section about that. I hope people will read it.
tocomment 13y ago
Perhaps you could use crawl frequency in your pricing. For example daily (or even hourly) crawls for the top tier plan. Most people probably don't have that many pages so I'm thinking the existing pricing tiers might not work well? And maybe offer a free plan that only crawls your site 3x per year or something really slow like that.
- chaddeshon 13y ago
  I almost did that. Ultimately I decided to take an "it just works" philosophy to crawl frequency (unless you need realtime). Would you segment the number of pages differently? I think some sites with a lot of user generated content might actually have a ton of pages. But it is hard to get real numbers.
  - tocomment 13y ago
    Hmm I still like the crawl frequency as a price differentiator personally. Other ideas might be a service level guarantee for the high priced plans. And included email support for setup.
  - tocomment 13y ago
    Maybe Patio11 will chime in with some ideas for your pricing plans? Everything I know I got from reading his comments/blogs. (I think he searches for mentions of his username, so the bat signal is officially turned on :-)
    - patio11 13y ago
      They're undepriced, but I think the first problem is going to be going from 0 users to 10 users (bang down doors) rather than extracting the maximum proper value from those first ten users. The optimal pricing strategy will still lead to him having 0 customers until he figures out who needs this and how to reach them repeatedly.
      - chaddeshon 13y ago
tocomment 13y ago
Do you think there would be any demand for a separate service that makes it easy to screen scrape javascript heavy sites? (might be legal issues there though?) (Sorry for all the comments, this concept intrigues me)
- chaddeshon 13y ago
  I don't know. That was brought up in my previous posting, and I've had some people email me about it. It's hard for me to tell. I'm just not in that community enough to understand their needs. I'm afraid that every scraping project is just enough different that it would be hard to write something that offered value to a lot of people.
  - tocomment 13y ago
    That's probably a good point. I guess if someone wants to screenscape a dynamic JS heavy page they'd probably want to interact with the page.
tocomment 13y ago
Marketing idea: How about you offer a free version but insert a link back to brombone in the crawable/rendered version? As long as you were very upfront about it, it might not be too offensive.
- swah 13y ago
  Wouldn't that blacklist his own site?
  - tocomment 13y ago
    Why would it? I guess I don't know a lot about SEO.
    - swah 13y ago
      Me neither :) But my line of thinking was that lots of sites that have nothing in common pointing to the same site... must mean that that site isn't isn't very relevant (Now that I'm trying to explain it I see my mistake: it could just mean that the site is very general, like Facebook.)
    - wikwocket 13y ago
      Some people think that embedding a link to your site in a web product/plugin you distribute can lead to bad things, especially if it is not a nofollow link and your plugin is used by nefarious sites. http://www.jitbit.com/news/184-how-google-authorship-and-penguin-have-killed-our-traffic/ http://www.jitbit.com/news/184-how-google-authorship-and-pen... has a bit about the Penguin algorithm and this effect.
tech-dragon 13y ago
You have me wondering about the feasibility of offering this as a CloudFlare app. If I was able to throw up a little ajax/client side JS driven app at a domain, throw CloudFlare over it (like I usually do for most smaller domains I own) and then have it all happen without even needing to change that .htaccess file, then your onto something even better. I also think the pricing model may need some work. Per page seems like it could cut you off from some potential customers. For instance, I'm already thinking of a blog engine I wanted to try out more thoroughly by making it the one i use on my new blog but wasnt prepared to loose "google-ability". I Would like to be able to use a service like this but when I do my mental math it doesn't add up. I wanted to use a blog engine that will produce a large number of pages by pulling in third party service activity into my site. So its not 100% what I want, but damn its tempting. If your pricing it to cover costs, might I suggest looking to optimize your stack to bring down those costs down?. If you can cut the price and make it more of a 'yeah for $X its easier than doing it myself' you'll get more takers. Right now it feels that your current lowest price per month is too high for a lot of devs to decide the $ is a throwaway expense to save them time. Id hazard, below $25, and probably with an order of magnitude more pages. All that said... I may still try it out. It looks interesting.
- chaddeshon 13y ago
  Thanks for your suggestion about CloudFlare. I am looking into it.

13 points by chaddeshon 13y ago

chaddeshon 13y ago

Clickable: http://www.brombone.com http://www.brombone.com

tocomment 13y ago

That sounds really useful! Will google penalize for this though?

chaddeshon 13y ago
This is actually the method that Google recommends for these AJAX powered websites. https://developers.google.com/webmasters/ajax-crawling/ https://developers.google.com/webmasters/ajax-crawling/ I am just letting people out source the hard work. Since the rendered page will be proxied from BromBone, there will be no way to tell that the same it rendered the file instead of the normal webserver.
- wikwocket 13y ago
  That's very useful info; I had no idea. I thought the Google crawlers were basically headless Chrome bots so I wouldn't have expected this. Do you mention this on your site? Seems like a good endorsement, "This is the correct way to get indexed, as per Google!"
  - chaddeshon 13y ago
    I have a section about that. I hope people will read it.

tocomment 13y ago

Perhaps you could use crawl frequency in your pricing. For example daily (or even hourly) crawls for the top tier plan. Most people probably don't have that many pages so I'm thinking the existing pricing tiers might not work well? And maybe offer a free plan that only crawls your site 3x per year or something really slow like that.

chaddeshon 13y ago
I almost did that. Ultimately I decided to take an "it just works" philosophy to crawl frequency (unless you need realtime). Would you segment the number of pages differently? I think some sites with a lot of user generated content might actually have a ton of pages. But it is hard to get real numbers.
- tocomment 13y ago
  Hmm I still like the crawl frequency as a price differentiator personally. Other ideas might be a service level guarantee for the high priced plans. And included email support for setup.
- tocomment 13y ago
  Maybe Patio11 will chime in with some ideas for your pricing plans? Everything I know I got from reading his comments/blogs. (I think he searches for mentions of his username, so the bat signal is officially turned on :-)
  - patio11 13y ago
    They're undepriced, but I think the first problem is going to be going from 0 users to 10 users (bang down doors) rather than extracting the maximum proper value from those first ten users. The optimal pricing strategy will still lead to him having 0 customers until he figures out who needs this and how to reach them repeatedly.
    - chaddeshon 13y ago

tocomment 13y ago

Do you think there would be any demand for a separate service that makes it easy to screen scrape javascript heavy sites? (might be legal issues there though?) (Sorry for all the comments, this concept intrigues me)

chaddeshon 13y ago
I don't know. That was brought up in my previous posting, and I've had some people email me about it. It's hard for me to tell. I'm just not in that community enough to understand their needs. I'm afraid that every scraping project is just enough different that it would be hard to write something that offered value to a lot of people.
- tocomment 13y ago
  That's probably a good point. I guess if someone wants to screenscape a dynamic JS heavy page they'd probably want to interact with the page.

tocomment 13y ago

Marketing idea: How about you offer a free version but insert a link back to brombone in the crawable/rendered version? As long as you were very upfront about it, it might not be too offensive.

swah 13y ago
Wouldn't that blacklist his own site?
- tocomment 13y ago
  Why would it? I guess I don't know a lot about SEO.
  - swah 13y ago
    Me neither :) But my line of thinking was that lots of sites that have nothing in common pointing to the same site... must mean that that site isn't isn't very relevant (Now that I'm trying to explain it I see my mistake: it could just mean that the site is very general, like Facebook.)
  - wikwocket 13y ago
    Some people think that embedding a link to your site in a web product/plugin you distribute can lead to bad things, especially if it is not a nofollow link and your plugin is used by nefarious sites. http://www.jitbit.com/news/184-how-google-authorship-and-penguin-have-killed-our-traffic/ http://www.jitbit.com/news/184-how-google-authorship-and-pen... has a bit about the Penguin algorithm and this effect.

tech-dragon 13y ago

You have me wondering about the feasibility of offering this as a CloudFlare app. If I was able to throw up a little ajax/client side JS driven app at a domain, throw CloudFlare over it (like I usually do for most smaller domains I own) and then have it all happen without even needing to change that .htaccess file, then your onto something even better. I also think the pricing model may need some work. Per page seems like it could cut you off from some potential customers. For instance, I'm already thinking of a blog engine I wanted to try out more thoroughly by making it the one i use on my new blog but wasnt prepared to loose "google-ability". I Would like to be able to use a service like this but when I do my mental math it doesn't add up. I wanted to use a blog engine that will produce a large number of pages by pulling in third party service activity into my site. So its not 100% what I want, but damn its tempting. If your pricing it to cover costs, might I suggest looking to optimize your stack to bring down those costs down?. If you can cut the price and make it more of a 'yeah for $X its easier than doing it myself' you'll get more takers. Right now it feels that your current lowest price per month is too high for a lot of devs to decide the $ is a throwaway expense to save them time. Id hazard, below $25, and probably with an order of magnitude more pages. All that said... I may still try it out. It looks interesting.

chaddeshon 13y ago
Thanks for your suggestion about CloudFlare. I am looking into it.