6 ms·

Show HN: Checkbot for Chrome – web crawler that tests for web best practices

83 points by seanwilson 9y ago

seanwilson 9y ago
Hi, I didn't get much feedback when I posted last time so I'm giving it another try. This is aimed at helping web developers follow SEO, performance and security best practices so I'd love to know what the community thinks. Can you think of any changes that would make Checkbot more helpful? Did you notice any bugs? Thanks!
- cdawg0 9y ago
  Cool, I'm checking it out right now. Can you share what your tool provides that others don't?
  - seanwilson 9y ago
    The major one is that instead of manually checking pages one at a time, Checkbot lets you easily test 1,000s of pages in a few minutes to root out issues you'd normally miss. As you're doing web crawls from your own machine, you can also crawl any site you want as often as you want including localhost/development, staging and production sites. This lets you identify issues early and confirm fixes during development before problems go live.
    - ajeet_dhaliwal 9y ago
      Who’s the customer you have in mind who will want to do this for thousands of pages?
      - seanwilson 9y ago
        People involved in web development, SEO and marketing. It doesn't need to be in the thousands but checking large number of pages for problems has obvious productivity benefits over checking one page at a time. It also lets you identify issues that impact multiple pages like duplicate titles, descriptions and content that you'd normally miss.
    - cdawg0 9y ago
      Interesting. So the intent is to tackle problems before deployment. Do you plan on any devtools integrations so it can be used as part of an automatic CI/CD process? Also, does Checkbot dig into all dependancies or skip them like some others?
      - seanwilson 9y ago
        > Interesting. So the intent is to tackle problems before deployment. Do you plan on any devtools integrations so it can be used as part of an automatic CI/CD process? A lot of the time, websites owners don't know there's a problem until their search results or Google Search Console updates. So I'm seeing it being used by developers to check localhost/development sites, then on staging for other problems, then on production when changes are made there. A command-line version to support CI/CD is something I'm really interested in as well. > Also, does Checkbot dig into all dependancies or skip them like some others? Can you expand on what you mean here?
    - idle_processor 9y ago
      > The major one is that instead of manually checking pages one at a time, Checkbot lets you easily test 1,000s of pages in a few minutes to root out issues you'd normally miss. ScreamingFrog does this too, but it's still nice that your work is out there to add to the suite.
- idle_processor 9y ago
  - A regex blacklist for URL structure would be very helpful. - Acceptable title length seems a bit short. 70 characters seems closer to what we're allowed today than the old 60 (per http://www.bigleap.com/blog/5-tips-take-advantage-googles-new-meta-title-guidelines-2016/ http://www.bigleap.com/blog/5-tips-take-advantage-googles-ne...). Might also be worth segmenting URLs with query parameters in them into a low priority batch to check later (or skip). (E.g., when spidering a WordPress site, the crawler wastes time on .../article/?replytocom=* URLs. URL filtering solves this, somewhat, but it might require multiple passes to identify all of the problematic query strings.)
ramon 9y ago
Hi, I have downloaded already but didn't test it yet. I will test as soon as I can.
mahesh_rm 9y ago
I rapidly tested it. It's staying on my chrome. Good Job.
- seanwilson 9y ago
  Thanks, let me know how you get on!
queezey 9y ago
Very cool. I noticed that it's mangling some of my URLs, though. `/!0ead1aEq` is getting turned into `/%210ead1aEq` (the exclamation point is getting percent-encoded), which leads to a bunch of spurious 404 errors. https://tools.ietf.org/html/rfc3986#section-3.3 https://tools.ietf.org/html/rfc3986#section-3.3
- seanwilson 9y ago
  Ah, thanks a good bug report! I'll get this fixed.
- seanwilson 9y ago
  Hmm, so when crawling, URLs are normalised and the URL library I'm using is normalising the "!" to "%21". Could you send me a working URL to test on? My email is sw@seanw.org if you want to use that.
Raphmedia 9y ago
Works pretty good. Incidentally, I just locked myself from my own server. This double as a security tester! :o)
- seanwilson 9y ago
  On the left sidebar at the start you can modify the number of URLs crawled per second if that helps!
ratata 9y ago
Nice. very similar to Lighthouse https://developers.google.com/web/tools/lighthouse/#devtools https://developers.google.com/web/tools/lighthouse/#devtools
- kaycebasques 9y ago
  Which is also available in Chrome DevTools via the Audits panel.
  - PudgePacket 9y ago
    I've never been able to use that, every time I run the test it says initial page load is 10s, but I'm running it against a static website hosted locally.. how is it taking so long? Even if I visited my site from the other side of the world over 3g it would be faster than lighthouse accessing it locally.
    - pimlottc 9y ago
      Part of the audit includes throttling your connection to simulate a device on a slower link. https://github.com/GoogleChrome/lighthouse/blob/master/docs/throttling.md https://github.com/GoogleChrome/lighthouse/blob/master/docs/...
      - PudgePacket 9y ago
        Still couldn't account for 10s for 5kb of html, something must have been really wrong, maybe I configured it incorrectly.
- wgjordan 9y ago
  Similar feature-set at first glance, but not open-source, not free after beta ends, and created/maintained by an unknown solo developer. Nice site design, but I think the cards are pretty heavily stacked against this one.
  - 9y ago
chasers 9y ago
It's nice and fast! Are you actually using chrome to render each page or making requests some other way?
- seanwilson 9y ago
  Thanks! All requests are done from your own browser if that's what you mean.
  - chasers 9y ago
    No, I mean like render the whole page with JS and all, but from your FAQ it seems like you're not. Which is why it's fast, ha.
gerdesj 9y ago
Very cool. I'm giving it a quick go whilst doing yet another patchathon on customer systems. First impressions are that it is very quick and gives some great advice. I'm not really a web. dev. but I even I can see how this can make a good audit tool. Looks great as well. I suggest caution against using the term "best practice" though. It's one of my pet hates - there is good practice and there is bad practice but its a brave person who claims to know best practice. I think my hatred of that term stems from seeing it plastered all over older MS docs and the usual crop of "me too" copy n paste blog postings that litter the web, not to mention various forum postings. We're all bloody experts who know best in this game 8)
- keithnz 9y ago
  I agree, it runs nicely, and is nicely presented. The term best practice does irk me. But weirdly enough I think the use of that term is why I decided to give it a go.
  - acct1771 9y ago
    Love-hate relationship with marketing is probably pretty common in this crowd!
- seanwilson 9y ago
  > First impressions are that it is very quick and gives some great advice. I'm not really a web. dev. but I even I can see how this can make a good audit tool. Looks great as well. Awesome, thanks for trying it! > I suggest caution against using the term "best practice" though. It's one of my pet hates - there is good practice and there is bad practice but its a brave person who claims to know best practice. Yes, I'll admit I don't love the phrase myself. I don't like when someone is pushing what's really just their opinion as the one and only way to do things but the phrase does get the point across when you've got a character limit. I've been really careful with the rules in Checkbot so far and the guide (https://www.checkbot.io/guide/ https://www.checkbot.io/guide/) has links to where web experts like Google, Mozilla, OWASP and W3C recommend each practice. For example, making sure every page has a title, HTTPS is enabled and compression is used is difficult to argue against. Let me know if you think any of the rules need some changes however.
scrollaway 9y ago
Really cool stuff. Here's some initial feedback: "Avoid internal link redirects" -> All the errors I'm getting on my site are due to the login-wall on some of the pages because it's detecting /account/login/?next=... links as internal redirects. "Use unique titles" / "Set page descriptions" / "Avoid URL parameters" / "Avoid thin content pages" / etc -> Same problem as above with login walls ("Sign in to ..."). I get why, but it's adding a ton of noise. I added /account/login to URLs to ignore but it didn't achieve anything, I'm guessing I must have misunderstood the syntax or it has to be an exact match or some such? "Avoid inline JavaScript" -> I'm making quite a bit of use of the <script type="application/json" id="foo"/> pattern, which allows me to declare json objects in the body I can later parse in my scripts. This pattern doesn't have all the issues carried with inline js. Can you ignore script tags where the type= is unknown or application/json\? hsts preload: This is picking up individual pages on the checked site as errors, even though hsts preload really is a domain-wide thing. "Hide server version data" -> This is picking up "server: cloudflare" as an error. Means no site behind cloudflare will ever pass this which seems overkill. "Use lowercase URLs" -> So, on my site you can access objects with IDs like youtube's (/id/vZEz7JoNnfgVo...). It's picking up all those as errors. Feels wrong? UI: Not a fan of the "x inlinks / y outlinks / headers / recrawl / html / copy" links below the URLs on the results page. Low contrast, unclear what I'm clicking and where it's gonna take me. The "copy" button: What am I copying? (Clearly the URL as I tried it, but that'd be more useful as a clipboard button next to the URL for example) Finally, I ran it on my company's blog and it ended up crawling a ton of the company's various exosites on different domains which wasn't super useful, especially since none of it showed up in the final results. Hey, this is a really great tool. Fast, slick UI and very clear what it does. I'll keep an eye on it and would love to see what else it can do in the future. Pricing: It's hard to see myself pay for this; not because it's not worth it (I think it is easily worth a dozen USD / site checked), but because it's so easy to look at what it does and think "Yeah, but, I can probably do all that myself, and if I don't, it's not so important that I need to pay for a tool to tell me what to fix". I think this is the curse of developing products targeted at developers: Devs will tend to think "I can do this myself // I don't need this". In fact, if you hook me up with a free account, I'll use it a bunch ;) Shoot me an email (see profile) if you want to talk through some more feedback (especially UX feedback). You just provided me with a pretty cool service for free so I feel I have to give back :)
wpasc 9y ago
Awesome
santoshmaharshi 9y ago
Very nicely thought through and implemented i agree with you on the best practice point. Already forwarded to few folks and going to stay with my browser.
- seanwilson 9y ago
  Thanks! Let me know if you can think of any improvements I can make.
rambojazz 9y ago
Is there a version that works on Firefox?
mkorsak 9y ago
Seems really cool, I like this. One issue I'm having so far though is that after crawling my site it found one 404, and the link is (my domain)/page-not-found-test. It also says there's 0 inlinks to it, so I have no idea where it's getting the idea for this page from. It doesn't exist, but I've never linked to anything like it?

83 points by seanwilson 9y ago

Hi, I didn't get much feedback when I posted last time so I'm giving it another try. This is aimed at helping web developers follow SEO, performance and security best practices so I'd love to know what the community thinks. Can you think of any changes that would make Checkbot more helpful? Did you notice any bugs? Thanks!

cdawg0 9y ago
Cool, I'm checking it out right now. Can you share what your tool provides that others don't?
- seanwilson 9y ago
  The major one is that instead of manually checking pages one at a time, Checkbot lets you easily test 1,000s of pages in a few minutes to root out issues you'd normally miss. As you're doing web crawls from your own machine, you can also crawl any site you want as often as you want including localhost/development, staging and production sites. This lets you identify issues early and confirm fixes during development before problems go live.
  - ajeet_dhaliwal 9y ago
    Who’s the customer you have in mind who will want to do this for thousands of pages?
    - seanwilson 9y ago
      People involved in web development, SEO and marketing. It doesn't need to be in the thousands but checking large number of pages for problems has obvious productivity benefits over checking one page at a time. It also lets you identify issues that impact multiple pages like duplicate titles, descriptions and content that you'd normally miss.
  - cdawg0 9y ago
    Interesting. So the intent is to tackle problems before deployment. Do you plan on any devtools integrations so it can be used as part of an automatic CI/CD process? Also, does Checkbot dig into all dependancies or skip them like some others?
    - seanwilson 9y ago
      > Interesting. So the intent is to tackle problems before deployment. Do you plan on any devtools integrations so it can be used as part of an automatic CI/CD process? A lot of the time, websites owners don't know there's a problem until their search results or Google Search Console updates. So I'm seeing it being used by developers to check localhost/development sites, then on staging for other problems, then on production when changes are made there. A command-line version to support CI/CD is something I'm really interested in as well. > Also, does Checkbot dig into all dependancies or skip them like some others? Can you expand on what you mean here?
  - idle_processor 9y ago
    > The major one is that instead of manually checking pages one at a time, Checkbot lets you easily test 1,000s of pages in a few minutes to root out issues you'd normally miss. ScreamingFrog does this too, but it's still nice that your work is out there to add to the suite.
idle_processor 9y ago
- A regex blacklist for URL structure would be very helpful. - Acceptable title length seems a bit short. 70 characters seems closer to what we're allowed today than the old 60 (per http://www.bigleap.com/blog/5-tips-take-advantage-googles-new-meta-title-guidelines-2016/ http://www.bigleap.com/blog/5-tips-take-advantage-googles-ne...). Might also be worth segmenting URLs with query parameters in them into a low priority batch to check later (or skip). (E.g., when spidering a WordPress site, the crawler wastes time on .../article/?replytocom=* URLs. URL filtering solves this, somewhat, but it might require multiple passes to identify all of the problematic query strings.)

ramon 9y ago

Hi, I have downloaded already but didn't test it yet. I will test as soon as I can.

mahesh_rm 9y ago

I rapidly tested it. It's staying on my chrome. Good Job.

seanwilson 9y ago
Thanks, let me know how you get on!

queezey 9y ago

Very cool. I noticed that it's mangling some of my URLs, though. `/!0ead1aEq` is getting turned into `/%210ead1aEq` (the exclamation point is getting percent-encoded), which leads to a bunch of spurious 404 errors. https://tools.ietf.org/html/rfc3986#section-3.3 https://tools.ietf.org/html/rfc3986#section-3.3

seanwilson 9y ago
Ah, thanks a good bug report! I'll get this fixed.
seanwilson 9y ago
Hmm, so when crawling, URLs are normalised and the URL library I'm using is normalising the "!" to "%21". Could you send me a working URL to test on? My email is sw@seanw.org if you want to use that.

Raphmedia 9y ago

Works pretty good. Incidentally, I just locked myself from my own server. This double as a security tester! :o)

seanwilson 9y ago
On the left sidebar at the start you can modify the number of URLs crawled per second if that helps!

ratata 9y ago

Nice. very similar to Lighthouse https://developers.google.com/web/tools/lighthouse/#devtools https://developers.google.com/web/tools/lighthouse/#devtools

kaycebasques 9y ago
Which is also available in Chrome DevTools via the Audits panel.
- PudgePacket 9y ago
  I've never been able to use that, every time I run the test it says initial page load is 10s, but I'm running it against a static website hosted locally.. how is it taking so long? Even if I visited my site from the other side of the world over 3g it would be faster than lighthouse accessing it locally.
  - pimlottc 9y ago
    Part of the audit includes throttling your connection to simulate a device on a slower link. https://github.com/GoogleChrome/lighthouse/blob/master/docs/throttling.md https://github.com/GoogleChrome/lighthouse/blob/master/docs/...
    - PudgePacket 9y ago
      Still couldn't account for 10s for 5kb of html, something must have been really wrong, maybe I configured it incorrectly.
wgjordan 9y ago
Similar feature-set at first glance, but not open-source, not free after beta ends, and created/maintained by an unknown solo developer. Nice site design, but I think the cards are pretty heavily stacked against this one.
- 9y ago

chasers 9y ago

It's nice and fast! Are you actually using chrome to render each page or making requests some other way?

seanwilson 9y ago
Thanks! All requests are done from your own browser if that's what you mean.
- chasers 9y ago
  No, I mean like render the whole page with JS and all, but from your FAQ it seems like you're not. Which is why it's fast, ha.

gerdesj 9y ago

Very cool. I'm giving it a quick go whilst doing yet another patchathon on customer systems. First impressions are that it is very quick and gives some great advice. I'm not really a web. dev. but I even I can see how this can make a good audit tool. Looks great as well. I suggest caution against using the term "best practice" though. It's one of my pet hates - there is good practice and there is bad practice but its a brave person who claims to know best practice. I think my hatred of that term stems from seeing it plastered all over older MS docs and the usual crop of "me too" copy n paste blog postings that litter the web, not to mention various forum postings. We're all bloody experts who know best in this game 8)

keithnz 9y ago
I agree, it runs nicely, and is nicely presented. The term best practice does irk me. But weirdly enough I think the use of that term is why I decided to give it a go.
- acct1771 9y ago
  Love-hate relationship with marketing is probably pretty common in this crowd!
seanwilson 9y ago
> First impressions are that it is very quick and gives some great advice. I'm not really a web. dev. but I even I can see how this can make a good audit tool. Looks great as well. Awesome, thanks for trying it! > I suggest caution against using the term "best practice" though. It's one of my pet hates - there is good practice and there is bad practice but its a brave person who claims to know best practice. Yes, I'll admit I don't love the phrase myself. I don't like when someone is pushing what's really just their opinion as the one and only way to do things but the phrase does get the point across when you've got a character limit. I've been really careful with the rules in Checkbot so far and the guide (https://www.checkbot.io/guide/ https://www.checkbot.io/guide/) has links to where web experts like Google, Mozilla, OWASP and W3C recommend each practice. For example, making sure every page has a title, HTTPS is enabled and compression is used is difficult to argue against. Let me know if you think any of the rules need some changes however.

scrollaway 9y ago

Really cool stuff. Here's some initial feedback: "Avoid internal link redirects" -> All the errors I'm getting on my site are due to the login-wall on some of the pages because it's detecting /account/login/?next=... links as internal redirects. "Use unique titles" / "Set page descriptions" / "Avoid URL parameters" / "Avoid thin content pages" / etc -> Same problem as above with login walls ("Sign in to ..."). I get why, but it's adding a ton of noise. I added /account/login to URLs to ignore but it didn't achieve anything, I'm guessing I must have misunderstood the syntax or it has to be an exact match or some such? "Avoid inline JavaScript" -> I'm making quite a bit of use of the <script type="application/json" id="foo"/> pattern, which allows me to declare json objects in the body I can later parse in my scripts. This pattern doesn't have all the issues carried with inline js. Can you ignore script tags where the type= is unknown or application/json\? hsts preload: This is picking up individual pages on the checked site as errors, even though hsts preload really is a domain-wide thing. "Hide server version data" -> This is picking up "server: cloudflare" as an error. Means no site behind cloudflare will ever pass this which seems overkill. "Use lowercase URLs" -> So, on my site you can access objects with IDs like youtube's (/id/vZEz7JoNnfgVo...). It's picking up all those as errors. Feels wrong? UI: Not a fan of the "x inlinks / y outlinks / headers / recrawl / html / copy" links below the URLs on the results page. Low contrast, unclear what I'm clicking and where it's gonna take me. The "copy" button: What am I copying? (Clearly the URL as I tried it, but that'd be more useful as a clipboard button next to the URL for example) Finally, I ran it on my company's blog and it ended up crawling a ton of the company's various exosites on different domains which wasn't super useful, especially since none of it showed up in the final results. Hey, this is a really great tool. Fast, slick UI and very clear what it does. I'll keep an eye on it and would love to see what else it can do in the future. Pricing: It's hard to see myself pay for this; not because it's not worth it (I think it is easily worth a dozen USD / site checked), but because it's so easy to look at what it does and think "Yeah, but, I can probably do all that myself, and if I don't, it's not so important that I need to pay for a tool to tell me what to fix". I think this is the curse of developing products targeted at developers: Devs will tend to think "I can do this myself // I don't need this". In fact, if you hook me up with a free account, I'll use it a bunch ;) Shoot me an email (see profile) if you want to talk through some more feedback (especially UX feedback). You just provided me with a pretty cool service for free so I feel I have to give back :)

wpasc 9y ago

Awesome

santoshmaharshi 9y ago

Very nicely thought through and implemented i agree with you on the best practice point. Already forwarded to few folks and going to stay with my browser.

seanwilson 9y ago
Thanks! Let me know if you can think of any improvements I can make.

rambojazz 9y ago

Is there a version that works on Firefox?

mkorsak 9y ago

Seems really cool, I like this. One issue I'm having so far though is that after crawling my site it found one 404, and the link is (my domain)/page-not-found-test. It also says there's 0 inlinks to it, so I have no idea where it's getting the idea for this page from. It doesn't exist, but I've never linked to anything like it?