14 ms·

Hi, I didn't get much feedback when I posted last time so I'm giving it another try. This is aimed at helping web developers follow SEO, performance and securit

Hi, I didn't get much feedback when I posted last time so I'm giving it another try. This is aimed at helping web developers follow SEO, performance and security best practices so I'd love to know what the community thinks. Can you think of any changes that would make Checkbot more helpful? Did you notice any bugs? Thanks!

cdawg0 9y ago
Cool, I'm checking it out right now. Can you share what your tool provides that others don't?
- seanwilson 9y ago
  The major one is that instead of manually checking pages one at a time, Checkbot lets you easily test 1,000s of pages in a few minutes to root out issues you'd normally miss. As you're doing web crawls from your own machine, you can also crawl any site you want as often as you want including localhost/development, staging and production sites. This lets you identify issues early and confirm fixes during development before problems go live.
  - ajeet_dhaliwal 9y ago
    Who’s the customer you have in mind who will want to do this for thousands of pages?
    - seanwilson 9y ago
      People involved in web development, SEO and marketing. It doesn't need to be in the thousands but checking large number of pages for problems has obvious productivity benefits over checking one page at a time. It also lets you identify issues that impact multiple pages like duplicate titles, descriptions and content that you'd normally miss.
  - cdawg0 9y ago
    Interesting. So the intent is to tackle problems before deployment. Do you plan on any devtools integrations so it can be used as part of an automatic CI/CD process? Also, does Checkbot dig into all dependancies or skip them like some others?
    - seanwilson 9y ago
      > Interesting. So the intent is to tackle problems before deployment. Do you plan on any devtools integrations so it can be used as part of an automatic CI/CD process? A lot of the time, websites owners don't know there's a problem until their search results or Google Search Console updates. So I'm seeing it being used by developers to check localhost/development sites, then on staging for other problems, then on production when changes are made there. A command-line version to support CI/CD is something I'm really interested in as well. > Also, does Checkbot dig into all dependancies or skip them like some others? Can you expand on what you mean here?
  - idle_processor 9y ago
    > The major one is that instead of manually checking pages one at a time, Checkbot lets you easily test 1,000s of pages in a few minutes to root out issues you'd normally miss. ScreamingFrog does this too, but it's still nice that your work is out there to add to the suite.
idle_processor 9y ago
- A regex blacklist for URL structure would be very helpful. - Acceptable title length seems a bit short. 70 characters seems closer to what we're allowed today than the old 60 (per http://www.bigleap.com/blog/5-tips-take-advantage-googles-new-meta-title-guidelines-2016/ http://www.bigleap.com/blog/5-tips-take-advantage-googles-ne...). Might also be worth segmenting URLs with query parameters in them into a low priority batch to check later (or skip). (E.g., when spidering a WordPress site, the crawler wastes time on .../article/?replytocom=* URLs. URL filtering solves this, somewhat, but it might require multiple passes to identify all of the problematic query strings.)

cdawg0 9y ago

Cool, I'm checking it out right now. Can you share what your tool provides that others don't?

seanwilson 9y ago
The major one is that instead of manually checking pages one at a time, Checkbot lets you easily test 1,000s of pages in a few minutes to root out issues you'd normally miss. As you're doing web crawls from your own machine, you can also crawl any site you want as often as you want including localhost/development, staging and production sites. This lets you identify issues early and confirm fixes during development before problems go live.
- ajeet_dhaliwal 9y ago
  Who’s the customer you have in mind who will want to do this for thousands of pages?
  - seanwilson 9y ago
    People involved in web development, SEO and marketing. It doesn't need to be in the thousands but checking large number of pages for problems has obvious productivity benefits over checking one page at a time. It also lets you identify issues that impact multiple pages like duplicate titles, descriptions and content that you'd normally miss.
- cdawg0 9y ago
  Interesting. So the intent is to tackle problems before deployment. Do you plan on any devtools integrations so it can be used as part of an automatic CI/CD process? Also, does Checkbot dig into all dependancies or skip them like some others?
  - seanwilson 9y ago
    > Interesting. So the intent is to tackle problems before deployment. Do you plan on any devtools integrations so it can be used as part of an automatic CI/CD process? A lot of the time, websites owners don't know there's a problem until their search results or Google Search Console updates. So I'm seeing it being used by developers to check localhost/development sites, then on staging for other problems, then on production when changes are made there. A command-line version to support CI/CD is something I'm really interested in as well. > Also, does Checkbot dig into all dependancies or skip them like some others? Can you expand on what you mean here?
- idle_processor 9y ago
  > The major one is that instead of manually checking pages one at a time, Checkbot lets you easily test 1,000s of pages in a few minutes to root out issues you'd normally miss. ScreamingFrog does this too, but it's still nice that your work is out there to add to the suite.

idle_processor 9y ago

- A regex blacklist for URL structure would be very helpful. - Acceptable title length seems a bit short. 70 characters seems closer to what we're allowed today than the old 60 (per http://www.bigleap.com/blog/5-tips-take-advantage-googles-new-meta-title-guidelines-2016/ http://www.bigleap.com/blog/5-tips-take-advantage-googles-ne...). Might also be worth segmenting URLs with query parameters in them into a low priority batch to check later (or skip). (E.g., when spidering a WordPress site, the crawler wastes time on .../article/?replytocom=* URLs. URL filtering solves this, somewhat, but it might require multiple passes to identify all of the problematic query strings.)