5 ms·
Awesome, thanks for the detailed feedback! As you can probably imagine, tweaking the rules to work with every imaginable website configuration is an ongoing pro
by seanwilson 9y ago
Awesome, thanks for the detailed feedback! As you can probably imagine, tweaking the rules to work with every imaginable website configuration is an ongoing process so this is super helpful.
> All the errors I'm getting on my site are due to the login-wall on some of the pages because it's detecting /account/login/?next=... links as internal redirects.
> "Use unique titles" / "Set page descriptions" / "Avoid URL parameters" / "Avoid thin content pages" / etc
Allowing Checkbot to login could help but I'll look into how to improve this.
> "Avoid inline JavaScript" -> I'm making quite a bit of use of the <script type="application/json" id="foo"/>
Ah, thanks, this is an easy fix. I'm planning to add structured data checks in the future as well because checking you've configured these correctly on all your pages is cumbersome.
> "Hide server version data" -> This is picking up "server: cloudflare" as an error. Means no site behind cloudflare will ever pass this which seems overkill.
Yes, for what it's worth this is defined as "low priority" internally and the rule description is written to emphasise this. I could change it to only fire when there's version numbers in the headers perhaps. I agree knowing you're using Cloudflare isn't a big deal but some servers will advertise very specific OS and PHP versions for example.
> "Use lowercase URLs" -> So, on my site you can access objects with IDs like youtube's (/id/vZEz7JoNnfgVo...). It's picking up all those as errors. Feels wrong?
Yes, I'll need to think how to avoid that case. It's a good general rule when you're writing human readable URLs however so I wouldn't want to disable it completely.
> UI: Not a fan of the "x inlinks / y outlinks / headers / recrawl / html / copy" links below the URLs on the results page. Low contrast, unclear what I'm clicking and where it's gonna take me. The "copy" button: What am I copying? (Clearly the URL as I tried it, but that'd be more useful as a clipboard button next to the URL for example)
Hmm, any more suggestions on what to change here? I made these links prominent because they were common user actions and added tooltips to them to help describe what they do. I agree it's not completely obvious what they do at first but there's only so much space available. I experimented with only showing these when you hover over a table cell. I do want to add more shortcuts in the future such as a quick way to look up a URL on Google or on archive.org so I'll likely have a "more" button for extra options later.
> Finally, I ran it on my company's blog and it ended up crawling a ton of the company's various exosites on different domains which wasn't super useful, especially since none of it showed up in the final results.
Can you give more details here? Checkbot will probe <a href="..."> links to check they're working for example but shouldn't spider sites that are considered external. I originally had it crawling subdomains of the start URL but changed that default because it wasn't what most people wanted.
> Shoot me an email (see profile) if you want to talk through some more feedback (especially UX feedback).
Great, let's keep in contact (see my profile as well)! Hopefully it's obvious UX is important to me too. It's been challenging to find a balance in showing the right amount of information on the screen while battling with the horizontal space constraints you get with long URLs. The "Avoid temporary redirects" report is a good example of this e.g. for each row, you want to know the redirect status code, the start URL, redirect destination and redirect path.
- scrollaway 9y ago> Allowing Checkbot to login could help but I'll look into how to improve this. Wouldn't help in my scenario FWIW, my site is oauth-only login. > Hmm, any more suggestions on what to change here? I move the copy link into a clipboard button next to the url (like Github's "clipboard" button next to URLs), and make the remainder of the links prominent buttons. I would also avoid taking users to a separate page when clicking any of them; rather, open a "sub view" below the url (eg a nested list). If you have less-often-used actions, you could also add a "..." menu on the right side, or next to the buttons. > Can you give more details here Try it against articles.hsreplay.net and look at all the URLs it ends up checking against, you'll see what I mean. It didn't end up spidering all of hsreplay.net, but it did go through a ton of it.
- seanwilson 9y ago> Wouldn't help in my scenario FWIW, my site is oauth-only login. Would being able to set cookies or sending custom headers help? > Try it against articles.hsreplay.net and look at all the URLs it ends up checking against, you'll see what I mean. It didn't end up spidering all of hsreplay.net, but it did go through a ton of it. Hmm, so if you check "Explore" -> "External URLs" there's a ton of external links being checked for this URL. I'm not sure what you could do here except excluding hsreplay.net URLs from being checked. Thanks for the other tips and examples, I'm actively working on this.
- scrollaway 9y ago> Would being able to set cookies or sending custom headers help? I don't think so. And if it's for SEO purposes, I don't care; these pages won't get crawled by Google anyway. I'm ok ignoring the URLs, but it'd be nice if maybe for example you detect there's a bunch of redirects to a pattern that contains "~login~" and ask the user if they want to add the login URL to the blocklist. I didn't have much success adding it myself.
- seanwilson 9y ago> these pages won't get crawled by Google anyway. Hmm, how do you indicate this to Google? I'm thinking about how you could tell Checkbot to ignore pages like this. > I didn't have much success adding it myself. The "URL patterns to ignore" setting is just a JavaScript regex string if that helps. It needs some help text at a minimum. A common scenario I see as well is you start a crawl, see the URLs flying by and think "oops, don't want to crawl those URLs". A cancel button would help but hopefully more can be done.