7 ms·
Fast website link checker in Go
- gtirloni 8y agoPretty nice tool that Just Works (tm). Although the concurrency level (512 connections) is a bit too aggressive for most servers. You'll get throttled, blocked or your backend will crash (which isn't too bad in any case, except that might not be what you were after with a link checker).
- raviqqe42 8y agoActually, I totally agree with you. I decided the number based on the default maximum number of open files on Linux because I was not sure about common limits of concurrent connections between clients and HTTP servers. Or, it should probably regulate numbers of requests per second sent to the same hosts. If someone suggests other options, I would adopt it.
- gtirloni 8y agoI think your approach is fine for local dev servers. Maybe just introduce something like an exponential backoff algorithm if you start getting too many 5xx errors or the requests are hanging.
- anonacct37 8y agoA good default might be based on current browser behavior. Keep in mind http2 might make everything use one connection but allow 100 concurrent requests.
- jlgaddis 8y agoBoth Chrome and Firefox limited the number of connections to a server to six (6), if memory serves. I'm not certain if those limits have changed or if the number is different based on HTTP 1.1 versus HTTP 2.
- brobinson 8y agoThe limit is per _hostname_ not per _server_ (unless things have changed in the last 10 years). This is why you'll see assets1.foo.com, assets2.foo.com, etc. all pointing to the same IP address(es). Server-side code picks one based on a modulus of a hash of the filename or something similar when rendering the HTML to get additional pipelines in the user's browser. Not sure how or if this is done in SPA.
- jchw 8y agoI find the use of rake to be kind of unorthodox, and yet I don't know what else you'd use in the Go world, other than maybe just Makefiles. Any particular reason to choose Rake? It's probably not easy to get it running on Windows based on my experience playing with Rails on Windows. Other than that it looks quite useful, and it's definitely something to keep in the tool belt. Bonus points for the subtle Undertale references too :)
- reificator 8y agoI find it helpful when CI providers run Go tooling by default, except if there's a makefile in which case they run that instead.
- mjk7841 8y agoI'm also curious about this. It seems that many go developers out there are using Makefiles. Makefiles are a good solution for golang projects in some cases, but I've seen a lot of people really abusing Makefiles and trying to use them for more generic task running. In a past life, we used invoke [1] for task running. It was incredible but has the same problem as rake: it introduces another language (Python) and more dependencies. There's a fairly new task runner being developed in go called mage [2], but it didn't seem worth the jump yet to me as it's still pretty immature (I haven't played with it in a few months, though). Did you consider trying that out? [1] https://github.com/pyinvoke/invoke https://github.com/pyinvoke/invoke [2] https://github.com/magefile/mage https://github.com/magefile/mage
- anonfunction 8y agoI like using make even for basic task running. It’s usually installed where I’m working and is familiar. I don’t want to install rake or mage.
- arbie 8y agoMake gets a bad rap but I've seen it being used for susbtantially complex workflows. If you wrap your commands in something that can reattach to running processes and use dependencies correctly, it is hard to displace.
- fwip 8y agoLooks rad. Is there any plan to add authorization headers, so that I can test a site as a particular user?
- raviqqe42 8y agoThank you for your feedback! Can you open an issue on the GitHub repository? I would appreciate it if you add some concrete use cases as then it'll be clear what kinds of options need to be implemented.
- jpsim 8y agoAnyone know of an equivalent for Ruby? I’d love to add this as an integration test for jazzy[0]. Separately, it’d be awesome if this also checked that anchor links resolve to id values to validate linking to specific elements on a page. [0]: https://github.com/realm/jazzy https://github.com/realm/jazzy
- jamietanna 8y agohttps://github.com/gjtorikian/html-proofer https://github.com/gjtorikian/html-proofer is what I use
- peterwwillis 8y agoSo, I'll bite. Why user this and not wget, curl, or any other http spider?
- pknopf 8y agoIs there a good html parser that you can pipe into/out of? Using Go, it would be easier to parse the html, and tell the difference between <p>http://somewhere.com/</p> http://somewhere.com/</p> and <a href="http://somewhere.com/">test</p> http://somewhere.com/">test</p>
- laumars 8y agoPersonally I don't care about the difference between your two examples because a published URL is still a published URL regardless of whether it is a hyperlink or not. However where I do care about the difference between an anchor and a paragraph block is with relative links or other URLs without the protocol prefix as those are harder to programmatically guess what is a web URL and what is an example file system path in a technical document (for example). In an ideal world the JavaScript, CSS and any web-APIs (eg JSON returns) would be executed locally to check any modern way of abstracting away URLs (page redirection et al). But that's not to say there isn't a place for a less sophisticated parser (though I would say that as I've also written a link checker similar to the one posted hehehe).
- ryanlol 8y agoBecause sometimes you care about performance? Although, based on a quick look at the code this thing isn't going to go particularly fast.
- peterwwillis 8y agoLike, extreme performance, or just parallelism? One example of parallelism: xargs -a urls.txt -n 5 -P 20 wget -nv --spider -T 10 -e robots=off. This will run up to 20 processes with 5 URLs each. It's not "efficient" but it's faster than nothing, and you get the whole feature set of Wget. For more customizeable spidering, Scrapy allows you to customize a spider, and even deploy spider daemons to run in production (https://doc.scrapy.org/en/latest/topics/deploy.html https://doc.scrapy.org/en/latest/topics/deploy.html). For an out-of-the-box version, try Spidy (https://github.com/rivermont/spidy https://github.com/rivermont/spidy). For super serious spidering, try Heritrix (https://webarchive.jira.com/wiki/spaces/Heritrix/overview https://webarchive.jira.com/wiki/spaces/Heritrix/overview) or Nutch (https://nutch.apache.org/ https://nutch.apache.org/). Here's an interesting read on crawling a quarter billion pages in 40 hours: http://www.michaelnielsen.org/ddi/how-to-crawl-a-quarter-billion-webpages-in-40-hours/ http://www.michaelnielsen.org/ddi/how-to-crawl-a-quarter-bil... From my own experience crawling massive dynamic state-driven websites, even if you're trying to just grab a single page, you will eventually want the extra features.
- zdw 8y agoHow fast is this? Is there even a common benchmark for this sort of thing? How does it compare to the older python "linkchecker", which was resurrected here (and is quite fast): https://github.com/linkchecker/linkchecker https://github.com/linkchecker/linkchecker
- chrismorgan 8y agoI have found linkchecker to be unreasonably slow by default, very fragile if you try to make it any faster (e.g. occasional socket errors with almost any concurrency, regardless of the purported ulimits, in a way that suggested some kind of socket leaking to me at the time), and possessing fairly bad reporting. Also, on Windows, being built with ancient Python meant it didn’t support SNI, so I had to delve into it to figure out a way of turning TLS verification off to make it work pretty much at all, which also hints at its generally poor configurability and surprisingly poor documentation (given the fact that there is actually a moderate amount of it). I still use linkchecker (because I’ve never completed my Rust-based link checker that I started several years ago), and have extended it at work to support client-side certificates which we use on CI, but I’m generally fairly unimpressed with linkchecker.
- omeid2 8y agoA comprehensive prompt is an extremely useful and an underappreciated productivity booster. But please, when making screencast or screenshots, use a simple prompt. Because the information included in the prompt is often not only irrelevant, it can be actively distracting and makes reading through less pleasant by increasing the required eye-movement and cognitive effort to filter out the useless content. Small little things makes a difference.
- jakear 8y agoThe prompt is so long that you can’t actually see the command being typed on mobile, asciienema just clips the stream.
- JasonFruit 8y agoYou mean `asciinema`. That, my friend, is one unfortunate typo.
- jakear 8y agoAhh. That makes more sense. I took it as a bit of (very) crude humor.
- chrismorgan 8y agoI wrote a script bash-for-recording for myself a few years back, for invocation by termrec, which set the window size to 80×24, set HOME to a new empty directory, set USER and NAME to dummy values (creating an actual new throwaway user account would be better here, but I was lazy), set TERM to xterm-256color (I think it was), cleared the environment (env -i) and possibly one or two other things, set a deliberately very simple and obvious prompt (which sets the title as well), cleared the screen, then finally started a nice clean bash profile. I should pull out my old laptop or my backups and resuscitate the script. The example at http://tty-player.chrismorgan.info/ http://tty-player.chrismorgan.info/ was generated using that script.
- NetOpWibby 8y ago
- andyonthewings 8y agoI have been using a nodejs equivalent, named broken-link-checker https://github.com/stevenvachon/broken-link-checker https://github.com/stevenvachon/broken-link-checker I run it in a CI job which builds a static site. It works pretty well for me.
- thamizhan2611 8y agoAwesome tool. I have one small request, is it possible to put an architecture diagram connecting different moving parts like parsing, fetcher, daemon etc. IMHO this might be useful for people who are trying to go through the source code to understand how the tool functions. Thanks anyways :)
- bryanrasmussen 8y agoI don't know Go, but looking at the code it doesn't look like it handles sites that are rendered on the client, if so it has limited utility in today's web.
- OoooooooO 8y agoOf course it does not, for that you need a headless browser ( = Chrome Headless). Neither Python/Go/Java/Rust/C/C++/D/Ruby/C#/F#/... nor any other language (not even Node afaik, but Puppeteer handles that) can handle Javascript rendered sites.
- gruzh 8y agopylinkvalidator seems to be a pretty good python alternative for this project. It's also very fast, scans 700+ URLs in under a minute https://github.com/bartdag/pylinkvalidator https://github.com/bartdag/pylinkvalidator
- jwilk 8y agoAre there any good checkers for URLs in text files? I wrote https://github.com/jwilk/urlycue https://github.com/jwilk/urlycue , but I'm not quite happy about it, I don't have energy to improve it either; so I'm looking for alternatives.
- oneeyedpigeon 8y agoI guess one of the bigger challenges when it comes to unstructured data is identifying URLs. Is there a canonical way of identifying a URL embedded in text? Is it an impossible problem?
- ethhics 8y agoPerhaps I’m missing the question, but there is a regular expression for matching a URI. Remove the leading carat and it can match anywhere in a text. https://tools.ietf.org/html/rfc3986#appendix-B https://tools.ietf.org/html/rfc3986#appendix-B Edit: I see it's not quite that simple. However, I still think that with some stricter matching requirements this could work.
- jwilk 8y agoThis regexp lets you parse a valid URI, but it matches also a lot things at are not URIs at all. The URI language is of course regular, so it would be possible to construct a regexp that matches only URIs. But naively applying such regexp wouldn't work in practice, because many punctuation characters are allowed in URIs. For example, single quotes are allowed, so in this Python code the regexp would match too much: homepage = 'http://example.com/'
- ethhics 8y agoI see—The standard allows most characters we normally use to surround URIs. It sure does look like a difficult problem then, and one that a regexp can't solve.