4 ms·

True, and the suggestions in the article are pretty tame. But let’s set aside the article and talk about invalid HTML for a minute. I write scrapers a lot (not

by oefrha 7y ago

True, and the suggestions in the article are pretty tame. But let’s set aside the article and talk about invalid HTML for a minute. I write scrapers a lot (not the irresponsible kind and never for monetary gains) and invalid HTML, while technically parseable and to spec, are often a pain in the ass. You have to bring in a full blown HTML5 parser, and they could be way slower (e.g. lxml.html vs html5lib). Depending on your language of choice there might not even be a to-spec HTML5 parser available. So, just close your damn tags (except self-closing ones), and close them in order, it’s not hard, the size increase is minimal, it will help with your own sanity and people will thank you for it.

thinkloop 7y ago
For many sites this is an endorsement not to close tags.
- oefrha 7y ago
  It’s an annoyance, not a meaningful hindrance.
goto11 7y ago
There are a lot more web sites than scrapers though. Why push the work to all the web sites in the world instead of fixing your scraper? But in any case, you are never getting all the web sites in the world to validate, so what does it matter if you can get some of them to validate? You still have to handle the rest.
- oefrha 7y ago
  "it will help with your own sanity" Writing unclosed/out-of-order tag soup is extra work, not less work. What a weird hill to die on.
  - goto11 7y ago
    What people don't get about the web is that there is a significant (and growing) number of pages written by people who are now dead. So all the well-intentioned ideas about "cleaning up" html - it's not going to happen. It's not a question about being pro or con closing tags or which is "best". The question is about the billions of pages which already exist and is not going to change. If you write a web scraper, that is just the reality you work with.
    - oefrha 7y ago
      Where did I call for people to clean up existing HTML? Nowhere. Did I ask people to not follow bad advice (intentionally not closing non-self-closing tags just for the hell of it)? Absolutely.
deleted 7y ago
[deleted]
chrismorgan 7y ago
I extremely strongly disagree with your points and conclusion here: why should I, a web author, care because you’re using a slow language that doesn’t have a real HTML parser available? You want to parse my HTML, use an HTML parser. Parsing an HTML document without a correct HTML parser is just wrong. (I decline to call it an HTML 5 parser, because it’s not strictly that any more, and this stuff is from more than ten years ago: get with the program!) I’d lump it in with using regular expressions to parse HTML: useful in certain situations, but not wise for the general case. So you have a fast but wrong HTML parser? Find a correct HTML parser. If lxml.html doesn’t parse correctly (I don’t know whether it does or not, I haven’t checked—but suppose it doesn’t), then I maintain that you should never under any circumstances use it in new code. It’s an artefact of twelve years ago, and it’s bad. So you have a correct but slow HTML parser, and you’re not happy with that? I could say “if you care about performance why are you using Python anyway”, but that would be disingenuous—though some of those sentiments can still reasonably apply. Anyway, your remedy is to find a fast and correct one. I confess myself a tad surprised to find no stable or featureful bindings to html5ever, which is one of the best things along these lines. https://github.com/SimonSapin/html5ever-python https://github.com/SimonSapin/html5ever-python and https://pypi.org/project/htmlpyever/ https://pypi.org/project/htmlpyever/ are two things that at least start on this, but it looks like nothing interesting’s happened in this space for about three years. Huh. But anyway: I decline to adjust my authoring practices because you refuse to use the right tools. Enough other people use the right tools that I don’t need to care. It’s rather like the XHTML/HTML situation if you squint: the only reason to use XHTML, which would reject an invalid document (while HTML would let your nominally invalid document do something useful), would be if something you were interacting with required it.
- oefrha 7y ago
  If your authoring practice is producing wrongly closed tag soup that happens to work (making sure it happens to work takes longer than writing it correctly in the first place), making it unnecessarily harder for everyone including you and your coworkers, then you simply suck as a developer. Of course I'm not paying you, so whatever.
  - 7y ago

by oefrha 7y ago

thinkloop 7y ago

For many sites this is an endorsement not to close tags.

oefrha 7y ago
It’s an annoyance, not a meaningful hindrance.

goto11 7y ago

There are a lot more web sites than scrapers though. Why push the work to all the web sites in the world instead of fixing your scraper? But in any case, you are never getting all the web sites in the world to validate, so what does it matter if you can get some of them to validate? You still have to handle the rest.

oefrha 7y ago
"it will help with your own sanity" Writing unclosed/out-of-order tag soup is extra work, not less work. What a weird hill to die on.
- goto11 7y ago
  What people don't get about the web is that there is a significant (and growing) number of pages written by people who are now dead. So all the well-intentioned ideas about "cleaning up" html - it's not going to happen. It's not a question about being pro or con closing tags or which is "best". The question is about the billions of pages which already exist and is not going to change. If you write a web scraper, that is just the reality you work with.
  - oefrha 7y ago
    Where did I call for people to clean up existing HTML? Nowhere. Did I ask people to not follow bad advice (intentionally not closing non-self-closing tags just for the hell of it)? Absolutely.

deleted 7y ago

[deleted]

chrismorgan 7y ago

I extremely strongly disagree with your points and conclusion here: why should I, a web author, care because you’re using a slow language that doesn’t have a real HTML parser available? You want to parse my HTML, use an HTML parser. Parsing an HTML document without a correct HTML parser is just wrong. (I decline to call it an HTML 5 parser, because it’s not strictly that any more, and this stuff is from more than ten years ago: get with the program!) I’d lump it in with using regular expressions to parse HTML: useful in certain situations, but not wise for the general case. So you have a fast but wrong HTML parser? Find a correct HTML parser. If lxml.html doesn’t parse correctly (I don’t know whether it does or not, I haven’t checked—but suppose it doesn’t), then I maintain that you should never under any circumstances use it in new code. It’s an artefact of twelve years ago, and it’s bad. So you have a correct but slow HTML parser, and you’re not happy with that? I could say “if you care about performance why are you using Python anyway”, but that would be disingenuous—though some of those sentiments can still reasonably apply. Anyway, your remedy is to find a fast and correct one. I confess myself a tad surprised to find no stable or featureful bindings to html5ever, which is one of the best things along these lines. https://github.com/SimonSapin/html5ever-python https://github.com/SimonSapin/html5ever-python and https://pypi.org/project/htmlpyever/ https://pypi.org/project/htmlpyever/ are two things that at least start on this, but it looks like nothing interesting’s happened in this space for about three years. Huh. But anyway: I decline to adjust my authoring practices because you refuse to use the right tools. Enough other people use the right tools that I don’t need to care. It’s rather like the XHTML/HTML situation if you squint: the only reason to use XHTML, which would reject an invalid document (while HTML would let your nominally invalid document do something useful), would be if something you were interacting with required it.

oefrha 7y ago
If your authoring practice is producing wrongly closed tag soup that happens to work (making sure it happens to work takes longer than writing it correctly in the first place), making it unnecessarily harder for everyone including you and your coworkers, then you simply suck as a developer. Of course I'm not paying you, so whatever.
- 7y ago