4 ms·

People trying to build new tools that parse the web. Try building a crawler without reusing an existing browser engine. Try running unit tests against your ow

People trying to build new tools that parse the web. Try building a crawler without reusing an existing browser engine. Try running unit tests against your own web projects with Selenium without running a headless browser. Phantom.JS gave up because they couldn’t keep up with the complexity, and Chrome headless "just works". Opera gave up on their own browser engine because of the complexity of parsing HTML5 accurately, when the spec is just "whatever Chrome does". We’ve thrown away an entire ecosystem, just for more flashy graphics.

pvg 9y ago
That's a pretty tiny 'everybody else' compared to users, web and browser developers. They seem to be doing mostly ok and I still don't follow your argument that their concerns should somehow reign supreme over those of, you know, the actual everybody else.
- kuschku 9y ago
  It’s a tiny "everybody else" because it never was given a chance to develop. Maybe you’d also say that the number of people that want to do their online banking with a desktop program that isn’t provided by their bank is a tiny "everybody else". Yet in places where OpenHBCI exists, many people use it – and there is an ecosystem around it, e.g. KMyMoney can integrate with it, and there’s a small widget for KDE to show your current account balance in your tray. If we had a machine parseable web, a similar ecosystem would have developed. When embedding links today, we use OpenGraph tags. But why? If the web was directly machine readable, we could’ve directly embedded that content. Google search shows you as preview excerpts of tables on a page, with an easily machine readable web, similar stuff could’ve happened here. Maybe instead of only embedding YouTube videos, I would have been able to easily embed any part of any web page into any other. Maybe I would have been able to easily embed any part of any web page in a desktop application. Maybe I would have been able to build addons that easily search over content of web pages, in a structured way.
  - tptacek 9y ago
    This would be a more compelling counterfactual if the approach that won wasn't in many senses the most successful technology in the history of the computer industry.
detaro 9y ago
I very much doubt Opera gave up because of "parsing HTML5". Parsing HTML5 is well-defined, certainly better than what was there before (unless you are willing to say "if you don't write perfect XML you don't get to be on the web", but good luck with that – no browser ever was and never will be in the position to do that) The vast majority of the difficulty of the web platform is in the layers above, which don't care if the DOM they are looking at came from HTML5 or XHTML: layouting/rendering, interactions of JS and DOM, ...
- annevk 9y ago
  Indeed, having worked for Opera during the time where we implemented HTML5-compliant parsing, the net result was that we fixed a bunch of site compatibility issues. Implementing it made competing with other browsers easier. (And as you say, parsing HTML is complex, but the rest of the web platform is vastly more so.)
simonw 9y ago
One of the most valuable aspects of HTML5 is that it defines a parsing model for "broken" HTML. This means tat, for the first time, it's possible to build a brand new HTML parser that has a high chance of working against all existing HTML without needing to first reverse-engineer existing browsers. Remember, when HTML5 was first designed Internet Ecplorer was by far the most widely used browser. And IE was closed source. If you wanted to build a parser you needed to first reverse engineer IE and figure out how it handles invalid HTML. The HTML5 spec fixed that. The thing you are complaining about here (HTML5 making it harder to build a new browser from scratch) is one of the things HTML5 actually solved!
coding123 9y ago
It seems like the web is 10 times more parseable. I feel like almost all content is now delivered via JSON services, and the HTML around it is constructed on the fly and is totally pointless to look at. At least that's the direction I've seen. I don't know how "everyone else" are people that are writing parsers. Seems like that would be a tiny percentage of the population, not "everyone else".

pvg 9y ago

That's a pretty tiny 'everybody else' compared to users, web and browser developers. They seem to be doing mostly ok and I still don't follow your argument that their concerns should somehow reign supreme over those of, you know, the actual everybody else.

kuschku 9y ago
It’s a tiny "everybody else" because it never was given a chance to develop. Maybe you’d also say that the number of people that want to do their online banking with a desktop program that isn’t provided by their bank is a tiny "everybody else". Yet in places where OpenHBCI exists, many people use it – and there is an ecosystem around it, e.g. KMyMoney can integrate with it, and there’s a small widget for KDE to show your current account balance in your tray. If we had a machine parseable web, a similar ecosystem would have developed. When embedding links today, we use OpenGraph tags. But why? If the web was directly machine readable, we could’ve directly embedded that content. Google search shows you as preview excerpts of tables on a page, with an easily machine readable web, similar stuff could’ve happened here. Maybe instead of only embedding YouTube videos, I would have been able to easily embed any part of any web page into any other. Maybe I would have been able to easily embed any part of any web page in a desktop application. Maybe I would have been able to build addons that easily search over content of web pages, in a structured way.
- tptacek 9y ago
  This would be a more compelling counterfactual if the approach that won wasn't in many senses the most successful technology in the history of the computer industry.

detaro 9y ago

I very much doubt Opera gave up because of "parsing HTML5". Parsing HTML5 is well-defined, certainly better than what was there before (unless you are willing to say "if you don't write perfect XML you don't get to be on the web", but good luck with that – no browser ever was and never will be in the position to do that) The vast majority of the difficulty of the web platform is in the layers above, which don't care if the DOM they are looking at came from HTML5 or XHTML: layouting/rendering, interactions of JS and DOM, ...

annevk 9y ago
Indeed, having worked for Opera during the time where we implemented HTML5-compliant parsing, the net result was that we fixed a bunch of site compatibility issues. Implementing it made competing with other browsers easier. (And as you say, parsing HTML is complex, but the rest of the web platform is vastly more so.)

simonw 9y ago

One of the most valuable aspects of HTML5 is that it defines a parsing model for "broken" HTML. This means tat, for the first time, it's possible to build a brand new HTML parser that has a high chance of working against all existing HTML without needing to first reverse-engineer existing browsers. Remember, when HTML5 was first designed Internet Ecplorer was by far the most widely used browser. And IE was closed source. If you wanted to build a parser you needed to first reverse engineer IE and figure out how it handles invalid HTML. The HTML5 spec fixed that. The thing you are complaining about here (HTML5 making it harder to build a new browser from scratch) is one of the things HTML5 actually solved!

coding123 9y ago

It seems like the web is 10 times more parseable. I feel like almost all content is now delivered via JSON services, and the HTML around it is constructed on the fly and is totally pointless to look at. At least that's the direction I've seen. I don't know how "everyone else" are people that are writing parsers. Seems like that would be a tiny percentage of the population, not "everyone else".