5 ms·

"More and more webpages are requiring JS even to display anything beyond a blank page." Can you provide some example webpages so we can take a look? Also, I a

"More and more webpages are requiring JS even to display anything beyond a blank page." Can you provide some example webpages so we can take a look? Also, I agree with @asciimoo's point about endpoints. One could make an argument that compared to the website design of the 1990's and 2000's, retrieving structured data from websites is actually getting easier not more difficult. I recall one period of time where the trend was to design websites entirely in Macromedia/Adobe Flash. Here is what is missing from this project (and many others like it): when providing software that performs text processing one needs to not only provide example code but also example output. This enables a user to quickly compare her current text-processing solution with the software being provided without having to install, review and run the unfamiliar software. For example, without some sample output she cannot test, e.g., whether her current solution produces the same output faster or using less code.

afandian 9y ago
With a minute's searching, here's a webpage that's meaningless without JavaScript: https://blogger.googleblog.com/ https://blogger.googleblog.com/ It's arguably a page of text not a web app that needs user interaction. I'm not trying to start an argument about JS: the consensus on HN seems to be that if you don't execute JavaScript you don't deserve to read webpages. I'm just saying that your website's clients are more diverse than 'normal' well-sighted humans. There may be machines reading the site, for all kinds of reasons. And regarding endpoints. One could make the argument that with AJAX we now have richer APIs. I disagree. We have a well-understood API for getting hypertext (HTTP) in a well-understood format (HTML) that works/worked across all websites. Replacing that with a custom-built API for every website isn't apples-for-apples.
- feelin_googley 9y ago
 The reason I asked is that I rarely use a graphical browser so I am always curious about wesbites that are inaccessible without Javascript. Blogger has a feed. Here is one way to retrieve it, in two steps. 1. get TargetBlogID 2. retrieve data optional: 3. format data for viewing Example: 1. x=$(exec curl https://blogger.googleblog.com \ |exec sed ' s/\\046/\&/g; s/\\46/\&/g; s/\\075/=/g; s/\\75/=/g; /targetBlogID/!d; s/.*targetBlogID=//; s/&.*//; '); 2. curl -o y.htm https://www.blogger.com/feeds/$x/posts/default 3. exec sed ' # ^M is "\r" s/^[0-9a-f]*^M//; s/</</g; s/>/>/g; s/&/\&/g; s/"/\"/g; 1i\ s/<name>/ name &/g; s/<uri>/ uri &/g; s/<generator>/ generator &/g; s/Blogger//; s/<id>/ id &/g; s/<published>/ published &/g; s/<email>/ email &/g; s/<title type=.text.>/ &/g; s/<openSearch:totalResults>/ total results &/g; s/<openSearch:startIndex>/ start index &/g; s/<openSearch:itemsPerPage>/ items per page &/g; s/<updated>/ updated &/g; s/<thr:total>/ thr:total &/g; s/<\/feed>/& /; s/^M*/ /; ' y.htm \ |exec tr -cd '\12\40-\176'
 - always_good 9y ago
 It's almost like you asked that question to bait someone into giving you an excuse to show "just how easy it is" with some cli hacking.
 - dchuk 9y ago
 Just 3 easy steps
faitswulff 9y ago
Feedly, Netflix, Google Maps, as of this article: https://sonniesedge.co.uk/blog/a-day-without-javascript https://sonniesedge.co.uk/blog/a-day-without-javascript
- feelin_googley 9y ago
  I remember that article. I do not use any of those three websites so I would be curious to know some specific examples of what the desired structured data looks like. If for Netflix the desired data is only DVD summaries, I recall that these were easily retrievable without Javascript not too long ago.
  - afandian 9y ago
    I would argue that it's reasonable for these examples to require JS. My complaint above was the accessibility text-only sites.
    - feelin_googley 9y ago
      I would agree. That article had a very poor choice of websites, IMO. The web is quite usable without Javascript IME.
dithering 9y ago
I know of a site where all the data is embedded inline ("var Person = {firstName: 'Fred', lastName: 'Smith' }") then rendered to HTML with JS. No endpoints, Javascript not JSON, complete pain in the ass to extract without rendering in a headless browser first.
- sitepodmatt 9y ago
  Isn't this better? Why do you need it render when they're giving you basically JSON of the data (might need to quote the keys in strict parsers), just extract that.
  - dithering 9y ago
    The short answer is that, yes, the only parser available to me is that strict. If I'm going to jump languages, I might as well go the whole hog and jump to Puppeteer.
slezyr 9y ago
> Can you provide some example webpages so we can take a look? learnopengl.com
- feelin_googley 9y ago
  curl -d 'content_name=Heading' https://example.com/content_load.php https://example.com/content_load.php? curl -d 'content_name=Heading/Subheading' https://example.com/content_load.php https://example.com/content_load.php?
  - feelin_googley 9y ago
    s/example/learnopengl/ s/Heading/Introduction/
Animats 9y ago
Wix. Everything with Wix.[1] [1] http://www.wix.com/stunningwebsites/400 http://www.wix.com/stunningwebsites/400
- feelin_googley 9y ago
 I looked at some exemplary Wix sites since I rarely ever encounter these. The ones they identify as "stunning" seem to be mostly devoid of text and are mainly images and videos. As others in this thread have noted is common on today's web, they use json endpoints. Some sites also have an optional XML feed activated at /feed.xml. Viewing the images and videos is possible without any use of Javascript. Simply extract the image and video urls and their descriptions from the json. Below I only demonstrate how to extract the text, as html. One could view this html in a browser, without Javascript. Random wix site: www.cricketcanadakids.com curl -o x.htm https://www.cricketcanadakids.com # produce a listing of the page titles and json endpoint urls # in addition to a title, each page has a "pageID" # beginning on 3rd line is CSV e.g. for database: # ID, title1, title2, url # last line of CSV is "master" json x=https://static.wixstatic.com/sites/; exec awk '{gsub(/:{/,/"\n"{/);print}' \ |exec sed -n ' s/{/\ &/g; s/\],/\ &/; /<title>/p; /mainPage/p; ' \ |exec sed ' /var publicModel/d; /.*\"baseUrl\"/d; /{filename}/d; s/<title>//;s/<\/title>//; /ExternalBaseUrl/{s/.*Url\":\"//;s/\".*/\ \ \"ID\",\"title1\",\"title2\",\"url\"/;}; s/\\//g; s/{\"pageId\":\"//; 3!s/\",\"/,/g; s/title\"://; s/,pageUriSEO//; s/:\"/,/; s>pageJsonFileName\":\">'"$x"'>; s/\"}.*/.z/; s/\],\"mainPageId\",//; s>masterPageJsonFileName\":\">master,master,'"$x"'>; /topology\":\[/s/json.*/json.z/; ' x.htm Random page from the site: coach-profiles. JSON url taken from CSV above. curl -4o y.json https://static.wixstatic.com/sites/7f1cbe_7131eb80aa297a10c02a08c8ffbc3ef6_122.json.z # produce simple html version of page # text-only, no images or videos x=$(exec echo b|exec tr b '\34'); exec tr -cd '\12\40-\176' < y.json \ |exec sed 's/{/'"$x"'&/g;s/ *//;' \ |exec tr -d '\12' \ |exec tr '\34' '\12' \ |exec sed '1d;$d' \ |exec sed ' s/</\ &/g; ' \ |exec sed 's/\\["tn]//g;s/\",\".*//;/{/d;'
 - feelin_googley 9y ago

afandian 9y ago

With a minute's searching, here's a webpage that's meaningless without JavaScript: https://blogger.googleblog.com/ https://blogger.googleblog.com/ It's arguably a page of text not a web app that needs user interaction. I'm not trying to start an argument about JS: the consensus on HN seems to be that if you don't execute JavaScript you don't deserve to read webpages. I'm just saying that your website's clients are more diverse than 'normal' well-sighted humans. There may be machines reading the site, for all kinds of reasons. And regarding endpoints. One could make the argument that with AJAX we now have richer APIs. I disagree. We have a well-understood API for getting hypertext (HTTP) in a well-understood format (HTML) that works/worked across all websites. Replacing that with a custom-built API for every website isn't apples-for-apples.

feelin_googley 9y ago
The reason I asked is that I rarely use a graphical browser so I am always curious about wesbites that are inaccessible without Javascript. Blogger has a feed. Here is one way to retrieve it, in two steps. 1. get TargetBlogID 2. retrieve data optional: 3. format data for viewing Example: 1. x=$(exec curl https://blogger.googleblog.com \ |exec sed ' s/\\046/\&/g; s/\\46/\&/g; s/\\075/=/g; s/\\75/=/g; /targetBlogID/!d; s/.*targetBlogID=//; s/&.*//; '); 2. curl -o y.htm https://www.blogger.com/feeds/$x/posts/default 3. exec sed ' # ^M is "\r" s/^[0-9a-f]*^M//; s/</</g; s/>/>/g; s/&/\&/g; s/"/\"/g; 1i\ s/<name>/ name &/g; s/<uri>/ uri &/g; s/<generator>/ generator &/g; s/Blogger//; s/<id>/ id &/g; s/<published>/ published &/g; s/<email>/ email &/g; s/<title type=.text.>/ &/g; s/<openSearch:totalResults>/ total results &/g; s/<openSearch:startIndex>/ start index &/g; s/<openSearch:itemsPerPage>/ items per page &/g; s/<updated>/ updated &/g; s/<thr:total>/ thr:total &/g; s/<\/feed>/& /; s/^M*/ /; ' y.htm \ |exec tr -cd '\12\40-\176'
- always_good 9y ago
 It's almost like you asked that question to bait someone into giving you an excuse to show "just how easy it is" with some cli hacking.
- dchuk 9y ago
 Just 3 easy steps

faitswulff 9y ago

Feedly, Netflix, Google Maps, as of this article: https://sonniesedge.co.uk/blog/a-day-without-javascript https://sonniesedge.co.uk/blog/a-day-without-javascript

feelin_googley 9y ago
I remember that article. I do not use any of those three websites so I would be curious to know some specific examples of what the desired structured data looks like. If for Netflix the desired data is only DVD summaries, I recall that these were easily retrievable without Javascript not too long ago.
- afandian 9y ago
  I would argue that it's reasonable for these examples to require JS. My complaint above was the accessibility text-only sites.
  - feelin_googley 9y ago
    I would agree. That article had a very poor choice of websites, IMO. The web is quite usable without Javascript IME.

dithering 9y ago

I know of a site where all the data is embedded inline ("var Person = {firstName: 'Fred', lastName: 'Smith' }") then rendered to HTML with JS. No endpoints, Javascript not JSON, complete pain in the ass to extract without rendering in a headless browser first.

sitepodmatt 9y ago
Isn't this better? Why do you need it render when they're giving you basically JSON of the data (might need to quote the keys in strict parsers), just extract that.
- dithering 9y ago
  The short answer is that, yes, the only parser available to me is that strict. If I'm going to jump languages, I might as well go the whole hog and jump to Puppeteer.

slezyr 9y ago

> Can you provide some example webpages so we can take a look? learnopengl.com

feelin_googley 9y ago
curl -d 'content_name=Heading' https://example.com/content_load.php https://example.com/content_load.php? curl -d 'content_name=Heading/Subheading' https://example.com/content_load.php https://example.com/content_load.php?
- feelin_googley 9y ago
  s/example/learnopengl/ s/Heading/Introduction/

Animats 9y ago

Wix. Everything with Wix.[1] [1] http://www.wix.com/stunningwebsites/400 http://www.wix.com/stunningwebsites/400

feelin_googley 9y ago
I looked at some exemplary Wix sites since I rarely ever encounter these. The ones they identify as "stunning" seem to be mostly devoid of text and are mainly images and videos. As others in this thread have noted is common on today's web, they use json endpoints. Some sites also have an optional XML feed activated at /feed.xml. Viewing the images and videos is possible without any use of Javascript. Simply extract the image and video urls and their descriptions from the json. Below I only demonstrate how to extract the text, as html. One could view this html in a browser, without Javascript. Random wix site: www.cricketcanadakids.com curl -o x.htm https://www.cricketcanadakids.com # produce a listing of the page titles and json endpoint urls # in addition to a title, each page has a "pageID" # beginning on 3rd line is CSV e.g. for database: # ID, title1, title2, url # last line of CSV is "master" json x=https://static.wixstatic.com/sites/; exec awk '{gsub(/:{/,/"\n"{/);print}' \ |exec sed -n ' s/{/\ &/g; s/\],/\ &/; /<title>/p; /mainPage/p; ' \ |exec sed ' /var publicModel/d; /.*\"baseUrl\"/d; /{filename}/d; s/<title>//;s/<\/title>//; /ExternalBaseUrl/{s/.*Url\":\"//;s/\".*/\ \ \"ID\",\"title1\",\"title2\",\"url\"/;}; s/\\//g; s/{\"pageId\":\"//; 3!s/\",\"/,/g; s/title\"://; s/,pageUriSEO//; s/:\"/,/; s>pageJsonFileName\":\">'"$x"'>; s/\"}.*/.z/; s/\],\"mainPageId\",//; s>masterPageJsonFileName\":\">master,master,'"$x"'>; /topology\":\[/s/json.*/json.z/; ' x.htm Random page from the site: coach-profiles. JSON url taken from CSV above. curl -4o y.json https://static.wixstatic.com/sites/7f1cbe_7131eb80aa297a10c02a08c8ffbc3ef6_122.json.z # produce simple html version of page # text-only, no images or videos x=$(exec echo b|exec tr b '\34'); exec tr -cd '\12\40-\176' < y.json \ |exec sed 's/{/'"$x"'&/g;s/ *//;' \ |exec tr -d '\12' \ |exec tr '\34' '\12' \ |exec sed '1d;$d' \ |exec sed ' s/</\ &/g; ' \ |exec sed 's/\\["tn]//g;s/\",\".*//;/{/d;'
- feelin_googley 9y ago