4 ms·
>Sorry, can't kimonify >According that web site's data protection policy, we were unable to kimonify that particular page. Sigh... Oh well... Back to scraping
by fnordfnordfnord 13y ago
>Sorry, can't kimonify
>According that web site's data protection policy, we were unable to kimonify that particular page.
Sigh... Oh well... Back to scraping.
- pranade 13y agoWhat page were you trying to hit? We'll check it out
- fnordfnordfnord 13y agoPages buried in here: https://fannin4.wcjc.edu/ https://fannin4.wcjc.edu/ The course catalog is public, so no login is needed. I want to scrape various data related to courses, to populate forms automatically and such.
- nobodysfool 13y agoYeah, of course it won't work with HTTPS sites. They'd have to proxy those HTTPS sites and perform a MITM just to do it.
- Dylan16807 13y agoAs opposed to HTTP where they proxy it and MITM it? I don't understand the objection.
- fnordfnordfnord 13y agoI kind of just assumed that's what they were doing.
- garyjob 13y agoHTTPS is definitely a problem for proxy servers unless you the proxy server rewrites all the URLs in the html pages loaded as well as all the URLs of the Ajax calls to point back to the proxy server.
- fnordfnordfnord 13y agoIt may be to their advantage to come up with a solution for this, given the popularity of https these days.