5 ms·

I blame modern CS programs that don't teach kids about parsing. The last time I looked at some scraping code, the dev was using regexes to "parse" html to find

I blame modern CS programs that don't teach kids about parsing. The last time I looked at some scraping code, the dev was using regexes to "parse" html to find various references. Maybe that's a way to defend against bots that ignore robots.txt, include a reference to a Honeypot HTML file with garbage text, but include the link to it in a comment.

tuwtuwtuwtuw 1y ago
Do you think that if some CS programs taught parsing, the authors of the bot would parse the HTML to properly extract links, instead of just doing plain text search? I doubt it.
- OhMeadhbh 1y ago
  Sure. Ideally CS programs would be more than the trade schools they are today. When I was in school, the "CS Program" was a specialization in the math department. I was already in a Physics program when they created a "Computer Science & Engineering" degree. And I remember both the Math and CSE degrees being pretty decent. But that was back in the day when people took CS related courses because they were interested in computers, not because they wanted to get a super high paying job for a FANG company. Maybe you're right. You can lead a horse to water, but you can't make it think deep thoughts about parsing.
ericmcer 1y ago
How would recommend doing it? If I was just trying to pull <a/> tag links out I feel like treating it like text and using regex would be way more efficient than a full on HTML parser like JSDom or something.
- singron 1y ago
  You don't need javascript to parse HTML. Just use an HTML parser. They are very fast. HTML isn't a regular language, so you can't parse it with regular expressions. Obligatory: https://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags https://stackoverflow.com/questions/1732348/regex-match-open...
  - zahlman 1y ago
    The point is: if you're trying to find all the URLs within the page source, it doesn't really matter to you what tags they're in, or how the document is structured, or even whether they're given as link targets or in the readable text or just what.
vaylian 1y ago
The people who do this type of scraping to feed their AI are probably also using AI to write their scraper.
mikeiz404 1y ago
It’s been some time since I have dealt with web scrapers but it takes less resources to run a regex than it does to parse the DOM (which may have syntactically incorrect parts anyway). This can add up when running many scraping requests in parallel. So depending on your goals using a regex can be much preferred.
mrweasel 1y ago
You don't need to teach parsing, that won't help much any way. We need to teach people to be good netizen again. I'd argue that it was always viewed as reasonable to scrape content, as long as you didn't misrepresent content as your own and if you scraped responsibly, backing of if the server started to slow down, or simply not crawling to fast to begin with. Currently we have at least three problems: 1) Companies have no issue with not providing sources and not linking back. 2) There are too many scrapers, even if they behaved, some site would struggle to handle all of them. 3) Srapers go full throttle 24/7, expecting the sites to rate-limit them if they are going to fast. Hammer a site into the ground, just wait until it's back and hammer it again, grabbing what you can before it crashes once more. There's no longer a sense of the internet being for all of us and that we need to make room for each other. Website / human generated content exists as a resource to be strip mined.

tuwtuwtuwtuw 1y ago

Do you think that if some CS programs taught parsing, the authors of the bot would parse the HTML to properly extract links, instead of just doing plain text search? I doubt it.

OhMeadhbh 1y ago
Sure. Ideally CS programs would be more than the trade schools they are today. When I was in school, the "CS Program" was a specialization in the math department. I was already in a Physics program when they created a "Computer Science & Engineering" degree. And I remember both the Math and CSE degrees being pretty decent. But that was back in the day when people took CS related courses because they were interested in computers, not because they wanted to get a super high paying job for a FANG company. Maybe you're right. You can lead a horse to water, but you can't make it think deep thoughts about parsing.

ericmcer 1y ago

How would recommend doing it? If I was just trying to pull <a/> tag links out I feel like treating it like text and using regex would be way more efficient than a full on HTML parser like JSDom or something.

singron 1y ago
You don't need javascript to parse HTML. Just use an HTML parser. They are very fast. HTML isn't a regular language, so you can't parse it with regular expressions. Obligatory: https://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags https://stackoverflow.com/questions/1732348/regex-match-open...
- zahlman 1y ago
  The point is: if you're trying to find all the URLs within the page source, it doesn't really matter to you what tags they're in, or how the document is structured, or even whether they're given as link targets or in the readable text or just what.

vaylian 1y ago

The people who do this type of scraping to feed their AI are probably also using AI to write their scraper.

mikeiz404 1y ago

It’s been some time since I have dealt with web scrapers but it takes less resources to run a regex than it does to parse the DOM (which may have syntactically incorrect parts anyway). This can add up when running many scraping requests in parallel. So depending on your goals using a regex can be much preferred.

mrweasel 1y ago

You don't need to teach parsing, that won't help much any way. We need to teach people to be good netizen again. I'd argue that it was always viewed as reasonable to scrape content, as long as you didn't misrepresent content as your own and if you scraped responsibly, backing of if the server started to slow down, or simply not crawling to fast to begin with. Currently we have at least three problems: 1) Companies have no issue with not providing sources and not linking back. 2) There are too many scrapers, even if they behaved, some site would struggle to handle all of them. 3) Srapers go full throttle 24/7, expecting the sites to rate-limit them if they are going to fast. Hammer a site into the ground, just wait until it's back and hammer it again, grabbing what you can before it crashes once more. There's no longer a sense of the internet being for all of us and that we need to make room for each other. Website / human generated content exists as a resource to be strip mined.