5 ms·
> Invalid or broken content should return an error, the parser shouldn't try to "fix" it. You would have a really difficult life in web scraping. You do not ha
by meritt 7y ago
> Invalid or broken content should return an error, the parser shouldn't try to "fix" it.
You would have a really difficult life in web scraping. You do not have the guarantees of well-formed data. Instead you get HTML with mismatched tags, JSON with newlines in the middle of strings, content that claims it's UTF-8 but upon closer inspection is actually GB2312, pagination endpoints with off-by-one errors, etc. It's an absolute mess and taking the stance of "well, they didn't encode their JSON correctly, so we're not going to operate on their data" isn't a very effective strategy.
> Which is especially bad when you're dealing with money transfers
Afaik Plaid is read-only. They fetch information from financial institutions and make it available through an API.
- calibas 7y agoI'm actually quite experienced with web scraping, mostly using PHP and XPath, but also with Javascript as well as a custom approach written in Rust. I know in detail what an inconsistent mess everything is. That's why I'm so uncomfortable handling things like bank transfers over such inconsistent, buggy systems, which is what Plaid does. It's not read-only: https://plaid.com/use-cases/consumer-payments/ https://plaid.com/use-cases/consumer-payments/ Not to say I don't trust Plaid, I'm sure they're aware of all this and very careful about how they do things.
- meritt 7y agoThen I feel like we're in agreement then that web scraping is a messy and imprecise art and the flexibility of a scripting language like PHP provides is immense. I have no affiliation with plaid, I've honestly only heard negative things about those guys, I was only empathizing with the difficulties in maintaining thousands of different scrapers and why I felt a scripting language provided far more latitude to get things done.