4 ms·

I wouldn't characterize it as "better" but specifically easier and more flexible for the people writing and maintaining these scrapers. I'm also speaking more b

by meritt 7y ago

I wouldn't characterize it as "better" but specifically easier and more flexible for the people writing and maintaining these scrapers. I'm also speaking more broadly about scripting languages (not javascript specifically) vs the aforementioned JVM or Go, and the ease with which you can deal with inconsistent, frequently changing, and often completely invalid inputs from a wide variety of data sources. Plaid's use case here is automating logins, responding to captchas, manipulating those on-screen virtual keypads to respond to security questions, chaining together multiple HTTP requests, and then parsing out frequently invalid, rapidly changing, and just plain broken content from a wide multitude of banking websites.

calibas 7y ago
To me, that seems like a case against Javascript. Invalid or broken content should return an error, the parser shouldn't try to "fix" it. And things like data types should be strictly enforced, otherwise you can get unpredictable results, which is especially bad when you're dealing with money transfers.
- meritt 7y ago
  > Invalid or broken content should return an error, the parser shouldn't try to "fix" it. You would have a really difficult life in web scraping. You do not have the guarantees of well-formed data. Instead you get HTML with mismatched tags, JSON with newlines in the middle of strings, content that claims it's UTF-8 but upon closer inspection is actually GB2312, pagination endpoints with off-by-one errors, etc. It's an absolute mess and taking the stance of "well, they didn't encode their JSON correctly, so we're not going to operate on their data" isn't a very effective strategy. > Which is especially bad when you're dealing with money transfers Afaik Plaid is read-only. They fetch information from financial institutions and make it available through an API.
  - calibas 7y ago
    I'm actually quite experienced with web scraping, mostly using PHP and XPath, but also with Javascript as well as a custom approach written in Rust. I know in detail what an inconsistent mess everything is. That's why I'm so uncomfortable handling things like bank transfers over such inconsistent, buggy systems, which is what Plaid does. It's not read-only: https://plaid.com/use-cases/consumer-payments/ https://plaid.com/use-cases/consumer-payments/ Not to say I don't trust Plaid, I'm sure they're aware of all this and very careful about how they do things.
    - meritt 7y ago
      Then I feel like we're in agreement then that web scraping is a messy and imprecise art and the flexibility of a scripting language like PHP provides is immense. I have no affiliation with plaid, I've honestly only heard negative things about those guys, I was only empathizing with the difficulties in maintaining thousands of different scrapers and why I felt a scripting language provided far more latitude to get things done.

by meritt 7y ago

calibas 7y ago

To me, that seems like a case against Javascript. Invalid or broken content should return an error, the parser shouldn't try to "fix" it. And things like data types should be strictly enforced, otherwise you can get unpredictable results, which is especially bad when you're dealing with money transfers.

meritt 7y ago
> Invalid or broken content should return an error, the parser shouldn't try to "fix" it. You would have a really difficult life in web scraping. You do not have the guarantees of well-formed data. Instead you get HTML with mismatched tags, JSON with newlines in the middle of strings, content that claims it's UTF-8 but upon closer inspection is actually GB2312, pagination endpoints with off-by-one errors, etc. It's an absolute mess and taking the stance of "well, they didn't encode their JSON correctly, so we're not going to operate on their data" isn't a very effective strategy. > Which is especially bad when you're dealing with money transfers Afaik Plaid is read-only. They fetch information from financial institutions and make it available through an API.
- calibas 7y ago
  I'm actually quite experienced with web scraping, mostly using PHP and XPath, but also with Javascript as well as a custom approach written in Rust. I know in detail what an inconsistent mess everything is. That's why I'm so uncomfortable handling things like bank transfers over such inconsistent, buggy systems, which is what Plaid does. It's not read-only: https://plaid.com/use-cases/consumer-payments/ https://plaid.com/use-cases/consumer-payments/ Not to say I don't trust Plaid, I'm sure they're aware of all this and very careful about how they do things.
  - meritt 7y ago
    Then I feel like we're in agreement then that web scraping is a messy and imprecise art and the flexibility of a scripting language like PHP provides is immense. I have no affiliation with plaid, I've honestly only heard negative things about those guys, I was only empathizing with the difficulties in maintaining thousands of different scrapers and why I felt a scripting language provided far more latitude to get things done.