3 ms·

Many years ago I was working on a complex screen scraper (the good kind) app and every site we scraped was ~1000 lines of error prone code, plus all the usual p

by flukus 7y ago

Many years ago I was working on a complex screen scraper (the good kind) app and every site we scraped was ~1000 lines of error prone code, plus all the usual problems of sites changing randomly under us and the need to respond quickly. They wanted to scale up to scrape dozens of sites and we weren't given the resources so I came up with a little xpath like language "//content-area/table/tbody/@foreach/tr/@get-value" or something like that, after that the largest scraper was a dozen lines of code. I think it came down on the right side of reducing complexity rather than creating more but I was let go shortly after for the heinous crime of not being at my desk at 9am.

specialist 7y ago
This is a Most Righteous Answer, kissing cousin to The Correct Answer. Repeating myself, sorry: I've done a lot of electronic medical records. At the time, SOAP, WSDL, HL7 3.x (XML format) reigned. Techniques like yours is how our two person team was able to run circles around our much larger partners. In other words, while they were stuck trying to update and compile the XSDs, we just treated inbound data as screen scrapping. Sharing a tip of my own: while xpath expressions are mostly great, we migrated to globbing expressions. The wildcarding allowed our scrappers to be a lot more flexible, robust in regards to all the chaotic mutant data we were receiving. https://en.wikipedia.org/wiki/Glob_%28programming%29 https://en.wikipedia.org/wiki/Glob_%28programming%29 PS- IMHO, The Correct Answer is to have path expression intrinsics, built right into the language. What LINQ should have been. Imagine if Java had regex intrinsics like Perl, instead of that weird library. Some idea.
- yumaikas 7y ago
  So how did you use globbing on XML? Was there an existing library for it, or did you write your own?
  - specialist 7y ago
    Rolled my own that I'd copypasta. I always ended up with my own graph model (eg HL7, XML DOM, parse trees) and add query methods to the node objects. I really dislike having separate query objects, preferring a tighter fit. If you need a globber, maybe you can extract it from this library. https://github.com/EsotericSoftware/wildcard https://github.com/EsotericSoftware/wildcard It's nicer than mine, but I regarded mine as easier to debug.

by flukus 7y ago

specialist 7y ago

This is a Most Righteous Answer, kissing cousin to The Correct Answer. Repeating myself, sorry: I've done a lot of electronic medical records. At the time, SOAP, WSDL, HL7 3.x (XML format) reigned. Techniques like yours is how our two person team was able to run circles around our much larger partners. In other words, while they were stuck trying to update and compile the XSDs, we just treated inbound data as screen scrapping. Sharing a tip of my own: while xpath expressions are mostly great, we migrated to globbing expressions. The wildcarding allowed our scrappers to be a lot more flexible, robust in regards to all the chaotic mutant data we were receiving. https://en.wikipedia.org/wiki/Glob_%28programming%29 https://en.wikipedia.org/wiki/Glob_%28programming%29 PS- IMHO, The Correct Answer is to have path expression intrinsics, built right into the language. What LINQ should have been. Imagine if Java had regex intrinsics like Perl, instead of that weird library. Some idea.

yumaikas 7y ago
So how did you use globbing on XML? Was there an existing library for it, or did you write your own?
- specialist 7y ago
  Rolled my own that I'd copypasta. I always ended up with my own graph model (eg HL7, XML DOM, parse trees) and add query methods to the node objects. I really dislike having separate query objects, preferring a tighter fit. If you need a globber, maybe you can extract it from this library. https://github.com/EsotericSoftware/wildcard https://github.com/EsotericSoftware/wildcard It's nicer than mine, but I regarded mine as easier to debug.