5 ms·

I wrote a scraper for Air Canada's aeroplan program a few years ago. I wanted to track my points in my own custom native app. I probably had $10,000 worth of po

I wrote a scraper for Air Canada's aeroplan program a few years ago. I wanted to track my points in my own custom native app. I probably had $10,000 worth of points in my account. One day I logged in to find out my account had been deactivated on suspicions of fraud. After several lengthy phone calls with their team (including sending them the node.js script I was using), I was able to get my account restored. For the weeks it took to fix my account it was pretty frustrating. I just don't understand why you can't write a script that acts as your web user agent.

lowercased 5y ago
"agents" and "bots" were proposed as a thing of the future back in the 90s. "you'll have all these agents that can buy stuff for you and book travel and ..." but not if you write it yourself, apparently.
- enos_feedler 5y ago
  And they are still a growing thing in business with RPA. I guess consumers aren't allowed to access the technology afforded to enterprise employees.
  - peterangular 5y ago
    > I guess consumers aren't allowed to access the technology afforded to enterprise employees. Lots of us built our own stacks and attempt to fly under the radar - things have gotten legally gray over the last two years and corporations have no problem sending their legal team to kick your door down for trivial stuff.
PragmaticPulp 5y ago
I’ve been on the other side of this, defending against bots. Basically: Well-behaved and well-intentioned scraping bots are rare. You’d get a lot of users setting update rates to 60 seconds that did a new login every time and creating as much traffic as 1000 users. Then they’d release the script for integration with something people and suddenly you have 1000 people each creating 1000 times as many login requests as a single user. Another common problem was forgetting to implement reasonable back off for failures. A lot of newbies write scripts that immediately retry on a tight infinite loop whenever something goes wrong, sending a huge stream of requests to your server if the API changes or when it goes down. Again, multiply this by many users sharing a script and it becomes a problem. Then of course there are people trying to make a business out of extracting your company’s data, such as putting it in some other website where they can serve ads over your content or whatever (think of all of those StackOverflow scraping websites in Google) Basically, you can’t investigate the motivations of each individual user. You just block them all.
- enos_feedler 5y ago
  Makes sense. Thanks for this context.
  - listenallyall 5y ago
    Does it make sense? Bots rarely make HTTP requests for images, css, video clips, large JS files, custom fonts, etc. Real people do. A well-written bot just seeking some specific data can often complete it's task with less than 1% of the resources that would be sent to a "real" user.
    - drivebycomment 5y ago
      Lol. Well written bot. You assume so much. Most bots are not well written at all, and more often buggy as hell generating unnecessary extra load.
      - listenallyall 5y ago
        True, although the more you lock down, obfuscate, and hide your data, the more the bot-writer is going to use the heavy guns to penetrate you. Points Guy, and Award Wallet are both attempting to provide a service that people, especially American Airlines's most valuable customers, obviously want -- AA could easily work with them, instead of against them (or provide the same service themselves).
    - enos_feedler 5y ago
      I am not sure what "bots" you are talking about here. When I wrote a web scraper to get my Air Canada point values I used a script that fetches the web page and parses it. It was the only way i could get it to work. I had to steel the session token from the browser cookie in order to make it auth
      - listenallyall 5y ago
        I guarantee that, with some effort, you could write a script to emulate every HTTP call -- logging in, accepting the cookie (just a value in a Set-Cookie header), and requesting the point values, making sure that cookie value is in your Cookie header. Just because you could "only get it to work" one way, does not mean there isn't a far more efficient way.
Beached 5y ago
at a previous employer we had a similar issue as you. from the employer side, we had a managed api platform that normalized all our messy data stores into a single api front end. our websites basically all called this api to populate the data in the page requests. except we didn't write this platform, we paid for it. and it is licensed per 100 queries. so when. a single account starts doing 30k queries every hour, 24/7. the pocketbook was directly hit. and yes, web scraping was in the tos that every account agreed to but never read
ryandrake 5y ago
Many moons ago I wrote and published a little app that allowed users of a popular VOIP provider to listen to their voicemail. Previously, voicemail functionality was limited to their awful web-only interface, but the audio files could be easily scraped once the user provided their credentials. My biggest fear/risk was getting noticed and sued by the company, merely for letting users request HTML and display it differently than the company wished. The app is no longer available, so I guess I survived but the idea of $BIGCO crushing me with lawyers was chilling.

lowercased 5y ago

"agents" and "bots" were proposed as a thing of the future back in the 90s. "you'll have all these agents that can buy stuff for you and book travel and ..." but not if you write it yourself, apparently.

enos_feedler 5y ago
And they are still a growing thing in business with RPA. I guess consumers aren't allowed to access the technology afforded to enterprise employees.
- peterangular 5y ago
  > I guess consumers aren't allowed to access the technology afforded to enterprise employees. Lots of us built our own stacks and attempt to fly under the radar - things have gotten legally gray over the last two years and corporations have no problem sending their legal team to kick your door down for trivial stuff.

PragmaticPulp 5y ago

I’ve been on the other side of this, defending against bots. Basically: Well-behaved and well-intentioned scraping bots are rare. You’d get a lot of users setting update rates to 60 seconds that did a new login every time and creating as much traffic as 1000 users. Then they’d release the script for integration with something people and suddenly you have 1000 people each creating 1000 times as many login requests as a single user. Another common problem was forgetting to implement reasonable back off for failures. A lot of newbies write scripts that immediately retry on a tight infinite loop whenever something goes wrong, sending a huge stream of requests to your server if the API changes or when it goes down. Again, multiply this by many users sharing a script and it becomes a problem. Then of course there are people trying to make a business out of extracting your company’s data, such as putting it in some other website where they can serve ads over your content or whatever (think of all of those StackOverflow scraping websites in Google) Basically, you can’t investigate the motivations of each individual user. You just block them all.

enos_feedler 5y ago
Makes sense. Thanks for this context.
- listenallyall 5y ago
  Does it make sense? Bots rarely make HTTP requests for images, css, video clips, large JS files, custom fonts, etc. Real people do. A well-written bot just seeking some specific data can often complete it's task with less than 1% of the resources that would be sent to a "real" user.
  - drivebycomment 5y ago
    Lol. Well written bot. You assume so much. Most bots are not well written at all, and more often buggy as hell generating unnecessary extra load.
    - listenallyall 5y ago
      True, although the more you lock down, obfuscate, and hide your data, the more the bot-writer is going to use the heavy guns to penetrate you. Points Guy, and Award Wallet are both attempting to provide a service that people, especially American Airlines's most valuable customers, obviously want -- AA could easily work with them, instead of against them (or provide the same service themselves).
  - enos_feedler 5y ago
    I am not sure what "bots" you are talking about here. When I wrote a web scraper to get my Air Canada point values I used a script that fetches the web page and parses it. It was the only way i could get it to work. I had to steel the session token from the browser cookie in order to make it auth
    - listenallyall 5y ago
      I guarantee that, with some effort, you could write a script to emulate every HTTP call -- logging in, accepting the cookie (just a value in a Set-Cookie header), and requesting the point values, making sure that cookie value is in your Cookie header. Just because you could "only get it to work" one way, does not mean there isn't a far more efficient way.

Beached 5y ago

at a previous employer we had a similar issue as you. from the employer side, we had a managed api platform that normalized all our messy data stores into a single api front end. our websites basically all called this api to populate the data in the page requests. except we didn't write this platform, we paid for it. and it is licensed per 100 queries. so when. a single account starts doing 30k queries every hour, 24/7. the pocketbook was directly hit. and yes, web scraping was in the tos that every account agreed to but never read

ryandrake 5y ago

Many moons ago I wrote and published a little app that allowed users of a popular VOIP provider to listen to their voicemail. Previously, voicemail functionality was limited to their awful web-only interface, but the audio files could be easily scraped once the user provided their credentials. My biggest fear/risk was getting noticed and sued by the company, merely for letting users request HTML and display it differently than the company wished. The app is no longer available, so I guess I survived but the idea of $BIGCO crushing me with lawyers was chilling.