6 ms·

I'm sad that this is happening to an app that's useful to its users, but the reality is that scraping is legal, always possible, but difficult. This particular

I'm sad that this is happening to an app that's useful to its users, but the reality is that scraping is legal, always possible, but difficult. This particular case is a bit harder since it's not purely using public data, but may still qualify since it's likely scraping with legally-obtained credentials. I know of businesses (scraping for ride-sharing, scraping for business intelligence for retailers, scraping from LinkedIn - see HiQ Labs v. LinkedIn) that have continuously succeeded via scraping in ways that large businesses oppose. The key is: you must make enough profit to justify dedicating engineering and legal techniques to defend your scraping. - Scraping public data is legal, as affirmed by the Supreme Court in Van Buren v. United States [1] and HiQ Labs v. LinkedIn [2]. Defending yourself or suing the data owner in court are both expensive though - Defeating anti-scraping via technical means is pretty much always possible, but can be costly depending on the scraped site's technical expertise and value in keeping their data private. The benefit to you must exceed the cost to you, and ideally should also exceed the cost to the data owner - Mobilizing PR and internal resistance may also be effective, but it's usually hard to have outcry from a large enough group to change an organization's policies. In this case, the union can push for it, but AA may try to withhold improvements until the next set of union negotiations 1. https://en.wikipedia.org/wiki/Van_Buren_v._United_States https://en.wikipedia.org/wiki/Van_Buren_v._United_States 2. https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn

YeBanKo 4y ago
Playing devil’s advocate here: this is not the same as scraping LinkedIn data. Linkedin data is public. This app requires a login info from a flight attendants to scrape their schedules. When you try to log in, you can choose to login as public or as a AA flight attendant. It sucks, but I also understand why a company may be unhappy, that a third party handles credentials and accesses internal data. What they can: - build a 3rd part integration API, which opens up a whole can of worms. Not many tech-first companies can do it right, for an airline it’s a very challenging steps. - build their own, but they already failed there if their employees turn to 3rd party - ignore and let it run. This is basically unauthorized access to go and hope that the guy names Jeff won’t screw up. - deny and prevent access. This is probably technically the easiest and safest from legal standpoint.
- matheusmoreira 4y ago
  > This app requires a login info from a flight attendants to scrape their schedules. So? If the flight attendants have provided their credentials to the scraping software, they have essentially authorized the software to scrape the data on their accounts. It's just a custom user agent running locally and the airline company has no business blocking anything.
  - wraptile 4y ago
    In other words: "you can write this down by hand, copy paste or browser plugins but you cannot automate this". I wonder if this stood up in any other context and I can't imagine of a similar scenario from the top of my head where automation would be forbidden. I could totally hire a part time student from a developing country to do data entry for me and that would be alright? Strange world - somehow these corporations have people brainwashed.
    - YeBanKo 4y ago
      The issue is not that some app has access to a timetable of work shifts. It is that it has access to credentials and potentially can so something else. In your analogy a part time student from a developing country data entry - this is scrapping public linkedin data. What happens here is an employee giving their office badge, so they can go get a folder from the employee’s desk, open it and make a presentation based on its content. To make it worse, many employees give their badges to the exactly same student.
  - YeBanKo 4y ago
    Third party having an unrestricted access to the internal system. No sane business owner would be ok with it. This is literally the reason why protocols like oauth2 exists.
wraptile 4y ago
> The key is: you must make enough profit to justify dedicating engineering and legal techniques to defend your scraping. That's why web scraping is a huge SaaS market these days (I'm part of one too @ scrapfly.io). Loads of our customers are tiny businesses and entrepreneurs that could no way afford the engineering effort required to scrape any of these websites and honestly empowering small folk against these giant, untouchable corporations is the best part about my job :)
toomuchtodo 4y ago
> The key is: you must make enough profit to justify dedicating engineering and legal techniques to defend your scraping. It also works if you have philanthropic, non profit, or unconventional backing to pay for these defensive resources. If this app is providing substantial benefits to the AA crew around scheduling and QoL, their union might consider providing some backstop/support. https://www.apfa.org/ https://www.apfa.org/
- Kalium 4y ago
  If memory serves, FA unions often use seniority-oriented contracts. The more senior members will tend to be more active and better-represented among union leadership. Reserve members are often more junior. Putting on my cynical prick hat for a moment, I would guess the union as an institution is far more willing to throw the app-oriented concerns of the junior members under the bus than the health care and pension concerns of the senior ones.
  - deleted 4y ago
    [deleted]
OJFord 4y ago
> the reality is that scraping is legal, always possible, but difficult. > This particular case is a bit harder since it's not purely using public data, but may still qualify since it's likely scraping with legally-obtained credentials. No, it's easy: they're employees, they can be told they're not allowed to do that. Doesn't matter if the app's legally allowed to exist or not.
- lostdog 4y ago
  But they also deserve to have access to their work schedules, and I bet a good lawyer could argue that "access" should be interpreted broadly here.
  - dpifke 4y ago
    Presumably the non-public data is being scraped using the employees' credentials (i.e. username and password). It is perfectly reasonable for an employer to have a policy which states, "do not give your work username and password to a third party." I can't imagine a court ordering otherwise. Providing an API for this data is a non-trivial amount of work, involving significant technical and compliance challenges. Employee schedules would be useful as a signal for trading in AA stock. How do you enforce that the third party is properly protecting that information, e.g. during SEC-mandated blackout periods around earnings? The union might be able to negotiate for AA to hire lawyers and IT staff to work on such an API, but I really can't see the employees being automatically entitled to it.
    - dkonofalski 4y ago
      If the scraping is happening "on-device", though, then they're not providing their details to a third party. They're simply accessing their schedules. Otherwise, pulling up their schedules in any web browser would be considered giving their credentials to a third party since that's basically what's happening here. It would be like logging in to the aa.com employee site and then installing a Chrome extension that reads the page that was downloaded. Nothing is given to the Chrome extension in terms of credentials, only page content.
nofinator 4y ago
It's noteworthy that American Airlines has taken the hardest line against blocking AwardWallet, too [1]. https://yourmileagemayvary.net/2021/12/21/is-this-the-reason-american-airlines-forced-awardwallet-to-stop-tracking-accounts/ https://yourmileagemayvary.net/2021/12/21/is-this-the-reason...

YeBanKo 4y ago

Playing devil’s advocate here: this is not the same as scraping LinkedIn data. Linkedin data is public. This app requires a login info from a flight attendants to scrape their schedules. When you try to log in, you can choose to login as public or as a AA flight attendant. It sucks, but I also understand why a company may be unhappy, that a third party handles credentials and accesses internal data. What they can: - build a 3rd part integration API, which opens up a whole can of worms. Not many tech-first companies can do it right, for an airline it’s a very challenging steps. - build their own, but they already failed there if their employees turn to 3rd party - ignore and let it run. This is basically unauthorized access to go and hope that the guy names Jeff won’t screw up. - deny and prevent access. This is probably technically the easiest and safest from legal standpoint.

matheusmoreira 4y ago
> This app requires a login info from a flight attendants to scrape their schedules. So? If the flight attendants have provided their credentials to the scraping software, they have essentially authorized the software to scrape the data on their accounts. It's just a custom user agent running locally and the airline company has no business blocking anything.
- wraptile 4y ago
  In other words: "you can write this down by hand, copy paste or browser plugins but you cannot automate this". I wonder if this stood up in any other context and I can't imagine of a similar scenario from the top of my head where automation would be forbidden. I could totally hire a part time student from a developing country to do data entry for me and that would be alright? Strange world - somehow these corporations have people brainwashed.
  - YeBanKo 4y ago
    The issue is not that some app has access to a timetable of work shifts. It is that it has access to credentials and potentially can so something else. In your analogy a part time student from a developing country data entry - this is scrapping public linkedin data. What happens here is an employee giving their office badge, so they can go get a folder from the employee’s desk, open it and make a presentation based on its content. To make it worse, many employees give their badges to the exactly same student.
- YeBanKo 4y ago
  Third party having an unrestricted access to the internal system. No sane business owner would be ok with it. This is literally the reason why protocols like oauth2 exists.

wraptile 4y ago

> The key is: you must make enough profit to justify dedicating engineering and legal techniques to defend your scraping. That's why web scraping is a huge SaaS market these days (I'm part of one too @ scrapfly.io). Loads of our customers are tiny businesses and entrepreneurs that could no way afford the engineering effort required to scrape any of these websites and honestly empowering small folk against these giant, untouchable corporations is the best part about my job :)

toomuchtodo 4y ago

> The key is: you must make enough profit to justify dedicating engineering and legal techniques to defend your scraping. It also works if you have philanthropic, non profit, or unconventional backing to pay for these defensive resources. If this app is providing substantial benefits to the AA crew around scheduling and QoL, their union might consider providing some backstop/support. https://www.apfa.org/ https://www.apfa.org/

Kalium 4y ago
If memory serves, FA unions often use seniority-oriented contracts. The more senior members will tend to be more active and better-represented among union leadership. Reserve members are often more junior. Putting on my cynical prick hat for a moment, I would guess the union as an institution is far more willing to throw the app-oriented concerns of the junior members under the bus than the health care and pension concerns of the senior ones.
- deleted 4y ago
  [deleted]

OJFord 4y ago

> the reality is that scraping is legal, always possible, but difficult. > This particular case is a bit harder since it's not purely using public data, but may still qualify since it's likely scraping with legally-obtained credentials. No, it's easy: they're employees, they can be told they're not allowed to do that. Doesn't matter if the app's legally allowed to exist or not.

lostdog 4y ago
But they also deserve to have access to their work schedules, and I bet a good lawyer could argue that "access" should be interpreted broadly here.
- dpifke 4y ago
  Presumably the non-public data is being scraped using the employees' credentials (i.e. username and password). It is perfectly reasonable for an employer to have a policy which states, "do not give your work username and password to a third party." I can't imagine a court ordering otherwise. Providing an API for this data is a non-trivial amount of work, involving significant technical and compliance challenges. Employee schedules would be useful as a signal for trading in AA stock. How do you enforce that the third party is properly protecting that information, e.g. during SEC-mandated blackout periods around earnings? The union might be able to negotiate for AA to hire lawyers and IT staff to work on such an API, but I really can't see the employees being automatically entitled to it.
  - dkonofalski 4y ago
    If the scraping is happening "on-device", though, then they're not providing their details to a third party. They're simply accessing their schedules. Otherwise, pulling up their schedules in any web browser would be considered giving their credentials to a third party since that's basically what's happening here. It would be like logging in to the aa.com employee site and then installing a Chrome extension that reads the page that was downloaded. Nothing is given to the Chrome extension in terms of credentials, only page content.

nofinator 4y ago

It's noteworthy that American Airlines has taken the hardest line against blocking AwardWallet, too [1]. https://yourmileagemayvary.net/2021/12/21/is-this-the-reason-american-airlines-forced-awardwallet-to-stop-tracking-accounts/ https://yourmileagemayvary.net/2021/12/21/is-this-the-reason...