3 ms·

What’s unethical about scraping data from a public website?

blackeyeblitzar 3y ago
Rather than organically pricing their products, these bigger companies are setting prices to always match or undercut pricing elsewhere. This squeezes the advantage smaller players may have on pricing, and big players can afford to do so even at a loss. Another issue is that the bigger players may be essentially signaling their pricing to each other through their scraping, which can be a form of collusion and price fixing - this is what’s under scrutiny in the rental market where software is performing actions that would be anti-competitive for a human to do.
- c_o_n_v_e_x 3y ago
  >these bigger companies are setting prices to always match or undercut pricing elsewhere. This squeezes the advantage smaller players may have on pricing, and big players can afford to do so even at a loss. That sounds like a win for the consumer >This squeezes the advantage smaller players may have on pricing Small players competing on price alone with Walmart or Amazon is suicidal. Offering something other than low price is a way to compete.
  - thestepafter 3y ago
    Until the small players all close down and then the big players raise prices.
    - mattmaroon 3y ago
      Amazon has been around nearly 30 years and the prices have gotten steadily lower. Longer for Walmart and Target. That’s just a thing people imagine will happen. What actually happens is the big players just competing on price against each other forever.
  - mattmaroon 3y ago
    In retail the small companies have never successfully competed on price.
  - avidiax 3y ago
    > That sounds like a win for the consumer Let's say that you have a comparative advantage against Amazon. A company went bankrupt and you bought the remaining inventory of lawn darts at a fire sale price. Now you try to sell your lawn darts at 90% of the Amazon price. And they lower their price to match or be below yours. And this continues. And pretty soon you have to sell your lawn darts at the price you bought them for, and even that isn't really moving them. Sure, Amazon is taking a loss on lawn darts. But they've also squashed you, preventing you from specializing in outdoor equipment and building comparative advantage against Amazon. You go out of business, and Amazon's scraper see's no price data so they go back to a default price of cost+100%. And all of this without a human in the loop at Amazon. They just matched your price, and let the network effect they have do the rest to completely drain your customer flow.
    - Ekaros 3y ago
      Shouldn't the market be big enough for this to eventually drive Amazon to bankruptcy? They might do this with few items, but not with all items. And then at certain point someone would buy those items they overly discounted... And later sell for profit.
      - avidiax 3y ago
        That's the classic economic theory. A company that tries to dump will go bankrupt, which is irrational, so no company will do this. But in this case, Amazon is 40% of the market. There is no upstart competitor that can outlast them in a price war. There is also likely little to no comparative advantage you can have against them, except a legal monopoly like a patented or trademarked product. Even then, if they clone your product, when you sue them it will be a rounding error in their legal budget while you have no cash flow to sue them. The classical school seems to think that there's a limitless amount of irrational entrepreneurship that will drain Amazon if they price too low. In reality, entrepreneurs are generally not that dumb. Amazon, of course, doesn't dominate everything. They aren't going to sell every product. There are marginal, low-volume, low return products that they will happily let other companies sell on their platform or elsewhere. But if your product becomes high margin and high volume on their platform, they will know, and they'll be coming for your profits.
- roenxi 3y ago
  Ironically, Amazon doesn't have a way to figure out what price is appropriate ahead of time without consulting a broader market. They face a similar problem as central planners - they don't know what the market price for anything is. They could run their website as a big real time auction house, but that'd be a touch unsettling for customers because prices would be continuously fluctuating. It is probably better if they use scraping to short-circuit that process. I wouldn't describe it as unethical. The problem with collusion is more about what happens to companies that undercut the market by a little bit. As long as there are no non-market consequences the incentives should play out as expected. It isn't a problem (indeed, in theory it is expected) that merchants all offer the same good/service for the same price.
- mattmaroon 3y ago
  The problem with the rental market is that they are all using the same pricing algorithm. From the same company. That creates collusion. If Amazon and Walmart and Target were all using some third party pricing software, the potential for collusion would be severe . This is just automating something that’s been done manually for as long as competitive retail has existed.

Rather than organically pricing their products, these bigger companies are setting prices to always match or undercut pricing elsewhere. This squeezes the advantage smaller players may have on pricing, and big players can afford to do so even at a loss. Another issue is that the bigger players may be essentially signaling their pricing to each other through their scraping, which can be a form of collusion and price fixing - this is what’s under scrutiny in the rental market where software is performing actions that would be anti-competitive for a human to do.

c_o_n_v_e_x 3y ago
>these bigger companies are setting prices to always match or undercut pricing elsewhere. This squeezes the advantage smaller players may have on pricing, and big players can afford to do so even at a loss. That sounds like a win for the consumer >This squeezes the advantage smaller players may have on pricing Small players competing on price alone with Walmart or Amazon is suicidal. Offering something other than low price is a way to compete.
- thestepafter 3y ago
  Until the small players all close down and then the big players raise prices.
  - mattmaroon 3y ago
    Amazon has been around nearly 30 years and the prices have gotten steadily lower. Longer for Walmart and Target. That’s just a thing people imagine will happen. What actually happens is the big players just competing on price against each other forever.
- mattmaroon 3y ago
  In retail the small companies have never successfully competed on price.
- avidiax 3y ago
  > That sounds like a win for the consumer Let's say that you have a comparative advantage against Amazon. A company went bankrupt and you bought the remaining inventory of lawn darts at a fire sale price. Now you try to sell your lawn darts at 90% of the Amazon price. And they lower their price to match or be below yours. And this continues. And pretty soon you have to sell your lawn darts at the price you bought them for, and even that isn't really moving them. Sure, Amazon is taking a loss on lawn darts. But they've also squashed you, preventing you from specializing in outdoor equipment and building comparative advantage against Amazon. You go out of business, and Amazon's scraper see's no price data so they go back to a default price of cost+100%. And all of this without a human in the loop at Amazon. They just matched your price, and let the network effect they have do the rest to completely drain your customer flow.
  - Ekaros 3y ago
    Shouldn't the market be big enough for this to eventually drive Amazon to bankruptcy? They might do this with few items, but not with all items. And then at certain point someone would buy those items they overly discounted... And later sell for profit.
    - avidiax 3y ago
      That's the classic economic theory. A company that tries to dump will go bankrupt, which is irrational, so no company will do this. But in this case, Amazon is 40% of the market. There is no upstart competitor that can outlast them in a price war. There is also likely little to no comparative advantage you can have against them, except a legal monopoly like a patented or trademarked product. Even then, if they clone your product, when you sue them it will be a rounding error in their legal budget while you have no cash flow to sue them. The classical school seems to think that there's a limitless amount of irrational entrepreneurship that will drain Amazon if they price too low. In reality, entrepreneurs are generally not that dumb. Amazon, of course, doesn't dominate everything. They aren't going to sell every product. There are marginal, low-volume, low return products that they will happily let other companies sell on their platform or elsewhere. But if your product becomes high margin and high volume on their platform, they will know, and they'll be coming for your profits.
roenxi 3y ago
Ironically, Amazon doesn't have a way to figure out what price is appropriate ahead of time without consulting a broader market. They face a similar problem as central planners - they don't know what the market price for anything is. They could run their website as a big real time auction house, but that'd be a touch unsettling for customers because prices would be continuously fluctuating. It is probably better if they use scraping to short-circuit that process. I wouldn't describe it as unethical. The problem with collusion is more about what happens to companies that undercut the market by a little bit. As long as there are no non-market consequences the incentives should play out as expected. It isn't a problem (indeed, in theory it is expected) that merchants all offer the same good/service for the same price.
mattmaroon 3y ago
The problem with the rental market is that they are all using the same pricing algorithm. From the same company. That creates collusion. If Amazon and Walmart and Target were all using some third party pricing software, the potential for collusion would be severe . This is just automating something that’s been done manually for as long as competitive retail has existed.