22 ms·
A Google bot scrapes pricing info by adding items to carts
- vmateixeira 6y agoGenuine question, is this not considered a DoS attack? Let's imagine I have my online stock linked to limited physical items/assets, ex tickets for a show, which will get reserved for a period of time. This will be preventing genuine clients from buying them.
- zelly 6y agoYou can always update your robots.txt or block the Googlebot UA. (lol)
- nxpnsv 6y agoPossibly it is lower traffic than a full on dos?
- vmateixeira 6y agoYes, in regards to traffic. But it's still denying me from providing a service to real customers.
- Mizza 6y agoI'm thinking - if I forbid this in my site's Terms of Service, will DoJ go after Google for CFAA violations like they did to Aaron?
- vmateixeira 6y agoYeah.. probably depend$ on how loud you can make yourself heard.. RIP Aaron
- vmception 6y agoThat sparked a funny idea in my head, what if we tricked product managers industry wide to follow KPIs and A/B tests that resulted in a better user experience for consumers, instead of experiences that coincidentally slightly upticked "engagement". Because it seems like this mystery shopper is already doing that.
- itronitron 6y agoGiven that engagement metrics have been heavily interfered with for many years, as a result of bots and other activities, and yet PMs still rely on them it seems unlikely that they will be pulled away from that spectacle anytime soon. I like your idea though.
- hundchenkatze 6y ago> "and yet PMs still rely on them it seems unlikely that they will be pulled away from that spectacle anytime soon." I think they meant, since PMs will never stop using metrics, we should write bots that skew those metrics in favor of an experience for the consumer rather than the perceived increase in engagement.
- chundicus 6y agoWhat are some examples good KPIs and A/B tests for better consumer user experiences? Engagement is obviously deeply flawed if a good consumer user experience is your goal, but it does have the nice property of being easily measured. Do you rely on users constantly rating their experience on a numeric scale?
- maltelandwehr 6y ago„Messing up your competitors A/B test“ is not unheard of as a tactic in highly competitive ecommerce settings.
- withinboredom 6y agoDo software engineers actually implement that? That seems pretty immoral. I'd rather let them run the a/b test and steal whatever solution they end up with.
- advisedwang 6y agohttp://archive.is/YRkQe http://archive.is/YRkQe
- maltelandwehr 6y agoThanks! I was not aware you could use Web Archive for that. All the more reason to Love that site!
- bravoetch 6y agoTL;DR - it's a google bot
- CraneWorm 6y agoit was a google bot. But now the idea is out there to copy.
- amflare 6y agoSpecifically one trying to get the price at checkout
- whoisjuan 6y agoThis is the problem that I have with HN editorializing titles. This comment made perfect sense and was useful before they changed the title, but now that is changed it looks like the poster is an idiot who is just saying what the title says and some people downvote it. I know HN is not very keen on adding features, but this is one that is missing for the sake of transparency (seeing if the original title was editorialized) I understand that the original title was click-bait trash and this one makes sense, but it would be nice to understand how it changed so certain comments don’t get de-contextualized. But I guess is the same problem with editing comments.
- caser 6y agoThis feels like a great way to get data on how all these different e-commerce companies approach remarketing.
- leoh 6y agoSuch a bot could be used to damage ad tracking
- whoisjuan 6y agoThis bot is simply trying to get the final price (with tax and shipping) which is ridiculous because e-commerce storefronts should do that in the first place without going through the whole checkout process. I always have found that kind of shady but it's probably known to increase conversions. What I found interesting is that this an open attack vector for e-commerces. Multiple bots can hit a website and start adding items and start the checkout process. This basically creates an unprecedented cart behavior data influx that ruins any possible usage for data coming from legit customers. Maybe cleaning the data wouldn't be that hard but if someone knows what they are doing they can really make it hard (separate IPs, emails and cart behavior) I doubt Shopify or Magento have anything to prevent this.
- gowld 6y agoThere's a tiktok meme doing this to harass the Trump campaign's online store.
- sp332 6y agoe-commerce storefronts should do that in the first place without going through the whole checkout process Yes but how would you verify this or hold them accountable?
- gruez 6y ago> This bot is simply trying to get the final price (with tax and shipping) which is ridiculous because e-commerce storefronts should do that in the first place without going through the whole checkout process. It's usually not possible because you don't know how much the shipping + taxes are until the customer enters the billing information.
- whoisjuan 6y agoYeah that's true. But from a UX perspective there are ways to make this less opaque. Perhaps a call to action at the top of the listing with an entry box to enter the Zip Code so an approximate final price can be calculated. Any good UX designer can come up with a solution for this in a couple of hours or less. There's just no motivation to make it happen because this obfuscation of data is particularly optimized and useful for the sellers.
- abofh 6y agoGoogle. Saved you a click.
- lawnchair_larry 6y agoThanks. “A Google bot scrapes pricing info by adding items to carts” could have replaced that whole fairy tale that they wanted us to pay for.
- dang 6y agoOk, we'll use yours. Thanks! I kind of liked the mystery shopper angle, but since there's more than one complaint in the thread, the guidelines win ("Please use the original title, unless it is misleading or linkbait") https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- ardy42 6y ago> When The Wall Street Journal contacted Google in June, a spokesman at the internet giant, after a few days of digging, provided an update: The mystery shopper is a bot of its own creation. > The purpose: making sure the all-in price for the product, including tax and shipping, matches the listing on its Google Shopping platform or in advertisements. It wasn’t to cause angst to merchants due to thousands of abandoned carts. > “We use automated systems to ensure consumers are getting accurate pricing information from our merchants,” a company spokesman said. “This sometimes leads to merchants seeing abandoned carts as a result of our system testing whether the price displayed matches the price at checkout.” You'd think they could have better identified themselves in accounts they were creating rather than creating this mysterious "John Smith" persona. Maybe "GoogleBot PriceVerifier" would have been a better choice. edit: remove my inaccurate confusion about something, and fix quotes that I'd copied from a plagiarized version of the article.
- bluGill 6y agoThey need to be non traceable. If I'm doing something underhanded with pricing information I want to detect Google and other such bots and give them different information.
- inetknght 6y agoYou really think it's wise to lie to your customers?
- its_dario 6y agoNo, and that's not their point. They're saying if they were to lie to their customers, they'd want to make sure they're deceiving Google. In that case, having an easy way to detect that it's Google would make that trivial.
- eatingCake 6y agoIf you're a medium sized corporation that is subject to the whims of individual customers, no. If you are a gigantic global company that can position itself to weather any temporary inconvenience, absolutely yes.
- tacon 6y agoWould it be too much for Google to program the bot to get the final price, and then delete all the items from the cart? Seems rather rude, even for Google.
- disposekinetics 6y agoIs abandoning a cart really rude behavior? I sometimes do it just to see if they'll spam me as a test of if I want to do business with a site.
- jawns 6y agoIt's not rude at a consumer level, where (in general) you're at least considering making the purchase. It's arguably rude at a bot level, depending on the frequency, where there is 0% chance of conversion.
- dragonwriter 6y agoThe entire purpose of the bot is to provide listings to consumers who are looking to buy. If it was consumer journalist doing it to get the price for a news article (in a for-profit publication) about the product, would it be “rude”? If not, how is it for Google bot?
- jsnell 6y agoBecause bots will do it at a much larger scale than individual humans. The first law of web robotics applies here: the bot should not harm the website it's crawling, or through inaction allow it to come to harm. I didn't read the article due to the paywall, but I assume that the problem is that the problem is that these goods are reserved for that (non)-customer until the shopping cart times out? That is directly costing the merchant money, either in lost sales or having to maintain extra inventory. So yeah, that bot really should have been programmed to end the session with an empty basket one way or another.
- rubyron 6y ago
- justinwp 6y agoProtip: You will often get a discount coupon if you go through most of the checkout process(need to provide email), but wait a couple days. Many stores automate abandoned checkout promotions.
- bradlys 6y agoYes! This is also something that is common with smaller online retailers. Don't expect this with B&H, Adorama, or Newegg. Frequently these small companies give one time codes you won't see or be able to gain elsewhere.
- webmaven 6y agoFor a while there were registrars that gave a discount when you abandoned your cart.
- kutorio 6y agoA good example of "proof of work" used for price differentiation.
- Animats 6y agoNow even the WSJ has clickbait titles. Should have been "Google price-checking system annoying merchants".
- hyperrail 6y agoThis is an A Hed, one of the Wall Street Journal's daily funny news stories on the front page. Other recent ones include: * Baseball Stadiums Are Closed to Fans - but This Guy's Balcony Is Open for Business * Americans Craving Contact Ponder New Rules for Throwing a Party in Real Life * When Your Best Friend in Quarantine Is a Squirrel, You May Be Going Nuts * Beware of Falling Tofu: China Takes on High-Altitude Littering * Did You Forget Things During Lockdown? So Did People With Superior Memories In that context I don't have a problem with the title "Who Is the Mystery Shopper Leaving Behind Thousands of Online Shopping Carts?".
- agustif 6y agoHahaha That's the NYT website headlines nowadays LOL
- danimal88 6y agoIt's just price data collection. In particular, MAP policies can be skirted by not publishing a final price but having a price below MAP in the cart which is a common tactic that online sellers utilize. By pretending to walk through the cart, all sorts of data about pricing, taxes, etc. can be learned. It's not entirely uncommon to see different prices at different times, for different user agents, for different locations, etc. Used to work for a company that build huge price collection systems and built many of them...
- Drdrdrq 6y agoMAP == Minimum Advertised Price
- rkagerer 6y agoAre there legal implications to Google bots transacting with websites under false pretenses? I mean their normal web crawler identifies itself as such. Here, I feel like they're committing (very) minor fraud by putting in fake shopper information and actively hiding their identity. Not a big deal if it were just some Joe Schmoe somewhere, but at their scale might it border on harassment? The robot equivalent of a prank call?
- inetknght 6y agoYou should worry more about sellers engaging in anti-competitive behavior like bait-and-switch or price fixing.
- the_pwner224 6y agoProbably a violation of the CFAA. Lots of people hate it because they think it's overreaching, and lots of companies use it to legally threaten scrapers and security research. But in this case Google is doing mass unauthorized use of other people's computers.
- shadowgovt 6y agoIf I'm doing price comparison between online vendors, I will---as a human---put some items in the cart and get right to the edge of checkout to determine what my final bill would be. I may not close the sale if I'm looking at a better option elsewhere. How is what I'm doing materially different from what Google's doing? Is scale a factor that matters for CFAA?
- lmm 6y agoMaybe you are violating the CFAA by doing that? It's a very broad law.
- jogjayr 6y agoI think that's outdated information. ToS violations aren't prosecutable under CFAA since April.[1] 1. https://www.eff.org/deeplinks/2020/04/federal-judge-rules-it-not-crime-violate-websites-terms-service https://www.eff.org/deeplinks/2020/04/federal-judge-rules-it...
- yongjik 6y agorobots.txt, man, if you don't want search engines to visit certain part of your page, use robots.txt! Once heard a tale of an angry site owner calling Google (back when Google itself was novel) - Google deleted his whole website! Turned out he had "DELETE" button in each page, which generated plain GET request. So Googlebot visited the site, followed links to every page, and then of course followed every link that generated GET requests - because they are supposed to be safe. Don't be like that site owner.
- YetAnotherNick 6y agoHow do I use robots.txt to tell google to not add item to the shopping cart?
- yongjik 6y agoErm... hide the shopping cart page behind robots.txt?
- kabacha 6y agoAs someone who has seen way too many robots.txt files that's exactly how you do it.
- rubyron 6y agoWell, theoretically, your Add To Cart button could have an href with a path that’s banned in robots.txt, but overridden with JS. But most online stores should be happy to have Google crawling their prices and showing up under the Shopping results.
- a1369209993 6y agoThat has nothing to do with robots.txt, the problem is doing things in response to GET requests. I've said it before and I'll say it again: you do not do things on GET requests.
- deleted 6y ago[deleted]
- soganess 6y agoFor people saying this to calculate the final price with shipping and tax, it's not (or at least not entirely). It is for this new sales conversion dark pattern where prices aren't listed until you add to cart. Ebay sellers are particularly bad offenders: https://www.ebay.com/itm/Open-Box-Certified-Samsung-Galaxy-13-3-4K-Ultra-HD-Touch-Screen-Chromeboo/203028862820?epid=21037915306&hash=item2f45769764:g:IM4AAOSwq4Nesuii https://www.ebay.com/itm/Open-Box-Certified-Samsung-Galaxy-1...
- an_opabinia 6y agoThat clearly has a see details button that shows the price.
- soganess 6y agoMost dark patterns have a non-intuitive way of circumventing them (the small-font faded-color "no, thank you" button comes to mind). That is Ebay's. Other examples here: https://ux.stackexchange.com/questions/83050/price-too-low-to-show https://ux.stackexchange.com/questions/83050/price-too-low-t... Amazon example from a few years ago: https://lh5.googleusercontent.com/ztyT6xTPaTr9TtP8LwlRJBE6RVe8RwgJxd8HwomRTxDzdfzMVojXZduT1CleO1SP8VTgH-MKVuVWaMvTwKoD0O1t2H3HuOidqYOtVFKQAGcKS33Vb4PaFCrYzA https://lh5.googleusercontent.com/ztyT6xTPaTr9TtP8LwlRJBE6RV...
- nkozyra 6y agoWhat am I missing on here? That item has the price listed without having to Add To Cart.
- Felk 6y agoThe modal that pops up is not in the dom until you click the "See details" link, which has target="javascript:;". The "Add to cart" button is an actual link. I wouldn't be surprised if Google just doesn't want to run javascript to extract pricing information if it doesn't necessarily have to.
- dheera 6y ago
- baybal2 6y agoFunny, a one quick gig I did in my college years was to write a shopping bot protection against "guaranteed lowest price" scraper like tigerdirect, or RFD. Back then, the goal was exactly the opposite.
- Youden 6y agoWhen and why did news cease being news and start being short stories and opinion? This entire article could have been cut down to the last few paragraphs and nothing of value would have been lost. Look at The New York Times in 1921 [0]. Generally the stories are factual and to the point. The entire front page seems to be pure news. There's very little storytelling here, at most there are a few timelines of events. Look at The New York Times today [1]. There's a bunch of factual and useful Coronavirus information but ~15% of the page is dedicated to "Opinion", the second article appears to be pure speculation, the third article is a bunch of storytime fluff around a little bit of news and the front page has a mix of actual news and opinion pieces being passed off as news. When did this happen? Why? Did people lose interest in actual news? Is there less actual news to report? Perhaps this is regional? Take for example the story about the San Quentin prison. NYTimes [2] has the same drawn out nonsense as this Google story while Aljazeera [3] adds a lot of background but sticks to factual reporting. [0]: https://archive.org/details/NYTimes_jul16_31_1921 https://archive.org/details/NYTimes_jul16_31_1921 [1]: http://archive.is/oiiXU http://archive.is/oiiXU [2]: https://www.nytimes.com/2020/06/30/us/san-quentin-prison-coronavirus.html https://www.nytimes.com/2020/06/30/us/san-quentin-prison-cor... [3]: https://www.aljazeera.com/news/2020/07/san-quentin-prison-sees-600-coronavirus-cases-5-days-200701192059040.html https://www.aljazeera.com/news/2020/07/san-quentin-prison-se...
- Aloha 6y agoOpinion masquerading as news sells, see Fox News, MSNBC, et al.
- pgbi 6y agoBecause pure news are pretty much free today. So newspaper need to provide something else: analysis, opinions, etc. to be able to sell something.
- kulahan 6y agoNot to mention we have metrics which tell us exactly what kind of reporting is most likely to have good reader engagement (and thus higher ad payouts). Most news sources are just giving us exactly what the majority of people want.
- s1k3s 6y agoIs this supposed to intrigue me? Good bot
- mlvljr 6y agoThat basta*d.
- amelius 6y agoIt would be cool if Google could manage to become a storefront for the entire web, thereby eliminating Amazon.
- ksk 6y agoBetween the two, personally, I would rather an advertising company like Google be eliminated, or at-least be regulated to protect user privacy. For me, it's easy enough to avoid Amazon, harder to avoid Google since every website uses them to spy on users.
- belval 6y agoYou and I have a very different definition of "cool".
- murgindrag 6y agoFor Google (or anyone) to become a storefront for the entire web, they'd need to handle scams (and errors) well. eBay is a cesspool. Aliexpress is worse. Random web sites are bad. Amazon isn't perfect, but it's better. Amazon also has customer service; they've always made me whole. Random web sites, I'm basically SOL. Aliexpress and eBay are random. Someone flips a coin, heads seller wins, tails buyer wins, regardless of who the scammer is. I mostly buy from Amazon since my odds of not having problems are that much higher.
- ihumanable 6y agoExactly this, the customer service for the average consumer from Amazon is very difficult to beat and is Google's biggest weakness. Bought some cables from Amazon Basic, one ended up not working, another had some cosmetic damage but works fine. They refunded both, sent out replacements, and just told me to discard them, it wasn't worth it for Amazon to pay to have it shipped back. Of course if you abuse this too much Amazon will ban you. If you are an honest consumer though, their customer service generally provides a great experience. I still remember a time when everyone was afraid of purchasing stuff over the internet, Amazon has so greatly reduced the friction and concern that sometimes I find myself going from "hmm, I need something" to "it will be here tomorrow" in the matter of a minute or two. Although more competition in this space would ultimately benefit the consumer, it seems unlikely that Google is going to be the source of that competition. They've got shopping results integrated into their search engine, and it's a feature I've maybe browsed from time to time, but I often just end up searching and purchasing on amazon directly. I don't know if I would be super comfortable purchasing from Google in the same way that I am with Amazon, too many horror stories of App Developers / YouTube Creators / etc getting caught in some sort of Machine Learning Customer Support system. Curious if others use the Google Shopping thing in the search engine and what their experiences are with it.
- Alupis 6y agoThe real problem with this is from the merchant side of things. This bot generates thousands of "Abandoned Carts" on one of our sites... thousands... We send cart reminders to Abandoned Carts after a few days, sometimes with a coupon offer to complete checkout. This bot is responsible for thousands of bounced emails each week, which impacts our metrics with Mandrill among other things. Maybe we shouldn't care, but it's sloppy and ruins all sorts of stats we keep track of regarding cart abandonment rates, recapture rates and more.
- px1999 6y agoDo your users consent to contact before you send them reminders or coupons? If not, you've earned your bounce rate.
- Alupis 6y agoOf course we have consent. Not sure what kind of question that is? Violate CAN-SPAM Act and risk a $16,000 fine per instance? No legit business is going to do that. Just ask Papa Johns how painful those fines/settlements can be[1]. [1] https://topclassactions.com/lawsuit-settlements/lawsuit-news/4146-papa-john-s-agrees-to-16-5m-text-spam-class-action-settlement/ https://topclassactions.com/lawsuit-settlements/lawsuit-news...
- cortesoft 6y agoI personally find the 'email abandoned carts' behavior to be a dark pattern
- Alupis 6y agoDark pattern or not, it's super effective - particularly when accompanied with a coupon ;) Besides, the user is opting-in to receiving these emails. They don't have to provide an email address - so some are probably playing the game and seeing if they get a coupon or not. As an aside - if the internet worked the way SV hipster brogrammers thought it should work, nobody would use it. Yes, ads are crazy effective - you can ignore or block them, we don't care because enough people don't block them and are happy to click. Yes, emails are crazy effective - you can ignore or opt-out or never opt-in, we don't care, you just cost us money if you're not engaged anyway so we'd rather you not be on our mailing list.
- Keyframe 6y agoI think I've seen most Google's technologies dissected and/or explained in detail over the years. Lots of their own papers too. If you look into how and what they're doing regarding data collection, including scraping, there's nothing.
- hbarka 6y agoDidn’t some #tiktokteens do the same with some guy’s web store?
- moneywoes 6y agoSorry, what is the context here?
- doe88 6y agoI wouldn't fault them for that, I've observed some sites most likely are gaming the system by detecting and providing Google bots with artificially lower prices so that they would appear in indexes summaries and then when you access the product, its real price is always higher than the one reported in the index.
- dylz 6y agoyep, I see this type of behaviour constantly - faked prices for Gbot, fake prices on Cache, significantly higher price for end user. It's also infuriating to sort by price and get inflated fake shipping prices to "make up the total"
- ycombonator 6y agoGoogle product Growth hack: Fake it Until you make it
- aaron695 6y agoDigital shopping cart abandonment/Inventory Exhaustion/Hoarder bots is an interesting type of DDOS. There's a popular moment of people using it atm https://heavy.com/news/2020/06/shopping-card-abandonment-tiktok/ https://heavy.com/news/2020/06/shopping-card-abandonment-tik...
- fukpaywalls 6y agoPlease do not post articles that require paywalls!
- tudorw 6y agoNice, I think it has my CC details )
- madmax108 6y agoI used to work at a company that provided APIs used for search/personalization/autosuggest for a whole bunch of huge e-commerce companies. Since the entire integration with the customer site was API based, we worked off of tracking pixels, API requests and cookies to determine shopping behaviour. A lot of this went into determining things like ranking (If someone searches "Tshirt" what shows up on the first page and in what order etc.) Since we were only running search and not payment processing, the tracking pixel/API for "Add to Cart" was a big thing for us. The whole product ran on revenue-share so we were paid per X ATCs Interesting to see if any of the customers were affected by bots doing ATC and how it was handled if it was.