24 ms·
I had to build a web scraper to buy groceries
- ineedasername 7y agoAmazon Fresh (until a few days ago) was releasing slots 3 days out each night at midnight, so I've been saying up to place orders. Now it seems incremental throughout the day, but I've gotten lucky and nicked a spot. However, they don't have everything I need even normally. Now, it's even worse, so I have to order from multiple places. The local Mart's that deliver are still opening their slots at midnight, though usually for 8 days out, and again items ordered often go out of stock before delivery. It feels like a sureal video game.
- zxcvbn4038 7y agoStill better then Wal-Mart’s site where everything is in-store pick up, and Target’s site where everything is out but you can’t tell until you click into the item and try to put it into a basket. My only gripe with Amazon is that it’s too hard to distinguish between them and their third party sellers, and it’s a pain to deal with the sellers. I also wish there was a button to report price gouging - I was looking for some emergen-c packs to flavor my water, normally it’s cheaper then cool-aid and less hassle to mix, some fool is charging $109 for a box of thirty packets.
- fao_ 7y agoAmazon profits off price gouging, though, because they get a percentage of the profits. And since price gouging isn't strictly illegal -- it's seen as a perfectly valid part of capitalism by the state -- there isn't really any way to disincentivise it.
- ineedasername 7y agoYeah, emergen-c is either sold out or price gouged, a pattern seen across most vitamin-c products. But take a look at this-- it's a powdered form that can be mixed. Water works, but OJ might taste better. And the price per dose is very good: https://www.amazon.com/Ascorbic-Antioxidant-Smoothie-Pharmaceutical-Friendly/dp/B01HFOJBG2 https://www.amazon.com/Ascorbic-Antioxidant-Smoothie-Pharmac... As of this comment at least, it's in stock, about $28, there are other brands available too (search vitamin c powder) and it's at least as price-efficient per dose as normal pills/chewables. Also, if you have Amazon Fresh, Amazon's Whole Foods, or Amazon Pantry (just opened again) available in your area, 3rd party price gougers aren't really an issue.
- somebodynew 7y agoIt seems like a lot of demand issues highlighted by the pandemic could be solved by queues. Instead of saying "there are no slots available", put the user in the queue. Process the queue in order. The improvement in wait time and demand accuracy would be a win for both consumers and the business.
- megous 7y agoSome websites do put you in a queue to get to the actual website/shopping. You don't even get to access the website until it's your turn. Like waiting in front of a shop. Not a great experience if you want to just access your account, or some general information on the website, like contact info, or delivery times.
- xzel 7y agoWeb queues are ineffective as shown by sneaker bots. You’d need actual ID style authentication of accounts. If you’re talking about order queues than people will stuff the queue and cancel their orders or just have more hoarding if their multiple deliveries get processed. If it’s FIFO same technical advantages will apply: bots can click much faster than humans ever will. I think the issue is more human nature and things moving to the web creating inequality between the technically adept and non-adept.
- toomuchtodo 7y agoRequire SMS or callback confirmation of the order slot using a non-Google/Twilio/SIP number (mobile phones or landlines both work here). Provide for one time order cancellation at no cost, cancellations after that incur a $10 cancellation fee. This should negate any technical advantage from a sophisticated entity. Queues still work, if managed properly.
- amelius 7y agoCancellation costs may be against local consumer laws. But I think you wouldn't need it with upfront payment and some simple anti-hoarding rules. Also the delivery address or payment id could be used in a way to make this fair.
- Brakenshire 7y agoYoung healthy people shouldn’t be doing this, the slots should go to the ill and elderly, people who are in immediate danger from catching the illness.
- capableweb 7y agoPeople will do what the system forces people to do. You don't want people who are in danger to starve? Let's change the system to one where we help those in need. The current way of living, in many parts of the world (especially the US), is that you have to eat others, otherwise they'll eat you. This is the system the leaders keep defending, and the people keep cheering the leaders on. So you get what you want. Otherwise, go out on the streets and show you're not happy with it. Getting people who are writing scrapers for groceries to stop is not realistically gonna change anything.
- nsnick 7y agoTo add to this, if the sick can’t get grocery delivery, they will go to the grocery stores.
- jbm 7y ago38% of the people hospitalized with COVID-19 are between 20 and 54. [1] Everyone can catch the illness, everyone can wind up in the hospital, and depending on the state of the local hospitals, everyone is in immediate danger. [1] https://www.msn.com/en-ca/health/medical/close-to-40percent-of-hospitalized-covid-19-patients-in-the-us-are-aged-20-to-54-report/ar-BB11r1LN https://www.msn.com/en-ca/health/medical/close-to-40percent-...
- nsnick 7y agoBut not everyone can get grocery delivery, so who should have priority?
- gruez 7y ago>38% of the people hospitalized with COVID-19 are between 20 and 54. [1] This is odd statistic to give, considering it's not normalized by how many 20-54 year olds are there. So I clicked through to the original study[1] and found a more helpful figure: people between the ages of 20-44 have a hospitalization rate of 14.3%-20.8%, and an ICU admission rate of 2.0%-4.2%. [1] https://www.cdc.gov/mmwr/volumes/69/wr/mm6912e2.htm?s_cid=mm6912e2_w#T1_down https://www.cdc.gov/mmwr/volumes/69/wr/mm6912e2.htm?s_cid=mm...
- Swizec 7y agoI’ve just been going to the grocery store as usual and it works great. You show up in the morning and there’s no line yet and shelfs are full. Later in the day they meter entry so you can keep distance inside. Don’t be that ass who makes others do your shopping just coz you’re too afraid to.
- klipt 7y agoIn store: you need to have someone stock the shelves, someone walk through the store and pick groceries, and someone handle the payment in person. Chance of infection: depends how long since someone coughed in the store. Online: you need someone to stock the orders and wait for people to drive up for curbside pickup. Chance of infection: low. Surely the second option is both safer and more efficient?
- 6510 7y agoIf you work in a supermarket you know exactly where each product is. Standing in line is also a huge waste of time.
- Swizec 7y agoA lot of online options send a person who goes into the store and walks around to get all your stuff. Instacart for example.
- PeterisP 7y agoAh, that's quite inefficient (and probably costly) because it's a third party without integration in the logistics chain. I've tried food deliveries provided by two local major grocery chains. As far as I understand, one of them has the stock-shelving people pack the deliveries (so, before hitting the shelf), and the other seems to pack pack the direct deliveries in the same logistics centres from which they supply their smaller shops, so that stuff never even gets delivered to a store; and the limited overhead means that they offer free delivery for non-small purchases.
- squeaky-clean 7y ago
- vitiell0 7y agoWe recently ran into this problem at Cooklist in the US. We ended up setting up the ability for our users to place orders up to 3 days out. Our system then books their pickup order at midnight when the new pickup slots are opened. We are seeing that most pickup slots are filled for the next day by 9am. It is interesting that the stores don't allow booking several days in advance anymore. It makes sense though because even with booking for the next day we are seeing that 25% to 50% of the order is substituted due to out of stocks.
- 6510 7y agoGreasemonkey seems a much better weapon.
- op00to 7y agohow?
- banana_giraffe 7y agoI had to do the same thing to get an Amazon Fresh order. Well, not quite the same thing, I ended up writing an Selenium project that automated the Amazon Fresh checkout process till it got to a delivery window page that didn’t complain there were no slots available. If I can’t get groceries, it’s not the end of the world for me, since I can call on family if necessary, but I’d rather not make their stressful life any worse right now, since I’m not the only one in my family’s circle that needs help. There are no great answers, since I might have taken a slot from someone that can’t write a web tool to scrape Amazon, and doesn’t have a family to fall back on. It sucks on many levels.
- ct520 7y agoI couldnt find an appointment slot with safeway this past week. Ended up writing a quick script on lunch that did the trick. Ran it as a schedule task to run every 5 mins. Pretty hacky and unpolished but worked, got a text message a day later and checked out. Not sure if anyone could use it but here it is. https://github.com/christru/safeway-delivery-check https://github.com/christru/safeway-delivery-check
- deleted 7y ago[deleted]
- reaperducer 7y agoOf all the grocery companies in my area, only Walmart has it right: Choose an available pick up time, then start online shopping. All the others let you fill your cart, then at checkout tell you there are no slots available until the end of time. Waste of time, energy, and hope. F U, Target, Safeway, Albertsons, and Kroger.
- tzs 7y agoWalmart is also tied for my top spot for in-person shopping during this. Walmart gets that spot because of their mobile app which does a pretty good job of telling me what is in stock at my local store. My other top choice is a small (three store) chain in Western Washington called Central Market [1]. (If the name seems familiar, it may be from season 10 of "Top Chef" which was set in Seattle. Top Chef normally sends the contestants to Whole Foods for most shopping, but this was before there were any Whole Foods in the Seattle Area. They used the Shoreline Central Market instead). They don't have a mobile app or online way to check stock that I know, but they seem to be doing the most of any grocery store I've seen to deal with the virus when it comes to in-store shopping. 1. They've got separate cart return and cart pickup areas in front of the store. Every cart returned gets disinfected before being put into the pickup. 2. They have installed plexiglass shields in the checkout lanes between you and the cashier. 3. They have prominent marks on the floor for you to stand at while inline that provide good spacing. 4. After each customer checks out, they disinfect the conveyor belt area and the credit card terminal area. They are also nowhere near as crowded as Walmart, and the people that shop there seem to be paying more attention to social distancing. [1] https://central-market.com/ https://central-market.com/
- skookum 7y ago> ...this was before there were any Whole Foods in the Seattle Area. I'm not sure where this story originated. The first Whole Foods in the Seattle area* predates the filming of that season by about a decade and is closer to downtown Seattle than any of the Central Markets. (*the Roosevelt Square store, still at the original location)
- markkanof 7y ago
- omneity 7y agoShameless but very relevant plug: Monitoring a website for changes and sending data somewhere else is exactly what we do at https://monitoro.xyz https://monitoro.xyz It’s the difference between maintaining a bunch of scripts and making sure they don’t stop working abruptly, and having the data you wanted without headaches. It also wouldn’t make for such an elaborate article, but that’s the cost of simplicity if you just want to get things done.
- jve 7y agoFew months ago someone had posted open source scraper that got 1k+ votes (or maybe not) - it was used by NYTimes for realtime alerts. Someone remind me please.
- top_kekeroni_m8 7y agoI’ve done a few of those scrapers and mail to notify projects in the past, but I’ve recently switched from using email to a telegram bot, it’s pretty great. Very easy to implement and doesn’t spam your inbox.
- punarinta 7y agoHehe, I had to do the same a week ago. The only difference is Slack’ing instead of mailing and also cart prepopulation because those delivery slots are taken out almost instantly in my city. This is in the EU btw, not specifying the country or chain to, well, to be able to survive longer.
- polote 7y agoTips: the first session cookie is probably returned if you do a GET on the homepage of the website, so you can simplify your script by Declaring the requests session, doing get on the homepage and then doing get on the endpoint It will also prevent issue with session expiration
- rayuela 7y agoHey do you have any suggested readings on how to better understand these structures and your reasoning?
- mettamage 7y agoI've done the same thing in The Netherlands, as I'm doing the shopping for my grandparents. I noticed that I don't need it yet, but I did see that I might need it in the future since I need to stay awake until 23:59 at night, so I can snatch a slot at 00:00 at the most famous supermarket here. My script isn't fully ready, but it's 75% complete and I'll complete the final 25% when I really do need it. Also, these things are quite easy to make, so I expect a lot of techies to eventually do this.
- YeGoblynQueenne 7y ago>> everybody stays at home and especially avoids crowded indoor places like supermarkets. I know I shouldn't do that, my doctor has advised me against it, but I have a headache today and this made it worse and I can't help myself so: this is an oxymoron. If everybody is staying at home, then supermarkets cannot be crowded. If supermarkets are crowded, then everybody is not staying at home. Both things can't be happening at once.
- IshKebab 7y agoEverybody stays at home except for essential shopping in supermarkets. Not every statement is meant to be taken 100% literally, but I'm guessing you knew that and are being pointlessly pedantic.
- YeGoblynQueenne 7y agoYou mean you're guessing I'm a pointless pedant? Thank you for your contribution to my Sunday headache.
- jniedrauer 7y agoI think we've all underestimated just how many people "everybody" is. Normally we're spread out. Now we're either at home, the park, or the grocery store, and suddenly it seems more crowded even though "everybody" is staying at home.
- ant6n 7y agoNobody drives in New York - too much traffic!
- adamleithp 7y agoI literally build the same thing here in the UK. I knew I couldn't have been the only one lol. Good job.
- kwhitefoot 7y agoWould be simpler to just plan four days in advance. Shopping used to be done once a week, or even once a month for bulk and frozen goods. Now it seems to a be what people do instead of going for a walk in the countryside. Makes even less sense now that you don't have to leave your bed to do it.
- IshKebab 7y agoThey didn't mean that there were free slots after 4 days.
- danieldk 7y agoMaybe there are local dynamics at play (we are in a city in The Netherlands). But my wife has been doing the grocery shopping (since I don't have a driver's license). We would normally shop every day, but she is now going every 3-4 days. And after the initial 3-4 days of hoarding, the grocery stores are far less busy, probably because people are buying for multiple days at a time. She has had no problem keeping the necessary 1.5-2m distance.
- 40four 7y agoWhy is grocery delivery considered a ‘safer’ option than going to the store yourself. I actually prefer to do it myself. I’d rather be the one to lay hands on all my own products. Why would you trust an unknown group of people to take all the proper precautions? Who picked the item by hand? Who is this bringing it to your door? Are you sure they washed hands? Did they accidentally sneeze on anything? Is the vehicle sanitary? Also, I would imagine people doing these jobs are probably at higher risk of being exposed, just by nature of traveling & being in public all the time. I would rather trust my own skills to be responsible and protect myself, rather than other unknown entities to touch all my stuff. All the supermarkets I’ve been to have someone at the door sanitizing carts as the hand them too you. I am hyper aware of avoiding touch anything unnecessary. I can keep as much distance from people as I want. I know my hands are clean when I start, and then avoid touching my face until I sanitize after.
- mderazon 7y agoHow do you trust the employee who restocked the shelves with all the groceries or filled the stands with fruits?
- halfmatthalfcat 7y agoThe point is you’re eliminating a completely unnecessary vector for getting infected. The risk of getting infected from within the grocery is assumed in both scenarios.
- cogman10 7y agoI think the flip is more the reason to do delivery/curbside. Symptoms aren't always present so you going into the store could be exposing many people to infection. Delivery isolates. Yes your driver and handler could be infected, but so could you.
- 40four 7y agoThat’s true, but that can’t be eliminated.
- chmod775 7y agoThe picture with his cookies contains the suspicious keys "hphone" and "hmail", which likely show md5 sums of his phone number and mail respectively. Especially the phone one can easily be brute-forced (small search space). I wonder why they hash this in the first place? I hope they're salted...
- e12e 7y agoI must say, from the title I was expecting to be disappointed - but this is a very nice and clear example of a useful little hack combining python requests module and a little web page introspection. I think maybe I'd have gone down to every 5 minutes, to be a little nicer to the end-point (especially when sharing the code) - otoh once a minute isn't to much of a hammer, I guess.
- mymythisisthis 7y agoThe supermarket is a post WWII invention. Labour was expensive, so why not have the customer walk around and pick. Typically before WWII shop clerks handed items to waiting customers across the counter. As any invention it can be reinvented. Organic farmers started doing deliveries 20 years ago, to compete with the big warehouse boxes.
- deleted 7y ago[deleted]
- qiguai 7y agoThe best part of "blood in the streets" is old school hacker news posts like this. Kpi of creative progress to come!
- xarthna 7y agoEveryone should be ordering online regardless of age or increased risk. Limiting contact with individuals is the only way to stop the spread of this disease. The answer is that the surge in demand for delivery should quickly be filled on the supply side by delivery companies. The increase in business should go to providing hazard pay, PPE, and benefits for the supplying workers. This limits the number of people in contact with others while supplying jobs for those who have been unemployed during this time while increasing pay and benefits for those workers due to their increased risk vs reward. This is how capitalism is supposed to work in theory at least, to quickly adjust to market needs.
- chris1993 7y agonext, booking your grocery pickup as a service /s
- pmuk 7y agoI had a similar idea on Saturday and wrote a puppeteer based script which monitors Tesco.com and then pushes a screenshot to my phone if it finds a slot: https://github.com/paulmaunders/delivery-slot-bot https://github.com/paulmaunders/delivery-slot-bot
- rayic3 7y agohey - just tested your code - works great. I am no coder, but have installed and it runs fine. Quick question 1) Its now only checking for Group-1 which is 1hr fixed. Would like to include Group-4 which is the flexible slot. 2) Not just send notification - book the earliest one available. How to do this?