4 ms·

The irony of having an ad supported app to scrape recipes from ad supported websites. I'm not sure how I feel about this.

spacemadness 5y ago
Technology companies selling a solution to a problem created by technology companies instead of coming together to fix the original problem. This is the world we live in.
- massysett 5y ago
  The original problem is that people want to Google for recipes and not pay for them. As a cook there are several solutions to this: 1) pay for a subscription from vendors who are in business to sell recipes, such as Cooks Illustrated 2) get recipes from companies that provide them as a complement because they sell something else, generally food-related products. For instance King Arthur Baking has good recipes, and many are available from food companies such as Tyson, Kraft, and Betty Crocker. Grocers also have many recipes. These sources aren't interested in spamming users with ads because the website is one big ad. 3) Buy cookbooks, they're not expensive. But yeah as long as people want to crank search terms into Google and get "free" stuff, it's going to be ad-infested, and then other ad-seeking folks are going to run their shakedown operations just like the adblock extensions charge money to advertisers to let their ads through.
  - dron57 5y ago
    3) buy cookbooks While cookbooks are great they're not easily searchable. If only I could ingest every recipe from every cookbook I bought into a personal database. Hey that'd be a neat app idea!
    - sinbadpalomar 5y ago
      this is actually already a thing. Damn i forget the app but it exists I did the trial once https://www.eatyourbooks.com/ https://www.eatyourbooks.com/
PolandKid 5y ago
@AwkwardPanda And how does a site opt out of your scraping? Do you have a unique user-agent when you scrape? A set of IPs?
- notyourwork 5y ago
  You know the answer to this.
- AwkwardPanda 5y ago
  Hi, currently there is nothing of this sort. The user agents are random. I have a couple of servers doing the scraping in real-time. The IPs are not static. Let me see if I can build a opt-out list. But wouldn't it beat the entire purpose of this app?
  - PolandKid 5y ago
    You should just declare your bot as a user-agent. Most publishers won't even bother to do it, but leaving a publisher an option is the correct etiquette for any bot. Random user agents is cloaked scraping.

spacemadness 5y ago

Technology companies selling a solution to a problem created by technology companies instead of coming together to fix the original problem. This is the world we live in.

massysett 5y ago
The original problem is that people want to Google for recipes and not pay for them. As a cook there are several solutions to this: 1) pay for a subscription from vendors who are in business to sell recipes, such as Cooks Illustrated 2) get recipes from companies that provide them as a complement because they sell something else, generally food-related products. For instance King Arthur Baking has good recipes, and many are available from food companies such as Tyson, Kraft, and Betty Crocker. Grocers also have many recipes. These sources aren't interested in spamming users with ads because the website is one big ad. 3) Buy cookbooks, they're not expensive. But yeah as long as people want to crank search terms into Google and get "free" stuff, it's going to be ad-infested, and then other ad-seeking folks are going to run their shakedown operations just like the adblock extensions charge money to advertisers to let their ads through.
- dron57 5y ago
  3) buy cookbooks While cookbooks are great they're not easily searchable. If only I could ingest every recipe from every cookbook I bought into a personal database. Hey that'd be a neat app idea!
  - sinbadpalomar 5y ago
    this is actually already a thing. Damn i forget the app but it exists I did the trial once https://www.eatyourbooks.com/ https://www.eatyourbooks.com/

PolandKid 5y ago

@AwkwardPanda And how does a site opt out of your scraping? Do you have a unique user-agent when you scrape? A set of IPs?

notyourwork 5y ago
You know the answer to this.
AwkwardPanda 5y ago
Hi, currently there is nothing of this sort. The user agents are random. I have a couple of servers doing the scraping in real-time. The IPs are not static. Let me see if I can build a opt-out list. But wouldn't it beat the entire purpose of this app?
- PolandKid 5y ago
  You should just declare your bot as a user-agent. Most publishers won't even bother to do it, but leaving a publisher an option is the correct etiquette for any bot. Random user agents is cloaked scraping.