PigiVinci83

17 ms·

1. 31.▲
   Show HN: Databoutique.com, a Marketplace for Web Data

   3 points by PigiVinci83 4y ago|0 comments
2. 32.▲
   by PigiVinci83 4y ago

   This project is interesting, we at https://www.databoutique.com are building something similar, a curated dataset marketplace for web scraping data. We believe that using standardization, quality controls and high density verti
3. 33.▲
   by PigiVinci83 4y ago

   Since I’m quite experienced in the field, I opened a substack called The Web Scraping Club as a side gig. It is mostly free but has some paid articles and have already hundreds of $ in MRR after 2 months. It is a niche where tutorials and i
4. 34.▲
   by PigiVinci83 4y ago

   Original link to the post: https://thewebscraping.club/p/ikea-scraping-kallax
5. 35.▲
   The great African IP address heist (2020)(mybroadband.co.za)

   10 points by PigiVinci83 4y ago|0 comments
6. 36.▲
   by PigiVinci83 4y ago

   Good stuff, at least there's a sort of solution inside and not only content marketing
7. 37.▲
   by PigiVinci83 4y ago

   Thanks for sharing!
8. 38.▲
   by PigiVinci83 4y ago

   Thank you for this article, it confirms my direct experience. Never run a benchmarking test but I can see this every day.
9. 39.▲
   by PigiVinci83 4y ago

   Just shared some of these stories here https://twitter.com/pigivinciguerra/status/15715437943480893...
10. 40.▲
    by PigiVinci83 4y ago

    Totally agree.
11. 41.▲
    by PigiVinci83 4y ago

    Companies spend thousands of dollars on these anti-bot solutions and then they are so misconfigured that using a specific user agent or faking browsing via mobile, bypasses them. Real life stories.
12. 42.▲
    by PigiVinci83 4y ago

    Thank you for sharing my post, appreciated
13. 43.▲
    The Python Starter Toolkit for Web Scraping (2022)(substack.thewebscraping.club)

    1 points by PigiVinci83 4y ago|0 comments
14. 44.▲
    The Supreme Korean court says that scraping publicly available data is legal(lexology.com)

    292 points by PigiVinci83 4y ago|53 comments
15. 45.▲
    by PigiVinci83 4y ago

    Other news on web scraping world at https://thewebscraping.club/p/a-brief-wrap-up-of-the-latest-...
16. 46.▲
    by PigiVinci83 4y ago

    I’ve got several years of experience of webscraping, mainly in python. Scrapy is the first choice for “basic websites” while playwright is used then things get difficult. I’m collecting my experience in using these tools in this “web scrapi
17. 47.▲
    From 0 to 2Bs prices scraped a month(thewebscrapingclub.substack.com)

    1 points by PigiVinci83 4y ago|1 comments
18. 48.▲
    by PigiVinci83 4y ago

    Some high level principles for scaling a web data company
19. 49.▲
    by PigiVinci83 4y ago

    Yes, it's another post about web scraping
20. 50.▲
    Wanted a new parka, got a HTTP 429 error: a Kasada experience(thewebscrapingclub.substack.com)

    3 points by PigiVinci83 4y ago|1 comments
21. 51.▲
    by PigiVinci83 4y ago

    Oh, i read your substack and find it brilliant!
22. 52.▲
    by PigiVinci83 4y ago

    Thank you, much appreciated your comment
23. 53.▲
    by PigiVinci83 4y ago

    I'm sure no one will add here its secret sauce :)
24. 54.▲
    by PigiVinci83 4y ago

    Thanks for your reply, mobile data it's a thing i need to add soon. Usually we check using Fiddler if there's an API inside, but only for really problematic website.
25. 55.▲
    by PigiVinci83 4y ago

    Having a large codebase like ours, we find out that XPATH are more readable, but i understand it's a personal feeling. We don't have high frequency scraping, so the performances of CSS vs XPATH were not considered. It's an in
26. 56.▲
    by PigiVinci83 4y ago

    Thanks for the question, i can speak for what we've encountered in these years of web scraping and nothing beats API and JSON, but i'm sure there are formats even more friendly to read.
27. 57.▲
    by PigiVinci83 4y ago

    Because it's not a tutorial on web scraping but a mix of what we suggest internally to do and what we've learnt from our experience in this field in these years. For our codebase we prefer tabs instead of spaces, but i understand
28. 58.▲
    by PigiVinci83 4y ago

    Thanks for sharing, i'll update soon the page.
29. 59.▲
    by PigiVinci83 4y ago

    A work in progress guide about web scraping in python, anti bot softwares and techniques and so on. Please feel free to share and contribute with your own experience too.
30. 60.▲
    Web scraping with Python open knowledge(github.com)

    219 points by PigiVinci83 4y ago|74 comments