zachperkitny

3 ms·

1. 1.▲
   by zachperkitny 8mo ago

   Thanks! I've uesd Scrapy before, I like it a lot. This is built around CDP and uses an actual browser so it supports client side rendered content as well. I am adding a feature specifically for static HTML parsing for performance reaso
2. 2.▲
   Tadpole the Language for Scraping 0.2.0 – Complex Control Flow, Stealth and More

   7 points by zachperkitny 8mo ago|2 comments
3. 3.▲
   by zachperkitny 8mo ago

   Domain Specific Language - It's a fairly common acronym I feel like. SQL is a DSL, so is CSS, Rust macros lets you create DSLs, for example. The opposite is General Purpose, like Python, JavaScript
4. 4.▲
   by zachperkitny 8mo ago

   I don't think he understands what CDP is vs Client Side JavaScript.
5. 5.▲
   by zachperkitny 8mo ago

   Not really. Operating with Chrome Devtools Protocol over WS is completely different. You cannot move your mouse, you cannot open tabs, you cannot control the browser in the same way. It's not really comparable, sorry. Here is a referen
6. 6.▲
   by zachperkitny 8mo ago

   I'm glad you like the project! You know that's an interesting idea, NodeJS does support WASM. Compiling Python to WASM is possible. I could add custom actions for importing and executing external code. We'll see, it's gi
7. 7.▲
   by zachperkitny 8mo ago

   Effectively, it's not meant to restrict people from using it, even in a commercial setting, just to protect my personal interests in what I want to do with it in a commercial setting. KDL is more than just JSON or YAML. It's node
8. 8.▲
   by zachperkitny 8mo ago

   Thanks for the idea actually! It's difficult to cover every use case in the 0.1.0 release. I'll take this into account. Downloading Files/Images could likely be abstracted into just an HTTP source and the data sources could b
9. 9.▲
   Tadpole – A modular and extensible DSL built for web scraping(tadpolehq.com)

   63 points by zachperkitny 8mo ago|13 comments
10. 10.▲
    by zachperkitny 8mo ago

    Hello! I wanted to share my recent project: Tadpole. It is a custom DSL built on top of KDL specifically for web scraping and browser automation. I wanted there to be a standardized way of writing scrapers and reusing existing scraper logic