andrethegiant

7 ms·

1. 61.▲
   by andrethegiant 2y ago

   By default, href values of <a> tags are removed, because they add significant token length without adding more context. Coming soon, you can specify a request header to set whether or not you want links removed from the response. Thos
2. 62.▲
   by andrethegiant 2y ago

   With AWS Athena, you can query the contents of someone else’s public S3 bucket. You pay per read, but if you craft your query the right way then it’s very inexpensive. Each query I run only scans about 1MB of data.
3. 63.▲
   by andrethegiant 2y ago

   Much appreciated!
4. 64.▲
   by andrethegiant 2y ago

   Thanks! Still a work in progress :-)
5. 65.▲
   by andrethegiant 2y ago

   I'm working on pure.md[1], which lets your scripts, APIs, apps, agents, etc reliably access web content in markdown format. Simply prefix any URL with `pure.md/` and you get the unblocked markdown content of that webpage. It avoid
6. 66.▲
   by andrethegiant 2y ago

   I don’t have a problem with that. One user said that they sometimes use it simply to make copy/pasting text easier, which I thought was a neat way to hold it
7. 67.▲
   by andrethegiant 2y ago

   Thanks! Send me an email at jason at crawlspace dot dev
8. 68.▲
   by andrethegiant 2y ago

   Thanks for trying it out, will look into that character encoding issue! Edit: fixed :-)
9. 69.▲
   by andrethegiant 2y ago

   Author here, happy to answer any questions
10. 70.▲
    Show HN: Prefix any URL with `pure.md/` to get unblocked Markdown(pure.md)

    31 points by andrethegiant 2y ago|10 comments
11. 71.▲
    by andrethegiant 2y ago

    Check out https://pure.md for a REST API version of this
12. 72.▲
    by andrethegiant 2y ago

    $25 per thousand searches seems excessive
13. 73.▲
    by andrethegiant 2y ago

    Clever! I made a similar emoji encoding/decoding microsite: https://face64.me
14. 74.▲
    by andrethegiant 2y ago

    Amex?
15. 75.▲
    by andrethegiant 2y ago

    This is why I’m making crawlspace.dev, a crawling PaaS that respects robots.txt, implements proper caching, etc by default.
16. 76.▲
    by andrethegiant 2y ago

    Could it potentially fit in a Cloudflare worker? Workers are also V8 and can run wasm, but are constrained to 128MB RAM and 10MB zipped bundle size
17. 77.▲
    Show HN: Crawlspace – A centralized web crawling platform built on Cloudflare(crawlspace.dev)

    9 points by andrethegiant 2y ago|1 comments
18. 78.▲
    by andrethegiant 2y ago

    > Would it bail out if it saw this robots.txt? Yes, it should — I use the library below, and it should split at the slash character, treating it as a prefix match, per spec. https://github.com/samclarke/robots-parser
19. 79.▲
    by andrethegiant 2y ago

    The answer is to not use TikTok.
20. 80.▲
    by andrethegiant 2y ago

    Thanks for the feedback, it’s mentioned in the platform FAQ but I should make it more prominent in the docs. The UA will always be prefixed with the string `Crawlspace`. May I ask why you’d want to block it, even if it crawls respectfully?
21. 81.▲
    by andrethegiant 2y ago

    Even so, traffic funnels through the same cache, so website owners would see the same amount of hits whether there was 1 crawler or 1000 crawlers on the platform
22. 82.▲
    by andrethegiant 2y ago

    CommonCrawl’s data is up to a month old. Here you can write a custom crawler, and have a REST/RAG API that you can use with your apps and agents for the data that your crawler finds. All the while, it builds up a platform-wide cache, s
23. 83.▲
    by andrethegiant 2y ago

    I’m working on a centralized platform[1] to help web crawlers be polite by default by respecting robots.txt, 429s, etc, and sharing a platform-wide TTL cache just for crawlers. The goal is to reduce global bot traffic by providing a conveni
24. 84.▲
    by andrethegiant 2y ago

    Same here!
25. 85.▲
    by andrethegiant 2y ago

    I'm working on fixing this exact problem[1]. Crawlers are gonna keep crawling no matter what, so a solution to meet them where they are is to create a centralized platform that builds in an edge TTL cache, respects robots.txt and retry
26. 86.▲
    by andrethegiant 2y ago

    https://crawlspace.dev has a similar LLM-aware scraping where you can pass a Zod object and it’ll match the schema, but is available as a PaaS offering with queueing / concurrency / storage built in [disclaimer: I’m th
27. 87.▲
    by andrethegiant 2y ago

    > I find the term “agents” extremely frustrating. It lacks a single, clear and widely understood meaning... but the people who use the term never seem to acknowledge that. This 100%. “Agentic” especially as a buzzword can piss off
28. 88.▲
    by andrethegiant 2y ago

    CommonCrawl is supposed to help for this, i.e. crawl once and host the dataset for any interested party to download out of band. However, data can be up to a month stale, and it costs $$ to move the data out of us-east-1. I’m working on a c
29. 89.▲
    by andrethegiant 2y ago

    Dibs on S0
30. 90.▲
    by andrethegiant 2y ago

    FWIW, that’s why I’m working on a platform[1] to help devs deploy polite crawlers and scrapers out of the box that respect robots.txt (and 429s, Retry-After response headers, etc). It also happens to be entirely built on Cloudflare. [1] ht