7 ms·
Show HN: AI-powered API that instantly obtains website information
I built SiteProfile(siteprofile.io) to simplify how we access detailed website information. The main feature of SiteProfile is that you can obtain all relevant website information with a single, simple API call. With one API call, you can immediately get the following information:
1.Real-time Webpage Screenshots
Instantly capture real-time screenshots in both PC or Mobile views.
2. AI-Generated Content:
Generate content based on user prompts and website data, e.g. Describe the core functions of this website.
3.Comprehensive Website Info:
Social Media Links, Contact Info, Basic Details, and Assets, all in one place.
With just this API, you can instantly create an ai directory or similar website.
SiteProfile is currently in the testing phase. During this period, all subscriptions are discounted. There are still a few minor issues, and I’m working hard to fix them.
If you try it out, please let me know if you found it useful, if you have anything you’d like me to add, or if you have any other feedback.
And if you have any questions, I’d be happy to answer. Much appreciated.
- nhggfu 2y agodoes your bot respect robots.txt directives? pray tell, what is your bot's user agent string, [so i can nicely block you from my web-properties.]
- kianworkk 2y agoThe goal of SiteProfile is not to scrape data. It only accesses publicly available web pages, such as the homepage, about page, and pricing page. It does not access non-public content on websites, nor does it offer users the functionality to scrape website data.
- pavel_lishin 2y agoYour statement doesn't answer their question.
- kianworkk 2y agoI meant that it is not supported yet. I will add this to the to-do list, and I believe it does not conflict with the goals of SiteProfile. Thank you for your feedback.
- throwaway211 2y agoIt does scrape the site in order to summarise it, no?
- _heimdall 2y agoWhat does it mean exactly for the service to provide information about a website without scraping it? How could summaries or LLM responses be generated be made without scraping pages?
- dotancohen 2y agoPresumably the same way that Firefox makes an HTTP request to the webserver then formats the page for the human user. This is just formatting that page differently. This is no more a scraper than is Firefox's Reader Mode. That said, lying about the UA is not cool.
- Animats 2y agoI have something that sends a UA of "Sitetruth.com site rating system". Many sites won't talk to that.
- _heimdall 2y agoI've used a reader mode library that I think as created by Mozilla and handles converting a site to reader mode locally. Does the Firefox browser do it locally, or at least on demand? If so I wouldn't really consider that scraping since they aren't parsing the site and storing data for later use.
- nhggfu 2y agoso you scrape + store + process contact info etc, presumably. sounds like a privacy nightmare no doubt this is not GDPR compliant. no doubt this is not legal in some parts of the world - unless people can opt out and get their data removed.
- CaptainOfCoit 2y ago> no doubt this is not GDPR compliant. Unless the project is open source, no doubt you cannot know this. If they don't store any of those details anywhere (including not in logs) but just pass it along, GDPR won't apply.
- CaptainOfCoit 2y ago> does your bot respect robots.txt directives? Would be a bit strange if it did, as the service is not a crawler/robot by any measures. Bit like asking if cURL is "respecting" robots.txt. It's just another user-agent after all.
- dumbfounder 2y agoIt is a service that seems to crawl a website for content and feed that content into some LLMs. It should absolutely respect robots.txt. This is exactly what robots.txt is used for, to tell automated crawlers of a website what they should and should not do.
- IncreasePosts 2y agoI disagree - this is not a crawler that just blindly stumbles around any random website that it finds. It is more akin to a user agent. The only requests it makes are derived from specific instructions by the user to do so. Having said that, people may use it as a crawler, just like you might be able to script Firefox to be a crawler, but it is not in itself a crawler.
- dumbfounder 2y agoIt doesn't need to be blind stumbling around the web. But you might be right about only grabbing one page, and if you are then I agree that abiding by robots.txt is only going to upset a tiny minority. When they talk about websites it makes me think they are crawling to see all the pages linked to the homepage, because the asking questions part is extremely limited if all it does is look at one page. If they crawl, then I think they need to abide. If they don't, I think it's ok.
- CaptainOfCoit 2y ago> It is a service that seems to crawl a website for content and feed that content into some LLMs. It doesn't seem to work like that at all, to me. As far as I understand, you give it a specific URL, and it extracts content from that URL and that URL only. A "crawl" would mean it would also follow links automatically, which I don't see any evidence of being done, from the landing page at least.
- oefrha 2y agoI tried with a nonexistent path, here are the user agents I got: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36 It's apparently built to evade detection.
- moneywoes 2y agowhat sort of evasion technique is being used here, random user agents on each attempt?
- CaptainOfCoit 2y ago> what sort of evasion technique is being used here Using a user-agent that looks like a desktop user-agent, rather than including the name of the actual project/product that is being used. I understand why though, plenty of websites block anything that looks like non-desktop/mobile user-agents, so makes sense. Besides the pragmatic reason, I also agree with menacingly that it doesn't matter, people should be able to use whatever user-agent they want to access the content you've put publicly on the internet.
- menacingly 2y agothe more reasonable assumption is that it uses a third party crawl service. but still, it doesn't matter, it's acting on behalf of a user, and you aren't entitled to know what software your users run. One consequence of putting stuff on the public internet is that it's like public.
- pavel_lishin 2y agoThe link to "Can’t find the answer you’re looking for? Reach out to our customer support team." just takes me to the top of the page. And I have to sign in to see the Privacy Policy or the Terms of Service?
- kianworkk 2y agoSorry, this is a bug. I will fix it as soon as possible all my focus is on building the API.
- cbsmith 2y agoHrm... "Application error: a server-side exception has occurred (see the server logs for more information). Digest: 2269195897" That was during sign up.
- kianworkk 2y agoYes, today is the first day of testing for SiteProfile. I have seen the issues everyone is encountering in the logs, and I will fix them as soon as possible. Thank you very much for trying it out.
- erie 2y agoHow is it different from or better than builtwith.com ?
- kianworkk 2y agoBuildWith's goal is to tell you what technologies a website is built with. SiteProfile's goal is to provide you with the meta information of a website. Just like getting to know a person, I created a profile for websites, including images, social media links, and other information so you can quickly understand the site.
- erie 2y agoGreat, I could see that now. Thank you.
- menacingly 2y agoI like the idea. Looks like you've got some kinks, happens to the best of us. I look forward to playing with it when you get those ironed out. Also, I don't care if you respect robots.txt.
- kianworkk 2y agoI am working to fix them. Thank you very much for trying it out.
- menacingly 2y agowe've all been there, go get 'em
- 9cb14c1ec0 2y agoWow! This is great. I was looking for a service with this exact feature set. Keep up the good work.
- kianworkk 2y agoI'm very glad you like this idea. I think we should continue to add more website meta information in the future, such as WHOIS data and SimilarWeb traffic, but I'm not sure if everyone needs this.
- 9cb14c1ec0 2y agoI don't. I just need the AI summary-of-a-website service as an API that I can call.
- meiraleal 2y agoWtf is this?
- brian_herman 2y agoYou should make the json graphic on your homepage bigger I can't read it even though I made it bigger.
- kianworkk 2y agoI completely agree, web design is very challenging for me. I am trying to create a better hero image...
- mutant 2y agoI've done things like this adhoc by just feeding gpt html. Anyone can do this.
- moneywoes 2y agoI don't understand, does this just provide LLMM type questions on a website?
- sneak 2y agoI have actually this week been building a similar API for my own internal use. Cool project! Any intention of open sourcing it?
- kianworkk 2y agoI'm not sure yet, but I might consider it in the future
- Animats 2y agoSite tries to access "quality-sawfly-29.clerk.accounts.dev", rejected by the EFF's Privacy Badger.
- max__dev 2y agoLooks pretty similar to ez-extract.com, but more focused on metadata?