11 ms·
Show HN: Page Replica – Tool for Web Scraping, Prerendering, and SEO Boost
- quickthrower2 3y agoCool! I thought about this idea as an IaaS! A CDN that can figure out your rendered page and serve it. Regardless of tech stack.
- nirvanist 3y agoThx , and this is the idea
- ssgodderidge 3y agoCurious, how would caching the pages and serving over NGINX help with SEO? Is there any benefits over serving a static site?
- nirvanist 3y agoYes, Google can render your content handled by JavaScript, but Googlebot allows only a few milliseconds for rendering. If your page isn't rendered within that time frame, it may be penalized. For this reason, many news and broadcasting media outlets still use prerendering services. I speak from experience as I worked in a large Canadian media company. Another important factor to consider is that the SEO world isn't limited to Google. Various bots, including those from other search engines and platforms like Facebook, require correctly rendered pages for optimal sharing and visibility. Lastly, the choice between client-side rendering (CSR) and server-side rendering (SSR) depends on your specific needs. Google Search Console provides valuable metrics and information about your app, so it might be worth considering SSR if that better aligns with your requirements.
- nodja 3y agoIf your web app serves dynamic routes (i.e. client only) this helps with SEO because those routes are now directly visible through most crawlers.
- nirvanist 3y agoroutes yes but maybe not content, most of the time google search console will show that you have no content , also google ads boot need to access to content this useful to get your adsense account quickly validated
- jotto 3y agoA "static site" implies HTML rather than a JavaScript app. With respect to JavaScript apps (React, Angular, etc.): It's not clear these days because the major search engines don't explicitly clarify whether they parse JavaScript apps (or if they only parse high-ranking JS apps/sites. But 10 years ago it was a must-have to be indexed. One theory on pre-rendering is it reduces cost for the crawlers since they don't need to spend 1-3s of CPU time pre-rendering your site. And by reducing costs, it may increase chances of being indexed or higher rank. My hunch is that long-term, pre-rendering is not necessary for getting indexed. But it is typically still necessary for URL unfurls (link previews) for various social media and chat apps. disclosure: I operate https://headless-render-api.com https://headless-render-api.com
- nirvanist 3y agoabsolutly , bots still need to access content as faster as possible
- colesantiago 3y agoIt seems in the AI era SEO is now starting to become irrelevant and a relic of the 2000-2020 era. Why is SEO needed still needed here when AI / LLMs can just conjure up answers with references to valid links, bypassing search engines. Even privacy based search engines like DuckDuckGo, Brave and Kagi doesn't prioritise 'SEO'.
- quickthrower2 3y agoLong term you may be right. SEO isn’t dead but it will slowly die off. Being a reference link is second place but by that point you only get visited if the AI wasn’t trusted or didn’t solve the problem. Therefore I think viral/word of month or links from other engaged sources will become more relevant. Right now though why lose out on free SEO traffic just because you used JS to render most of your site?
- paulcole 3y agoIn the AI era, which services provide up to date information about local queries — like which dentists near me are open today?
- wongarsu 3y agoI still use Google (and ddg and kagi), so people who want to sell me stuff try to get better rankings in these search machines. I'd also wager that people who primarily use LLMs to answer their questions are only a rounding error.
- mrtksn 3y ago[flagged]
- pmx 3y ago> In places where JS is truly useful, that is when the UI is more than a text document, SEO is not a concern or possibility. I don't think you're thinking about this. e-commerce is more than just a document and SEO is massively important there but javascript makes the user experience miles better; Think product variation selectors, bulk pricing selectors, product filtering, realtime cart, etc, etc. It's insane to say we shouldn't use new tech so that the search engines can index us, do we just forever stick with what we had 15 years ago and never progress? Madness.
- mrtksn 3y agoI should have been clearer: I'm not against JS, I'm huge fan actually but I don't see the promised lands of better user experience with having these overly complex architectures where the UI is JS heavy and has to be pre-rendered on the server side so that the Search engines can make a sense of it. Larger and larger, heavier and heavier JS never delivered those perfected UI experiences that Web technologist promised. The Web technology folks still circle around the same problems like 10 years ago when the actually popular and successful websites like HN and others deliver superb UX with the dinosaur tech. Anything that is actually using the advanced Web technologies are not in the domain of SEO, stuff like true web apps like Figma, Google Docs let's say. At best, you can index list of content for those but that list could have been a JS-free HTML rendered by PHP.
- xnx 3y agoWhat's the use case? Scrape someone else's dynamic site and serve it statically as your own?
- digitcatphd 3y agoCurious
- nirvanist 3y agoand reddit too , thank you for your feedback
- nirvanist 3y agofor me is t not the purpose, but you can do it if you want. the use case for me was that meteorjs app are pourly SEO friendly and I need it to have prerindering html to serve it for bots
- binarymax 3y agoI read it as being able to dev your site in whatever you want, then scrape it and publish it as a static and seo optimized site.
- throwoutway 3y agoThat sounds sort like an inefficient static site generator
- quickthrower2 3y agoSounds like an inside-out NextJS! What you can do with this is design your app as an SPA and have this give a quicker loading experience to any route that is “logged out” so to speak. The problem is in reality you are logged in and what NextJS can do is allow you to define the subset of the page that can be static.
- amadeuspagel 3y agoIt sounds like a hack, which is after all what this website is dedicated to.
- Lukkaroinen 3y agoI wonder where this is actually needed, since most React frameworks support metadata with server-side rendering.
- nirvanist 3y agothank your for the comment while it's true that not all web applications leverage the React library, it's important to note that Next.js inherently supports React. However, the choice of technology stack depends on the specific use case and requirements of your project.
- deleted 3y ago[deleted]
- CharlesW 3y agoIt's not about React specifically, but about whether pre-rendering will make any difference from an SEO perspective. Not only do most frameworks do SSR, but Google is able to crawl dynamic content just fine. Here's an article from 2015 on the topic: https://searchengineland.com/tested-googlebot-crawls-javascript-heres-learned-220157 https://searchengineland.com/tested-googlebot-crawls-javascr...
- ahoka 3y agoGoogle renders all sites in Chrome according to their documentation. So why would you render it? AFAIK they even penalize serving different content to their crawlers.
- nirvanist 3y agosure , but you want aslo your web app available for other bots, and you are not serving other content it same content just you avoid the js layer so it will be quickly serverd, so far for my projects it worked very well ,google serach console showed a better ranking , it s not a new technique and other services like prerender.io do the same things but not free.
- lnxg33k1 3y agohttps://github.com/html5-ninja/page-replica/blob/main/api.js#L45C12-L45C12 https://github.com/html5-ninja/page-replica/blob/main/api.js... This code base has the most useful comments ever, are these normally accepted? Enforced? Adding stuff that has no value, but needs to be mantained and updated when code changes without the ability for it to be validated by compilers or parsers?