4 ms·

Next.js v13 websites are a heaven for scraping

81 points by punkpeye 3y ago

Just a random thought while analyzing some Next.js v13 website HTML markup. Due to the way that RSC hydrates elements during client-side rendering, Next.js has to provide all their property data using JSON (see this issue https://github.com/vercel/next.js/discussions/42170) As a result, any website that uses Next.js with RSC is extremely easy to extract data from since you can tap into JSON of every element. Just an interesting observation without implying whether it is good or bad for the ecosystem.

ChrisArchitect 3y ago
Tell HN:
isbvhodnvemrwvn 3y ago
It also shows how awful some sites are. I'm scraping a jobs offer portal written in next.js (although I use the endpoint for hydration), one of the lovely things they do is they pass in props containing all possible skills candidates can have, they send 200KB per listing, less than 10KB is the average volume of job-specific (actually required) data. Previously they were rendering it from a REST endpoint which only had the necessary data, but now I get a lovely dictionary of all possible values of various properties, provided on a silver platter. Another portal I'm scraping uses GraphQL, and they don't have access control, I get all the data they have on impressions, I know how much money somebody paid for listing, I know when the offer is going to be auto-bumped - just lovely. And no need to use Playwright either.
- punkpeye 3y ago
  The irony is that I discovered this also while scraping a job portal. I wonder if we are scraping the same site lol. I am building a job board for tech jobs, but with very granular filters.
  - isbvhodnvemrwvn 3y ago
    Interesting. I was initially intending to do the same, but now I just use the project as OSINT source for research on various companies, if I or any of my friends is looking for a job they get a dump of everything they posted on popular portals in the last 3 years or so.
    - rrrix1 3y ago
      Can I be your friend? :) I'm looking for a job, and just started hacking together a job posts database with data scraped from a few aggregators and a couple of specific companies. Currently it's all just local and for myself, mostly raw data with some basic statistics and keyword similarity heuristics. I decided to try something new and different and take it into my own hands (instead of hoping LinkedIn shows me the "right job"). I'm frustrated by the sheer volume of clearly duplicate job posts which often only vary by the "city" (even if they're 100% remote) and the lack of any kind of basic filtering for programming languages, tech stacks, platforms, years of experience requirements, company industry / size / age etc. I'm really curious what others who have been doing this for much longer have learned! How do you store your data? How do you process it? Have you eliminated manual cleansing and grooming? So many questions...
      - isbvhodnvemrwvn 3y ago
        I'm scraping jobs for a specific country I don't think you're in, but a couple of things: - always keep the original data you scraped in as close to original form as you can (in next.js example, I just strip the unnecessary props - but only those I know are useless, see point 2) - there aren't that many job offers out there, text is easy to compress, storage is cheap. Eventually you will change the idea for what's important, and having original files to work with allows you to re-process them. Kind of like data vault approach in data engineering. - prepare for the APIs to evolve, version your stuff. Have stringent validation (like json schema with enums and not allowing extra fields) to detect this evolution, so that you can adapt. This is extra important for portals with complex data models where you might not fully understand them at first. - understand key fields - it's often the case that "posted date" is auto-bumped every now and then, depending on which package you bought. Try going through buy-a-listing workflow to understand the options. For multi-city listings there is often some sort of way of de-duplicating (like URL slug having a common prefix, or the data containing "main" listing). Spending an hour going through the data you scraped goes a long way. - a lot of the data comes from manual user input or from some imports from the company's ATS. You will need to normalize those values in some way, and it's not always possible to do it in a lossless way. A good example is company name, which tends to change over time or be used in different formats. This is a pretty difficult problem to solve and I don't have any good pointers for this. For the most part I keep the data in sqlite databases in format specific to the job portal, I haven't found a decent way of normalizing the data as the portals have very different ideas about the level of information they keep in a structured way. The only processing I do is indexing the values I need and deduplicating multi-location listings and same offers across many days/weeks/months. This is good enough for my use case, but falls short of making it useful for any commercial use.
  - nicbou 3y ago
    How do you plan to implement very granular filters when the data is rarely complete or neatly organised? I had this discussion with my friend who runs ArbeitNow. It's hard to create good filters when the data is just paragraphs of text. It's hard to augment these listings when the data is not reliable.
    - punkpeye 3y ago
      Won't comment on this now, but will follow up once it launches. I've been working on this now for 3 months. So it isn't simple aggregator. There is a lot of data enrichment happening by collecting auxiliary data from LinkedIn, Crunchbase, GitHub, etc.
      - altdataseller 3y ago
        What are some examples of some of these filters?
  - altdataseller 3y ago
    What differentiates this job board from Indeed and other job boards?
- jiggawatts 3y ago
  Modern JS development just blows my mind. There was a person here complaining that hosting his anime fan site was costing too much. He just couldn’t understand the concept of not sending the entire dataset to the web page on every page view.
  - inemesitaffia 3y ago
    Link?
    - jiggawatts 3y ago
      An in-context link is here: https://news.ycombinator.com/item?id=28440742#28442591 https://news.ycombinator.com/item?id=28440742#28442591 However the entire post is worth reading because other people contributed some good points also. The Mangadex devs seem to have followed some of the advice and the referenced inefficient JS API call has been made more efficient since my comment, but notably it is still sending many times more data than what is required.
- steve_adams_86 3y ago
  One of the easiest fixes for Next page sizes is pruning unused prop data. It’s super common, even in relatively professional settings. I think it isn’t intuitive that unused data will be sent. The whole tree shaking and optimization pipeline seems like it should handle stuff like that, but sure enough, it doesn’t (for good reason, though).
- csomar 3y ago
  I once had a discussion with a management that thought it’s fine not to implement access control for GraphQl because no one knows the endpoint. I explained it’s possible to figure out the endpoint through the Network tab in the console. They assumed it’s fine because that’s too technical.
  - speedgoose 3y ago
    That’s when you stop involving management in technical discussions. As a software engineer, you do what must be done. If access control is a must, management can wait on the less important features. If they are stupid and don’t understand despite explaining them, they don’t have to know.
    - awill88 3y ago
      Yeah, but don’t get yourself in trouble and be a renegade either. I do not agree that engineers should avoid technical talks with managers. They should just always be prepared to break it down and provide examples. Look people in the eye, empathize with their sentiments, don’t be an asshole if you can help it, in my humble opinion.
      - pavel_lishin 3y ago
        And get stupid decisions in writing.
  - awill88 3y ago
    Tell them all the juniors are using it and it’s causing widespread security breaches and could cost the company money (it’s half true at least). If they don’t care about money when you talk about it, well, then that may signal something deeper.
nine_k 3y ago
I wish websites could just serve the data worth scraping as nice data files, along the content. But it's impossible due to most websites' business model. Those who can do it for free, do; see e.g. Wikipedia. More websites could offer a paid API though; stopping to waste your engineers' time on adding anti-scraping measures, and collecting payments from serious actors could offset the possible losses from copycats.
- dewey 3y ago
  > More websites could offer a paid API though If you are offering a paid API it becomes a product that's most likely very different from your main product you should be focusing on. So instead of deprecating fields, breaking the format and quickly iterating you'll have to version APIs, add authentication, keep a change log for your paid customers, have customer support for your API customers and make sure you are not doing backwards incompatible changes.
  - nine_k 3y ago
    This is correct. But if you have a reasonable PMF and stability, and already serve your pages via some API anyway, maybe productizing a portion of that might be worth it.
- isbvhodnvemrwvn 3y ago
  Funnily enough the site which changed to next.js did, they actually had a lovely offers.json file (~1MB, thousands of offers, just without a long-form HTMLy description) but they were also using it for their transactional needs (meaning that everyone had to load this file before the offers showed up). So they rewrote everything to next.js instead of adding some pagination...
- Alifatisk 3y ago
  Rails has a cool built-in feature where if you add ".json" to the end of the URL, the server will return the same page but in json format with only the data.
moomoo11 3y ago
I’m glad stuff like next and other bloat exists so I can avoid it and stick to plain ol server rendered tech. Go backend, sprinkle some svelte, call it a day. Serve m/billions with server rendered page without breaking a sweat or over engineering. $24 a month.
- is_true 3y ago
  How is your architecture? I do something similar but with PHP + caching as much as I can, which right now is everything.
  - moomoo11 3y ago
    I cache certain info using embedded databases like RocksDB (formerly) and currently a LevelDB. Then just load that data into the templates and serve it. I used to host on cheap 1cpu 512mb ram and 20gb SSD. I would be able to handle 7-9k rps. Now I’m on a bit beefier dedicated hardware doing a few more things. But under simulated load this can handle close to 200k rps.
    - is_true 3y ago
      ty
  - quickthrower2 3y ago
    If you simply send out the correct cache headers, then there are a lot of good CDN choices you can chuck in front of your site, and call it a day. You then get edge delivery too! I have a site hosted on digital ocean app service. $5 a month. I just pinged it, with times being about 24ms. It must be serving that pretty local. Ooh IPv6 too! All I did is make sure good cache headers are going out. As a bonus the tech being used here is something you need to know anyway. Whereas with funky tech, you need to learn the funky thing, but you still need to know about cache headers!
quickthrower2 3y ago
Hasn’t it always been that way. Had fun on manifold.markets a year ago beating some markets using that kind of data
javascriptdante 3y ago
I sometimes wonder if the Javascript world has some sort of divine significance, as though they carry the weight of all the developer sins in the world for some particular transcendental purpose. After all, Dante's nine circles of hell are insufficient for programmers who are able to recurse infinitely. How much further can we go? Consider the descent down, and the contradictions therein, considering JUST data concerns for brevity: 1. SPA ostensibly being marginally more data efficient by shipping the markup/interactivity first, and fetching only the data as needed. 2. REST being too inefficient and cumbersome in practice, the rise of GraphQL to minimize payload over the wire. 3. Lack of SSR impacting SEO, the need to render on the server and sending HTML over on initial payload. (which btw goes against (1) for first render) 4. Impedance mismatch creeping in, seemingly necessitating the need to send data along with the rendered HTML for hydration purposes (again, going against the idea of saving data over the wire) I eagerly await, from the sidelines, news from even deeper depths. Perhaps shipping the entirety of the server code that operates then against a local SQLite database? Which is then kept in sync with the authoritative copy "on the edge"? After all, if simple old SSR is beneath consideration, than we would do well to disdain the notion of a boring centralized database as well.
- lioeters 3y ago
  > shipping the entirety of the server code that operates then against a local SQLite database For sure this new circle of hell is emerging. The browser is metastasizing into what resembles an application platform, operating system, a sandboxed virtual machine. It's becoming quite capable too, including a local file system (Origin Private File System), widely supported now. > kept in sync with the authoritative copy "on the edge" There could be a new generation of portable, cross-platform and local-first apps built with HTML, CSS, and/or any language that compiles to WebAssembly. Perhaps after decades of penance in the flames of their own making, the web world will find redemption by re-discovering and re-inventing ancient dreams of universal computational media, networked software objects inter-communicating with distributed data.. Speaking like a true denizen of the ninth circle.
- cco 3y ago
  Your first sentence is sublime writing. It made me laugh and cry.
revskill 3y ago
No need for it. I made a RSC framework which offers both for you, the html or the data. Html: https://www.revskill.dev https://www.revskill.dev Only the json: https://www.revskill.dev/?data=true https://www.revskill.dev/?data=true Want the HTML without the embed json ? https://revskill.dev/?isBot=true https://revskill.dev/?isBot=true You can see, no need for Graphql, HTML and API seperation. Your html is the API !

81 points by punkpeye 3y ago

ChrisArchitect 3y ago

Tell HN:

isbvhodnvemrwvn 3y ago

It also shows how awful some sites are. I'm scraping a jobs offer portal written in next.js (although I use the endpoint for hydration), one of the lovely things they do is they pass in props containing all possible skills candidates can have, they send 200KB per listing, less than 10KB is the average volume of job-specific (actually required) data. Previously they were rendering it from a REST endpoint which only had the necessary data, but now I get a lovely dictionary of all possible values of various properties, provided on a silver platter. Another portal I'm scraping uses GraphQL, and they don't have access control, I get all the data they have on impressions, I know how much money somebody paid for listing, I know when the offer is going to be auto-bumped - just lovely. And no need to use Playwright either.

punkpeye 3y ago
The irony is that I discovered this also while scraping a job portal. I wonder if we are scraping the same site lol. I am building a job board for tech jobs, but with very granular filters.
- isbvhodnvemrwvn 3y ago
  Interesting. I was initially intending to do the same, but now I just use the project as OSINT source for research on various companies, if I or any of my friends is looking for a job they get a dump of everything they posted on popular portals in the last 3 years or so.
  - rrrix1 3y ago
    Can I be your friend? :) I'm looking for a job, and just started hacking together a job posts database with data scraped from a few aggregators and a couple of specific companies. Currently it's all just local and for myself, mostly raw data with some basic statistics and keyword similarity heuristics. I decided to try something new and different and take it into my own hands (instead of hoping LinkedIn shows me the "right job"). I'm frustrated by the sheer volume of clearly duplicate job posts which often only vary by the "city" (even if they're 100% remote) and the lack of any kind of basic filtering for programming languages, tech stacks, platforms, years of experience requirements, company industry / size / age etc. I'm really curious what others who have been doing this for much longer have learned! How do you store your data? How do you process it? Have you eliminated manual cleansing and grooming? So many questions...
    - isbvhodnvemrwvn 3y ago
      I'm scraping jobs for a specific country I don't think you're in, but a couple of things: - always keep the original data you scraped in as close to original form as you can (in next.js example, I just strip the unnecessary props - but only those I know are useless, see point 2) - there aren't that many job offers out there, text is easy to compress, storage is cheap. Eventually you will change the idea for what's important, and having original files to work with allows you to re-process them. Kind of like data vault approach in data engineering. - prepare for the APIs to evolve, version your stuff. Have stringent validation (like json schema with enums and not allowing extra fields) to detect this evolution, so that you can adapt. This is extra important for portals with complex data models where you might not fully understand them at first. - understand key fields - it's often the case that "posted date" is auto-bumped every now and then, depending on which package you bought. Try going through buy-a-listing workflow to understand the options. For multi-city listings there is often some sort of way of de-duplicating (like URL slug having a common prefix, or the data containing "main" listing). Spending an hour going through the data you scraped goes a long way. - a lot of the data comes from manual user input or from some imports from the company's ATS. You will need to normalize those values in some way, and it's not always possible to do it in a lossless way. A good example is company name, which tends to change over time or be used in different formats. This is a pretty difficult problem to solve and I don't have any good pointers for this. For the most part I keep the data in sqlite databases in format specific to the job portal, I haven't found a decent way of normalizing the data as the portals have very different ideas about the level of information they keep in a structured way. The only processing I do is indexing the values I need and deduplicating multi-location listings and same offers across many days/weeks/months. This is good enough for my use case, but falls short of making it useful for any commercial use.
- nicbou 3y ago
  How do you plan to implement very granular filters when the data is rarely complete or neatly organised? I had this discussion with my friend who runs ArbeitNow. It's hard to create good filters when the data is just paragraphs of text. It's hard to augment these listings when the data is not reliable.
  - punkpeye 3y ago
    Won't comment on this now, but will follow up once it launches. I've been working on this now for 3 months. So it isn't simple aggregator. There is a lot of data enrichment happening by collecting auxiliary data from LinkedIn, Crunchbase, GitHub, etc.
    - altdataseller 3y ago
      What are some examples of some of these filters?
- altdataseller 3y ago
  What differentiates this job board from Indeed and other job boards?
jiggawatts 3y ago
Modern JS development just blows my mind. There was a person here complaining that hosting his anime fan site was costing too much. He just couldn’t understand the concept of not sending the entire dataset to the web page on every page view.
- inemesitaffia 3y ago
  Link?
  - jiggawatts 3y ago
    An in-context link is here: https://news.ycombinator.com/item?id=28440742#28442591 https://news.ycombinator.com/item?id=28440742#28442591 However the entire post is worth reading because other people contributed some good points also. The Mangadex devs seem to have followed some of the advice and the referenced inefficient JS API call has been made more efficient since my comment, but notably it is still sending many times more data than what is required.
steve_adams_86 3y ago
One of the easiest fixes for Next page sizes is pruning unused prop data. It’s super common, even in relatively professional settings. I think it isn’t intuitive that unused data will be sent. The whole tree shaking and optimization pipeline seems like it should handle stuff like that, but sure enough, it doesn’t (for good reason, though).
csomar 3y ago
I once had a discussion with a management that thought it’s fine not to implement access control for GraphQl because no one knows the endpoint. I explained it’s possible to figure out the endpoint through the Network tab in the console. They assumed it’s fine because that’s too technical.
- speedgoose 3y ago
  That’s when you stop involving management in technical discussions. As a software engineer, you do what must be done. If access control is a must, management can wait on the less important features. If they are stupid and don’t understand despite explaining them, they don’t have to know.
  - awill88 3y ago
    Yeah, but don’t get yourself in trouble and be a renegade either. I do not agree that engineers should avoid technical talks with managers. They should just always be prepared to break it down and provide examples. Look people in the eye, empathize with their sentiments, don’t be an asshole if you can help it, in my humble opinion.
    - pavel_lishin 3y ago
      And get stupid decisions in writing.
- awill88 3y ago
  Tell them all the juniors are using it and it’s causing widespread security breaches and could cost the company money (it’s half true at least). If they don’t care about money when you talk about it, well, then that may signal something deeper.

nine_k 3y ago

I wish websites could just serve the data worth scraping as nice data files, along the content. But it's impossible due to most websites' business model. Those who can do it for free, do; see e.g. Wikipedia. More websites could offer a paid API though; stopping to waste your engineers' time on adding anti-scraping measures, and collecting payments from serious actors could offset the possible losses from copycats.

dewey 3y ago
> More websites could offer a paid API though If you are offering a paid API it becomes a product that's most likely very different from your main product you should be focusing on. So instead of deprecating fields, breaking the format and quickly iterating you'll have to version APIs, add authentication, keep a change log for your paid customers, have customer support for your API customers and make sure you are not doing backwards incompatible changes.
- nine_k 3y ago
  This is correct. But if you have a reasonable PMF and stability, and already serve your pages via some API anyway, maybe productizing a portion of that might be worth it.
isbvhodnvemrwvn 3y ago
Funnily enough the site which changed to next.js did, they actually had a lovely offers.json file (~1MB, thousands of offers, just without a long-form HTMLy description) but they were also using it for their transactional needs (meaning that everyone had to load this file before the offers showed up). So they rewrote everything to next.js instead of adding some pagination...
Alifatisk 3y ago
Rails has a cool built-in feature where if you add ".json" to the end of the URL, the server will return the same page but in json format with only the data.

moomoo11 3y ago

I’m glad stuff like next and other bloat exists so I can avoid it and stick to plain ol server rendered tech. Go backend, sprinkle some svelte, call it a day. Serve m/billions with server rendered page without breaking a sweat or over engineering. $24 a month.

is_true 3y ago
How is your architecture? I do something similar but with PHP + caching as much as I can, which right now is everything.
- moomoo11 3y ago
  I cache certain info using embedded databases like RocksDB (formerly) and currently a LevelDB. Then just load that data into the templates and serve it. I used to host on cheap 1cpu 512mb ram and 20gb SSD. I would be able to handle 7-9k rps. Now I’m on a bit beefier dedicated hardware doing a few more things. But under simulated load this can handle close to 200k rps.
  - is_true 3y ago
    ty
- quickthrower2 3y ago
  If you simply send out the correct cache headers, then there are a lot of good CDN choices you can chuck in front of your site, and call it a day. You then get edge delivery too! I have a site hosted on digital ocean app service. $5 a month. I just pinged it, with times being about 24ms. It must be serving that pretty local. Ooh IPv6 too! All I did is make sure good cache headers are going out. As a bonus the tech being used here is something you need to know anyway. Whereas with funky tech, you need to learn the funky thing, but you still need to know about cache headers!

quickthrower2 3y ago

Hasn’t it always been that way. Had fun on manifold.markets a year ago beating some markets using that kind of data

javascriptdante 3y ago

I sometimes wonder if the Javascript world has some sort of divine significance, as though they carry the weight of all the developer sins in the world for some particular transcendental purpose. After all, Dante's nine circles of hell are insufficient for programmers who are able to recurse infinitely. How much further can we go? Consider the descent down, and the contradictions therein, considering JUST data concerns for brevity: 1. SPA ostensibly being marginally more data efficient by shipping the markup/interactivity first, and fetching only the data as needed. 2. REST being too inefficient and cumbersome in practice, the rise of GraphQL to minimize payload over the wire. 3. Lack of SSR impacting SEO, the need to render on the server and sending HTML over on initial payload. (which btw goes against (1) for first render) 4. Impedance mismatch creeping in, seemingly necessitating the need to send data along with the rendered HTML for hydration purposes (again, going against the idea of saving data over the wire) I eagerly await, from the sidelines, news from even deeper depths. Perhaps shipping the entirety of the server code that operates then against a local SQLite database? Which is then kept in sync with the authoritative copy "on the edge"? After all, if simple old SSR is beneath consideration, than we would do well to disdain the notion of a boring centralized database as well.

lioeters 3y ago
> shipping the entirety of the server code that operates then against a local SQLite database For sure this new circle of hell is emerging. The browser is metastasizing into what resembles an application platform, operating system, a sandboxed virtual machine. It's becoming quite capable too, including a local file system (Origin Private File System), widely supported now. > kept in sync with the authoritative copy "on the edge" There could be a new generation of portable, cross-platform and local-first apps built with HTML, CSS, and/or any language that compiles to WebAssembly. Perhaps after decades of penance in the flames of their own making, the web world will find redemption by re-discovering and re-inventing ancient dreams of universal computational media, networked software objects inter-communicating with distributed data.. Speaking like a true denizen of the ninth circle.
cco 3y ago
Your first sentence is sublime writing. It made me laugh and cry.

revskill 3y ago

No need for it. I made a RSC framework which offers both for you, the html or the data. Html: https://www.revskill.dev https://www.revskill.dev Only the json: https://www.revskill.dev/?data=true https://www.revskill.dev/?data=true Want the HTML without the embed json ? https://revskill.dev/?isBot=true https://revskill.dev/?isBot=true You can see, no need for Graphql, HTML and API seperation. Your html is the API !