8 ms·

Incremental Builds in Gatsby Cloud

130 points by Dockson 6y ago

arpowers 6y ago
You still need to use an api for everything. Good apps need a backend; not a JAMStack fan for anything but the most basic of sites.
- bmelton 6y ago
  The 'backend' here is ... HTML. For read-only a blog, that's likely more than enough. Otherwise, for dynamic content like contact forms and such, I don't know if there's a meaningful benefit to building out a whole site in PHP/Python/Rails or something (and paying commensurately more in hosting) than to use Formspree or something similar. Yes, it calls an API. And thankfully with Formspree, it's pretty easy to see the price breakeven points vs. hosting, but there are benefits to be had.
  - dgb23 6y ago
    Content has to come from somewhere. Operational complexity is increased, not decreased, since you are running a build server in addition to your CMS. The benefit is easier optimizations in the frontend.
    - Cthulhu_ 6y ago
      But you can outsource those (build server, CMS) to e.g. Netlify and Contentful. But if you don't want that, there's a billion alternatives; the CMS market is one of the most saturated ones out there.
      - subpixel 6y ago
        I actually feel like Gatsby will be developing a CMS to round out their paid feature-set. I have zero-inside knowledge, just a hunch that to get customers to pay up handsomely, the product needs a deeper "fit" in the publishing pipeline.
    - bmelton 6y ago
      Content can come from text files. Operational complexity ought to be neutral (or even negative) if that content compilation step replaces your CI pipeline.
    - lotyrin 6y ago
      Yeah, but you're only running your build server some of the time (which is favorable in this era of by-the-second pricing) and you're running it inside a private network where nobody can try to infiltrate it 24/7 for no good reason.
turadg 6y ago
This is great! Is there any technical limitation keeping this from being part of the open source version? I get that Gatsby company put a lot of effort into this and wants a return on that investment, and good for them. I assume a third party could offer the same but why would they compete at the same value prop. However an open source version to not be reliant on any company would be compelling to many.
- kylemathews 6y ago
  We recently introduced build optimizations for incremental data changes for self-hosted environments: https://www.gatsbyjs.org/docs/page-build-optimizations-for-incremental-data-changes/ https://www.gatsbyjs.org/docs/page-build-optimizations-for-i... and are continuing to improve build speed across platforms. To reliably provide near real-time deployments, we need tight integration with the CI/CD environment to optimize and parallelize the work; that's why you’ll see the fastest builds and deploys through Gatsby Cloud — the platform is purpose built for Gatsby!
alexgvozden 6y ago
so this is cool release, and no objection on that but if your pipeline has automated testing, security scans and more then you are not actually deploying in 10s more technical details would be good but I guess either I missed it or they look at it as IP
- ascorbic 6y ago
  You wouldn't be running automated testing etc on data updates though, surely? That's what this feature is for, not code updates.
kylemathews 6y ago
Gatsby founder here. Really appreciate the feedback and support for our launch today! The team worked super hard to get Incremental Builds live in public beta but are taking all the feedback (here and all over the web) as we go into full launch. Let us know what you think. Thanks!
- denster 6y ago
  Kyle, Just read the post, congrats on the launch! We've been using Gatsby on: https://mintdata.com https://mintdata.com for the past few years, and are huge fans of your work. I still recall the day when I brought Gatsby into our org, our front-end guys almost ate me alive :D They said: a React.render(...) + GraphQL thing, why do we need it? What's the big deal? Fast forward a few years later, and Gatsby dominates (in my opinion) the best way to build a static website based on React. Keep up the awesome work! Your true fan, Denis
  - kylemathews 6y ago
    Glad it's been a great experience!
  - rayshan 6y ago
    Wow MintData looks so cool! I was just trying to figure out whether Webflow can be used to build simple apps, then I saw this, a whole new level. Is there a way to try it?
- noworriesnate 6y ago
  Thanks for the great piece of software! I really like how the Gatsby community is dabbling in templates for more than just blogs. I've seen documentation sites, landing pages, and notes. Here are some thoughts on my experience with Gatsby: - You've done a lot of work to make configuring Gatsby easier, but I still seem to constantly hit roadblocks trying to get the config I want. For example I was running into problems getting MermaidJS, embedded video (that I was hosting on my own machine, not on YouTube), and mdx files all working together. - I've been thinking that Gatsby is the perfect framework for creating semantic web content. E.g., you could have calendar events sprinkled through a website and create a GraphQL API for listing those calendar events, and that API would be accessible during the build process.
skrebbel 6y ago
Having never used a static site generator in anger, can someone explain to me like I'm five what's going on here? My understanding is that Gatsby is a tool that converts a bunch of markdown files into a static HTML website. Why is slow builds a problem for any static site generator? Why does it need a cloud? In other words, what problem am I supposed to be having that any of this solves? Note, I'm trying not to be skeptical here - my company's website is hand-maintained HTML with a bunch of PHP mixed in so I can totally imagine that things may be better. But I don't understand the kinds of situations where using a 3rd party cloud to generate some static HTML solves a problem.
- akiselev 6y ago
  Static site generators output static HTML files instead of running a server that renders every request, it doesn't say anything about what language they consume. Gatsby is built around Javascript and React, not Markdown.
  - nogabebop23 6y ago
    >> Static site generators output static HTML files This is not really true; they often generate a static client-side web application vs. a dynamic first-time (or every time) app based on server-side processing. This provides a highly optimized, largely self-contained application that avoids a lot of the runtime dependencies and complexity we typically get (ex: web servers and databases). They are still highly dynamic through the use of APIs and such. Gatsby has an extensive build pipeline and can query almost any data source during the build, but the original base source is markdown, and react is the Javascript.
- searchableguy 6y ago
  It doesn't have to be markdown files. Gatsby supports a wide range of data sources which is available to use in your templates via graphql. If your website is big and gets frequently updated with data from the backend triggering the builds, new content on the site can take few minutes to appear as the generator will need to build a static site (html/css/js files) which I assume is a problem for big publication sites.
gnalck 6y ago
Is the technology behind "incremental builds" being upstreamed into the open source project?
dergachev 6y ago
Our team's been waiting on this for a year to start moving larger sites to Gatsby. Can't wait to try it.
WnZ39p0Dgydaz1 6y ago
Javascript re-invents "code typing" (Typescript) Javascript re-invents "Promises" because callback hell Javascript re-invents "compilers" (babel) Javascript re-invents "build systems" (webpack, etc) Javascript re-invents "caching" (incremental builds) - but paid, and in the cloud Because why not.
- kaishiro 6y ago
  /s/re-invents/implements/g and suddenly JavaScript is running a successful dev cycle.
seanwilson 6y ago
How much is the speed issue related to the language used? I know Hugo is an order of magnitude faster than most static site generators for example - it's written in Go with e.g. 2 seconds to generate about 10K pages https://forestry.io/blog/hugo-vs-jekyll-benchmark/ https://forestry.io/blog/hugo-vs-jekyll-benchmark/. I would have thought the generation process could be massively parallelised and a typical blog page would only need a modest amount of computation e.g. concat header, footer, pull in body text, resolve a few URLs. I can't help but think about how much work a typical computer game is doing in comparison 60 times per second even without a GPU.
- turnipla 6y ago
  I don’t think it’s a language issue. Even for JavaScript bundlers you have the slow extensible bundle and the “new super fast bundler” that dies in a month because it only fits one use case. How flexible is Hugo? And how many plugins does someone generally use?
  - seanwilson 6y ago
    > How flexible is Hugo? And how many plugins does someone generally use? It processes Markdown, JSON, YAML and SASS, can pull in data files from URLs, and has custom templates/themes, custom macros/shortcuts, image processing and live reload. It doesn't have a plugin system as far as I know but nothing stops you combining Hugo with other tools e.g. run a JS script to pull in and transform a JSON file before Hugo runs.
    - turnipla 6y ago
      I think that’s the point. No plugin system. Compare Babel to Bublé or even Sucrase for example: https://github.com/alangpierce/sucrase https://github.com/alangpierce/sucrase Preparing data for external use always takes extra effort. You can build an efficient self-contained tool in JavaScript too.
      - ratww 6y ago

130 points by Dockson 6y ago

You still need to use an api for everything. Good apps need a backend; not a JAMStack fan for anything but the most basic of sites.

bmelton 6y ago
The 'backend' here is ... HTML. For read-only a blog, that's likely more than enough. Otherwise, for dynamic content like contact forms and such, I don't know if there's a meaningful benefit to building out a whole site in PHP/Python/Rails or something (and paying commensurately more in hosting) than to use Formspree or something similar. Yes, it calls an API. And thankfully with Formspree, it's pretty easy to see the price breakeven points vs. hosting, but there are benefits to be had.
- dgb23 6y ago
  Content has to come from somewhere. Operational complexity is increased, not decreased, since you are running a build server in addition to your CMS. The benefit is easier optimizations in the frontend.
  - Cthulhu_ 6y ago
    But you can outsource those (build server, CMS) to e.g. Netlify and Contentful. But if you don't want that, there's a billion alternatives; the CMS market is one of the most saturated ones out there.
    - subpixel 6y ago
      I actually feel like Gatsby will be developing a CMS to round out their paid feature-set. I have zero-inside knowledge, just a hunch that to get customers to pay up handsomely, the product needs a deeper "fit" in the publishing pipeline.
  - bmelton 6y ago
    Content can come from text files. Operational complexity ought to be neutral (or even negative) if that content compilation step replaces your CI pipeline.
  - lotyrin 6y ago
    Yeah, but you're only running your build server some of the time (which is favorable in this era of by-the-second pricing) and you're running it inside a private network where nobody can try to infiltrate it 24/7 for no good reason.

turadg 6y ago

This is great! Is there any technical limitation keeping this from being part of the open source version? I get that Gatsby company put a lot of effort into this and wants a return on that investment, and good for them. I assume a third party could offer the same but why would they compete at the same value prop. However an open source version to not be reliant on any company would be compelling to many.

kylemathews 6y ago
We recently introduced build optimizations for incremental data changes for self-hosted environments: https://www.gatsbyjs.org/docs/page-build-optimizations-for-incremental-data-changes/ https://www.gatsbyjs.org/docs/page-build-optimizations-for-i... and are continuing to improve build speed across platforms. To reliably provide near real-time deployments, we need tight integration with the CI/CD environment to optimize and parallelize the work; that's why you’ll see the fastest builds and deploys through Gatsby Cloud — the platform is purpose built for Gatsby!

alexgvozden 6y ago

so this is cool release, and no objection on that but if your pipeline has automated testing, security scans and more then you are not actually deploying in 10s more technical details would be good but I guess either I missed it or they look at it as IP

ascorbic 6y ago
You wouldn't be running automated testing etc on data updates though, surely? That's what this feature is for, not code updates.

kylemathews 6y ago

Gatsby founder here. Really appreciate the feedback and support for our launch today! The team worked super hard to get Incremental Builds live in public beta but are taking all the feedback (here and all over the web) as we go into full launch. Let us know what you think. Thanks!

denster 6y ago
Kyle, Just read the post, congrats on the launch! We've been using Gatsby on: https://mintdata.com https://mintdata.com for the past few years, and are huge fans of your work. I still recall the day when I brought Gatsby into our org, our front-end guys almost ate me alive :D They said: a React.render(...) + GraphQL thing, why do we need it? What's the big deal? Fast forward a few years later, and Gatsby dominates (in my opinion) the best way to build a static website based on React. Keep up the awesome work! Your true fan, Denis
- kylemathews 6y ago
  Glad it's been a great experience!
- rayshan 6y ago
  Wow MintData looks so cool! I was just trying to figure out whether Webflow can be used to build simple apps, then I saw this, a whole new level. Is there a way to try it?
noworriesnate 6y ago
Thanks for the great piece of software! I really like how the Gatsby community is dabbling in templates for more than just blogs. I've seen documentation sites, landing pages, and notes. Here are some thoughts on my experience with Gatsby: - You've done a lot of work to make configuring Gatsby easier, but I still seem to constantly hit roadblocks trying to get the config I want. For example I was running into problems getting MermaidJS, embedded video (that I was hosting on my own machine, not on YouTube), and mdx files all working together. - I've been thinking that Gatsby is the perfect framework for creating semantic web content. E.g., you could have calendar events sprinkled through a website and create a GraphQL API for listing those calendar events, and that API would be accessible during the build process.

skrebbel 6y ago

Having never used a static site generator in anger, can someone explain to me like I'm five what's going on here? My understanding is that Gatsby is a tool that converts a bunch of markdown files into a static HTML website. Why is slow builds a problem for any static site generator? Why does it need a cloud? In other words, what problem am I supposed to be having that any of this solves? Note, I'm trying not to be skeptical here - my company's website is hand-maintained HTML with a bunch of PHP mixed in so I can totally imagine that things may be better. But I don't understand the kinds of situations where using a 3rd party cloud to generate some static HTML solves a problem.

akiselev 6y ago
Static site generators output static HTML files instead of running a server that renders every request, it doesn't say anything about what language they consume. Gatsby is built around Javascript and React, not Markdown.
- nogabebop23 6y ago
  >> Static site generators output static HTML files This is not really true; they often generate a static client-side web application vs. a dynamic first-time (or every time) app based on server-side processing. This provides a highly optimized, largely self-contained application that avoids a lot of the runtime dependencies and complexity we typically get (ex: web servers and databases). They are still highly dynamic through the use of APIs and such. Gatsby has an extensive build pipeline and can query almost any data source during the build, but the original base source is markdown, and react is the Javascript.
searchableguy 6y ago
It doesn't have to be markdown files. Gatsby supports a wide range of data sources which is available to use in your templates via graphql. If your website is big and gets frequently updated with data from the backend triggering the builds, new content on the site can take few minutes to appear as the generator will need to build a static site (html/css/js files) which I assume is a problem for big publication sites.

gnalck 6y ago

Is the technology behind "incremental builds" being upstreamed into the open source project?

dergachev 6y ago

Our team's been waiting on this for a year to start moving larger sites to Gatsby. Can't wait to try it.

WnZ39p0Dgydaz1 6y ago

Javascript re-invents "code typing" (Typescript) Javascript re-invents "Promises" because callback hell Javascript re-invents "compilers" (babel) Javascript re-invents "build systems" (webpack, etc) Javascript re-invents "caching" (incremental builds) - but paid, and in the cloud Because why not.

kaishiro 6y ago
/s/re-invents/implements/g and suddenly JavaScript is running a successful dev cycle.

seanwilson 6y ago

How much is the speed issue related to the language used? I know Hugo is an order of magnitude faster than most static site generators for example - it's written in Go with e.g. 2 seconds to generate about 10K pages https://forestry.io/blog/hugo-vs-jekyll-benchmark/ https://forestry.io/blog/hugo-vs-jekyll-benchmark/. I would have thought the generation process could be massively parallelised and a typical blog page would only need a modest amount of computation e.g. concat header, footer, pull in body text, resolve a few URLs. I can't help but think about how much work a typical computer game is doing in comparison 60 times per second even without a GPU.

turnipla 6y ago
I don’t think it’s a language issue. Even for JavaScript bundlers you have the slow extensible bundle and the “new super fast bundler” that dies in a month because it only fits one use case. How flexible is Hugo? And how many plugins does someone generally use?
- seanwilson 6y ago
  > How flexible is Hugo? And how many plugins does someone generally use? It processes Markdown, JSON, YAML and SASS, can pull in data files from URLs, and has custom templates/themes, custom macros/shortcuts, image processing and live reload. It doesn't have a plugin system as far as I know but nothing stops you combining Hugo with other tools e.g. run a JS script to pull in and transform a JSON file before Hugo runs.
  - turnipla 6y ago
    I think that’s the point. No plugin system. Compare Babel to Bublé or even Sucrase for example: https://github.com/alangpierce/sucrase https://github.com/alangpierce/sucrase Preparing data for external use always takes extra effort. You can build an efficient self-contained tool in JavaScript too.
    - ratww 6y ago