For many Gatsby sites the longest part of the build is resizing and optimizing images, which can take many minutes at a time. Gatsby recently added support for Image CDN, which can save a lot of this time by instead handling the image processing at runtime. Unfortunately this only works for certain CMS integrations, meaning local images still need to be processed at build time. I’ve been working on a new experimental Gatsby Runner to do the same for all images in Gatsby, including local images. Unlike the Image CDN it requires no changes to your Gatsby code.
The results
In my tests on a site with 100 images, Gatsby Runner cuts build times by 84% and cuts total deploy time by 62%.
For these tests I created a basic Gatsby site which includes 100 images from Unsplash, each around 8-10 MP in size. It creates a page for each image and an index page linking them all as thumbnails. In total Gatsby generates 400 image jobs when building the site. I created two sites on Netlify linked to the same repo, with the only difference being the build command. One runs gatsby build as normal, and the other runs gatsby-runner instead. For each of these I performed a “Clean cache and deploy” and ran on a starter account with form detection disabled.
gatsby build. Gatsby Build completes in 6m 50.8s, with a total build time of 8m 52s and total deploy time of 9m. Visit the deployed site.gatsby-runner. Gatsby Runner completes in 1m 4.9s, with a total build time of 3m 22s total deploy time of 3m 26s. Visit the deployed site.
These results are great, but Gatsby Runner really shines for sites with even more images, where it can cut 90% or more of the build time.
Try Gatsby Runner on Netlify today
If you have a site with lots of local images and want to try the runner, then hop on over to the plugin repo to get started. If you’d like to learn more about how it works then read on.
How image processing works in Gatsby
Images in Gatsby are processed at build time using gatsby-plugin-sharp. In the site code you use either the gatsbyImageData resolver in GraphQL, or the StaticImage component. Depending on the size and layout of the requested image, Gatsby will then generate several different output images or different sizes and formats which are then used in a <picture> tag. Each of the required images is turned into a job which contains a reference to the source image and the operations to perform on it. By default these are then processed by workers during the build. However, Gatsby also has support for external jobs. If this feature is enabled then instead of running the job locally it is dispatched to a parent process using IPC.
Why we need a build runner
Gatsby will only enable external jobs if the build is being run from a parent process, so we need to create a runner to do this. Matt Biilmann created something similar when Gatsby first implemented the external jobs API. That runner used Google Cloud Functions to run the transformations in parallel. We will take a slightly different approach and instead of processing the images at build time, we’ll skip processing and process them at runtime. This is the same approach as the image CDN, but it works for any image.
The gatsby-runner CLI
The gatsby-runner CLI runs the site’s gatsby build command in a Node child process and passes all other arguments through. It sets the environment variable that enables external jobs, and then registers an onMessage handler to listen for jobs. We only care about image processing jobs, so all others are still handled by the main gatsby process.
Handling image jobs
The Gatsby Image CDN works by generating a URL that contains the source image URL and all of the operations to perform on the image. Unfortunately the Gatsby sharp plugin doesn’t do this, but instead generates a URL with a hash of the source file and operations. This means we can’t directly extract the operations to perform from the URL so need to take another approach. When we receive an image job, instead of processing the image we save the job to disk as a JSON file. We also copy the source image into the site’s public folder so that it’s available at runtime. We don’t want it to be easy for visitors to download the source image, so we put it in a folder with a randomly-generated name which we also store in the JSON file.
The gatsby-image function
When you build your site with Gatsby Runner, image processing is performed at runtime by On-demand Builders. These are perfect for this task, as they only need to run once on the first request and then persist the result until the site is next deployed.
We include all of the jobs JSON files inside the builder function bundle, as well as the site’s gatsby-plugin-sharp module. When a user requests an image, the handler uses the requested URL path to see which JSON file contains the details of the job needed to generate the requested image. It then loads the source image from the CDN (caching it locally), then passes the image and the job arguments to gatsby-plugin-sharp, which performs the transform. The function then returns the transformed image to the browser, and it is cached globally so it is available instantly to the next user who requests it.
The Gatsby Runner plugin
If we validate Gatsby Runner as a solution, we may choose to add support for Gatsby Runner to Netlify’s Essential Gatsby plugin. Until then, while Gatsby Runner is still experimental, the CLI and runtime image processing are delivered as a separate build plugin, which generates the Netlify Function and adds the correct rewrites to map image requests to the function. All you need to do is install the build plugin and change the build command from gatsby build to gatsby-runner.
Note: While the Gatsby Runner plugin is experimental, it will not be listed in the Netlify plugins directory within the app UI. You can install it with a file-based configuration. Install with npm and then update your
.tomlfile.
Ready to try the new Gatsby Runner for yourself? Visit the plugin repo for instructions on getting started. Please share your feedback and let us know how much time it saves you!
