--- title: "How we built Git storage for millions of Netlify projects" description: "A technical look at how Netlify built Netlify Source: durable Git storage on S3, scalable caching, push consistency, and support for millions of projects." source: "https://www.netlify.com/blog/how-we-built-git-storage-for-millions/" last_updated: "2026-09-22T21:25:03.000Z" --- For the last ten years, Netlify has grown from continuous deployment and hosting into a broader runtime platform: functions, edge functions, blobs, databases, authentication, AI, and more. But development itself still lived somewhere else: the user’s own Git host. Code was pushed over there, we were notified, and we built, deployed, and served the resulting site. It was honest work. In the last 12 to 18 months, something changed. More and more, our users started _sending us the files directly_: a folder or zip file, through [Netlify Drop](https://app.netlify.com/drop), the [Netlify API](https://docs.netlify.com/api-and-cli-guides/api-guides/get-started-with-api/), the [Netlify MCP](https://docs.netlify.com/build/build-with-ai/agent-setup-guides/agent-setup-overview/#mcp-server-support)… or [your agent did](https://docs.netlify.com/build/build-with-ai/agent-setup-guides/agent-setup-overview/). They also started [building and updating projects directly with Agent Runners](https://docs.netlify.com/start/choose-your-path/#start-with-ai-agent). They started doing this _a lot_. In the last month, 93% of new projects were created one of these ways; two years ago, it was 70%. In absolute terms, that’s multiple millions of new projects per month, a 14x increase. ![Netlify projects with a connected repository](https://cdn.sanity.io/images/o0o2tn5x/production/7556f593b8197b308bc5de7ebd7f45b1582e9132-800x440.png) ## Now, every project needs a Git repo It turns out Git hosts were doing a lot of valuable heavy lifting. We don’t want to become a Git hosting provider, but we do want every project on Netlify to have the benefits of Git. When you and an agent are both making changes, you need somewhere to put the work in progress (so… a commit on a branch?), a shared understanding of what it started from and what it changed (so… a parent commit and a diff?), a way to take over from the agent (so… `git pull`?), a way to hand your changes back to the agent (so… `git push`?), and a way to incorporate upstream changes before publishing yours (so… `git merge` or rebase?). Oh, and a way to operationalize all this with tens of millions of repos, with high performance, unimpeachable data integrity, and scalable unit economics. … Yep, that’s Git. We’d been accidentally, incrementally rebuilding Git, piece by piece. Our bad. So we built an integrated, Netlify-managed Git repository for every project that comes to Netlify without a connected host (the same thing we introduced last month in [Dana Lawson’s conversation about the future of Git](https://www.netlify.com/blog/netlify-source-with-netlify-cto-dana-lawson/), where we called it the internal name, Netlify Source). These repositories speak the same Git protocol as any other remote. Our build infrastructure, Agent Runners, your editor, and your own agent can all work with them. Picking up where an agent left off should be as ordinary as cloning a repo. In this technical deep dive, we’ll dig into the engineering work that went into architecting, scaling, and securing this system. ## Architecture ### S3 as durable source of truth First, a one-minute primer on Git internals. Git stores file contents (blobs), directory listings (trees), and commits as [objects identified by hashes](https://git-scm.com/book/en/v2/Git-Internals-Git-Objects). A commit points to the tree for its snapshot and to its parent commits. Those objects don’t change; a branch like `main` is a small pointer that does. Branch pointers are a type of _ref_. Git can store objects individually (“loose objects”) or bundle them into [packfiles](https://git-scm.com/book/en/v2/Git-Internals-Packfiles). ![Git object model](https://cdn.sanity.io/images/o0o2tn5x/production/fbb98724a4eeb8176a42e4ec1ea54f0842b67e72-800x440.png) Our Git service is written in Go and [runs the Git binary](https://blog.gitbutler.com/true-grit) against a local [bare](https://git-scm.com/docs/git-clone#Documentation/git-clone.txt---bare) repository (full history, no checked-out working directory). Git itself speaks [Smart HTTP](https://git-scm.com/book/en/v2/Git-on-the-Server-Smart-HTTP), negotiates object transfers with the client, and processes packfiles. Our service’s responsibility is moving data between S3 and that local repository before and after Git runs. S3 holds the durable data: loose objects and packfiles in Git’s native formats, and refs as small JSON records. Each pod keeps a local cache of repositories on disk, evicting the least recently used ones when it needs space. For a clone or fetch, the service first synchronizes the local refs and objects from S3, then lets Git serve them from disk. On a cold cache, that means initializing a bare repository and downloading its data. On a warm cache, we still check S3 and download anything missing. If a pod disappears, another will reconstruct the repository from S3 on the next request to that repo. ![Git served from S3](https://cdn.sanity.io/images/o0o2tn5x/production/761134a3c0f2fe443207518e6fc70fc6ca14b6a2-800x440.png) ### Making git push durable `git receive-pack` is the Git RPC command that accepts a push. Once it has updated the local repository, it’s ready to report success. But the pod could disappear before those changes make it to S3. We have two requirements: a ref (e.g. branch pointer) must never get ahead of the objects it points to, and we must never report a success to the client until the changes have been persisted to durable storage (S3). We thus run the following sequence, while holding a local write lock (a Go mutex) on that repository: 1. Synchronize refs and objects from S3, then stream the request payload into `git receive-pack`. 2. Upload new packfiles and loose objects to S3. 3. Update the refs in S3 using conditional writes, only if their existing values match what we expect. 4. Stream the `receive-pack` response from step 1 to the client. ![Git push timeline view](https://cdn.sanity.io/images/o0o2tn5x/production/dbe42dae60e6c9e1787062df63cc26ef5e2fd2a8-800x500.png) Object uploads can overlap: the objects are immutable and identified by their contents. Refs are mutable; two writers can disagree about where `main` should point. The mutex only coordinates requests within one pod. During a deployment or routing change, two pods might both start with `main` pointing to commit A and try to advance it to B and C. Both can upload their objects, but only one can move `main` from A. We read the ref from S3 and check that it still points to A. That read also returns an ETag, a token for checking whether the stored ref record has changed. This is distinct from the Git commit hash. We write the new ref with `If-Match` against that ETag. S3 [checks the condition and writes atomically](https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-writes.html), so a change between our read and write causes our update to fail. A new ref uses `If-None-Match: *` to require that it doesn’t yet exist. ![collision handling](https://cdn.sanity.io/images/o0o2tn5x/production/3e579ef988ea76304b830e04d5557dbf8ce755d3-800x440.png) If a conditional write is rejected, the push reports a storage-sync error. On a persistence error, we discard the local cache so the next request reconstructs it from S3. Once Git has accepted changes locally, persistence continues under its own timeout even if the client disconnects. The push can reach S3 without the client receiving confirmation. Holding that response can leave the connection quiet for minutes on a large push. Clients and proxies tend to have opinions about that. When clients support it (most should these days), we send (and flush!) [Git side-band](https://git-scm.com/docs/protocol-capabilities/2.17.0#_side_band_side_band_64k) progress messages every ten seconds during the S3 sync. ## Performance wins One immediate win came from replacing our homegrown “rebase” for Agent Runners. We’d been asking an agent to reapply a run’s accumulated diff on top of a newer deploy’s source. With an integrated Git repository, `git merge` handles the update. An agent can still help with conflicts, if any. Clean merges skip the agent session entirely, **resulting in a majority of these operations taking about one second (and zero tokens!) rather than a few minutes**. Publishing an agent run is now (most of the time) just `git merge` into `main` followed by a push. Preparing the source code workspace for builds and agent runs took **0.76s at the median and 2.26s at p95** during September 7–13, 2026. The same stage took **0.84s and 2.51s** for projects connected to GitHub. For a relatively simple Git service whose durable store is S3, that is close enough to GitHub to feel boring — which was the point. ![Git clone performance comparison](https://cdn.sanity.io/images/o0o2tn5x/production/4b09e7c2a5993399d490432b220e1a1e6ed94fba-800x440.png) ## Git hosting in 2026 We didn’t build this to compete with any of these. Our goal was to give every project that lands on Netlify without a Git host a real one, invisibly. But since several teams are solving adjacent problems right now, the constraints are worth comparing. [Cursor Origin](https://cursor.com/blog/git-at-any-scale) is building a full forge. Its Continuity backend also treats object storage as durable and local repositories as caches, but it is optimizing for a different problem: high write throughput and contention within one repository. It publishes pushes through a write-ahead log and batches updates to its index; we upload Git objects and update each ref directly. [East River Source Control](https://ersc.io/blog/what-comes-after-git) is rethinking storage for growing repositories and more concurrent development. They’re building on top of [Jujutsu](https://www.jj-vcs.dev/), intending to solve performance bottlenecks for huge monorepos, and looking beyond Git. [Code Storage](https://code.storage/), from The Pierre Computer Company, offers programmable Git storage with APIs designed for machines, backed by a quorum of replicas. [Entire](https://entire.io/) combines distributed Git hosting with the agent sessions and conversations behind the code. For comparison, [GitHub’s published Spokes design](https://github.blog/engineering/infrastructure/stretching-spokes/) replicates pushes across file servers and requires a quorum. [GitLab’s Gitaly](https://docs.gitlab.com/administration/gitaly/) keeps authoritative repositories on the serving nodes’ filesystems, while Praefect coordinates multiple replicas. Those systems have to keep track of and protect the repository copies on those nodes. We chose to lean on the availability, durability, and atomicity guarantees of S3 and its conditional writes. This also lets us keep our architecture relatively simple. GitLab is exploring that separation too. Its [Scaling Git proposal](https://gitlab.com/gitlab-com/content-sites/handbook/-/blob/main/content/handbook/engineering/architecture/design-documents/scaling-git/_index.md) would make object storage authoritative and Gitaly nodes caches. It proposes custom Git storage backends and a single manifest pointer to publish a consistent repository snapshot atomically. We kept Git’s native object formats and use per-ref conditional writes, with the narrower atomicity described above. ## What’s next Rearchitecting how we store and manipulate source code under the hood was just a first, invisible step. With this foundation in place, a number of potential capabilities have been unlocked. ## What’s _not_ next Netlify has always been about freedom of choice and composition. Nothing is changing or will change about the Git hosts you can connect to Netlify — GitHub, GitLab, Bitbucket, Azure DevOps, and, as of last week, [Cursor Origin](https://www.netlify.com/blog/cursor-origin-repositories-on-netlify). If your team’s workflow already lives on one of those, that’s still exactly how you should use Netlify. This is about giving projects that land on Netlify without one of those hosts the same foundation — not about pulling anyone off a host they’ve already chosen. And because it’s a real Git repository speaking the standard protocol, nothing is locked in on our side either. Clone it, add whatever remote you want, and push. You can even export your code to GitHub in a click. Whatever comes next won’t be exclusive to projects with an integrated Git repository. We can _commit_ to that. ### More to come in part two We have plenty more to share about the technical architecture, scaling challenges, security, and how we migrated millions of existing projects onto this system. Watch out for part two of this series. ### Share - [X (fka Twitter)](https://twitter.com/intent/tweet?text=How we built Git storage for millions of Netlify projects&url=https://www.netlify.com/blog/how-we-built-git-storage-for-millions/) - [LinkedIn](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Fwww.netlify.com%2Fblog%2Fhow-we-built-git-storage-for-millions%2F) - [Facebook](https://www.facebook.com/sharer.php?u=https://www.netlify.com/blog/how-we-built-git-storage-for-millions/) - [Bluesky](https://bsky.app/intent/compose?text=How we built Git storage for millions of Netlify projects+https://www.netlify.com/blog/how-we-built-git-storage-for-millions/) ## Keep reading ![](/_astro/39409700c858c295fc5c37de304f679e44e21c32-2400x1350_Z1i5x2x.webp) Opinions & Insights August 14, 2026 [ ### The full power of Git, without the friction: A conversation with Netlify CTO Dana Lawson ](/blog/netlify-source-with-netlify-cto-dana-lawson) - ![Profile picture of Dana Lawson](/_astro/856bf146d0c05c9dc25d45b59f7eac955fbbd644-512x512_1n84rs.webp) Dana Lawson ![](/_astro/107ebf9c7eeee779ea48e5ff2d617a43e0b81e47-1800x1013_Z90CT4.webp) Opinions & Insights September 2, 2026 [ ### Build with Netlify came to Atlanta ](/blog/build-with-netlify-came-to-atlanta) - ![Profile picture of Tania Chakraborty](/_astro/707202a137a002306d64cd12d886e28258c1bd8d-800x800_ZfolOr.webp) Tania Chakraborty ## Recent posts Tools & Services September 10, 2026 [ ### Cursor Origin repositories build, preview, and deploy on Netlify ](/blog/cursor-origin-repositories-on-netlify) - ![Profile picture of Gehrig Kunz](/_astro/b4e9f58d914d1334ea70d53ea55a1f26b26f1445-512x512_17SwOI.webp) Gehrig Kunz Opinions & Insights September 2, 2026 [ ### Build with Netlify came to Atlanta ](/blog/build-with-netlify-came-to-atlanta) - ![Profile picture of Tania Chakraborty](/_astro/707202a137a002306d64cd12d886e28258c1bd8d-800x800_ZfolOr.webp) Tania Chakraborty News & Announcements August 25, 2026 [ ### Compete in OpenAI’s WebMCP Challenge with Netlify ](/blog/compete-openai-webmcp-challenge) - ![Profile picture of Karthik Puvvada](/_astro/e5524d3c315ad5114e1d5300d8991bfe902916fe-192x192_Z1w18MB.webp) Karthik Puvvada ![](/_astro/3f255b372fa958df35802666ee33b4609b2d71bd-1200x1586_1VtE2D.webp) ### How do the best dev and marketing teams work together? [Access the report](https://www.netlify.com/reports/2024-leadership-trend-report/access/)