debian-mirror-gitlab/doc/development/git_object_deduplication.md

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

200 lines
8.7 KiB
Markdown
Raw Normal View History

2021-01-29 00:20:46 +05:30
---
2022-08-13 15:12:31 +05:30
stage: Systems
2021-04-17 20:07:23 +05:30
group: Gitaly
2021-02-22 17:27:13 +05:30
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
2021-01-29 00:20:46 +05:30
---
2019-07-07 11:18:12 +05:30
# How Git object deduplication works in GitLab
2019-12-26 22:10:19 +05:30
When a GitLab user [forks a project](../user/project/repository/forking_workflow.md),
2019-07-07 11:18:12 +05:30
GitLab creates a new Project with an associated Git repository that is a
copy of the original project at the time of the fork. If a large project
gets forked often, this can lead to a quick increase in Git repository
storage disk use. To counteract this problem, we are adding Git object
2021-02-22 17:27:13 +05:30
deduplication for forks to GitLab. In this document, we describe how
2019-07-07 11:18:12 +05:30
GitLab implements Git object deduplication.
## Pool repositories
### Understanding Git alternates
2022-08-27 11:52:29 +05:30
At the Git level, we achieve deduplication by using
[Git alternates](https://git-scm.com/docs/gitrepository-layout#gitrepository-layout-objects).
2019-07-07 11:18:12 +05:30
Git alternates is a mechanism that lets a repository borrow objects from
another repository on the same machine.
If we want repository A to borrow from repository B, we first write a
path that resolves to `B.git/objects` in the special file
`A.git/objects/info/alternates`. This establishes the alternates link.
Next, we must perform a Git repack in A. After the repack, any objects
2021-02-22 17:27:13 +05:30
that are duplicated between A and B are deleted from A. Repository
2019-07-07 11:18:12 +05:30
A is now no longer self-contained, but it still has its own refs and
2021-02-22 17:27:13 +05:30
configuration. Objects in A that are not in B remain in A. For this
2019-07-07 11:18:12 +05:30
to work, it is of course critical that **no objects ever get deleted from
B** because A might need them.
2021-02-22 17:27:13 +05:30
WARNING:
2021-06-08 01:23:25 +05:30
Do not run `git prune` or `git gc` in object pool repositories, which are
stored in the `@pools` directory. This can cause data loss in the regular
repositories that depend on the object pool.
2019-09-04 21:01:54 +05:30
The danger lies in `git prune`, and `git gc` calls `git prune`. The
problem is that `git prune`, when running in a pool repository, cannot
reliable decide if an object is no longer needed.
2019-07-07 11:18:12 +05:30
### Git alternates in GitLab: pool repositories
2022-08-27 11:52:29 +05:30
GitLab organizes this object borrowing by [creating special **pool repositories**](../administration/repository_storage_types.md)
which are hidden from the user. We then use Git
2019-07-07 11:18:12 +05:30
alternates to let a collection of project repositories borrow from a
single pool repository. We call such a collection of project
repositories a pool. Pools form star-shaped networks of repositories
2021-02-22 17:27:13 +05:30
that borrow from a single pool, which resemble (but not be
2019-07-07 11:18:12 +05:30
identical to) the fork networks that get formed when users fork
projects.
At the Git level, pool repositories are created and managed using Gitaly
RPC calls. Just like with normal repositories, the authority on which
pool repositories exist, and which repositories borrow from them, lies
at the Rails application level in SQL.
In conclusion, we need three things for effective object deduplication
across a collection of GitLab project repositories at the Git level:
2019-09-30 21:07:59 +05:30
1. A pool repository must exist.
1. The participating project repositories must be linked to the pool
repository via their respective `objects/info/alternates` files.
1. The pool repository must contain Git object data common to the
participating project repositories.
2019-07-07 11:18:12 +05:30
### Deduplication factor
The effectiveness of Git object deduplication in GitLab depends on the
amount of overlap between the pool repository and each of its
2019-09-04 21:01:54 +05:30
participants. Each time garbage collection runs on the source project,
2021-02-22 17:27:13 +05:30
Git objects from the source project are migrated to the pool
2019-09-04 21:01:54 +05:30
repository. One by one, as garbage collection runs, other member
2021-02-22 17:27:13 +05:30
projects benefit from the new objects that got added to the pool.
2019-07-07 11:18:12 +05:30
## SQL model
As of GitLab 11.8, project repositories in GitLab do not have their own
SQL table. They are indirectly identified by columns on the `projects`
table. In other words, the only way to look up a project repository is to
first look up its project, and then call `project.repository`.
With pool repositories we made a fresh start. These live in their own
`pool_repositories` SQL table. The relations between these two tables
are as follows:
2019-09-30 21:07:59 +05:30
- a `Project` belongs to at most one `PoolRepository`
(`project.pool_repository`)
- as an automatic consequence of the above, a `PoolRepository` has
many `Project`s
- a `PoolRepository` has exactly one "source `Project`"
(`pool.source_project`)
2019-07-07 11:18:12 +05:30
2019-09-04 21:01:54 +05:30
> TODO Fix invalid SQL data for pools created prior to GitLab 11.11
2020-06-23 00:09:42 +05:30
> <https://gitlab.com/gitlab-org/gitaly/-/issues/1653>.
2019-09-04 21:01:54 +05:30
2019-07-07 11:18:12 +05:30
### Assumptions
2022-08-27 11:52:29 +05:30
- All repositories in a pool must use [hashed storage](../administration/repository_storage_types.md).
This is so that we don't have to ever worry about updating paths in
2019-09-30 21:07:59 +05:30
`object/info/alternates` files.
- All repositories in a pool must be on the same Gitaly storage shard.
The Git alternates mechanism relies on direct disk access across
multiple repositories, and we can only assume direct disk access to
be possible within a Gitaly storage shard.
- The only two ways to remove a member project from a pool are (1) to
delete the project or (2) to move the project to another Gitaly
storage shard.
2019-07-07 11:18:12 +05:30
### Creating pools and pool memberships
2019-09-30 21:07:59 +05:30
- When a pool gets created, it must have a source project. The initial
contents of the pool repository are a Git clone of the source
project repository.
- The occasion for creating a pool is when an existing eligible
2020-01-01 13:55:28 +05:30
(non-private, hashed storage, non-forked) GitLab project gets forked and
2019-09-30 21:07:59 +05:30
this project does not belong to a pool repository yet. The fork
parent project becomes the source project of the new pool, and both
the fork parent and the fork child project become members of the new
pool.
- Once project A has become the source project of a pool, all future
2021-02-22 17:27:13 +05:30
eligible forks of A become pool members.
2019-09-30 21:07:59 +05:30
- If the fork source is itself a fork, the resulting repository will
2021-02-22 17:27:13 +05:30
neither join the repository nor is a new pool repository
2019-09-30 21:07:59 +05:30
seeded.
2021-02-22 17:27:13 +05:30
Such as:
2019-09-30 21:07:59 +05:30
Suppose fork A is part of a pool repository, any forks created off
2021-02-22 17:27:13 +05:30
of fork A *are not* a part of the pool repository that fork A is
2019-09-30 21:07:59 +05:30
a part of.
Suppose B is a fork of A, and A does not belong to an object pool.
2021-02-22 17:27:13 +05:30
Now C gets created as a fork of B. C is not part of a pool
2019-09-30 21:07:59 +05:30
repository.
2019-07-07 11:18:12 +05:30
> TODO should forks of forks be deduplicated?
2020-06-23 00:09:42 +05:30
> <https://gitlab.com/gitlab-org/gitaly/-/issues/1532>
2019-07-07 11:18:12 +05:30
### Consequences
2019-09-30 21:07:59 +05:30
- If a normal Project participating in a pool gets moved to another
Gitaly storage shard, its "belongs to PoolRepository" relation will
be broken. Because of the way moving repositories between shard is
2021-02-22 17:27:13 +05:30
implemented, we get a fresh self-contained copy
2019-09-30 21:07:59 +05:30
of the project's repository on the new storage shard.
- If the source project of a pool gets moved to another Gitaly storage
shard or is deleted the "source project" relation is not broken.
2021-02-22 17:27:13 +05:30
However, as of GitLab 12.0 a pool does not fetch from a source
2019-09-30 21:07:59 +05:30
unless the source is on the same Gitaly shard.
2019-07-07 11:18:12 +05:30
## Consistency between the SQL pool relation and Gitaly
As far as Gitaly is concerned, the SQL pool relations make two types of
claims about the state of affairs on the Gitaly server: pool repository
existence, and the existence of an alternates connection between a
repository and a pool.
### Pool existence
2021-09-30 23:02:18 +05:30
If GitLab thinks a pool repository exists (that is, it exists according to
2021-02-22 17:27:13 +05:30
SQL), but it does not on the Gitaly server, then it is created on
2019-09-04 21:01:54 +05:30
the fly by Gitaly.
2019-07-07 11:18:12 +05:30
### Pool relation existence
There are three different things that can go wrong here.
2021-02-22 17:27:13 +05:30
#### 1. SQL says repository A belongs to pool P but Gitaly says A has no alternate objects
2019-07-07 11:18:12 +05:30
2019-09-04 21:01:54 +05:30
In this case, we miss out on disk space savings but all RPC's on A
2021-02-22 17:27:13 +05:30
itself function fine. The next time garbage collection runs on A,
2019-09-04 21:01:54 +05:30
the alternates connection gets established in Gitaly. This is done by
2019-12-04 20:38:33 +05:30
`Projects::GitDeduplicationService` in GitLab Rails.
2019-07-07 11:18:12 +05:30
2021-02-22 17:27:13 +05:30
#### 2. SQL says repository A belongs to pool P1 but Gitaly says A has alternate objects in pool P2
2019-07-07 11:18:12 +05:30
2021-02-22 17:27:13 +05:30
In this case `Projects::GitDeduplicationService` throws an exception.
2019-07-07 11:18:12 +05:30
2021-02-22 17:27:13 +05:30
#### 3. SQL says repository A does not belong to any pool but Gitaly says A belongs to P
2019-07-07 11:18:12 +05:30
2021-02-22 17:27:13 +05:30
In this case `Projects::GitDeduplicationService` tries to
2019-09-04 21:01:54 +05:30
"re-duplicate" the repository A using the DisconnectGitAlternates RPC.
2019-07-07 11:18:12 +05:30
## Git object deduplication and GitLab Geo
When a pool repository record is created in SQL on a Geo primary, this
2021-02-22 17:27:13 +05:30
eventually triggers an event on the Geo secondary. The Geo secondary
then creates the pool repository in Gitaly. This leads to an
2019-07-07 11:18:12 +05:30
"eventually consistent" situation because as each pool participant gets
2021-02-22 17:27:13 +05:30
synchronized, Geo eventually triggers garbage collection in Gitaly on
the secondary, at which stage Git objects are deduplicated.
2019-07-07 11:18:12 +05:30
> TODO How do we handle the edge case where at the time the Geo
> secondary tries to create the pool repository, the source project does
2020-06-23 00:09:42 +05:30
> not exist? <https://gitlab.com/gitlab-org/gitaly/-/issues/1533>