51 lines
3.5 KiB
Markdown
51 lines
3.5 KiB
Markdown
---
|
|
stage: none
|
|
group: unassigned
|
|
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
|
|
---
|
|
|
|
# Repository mirroring
|
|
|
|
## Deep Dive
|
|
|
|
<!-- vale gitlab.Spelling = NO -->
|
|
|
|
In December 2018, Tiago Botelho hosted a Deep Dive (GitLab team members only: `https://gitlab.com/gitlab-org/create-stage/issues/1`)
|
|
on the GitLab [Pull Repository Mirroring functionality](../user/project/repository/repository_mirroring.md#pull-from-a-remote-repository)
|
|
to share his domain specific knowledge with anyone who may work in this part of the
|
|
codebase in the future. You can find the <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [recording on YouTube](https://www.youtube.com/watch?v=sSZq0fpdY-Y),
|
|
and the slides in [PDF](https://gitlab.com/gitlab-org/create-stage/uploads/8693404888a941fd851f8a8ecdec9675/Gitlab_Create_-_Pull_Mirroring_Deep_Dive.pdf).
|
|
Everything covered in this deep dive was accurate as of GitLab 11.6, and while specific
|
|
details may have changed since then, it should still serve as a good introduction.
|
|
|
|
<!-- vale gitlab.Spelling = YES -->
|
|
|
|
## Explanation of mirroring process
|
|
|
|
GitLab version 14 performs these steps when an
|
|
[API call](../api/projects.md#start-the-pull-mirroring-process-for-a-project)
|
|
triggers a pull mirror. Scheduled mirror updates are similar, but do not start with the API call:
|
|
|
|
1. The request originates from an API call, and triggers the `start_pull_mirroring_service` in
|
|
[`project_mirror.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/api/project_mirror.rb).
|
|
1. The pull mirroring service
|
|
([`start_pull_mirroring_service.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/services/start_pull_mirroring_service.rb)) starts. It updates the project state, and forces the job to start immediately.
|
|
1. The project import state is updated, and then triggers an `update_all_mirrors_worker` in
|
|
[`project_import_state.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/models/ee/project_import_state.rb#L170).
|
|
1. The update all mirrors worker
|
|
([`update_all_mirrors_worker.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/update_all_mirrors_worker.rb))
|
|
attempts to avoid stampedes by calling the `project_import_schedule` worker.
|
|
1. The project import schedule worker
|
|
([`project_import_schedule_worker.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/project_import_schedule_worker.rb#L21)) updates the state of the project, and
|
|
starts a Ruby `state_machine` to manage the import transition process.
|
|
1. While updating the project state,
|
|
[this call in `project.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/models/ee/project.rb#L426)
|
|
starts the `repository_update_mirror` worker.
|
|
1. The Sidekiq background mirror workers
|
|
([`repository_update_mirror_worker.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/repository_update_mirror_worker.rb)) track the state of the mirroring task, and
|
|
provide good error state information. Processes can hang here, because this step manages the Git steps.
|
|
1. The update mirror service
|
|
([`update_mirror_service.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/services/projects/update_mirror_service.rb))
|
|
performs the Git operations.
|
|
|
|
The import and mirror update processes are complete after the update mirror service step. However, depending on the changes included, more tasks (such as pipelines for commits) can be triggered.
|