debian-mirror-gitlab/doc/development/geo/framework.md

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

180 lines
7.1 KiB
Markdown
Raw Normal View History

2021-01-03 14:25:43 +05:30
---
stage: Enablement
group: Geo
2021-02-22 17:27:13 +05:30
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
2021-01-03 14:25:43 +05:30
---
2021-01-29 00:20:46 +05:30
# Geo self-service framework
2020-03-13 15:44:24 +05:30
2021-02-22 17:27:13 +05:30
NOTE:
2021-01-29 00:20:46 +05:30
This document is subject to change as we continue to implement and iterate on the framework.
Follow the progress in the [epic](https://gitlab.com/groups/gitlab-org/-/epics/2161).
If you need to replicate a new data type, reach out to the Geo
2020-03-13 15:44:24 +05:30
team to discuss the options. You can contact them in `#g_geo` on Slack
or mention `@geo-team` in the issue or merge request.
Geo provides an API to make it possible to easily replicate data types
2022-03-02 08:16:31 +05:30
across Geo sites. This API is presented as a Ruby Domain-Specific
2020-03-13 15:44:24 +05:30
Language (DSL) and aims to make it possible to replicate data with
minimal effort of the engineer who created a data type.
## Nomenclature
Before digging into the API, developers need to know some Geo-specific
2021-01-29 00:20:46 +05:30
naming conventions:
2020-03-13 15:44:24 +05:30
2021-01-29 00:20:46 +05:30
- **Model**:
A model is an Active Model, which is how it is known in the entire
2020-03-13 15:44:24 +05:30
Rails codebase. It usually is tied to a database table. From Geo
perspective, a model can have one or more resources.
2021-01-29 00:20:46 +05:30
- **Resource**:
A resource is a piece of data that belongs to a model and is
2020-03-13 15:44:24 +05:30
produced by a GitLab feature. It is persisted using a storage
2021-01-29 00:20:46 +05:30
mechanism. By default, a resource is not a Geo replicable.
2020-03-13 15:44:24 +05:30
2021-01-29 00:20:46 +05:30
- **Data type**:
Data type is how a resource is stored. Each resource should
2020-03-13 15:44:24 +05:30
fit in one of the data types Geo supports:
2021-01-29 00:20:46 +05:30
- Git repository
- Blob
- Database
For more detail, see [Data types](../../administration/geo/replication/datatypes.md).
2020-03-13 15:44:24 +05:30
2021-01-29 00:20:46 +05:30
- **Geo Replicable**:
2022-03-02 08:16:31 +05:30
A Replicable is a resource Geo wants to sync across Geo sites. There
2020-03-13 15:44:24 +05:30
is a limited set of supported data types of replicables. The effort
required to implement replication of a resource that belongs to one
of the known data types is minimal.
2021-01-29 00:20:46 +05:30
- **Geo Replicator**:
A Geo Replicator is the object that knows how to replicate a
2020-03-13 15:44:24 +05:30
replicable. It's responsible for:
2021-01-29 00:20:46 +05:30
- Firing events (producer)
- Consuming events (consumer)
It's tied to the Geo Replicable data type. All replicators have a
2020-03-13 15:44:24 +05:30
common interface that can be used to process (that is, produce and
consume) events. It takes care of the communication between the
2022-03-02 08:16:31 +05:30
primary site (where events are produced) and the secondary site
2020-03-13 15:44:24 +05:30
(where events are consumed). The engineer who wants to incorporate
Geo in their feature will use the API of replicators to make this
happen.
2021-01-29 00:20:46 +05:30
- **Geo Domain-Specific Language**:
The syntactic sugar that allows engineers to easily specify which
2020-03-13 15:44:24 +05:30
resources should be replicated and how.
## Geo Domain-Specific Language
### The replicator
First of all, you need to write a replicator. The replicators live in
[`ee/app/replicators/geo`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/app/replicators/geo).
For each resource that needs to be replicated, there should be a
separate replicator specified, even if multiple resources are tied to
the same model.
For example, the following replicator replicates a package file:
```ruby
module Geo
class PackageFileReplicator < Gitlab::Geo::Replicator
# Include one of the strategies your resource needs
include ::Geo::BlobReplicatorStrategy
# Specify the CarrierWave uploader needed by the used strategy
def carrierwave_uploader
model_record.file
end
# Specify the model this replicator belongs to
2020-05-24 23:13:21 +05:30
def self.model
2020-03-13 15:44:24 +05:30
::Packages::PackageFile
end
2020-10-24 23:57:45 +05:30
# The feature flag follows the format `geo_#{replicable_name}_replication`,
# so here it would be `geo_package_file_replication`
def self.replication_enabled_by_default?
false
end
2020-03-13 15:44:24 +05:30
end
end
```
The class name should be unique. It also is tightly coupled to the
table name for the registry, so for this example the registry table
will be `package_file_registry`.
For the different data types Geo supports there are different
strategies to include. Pick one that fits your needs.
### Linking to a model
To tie this replicator to the model, you need to add the following to
the model code:
```ruby
class Packages::PackageFile < ApplicationRecord
2022-03-02 08:16:31 +05:30
include ::Geo::ReplicableModel
2020-03-13 15:44:24 +05:30
with_replicator Geo::PackageFileReplicator
end
```
### API
When this is set in place, it's easy to access the replicator through
the model:
```ruby
2020-11-24 15:15:51 +05:30
package_file = Packages::PackageFile.find(4) # just a random ID as example
2020-03-13 15:44:24 +05:30
replicator = package_file.replicator
```
Or get the model back from the replicator:
```ruby
replicator.model_record
=> <Packages::PackageFile id:4>
```
The replicator can be used to generate events, for example in
2021-01-29 00:20:46 +05:30
`ActiveRecord` hooks:
2020-03-13 15:44:24 +05:30
```ruby
after_create_commit -> { replicator.publish_created_event }
```
#### Library
The framework behind all this is located in
[`ee/lib/gitlab/geo/`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/lib/gitlab/geo).
2020-04-08 14:13:33 +05:30
## Existing Replicator Strategies
Before writing a new kind of Replicator Strategy, check below to see if your
resource can already be handled by one of the existing strategies. Consult with
the Geo team if you are unsure.
### Blob Replicator Strategy
2021-04-29 21:17:54 +05:30
Models that use [CarrierWave's](https://github.com/carrierwaveuploader/carrierwave) `Uploader::Base` are supported by Geo with the `Geo::BlobReplicatorStrategy` module. For example, see how [Geo replication was implemented for Pipeline Artifacts](https://gitlab.com/gitlab-org/gitlab/-/issues/238464).
2020-04-08 14:13:33 +05:30
2021-04-29 21:17:54 +05:30
Each file is expected to have its own primary ID and model. Geo strongly recommends treating *every single file* as a first-class citizen, because in our experience this greatly simplifies tracking replication and verification state.
2020-04-08 14:13:33 +05:30
2021-06-08 01:23:25 +05:30
To implement Geo replication of a new blob-type Model, [open an issue with the provided issue template](https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Geo%20Replicate%20a%20new%20blob%20type).
To view the implementation steps without opening an issue, [view the issue template file](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/Geo%20Replicate%20a%20new%20blob%20type.md).
2021-03-08 18:12:59 +05:30
### Repository Replicator Strategy
2021-04-29 21:17:54 +05:30
Models that refer to any Git repository on disk are supported by Geo with the `Geo::RepositoryReplicatorStrategy` module. For example, see how [Geo replication was implemented for Group-level Wikis](https://gitlab.com/gitlab-org/gitlab/-/issues/208147). Note that this issue does not implement verification, since verification of Git repositories was not yet added to the Geo self-service framework. An example implementing verification can be found in the merge request to [Add Snippet repository verification](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/56596).
2021-03-08 18:12:59 +05:30
2021-04-29 21:17:54 +05:30
Each Git repository is expected to have its own primary ID and model.
2021-03-08 18:12:59 +05:30
2021-06-08 01:23:25 +05:30
To implement Geo replication of a new Git repository-type Model, [open an issue with the provided issue template](https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Geo%20Replicate%20a%20new%20Git%20repository%20type).
To view the implementation steps without opening an issue, [view the issue template file](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/Geo%20Replicate%20a%20new%20Git%20repository%20type.md).