debian-mirror-gitlab/doc/administration/geo/index.md

321 lines
17 KiB
Markdown
Raw Normal View History

2020-11-24 15:15:51 +05:30
---
stage: Enablement
group: Geo
2021-02-22 17:27:13 +05:30
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
2020-11-24 15:15:51 +05:30
type: howto
---
2021-03-11 19:13:27 +05:30
# Geo **(PREMIUM SELF)**
2020-11-24 15:15:51 +05:30
> - Introduced in GitLab Enterprise Edition 8.9.
> - Using Geo in combination with
> [multi-node architectures](../reference_architectures/index.md)
> is considered **Generally Available** (GA) in
> [GitLab Premium](https://about.gitlab.com/pricing/) 10.4.
Geo is the solution for widely distributed development teams and for providing a warm-standby as part of a disaster recovery strategy.
## Overview
2021-02-22 17:27:13 +05:30
WARNING:
2020-11-24 15:15:51 +05:30
Geo undergoes significant changes from release to release. Upgrades **are** supported and [documented](#updating-geo), but you should ensure that you're using the right version of the documentation for your installation.
Fetching large repositories can take a long time for teams located far from a single GitLab instance.
2021-06-08 01:23:25 +05:30
Geo provides local, read-only sites of your GitLab instances. This can reduce the time it takes
2020-11-24 15:15:51 +05:30
to clone and fetch large repositories, speeding up development.
For a video introduction to Geo, see [Introduction to GitLab Geo - GitLab Features](https://www.youtube.com/watch?v=-HDLxSjEh6w).
2021-09-04 01:27:46 +05:30
To make sure you're using the right version of the documentation, navigate to [the Geo page on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/administration/geo/index.md) and choose the appropriate release from the **Switch branch/tag** dropdown. For example, [`v13.7.6-ee`](https://gitlab.com/gitlab-org/gitlab/-/blob/v13.7.6-ee/doc/administration/geo/index.md).
2021-06-08 01:23:25 +05:30
Geo uses a set of defined terms that is described in the [Geo Glossary](glossary.md), please familiarize yourself with those terms.
2020-11-24 15:15:51 +05:30
## Use cases
Implementing Geo provides the following benefits:
- Reduce from minutes to seconds the time taken for your distributed developers to clone and fetch large repositories and projects.
- Enable all of your developers to contribute ideas and work in parallel, no matter where they are.
2021-06-08 01:23:25 +05:30
- Balance the read-only load between your **primary** and **secondary** sites.
2020-11-24 15:15:51 +05:30
In addition, it:
2021-01-03 14:25:43 +05:30
- Can be used for cloning and fetching projects, in addition to reading any data available in the GitLab web interface (see [limitations](#limitations)).
2020-11-24 15:15:51 +05:30
- Overcomes slow connections between distant offices, saving time by improving speed for distributed teams.
- Helps reducing the loading time for automated tasks, custom integrations, and internal workflows.
2021-06-08 01:23:25 +05:30
- Can quickly fail over to a **secondary** site in a [disaster recovery](disaster_recovery/index.md) scenario.
- Allows [planned failover](disaster_recovery/planned_failover.md) to a **secondary** site.
2020-11-24 15:15:51 +05:30
Geo provides:
2021-06-08 01:23:25 +05:30
- Read-only **secondary** sites: Maintain one **primary** GitLab site while still enabling read-only **secondary** sites for each of your distributed teams.
- Authentication system hooks: **Secondary** sites receives all authentication data (like user accounts and logins) from the **primary** instance.
- An intuitive UI: **Secondary** sites use the same web interface your team has grown accustomed to. In addition, there are visual notifications that block write operations and make it clear that a user is on a **secondary** sites.
2020-11-24 15:15:51 +05:30
2021-02-22 17:27:13 +05:30
### Gitaly Cluster
Geo should not be confused with [Gitaly Cluster](../gitaly/praefect.md). For more information about
2021-09-04 01:27:46 +05:30
the difference between Geo and Gitaly Cluster, see
[How does Gitaly Cluster compare to Geo?](../gitaly/faq.md#how-does-gitaly-cluster-compare-to-geo).
2021-02-22 17:27:13 +05:30
2020-11-24 15:15:51 +05:30
## How it works
2021-09-04 01:27:46 +05:30
Your Geo instance can be used for cloning and fetching projects, in addition to reading any data. This makes working with large repositories over large distances much faster.
2020-11-24 15:15:51 +05:30
![Geo overview](replication/img/geo_overview.png)
When Geo is enabled, the:
2021-06-08 01:23:25 +05:30
- Original instance is known as the **primary** site.
- Replicated read-only sites are known as **secondary** sites.
2020-11-24 15:15:51 +05:30
Keep in mind that:
2021-06-08 01:23:25 +05:30
- **Secondary** sites talk to the **primary** site to:
2020-11-24 15:15:51 +05:30
- Get user data for logins (API).
- Replicate repositories, LFS Objects, and Attachments (HTTPS + JWT).
2021-06-08 01:23:25 +05:30
- In GitLab Premium 10.0 and later, the **primary** site no longer talks to **secondary** sites to notify for changes (API).
- Pushing directly to a **secondary** site (for both HTTP and SSH, including Git LFS) was [introduced](https://about.gitlab.com/releases/2018/09/22/gitlab-11-3-released/) in [GitLab Premium](https://about.gitlab.com/pricing/#self-managed) 11.3.
2021-01-03 14:25:43 +05:30
- There are [limitations](#limitations) when using Geo.
2020-11-24 15:15:51 +05:30
### Architecture
The following diagram illustrates the underlying architecture of Geo.
![Geo architecture](replication/img/geo_architecture.png)
In this diagram:
2021-06-08 01:23:25 +05:30
- There is the **primary** site and the details of one **secondary** site.
- Writes to the database can only be performed on the **primary** site. A **secondary** site receives database
2020-11-24 15:15:51 +05:30
updates via PostgreSQL streaming replication.
- If present, the [LDAP server](#ldap) should be configured to replicate for [Disaster Recovery](disaster_recovery/index.md) scenarios.
2021-06-08 01:23:25 +05:30
- A **secondary** site performs different type of synchronizations against the **primary** site, using a special
2020-11-24 15:15:51 +05:30
authorization protected by JWT:
- Repositories are cloned/updated via Git over HTTPS.
- Attachments, LFS objects, and other files are downloaded via HTTPS using a private API endpoint.
From the perspective of a user performing Git operations:
2021-06-08 01:23:25 +05:30
- The **primary** site behaves as a full read-write GitLab instance.
- **Secondary** sites are read-only but proxy Git push operations to the **primary** site. This makes **secondary** sites appear to support push operations themselves.
2020-11-24 15:15:51 +05:30
To simplify the diagram, some necessary components are omitted. Note that:
- Git over SSH requires [`gitlab-shell`](https://gitlab.com/gitlab-org/gitlab-shell) and OpenSSH.
- Git over HTTPS required [`gitlab-workhorse`](https://gitlab.com/gitlab-org/gitlab-workhorse).
2021-06-08 01:23:25 +05:30
Note that a **secondary** site needs two different PostgreSQL databases:
2020-11-24 15:15:51 +05:30
- A read-only database instance that streams data from the main GitLab database.
2021-06-08 01:23:25 +05:30
- [Another database instance](#geo-tracking-database) used internally by the **secondary** site to record what data has been replicated.
2020-11-24 15:15:51 +05:30
2021-06-08 01:23:25 +05:30
In **secondary** sites, there is an additional daemon: [Geo Log Cursor](#geo-log-cursor).
2020-11-24 15:15:51 +05:30
## Requirements for running Geo
The following are required to run Geo:
- An operating system that supports OpenSSH 6.9+ (needed for
[fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md))
The following operating systems are known to ship with a current version of OpenSSH:
- [CentOS](https://www.centos.org) 7.4+
- [Ubuntu](https://ubuntu.com) 16.04+
2021-09-04 01:27:46 +05:30
- PostgreSQL 12+ with [Streaming Replication](https://wiki.postgresql.org/wiki/Streaming_Replication)
2020-11-24 15:15:51 +05:30
- Git 2.9+
2021-01-29 00:20:46 +05:30
- Git-lfs 2.4.2+ on the user side when using LFS
2021-06-08 01:23:25 +05:30
- All sites must run the same GitLab version.
2020-11-24 15:15:51 +05:30
2021-02-22 17:27:13 +05:30
Additionally, check the GitLab [minimum requirements](../../install/requirements.md),
2020-11-24 15:15:51 +05:30
and we recommend you use:
- At least GitLab Enterprise Edition 10.0 for basic Geo features.
- The latest version for a better experience.
### Firewall rules
2021-06-08 01:23:25 +05:30
The following table lists basic ports that must be open between the **primary** and **secondary** sites for Geo.
2020-11-24 15:15:51 +05:30
2021-06-08 01:23:25 +05:30
| **Primary** site | **Secondary** site | Protocol |
2020-11-24 15:15:51 +05:30
|:-----------------|:-------------------|:-------------|
| 80 | 80 | HTTP |
| 443 | 443 | TCP or HTTPS |
| 22 | 22 | TCP |
| 5432 | | PostgreSQL |
See the full list of ports used by GitLab in [Package defaults](https://docs.gitlab.com/omnibus/package-information/defaults.html)
2021-02-22 17:27:13 +05:30
NOTE:
2020-11-24 15:15:51 +05:30
[Web terminal](../../ci/environments/index.md#web-terminals) support requires your load balancer to correctly handle WebSocket connections.
When using HTTP or HTTPS proxying, your load balancer must be configured to pass through the `Connection` and `Upgrade` hop-by-hop headers. See the [web terminal](../integration/terminal.md) integration guide for more details.
2021-02-22 17:27:13 +05:30
NOTE:
2021-09-04 01:27:46 +05:30
When using HTTPS protocol for port 443, you need to add an SSL certificate to the load balancers.
2020-11-24 15:15:51 +05:30
If you wish to terminate SSL at the GitLab application server instead, use TCP protocol.
### LDAP
2021-09-04 01:27:46 +05:30
We recommend that if you use LDAP on your **primary** site, you also set up secondary LDAP servers on each **secondary** site. Otherwise, users are unable to perform Git operations over HTTP(s) on the **secondary** site using HTTP Basic Authentication. However, Git via SSH and personal access tokens still works.
2020-11-24 15:15:51 +05:30
2021-02-22 17:27:13 +05:30
NOTE:
2021-09-04 01:27:46 +05:30
It is possible for all **secondary** sites to share an LDAP server, but additional latency can be an issue. Also, consider what LDAP server is available in a [disaster recovery](disaster_recovery/index.md) scenario if a **secondary** site is promoted to be a **primary** site.
2020-11-24 15:15:51 +05:30
2021-09-04 01:27:46 +05:30
Check for instructions on how to set up replication in your LDAP service. Instructions are different depending on the software or service used. For example, OpenLDAP provides [these instructions](https://www.openldap.org/doc/admin24/replication.html).
2020-11-24 15:15:51 +05:30
### Geo Tracking Database
The tracking database instance is used as metadata to control what needs to be updated on the disk of the local instance. For example:
- Download new assets.
- Fetch new LFS Objects.
- Fetch changes from a repository that has recently been updated.
2021-06-08 01:23:25 +05:30
Because the replicated database instance is read-only, we need this additional database instance for each **secondary** site.
2020-11-24 15:15:51 +05:30
### Geo Log Cursor
This daemon:
2021-06-08 01:23:25 +05:30
- Reads a log of events replicated by the **primary** site to the **secondary** database instance.
2020-11-24 15:15:51 +05:30
- Updates the Geo Tracking Database instance with changes that need to be executed.
2021-09-04 01:27:46 +05:30
When something is marked to be updated in the tracking database instance, asynchronous jobs running on the **secondary** site execute the required operations and update the state.
2021-06-08 01:23:25 +05:30
2021-09-04 01:27:46 +05:30
This new architecture allows GitLab to be resilient to connectivity issues between the sites. It doesn't matter how long the **secondary** site is disconnected from the **primary** site as it is able to replay all the events in the correct order and become synchronized with the **primary** site again.
2021-06-08 01:23:25 +05:30
## Limitations
WARNING:
This list of limitations only reflects the latest version of GitLab. If you are using an older version, extra limitations may be in place.
2020-11-24 15:15:51 +05:30
2021-06-08 01:23:25 +05:30
- Pushing directly to a **secondary** site redirects (for HTTP) or proxies (for SSH) the request to the **primary** site instead of [handling it directly](https://gitlab.com/gitlab-org/gitlab/-/issues/1381), except when using Git over HTTP with credentials embedded within the URI. For example, `https://user:password@secondary.tld`.
- The **primary** site has to be online for OAuth login to happen. Existing sessions and Git are not affected. Support for the **secondary** site to use an OAuth provider independent from the primary is [being planned](https://gitlab.com/gitlab-org/gitlab/-/issues/208465).
- The installation takes multiple manual steps that together can take about an hour depending on circumstances. We are working on improving this experience. See [Omnibus GitLab issue #2978](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/2978) for details.
- Real-time updates of issues/merge requests (for example, via long polling) doesn't work on the **secondary** site.
- [Selective synchronization](replication/configuration.md#selective-synchronization) applies only to files and repositories. Other datasets are replicated to the **secondary** site in full, making it inappropriate for use as an access control mechanism.
- Object pools for forked project deduplication work only on the **primary** site, and are duplicated on the **secondary** site.
- GitLab Runners cannot register with a **secondary** site. Support for this is [planned for the future](https://gitlab.com/gitlab-org/gitlab/-/issues/3294).
- Configuring Geo **secondary** sites to [use high-availability configurations of PostgreSQL](https://gitlab.com/groups/gitlab-org/-/epics/2536) is currently in **alpha** support.
2021-09-04 01:27:46 +05:30
- [Selective synchronization](replication/configuration.md#selective-synchronization) only limits what repositories are replicated. The entire PostgreSQL data is still replicated. Selective synchronization is not built to accommodate compliance / export control use cases.
2021-06-08 01:23:25 +05:30
### Limitations on replication/verification
There is a complete list of all GitLab [data types](replication/datatypes.md) and [existing support for replication and verification](replication/datatypes.md#limitations-on-replicationverification).
2020-11-24 15:15:51 +05:30
## Setup instructions
For setup instructions, see [Setting up Geo](setup/index.md).
## Post-installation documentation
2021-06-08 01:23:25 +05:30
After installing GitLab on the **secondary** site(s) and performing the initial configuration, see the following documentation for post-installation information.
2020-11-24 15:15:51 +05:30
### Configuring Geo
For information on configuring Geo, see [Geo configuration](replication/configuration.md).
### Updating Geo
2021-06-08 01:23:25 +05:30
For information on how to update your Geo site(s) to the latest GitLab version, see [Updating the Geo sites](replication/updating_the_geo_nodes.md).
2020-11-24 15:15:51 +05:30
### Pausing and resuming replication
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/35913) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.2.
2021-02-22 17:27:13 +05:30
WARNING:
2021-06-08 01:23:25 +05:30
In GitLab 13.2 and 13.3, promoting a secondary site to a primary while the
2021-01-29 00:20:46 +05:30
secondary is paused fails. Do not pause replication before promoting a
2021-06-08 01:23:25 +05:30
secondary. If the site is paused, be sure to resume before promoting. This
2021-01-29 00:20:46 +05:30
issue has been fixed in GitLab 13.4 and later.
2021-02-22 17:27:13 +05:30
WARNING:
2020-12-08 15:28:05 +05:30
Pausing and resuming of replication is currently only supported for Geo installations using an
Omnibus GitLab-managed database. External databases are currently not supported.
2020-11-24 15:15:51 +05:30
In some circumstances, like during [upgrades](replication/updating_the_geo_nodes.md) or a [planned failover](disaster_recovery/planned_failover.md), it is desirable to pause replication between the primary and secondary.
2021-06-08 01:23:25 +05:30
Pausing and resuming replication is done via a command line tool from the a node in the secondary site where the `postgresql` service is enabled.
2021-02-22 17:27:13 +05:30
If `postgresql` is on a standalone database node, ensure that `gitlab.rb` on that node contains the configuration line `gitlab_rails['geo_node_name'] = 'node_name'`, where `node_name` is the same as the `geo_name_name` on the application node.
2020-11-24 15:15:51 +05:30
**To Pause: (from secondary)**
```shell
gitlab-ctl geo-replication-pause
```
**To Resume: (from secondary)**
```shell
gitlab-ctl geo-replication-resume
```
### Configuring Geo for multiple nodes
For information on configuring Geo for multiple nodes, see [Geo for multiple servers](replication/multiple_servers.md).
### Configuring Geo with Object Storage
For information on configuring Geo with object storage, see [Geo with Object storage](replication/object_storage.md).
### Disaster Recovery
For information on using Geo in disaster recovery situations to mitigate data-loss and restore services, see [Disaster Recovery](disaster_recovery/index.md).
### Replicating the Container Registry
2021-06-08 01:23:25 +05:30
For more information on how to replicate the Container Registry, see [Docker Registry for a **secondary** site](replication/docker_registry.md).
2020-11-24 15:15:51 +05:30
### Security Review
For more information on Geo security, see [Geo security review](replication/security_review.md).
### Tuning Geo
For more information on tuning Geo, see [Tuning Geo](replication/tuning.md).
### Set up a location-aware Git URL
For an example of how to set up a location-aware Git remote URL with AWS Route53, see [Location-aware Git remote URL with AWS Route53](replication/location_aware_git_url.md).
2021-02-22 17:27:13 +05:30
### Backfill
2021-09-04 01:27:46 +05:30
Once a **secondary** site is set up, it starts replicating missing data from
2021-06-08 01:23:25 +05:30
the **primary** site in a process known as **backfill**. You can monitor the
synchronization process on each Geo site from the **primary** site's **Geo Nodes**
2021-02-22 17:27:13 +05:30
dashboard in your browser.
Failures that happen during a backfill are scheduled to be retried at the end
of the backfill.
2021-06-08 01:23:25 +05:30
## Remove Geo site
2020-11-24 15:15:51 +05:30
2021-06-08 01:23:25 +05:30
For more information on removing a Geo site, see [Removing **secondary** Geo sites](replication/remove_geo_site.md).
2020-11-24 15:15:51 +05:30
## Disable Geo
To find out how to disable Geo, see [Disabling Geo](replication/disable_geo.md).
## Frequently Asked Questions
For answers to common questions, see the [Geo FAQ](replication/faq.md).
## Log files
2021-01-03 14:25:43 +05:30
In GitLab 9.5 and later, Geo stores structured log messages in a `geo.log` file. For Omnibus installations, this file is at `/var/log/gitlab/gitlab-rails/geo.log`.
2020-11-24 15:15:51 +05:30
This file contains information about when Geo attempts to sync repositories and files. Each line in the file contains a separate JSON entry that can be ingested into. For example, Elasticsearch or Splunk.
For example:
```json
{"severity":"INFO","time":"2017-08-06T05:40:16.104Z","message":"Repository update","project_id":1,"source":"repository","resync_repository":true,"resync_wiki":true,"class":"Gitlab::Geo::LogCursor::Daemon","cursor_delay_s":0.038}
```
This message shows that Geo detected that a repository update was needed for project `1`.
## Troubleshooting
For troubleshooting steps, see [Geo Troubleshooting](replication/troubleshooting.md).