2020-06-23 00:09:42 +05:30
---
2022-07-23 23:45:48 +05:30
stage: Systems
2020-06-23 00:09:42 +05:30
group: Geo
2021-02-22 17:27:13 +05:30
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
2020-06-23 00:09:42 +05:30
type: howto
---
2021-03-11 19:13:27 +05:30
# Geo configuration **(PREMIUM SELF)**
2019-07-31 22:56:46 +05:30
2021-10-27 15:23:28 +05:30
## Configuring a new **secondary** site
2019-07-31 22:56:46 +05:30
2021-02-22 17:27:13 +05:30
NOTE:
2021-10-27 15:23:28 +05:30
This is the final step in setting up a **secondary** Geo site. Stages of the
2019-07-31 22:56:46 +05:30
setup process must be completed in the documented order.
2020-11-24 15:15:51 +05:30
Before attempting the steps in this stage, [complete all prior stages ](../setup/index.md#using-omnibus-gitlab ).
2019-07-31 22:56:46 +05:30
2021-10-27 15:23:28 +05:30
The basic steps of configuring a **secondary** site are to:
2019-07-31 22:56:46 +05:30
2021-10-27 15:23:28 +05:30
- Replicate required configurations between the **primary** site and the **secondary** sites.
- Configure a tracking database on each **secondary** site.
- Start GitLab on each **secondary** site.
2019-07-31 22:56:46 +05:30
You are encouraged to first read through all the steps before executing them
in your testing/production environment.
2021-02-22 17:27:13 +05:30
NOTE:
2021-10-27 15:23:28 +05:30
**Do not** set up any custom authentication for the **secondary** sites. This is handled by the **primary** site.
2019-09-30 21:07:59 +05:30
Any change that requires access to the **Admin Area** needs to be done in the
2021-10-27 15:23:28 +05:30
**primary** site because the **secondary** site is a read-only replica.
2019-07-31 22:56:46 +05:30
### Step 1. Manually replicate secret GitLab values
GitLab stores a number of secret values in the `/etc/gitlab/gitlab-secrets.json`
2021-10-27 15:23:28 +05:30
file which *must* be the same on all of a site's nodes. Until there is
a means of automatically replicating these between sites (see [issue #3789 ](https://gitlab.com/gitlab-org/gitlab/-/issues/3789 )),
they must be manually replicated to **all nodes of the secondary site** .
2019-07-31 22:56:46 +05:30
2021-10-27 15:23:28 +05:30
1. SSH into a **Rails node on your primary** site, and execute the command below:
2019-07-31 22:56:46 +05:30
2020-03-13 15:44:24 +05:30
```shell
2019-09-30 21:07:59 +05:30
sudo cat /etc/gitlab/gitlab-secrets.json
```
2019-07-31 22:56:46 +05:30
2021-06-08 01:23:25 +05:30
This displays the secrets that need to be replicated, in JSON format.
2019-07-31 22:56:46 +05:30
2021-10-27 15:23:28 +05:30
1. SSH **into each node on your secondary Geo site** and login as the `root` user:
2019-07-31 22:56:46 +05:30
2020-03-13 15:44:24 +05:30
```shell
2019-09-30 21:07:59 +05:30
sudo -i
```
2019-07-31 22:56:46 +05:30
1. Make a backup of any existing secrets:
2020-03-13 15:44:24 +05:30
```shell
2019-09-30 21:07:59 +05:30
mv /etc/gitlab/gitlab-secrets.json /etc/gitlab/gitlab-secrets.json.`date +%F`
```
2019-07-31 22:56:46 +05:30
2021-10-27 15:23:28 +05:30
1. Copy `/etc/gitlab/gitlab-secrets.json` from the **Rails node on your primary** site to **each node on your secondary** site, or
2019-07-31 22:56:46 +05:30
copy-and-paste the file contents between nodes:
2020-03-13 15:44:24 +05:30
```shell
2019-09-30 21:07:59 +05:30
sudo editor /etc/gitlab/gitlab-secrets.json
2019-07-31 22:56:46 +05:30
2019-09-30 21:07:59 +05:30
# paste the output of the `cat` command you ran on the primary
# save and exit
```
2019-07-31 22:56:46 +05:30
1. Ensure the file permissions are correct:
2020-03-13 15:44:24 +05:30
```shell
2019-09-30 21:07:59 +05:30
chown root:root /etc/gitlab/gitlab-secrets.json
chmod 0600 /etc/gitlab/gitlab-secrets.json
```
2019-07-31 22:56:46 +05:30
2021-10-27 15:23:28 +05:30
1. Reconfigure **each Rails, Sidekiq and Gitaly nodes on your secondary** site for the change to take effect:
2019-07-31 22:56:46 +05:30
2020-03-13 15:44:24 +05:30
```shell
2019-09-30 21:07:59 +05:30
gitlab-ctl reconfigure
gitlab-ctl restart
```
2019-07-31 22:56:46 +05:30
2021-10-27 15:23:28 +05:30
### Step 2. Manually replicate the **primary** site's SSH host keys
2019-07-31 22:56:46 +05:30
GitLab integrates with the system-installed SSH daemon, designating a user
2019-12-21 20:55:43 +05:30
(typically named `git` ) through which all access requests are handled.
2019-07-31 22:56:46 +05:30
2020-04-22 19:07:51 +05:30
In a [Disaster Recovery ](../disaster_recovery/index.md ) situation, GitLab system
2021-10-27 15:23:28 +05:30
administrators promote a **secondary** site to the **primary** site. DNS records for the
**primary** domain should also be updated to point to the new **primary** site
(previously a **secondary** site). Doing so avoids the need to update Git remotes and API URLs.
2019-07-31 22:56:46 +05:30
2021-10-27 15:23:28 +05:30
This causes all SSH requests to the newly promoted **primary** site to
2019-07-31 22:56:46 +05:30
fail due to SSH host key mismatch. To prevent this, the primary SSH host
2021-10-27 15:23:28 +05:30
keys must be manually replicated to the **secondary** site.
2019-07-31 22:56:46 +05:30
2021-10-27 15:23:28 +05:30
1. SSH into **each node on your secondary** site and login as the `root` user:
2019-07-31 22:56:46 +05:30
2020-03-13 15:44:24 +05:30
```shell
2019-09-30 21:07:59 +05:30
sudo -i
```
2019-07-31 22:56:46 +05:30
1. Make a backup of any existing SSH host keys:
2020-03-13 15:44:24 +05:30
```shell
2019-09-30 21:07:59 +05:30
find /etc/ssh -iname ssh_host_* -exec cp {} {}.backup.`date +%F` \;
```
2019-07-31 22:56:46 +05:30
2021-10-27 15:23:28 +05:30
1. Copy OpenSSH host keys from the **primary** site:
2019-07-31 22:56:46 +05:30
2021-10-27 15:23:28 +05:30
If you can access one of the **nodes on your primary** site serving SSH traffic (usually, the main GitLab Rails application nodes) using the **root** user:
2019-07-31 22:56:46 +05:30
2020-03-13 15:44:24 +05:30
```shell
2021-10-27 15:23:28 +05:30
# Run this from the secondary site, change `<primary_site_fqdn>` for the IP or FQDN of the server
2019-09-30 21:07:59 +05:30
scp root@< primary_node_fqdn > :/etc/ssh/ssh_host_*_key* /etc/ssh
```
2019-07-31 22:56:46 +05:30
2020-04-08 14:13:33 +05:30
If you only have access through a user with `sudo` privileges:
2019-07-31 22:56:46 +05:30
2020-03-13 15:44:24 +05:30
```shell
2021-10-27 15:23:28 +05:30
# Run this from the node on your primary site:
2019-09-30 21:07:59 +05:30
sudo tar --transform 's/.*\///g' -zcvf ~/geo-host-key.tar.gz /etc/ssh/ssh_host_*_key*
2019-07-31 22:56:46 +05:30
2021-10-27 15:23:28 +05:30
# Run this on each node on your secondary site:
scp < user_with_sudo > @< primary_site_fqdn > :geo-host-key.tar.gz .
2019-09-30 21:07:59 +05:30
tar zxvf ~/geo-host-key.tar.gz -C /etc/ssh
```
2019-07-31 22:56:46 +05:30
2021-10-27 15:23:28 +05:30
1. On **each node on your secondary** site, ensure the file permissions are correct:
2019-07-31 22:56:46 +05:30
2020-03-13 15:44:24 +05:30
```shell
2019-09-30 21:07:59 +05:30
chown root:root /etc/ssh/ssh_host_*_key*
2022-07-16 23:28:13 +05:30
chmod 0600 /etc/ssh/ssh_host_*_key
2019-09-30 21:07:59 +05:30
```
2019-07-31 22:56:46 +05:30
2021-10-27 15:23:28 +05:30
1. To verify key fingerprint matches, execute the following command on both primary and secondary nodes on each site:
2019-07-31 22:56:46 +05:30
2020-03-13 15:44:24 +05:30
```shell
2019-09-30 21:07:59 +05:30
for file in /etc/ssh/ssh_host_*_key; do ssh-keygen -lf $file; done
```
2019-07-31 22:56:46 +05:30
2019-09-30 21:07:59 +05:30
You should get an output similar to this one and they should be identical on both nodes:
2019-07-31 22:56:46 +05:30
2020-03-13 15:44:24 +05:30
```shell
2019-09-30 21:07:59 +05:30
1024 SHA256:FEZX2jQa2bcsd/fn/uxBzxhKdx4Imc4raXrHwsbtP0M root@serverhostname (DSA)
256 SHA256:uw98R35Uf+fYEQ/UnJD9Br4NXUFPv7JAUln5uHlgSeY root@serverhostname (ECDSA)
256 SHA256:sqOUWcraZQKd89y/QQv/iynPTOGQxcOTIXU/LsoPmnM root@serverhostname (ED25519)
2048 SHA256:qwa+rgir2Oy86QI+PZi/QVR+MSmrdrpsuH7YyKknC+s root@serverhostname (RSA)
```
2019-07-31 22:56:46 +05:30
1. Verify that you have the correct public keys for the existing private keys:
2020-03-13 15:44:24 +05:30
```shell
2019-09-30 21:07:59 +05:30
# This will print the fingerprint for private keys:
for file in /etc/ssh/ssh_host_*_key; do ssh-keygen -lf $file; done
2019-07-31 22:56:46 +05:30
2019-09-30 21:07:59 +05:30
# This will print the fingerprint for public keys:
for file in /etc/ssh/ssh_host_*_key.pub; do ssh-keygen -lf $file; done
```
2019-07-31 22:56:46 +05:30
2021-02-22 17:27:13 +05:30
NOTE:
2019-09-30 21:07:59 +05:30
The output for private keys and public keys command should generate the same fingerprint.
2019-07-31 22:56:46 +05:30
2021-10-27 15:23:28 +05:30
1. Restart `sshd` on **each node on your secondary** site:
2019-07-31 22:56:46 +05:30
2020-03-13 15:44:24 +05:30
```shell
2019-09-30 21:07:59 +05:30
# Debian or Ubuntu installations
sudo service ssh reload
2019-07-31 22:56:46 +05:30
2019-09-30 21:07:59 +05:30
# CentOS installations
sudo service sshd reload
```
2019-07-31 22:56:46 +05:30
2021-03-08 18:12:59 +05:30
1. Verify SSH is still functional.
SSH into your GitLab **secondary** server in a new terminal. If you are unable to connect,
verify the permissions are correct according to the previous steps.
2021-10-27 15:23:28 +05:30
### Step 3. Add the **secondary** site
2019-07-31 22:56:46 +05:30
2021-10-27 15:23:28 +05:30
1. SSH into **each Rails and Sidekiq node on your secondary** site and login as root:
2019-12-21 20:55:43 +05:30
2020-03-13 15:44:24 +05:30
```shell
2019-12-21 20:55:43 +05:30
sudo -i
```
2021-10-27 15:23:28 +05:30
1. Edit `/etc/gitlab/gitlab.rb` and add a **unique** name for your site. You need this in the next steps:
2019-12-21 20:55:43 +05:30
```ruby
2021-10-27 15:23:28 +05:30
##
## The unique identifier for the Geo site. See
## https://docs.gitlab.com/ee/user/admin_area/geo_nodes.html#common-settings
##
gitlab_rails['geo_node_name'] = '< site_name_here > '
2019-12-21 20:55:43 +05:30
```
2021-10-27 15:23:28 +05:30
1. Reconfigure **each Rails and Sidekiq node on your secondary** site for the change to take effect:
2019-12-21 20:55:43 +05:30
2020-03-13 15:44:24 +05:30
```shell
2019-12-21 20:55:43 +05:30
gitlab-ctl reconfigure
```
2022-03-02 08:16:31 +05:30
1. Navigate to the Primary Node GitLab Instance:
1. On the top bar, select **Menu > Admin** .
1. On the left sidebar, select **Geo > Sites** .
1. Select **Add site** .
2021-10-27 15:23:28 +05:30
![Add secondary site ](img/adding_a_secondary_v13_3.png )
2022-03-02 08:16:31 +05:30
1. Fill in **Name** with the `gitlab_rails['geo_node_name']` in
2019-12-26 22:10:19 +05:30
`/etc/gitlab/gitlab.rb` . These values must always match *exactly* , character
for character.
2022-03-02 08:16:31 +05:30
1. Fill in **URL** with the `external_url` in `/etc/gitlab/gitlab.rb` . These
2019-12-26 22:10:19 +05:30
values must always match, but it doesn't matter if one ends with a `/` and
the other doesn't.
2022-03-02 08:16:31 +05:30
1. (Optional) Choose which groups or storage shards should be replicated by the
2021-10-27 15:23:28 +05:30
**secondary** site. Leave blank to replicate all. Read more in
2019-07-31 22:56:46 +05:30
[selective synchronization ](#selective-synchronization ).
2022-03-02 08:16:31 +05:30
1. Select **Save changes** to add the **secondary** site.
2021-10-27 15:23:28 +05:30
1. SSH into **each Rails, and Sidekiq node on your secondary** site and restart the services:
2019-07-31 22:56:46 +05:30
2020-03-13 15:44:24 +05:30
```shell
2019-09-30 21:07:59 +05:30
gitlab-ctl restart
```
2019-07-31 22:56:46 +05:30
2019-09-30 21:07:59 +05:30
Check if there are any common issue with your Geo setup by running:
2019-07-31 22:56:46 +05:30
2020-03-13 15:44:24 +05:30
```shell
2019-09-30 21:07:59 +05:30
gitlab-rake gitlab:geo:check
```
2019-07-31 22:56:46 +05:30
2021-10-27 15:23:28 +05:30
1. SSH into a **Rails or Sidekiq server on your primary** site and login as root to verify the
**secondary** site is reachable or there are any common issue with your Geo setup:
2019-07-31 22:56:46 +05:30
2020-03-13 15:44:24 +05:30
```shell
2019-09-30 21:07:59 +05:30
gitlab-rake gitlab:geo:check
```
2019-07-31 22:56:46 +05:30
2021-10-27 15:23:28 +05:30
Once added to the Geo administration page and restarted, the **secondary** site automatically starts
replicating missing data from the **primary** site in a process known as **backfill** .
Meanwhile, the **primary** site starts to notify each **secondary** site of any changes, so
that the **secondary** site can act on those notifications immediately.
2019-07-31 22:56:46 +05:30
2021-10-27 15:23:28 +05:30
Be sure the _secondary_ site is running and accessible. You can sign in to the
_secondary_ site with the same credentials as were used with the _primary_ site.
2019-07-31 22:56:46 +05:30
2022-07-16 23:28:13 +05:30
### Step 4. (Optional) Using custom certificates
2019-07-31 22:56:46 +05:30
2022-07-16 23:28:13 +05:30
You can safely skip this step if:
2019-07-31 22:56:46 +05:30
2022-07-16 23:28:13 +05:30
- Your **primary** site uses a public CA-issued HTTPS certificate.
- Your **primary** site only connects to external services with CA-issued (not self-signed) HTTPS certificates.
#### Custom or self-signed certificate for inbound connections
If your GitLab Geo **primary** site uses a custom or [self-signed certificate to secure inbound HTTPS connections ](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates ), this certificate can either be single-domain certificate or multi-domain.
Install the correct certificate based on your certificate type:
- **Multi-domain certificate** that includes both primary and secondary site domains: Install the certificate at `/etc/gitlab/ssl` on all **Rails, Sidekiq, and Gitaly** nodes in the **secondary** site.
- **Single-domain certificate** where the certificates are specific to each Geo site domain: Generate a valid certificate for your **secondary** site's domain and install it at `/etc/gitlab/ssl` per [these instructions ](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates ) on all **Rails, Sidekiq, and Gitaly** nodes in the **secondary** site.
#### Connecting to external services that use customer certificates
A copy of the self-signed certificate for the external service needs to be added to the trust store on all the **primary** site's nodes that require access to the service.
For the **secondary** site to be able to access the same external services, these certificates *must* be added to the **secondary** site's trust store.
If your **primary** site is using a [custom or self-signed certificate for inbound HTTPS connections ](#custom-or-self-signed-certificate-for-inbound-connections ), the **primary** site's certificate needs to be added to the **secondary** site's trust store:
1. SSH into each **Rails, Sidekiq, and Gitaly node on your secondary** site and login as root:
```shell
sudo -i
```
1. Copy the trusted certs from the **primary** site:
If you can access one of the nodes on your **primary** site serving SSH traffic using the root user:
```shell
scp root@< primary_site_node_fqdn > :/etc/gitlab/trusted-certs/* /etc/gitlab/trusted-certs
```
If you only have access through a user with sudo privileges:
```shell
# Run this from the node on your primary site:
sudo tar --transform 's/.*\///g' -zcvf ~/geo-trusted-certs.tar.gz /etc/gitlab/trusted-certs/*
# Run this on each node on your secondary site:
scp < user_with_sudo > @< primary_site_node_fqdn > :geo-trusted-certs.tar.gz .
tar zxvf ~/geo-trusted-certs.tar.gz -C /etc/gitlab/trusted-certs
```
1. Reconfigure each updated **Rails, Sidekiq, and Gitaly node in your secondary** site:
```shell
sudo gitlab-ctl reconfigure
```
2019-07-31 22:56:46 +05:30
2021-01-29 00:20:46 +05:30
### Step 5. Enable Git access over HTTP/HTTPS
2019-07-31 22:56:46 +05:30
Geo synchronizes repositories over HTTP/HTTPS, and therefore requires this clone
2021-10-27 15:23:28 +05:30
method to be enabled. This is enabled by default, but if converting an existing site to Geo it should be checked:
2021-01-29 00:20:46 +05:30
2021-10-27 15:23:28 +05:30
On the **primary** site:
2021-09-04 01:27:46 +05:30
2021-11-11 11:23:49 +05:30
1. On the top bar, select **Menu > Admin** .
2021-09-04 01:27:46 +05:30
1. On the left sidebar, select **Settings > General** .
1. Expand **Visibility and access controls** .
2021-01-29 00:20:46 +05:30
1. Ensure "Enabled Git access protocols" is set to either "Both SSH and HTTP(S)" or "Only HTTP(S)".
2019-07-31 22:56:46 +05:30
2021-10-27 15:23:28 +05:30
### Step 6. Verify proper functioning of the **secondary** site
2019-07-31 22:56:46 +05:30
2021-10-27 15:23:28 +05:30
You can sign in to the **secondary** site with the same credentials you used with
the **primary** site. After you sign in:
2019-07-31 22:56:46 +05:30
2021-11-11 11:23:49 +05:30
1. On the top bar, select **Menu > Admin** .
2021-10-27 15:23:28 +05:30
1. On the left sidebar, select **Geo > Sites** .
1. Verify that it's correctly identified as a **secondary** Geo site, and that
2021-09-04 01:27:46 +05:30
Geo is enabled.
2019-07-31 22:56:46 +05:30
2021-06-08 01:23:25 +05:30
The initial replication, or 'backfill', is probably still in progress. You
2021-10-27 15:23:28 +05:30
can monitor the synchronization process on each Geo site from the **primary**
site's **Geo Sites** dashboard in your browser.
2019-07-31 22:56:46 +05:30
2021-10-27 15:23:28 +05:30
![Geo dashboard ](img/geo_dashboard_v14_0.png )
2019-07-31 22:56:46 +05:30
If your installation isn't working properly, check the
2019-10-12 21:52:04 +05:30
[troubleshooting document ](troubleshooting.md ).
2019-07-31 22:56:46 +05:30
The two most obvious issues that can become apparent in the dashboard are:
1. Database replication not working well.
1. Instance to instance notification not working. In that case, it can be
something of the following:
2019-09-30 21:07:59 +05:30
- You are using a custom certificate or custom CA (see the [troubleshooting document ](troubleshooting.md )).
- The instance is firewalled (check your firewall rules).
2019-07-31 22:56:46 +05:30
2021-10-27 15:23:28 +05:30
Disabling a **secondary** site stops the synchronization process.
2019-07-31 22:56:46 +05:30
2021-10-27 15:23:28 +05:30
If `git_data_dirs` is customized on the **primary** site for multiple
repository shards you must duplicate the same configuration on each **secondary** site.
2019-07-31 22:56:46 +05:30
2021-06-08 01:23:25 +05:30
Point your users to the [Using a Geo Site guide ](usage.md ).
2019-07-31 22:56:46 +05:30
Currently, this is what is synced:
- Git repositories.
- Wikis.
- LFS objects.
- Issues, merge requests, snippets, and comment attachments.
- Users, groups, and project avatars.
## Selective synchronization
2021-03-11 19:13:27 +05:30
Geo supports selective synchronization, which allows administrators to choose
2021-10-27 15:23:28 +05:30
which projects should be synchronized by **secondary** sites.
2019-07-31 22:56:46 +05:30
A subset of projects can be chosen, either by group or by storage shard. The
former is ideal for replicating data belonging to a subset of users, while the
latter is more suited to progressively rolling out Geo to a large GitLab
instance.
It is important to note that selective synchronization:
2021-10-27 15:23:28 +05:30
1. Does not restrict permissions from **secondary** sites.
1. Does not hide project metadata from **secondary** sites.
2019-07-31 22:56:46 +05:30
- Since Geo currently relies on PostgreSQL replication, all project metadata
2021-10-27 15:23:28 +05:30
gets replicated to **secondary** sites, but repositories that have not been
2021-06-08 01:23:25 +05:30
selected are empty.
2019-07-31 22:56:46 +05:30
1. Does not reduce the number of events generated for the Geo event log.
2021-10-27 15:23:28 +05:30
- The **primary** site generates events as long as any **secondary** sites are present.
Selective synchronization restrictions are implemented on the **secondary** sites,
not the **primary** site.
2019-07-31 22:56:46 +05:30
2020-05-24 23:13:21 +05:30
### Git operations on unreplicated repositories
2020-04-22 19:07:51 +05:30
2020-05-24 23:13:21 +05:30
> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2562) in GitLab 12.10 for HTTP(S) and in GitLab 13.0 for SSH.
2020-04-22 19:07:51 +05:30
2020-05-24 23:13:21 +05:30
Git clone, pull, and push operations over HTTP(S) and SSH are supported for repositories that
2021-10-27 15:23:28 +05:30
exist on the **primary** site but not on **secondary** sites. This situation can occur
2020-04-22 19:07:51 +05:30
when:
- Selective synchronization does not include the project attached to the repository.
- The repository is actively being replicated but has not completed yet.
2019-07-31 22:56:46 +05:30
## Upgrading Geo
2022-07-23 23:45:48 +05:30
See the [upgrading the Geo sites document ](upgrading_the_geo_sites.md ).
2019-07-31 22:56:46 +05:30
## Troubleshooting
See the [troubleshooting document ](troubleshooting.md ).