debian-mirror-gitlab/doc/administration/geo/replication/multiple_servers.md

360 lines
13 KiB
Markdown
Raw Normal View History

2020-06-23 00:09:42 +05:30
---
stage: Enablement
group: Geo
2021-02-22 17:27:13 +05:30
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
2020-06-23 00:09:42 +05:30
type: howto
---
2021-03-11 19:13:27 +05:30
# Geo for multiple nodes **(PREMIUM SELF)**
2020-05-24 23:13:21 +05:30
This document describes a minimal reference architecture for running Geo
2020-07-28 23:09:34 +05:30
in a multi-node configuration. If your multi-node setup differs from the one
2020-05-24 23:13:21 +05:30
described, it is possible to adapt these instructions to your needs.
## Architecture overview
2021-01-03 14:25:43 +05:30
![Geo multi-node diagram](img/geo-ha-diagram.png)
2020-05-24 23:13:21 +05:30
_[diagram source - GitLab employees only](https://docs.google.com/drawings/d/1z0VlizKiLNXVVVaERFwgsIOuEgjcUqDTWPdQYsE7Z4c/edit)_
2021-10-27 15:23:28 +05:30
The topology above assumes the **primary** and **secondary** Geo sites
2020-05-24 23:13:21 +05:30
are located in two separate locations, on their own virtual network
2021-09-04 01:27:46 +05:30
with private IP addresses. The network is configured such that all machines in
2020-05-24 23:13:21 +05:30
one geographic location can communicate with each other using their private IP addresses.
The IP addresses given are examples and may be different depending on the
network topology of your deployment.
2021-10-27 15:23:28 +05:30
The only external way to access the two Geo sites is by HTTPS at
2020-05-24 23:13:21 +05:30
`gitlab.us.example.com` and `gitlab.eu.example.com` in the example above.
2021-02-22 17:27:13 +05:30
NOTE:
2021-10-27 15:23:28 +05:30
The **primary** and **secondary** Geo sites must be able to communicate to each other over HTTPS.
2020-05-24 23:13:21 +05:30
2020-07-28 23:09:34 +05:30
## Redis and PostgreSQL for multiple nodes
2020-05-24 23:13:21 +05:30
Because of the additional complexity involved in setting up this configuration
2020-07-28 23:09:34 +05:30
for PostgreSQL and Redis, it is not covered by this Geo multi-node documentation.
2020-05-24 23:13:21 +05:30
2021-09-04 01:27:46 +05:30
For more information on setting up a multi-node PostgreSQL cluster and Redis cluster using the Omnibus GitLab package, see:
2021-10-27 15:23:28 +05:30
- [Geo multi-node database replication](../setup/database.md#multi-node-database-replication)
2021-09-04 01:27:46 +05:30
- [Redis multi-node documentation](../../redis/replication_and_failover.md)
2020-05-24 23:13:21 +05:30
2021-02-22 17:27:13 +05:30
NOTE:
2020-05-24 23:13:21 +05:30
It is possible to use cloud hosted services for PostgreSQL and Redis, but this is beyond the scope of this document.
2021-10-27 15:23:28 +05:30
## Prerequisites: Two independently working GitLab multi-node sites
2020-05-24 23:13:21 +05:30
2021-10-27 15:23:28 +05:30
One GitLab site serves as the Geo **primary** site. Use the
[GitLab reference architectures documentation](../../reference_architectures/index.md)
to set this up. You can use different reference architecture sizes for each Geo site. If
2020-05-24 23:13:21 +05:30
you already have a working GitLab instance that is in-use, it can be used as a
2021-10-27 15:23:28 +05:30
**primary** site.
2020-05-24 23:13:21 +05:30
2021-10-27 15:23:28 +05:30
The second GitLab site serves as the Geo **secondary** site. Again, use the
[GitLab reference architectures documentation](../../reference_architectures/index.md) to set this up.
2021-09-04 01:27:46 +05:30
It's a good idea to log in and test it. However, be aware that its data is
2021-10-27 15:23:28 +05:30
wiped out as part of the process of replicating from the **primary** site.
2020-05-24 23:13:21 +05:30
2021-10-27 15:23:28 +05:30
## Configure a GitLab site to be the Geo **primary** site
2020-05-24 23:13:21 +05:30
2021-10-27 15:23:28 +05:30
The following steps enable a GitLab site to serve as the Geo **primary** site.
2020-05-24 23:13:21 +05:30
2021-10-27 15:23:28 +05:30
### Step 1: Configure the **primary** frontend nodes
2020-05-24 23:13:21 +05:30
1. Edit `/etc/gitlab/gitlab.rb` and add the following:
```ruby
##
2021-10-27 15:23:28 +05:30
## The unique identifier for the Geo site. See
## https://docs.gitlab.com/ee/user/admin_area/geo_nodes.html#common-settings
2020-05-24 23:13:21 +05:30
##
2021-10-27 15:23:28 +05:30
gitlab_rails['geo_node_name'] = '<site_name_here>'
2020-05-24 23:13:21 +05:30
##
## Disable automatic migrations
##
gitlab_rails['auto_migrate'] = false
```
After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect.
2021-02-22 17:27:13 +05:30
NOTE:
2020-07-28 23:09:34 +05:30
PostgreSQL and Redis should have already been disabled on the
2021-10-27 15:23:28 +05:30
application nodes during normal GitLab multi-node setup. Connections
from the application nodes to services on the backend nodes should
2021-09-04 01:27:46 +05:30
have also been configured. See multi-node configuration documentation for
2020-06-23 00:09:42 +05:30
[PostgreSQL](../../postgresql/replication_and_failover.md#configuring-the-application-nodes)
2020-07-28 23:09:34 +05:30
and [Redis](../../redis/replication_and_failover.md#example-configuration-for-the-gitlab-application).
2020-05-24 23:13:21 +05:30
2021-10-27 15:23:28 +05:30
## Configure the other GitLab site to be a Geo **secondary** site
2020-05-24 23:13:21 +05:30
2021-10-27 15:23:28 +05:30
A **secondary** site is similar to any other GitLab multi-node site, with three
2020-05-24 23:13:21 +05:30
major differences:
2021-10-27 15:23:28 +05:30
- The main PostgreSQL database is a read-only replica of the Geo **primary** site's
2020-05-24 23:13:21 +05:30
PostgreSQL database.
2021-10-27 15:23:28 +05:30
- There is an additional PostgreSQL database for each Geo **secondary** site,
called the "Geo tracking database", which tracks the replication and verification
state of various resources.
- There is an additional GitLab service [`geo-logcursor`](../index.md#geo-log-cursor)
2020-05-24 23:13:21 +05:30
2021-09-04 01:27:46 +05:30
Therefore, we set up the multi-node components one by one and include deviations
from the normal multi-node setup. However, we highly recommend configuring a
2021-10-27 15:23:28 +05:30
brand-new GitLab site first, as if it were not part of a Geo setup. This allows
verifying that it is a working GitLab site. And only then should it be modified
for use as a Geo **secondary** site. This helps to separate Geo setup problems from
unrelated multi-node configuration problems.
2020-05-24 23:13:21 +05:30
2021-10-27 15:23:28 +05:30
### Step 1: Configure the Redis and Gitaly services on the Geo **secondary** site
2020-05-24 23:13:21 +05:30
2020-07-28 23:09:34 +05:30
Configure the following services, again using the non-Geo multi-node
2020-05-24 23:13:21 +05:30
documentation:
2020-07-28 23:09:34 +05:30
- [Configuring Redis for GitLab](../../redis/replication_and_failover.md#example-configuration-for-the-gitlab-application) for multiple nodes.
2021-06-08 01:23:25 +05:30
- [Gitaly](../../gitaly/index.md), which stores data that is
2021-10-27 15:23:28 +05:30
synchronized from the Geo **primary** site.
2020-05-24 23:13:21 +05:30
2021-02-22 17:27:13 +05:30
NOTE:
2020-10-24 23:57:45 +05:30
[NFS](../../nfs.md) can be used in place of Gitaly but is not
2020-05-24 23:13:21 +05:30
recommended.
2021-10-27 15:23:28 +05:30
### Step 2: Configure Postgres streaming replication
2020-05-24 23:13:21 +05:30
2021-10-27 15:23:28 +05:30
Follow the [Geo database replication instructions](../setup/database.md).
2020-05-24 23:13:21 +05:30
If using an external PostgreSQL instance, refer also to
2020-11-24 15:15:51 +05:30
[Geo with external PostgreSQL instances](../setup/external_database.md).
2020-05-24 23:13:21 +05:30
2021-10-27 15:23:28 +05:30
### Step 3: Configure the Geo tracking database on the Geo **secondary** site
2020-05-24 23:13:21 +05:30
2021-10-27 15:23:28 +05:30
If you want to run the Geo tracking database in a multi-node PostgreSQL cluster,
then follow [Configuring Patroni cluster for the tracking PostgreSQL database](../setup/database.md#configuring-patroni-cluster-for-the-tracking-postgresql-database).
2020-05-24 23:13:21 +05:30
2021-10-27 15:23:28 +05:30
If you want to run the Geo tracking database on a single node, then follow
the instructions below.
2020-05-24 23:13:21 +05:30
1. Generate an MD5 hash of the desired password for the database user that the
2021-06-08 01:23:25 +05:30
GitLab application uses to access the tracking database:
2020-05-24 23:13:21 +05:30
2021-10-27 15:23:28 +05:30
The username (`gitlab_geo` by default) is incorporated into the
2020-05-24 23:13:21 +05:30
hash.
```shell
gitlab-ctl pg-password-md5 gitlab_geo
# Enter password: <your_password_here>
# Confirm password: <your_password_here>
# fca0b89a972d69f00eb3ec98a5838484
```
Use this hash to fill in `<tracking_database_password_md5_hash>` in the next
step.
2021-10-27 15:23:28 +05:30
1. On the machine where the Geo tracking database is intended to run, add the
following to `/etc/gitlab/gitlab.rb`:
2020-05-24 23:13:21 +05:30
```ruby
##
## Enable the Geo secondary tracking database
##
geo_postgresql['enable'] = true
geo_postgresql['listen_address'] = '<ip_address_of_this_host>'
geo_postgresql['sql_user_password'] = '<tracking_database_password_md5_hash>'
##
2020-10-24 23:57:45 +05:30
## Configure PostgreSQL connection to the replica database
2020-05-24 23:13:21 +05:30
##
geo_postgresql['md5_auth_cidr_addresses'] = ['<replica_database_ip>/32']
gitlab_rails['db_host'] = '<replica_database_ip>'
2021-10-27 15:23:28 +05:30
# Prevent reconfigure from attempting to run migrations on the replica database
2020-05-24 23:13:21 +05:30
gitlab_rails['auto_migrate'] = false
```
After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect.
If using an external PostgreSQL instance, refer also to
2020-11-24 15:15:51 +05:30
[Geo with external PostgreSQL instances](../setup/external_database.md).
2020-05-24 23:13:21 +05:30
2021-10-27 15:23:28 +05:30
### Step 4: Configure the frontend application nodes on the Geo **secondary** site
2020-05-24 23:13:21 +05:30
2021-10-27 15:23:28 +05:30
In the minimal [architecture diagram](#architecture-overview) above, there are two
machines running the GitLab application services. These services are enabled
selectively in the configuration.
2020-05-24 23:13:21 +05:30
2021-10-27 15:23:28 +05:30
Configure the GitLab Rails application nodes following the relevant steps
2020-10-24 23:57:45 +05:30
outlined in the [reference architectures](../../reference_architectures/index.md),
then make the following modifications:
2020-05-24 23:13:21 +05:30
2021-10-27 15:23:28 +05:30
1. Edit `/etc/gitlab/gitlab.rb` on each application node in the Geo **secondary**
site, and add the following:
2020-05-24 23:13:21 +05:30
```ruby
##
2021-10-27 15:23:28 +05:30
## Enable GitLab application services. The application_role enables many services.
## Alternatively, you can choose to enable or disable specific services on
## different nodes to aid in horizontal scaling and separation of concerns.
2020-05-24 23:13:21 +05:30
##
2021-10-27 15:23:28 +05:30
roles ['application_role']
## `application_role` already enables this. You only need this line if
## you selectively enable individual services that depend on Rails, like
## `puma`, `sidekiq`, `geo-logcursor`, etc.
gitlab_rails['enable'] = true
2020-05-24 23:13:21 +05:30
##
2021-10-27 15:23:28 +05:30
## Enable Geo Log Cursor service
2020-05-24 23:13:21 +05:30
##
2021-10-27 15:23:28 +05:30
geo_logcursor['enable'] = true
##
## The unique identifier for the Geo site. See
## https://docs.gitlab.com/ee/user/admin_area/geo_nodes.html#common-settings
##
gitlab_rails['geo_node_name'] = '<site_name_here>'
2020-05-24 23:13:21 +05:30
##
## Disable automatic migrations
##
gitlab_rails['auto_migrate'] = false
##
2021-10-27 15:23:28 +05:30
## Configure the connection to the tracking database
2020-05-24 23:13:21 +05:30
##
2021-10-27 15:23:28 +05:30
geo_secondary['enable'] = true
2020-05-24 23:13:21 +05:30
geo_secondary['db_host'] = '<geo_tracking_db_host>'
geo_secondary['db_password'] = '<geo_tracking_db_password>'
##
## Configure connection to the streaming replica database, if you haven't
## already
##
gitlab_rails['db_host'] = '<replica_database_host>'
gitlab_rails['db_password'] = '<replica_database_password>'
##
## Configure connection to Redis, if you haven't already
##
gitlab_rails['redis_host'] = '<redis_host>'
gitlab_rails['redis_password'] = '<redis_password>'
##
## If you are using custom users not managed by Omnibus, you need to specify
2021-10-27 15:23:28 +05:30
## UIDs and GIDs like below, and ensure they match between nodes in a
2020-05-24 23:13:21 +05:30
## cluster to avoid permissions issues
##
user['uid'] = 9000
user['gid'] = 9000
web_server['uid'] = 9001
web_server['gid'] = 9001
registry['uid'] = 9002
registry['gid'] = 9002
```
2021-02-22 17:27:13 +05:30
NOTE:
2021-09-04 01:27:46 +05:30
If you had set up PostgreSQL cluster using the omnibus package and had set
`postgresql['sql_user_password'] = 'md5 digest of secret'`, keep in
2020-05-24 23:13:21 +05:30
mind that `gitlab_rails['db_password']` and `geo_secondary['db_password']`
2021-09-04 01:27:46 +05:30
contains the plaintext passwords. This is used to let the Rails
2021-10-27 15:23:28 +05:30
nodes connect to the databases.
2020-05-24 23:13:21 +05:30
2021-02-22 17:27:13 +05:30
NOTE:
2021-10-27 15:23:28 +05:30
Make sure that current node's IP is listed in
`postgresql['md5_auth_cidr_addresses']` setting of the read-replica database to
allow Rails on this node to connect to Postgres.
2020-05-24 23:13:21 +05:30
After making these changes [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect.
2021-10-27 15:23:28 +05:30
In the [architecture overview](#architecture-overview) topology, the following GitLab
services are enabled on the "frontend" nodes:
2020-05-24 23:13:21 +05:30
- `geo-logcursor`
- `gitlab-pages`
- `gitlab-workhorse`
- `logrotate`
- `nginx`
- `registry`
- `remote-syslog`
- `sidekiq`
- `puma`
2021-10-27 15:23:28 +05:30
Verify these services exist by running `sudo gitlab-ctl status` on the frontend
application nodes.
2020-05-24 23:13:21 +05:30
2021-10-27 15:23:28 +05:30
### Step 5: Set up the LoadBalancer for the Geo **secondary** site
2020-05-24 23:13:21 +05:30
2021-10-27 15:23:28 +05:30
The minimal [architecture diagram](#architecture-overview) above shows a load
balancer at each geographic location to route traffic to the application nodes.
2020-05-24 23:13:21 +05:30
2020-10-24 23:57:45 +05:30
See [Load Balancer for GitLab with multiple nodes](../../load_balancer.md) for
2020-05-24 23:13:21 +05:30
more information.
2021-10-27 15:23:28 +05:30
### Step 6: Configure the backend application nodes on the Geo **secondary** site
2020-05-24 23:13:21 +05:30
2021-10-27 15:23:28 +05:30
The minimal [architecture diagram](#architecture-overview) above shows all application services
2020-07-28 23:09:34 +05:30
running together on the same machines. However, for multiple nodes we
2020-05-24 23:13:21 +05:30
[strongly recommend running all services separately](../../reference_architectures/index.md).
2021-10-27 15:23:28 +05:30
For example, a Sidekiq node could be configured similarly to the frontend
application nodes above, with some changes to run only the `sidekiq` service:
2020-05-24 23:13:21 +05:30
2021-10-27 15:23:28 +05:30
1. Edit `/etc/gitlab/gitlab.rb` on each Sidekiq node in the Geo **secondary**
site, and add the following:
2020-05-24 23:13:21 +05:30
```ruby
##
## Enable the Sidekiq service
##
sidekiq['enable'] = true
2021-10-27 15:23:28 +05:30
gitlab_rails['enable'] = true
2020-05-24 23:13:21 +05:30
##
2021-10-27 15:23:28 +05:30
## The unique identifier for the Geo site. See
## https://docs.gitlab.com/ee/user/admin_area/geo_nodes.html#common-settings
2020-05-24 23:13:21 +05:30
##
2021-10-27 15:23:28 +05:30
gitlab_rails['geo_node_name'] = '<site_name_here>'
2020-05-24 23:13:21 +05:30
##
## Disable automatic migrations
##
gitlab_rails['auto_migrate'] = false
##
2021-10-27 15:23:28 +05:30
## Configure the connection to the tracking database
2020-05-24 23:13:21 +05:30
##
2021-10-27 15:23:28 +05:30
geo_secondary['enable'] = true
2020-05-24 23:13:21 +05:30
geo_secondary['db_host'] = '<geo_tracking_db_host>'
geo_secondary['db_password'] = '<geo_tracking_db_password>'
##
## Configure connection to the streaming replica database, if you haven't
## already
##
gitlab_rails['db_host'] = '<replica_database_host>'
gitlab_rails['db_password'] = '<replica_database_password>'
##
## Configure connection to Redis, if you haven't already
##
gitlab_rails['redis_host'] = '<redis_host>'
gitlab_rails['redis_password'] = '<redis_password>'
##
## If you are using custom users not managed by Omnibus, you need to specify
2021-10-27 15:23:28 +05:30
## UIDs and GIDs like below, and ensure they match between nodes in a
2020-05-24 23:13:21 +05:30
## cluster to avoid permissions issues
##
user['uid'] = 9000
user['gid'] = 9000
web_server['uid'] = 9001
web_server['gid'] = 9001
registry['uid'] = 9002
registry['gid'] = 9002
```
2021-10-27 15:23:28 +05:30
You can similarly configure a node to run only the `geo-logcursor` service
2020-05-24 23:13:21 +05:30
with `geo_logcursor['enable'] = true` and disabling Sidekiq with
`sidekiq['enable'] = false`.
2021-10-27 15:23:28 +05:30
These nodes do not need to be attached to the load balancer.