debian-mirror-gitlab/doc/administration/high_availability/gitlab.md

211 lines
9.3 KiB
Markdown
Raw Normal View History

2019-09-30 21:07:59 +05:30
---
type: reference
---
# Configuring GitLab for Scaling and High Availability
2016-06-02 11:05:42 +05:30
2019-12-04 20:38:33 +05:30
NOTE: **Note:** There is some additional configuration near the bottom for
additional GitLab application servers. It's important to read and understand
these additional steps before proceeding with GitLab installation.
2016-06-02 11:05:42 +05:30
2020-04-08 14:13:33 +05:30
NOTE: **Note:** [Cloud Object Storage service](object_storage.md) with [Gitaly](gitaly.md)
is recommended over [NFS](nfs.md) wherever possible for improved performance.
2016-06-02 11:05:42 +05:30
1. If necessary, install the NFS client utility packages using the following
commands:
2020-03-13 15:44:24 +05:30
```shell
2019-09-30 21:07:59 +05:30
# Ubuntu/Debian
apt-get install nfs-common
2016-06-02 11:05:42 +05:30
2019-09-30 21:07:59 +05:30
# CentOS/Red Hat
yum install nfs-utils nfs-utils-lib
```
2016-06-02 11:05:42 +05:30
1. Specify the necessary NFS shares. Mounts are specified in
`/etc/fstab`. The exact contents of `/etc/fstab` will depend on how you chose
to configure your NFS server. See [NFS documentation](nfs.md) for the various
options. Here is an example snippet to add to `/etc/fstab`:
2020-03-13 15:44:24 +05:30
```plaintext
2019-09-30 21:07:59 +05:30
10.1.0.1:/var/opt/gitlab/.ssh /var/opt/gitlab/.ssh nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2
10.1.0.1:/var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2
10.1.0.1:/var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2
10.1.0.1:/var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/gitlab-ci/builds nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2
10.1.0.1:/var/opt/gitlab/git-data /var/opt/gitlab/git-data nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2
```
2016-06-02 11:05:42 +05:30
1. Create the shared directories. These may be different depending on your NFS
mount locations.
2020-03-13 15:44:24 +05:30
```shell
2019-09-30 21:07:59 +05:30
mkdir -p /var/opt/gitlab/.ssh /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/git-data
```
2016-06-02 11:05:42 +05:30
1. Download/install GitLab Omnibus using **steps 1 and 2** from
2019-12-21 20:55:43 +05:30
[GitLab downloads](https://about.gitlab.com/install/). Do not complete other
2016-06-02 11:05:42 +05:30
steps on the download page.
1. Create/edit `/etc/gitlab/gitlab.rb` and use the following configuration.
Be sure to change the `external_url` to match your eventual GitLab front-end
URL. Depending your the NFS configuration, you may need to change some GitLab
data locations. See [NFS documentation](nfs.md) for `/etc/gitlab/gitlab.rb`
configuration values for various scenarios. The example below assumes you've
2018-11-08 19:23:39 +05:30
added NFS mounts in the default data locations. Additionally the UID and GIDs
given are just examples and you should configure with your preferred values.
2018-03-17 18:26:18 +05:30
2019-09-30 21:07:59 +05:30
```ruby
external_url 'https://gitlab.example.com'
# Prevent GitLab from starting if NFS data mounts are not available
high_availability['mountpoint'] = '/var/opt/gitlab/git-data'
# Disable components that will not be on the GitLab application server
roles ['application_role']
nginx['enable'] = true
# PostgreSQL connection details
gitlab_rails['db_adapter'] = 'postgresql'
gitlab_rails['db_encoding'] = 'unicode'
gitlab_rails['db_host'] = '10.1.0.5' # IP/hostname of database server
gitlab_rails['db_password'] = 'DB password'
# Redis connection details
gitlab_rails['redis_port'] = '6379'
gitlab_rails['redis_host'] = '10.1.0.6' # IP/hostname of Redis server
gitlab_rails['redis_password'] = 'Redis Password'
# Ensure UIDs and GIDs match between servers for permissions via NFS
user['uid'] = 9000
user['gid'] = 9000
web_server['uid'] = 9001
web_server['gid'] = 9001
registry['uid'] = 9002
registry['gid'] = 9002
```
2018-03-17 18:26:18 +05:30
2019-09-04 21:01:54 +05:30
1. [Enable monitoring](#enable-monitoring)
2019-12-04 20:38:33 +05:30
NOTE: **Note:** To maintain uniformity of links across HA clusters, the `external_url`
2019-09-30 21:07:59 +05:30
on the first application server as well as the additional application
servers should point to the external url that users will use to access GitLab.
In a typical HA setup, this will be the url of the load balancer which will
route traffic to all GitLab application servers in the HA cluster.
2019-12-04 20:38:33 +05:30
NOTE: **Note:** When you specify `https` in the `external_url`, as in the example
2019-09-30 21:07:59 +05:30
above, GitLab assumes you have SSL certificates in `/etc/gitlab/ssl/`. If
2019-12-21 20:55:43 +05:30
certificates are not present, NGINX will fail to start. See
[NGINX documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https)
2019-09-30 21:07:59 +05:30
for more information.
2019-12-04 20:38:33 +05:30
NOTE: **Note:** It is best to set the `uid` and `gid`s prior to the initial reconfigure
2019-09-30 21:07:59 +05:30
of GitLab. Omnibus will not recursively `chown` directories if set after the initial reconfigure.
2016-06-02 11:05:42 +05:30
2017-09-10 17:25:29 +05:30
## First GitLab application server
2016-06-02 11:05:42 +05:30
2019-12-21 20:55:43 +05:30
On the first application server, run:
2016-06-02 11:05:42 +05:30
2020-03-13 15:44:24 +05:30
```shell
2019-12-21 20:55:43 +05:30
sudo gitlab-ctl reconfigure
```
2016-06-02 11:05:42 +05:30
2019-12-21 20:55:43 +05:30
This should compile the configuration and initialize the database. Do
not run this on additional application servers until the next step.
2016-06-02 11:05:42 +05:30
2017-09-10 17:25:29 +05:30
## Extra configuration for additional GitLab application servers
2016-06-02 11:05:42 +05:30
2017-09-10 17:25:29 +05:30
Additional GitLab servers (servers configured **after** the first GitLab server)
need some extra configuration.
2016-06-02 11:05:42 +05:30
1. Configure shared secrets. These values can be obtained from the primary
2019-07-07 11:18:12 +05:30
GitLab server in `/etc/gitlab/gitlab-secrets.json`. Copy this file to the
secondary servers **prior to** running the first `reconfigure` in the steps
above.
2016-06-02 11:05:42 +05:30
2019-09-30 21:07:59 +05:30
```ruby
gitlab_shell['secret_token'] = 'fbfb19c355066a9afb030992231c4a363357f77345edd0f2e772359e5be59b02538e1fa6cae8f93f7d23355341cea2b93600dab6d6c3edcdced558fc6d739860'
gitlab_rails['otp_key_base'] = 'b719fe119132c7810908bba18315259ed12888d4f5ee5430c42a776d840a396799b0a5ef0a801348c8a357f07aa72bbd58e25a84b8f247a25c72f539c7a6c5fa'
gitlab_rails['secret_key_base'] = '6e657410d57c71b4fc3ed0d694e7842b1895a8b401d812c17fe61caf95b48a6d703cb53c112bc01ebd197a85da81b18e29682040e99b4f26594772a4a2c98c6d'
gitlab_rails['db_key_base'] = 'bf2e47b68d6cafaef1d767e628b619365becf27571e10f196f98dc85e7771042b9203199d39aff91fcb6837c8ed83f2a912b278da50999bb11a2fbc0fba52964'
```
2016-06-02 11:05:42 +05:30
2019-02-15 15:39:39 +05:30
1. Run `touch /etc/gitlab/skip-auto-reconfigure` to prevent database migrations
2016-06-02 11:05:42 +05:30
from running on upgrade. Only the primary GitLab application server should
handle migrations.
2019-09-30 21:07:59 +05:30
1. **Recommended** Configure host keys. Copy the contents (primary and public keys) of `/etc/ssh/` on
2018-11-18 11:00:15 +05:30
the primary application server to `/etc/ssh` on all secondary servers. This
prevents false man-in-the-middle-attack alerts when accessing servers in your
High Availability cluster behind a load balancer.
2018-11-08 19:23:39 +05:30
1. Run `sudo gitlab-ctl reconfigure` to compile the configuration.
2019-09-04 21:01:54 +05:30
## Enable Monitoring
> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0.
If you enable Monitoring, it must be enabled on **all** GitLab servers.
2019-09-30 21:07:59 +05:30
1. Make sure to collect [`CONSUL_SERVER_NODES`](database.md#consul-information), which are the IP addresses or DNS records of the Consul server nodes, for the next step. Note they are presented as `Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z`
2019-09-04 21:01:54 +05:30
1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration:
```ruby
# Enable service discovery for Prometheus
consul['enable'] = true
consul['monitoring_service_discovery'] = true
# Replace placeholders
# Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z
# with the addresses of the Consul server nodes
consul['configuration'] = {
retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z),
}
# Set the network addresses that the exporters will listen on
node_exporter['listen_address'] = '0.0.0.0:9100'
gitlab_workhorse['prometheus_listen_addr'] = '0.0.0.0:9229'
sidekiq['listen_address'] = "0.0.0.0"
unicorn['listen'] = '0.0.0.0'
2019-09-30 21:07:59 +05:30
# Add the monitoring node's IP address to the monitoring whitelist and allow it to
# scrape the NGINX metrics. Replace placeholder `monitoring.gitlab.example.com` with
# the address and/or subnets gathered from the monitoring node(s).
gitlab_rails['monitoring_whitelist'] = ['monitoring.gitlab.example.com', '127.0.0.0/8']
nginx['status']['options']['allow'] = ['monitoring.gitlab.example.com', '127.0.0.0/8']
2019-09-04 21:01:54 +05:30
```
1. Run `sudo gitlab-ctl reconfigure` to compile the configuration.
2019-12-04 20:38:33 +05:30
CAUTION: **Warning:**
After changing `unicorn['listen']` in `gitlab.rb`, and running `sudo gitlab-ctl reconfigure`,
2019-12-21 20:55:43 +05:30
it can take an extended period of time for Unicorn to complete reloading after receiving a `HUP`.
2019-12-04 20:38:33 +05:30
For more information, see the [issue](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/4401).
2019-09-04 21:01:54 +05:30
2016-06-02 11:05:42 +05:30
## Troubleshooting
- `mount: wrong fs type, bad option, bad superblock on`
You have not installed the necessary NFS client utilities. See step 1 above.
- `mount: mount point /var/opt/gitlab/... does not exist`
This particular directory does not exist on the NFS server. Ensure
the share is exported and exists on the NFS server and try to remount.
---
2019-07-07 11:18:12 +05:30
## Upgrading GitLab HA
GitLab HA installations can be upgraded with no downtime, but the
upgrade process must be carefully coordinated to avoid failures. See the
[Omnibus GitLab multi-node upgrade
document](https://docs.gitlab.com/omnibus/update/#multi-node--ha-deployment)
for more details.
2016-06-02 11:05:42 +05:30
Read more on high-availability configuration:
1. [Configure the database](database.md)
1. [Configure Redis](redis.md)
1. [Configure NFS](nfs.md)
1. [Configure the load balancers](load_balancer.md)