debian-mirror-gitlab/doc/administration/redis/replication_and_failover_external.md

370 lines
14 KiB
Markdown
Raw Normal View History

2020-07-28 23:09:34 +05:30
---
type: howto
stage: Enablement
group: Distribution
2021-02-22 17:27:13 +05:30
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
2020-07-28 23:09:34 +05:30
---
2021-03-11 19:13:27 +05:30
# Redis replication and failover providing your own instance **(FREE SELF)**
2020-07-28 23:09:34 +05:30
2021-04-29 21:17:54 +05:30
If you're hosting GitLab on a cloud provider, you can optionally use a managed
2020-07-28 23:09:34 +05:30
service for Redis. For example, AWS offers ElastiCache that runs Redis.
Alternatively, you may opt to manage your own Redis instance separate from the
Omnibus GitLab package.
## Requirements
The following are the requirements for providing your own Redis instance:
2020-11-24 15:15:51 +05:30
- Find the minimum Redis version that is required in the
[requirements page](../../install/requirements.md).
2020-07-28 23:09:34 +05:30
- Standalone Redis or Redis high availability with Sentinel are supported. Redis
Cluster is not supported.
2021-09-04 01:27:46 +05:30
- Managed Redis from cloud providers such as AWS ElastiCache works fine. If these
2020-07-28 23:09:34 +05:30
services support high availability, be sure it is **not** the Redis Cluster type.
Note the Redis node's IP address or hostname, port, and password (if required).
## Redis as a managed service in a cloud provider
1. Set up Redis according to the [requirements](#requirements).
1. Configure the GitLab application servers with the appropriate connection details
for your external Redis service in your `/etc/gitlab/gitlab.rb` file:
```ruby
redis['enable'] = false
gitlab_rails['redis_host'] = 'redis.example.com'
gitlab_rails['redis_port'] = 6379
# Required if Redis authentication is configured on the Redis node
gitlab_rails['redis_password'] = 'Redis Password'
```
1. Reconfigure for the changes to take effect:
```shell
sudo gitlab-ctl reconfigure
```
## Redis replication and failover with your own Redis servers
This is the documentation for configuring a scalable Redis setup when
you have installed Redis all by yourself and not using the bundled one that
comes with the Omnibus packages, although using the Omnibus GitLab packages is
2021-09-04 01:27:46 +05:30
highly recommend as we optimize them specifically for GitLab, and we take
2020-07-28 23:09:34 +05:30
care of upgrading Redis to the latest supported version.
Note also that you may elect to override all references to
`/home/git/gitlab/config/resque.yml` in accordance with the advanced Redis
settings outlined in
2021-09-04 01:27:46 +05:30
[Configuration Files Documentation](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/README.md).
2020-07-28 23:09:34 +05:30
We cannot stress enough the importance of reading the
[replication and failover](replication_and_failover.md) documentation of the
Omnibus Redis HA as it provides some invaluable information to the configuration
of Redis. Please proceed to read it before going forward with this guide.
Before proceeding on setting up the new Redis instances, here are some
requirements:
- All Redis servers in this guide must be configured to use a TCP connection
instead of a socket. To configure Redis to use TCP connections you need to
2021-01-03 14:25:43 +05:30
define both `bind` and `port` in the Redis configuration file. You can bind to all
2020-07-28 23:09:34 +05:30
interfaces (`0.0.0.0`) or specify the IP of the desired interface
2021-09-30 23:02:18 +05:30
(for example, one from an internal network).
2020-07-28 23:09:34 +05:30
- Since Redis 3.2, you must define a password to receive external connections
(`requirepass`).
2021-09-04 01:27:46 +05:30
- If you are using Redis with Sentinel, you also need to define the same
2020-07-28 23:09:34 +05:30
password for the replica password definition (`masterauth`) in the same instance.
In addition, read the prerequisites as described in the
[Omnibus Redis document](replication_and_failover.md#requirements) since they provide some
valuable information for the general setup.
### Step 1. Configuring the primary Redis instance
Assuming that the Redis primary instance IP is `10.0.0.1`:
1. [Install Redis](../../install/installation.md#7-redis).
1. Edit `/etc/redis/redis.conf`:
```conf
## Define a `bind` address pointing to a local IP that your other machines
## can reach you. If you really need to bind to an external accessible IP, make
## sure you add extra firewall rules to prevent unauthorized access:
bind 10.0.0.1
## Define a `port` to force redis to listen on TCP so other machines can
## connect to it (default port is `6379`).
port 6379
## Set up password authentication (use the same password in all nodes).
## The password should be defined equal for both `requirepass` and `masterauth`
## when setting up Redis to use with Sentinel.
requirepass redis-password-goes-here
masterauth redis-password-goes-here
```
1. Restart the Redis service for the changes to take effect.
### Step 2. Configuring the replica Redis instances
Assuming that the Redis replica instance IP is `10.0.0.2`:
1. [Install Redis](../../install/installation.md#7-redis).
1. Edit `/etc/redis/redis.conf`:
```conf
## Define a `bind` address pointing to a local IP that your other machines
## can reach you. If you really need to bind to an external accessible IP, make
## sure you add extra firewall rules to prevent unauthorized access:
bind 10.0.0.2
## Define a `port` to force redis to listen on TCP so other machines can
## connect to it (default port is `6379`).
port 6379
## Set up password authentication (use the same password in all nodes).
## The password should be defined equal for both `requirepass` and `masterauth`
## when setting up Redis to use with Sentinel.
requirepass redis-password-goes-here
masterauth redis-password-goes-here
## Define `replicaof` pointing to the Redis primary instance with IP and port.
replicaof 10.0.0.1 6379
```
1. Restart the Redis service for the changes to take effect.
1. Go through the steps again for all the other replica nodes.
### Step 3. Configuring the Redis Sentinel instances
Sentinel is a special type of Redis server. It inherits most of the basic
configuration options you can define in `redis.conf`, with specific ones
starting with `sentinel` prefix.
Assuming that the Redis Sentinel is installed on the same instance as Redis
primary with IP `10.0.0.1` (some settings might overlap with the primary):
1. [Install Redis Sentinel](https://redis.io/topics/sentinel).
1. Edit `/etc/redis/sentinel.conf`:
```conf
## Define a `bind` address pointing to a local IP that your other machines
## can reach you. If you really need to bind to an external accessible IP, make
## sure you add extra firewall rules to prevent unauthorized access:
bind 10.0.0.1
## Define a `port` to force Sentinel to listen on TCP so other machines can
## connect to it (default port is `6379`).
port 26379
## Set up password authentication (use the same password in all nodes).
## The password should be defined equal for both `requirepass` and `masterauth`
## when setting up Redis to use with Sentinel.
requirepass redis-password-goes-here
masterauth redis-password-goes-here
## Define with `sentinel auth-pass` the same shared password you have
## defined for both Redis primary and replicas instances.
sentinel auth-pass gitlab-redis redis-password-goes-here
## Define with `sentinel monitor` the IP and port of the Redis
## primary node, and the quorum required to start a failover.
sentinel monitor gitlab-redis 10.0.0.1 6379 2
## Define with `sentinel down-after-milliseconds` the time in `ms`
2021-09-04 01:27:46 +05:30
## that an unresponsive server is considered down.
2020-07-28 23:09:34 +05:30
sentinel down-after-milliseconds gitlab-redis 10000
## Define a value for `sentinel failover_timeout` in `ms`. This has multiple
## meanings:
##
## * The time needed to re-start a failover after a previous failover was
## already tried against the same primary by a given Sentinel, is two
## times the failover timeout.
##
## * The time needed for a replica replicating to a wrong primary according
## to a Sentinel current configuration, to be forced to replicate
## with the right primary, is exactly the failover timeout (counting since
## the moment a Sentinel detected the misconfiguration).
##
## * The time needed to cancel a failover that is already in progress but
## did not produced any configuration change (REPLICAOF NO ONE yet not
## acknowledged by the promoted replica).
##
## * The maximum time a failover in progress waits for all the replicas to be
## reconfigured as replicas of the new primary. However even after this time
2021-09-04 01:27:46 +05:30
## the replicas are reconfigured by the Sentinels anyway, but not with
2020-07-28 23:09:34 +05:30
## the exact parallel-syncs progression as specified.
sentinel failover_timeout 30000
```
1. Restart the Redis service for the changes to take effect.
1. Go through the steps again for all the other Sentinel nodes.
### Step 4. Configuring the GitLab application
You can enable or disable Sentinel support at any time in new or existing
installations. From the GitLab application perspective, all it requires is
the correct credentials for the Sentinel nodes.
While it doesn't require a list of all Sentinel nodes, in case of a failure,
it needs to access at least one of listed ones.
2020-10-24 23:57:45 +05:30
The following steps should be performed in the GitLab application server
2020-07-28 23:09:34 +05:30
which ideally should not have Redis or Sentinels in the same machine:
1. Edit `/home/git/gitlab/config/resque.yml` following the example in
2021-09-04 01:27:46 +05:30
[resque.yml.example](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/resque.yml.example), and uncomment the Sentinel lines, pointing to
2020-07-28 23:09:34 +05:30
the correct server credentials:
```yaml
# resque.yaml
production:
url: redis://:redi-password-goes-here@gitlab-redis/
sentinels:
-
host: 10.0.0.1
2021-01-03 14:25:43 +05:30
port: 26379 # point to sentinel, not to redis port
2020-07-28 23:09:34 +05:30
-
host: 10.0.0.2
2021-01-03 14:25:43 +05:30
port: 26379 # point to sentinel, not to redis port
2020-07-28 23:09:34 +05:30
-
host: 10.0.0.3
2021-01-03 14:25:43 +05:30
port: 26379 # point to sentinel, not to redis port
2020-07-28 23:09:34 +05:30
```
1. [Restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect.
## Example of minimal configuration with 1 primary, 2 replicas and 3 sentinels
In this example we consider that all servers have an internal network
interface with IPs in the `10.0.0.x` range, and that they can connect
to each other using these IPs.
In a real world usage, you would also set up firewall rules to prevent
unauthorized access from other machines, and block traffic from the
outside ([Internet](https://gitlab.com/gitlab-org/gitlab-foss/uploads/c4cc8cd353604bd80315f9384035ff9e/The_Internet_IT_Crowd.png)).
2021-09-04 01:27:46 +05:30
For this example, **Sentinel 1** is configured in the same machine as the
2020-07-28 23:09:34 +05:30
**Redis Primary**, **Sentinel 2** and **Sentinel 3** in the same machines as the
**Replica 1** and **Replica 2** respectively.
Here is a list and description of each **machine** and the assigned **IP**:
- `10.0.0.1`: Redis Primary + Sentinel 1
- `10.0.0.2`: Redis Replica 1 + Sentinel 2
- `10.0.0.3`: Redis Replica 2 + Sentinel 3
- `10.0.0.4`: GitLab application
2021-10-27 15:23:28 +05:30
After the initial configuration, if a failover is initiated
2021-09-04 01:27:46 +05:30
by the Sentinel nodes, the Redis nodes are reconfigured and the **Primary**
changes permanently (including in `redis.conf`) from one node to the other,
2020-07-28 23:09:34 +05:30
until a new failover is initiated again.
2021-09-04 01:27:46 +05:30
The same thing happens with `sentinel.conf` that is overridden after the
2020-07-28 23:09:34 +05:30
initial execution, after any new sentinel node starts watching the **Primary**,
or a failover promotes a different **Primary** node.
### Example configuration for Redis primary and Sentinel 1
1. In `/etc/redis/redis.conf`:
```conf
bind 10.0.0.1
port 6379
requirepass redis-password-goes-here
masterauth redis-password-goes-here
```
1. In `/etc/redis/sentinel.conf`:
```conf
bind 10.0.0.1
port 26379
sentinel auth-pass gitlab-redis redis-password-goes-here
sentinel monitor gitlab-redis 10.0.0.1 6379 2
sentinel down-after-milliseconds gitlab-redis 10000
sentinel failover_timeout 30000
```
1. Restart the Redis service for the changes to take effect.
### Example configuration for Redis replica 1 and Sentinel 2
1. In `/etc/redis/redis.conf`:
```conf
bind 10.0.0.2
port 6379
requirepass redis-password-goes-here
masterauth redis-password-goes-here
replicaof 10.0.0.1 6379
```
1. In `/etc/redis/sentinel.conf`:
```conf
bind 10.0.0.2
port 26379
sentinel auth-pass gitlab-redis redis-password-goes-here
sentinel monitor gitlab-redis 10.0.0.1 6379 2
sentinel down-after-milliseconds gitlab-redis 10000
sentinel failover_timeout 30000
```
1. Restart the Redis service for the changes to take effect.
### Example configuration for Redis replica 2 and Sentinel 3
1. In `/etc/redis/redis.conf`:
```conf
bind 10.0.0.3
port 6379
requirepass redis-password-goes-here
masterauth redis-password-goes-here
replicaof 10.0.0.1 6379
```
1. In `/etc/redis/sentinel.conf`:
```conf
bind 10.0.0.3
port 26379
sentinel auth-pass gitlab-redis redis-password-goes-here
sentinel monitor gitlab-redis 10.0.0.1 6379 2
sentinel down-after-milliseconds gitlab-redis 10000
sentinel failover_timeout 30000
```
1. Restart the Redis service for the changes to take effect.
### Example configuration of the GitLab application
1. Edit `/home/git/gitlab/config/resque.yml`:
```yaml
production:
2021-11-11 11:23:49 +05:30
url: redis://:redis-password-goes-here@gitlab-redis/
2020-07-28 23:09:34 +05:30
sentinels:
-
host: 10.0.0.1
2021-01-03 14:25:43 +05:30
port: 26379 # point to sentinel, not to redis port
2020-07-28 23:09:34 +05:30
-
host: 10.0.0.2
2021-01-03 14:25:43 +05:30
port: 26379 # point to sentinel, not to redis port
2020-07-28 23:09:34 +05:30
-
host: 10.0.0.3
2021-01-03 14:25:43 +05:30
port: 26379 # point to sentinel, not to redis port
2020-07-28 23:09:34 +05:30
```
1. [Restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect.
## Troubleshooting
See the [Redis troubleshooting guide](troubleshooting.md).