info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
feature of Puma does not work with the way it is configured in GitLab all-in-one
packages (cluster-mode with app preloading).
- While it is possible to minimize downtime on a single-node instance by following
these instructions, **it is not possible to always achieve true zero downtime
updates**. Users may see some connections timeout or be refused for a few minutes,
depending on which services need to restart.
1. Create an empty file at `/etc/gitlab/skip-auto-reconfigure`. This prevents upgrades from running `gitlab-ctl reconfigure`, which by default automatically stops GitLab, runs all database migrations, and restarts GitLab.
```shell
sudo touch /etc/gitlab/skip-auto-reconfigure
```
1. Update the GitLab package:
- For GitLab Community Edition:
```shell
# Debian/Ubuntu
sudo apt-get update
sudo apt-get install gitlab-ce
# Centos/RHEL
sudo yum install gitlab-ce
```
- For GitLab [Enterprise Edition](https://about.gitlab.com/pricing/):
```shell
# Debian/Ubuntu
sudo apt-get update
sudo apt-get install gitlab-ee
# Centos/RHEL
sudo yum install gitlab-ee
```
1. To get the regular migrations and latest code in place, run
1. Once the node is updated and `reconfigure` finished successfully, run post-deployment migrations with
```shell
sudo gitlab-rake db:migrate
```
1. Hot reload `puma` and `sidekiq` services
```shell
sudo gitlab-ctl hup puma
sudo gitlab-ctl restart sidekiq
```
If you do not want to run zero downtime upgrades in the future, make
sure you remove `/etc/gitlab/skip-auto-reconfigure` after
you've completed these steps.
## Multi-node / HA deployment
You can only upgrade 1 minor release at a time. So from 13.6 to 13.7, not to 13.8.
If you attempt more than one minor release, the upgrade may fail.
### Use a load balancer in front of web (Puma) nodes
With Puma, single node zero-downtime updates are no longer possible. To achieve
HA with zero-downtime updates, at least two nodes are required to be used with a
load balancer which distributes the connections properly across both nodes.
The load balancer in front of the application nodes must be configured to check
proper health check endpoints to check if the service is accepting traffic or
not. For Puma, the `/-/readiness` endpoint should be used, while
`/readiness` endpoint can be used for Sidekiq and other services.
Upgrades on web (Puma) nodes must be done in a rolling manner, one after
another, ensuring at least one node is always up to serve traffic. This is
required to ensure zero-downtime.
Puma will enter a blackout period as part of the upgrade, during which they
continue to accept connections but will mark their respective health check
endpoints to be unhealthy. On seeing this, the load balancer should disconnect
them gracefully.
Puma will restart only after completing all the currently processing requests.
This ensures data and service integrity. Once they have restarted, the health
check end points will be marked healthy.
The nodes must be updated in the following order to update an HA instance using
load balancer to latest GitLab version.
1. Select one application node as a deploy node and complete the following steps
on it:
1. Create an empty file at `/etc/gitlab/skip-auto-reconfigure`. This prevents upgrades from running `gitlab-ctl reconfigure`, which by default automatically stops GitLab, runs all database migrations, and restarts GitLab:
1. Complete the following steps on the other Puma/Sidekiq nodes, one
after another. Always ensure at least one of such nodes is up and running,
and connected to the load balancer before proceeding to the next node.
1. Update the GitLab package and ensure a `reconfigure` is run as part of
it. If not (due to `/etc/gitlab/skip-auto-reconfigure` file being
present), run `sudo gitlab-ctl reconfigure` manually.
1. Ensure services use latest code:
```shell
sudo gitlab-ctl hup puma
sudo gitlab-ctl restart sidekiq
```
1. On the deploy node, run the post-deployment migrations:
```shell
sudo gitlab-rake db:migrate
```
### Gitaly Cluster
[Gitaly Cluster](../administration/gitaly/praefect.md) is built using
Gitaly and the Praefect component. It has its own PostgreSQL database, independent of the rest of
the application.
Before you update the main application you need to update Praefect.
Out of your Praefect nodes, pick one to be your Praefect deploy node.
This is where you will install the new Omnibus package first and run
database migrations.
**Praefect deploy node**
- Create an empty file at `/etc/gitlab/skip-auto-reconfigure`. This prevents upgrades from running `gitlab-ctl reconfigure`, which by default automatically stops GitLab, runs all database migrations, and restarts GitLab:
```shell
sudo touch /etc/gitlab/skip-auto-reconfigure
```
- Ensure that `praefect['auto_migrate'] = true` is set in `/etc/gitlab/gitlab.rb`
**All Praefect nodes _excluding_ the Praefect deploy node**
- To prevent `reconfigure` from automatically running database migrations, ensure that `praefect['auto_migrate'] = false` is set in `/etc/gitlab/gitlab.rb`.
If you are an Enterprise Edition user, replace `gitlab-ce` with `gitlab-ee` in the above command.
- Ensure nodes are running the latest code:
```shell
sudo gitlab-ctl reconfigure
```
### Use PostgreSQL HA
Pick a node to be the `Deploy Node`. It can be any application node, but it must be the same
node throughout the process.
**Deploy node**
- Create an empty file at `/etc/gitlab/skip-auto-reconfigure`. This prevents upgrades from running `gitlab-ctl reconfigure`, which by default automatically stops GitLab, runs all database migrations, and restarts GitLab.
```shell
sudo touch /etc/gitlab/skip-auto-reconfigure
```
**All nodes _including_ the Deploy node**
- To prevent `reconfigure` from automatically running database migrations, ensure that `gitlab_rails['auto_migrate'] = false` is set in `/etc/gitlab/gitlab.rb`.
The order of steps is important. While following these steps, make
sure you follow them in the right order, on the correct node.
Log in to your **primary** node, executing the following:
1. Create an empty file at `/etc/gitlab/skip-auto-reconfigure`. This prevents upgrades from running `gitlab-ctl reconfigure`, which by default automatically stops GitLab, runs all database migrations, and restarts GitLab.
```shell
sudo touch /etc/gitlab/skip-auto-reconfigure
```
1. Edit `/etc/gitlab/gitlab.rb` and ensure the following is present:
On each **secondary** node, executing the following:
1. Create an empty file at `/etc/gitlab/skip-auto-reconfigure`. This prevents upgrades from running `gitlab-ctl reconfigure`, which by default automatically stops GitLab, runs all database migrations, and restarts GitLab.
```shell
sudo touch /etc/gitlab/skip-auto-reconfigure
```
1. Edit `/etc/gitlab/gitlab.rb` and ensure the following is present:
### Step 1: Choose a "deploy node" for each deployment
You now need to choose:
- One instance for use as the **primary** "deploy node" on the Geo **primary** multi-node deployment.
- One instance for use as the **secondary** "deploy node" on each Geo **secondary** multi-node deployment.
Deploy nodes must be configured to be running Puma or Sidekiq or the `geo-logcursor` daemon. In order
to avoid any downtime, they must not be in use during the update:
- If running Puma remove the deploy node from the load balancer.
- If running Sidekiq, ensure the deploy node is not processing jobs:
```shell
sudo gitlab-ctl stop sidekiq
```
- If running `geo-logcursor` daemon, ensure the deploy node is not processing events:
```shell
sudo gitlab-ctl stop geo-logcursor
```
For zero-downtime, Puma, Sidekiq, and `geo-logcursor` must be running on other nodes during the update.
### Step 2: Update the Geo primary multi-node deployment
**On all primary nodes _including_ the primary "deploy node"**
1. Create an empty file at `/etc/gitlab/skip-auto-reconfigure`. This prevents upgrades from running `gitlab-ctl reconfigure`, which by default automatically stops GitLab, runs all database migrations, and restarts GitLab.
```shell
sudo touch /etc/gitlab/skip-auto-reconfigure
```
1. To prevent `reconfigure` from automatically running database migrations, ensure that `gitlab_rails['auto_migrate'] = false` is set in `/etc/gitlab/gitlab.rb`.
**For all primary nodes that run Puma or Sidekiq _including_ the primary "deploy node"**
Hot reload `puma` and `sidekiq` services:
```shell
sudo gitlab-ctl hup puma
sudo gitlab-ctl restart sidekiq
```
### Step 3: Update each Geo secondary multi-node deployment
Only proceed if you have successfully completed all steps on the Geo **primary** multi-node deployment.
**On all secondary nodes _including_ the secondary "deploy node"**
1. Create an empty file at `/etc/gitlab/skip-auto-reconfigure`. This prevents upgrades from running `gitlab-ctl reconfigure`, which by default automatically stops GitLab, runs all database migrations, and restarts GitLab.
```shell
sudo touch /etc/gitlab/skip-auto-reconfigure
```
1. To prevent `reconfigure` from automatically running database migrations, ensure that `geo_secondary['auto_migrate'] = false` is set in `/etc/gitlab/gitlab.rb`.
