2019-09-30 21:07:59 +05:30
---
type: reference
---
2019-07-31 22:56:46 +05:30
2019-09-30 21:07:59 +05:30
# Working with the bundle Pgbouncer service
2019-07-31 22:56:46 +05:30
As part of its High Availability stack, GitLab Premium includes a bundled version of [Pgbouncer ](https://pgbouncer.github.io/ ) that can be managed through `/etc/gitlab/gitlab.rb` .
In a High Availability setup, Pgbouncer is used to seamlessly migrate database connections between servers in a failover scenario.
Additionally, it can be used in a non-HA setup to pool connections, speeding up response time while reducing resource usage.
It is recommended to run pgbouncer alongside the `gitlab-rails` service, or on its own dedicated node in a cluster.
## Operations
### Running Pgbouncer as part of an HA GitLab installation
2019-09-30 21:07:59 +05:30
1. Make sure you collect [`CONSUL_SERVER_NODES` ](database.md#consul-information ), [`CONSUL_PASSWORD_HASH` ](database.md#consul-information ), and [`PGBOUNCER_PASSWORD_HASH` ](database.md#pgbouncer-information ) before executing the next step.
1. Edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section:
2019-10-12 21:52:04 +05:30
```ruby
# Disable all components except Pgbouncer and Consul agent
roles ['pgbouncer_role']
# Configure Pgbouncer
pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul)
# Configure Consul agent
consul['watchers'] = %w(postgresql)
# START user configuration
# Please set the real values as explained in Required Information section
# Replace CONSUL_PASSWORD_HASH with with a generated md5 value
# Replace PGBOUNCER_PASSWORD_HASH with with a generated md5 value
pgbouncer['users'] = {
'gitlab-consul': {
password: 'CONSUL_PASSWORD_HASH'
},
'pgbouncer': {
password: 'PGBOUNCER_PASSWORD_HASH'
}
}
# Replace placeholders:
#
# Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z
# with the addresses gathered for CONSUL_SERVER_NODES
consul['configuration'] = {
retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z)
}
#
# END user configuration
```
NOTE: **Note:**
`pgbouncer_role` was introduced with GitLab 10.3.
1. Run `gitlab-ctl reconfigure`
2019-09-30 21:07:59 +05:30
1. Create a `.pgpass` file so Consul is able to
reload pgbouncer. Enter the `PGBOUNCER_PASSWORD` twice when asked:
2019-10-12 21:52:04 +05:30
```sh
gitlab-ctl write-pgpass --host 127.0.0.1 --database pgbouncer --user pgbouncer --hostuser gitlab-consul
```
2019-09-30 21:07:59 +05:30
#### PGBouncer Checkpoint
1. Ensure the node is talking to the current master:
2019-10-12 21:52:04 +05:30
```sh
gitlab-ctl pgb-console # You will be prompted for PGBOUNCER_PASSWORD
```
2019-09-30 21:07:59 +05:30
2019-10-12 21:52:04 +05:30
If there is an error `psql: ERROR: Auth failed` after typing in the
password, ensure you previously generated the MD5 password hashes with the correct
format. The correct format is to concatenate the password and the username:
`PASSWORDUSERNAME` . For example, `Sup3rS3cr3tpgbouncer` would be the text
needed to generate an MD5 password hash for the `pgbouncer` user.
2019-09-30 21:07:59 +05:30
1. Once the console prompt is available, run the following queries:
2019-10-12 21:52:04 +05:30
```sh
show databases ; show clients ;
```
2019-09-30 21:07:59 +05:30
2019-10-12 21:52:04 +05:30
The output should be similar to the following:
2019-09-30 21:07:59 +05:30
2019-10-12 21:52:04 +05:30
```
name | host | port | database | force_user | pool_size | reserve_pool | pool_mode | max_connections | current_connections
---------------------+-------------+------+---------------------+------------+-----------+--------------+-----------+-----------------+---------------------
gitlabhq_production | MASTER_HOST | 5432 | gitlabhq_production | | 20 | 0 | | 0 | 0
pgbouncer | | 6432 | pgbouncer | pgbouncer | 2 | 0 | statement | 0 | 0
(2 rows)
2019-09-30 21:07:59 +05:30
2019-10-12 21:52:04 +05:30
type | user | database | state | addr | port | local_addr | local_port | connect_time | request_time | ptr | link | remote_pid | tls
------+-----------+---------------------+---------+----------------+-------+------------+------------+---------------------+---------------------+-----------+------+------------+-----
C | pgbouncer | pgbouncer | active | 127.0.0.1 | 56846 | 127.0.0.1 | 6432 | 2017-08-21 18:09:59 | 2017-08-21 18:10:48 | 0x22b3880 | | 0 |
(2 rows)
```
2019-07-31 22:56:46 +05:30
### Running Pgbouncer as part of a non-HA GitLab installation
1. Generate PGBOUNCER_USER_PASSWORD_HASH with the command `gitlab-ctl pg-password-md5 pgbouncer`
1. Generate SQL_USER_PASSWORD_HASH with the command `gitlab-ctl pg-password-md5 gitlab` . We'll also need to enter the plaintext SQL_USER_PASSWORD later
1. On your database node, ensure the following is set in your `/etc/gitlab/gitlab.rb`
2019-09-30 21:07:59 +05:30
```ruby
postgresql['pgbouncer_user_password'] = 'PGBOUNCER_USER_PASSWORD_HASH'
postgresql['sql_user_password'] = 'SQL_USER_PASSWORD_HASH'
postgresql['listen_address'] = 'XX.XX.XX.Y' # Where XX.XX.XX.Y is the ip address on the node postgresql should listen on
postgresql['md5_auth_cidr_addresses'] = %w(AA.AA.AA.B/32) # Where AA.AA.AA.B is the IP address of the pgbouncer node
```
2019-07-31 22:56:46 +05:30
1. Run `gitlab-ctl reconfigure`
2019-09-30 21:07:59 +05:30
**Note:** If the database was already running, it will need to be restarted after reconfigure by running `gitlab-ctl restart postgresql` .
2019-07-31 22:56:46 +05:30
1. On the node you are running pgbouncer on, make sure the following is set in `/etc/gitlab/gitlab.rb`
2019-09-30 21:07:59 +05:30
```ruby
pgbouncer['enable'] = true
pgbouncer['databases'] = {
gitlabhq_production: {
host: 'DATABASE_HOST',
user: 'pgbouncer',
password: 'PGBOUNCER_USER_PASSWORD_HASH'
}
}
```
2019-07-31 22:56:46 +05:30
1. Run `gitlab-ctl reconfigure`
1. On the node running unicorn, make sure the following is set in `/etc/gitlab/gitlab.rb`
2019-09-30 21:07:59 +05:30
```ruby
gitlab_rails['db_host'] = 'PGBOUNCER_HOST'
gitlab_rails['db_port'] = '6432'
gitlab_rails['db_password'] = 'SQL_USER_PASSWORD'
```
2019-07-31 22:56:46 +05:30
1. Run `gitlab-ctl reconfigure`
1. At this point, your instance should connect to the database through pgbouncer. If you are having issues, see the [Troubleshooting ](#troubleshooting ) section
2019-09-30 21:07:59 +05:30
## Enable Monitoring
> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0.
If you enable Monitoring, it must be enabled on **all** pgbouncer servers.
1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration:
```ruby
# Enable service discovery for Prometheus
consul['enable'] = true
consul['monitoring_service_discovery'] = true
# Replace placeholders
# Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z
# with the addresses of the Consul server nodes
consul['configuration'] = {
retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z),
}
# Set the network addresses that the exporters will listen on
node_exporter['listen_address'] = '0.0.0.0:9100'
pgbouncer_exporter['listen_address'] = '0.0.0.0:9188'
```
1. Run `sudo gitlab-ctl reconfigure` to compile the configuration.
2019-07-31 22:56:46 +05:30
### Interacting with pgbouncer
#### Administrative console
As part of omnibus-gitlab, we provide a command `gitlab-ctl pgb-console` to automatically connect to the pgbouncer administrative console. Please see the [pgbouncer documentation ](https://pgbouncer.github.io/usage.html#admin-console ) for detailed instructions on how to interact with the console.
To start a session, run
```shell
# gitlab-ctl pgb-console
Password for user pgbouncer:
psql (9.6.8, server 1.7.2/bouncer)
Type "help" for help.
pgbouncer=#
```
The password you will be prompted for is the PGBOUNCER_USER_PASSWORD
To get some basic information about the instance, run
2019-09-30 21:07:59 +05:30
2019-07-31 22:56:46 +05:30
```shell
pgbouncer=# show databases; show clients; show servers;
name | host | port | database | force_user | pool_size | reserve_pool | pool_mode | max_connections | current_connections
---------------------+-----------+------+---------------------+------------+-----------+--------------+-----------+-----------------+---------------------
gitlabhq_production | 127.0.0.1 | 5432 | gitlabhq_production | | 100 | 5 | | 0 | 1
pgbouncer | | 6432 | pgbouncer | pgbouncer | 2 | 0 | statement | 0 | 0
(2 rows)
type | user | database | state | addr | port | local_addr | local_port | connect_time | request_time | ptr | link
| remote_pid | tls
------+-----------+---------------------+--------+-----------+-------+------------+------------+---------------------+---------------------+-----------+------
+------------+-----
C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44590 | 127.0.0.1 | 6432 | 2018-04-24 22:13:10 | 2018-04-24 22:17:10 | 0x12444c0 |
| 0 |
C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44592 | 127.0.0.1 | 6432 | 2018-04-24 22:13:10 | 2018-04-24 22:17:10 | 0x12447c0 |
| 0 |
C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44594 | 127.0.0.1 | 6432 | 2018-04-24 22:13:10 | 2018-04-24 22:17:10 | 0x1244940 |
| 0 |
C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44706 | 127.0.0.1 | 6432 | 2018-04-24 22:14:22 | 2018-04-24 22:16:31 | 0x1244ac0 |
| 0 |
C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44708 | 127.0.0.1 | 6432 | 2018-04-24 22:14:22 | 2018-04-24 22:15:15 | 0x1244c40 |
| 0 |
C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44794 | 127.0.0.1 | 6432 | 2018-04-24 22:15:15 | 2018-04-24 22:15:15 | 0x1244dc0 |
| 0 |
C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44798 | 127.0.0.1 | 6432 | 2018-04-24 22:15:15 | 2018-04-24 22:16:31 | 0x1244f40 |
| 0 |
C | pgbouncer | pgbouncer | active | 127.0.0.1 | 44660 | 127.0.0.1 | 6432 | 2018-04-24 22:13:51 | 2018-04-24 22:17:12 | 0x1244640 |
| 0 |
(8 rows)
type | user | database | state | addr | port | local_addr | local_port | connect_time | request_time | ptr | link | rem
ote_pid | tls
------+--------+---------------------+-------+-----------+------+------------+------------+---------------------+---------------------+-----------+------+----
--------+-----
S | gitlab | gitlabhq_production | idle | 127.0.0.1 | 5432 | 127.0.0.1 | 35646 | 2018-04-24 22:15:15 | 2018-04-24 22:17:10 | 0x124dca0 | |
19980 |
(1 row)
```
## Troubleshooting
In case you are experiencing any issues connecting through pgbouncer, the first place to check is always the logs:
```shell
# gitlab-ctl tail pgbouncer
```
Additionally, you can check the output from `show databases` in the [Administrative console ](#administrative-console ). In the output, you would expect to see values in the `host` field for the `gitlabhq_production` database. Additionally, `current_connections` should be greater than 1.
