debian-mirror-gitlab/doc/administration/operations/fast_ssh_key_lookup.md

227 lines
9.4 KiB
Markdown
Raw Normal View History

2021-01-03 14:25:43 +05:30
---
stage: none
group: unassigned
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
---
2018-03-17 18:26:18 +05:30
# Fast lookup of authorized SSH keys in the database
2020-06-23 00:09:42 +05:30
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/1631) in [GitLab Starter](https://about.gitlab.com/pricing/) 9.3.
> - [Available in](https://gitlab.com/gitlab-org/gitlab/-/issues/3953) GitLab Community Edition 10.4.
2018-11-18 11:00:15 +05:30
2020-07-28 23:09:34 +05:30
NOTE: **Note:**
This document describes a drop-in replacement for the
2020-04-22 19:07:51 +05:30
`authorized_keys` file. For normal (non-deploy key) users, consider using
[SSH certificates](ssh_certificates.md). They are even faster, but are not a
drop-in replacement.
2018-03-17 18:26:18 +05:30
Regular SSH operations become slow as the number of users grows because OpenSSH
searches for a key to authorize a user via a linear search. In the worst case,
such as when the user is not authorized to access GitLab, OpenSSH will scan the
entire file to search for a key. This can take significant time and disk I/O,
which will delay users attempting to push or pull to a repository. Making
matters worse, if users add or remove keys frequently, the operating system may
not be able to cache the `authorized_keys` file, which causes the disk to be
accessed repeatedly.
GitLab Shell solves this by providing a way to authorize SSH users via a fast,
indexed lookup in the GitLab database. This page describes how to enable the fast
lookup of authorized SSH keys.
> **Warning:** OpenSSH version 6.9+ is required because
`AuthorizedKeysCommand` must be able to accept a fingerprint. These
instructions will break installations using older versions of OpenSSH, such as
those included with CentOS 6 as of September 2017. If you want to use this
feature for CentOS 6, follow [the instructions on how to build and install a custom OpenSSH package](#compiling-a-custom-version-of-openssh-for-centos-6) before continuing.
2019-09-30 21:07:59 +05:30
## Fast lookup is required for Geo **(PREMIUM)**
2019-07-31 22:56:46 +05:30
By default, GitLab manages an `authorized_keys` file, which contains all the
public SSH keys for users allowed to access GitLab. However, to maintain a
2020-11-24 15:15:51 +05:30
single source of truth, [Geo](../geo/index.md) needs to be configured to perform SSH fingerprint
2019-07-31 22:56:46 +05:30
lookups via database lookup.
2020-11-24 15:15:51 +05:30
As part of [setting up Geo](../geo/index.md#setup-instructions),
2019-07-31 22:56:46 +05:30
you will be required to follow the steps outlined below for both the primary and
secondary nodes, but note that the `Write to "authorized keys" file` checkbox
only needs to be unchecked on the primary node since it will be reflected
automatically on the secondary if database replication is working.
2018-03-17 18:26:18 +05:30
## Setting up fast lookup via GitLab Shell
GitLab Shell provides a way to authorize SSH users via a fast, indexed lookup
to the GitLab database. GitLab Shell uses the fingerprint of the SSH key to
check whether the user is authorized to access GitLab.
2018-10-15 14:42:47 +05:30
Add the following to your `sshd_config` file. This is usually located at
2018-03-17 18:26:18 +05:30
`/etc/ssh/sshd_config`, but it will be `/assets/sshd_config` if you're using
Omnibus Docker:
2020-03-13 15:44:24 +05:30
```plaintext
2020-04-08 14:13:33 +05:30
Match User git # Apply the AuthorizedKeysCommands to the git user only
AuthorizedKeysCommand /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell-authorized-keys-check git %u %k
AuthorizedKeysCommandUser git
Match all # End match, settings apply to all users again
2018-03-17 18:26:18 +05:30
```
Reload OpenSSH:
2020-03-13 15:44:24 +05:30
```shell
2018-03-17 18:26:18 +05:30
# Debian or Ubuntu installations
sudo service ssh reload
# CentOS installations
sudo service sshd reload
```
2020-07-28 23:09:34 +05:30
Confirm that SSH is working by commenting out your user's key in the `authorized_keys`
(start the line with a `#` to comment it), and attempting to pull a repository.
2018-03-17 18:26:18 +05:30
2020-07-28 23:09:34 +05:30
A successful pull would mean that GitLab was able to find the key in the database,
since it is not present in the file anymore.
NOTE: **Note:**
For Omnibus Docker, `AuthorizedKeysCommand` is setup by default in
2019-07-31 22:56:46 +05:30
GitLab 11.11 and later.
2020-07-28 23:09:34 +05:30
NOTE: **Note:**
For Installations from source, the command would be located at
2020-05-24 23:13:21 +05:30
`/home/git/gitlab-shell/bin/gitlab-shell-authorized-keys-check` if [the install from source](../../install/installation.md#install-gitlab-shell) instructions were followed.
You might want to consider creating a wrapper script somewhere else since this command needs to be
owned by `root` and not be writable by group or others. You could also consider changing the ownership of this command
as required, but that might require temporary ownership changes during `gitlab-shell` upgrades.
2020-07-28 23:09:34 +05:30
CAUTION: **Caution:**
Do not disable writes until SSH is confirmed to be working
2018-03-17 18:26:18 +05:30
perfectly, because the file will quickly become out-of-date.
2018-03-27 19:54:05 +05:30
In the case of lookup failures (which are common), the `authorized_keys`
2019-12-21 20:55:43 +05:30
file will still be scanned. So Git SSH performance will still be slow for many
2018-03-17 18:26:18 +05:30
users as long as a large file exists.
You can disable any more writes to the `authorized_keys` file by unchecking
2019-07-31 22:56:46 +05:30
`Write to "authorized_keys" file` in the **Admin Area > Settings > Network > Performance optimization** of your GitLab
2018-03-17 18:26:18 +05:30
installation.
![Write to authorized keys setting](img/write_to_authorized_keys_setting.png)
Again, confirm that SSH is working by removing your user's SSH key in the UI,
2020-05-24 23:13:21 +05:30
adding a new one, and attempting to pull a repository.
2018-03-17 18:26:18 +05:30
Then you can backup and delete your `authorized_keys` file for best performance.
2020-07-28 23:09:34 +05:30
The current users' keys are already present in the database, so there is no need for migration
or for asking users to re-add their keys.
2018-03-17 18:26:18 +05:30
## How to go back to using the `authorized_keys` file
This is a brief overview. Please refer to the above instructions for more context.
1. [Rebuild the `authorized_keys` file](../raketasks/maintenance.md#rebuild-authorized_keys-file)
1. Enable writes to the `authorized_keys` file in Application Settings
1. Remove the `AuthorizedKeysCommand` lines from `/etc/ssh/sshd_config` or from `/assets/sshd_config` if you are using Omnibus Docker.
2020-04-22 19:07:51 +05:30
1. Reload `sshd`: `sudo service sshd reload`
2018-03-17 18:26:18 +05:30
## Compiling a custom version of OpenSSH for CentOS 6
Building a custom version of OpenSSH is not necessary for Ubuntu 16.04 users,
since Ubuntu 16.04 ships with OpenSSH 7.2.
It is also unnecessary for CentOS 7.4 users, as that version ships with
OpenSSH 7.4. If you are using CentOS 7.0 - 7.3, we strongly recommend that you
upgrade to CentOS 7.4 instead of following this procedure. This should be as
simple as running `yum update`.
CentOS 6 users must build their own OpenSSH package to enable SSH lookups via
the database. The following instructions can be used to build OpenSSH 7.5:
1. First, download the package and install the required packages:
2020-03-13 15:44:24 +05:30
```shell
2019-09-30 21:07:59 +05:30
sudo su -
cd /tmp
2020-06-23 00:09:42 +05:30
curl --remote-name https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-7.5p1.tar.gz
2019-09-30 21:07:59 +05:30
tar xzvf openssh-7.5p1.tar.gz
yum install rpm-build gcc make wget openssl-devel krb5-devel pam-devel libX11-devel xmkmf libXt-devel
```
2018-03-17 18:26:18 +05:30
2019-02-15 15:39:39 +05:30
1. Prepare the build by copying files to the right place:
2018-03-17 18:26:18 +05:30
2020-03-13 15:44:24 +05:30
```shell
2019-09-30 21:07:59 +05:30
mkdir -p /root/rpmbuild/{SOURCES,SPECS}
cp ./openssh-7.5p1/contrib/redhat/openssh.spec /root/rpmbuild/SPECS/
cp openssh-7.5p1.tar.gz /root/rpmbuild/SOURCES/
cd /root/rpmbuild/SPECS
```
2018-03-17 18:26:18 +05:30
2019-02-15 15:39:39 +05:30
1. Next, set the spec settings properly:
2018-03-17 18:26:18 +05:30
2020-03-13 15:44:24 +05:30
```shell
2019-09-30 21:07:59 +05:30
sed -i -e "s/%define no_gnome_askpass 0/%define no_gnome_askpass 1/g" openssh.spec
sed -i -e "s/%define no_x11_askpass 0/%define no_x11_askpass 1/g" openssh.spec
sed -i -e "s/BuildPreReq/BuildRequires/g" openssh.spec
```
2018-03-17 18:26:18 +05:30
2019-02-15 15:39:39 +05:30
1. Build the RPMs:
2018-03-17 18:26:18 +05:30
2020-03-13 15:44:24 +05:30
```shell
2019-09-30 21:07:59 +05:30
rpmbuild -bb openssh.spec
```
2018-03-17 18:26:18 +05:30
2019-02-15 15:39:39 +05:30
1. Ensure the RPMs were built:
2018-03-17 18:26:18 +05:30
2020-03-13 15:44:24 +05:30
```shell
2019-09-30 21:07:59 +05:30
ls -al /root/rpmbuild/RPMS/x86_64/
```
2018-03-17 18:26:18 +05:30
2019-09-30 21:07:59 +05:30
You should see something as the following:
2018-03-17 18:26:18 +05:30
2020-03-13 15:44:24 +05:30
```plaintext
2019-09-30 21:07:59 +05:30
total 1324
drwxr-xr-x. 2 root root 4096 Jun 20 19:37 .
drwxr-xr-x. 3 root root 19 Jun 20 19:37 ..
-rw-r--r--. 1 root root 470828 Jun 20 19:37 openssh-7.5p1-1.x86_64.rpm
-rw-r--r--. 1 root root 490716 Jun 20 19:37 openssh-clients-7.5p1-1.x86_64.rpm
-rw-r--r--. 1 root root 17020 Jun 20 19:37 openssh-debuginfo-7.5p1-1.x86_64.rpm
-rw-r--r--. 1 root root 367516 Jun 20 19:37 openssh-server-7.5p1-1.x86_64.rpm
```
2018-03-17 18:26:18 +05:30
2019-02-15 15:39:39 +05:30
1. Install the packages. OpenSSH packages will replace `/etc/pam.d/sshd`
2018-03-17 18:26:18 +05:30
with its own version, which may prevent users from logging in, so be sure
that the file is backed up and restored after installation:
2020-03-13 15:44:24 +05:30
```shell
2019-09-30 21:07:59 +05:30
timestamp=$(date +%s)
cp /etc/pam.d/sshd pam-ssh-conf-$timestamp
rpm -Uvh /root/rpmbuild/RPMS/x86_64/*.rpm
yes | cp pam-ssh-conf-$timestamp /etc/pam.d/sshd
```
2018-03-17 18:26:18 +05:30
2019-02-15 15:39:39 +05:30
1. Verify the installed version. In another window, attempt to login to the server:
2018-03-17 18:26:18 +05:30
2020-03-13 15:44:24 +05:30
```shell
2019-09-30 21:07:59 +05:30
ssh -v <your-centos-machine>
```
2018-03-17 18:26:18 +05:30
2019-09-30 21:07:59 +05:30
You should see a line that reads: "debug1: Remote protocol version 2.0, remote software version OpenSSH_7.5"
2018-03-17 18:26:18 +05:30
2020-04-22 19:07:51 +05:30
If not, you may need to restart `sshd` (e.g. `systemctl restart sshd.service`).
2018-03-17 18:26:18 +05:30
2019-09-30 21:07:59 +05:30
1. *IMPORTANT!* Open a new SSH session to your server before exiting to make
sure everything is working! If you need to downgrade, simple install the
older package:
2018-03-17 18:26:18 +05:30
2020-03-13 15:44:24 +05:30
```shell
2019-09-30 21:07:59 +05:30
# Only run this if you run into a problem logging in
yum downgrade openssh-server openssh openssh-clients
```
2020-07-28 23:09:34 +05:30
## SELinux support and limitations
> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/2855) in GitLab 10.5.
GitLab supports `authorized_keys` database lookups with [SELinux](https://en.wikipedia.org/wiki/Security-Enhanced_Linux).
Because the SELinux policy is static, GitLab doesn't support the ability to change
internal Unicorn ports at the moment. Admins would have to create a special `.te`
file for the environment, since it isn't generated dynamically.