debian-mirror-gitlab/doc/administration/raketasks/maintenance.md

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

393 lines
11 KiB
Markdown
Raw Normal View History

2021-01-29 00:20:46 +05:30
---
2022-07-23 23:45:48 +05:30
stage: Systems
2021-02-22 17:27:13 +05:30
group: Distribution
2022-11-25 23:54:43 +05:30
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
2021-01-29 00:20:46 +05:30
---
2021-03-11 19:13:27 +05:30
# Maintenance Rake tasks **(FREE SELF)**
2017-08-17 22:00:37 +05:30
2020-05-24 23:13:21 +05:30
GitLab provides Rake tasks for general maintenance.
2017-08-17 22:00:37 +05:30
2020-05-24 23:13:21 +05:30
## Gather GitLab and system information
This command gathers information about your GitLab installation and the system it runs on.
These may be useful when asking for help or reporting issues.
2017-08-17 22:00:37 +05:30
**Omnibus Installation**
2020-03-13 15:44:24 +05:30
```shell
2017-08-17 22:00:37 +05:30
sudo gitlab-rake gitlab:env:info
```
**Source Installation**
2020-03-13 15:44:24 +05:30
```shell
2017-08-17 22:00:37 +05:30
bundle exec rake gitlab:env:info RAILS_ENV=production
```
Example output:
2020-03-13 15:44:24 +05:30
```plaintext
2017-08-17 22:00:37 +05:30
System information
2020-10-24 23:57:45 +05:30
System: Ubuntu 20.04
Proxy: no
Current User: git
Using RVM: no
Ruby Version: 2.6.6p146
Gem Version: 2.7.10
Bundler Version:1.17.3
Rake Version: 12.3.3
Redis Version: 5.0.9
Git Version: 2.27.0
Sidekiq Version:5.2.9
Go Version: unknown
2017-08-17 22:00:37 +05:30
GitLab information
2020-10-24 23:57:45 +05:30
Version: 13.2.2-ee
Revision: 618883a1f9d
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 11.7
URL: http://gitlab.example.com
HTTP Clone URL: http://gitlab.example.com/some-group/some-project.git
SSH Clone URL: git@gitlab.example.com:some-group/some-project.git
Elasticsearch: no
Geo: no
Using LDAP: no
Using Omniauth: yes
Omniauth Providers:
2017-08-17 22:00:37 +05:30
GitLab Shell
2020-10-24 23:57:45 +05:30
Version: 13.3.0
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
2017-08-17 22:00:37 +05:30
```
2021-03-11 19:13:27 +05:30
## Show GitLab license information **(PREMIUM SELF)**
2020-06-23 00:09:42 +05:30
2021-03-11 19:13:27 +05:30
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/20501) in GitLab 12.6.
2021-09-30 23:02:18 +05:30
> - Moved to GitLab Premium in 13.9.
2020-06-23 00:09:42 +05:30
This command shows information about your [GitLab license](../../user/admin_area/license.md) and
how many seats are used. It is only available on GitLab Enterprise
installations: a license cannot be installed into GitLab Community Edition.
These may be useful when raising tickets with Support, or for programmatically
checking your license parameters.
**Omnibus Installation**
```shell
sudo gitlab-rake gitlab:license:info
```
**Source Installation**
```shell
bundle exec rake gitlab:license:info RAILS_ENV=production
```
Example output:
```plaintext
Today's Date: 2020-02-29
Current User Count: 30
Max Historical Count: 30
Max Users in License: 40
License valid from: 2019-11-29 to 2020-11-28
Email associated with license: user@example.com
```
2017-08-17 22:00:37 +05:30
## Check GitLab configuration
2020-05-24 23:13:21 +05:30
The `gitlab:check` Rake task runs the following Rake tasks:
2017-08-17 22:00:37 +05:30
- `gitlab:gitlab_shell:check`
2018-12-13 13:39:08 +05:30
- `gitlab:gitaly:check`
2017-08-17 22:00:37 +05:30
- `gitlab:sidekiq:check`
2022-03-02 08:16:31 +05:30
- `gitlab:incoming_email:check`
- `gitlab:ldap:check`
2017-08-17 22:00:37 +05:30
- `gitlab:app:check`
2021-02-22 17:27:13 +05:30
It checks that each component was set up according to the installation guide and suggest fixes
for issues found. This command must be run from your application server and doesn't work correctly on
2021-04-17 20:07:23 +05:30
component servers like [Gitaly](../gitaly/configure_gitaly.md#run-gitaly-on-its-own-server).
2022-03-02 08:16:31 +05:30
If you're running Geo, see also the [Geo Health check Rake task](../geo/replication/troubleshooting.md#health-check-rake-task).
2020-05-24 23:13:21 +05:30
You may also have a look at our troubleshooting guides for:
2017-08-17 22:00:37 +05:30
2020-05-24 23:13:21 +05:30
- [GitLab](../index.md#troubleshooting)
2021-11-18 22:05:49 +05:30
- [Omnibus GitLab](https://docs.gitlab.com/omnibus/index.html#troubleshooting)
2019-07-07 11:18:12 +05:30
2022-03-02 08:16:31 +05:30
Additionally you should also [verify database values can be decrypted using the current secrets](check.md#verify-database-values-can-be-decrypted-using-the-current-secrets).
2020-05-24 23:13:21 +05:30
To run `gitlab:check`, run:
2017-08-17 22:00:37 +05:30
**Omnibus Installation**
2020-03-13 15:44:24 +05:30
```shell
2017-08-17 22:00:37 +05:30
sudo gitlab-rake gitlab:check
```
**Source Installation**
2020-03-13 15:44:24 +05:30
```shell
2017-08-17 22:00:37 +05:30
bundle exec rake gitlab:check RAILS_ENV=production
```
2020-05-24 23:13:21 +05:30
Use `SANITIZE=true` for `gitlab:check` if you want to omit project names from the output.
2017-08-17 22:00:37 +05:30
Example output:
2020-03-13 15:44:24 +05:30
```plaintext
2017-08-17 22:00:37 +05:30
Checking Environment ...
Git configured for git user? ... yes
Has python2? ... yes
python2 is supported version? ... yes
Checking Environment ... Finished
Checking GitLab Shell ...
GitLab Shell version? ... OK (1.2.0)
Repo base directory exists? ... yes
Repo base directory is a symlink? ... no
Repo base owned by git:git? ... yes
Repo base access is drwxrws---? ... yes
post-receive hook up-to-date? ... yes
post-receive hooks in repos are links: ... yes
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes
Checking Sidekiq ... Finished
Checking GitLab ...
Database config exists? ... yes
Database is SQLite ... no
All migrations up? ... yes
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Init script exists? ... yes
Init script up-to-date? ... yes
Redis version >= 2.0.0? ... yes
Checking GitLab ... Finished
```
2022-06-21 17:19:12 +05:30
## Rebuild `authorized_keys` file
2017-08-17 22:00:37 +05:30
2022-06-21 17:19:12 +05:30
In some cases it is necessary to rebuild the `authorized_keys` file,
2022-08-13 15:12:31 +05:30
for example, if after an upgrade you receive `Permission denied (publickey)` when pushing [via SSH](../../user/ssh.md)
2022-08-27 11:52:29 +05:30
and find `404 Key Not Found` errors in [the `gitlab-shell.log` file](../logs/index.md#gitlab-shelllog).
2022-06-21 17:19:12 +05:30
To rebuild `authorized_keys`, run:
2017-08-17 22:00:37 +05:30
**Omnibus Installation**
2020-03-13 15:44:24 +05:30
```shell
2017-08-17 22:00:37 +05:30
sudo gitlab-rake gitlab:shell:setup
```
**Source Installation**
2020-03-13 15:44:24 +05:30
```shell
2017-08-17 22:00:37 +05:30
cd /home/git/gitlab
sudo -u git -H bundle exec rake gitlab:shell:setup RAILS_ENV=production
```
2020-05-24 23:13:21 +05:30
Example output:
2020-03-13 15:44:24 +05:30
```plaintext
2017-08-17 22:00:37 +05:30
This will rebuild an authorized_keys file.
You will lose any data stored in authorized_keys file.
Do you want to continue (yes/no)? yes
```
2019-12-21 20:55:43 +05:30
## Clear Redis cache
2017-08-17 22:00:37 +05:30
2020-05-24 23:13:21 +05:30
If for some reason the dashboard displays the wrong information, you might want to
clear Redis' cache. To do this, run:
2017-08-17 22:00:37 +05:30
**Omnibus Installation**
2020-03-13 15:44:24 +05:30
```shell
2017-08-17 22:00:37 +05:30
sudo gitlab-rake cache:clear
```
**Source Installation**
2020-03-13 15:44:24 +05:30
```shell
2017-08-17 22:00:37 +05:30
cd /home/git/gitlab
sudo -u git -H bundle exec rake cache:clear RAILS_ENV=production
```
## Precompile the assets
Sometimes during version upgrades you might end up with some wrong CSS or
missing some icons. In that case, try to precompile the assets again.
2022-08-13 15:12:31 +05:30
This only applies to source installations and does not apply to
2017-08-17 22:00:37 +05:30
Omnibus packages.
**Source Installation**
2020-03-13 15:44:24 +05:30
```shell
2017-08-17 22:00:37 +05:30
cd /home/git/gitlab
sudo -u git -H bundle exec rake gitlab:assets:compile RAILS_ENV=production
```
For omnibus versions, the unoptimized assets (JavaScript, CSS) are frozen at
the release of upstream GitLab. The omnibus version includes optimized versions
of those assets. Unless you are modifying the JavaScript / CSS code on your
production machine after installing the package, there should be no reason to redo
2019-12-21 20:55:43 +05:30
`rake gitlab:assets:compile` on the production machine. If you suspect that assets
2017-08-17 22:00:37 +05:30
have been corrupted, you should reinstall the omnibus package.
2018-03-17 18:26:18 +05:30
## Check TCP connectivity to a remote site
Sometimes you need to know if your GitLab installation can connect to a TCP
2021-04-17 20:07:23 +05:30
service on another machine (for example a PostgreSQL or web server)
2021-11-18 22:05:49 +05:30
to troubleshoot proxy issues.
2021-04-17 20:07:23 +05:30
A Rake task is included to help you with this.
2018-03-17 18:26:18 +05:30
**Omnibus Installation**
2020-03-13 15:44:24 +05:30
```shell
2018-03-17 18:26:18 +05:30
sudo gitlab-rake gitlab:tcp_check[example.com,80]
```
**Source Installation**
2020-03-13 15:44:24 +05:30
```shell
2018-03-17 18:26:18 +05:30
cd /home/git/gitlab
sudo -u git -H bundle exec rake gitlab:tcp_check[example.com,80] RAILS_ENV=production
```
2018-05-09 12:01:36 +05:30
## Clear exclusive lease (DANGER)
GitLab uses a shared lock mechanism: `ExclusiveLease` to prevent simultaneous operations
in a shared resource. An example is running periodic garbage collection on repositories.
2022-01-26 12:08:38 +05:30
In very specific situations, an operation locked by an Exclusive Lease can fail without
2018-05-09 12:01:36 +05:30
releasing the lock. If you can't wait for it to expire, you can run this task to manually
clear it.
To clear all exclusive leases:
2021-02-22 17:27:13 +05:30
WARNING:
2018-05-09 12:01:36 +05:30
Don't run it while GitLab or Sidekiq is running
2020-03-13 15:44:24 +05:30
```shell
2018-05-09 12:01:36 +05:30
sudo gitlab-rake gitlab:exclusive_lease:clear
```
To specify a lease `type` or lease `type + id`, specify a scope:
2020-03-13 15:44:24 +05:30
```shell
2018-05-09 12:01:36 +05:30
# to clear all leases for repository garbage collection:
sudo gitlab-rake gitlab:exclusive_lease:clear[project_housekeeping:*]
# to clear a lease for repository garbage collection in a specific project: (id=4)
sudo gitlab-rake gitlab:exclusive_lease:clear[project_housekeeping:4]
```
2019-09-30 21:07:59 +05:30
## Display status of database migrations
2021-03-11 19:13:27 +05:30
See the [upgrade documentation](../../update/index.md#checking-for-background-migrations-before-upgrading)
2020-04-22 19:07:51 +05:30
for how to check that migrations are complete when upgrading GitLab.
To check the status of specific migrations, you can use the following Rake task:
2019-09-30 21:07:59 +05:30
2020-03-13 15:44:24 +05:30
```shell
2019-09-30 21:07:59 +05:30
sudo gitlab-rake db:migrate:status
```
2022-11-25 23:54:43 +05:30
To check the [tracking database on a Geo secondary site](../geo/setup/external_database.md#configure-the-tracking-database), you can use the following Rake task:
```shell
sudo gitlab-rake db:migrate:status:geo
```
2021-02-22 17:27:13 +05:30
This outputs a table with a `Status` of `up` or `down` for
2019-09-30 21:07:59 +05:30
each Migration ID.
2020-03-13 15:44:24 +05:30
```shell
2019-09-30 21:07:59 +05:30
database: gitlabhq_production
Status Migration ID Migration Name
--------------------------------------------------
up migration_id migration_name
2019-12-04 20:38:33 +05:30
```
2020-01-01 13:55:28 +05:30
2020-06-23 00:09:42 +05:30
## Run incomplete database migrations
2021-02-22 17:27:13 +05:30
Database migrations can be stuck in an incomplete state, with a `down`
2020-06-23 00:09:42 +05:30
status in the output of the `sudo gitlab-rake db:migrate:status` command.
2022-10-11 01:57:18 +05:30
1. To complete these migrations, use the following Rake task:
2020-06-23 00:09:42 +05:30
2022-10-11 01:57:18 +05:30
```shell
sudo gitlab-rake db:migrate
```
1. After the command completes, run `sudo gitlab-rake db:migrate:status` to check if all migrations are completed (have an `up` status).
1. Hot reload `puma` and `sidekiq` services:
2020-06-23 00:09:42 +05:30
2022-10-11 01:57:18 +05:30
```shell
sudo gitlab-ctl hup puma
sudo gitlab-ctl restart sidekiq
```
2020-06-23 00:09:42 +05:30
2021-03-11 19:13:27 +05:30
## Rebuild database indexes
WARNING:
2021-09-30 23:02:18 +05:30
This is an experimental feature that isn't enabled by default. It requires PostgreSQL 12 or later.
2021-03-11 19:13:27 +05:30
Database indexes can be rebuilt regularly to reclaim space and maintain healthy levels of index bloat over time.
2021-11-18 22:05:49 +05:30
To rebuild the two indexes with the highest estimated bloat, use the following Rake task:
2021-03-11 19:13:27 +05:30
```shell
sudo gitlab-rake gitlab:db:reindex
```
2021-11-18 22:05:49 +05:30
To target a specific index, use the following Rake task:
2021-03-11 19:13:27 +05:30
```shell
sudo gitlab-rake gitlab:db:reindex['public.a_specific_index']
```
The following index types are not supported:
1. Indexes used for constraint exclusion
1. Partitioned indexes
1. Expression indexes
2021-11-18 22:05:49 +05:30
Optionally, this Rake task sends annotations to a Grafana (4.6 or later) endpoint. Use the following custom environment variables to enable annotations:
2021-03-11 19:13:27 +05:30
1. `GRAFANA_API_URL` - Grafana's base URL, for example `http://some-host:3000`.
1. `GRAFANA_API_KEY` - Grafana API key with at least `Editor role`.
You can also [enable reindexing as a regular cron job](https://docs.gitlab.com/omnibus/settings/database.html#automatic-database-reindexing).
2020-01-01 13:55:28 +05:30
## Import common metrics
Sometimes you may need to re-import the common metrics that power the Metrics dashboards.
2020-07-28 23:09:34 +05:30
This could be as a result of [updating existing metrics](../../development/prometheus_metrics.md#update-existing-metrics), or as a [troubleshooting measure](../../operations/metrics/dashboards/index.md#troubleshooting).
2020-01-01 13:55:28 +05:30
To re-import the metrics you can run:
2020-03-13 15:44:24 +05:30
```shell
2020-01-01 13:55:28 +05:30
sudo gitlab-rake metrics:setup_common_metrics
```