debian-mirror-gitlab/doc/administration/operations/extra_sidekiq_processes.md

464 lines
17 KiB
Markdown
Raw Normal View History

2020-05-24 23:13:21 +05:30
# Run multiple Sidekiq processes **(CORE ONLY)**
2019-09-30 21:07:59 +05:30
2020-04-22 19:07:51 +05:30
GitLab allows you to start multiple Sidekiq processes.
These processes can be used to consume a dedicated set
2019-07-31 22:56:46 +05:30
of queues. This can be used to ensure certain queues always have dedicated
workers, no matter the number of jobs that need to be processed.
2020-05-24 23:13:21 +05:30
NOTE: **Note:**
The information in this page applies only to Omnibus GitLab.
2019-09-30 21:07:59 +05:30
## Available Sidekiq queues
2019-07-31 22:56:46 +05:30
2019-09-30 21:07:59 +05:30
For a list of the existing Sidekiq queues, check the following files:
2019-07-31 22:56:46 +05:30
2019-12-04 20:38:33 +05:30
- [Queues for both GitLab Community and Enterprise Editions](https://gitlab.com/gitlab-org/gitlab/blob/master/app/workers/all_queues.yml)
- [Queues for GitLab Enterprise Editions only](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/app/workers/all_queues.yml)
2019-07-31 22:56:46 +05:30
2020-04-22 19:07:51 +05:30
Each entry in the above files represents a queue on which Sidekiq processes
2019-09-30 21:07:59 +05:30
can be started.
2019-07-31 22:56:46 +05:30
2020-05-24 23:13:21 +05:30
## Start multiple processes
2019-07-31 22:56:46 +05:30
2020-05-24 23:13:21 +05:30
> - [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4006) in GitLab 12.10, starting multiple processes with Sidekiq cluster.
> - [Sidekiq cluster moved](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/181) to GitLab [Core](https://about.gitlab.com/pricing/#self-managed) in GitLab 12.10.
> - [Sidekiq cluster became default](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4140) in GitLab 13.0.
2019-07-31 22:56:46 +05:30
2020-05-24 23:13:21 +05:30
To start multiple processes:
2019-07-31 22:56:46 +05:30
2020-05-24 23:13:21 +05:30
1. Using the `sidekiq['queue_groups']` array setting, specify how many processes to
create using `sidekiq-cluster` and which queue they should handle.
Each item in the array equates to one additional Sidekiq
2019-09-30 21:07:59 +05:30
process, and values in each item determine the queues it works on.
2019-07-31 22:56:46 +05:30
2020-05-24 23:13:21 +05:30
For example, the following setting creates three Sidekiq processes, one to run on
`elastic_indexer`, one to run on `mailers`, and one process running all on queues:
2019-07-31 22:56:46 +05:30
2019-09-30 21:07:59 +05:30
```ruby
2020-05-24 23:13:21 +05:30
sidekiq['queue_groups'] = [
2019-09-30 21:07:59 +05:30
"elastic_indexer",
2020-05-24 23:13:21 +05:30
"mailers",
"*"
2019-09-30 21:07:59 +05:30
]
```
To have an additional Sidekiq process handle multiple queues, add multiple
queue names to its item delimited by commas. For example:
```ruby
2020-05-24 23:13:21 +05:30
sidekiq['queue_groups'] = [
2019-09-30 21:07:59 +05:30
"elastic_indexer, elastic_commit_indexer",
2020-05-24 23:13:21 +05:30
"mailers",
"*"
2019-09-30 21:07:59 +05:30
]
```
2020-04-08 14:13:33 +05:30
[In GitLab 12.9](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26594) and
later, the special queue name `*` means all queues. This starts two
processes, each handling all queues:
```ruby
2020-05-24 23:13:21 +05:30
sidekiq['queue_groups'] = [
2020-04-08 14:13:33 +05:30
"*",
"*"
]
```
`*` cannot be combined with concrete queue names - `*, mailers` will
just handle the `mailers` queue.
2020-05-24 23:13:21 +05:30
When `sidekiq-cluster` is only running on a single node, make sure that at least
one process is running on all queues using `*`. This means a process will
automatically pick up jobs in queues created in the future.
If `sidekiq-cluster` is running on more than one node, you can also use
[`--negate`](#negate-settings) and list all the queues that are already being
processed.
2019-09-30 21:07:59 +05:30
1. Save the file and reconfigure GitLab for the changes to take effect:
2020-03-13 15:44:24 +05:30
```shell
2019-09-30 21:07:59 +05:30
sudo gitlab-ctl reconfigure
```
2020-05-24 23:13:21 +05:30
After the extra Sidekiq processes are added, navigate to
**{admin}** **Admin Area > Monitoring > Background Jobs** (`/admin/background_jobs`) in GitLab.
2019-07-31 22:56:46 +05:30
2020-04-22 19:07:51 +05:30
![Multiple Sidekiq processes](img/sidekiq-cluster.png)
2019-07-31 22:56:46 +05:30
2020-05-24 23:13:21 +05:30
## Negate settings
2019-07-31 22:56:46 +05:30
2019-09-30 21:07:59 +05:30
To have the additional Sidekiq processes work on every queue **except** the ones
2019-07-31 22:56:46 +05:30
you list:
2020-05-24 23:13:21 +05:30
1. After you follow the steps for [starting extra processes](#start-multiple-processes),
2019-09-30 21:07:59 +05:30
edit `/etc/gitlab/gitlab.rb` and add:
```ruby
2020-05-24 23:13:21 +05:30
sidekiq['negate'] = true
2019-09-30 21:07:59 +05:30
```
1. Save the file and reconfigure GitLab for the changes to take effect:
2020-03-13 15:44:24 +05:30
```shell
2019-09-30 21:07:59 +05:30
sudo gitlab-ctl reconfigure
```
2020-03-13 15:44:24 +05:30
## Queue selector (experimental)
2020-06-23 00:09:42 +05:30
> [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/45) in [GitLab Starter](https://about.gitlab.com/pricing/) 12.8.
2020-03-13 15:44:24 +05:30
CAUTION: **Caution:**
As this is marked as **experimental**, it is subject to change at any
time, including **breaking backwards compatibility**. This is so that we
can react to changes we need for our GitLab.com deployment. We have a
tracking issue open to [remove the experimental
2020-06-23 00:09:42 +05:30
designation](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/147)
2020-03-13 15:44:24 +05:30
from this feature; please comment there if you are interested in using
this in your own deployment.
In addition to selecting queues by name, as above, the
`experimental_queue_selector` option allows queue groups to be selected
in a more general way using the following components:
- Attributes that can be selected.
- Operators used to construct a query.
### Available attributes
2020-06-23 00:09:42 +05:30
- [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/261) in GitLab 13.1, `tags`.
2020-03-13 15:44:24 +05:30
From the [list of all available
attributes](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/all_queues.yml),
`experimental_queue_selector` allows selecting of queues by the
following attributes:
- `feature_category` - the [GitLab feature
category](https://about.gitlab.com/direction/maturity/#category-maturity) the
queue belongs to. For example, the `merge` queue belongs to the
`source_code_management` category.
- `has_external_dependencies` - whether or not the queue connects to external
services. For example, all importers have this set to `true`.
2020-04-08 14:13:33 +05:30
- `urgency` - how important it is that this queue's jobs run
quickly. Can be `high`, `low`, or `throttled`. For example, the
`authorized_projects` queue is used to refresh user permissions, and
is high urgency.
2020-03-13 15:44:24 +05:30
- `name` - the queue name. The other attributes are typically more useful as
they are more general, but this is available in case a particular queue needs
to be selected.
2020-06-23 00:09:42 +05:30
- `resource_boundary` - if the queue is bound by `cpu`, `memory`, or
2020-03-13 15:44:24 +05:30
`unknown`. For example, the `project_export` queue is memory bound as it has
to load data in memory before saving it for export.
2020-06-23 00:09:42 +05:30
- `tags` - short-lived annotations for queues. These are expected to frequently
change from release to release, and may be removed entirely.
2020-03-13 15:44:24 +05:30
2020-04-08 14:13:33 +05:30
`has_external_dependencies` is a boolean attribute: only the exact
string `true` is considered true, and everything else is considered
false.
2020-03-13 15:44:24 +05:30
2020-06-23 00:09:42 +05:30
`tags` is a set, which means that `=` checks for intersecting sets, and
`!=` checks for disjoint sets. For example, `tags=a,b` selects queues
that have tags `a`, `b`, or both. `tags!=a,b` selects queues that have
neither of those tags.
2020-03-13 15:44:24 +05:30
### Available operators
`experimental_queue_selector` supports the following operators, listed
from highest to lowest precedence:
- `|` - the logical OR operator. For example, `query_a|query_b` (where `query_a`
and `query_b` are queries made up of the other operators here) will include
queues that match either query.
- `&` - the logical AND operator. For example, `query_a&query_b` (where
`query_a` and `query_b` are queries made up of the other operators here) will
only include queues that match both queries.
- `!=` - the NOT IN operator. For example, `feature_category!=issue_tracking`
excludes all queues from the `issue_tracking` feature category.
- `=` - the IN operator. For example, `resource_boundary=cpu` includes all
queues that are CPU bound.
- `,` - the concatenate set operator. For example,
`feature_category=continuous_integration,pages` includes all queues from
either the `continuous_integration` category or the `pages` category. This
example is also possible using the OR operator, but allows greater brevity, as
well as being lower precedence.
The operator precedence for this syntax is fixed: it's not possible to make AND
have higher precedence than OR.
2020-04-08 14:13:33 +05:30
[In GitLab 12.9](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26594) and
later, as with the standard queue group syntax above, a single `*` as the
entire queue group selects all queues.
2020-03-13 15:44:24 +05:30
### Example queries
In `/etc/gitlab/gitlab.rb`:
```ruby
2020-05-24 23:13:21 +05:30
sidekiq['enable'] = true
sidekiq['experimental_queue_selector'] = true
sidekiq['queue_groups'] = [
2020-04-08 14:13:33 +05:30
# Run all non-CPU-bound queues that are high urgency
'resource_boundary!=cpu&urgency=high',
# Run all continuous integration and pages queues that are not high urgency
'feature_category=continuous_integration,pages&urgency!=high',
# Run all queues
'*'
2020-03-13 15:44:24 +05:30
]
```
2020-05-24 23:13:21 +05:30
### Disable Sidekiq cluster
2020-04-22 19:07:51 +05:30
CAUTION: **Warning:**
2020-05-24 23:13:21 +05:30
Sidekiq cluster is [scheduled](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/240)
to be the only way to start Sidekiq in GitLab 14.0.
2020-04-22 19:07:51 +05:30
2020-05-24 23:13:21 +05:30
By default, the Sidekiq service will run `sidekiq-cluster`. To disable this behavior,
add the following to the Sidekiq configuration:
2020-04-22 19:07:51 +05:30
```ruby
sidekiq['enable'] = true
2020-05-24 23:13:21 +05:30
sidekiq['cluster'] = false
2020-04-22 19:07:51 +05:30
```
2020-05-24 23:13:21 +05:30
All of the aforementioned configuration options for `sidekiq`
are available. By default, they will be configured as follows:
2020-04-22 19:07:51 +05:30
```ruby
sidekiq['experimental_queue_selector'] = false
sidekiq['interval'] = nil
2020-05-24 23:13:21 +05:30
sidekiq['max_concurrency'] = 50
2020-04-22 19:07:51 +05:30
sidekiq['min_concurrency'] = nil
sidekiq['negate'] = false
sidekiq['queue_groups'] = ['*']
2020-05-24 23:13:21 +05:30
sidekiq['shutdown_timeout'] = 25
2020-04-22 19:07:51 +05:30
```
`sidekiq_cluster` must be disabled if you decide to configure the
cluster as above.
When disabling `sidekiq_cluster`, you must copy your configuration for
`sidekiq_cluster`over to `sidekiq`. Anything configured for
`sidekiq_cluster` will be overridden by the options for `sidekiq` when
setting `sidekiq['cluster'] = true`.
When using this feature, the service called `sidekiq` will now be
running `sidekiq-cluster`.
2020-05-24 23:13:21 +05:30
The [concurrency](#manage-concurrency) and other options configured
2020-04-22 19:07:51 +05:30
for Sidekiq will be respected.
By default, logs for `sidekiq-cluster` go to `/var/log/gitlab/sidekiq`
like regular Sidekiq logs.
2019-09-30 21:07:59 +05:30
## Ignore all GitHub import queues
When [importing from GitHub](../../user/project/import/github.md), Sidekiq might
use all of its resources to perform those operations. To set up a separate
`sidekiq-cluster` process to ignore all GitHub import-related queues:
2019-07-31 22:56:46 +05:30
1. Edit `/etc/gitlab/gitlab.rb` and add:
```ruby
2020-05-24 23:13:21 +05:30
sidekiq['enable'] = true
sidekiq['negate'] = true
sidekiq['queue_groups'] = [
2019-09-30 21:07:59 +05:30
"github_import_advance_stage",
"github_importer:github_import_import_diff_note",
"github_importer:github_import_import_issue",
"github_importer:github_import_import_note",
"github_importer:github_import_import_lfs_object",
"github_importer:github_import_import_pull_request",
"github_importer:github_import_refresh_import_jid",
"github_importer:github_import_stage_finish_import",
"github_importer:github_import_stage_import_base_data",
"github_importer:github_import_stage_import_issues_and_diff_notes",
"github_importer:github_import_stage_import_notes",
"github_importer:github_import_stage_import_lfs_objects",
"github_importer:github_import_stage_import_pull_requests",
"github_importer:github_import_stage_import_repository"
]
2019-07-31 22:56:46 +05:30
```
2019-09-30 21:07:59 +05:30
1. Save the file and reconfigure GitLab for the changes to take effect:
2019-07-31 22:56:46 +05:30
2020-03-13 15:44:24 +05:30
```shell
2019-09-30 21:07:59 +05:30
sudo gitlab-ctl reconfigure
```
2019-07-31 22:56:46 +05:30
2019-09-30 21:07:59 +05:30
## Number of threads
2020-05-24 23:13:21 +05:30
Each process defined under `sidekiq` starts with a
2019-09-30 21:07:59 +05:30
number of threads that equals the number of queues, plus one spare thread.
For example, a process that handles the `process_commit` and `post_receive`
queues will use three threads in total.
2020-05-24 23:13:21 +05:30
## Manage concurrency
2020-03-13 15:44:24 +05:30
When setting the maximum concurrency, keep in mind this normally should
not exceed the number of CPU cores available. The values in the examples
below are arbitrary and not particular recommendations.
2019-09-30 21:07:59 +05:30
2020-03-13 15:44:24 +05:30
Each thread requires a Redis connection, so adding threads may increase Redis
latency and potentially cause client timeouts. See the [Sidekiq documentation
about Redis](https://github.com/mperham/sidekiq/wiki/Using-Redis) for more
details.
2020-05-24 23:13:21 +05:30
### When running Sidekiq cluster (default)
2020-01-01 13:55:28 +05:30
2020-05-24 23:13:21 +05:30
Running Sidekiq cluster is the default in GitLab 13.0 and later.
2020-01-01 13:55:28 +05:30
1. Edit `/etc/gitlab/gitlab.rb` and add:
```ruby
2020-05-24 23:13:21 +05:30
sidekiq['min_concurrency'] = 15
sidekiq['max_concurrency'] = 25
2019-07-31 22:56:46 +05:30
```
2019-09-30 21:07:59 +05:30
1. Save the file and reconfigure GitLab for the changes to take effect:
2019-07-31 22:56:46 +05:30
2020-03-13 15:44:24 +05:30
```shell
2019-09-30 21:07:59 +05:30
sudo gitlab-ctl reconfigure
```
2019-07-31 22:56:46 +05:30
2020-03-13 15:44:24 +05:30
`min_concurrency` and `max_concurrency` are independent; one can be set without
the other. Setting `min_concurrency` to 0 will disable the limit.
2019-09-30 21:07:59 +05:30
2020-03-13 15:44:24 +05:30
For each queue group, let N be one more than the number of queues. The
concurrency factor will be set to:
1. `N`, if it's between `min_concurrency` and `max_concurrency`.
1. `max_concurrency`, if `N` exceeds this value.
1. `min_concurrency`, if `N` is less than this value.
If `min_concurrency` is equal to `max_concurrency`, then this value will be used
regardless of the number of queues.
When `min_concurrency` is greater than `max_concurrency`, it is treated as
being equal to `max_concurrency`.
2019-09-30 21:07:59 +05:30
2020-05-24 23:13:21 +05:30
### When running a single Sidekiq process
Running a single Sidekiq process is the default in GitLab 12.10 and earlier.
CAUTION: **Warning:**
Running Sidekiq directly is scheduled to be removed in GitLab
[14.0](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/240).
1. Edit `/etc/gitlab/gitlab.rb` and add:
```ruby
sidekiq['cluster'] = false
sidekiq['concurrency'] = 25
```
1. Save the file and reconfigure GitLab for the changes to take effect:
```shell
sudo gitlab-ctl reconfigure
```
This will set the concurrency (number of threads) for the Sidekiq process.
## Modify the check interval
2019-07-31 22:56:46 +05:30
To modify the check interval for the additional Sidekiq processes:
1. Edit `/etc/gitlab/gitlab.rb` and add:
```ruby
2020-05-24 23:13:21 +05:30
sidekiq['interval'] = 5
2019-07-31 22:56:46 +05:30
```
1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
This tells the additional processes how often to check for enqueued jobs.
2020-05-24 23:13:21 +05:30
## Troubleshoot using the CLI
2019-07-31 22:56:46 +05:30
2019-09-30 21:07:59 +05:30
CAUTION: **Warning:**
It's recommended to use `/etc/gitlab/gitlab.rb` to configure the Sidekiq processes.
If you experience a problem, you should contact GitLab support. Use the command
line at your own risk.
For debugging purposes, you can start extra Sidekiq processes by using the command
2020-04-08 14:13:33 +05:30
`/opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster`. This command
2019-07-31 22:56:46 +05:30
takes arguments using the following syntax:
2020-03-13 15:44:24 +05:30
```shell
2020-04-08 14:13:33 +05:30
/opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster [QUEUE,QUEUE,...] [QUEUE, ...]
2019-07-31 22:56:46 +05:30
```
Each separate argument denotes a group of queues that have to be processed by a
Sidekiq process. Multiple queues can be processed by the same process by
separating them with a comma instead of a space.
Instead of a queue, a queue namespace can also be provided, to have the process
automatically listen on all queues in that namespace without needing to
explicitly list all the queue names. For more information about queue namespaces,
see the relevant section in the
[Sidekiq style guide](../../development/sidekiq_style_guide.md#queue-namespaces).
For example, say you want to start 2 extra processes: one to process the
2019-09-30 21:07:59 +05:30
`process_commit` queue, and one to process the `post_receive` queue. This can be
2019-07-31 22:56:46 +05:30
done as follows:
2020-03-13 15:44:24 +05:30
```shell
2020-04-08 14:13:33 +05:30
/opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster process_commit post_receive
2019-07-31 22:56:46 +05:30
```
2019-09-30 21:07:59 +05:30
If you instead want to start one process processing both queues, you'd use the
2019-07-31 22:56:46 +05:30
following syntax:
2020-03-13 15:44:24 +05:30
```shell
2020-04-08 14:13:33 +05:30
/opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster process_commit,post_receive
2019-07-31 22:56:46 +05:30
```
2019-09-30 21:07:59 +05:30
If you want to have one Sidekiq process dealing with the `process_commit` and
`post_receive` queues, and one process to process the `gitlab_shell` queue,
2019-07-31 22:56:46 +05:30
you'd use the following:
2020-03-13 15:44:24 +05:30
```shell
2020-04-08 14:13:33 +05:30
/opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster process_commit,post_receive gitlab_shell
2019-07-31 22:56:46 +05:30
```
2020-05-24 23:13:21 +05:30
### Monitor the `sidekiq-cluster` command
2019-07-31 22:56:46 +05:30
The `sidekiq-cluster` command will not terminate once it has started the desired
amount of Sidekiq processes. Instead, the process will continue running and
forward any signals to the child processes. This makes it easy to stop all
Sidekiq processes as you simply send a signal to the `sidekiq-cluster` process,
instead of having to send it to the individual processes.
If the `sidekiq-cluster` process crashes or receives a `SIGKILL`, the child
processes will terminate themselves after a few seconds. This ensures you don't
end up with zombie Sidekiq processes.
All of this makes monitoring the processes fairly easy. Simply hook up
2020-05-24 23:13:21 +05:30
`sidekiq-cluster` to your supervisor of choice (for example, runit) and you're good to
2019-07-31 22:56:46 +05:30
go.
If a child process died the `sidekiq-cluster` command will signal all remaining
process to terminate, then terminate itself. This removes the need for
`sidekiq-cluster` to re-implement complex process monitoring/restarting code.
Instead you should make sure your supervisor restarts the `sidekiq-cluster`
process whenever necessary.
### PID files
The `sidekiq-cluster` command can store its PID in a file. By default no PID
file is written, but this can be changed by passing the `--pidfile` option to
`sidekiq-cluster`. For example:
2020-03-13 15:44:24 +05:30
```shell
2020-04-08 14:13:33 +05:30
/opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster --pidfile /var/run/gitlab/sidekiq_cluster.pid process_commit
2019-07-31 22:56:46 +05:30
```
Keep in mind that the PID file will contain the PID of the `sidekiq-cluster`
command and not the PID(s) of the started Sidekiq processes.
### Environment
The Rails environment can be set by passing the `--environment` flag to the
`sidekiq-cluster` command, or by setting `RAILS_ENV` to a non-empty value. The
2019-09-30 21:07:59 +05:30
default value can be found in `/opt/gitlab/etc/gitlab-rails/env/RAILS_ENV`.