debian-mirror-gitlab/doc/development/redis.md

206 lines
9.1 KiB
Markdown
Raw Normal View History

2021-01-29 00:20:46 +05:30
---
stage: none
group: unassigned
2021-02-22 17:27:13 +05:30
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
2021-01-29 00:20:46 +05:30
---
2020-03-13 15:44:24 +05:30
# Redis guidelines
2020-11-24 15:15:51 +05:30
GitLab uses [Redis](https://redis.io) for the following distinct purposes:
2020-03-13 15:44:24 +05:30
2020-11-24 15:15:51 +05:30
- Caching (mostly via `Rails.cache`).
2020-03-13 15:44:24 +05:30
- As a job processing queue with [Sidekiq](sidekiq_style_guide.md).
- To manage the shared application state.
2020-11-24 15:15:51 +05:30
- As a Pub/Sub queue backend for ActionCable.
In most environments (including the GDK), all of these point to the same
Redis instance.
On GitLab.com, we use [separate Redis
instances](../administration/redis/replication_and_failover.md#running-multiple-redis-clusters).
2021-09-04 01:27:46 +05:30
See the [Redis SRE guide](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/redis/redis-survival-guide-for-sres.md)
for more details on our setup.
2020-03-13 15:44:24 +05:30
Every application process is configured to use the same Redis servers, so they
can be used for inter-process communication in cases where [PostgreSQL](sql.md)
2020-04-08 14:13:33 +05:30
is less appropriate. For example, transient state or data that is written much
2020-03-13 15:44:24 +05:30
more often than it is read.
If [Geo](geo.md) is enabled, each Geo node gets its own, independent Redis
database.
## Key naming
Redis is a flat namespace with no hierarchy, which means we must pay attention
to key names to avoid collisions. Typically we use colon-separated elements to
2020-06-23 00:09:42 +05:30
provide a semblance of structure at application level. An example might be
2020-03-13 15:44:24 +05:30
`projects:1:somekey`.
2020-11-24 15:15:51 +05:30
Although we split our Redis usage by purpose into distinct categories, and
those may map to separate Redis servers in a Highly Available
configuration like GitLab.com, the default Omnibus and GDK setups share
a single Redis server. This means that keys should **always** be
globally unique across all categories.
2020-03-13 15:44:24 +05:30
It is usually better to use immutable identifiers - project ID rather than
2021-02-22 17:27:13 +05:30
full path, for instance - in Redis key names. If full path is used, the key
stops being consulted if the project is renamed. If the contents of the key are
invalidated by a name change, it is better to include a hook that expires
2020-03-13 15:44:24 +05:30
the entry, instead of relying on the key changing.
2020-07-28 23:09:34 +05:30
### Multi-key commands
2020-03-13 15:44:24 +05:30
We don't use [Redis Cluster](https://redis.io/topics/cluster-tutorial) at the
2020-06-23 00:09:42 +05:30
moment, but may wish to in the future: [#118820](https://gitlab.com/gitlab-org/gitlab/-/issues/118820).
2020-03-13 15:44:24 +05:30
This imposes an additional constraint on naming: where GitLab is performing
operations that require several keys to be held on the same Redis server - for
instance, diffing two sets held in Redis - the keys should ensure that by
2020-10-24 23:57:45 +05:30
enclosing the changeable parts in curly braces.
For example:
```plaintext
project:{1}:set_a
project:{1}:set_b
project:{2}:set_c
```
`set_a` and `set_b` are guaranteed to be held on the same Redis server, while `set_c` is not.
2020-07-28 23:09:34 +05:30
Currently, we validate this in the development and test environments
with the [`RedisClusterValidator`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/instrumentation/redis_cluster_validator.rb),
which is enabled for the `cache` and `shared_state`
[Redis instances](https://docs.gitlab.com/omnibus/settings/redis.html#running-with-multiple-redis-instances)..
2020-11-24 15:15:51 +05:30
## Redis in structured logging
2021-03-11 19:13:27 +05:30
For GitLab Team Members: There are <i class="fa fa-youtube-play youtube" aria-hidden="true"></i>
[basic](https://www.youtube.com/watch?v=Uhdj19Dc6vU) and
<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [advanced](https://youtu.be/jw1Wv2IJxzs)
videos that show how you can work with the Redis
2021-01-03 14:25:43 +05:30
structured logging fields on GitLab.com.
2020-11-24 15:15:51 +05:30
Our [structured logging](logging.md#use-structured-json-logging) for web
requests and Sidekiq jobs contains fields for the duration, call count,
bytes written, and bytes read per Redis instance, along with a total for
all Redis instances. For a particular request, this might look like:
| Field | Value |
| --- | --- |
| `json.queue_duration_s` | 0.01 |
| `json.redis_cache_calls` | 1 |
| `json.redis_cache_duration_s` | 0 |
| `json.redis_cache_read_bytes` | 109 |
| `json.redis_cache_write_bytes` | 49 |
| `json.redis_calls` | 2 |
| `json.redis_duration_s` | 0.001 |
| `json.redis_read_bytes` | 111 |
| `json.redis_shared_state_calls` | 1 |
| `json.redis_shared_state_duration_s` | 0 |
| `json.redis_shared_state_read_bytes` | 2 |
| `json.redis_shared_state_write_bytes` | 206 |
| `json.redis_write_bytes` | 255 |
As all of these fields are indexed, it is then straightforward to
investigate Redis usage in production. For instance, to find the
requests that read the most data from the cache, we can just sort by
`redis_cache_read_bytes` in descending order.
### The slow log
2021-02-22 17:27:13 +05:30
NOTE:
2021-01-03 14:25:43 +05:30
There is a [video showing how to see the slow log](https://youtu.be/BBI68QuYRH8) (GitLab internal)
on GitLab.com
2020-11-24 15:15:51 +05:30
On GitLab.com, entries from the [Redis
slow log](https://redis.io/commands/slowlog) are available in the
`pubsub-redis-inf-gprd*` index with the [`redis.slowlog`
2021-01-03 14:25:43 +05:30
tag](https://log.gprd.gitlab.net/app/kibana#/discover?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-1d,to:now))&_a=(columns:!(json.type,json.command,json.exec_time_s),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:AWSQX_Vf93rHTYrsexmk,key:json.tag,negate:!f,params:(query:redis.slowlog),type:phrase),query:(match:(json.tag:(query:redis.slowlog,type:phrase))))),index:AWSQX_Vf93rHTYrsexmk)).
2020-11-24 15:15:51 +05:30
This shows commands that have taken a long time and may be a performance
concern.
The
2021-04-17 20:07:23 +05:30
[`fluent-plugin-redis-slowlog`](https://gitlab.com/gitlab-org/fluent-plugin-redis-slowlog)
project is responsible for taking the `slowlog` entries from Redis and
passing to Fluentd (and ultimately Elasticsearch).
2020-11-24 15:15:51 +05:30
## Analyzing the entire keyspace
2021-04-17 20:07:23 +05:30
The [Redis Keyspace Analyzer](https://gitlab.com/gitlab-com/gl-infra/redis-keyspace-analyzer)
2020-11-24 15:15:51 +05:30
project contains tools for dumping the full key list and memory usage of a Redis
2021-01-03 14:25:43 +05:30
instance, and then analyzing those lists while eliminating potentially sensitive
2020-11-24 15:15:51 +05:30
data from the results. It can be used to find the most frequent key patterns, or
those that use the most memory.
Currently this is not run automatically for the GitLab.com Redis instances, but
is run manually on an as-needed basis.
## Utility classes
We have some extra classes to help with specific use cases. These are
mostly for fine-grained control of Redis usage, so they wouldn't be used
in combination with the `Rails.cache` wrapper: we'd either use
`Rails.cache` or these classes and literal Redis commands.
`Rails.cache` or these classes and literal Redis commands. We prefer
using `Rails.cache` so we can reap the benefits of future optimizations
done to Rails. It is worth noting that Ruby objects are
[marshalled](https://github.com/rails/rails/blob/v6.0.3.1/activesupport/lib/active_support/cache/redis_cache_store.rb#L447)
when written to Redis, so we need to pay attention to not to store huge
objects, or untrusted user input.
Typically we would only use these classes when at least one of the
following is true:
1. We want to manipulate data on a non-cache Redis instance.
1. `Rails.cache` does not support the operations we want to perform.
### `Gitlab::Redis::{Cache,SharedState,Queues}`
These classes wrap the Redis instances (using
2021-09-04 01:27:46 +05:30
[`Gitlab::Redis::Wrapper`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/redis/wrapper.rb))
2020-11-24 15:15:51 +05:30
to make it convenient to work with them directly. The typical use is to
call `.with` on the class, which takes a block that yields the Redis
connection. For example:
```ruby
# Get the value of `key` from the shared state (persistent) Redis
Gitlab::Redis::SharedState.with { |redis| redis.get(key) }
# Check if `value` is a member of the set `key`
Gitlab::Redis::Cache.with { |redis| redis.sismember(key, value) }
```
### `Gitlab::Redis::Boolean`
In Redis, every value is a string.
2021-09-04 01:27:46 +05:30
[`Gitlab::Redis::Boolean`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/redis/boolean.rb)
2020-11-24 15:15:51 +05:30
makes sure that booleans are encoded and decoded consistently.
### `Gitlab::Redis::HLL`
The Redis [`PFCOUNT`](https://redis.io/commands/pfcount),
[`PFADD`](https://redis.io/commands/pfadd), and
[`PFMERGE`](https://redis.io/commands/pfmergge) commands operate on
HyperLogLogs, a data structure that allows estimating the number of unique
elements with low memory usage. (In addition to the `PFCOUNT` documentation,
2021-03-11 19:13:27 +05:30
Thoughtbot's article on [HyperLogLogs in Redis](https://thoughtbot.com/blog/hyperloglogs-in-redis)
provides a good background here.)
2020-11-24 15:15:51 +05:30
2021-09-04 01:27:46 +05:30
[`Gitlab::Redis::HLL`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/redis/hll.rb)
2020-11-24 15:15:51 +05:30
provides a convenient interface for adding and counting values in HyperLogLogs.
### `Gitlab::SetCache`
For cases where we need to efficiently check the whether an item is in a group
of items, we can use a Redis set.
2021-09-04 01:27:46 +05:30
[`Gitlab::SetCache`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/set_cache.rb)
2021-02-22 17:27:13 +05:30
provides an `#include?` method that uses the
2020-11-24 15:15:51 +05:30
[`SISMEMBER`](https://redis.io/commands/sismember) command, as well as `#read`
to fetch all entries in the set.
This is used by the
2021-09-04 01:27:46 +05:30
[`RepositorySetCache`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/repository_set_cache.rb)
2020-11-24 15:15:51 +05:30
to provide a convenient way to use sets to cache repository data like branch
names.