4.2 KiB
| stage | group | info |
|---|---|---|
| Plan | Project Management | To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments |
Real-Time Features
This guide contains instructions on how to safely roll out new real-time features.
Real-time features are implemented using GraphQL Subscriptions. Developer documentation is available.
WebSockets are a relatively new technology at GitLab, and supporting them at scale introduces some challenges. For that reason, new features should be rolled out using the instructions below.
Reuse an existing WebSocket connection
Features reusing an existing connection incur minimal risk. Feature flag rollout is recommended in order to give more control to self-hosting customers. However, it is not necessary to roll out in percentages, or to estimate new connections for GitLab.com.
Introduce a new WebSocket connection
Any change that introduces a WebSocket connection to part of the GitLab application incurs some scalability risk, both to nodes responsible for maintaining open connections and on downstream services; such as Redis and the primary database.
Estimate peak connections
The first real-time feature to be fully enabled on GitLab.com was real-time assignees. By comparing peak throughput to the issue page against peak simultaneous WebSocket connections it is possible to crudely estimate that each 1 request per second adds approximately 4200 WebSocket connections.
To understand the impact a new feature might have, sum the peak throughput (RPS)
to the pages it originates from (n) and apply the formula:
(n * 4200) / peak_active_connections
Current active connections are visible on this Grafana chart.
This calculation is crude, and should be revised as new features are deployed. It yields a rough estimate of the capacity that must be supported, as a proportion of existing capacity.
Graduated roll-out
New capacity may need to be provisioned to support your changes, depending on current saturation and the proportion of new connections required. While Kubernetes makes this relatively easy in most cases, there remains a risk to downstream services.
To mitigate this, ensure that the code establishing the new WebSocket connection
is feature flagged and defaulted to off. A careful, percentage-based roll-out
of the feature flag ensures that effects can be observed on the
WebSocket dashboard
- Create a feature flag roll-out issue.
- Add the estimated new connections required under the What are we expecting to happen section.
- Copy in a member of the Plan and Scalability teams to estimate a percentage-based roll-out plan.
Backward compatibility
For the duration of the feature flag roll-out and indefinitely thereafter, real-time features must be backward-compatible, or at least degrade gracefully. Not all customers have Action Cable enabled, and further work needs to be done before Action Cable can be enabled by default.
Making real-time a requirement represents a breaking change, so the next opportunity to do this is version 15.0.
Enable Real-Time by default
Mounting the Action Cable library adds minimal memory footprint. However, serving WebSocket requests introduces additional memory requirements. For this reason, enabling Action Cable by default requires additional work; perhaps to reduce overall memory usage, including a known issue with Workhorse, but at least to revise Reference Architectures.
Real-time infrastructure on GitLab.com
On GitLab.com, WebSocket connections are served from dedicated infrastructure, entirely separate from the regular Web fleet and deployed with Kubernetes. This limits risk to nodes handling requests but not to shared services. For more information on the WebSockets Kubernetes deployment see this epic.