debian-mirror-gitlab/doc/administration/geo/replication/geo_validation_tests.md

stage	group	info	type
Enablement	Geo	To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers	howto

Geo validation tests

The Geo team performs manual testing and validation on common deployment configurations to ensure that Geo works when upgrading between minor GitLab versions and major PostgreSQL database versions.

This section contains a journal of recent validation tests and links to the relevant issues.

GitLab upgrades

The following are GitLab upgrade validation tests we performed.

July 2020

Upgrade Geo multi-node installation:

• Description: Tested upgrading from GitLab 12.10.12 to 13.0.10 package in a multi-node configuration. As part of the issue to Fix zero-downtime upgrade process/instructions for multi-node Geo deployments, we monitored for downtime using the looping pipeline, HAProxy stats dashboards, and a script to log readiness status on both nodes.
• Outcome: Partial success because we observed downtime during the upgrade of the primary and secondary sites.
• Follow up issues/actions:
  • Investigate why reconfigure and hup cause downtime on multi-node Geo deployments
  • Geo multi-node deployment upgrade: investigate order when upgrading non-deploy nodes

June 2020

Upgrade Geo multi-node installation:

• Description: Tested upgrading from GitLab 12.9.10 to 12.10.12 package in a multi-node configuration. Monitored for downtime using the looping pipeline and HAProxy stats dashboards.
• Outcome: Partial success because we observed downtime during the upgrade of the primary and secondary sites.
• Follow up issues/actions:
  • Fix zero-downtime upgrade process/instructions for multi-node Geo deployments
  • Geo:check Rake task: Exclude AuthorizedKeysCommand check if node not running Puma/Unicorn
  • Update instructions in the next upgrade issue to include monitoring HAProxy dashboards

Upgrade Geo multi-node installation:

• Description: Tested upgrading from GitLab 12.8.1 to 12.9.10 package in a multi-node configuration.
• Outcome: Partial success because we did not run the looping pipeline during the demo to validate zero-downtime.
• Follow up issues:
  • Clarify hup Puma/Unicorn should include deploy node
  • Investigate MR creation failure after upgrade to 12.9.10 Closed as false positive.

February 2020

Upgrade Geo multi-node installation:

• Description: Tested upgrading from GitLab 12.7.5 to the latest GitLab 12.8 package in a multi-node configuration.
• Outcome: Partial success because we did not run the looping pipeline during the demo to monitor downtime.

January 2020

Upgrade Geo multi-node installation:

• Description: Tested upgrading from GitLab 12.6.x to the latest GitLab 12.7 package in a multi-node configuration.
• Outcome: Upgrade test was successful.
• Follow up issues:
  • Investigate Geo end-to-end test failures.
  • Add more logging to Geo end-to-end tests.
  • Excess service restarts during zero-downtime upgrade.

Upgrade Geo multi-node installation:

• Description: Tested upgrading from GitLab 12.5.7 to GitLab 12.6.6 in a multi-node configuration.
• Outcome: Upgrade test was successful.
• Follow up issue: Update documentation for zero-downtime upgrades to ensure deploy node it not in use.

Upgrade Geo multi-node installation:

• Description: Tested upgrading from GitLab 12.4.x to the latest GitLab 12.5 package in a multi-node configuration.
• Outcome: Upgrade test was successful.
• Follow up issues:
  • Investigate why HTTP push spec failed on primary node.
  • Investigate if documentation should be modified to include refresh foreign tables task.

October 2019

Upgrade Geo multi-node installation:

• Description: Tested upgrading from GitLab 12.3.5 to GitLab 12.4.1 in a multi-node configuration.
• Outcome: Upgrade test was successful.

Upgrade Geo multi-node installation:

• Description: Tested upgrading from GitLab 12.2.8 to GitLab 12.3.5.
• Outcome: Upgrade test was successful.

Upgrade Geo multi-node installation:

• Description: Tested upgrading from GitLab 12.1.9 to GitLab 12.2.8.
• Outcome: Partial success due to possible misconfiguration issues.

PostgreSQL upgrades

The following are PostgreSQL upgrade validation tests we performed.

April 2020

PostgreSQL 11 upgrade procedure for Geo installations:

• Description: Prior to making PostgreSQL 11 the default version of PostgreSQL in GitLab 12.10, we tested upgrading to PostgreSQL 11 in Geo deployments on GitLab 12.9.
• Outcome: Partially successful. Issues were discovered in multi-node configurations with a separate tracking database and concerns were raised about allowing automatic upgrades when Geo enabled.
• Follow up issues:
  • replicate-geo-database incorrectly tries to back up repositories.
  • pg-upgrade fails to upgrade a standalone Geo tracking database.
  • revert-pg-upgrade fails to downgrade the PostgreSQL data of a Geo secondary’s standalone tracking database.
  • Timeout error on Geo secondary read-replica near the end of gitlab-ctl pg-upgrade.

Verify Geo installation with PostgreSQL 11:

• Description: Prior to making PostgreSQL 11 the default version of PostgreSQL in GitLab 12.10, we tested fresh installations of GitLab 12.9 with Geo installed with PostgreSQL 11.
• Outcome: Installation test was successful.

September 2019

Test and validate PostgreSQL 10.0 upgrade for Geo:

• Description: With the 12.0 release, GitLab required an upgrade to PostgreSQL 10.0. We tested various upgrade scenarios from GitLab 11.11.5 through to GitLab 12.1.8.
• Outcome: Multiple issues were found when upgrading and addressed in follow-up issues.
• Follow up issues:
  • gitlab-ctl reconfigure fails on Redis node in multi-node Geo setup.
  • Geo multi-node upgrade from 12.0.9 to 12.1.9 does not upgrade PostgreSQL.
  • Refresh foreign tables fails on app server in multi-node setup after upgrade to 12.1.9.