debian-mirror-gitlab/doc/install/database_mysql.md

299 lines
14 KiB
Markdown
Raw Normal View History

2015-04-26 12:48:37 +05:30
# Database MySQL
2014-09-02 18:07:02 +05:30
2019-01-03 12:48:30 +05:30
> **Note:**
> - We do not recommend using MySQL due to various issues. For example, case
[(in)sensitivity](https://dev.mysql.com/doc/refman/5.0/en/case-sensitivity.html)
and [problems](https://bugs.mysql.com/bug.php?id=65830) that
[suggested](https://bugs.mysql.com/bug.php?id=50909)
[fixes](https://bugs.mysql.com/bug.php?id=65830) [have](https://bugs.mysql.com/bug.php?id=63164).
2014-09-02 18:07:02 +05:30
2017-08-17 22:00:37 +05:30
## Initial database setup
2019-01-03 12:48:30 +05:30
```
2017-08-17 22:00:37 +05:30
# Install the database packages
sudo apt-get install -y mysql-server mysql-client libmysqlclient-dev
2018-03-17 18:26:18 +05:30
# Ensure you have MySQL version 5.6 or later
2017-08-17 22:00:37 +05:30
mysql --version
# Pick a MySQL root password (can be anything), type it and press enter
# Retype the MySQL root password and press enter
# Secure your installation
sudo mysql_secure_installation
# Login to MySQL
mysql -u root -p
# Type the MySQL root password
# Create a user for GitLab
# do not type the 'mysql>', this is part of the prompt
# change $password in the command below to a real password you pick
mysql> CREATE USER 'git'@'localhost' IDENTIFIED BY '$password';
# Ensure you can use the InnoDB engine which is necessary to support long indexes
# If this fails, check your MySQL config files (e.g. `/etc/mysql/*.cnf`, `/etc/mysql/conf.d/*`) for the setting "innodb = off"
mysql> SET storage_engine=INNODB;
# If you have MySQL < 5.7.7 and want to enable utf8mb4 character set support with your GitLab install, you must set the following NOW:
mysql> SET GLOBAL innodb_file_per_table=1, innodb_file_format=Barracuda, innodb_large_prefix=1;
2017-09-10 17:25:29 +05:30
# If you use MySQL with replication, or just have MySQL configured with binary logging, you need to run the following to allow the use of `TRIGGER`:
mysql> SET GLOBAL log_bin_trust_function_creators = 1;
2017-08-17 22:00:37 +05:30
# Create the GitLab production database
mysql> CREATE DATABASE IF NOT EXISTS `gitlabhq_production` DEFAULT CHARACTER SET `utf8` COLLATE `utf8_general_ci`;
# Grant the GitLab user necessary permissions on the database
2017-09-10 17:25:29 +05:30
mysql> GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, CREATE TEMPORARY TABLES, DROP, INDEX, ALTER, LOCK TABLES, REFERENCES, TRIGGER ON `gitlabhq_production`.* TO 'git'@'localhost';
2017-08-17 22:00:37 +05:30
# Quit the database session
mysql> \q
# Try connecting to the new database with the new user
sudo -u git -H mysql -u git -p -D gitlabhq_production
# Type the password you replaced $password with earlier
# You should now see a 'mysql>' prompt
# Quit the database session
mysql> \q
```
You are done installing the database for now and can go back to the rest of the installation.
2017-09-10 17:25:29 +05:30
Please proceed to the rest of the installation **before** running through the steps below.
### `log_bin_trust_function_creators`
If you use MySQL with replication, or just have MySQL configured with binary logging, all of your MySQL servers will need to have `log_bin_trust_function_creators` enabled to allow the use of `TRIGGER` in migrations. You have already set this global variable in the steps above, but to make it persistent, add the following to your `my.cnf` file:
```
log_bin_trust_function_creators=1
```
2017-08-17 22:00:37 +05:30
### MySQL utf8mb4 support
2017-09-10 17:25:29 +05:30
After installation or upgrade, remember to [convert any new tables](#tables-and-data-conversion-to-utf8mb4) to `utf8mb4`/`utf8mb4_general_ci`.
2017-08-17 22:00:37 +05:30
---
2018-12-05 23:21:45 +05:30
GitLab 8.14 has introduced [a feature](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7420) requiring `utf8mb4` encoding to be supported in your GitLab MySQL Database, which is not the case if you have set up your database before GitLab 8.16.
2017-08-17 22:00:37 +05:30
Follow the below instructions to ensure you use the most up to date requirements for your GitLab MySQL Database.
**We are about to do the following:**
- Ensure you can enable `utf8mb4` encoding and `utf8mb4_general_ci` collation for your GitLab DB, tables and data.
2019-01-03 12:48:30 +05:30
- Convert your GitLab tables and data from `utf8`/`utf8_general_ci` to `utf8mb4`/`utf8mb4_general_ci`
2017-08-17 22:00:37 +05:30
### Check for utf8mb4 support
#### Check for InnoDB File-Per-Table Tablespaces
2019-01-03 12:48:30 +05:30
We need to check, enable and maybe convert your existing GitLab DB tables to the [InnoDB File-Per-Table Tablespaces](http://dev.mysql.com/doc/refman/5.7/en/innodb-multiple-tablespaces.html) as a prerequisite for supporting **utfb8mb4 with long indexes** required by recent GitLab databases.
2017-08-17 22:00:37 +05:30
# Login to MySQL
mysql -u root -p
# Type the MySQL root password
mysql > use gitlabhq_production;
# Check your MySQL version is >= 5.5.3 (GitLab requires 5.5.14+)
mysql > SHOW VARIABLES LIKE 'version';
+---------------+-----------------+
| Variable_name | Value |
+---------------+-----------------+
| version | 5.5.53-0+deb8u1 |
+---------------+-----------------+
# Note where is your MySQL data dir for later:
mysql > select @@datadir;
+----------------+
| @@datadir |
+----------------+
| /var/lib/mysql |
+----------------+
# Note whether your MySQL server runs with innodb_file_per_table ON or OFF:
mysql> SELECT @@innodb_file_per_table;
+-------------------------+
| @@innodb_file_per_table |
+-------------------------+
| 1 |
+-------------------------+
# You can now quit the database session
mysql> \q
> You need **MySQL 5.5.3 or later** to perform this update.
2019-01-03 12:48:30 +05:30
Whatever the results of your checks above, we now need to check if your GitLab database has been created using [InnoDB File-Per-Table Tablespaces](http://dev.mysql.com/doc/refman/5.7/en/innodb-multiple-tablespaces.html) (i.e. `innodb_file_per_table` was set to **1** at initial setup time).
2017-08-17 22:00:37 +05:30
2019-01-03 12:48:30 +05:30
> Note: This setting is [enabled by default](http://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_file_per_table) since MySQL 5.6.6.
2017-08-17 22:00:37 +05:30
# Run this command with root privileges, replace the data dir if different:
sudo ls -lh /var/lib/mysql/gitlabhq_production/*.ibd | wc -l
# Run this command with root privileges, replace the data dir if different:
sudo ls -lh /var/lib/mysql/gitlabhq_production/*.frm | wc -l
2014-09-02 18:07:02 +05:30
2019-01-03 12:48:30 +05:30
2017-08-17 22:00:37 +05:30
- **Case 1: a result > 0 for both commands**
2016-01-29 22:53:50 +05:30
2019-01-03 12:48:30 +05:30
Congrats, your GitLab database uses the right InnoDB tablespace format.
2014-09-02 18:07:02 +05:30
2017-08-17 22:00:37 +05:30
However, you must still ensure that any **future tables** created by GitLab will still use the right format:
2014-09-02 18:07:02 +05:30
2017-08-17 22:00:37 +05:30
- If `SELECT @@innodb_file_per_table` returned **1** previously, your server is running correctly.
2018-11-20 20:47:30 +05:30
2019-01-03 12:48:30 +05:30
> It's however a requirement to check *now* that this setting is indeed persisted in your [my.cnf](https://dev.mysql.com/doc/refman/5.7/en/tablespace-enabling.html) file!
2017-08-17 22:00:37 +05:30
- If `SELECT @@innodb_file_per_table` returned **0** previously, your server is not running correctly.
2018-11-20 20:47:30 +05:30
2019-01-03 12:48:30 +05:30
> [Enable innodb_file_per_table](https://dev.mysql.com/doc/refman/5.7/en/tablespace-enabling.html) by running in a MySQL session as root the command `SET GLOBAL innodb_file_per_table=1, innodb_file_format=Barracuda;` and persist the two settings in your [my.cnf](https://dev.mysql.com/doc/refman/5.7/en/tablespace-enabling.html) file
2017-08-17 22:00:37 +05:30
Now, if you have a **different result** returned by the 2 commands above, it means you have a **mix of tables format** uses in your GitLab database. This can happen if your MySQL server had different values for `innodb_file_per_table` in its life and you updated GitLab at different moments with those inconsistent values. So keep reading.
- **Case 2: a result equals to "0" OR not the same result for both commands**
2019-01-03 12:48:30 +05:30
Unfortunately, none or only some of your GitLab database tables use the GitLab requirement of [InnoDB File-Per-Table Tablespaces](http://dev.mysql.com/doc/refman/5.7/en/innodb-multiple-tablespaces.html).
2017-08-17 22:00:37 +05:30
Let's enable what we need on the running server:
2014-09-02 18:07:02 +05:30
# Login to MySQL
mysql -u root -p
2015-04-26 12:48:37 +05:30
# Type the MySQL root password
2014-09-02 18:07:02 +05:30
2017-08-17 22:00:37 +05:30
# Enable innodb_file_per_table and set innodb_file_format on the running server:
mysql > SET GLOBAL innodb_file_per_table=1, innodb_file_format=Barracuda;
# You can now quit the database session
mysql> \q
2019-01-03 12:48:30 +05:30
> Now, **persist** [innodb_file_per_table](https://dev.mysql.com/doc/refman/5.6/en/tablespace-enabling.html) and [innodb_file_format](https://dev.mysql.com/doc/refman/5.6/en/innodb-file-format-enabling.html) in your `my.cnf` file.
2017-08-17 22:00:37 +05:30
Ensure at this stage that your GitLab instance is indeed **stopped**.
Now, let's convert all the GitLab database tables to the new tablespace format:
# Login to MySQL
mysql -u root -p
# Type the MySQL root password
mysql > use gitlabhq_production;
2014-09-02 18:07:02 +05:30
2019-01-03 12:48:30 +05:30
# Safety check: you should still have those values set as follow:
2017-08-17 22:00:37 +05:30
mysql> SELECT @@innodb_file_per_table, @@innodb_file_format;
+-------------------------+----------------------+
| @@innodb_file_per_table | @@innodb_file_format |
+-------------------------+----------------------+
| 1 | Barracuda |
+-------------------------+----------------------+
2016-01-29 22:53:50 +05:30
2017-08-17 22:00:37 +05:30
mysql > SELECT CONCAT('ALTER TABLE `', TABLE_NAME,'` ENGINE=InnoDB;') AS 'Copy & run these SQL statements:' FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA="gitlabhq_production" AND TABLE_TYPE="BASE TABLE";
2014-09-02 18:07:02 +05:30
2017-08-17 22:00:37 +05:30
# If previous query returned results, copy & run all shown SQL statements
2014-09-02 18:07:02 +05:30
2017-08-17 22:00:37 +05:30
# You can now quit the database session
2014-09-02 18:07:02 +05:30
mysql> \q
2017-08-17 22:00:37 +05:30
---
#### Check for proper InnoDB File Format, Row Format, Large Prefix and tables conversion
2019-01-03 12:48:30 +05:30
We need to check, enable and probably convert your existing GitLab DB tables to use the [Barracuda InnoDB file format](https://dev.mysql.com/doc/refman/5.6/en/innodb-file-format.html), the [DYNAMIC row format](https://dev.mysql.com/doc/refman/5.6/en/glossary.html#glos_dynamic_row_format) and [innodb_large_prefix](http://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_large_prefix) as a second prerequisite for supporting **utfb8mb4 with long indexes** used by recent GitLab databases.
2017-08-17 22:00:37 +05:30
# Login to MySQL
mysql -u root -p
# Type the MySQL root password
mysql > use gitlabhq_production;
2014-09-02 18:07:02 +05:30
2017-08-17 22:00:37 +05:30
# Set innodb_file_format and innodb_large_prefix on the running server:
# Note: These are the default settings only for MySQL 5.7.7 and later.
2014-09-02 18:07:02 +05:30
2017-08-17 22:00:37 +05:30
mysql > SET GLOBAL innodb_file_format=Barracuda, innodb_large_prefix=1;
2014-09-02 18:07:02 +05:30
2017-08-17 22:00:37 +05:30
# Your DB must be (still) using utf8/utf8_general_ci as default encoding and collation.
# We will NOT change the default encoding and collation on the DB in order to support future GitLab migrations creating tables
# that require "long indexes support" on installations using MySQL <= 5.7.9.
# However, when such migrations occur, you will have to follow this guide again to convert the newly created tables to the proper encoding/collation.
# This should return the following:
mysql> SELECT @@character_set_database, @@collation_database;
+--------------------------+----------------------+
| @@character_set_database | @@collation_database |
+--------------------------+----------------------+
| utf8 | utf8_general_ci |
+--------------------------+----------------------+
2019-01-03 12:48:30 +05:30
> Now, ensure that [innodb_file_format](https://dev.mysql.com/doc/refman/5.6/en/tablespace-enabling.html) and [innodb_large_prefix](http://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_large_prefix) are **persisted** in your `my.cnf` file.
2017-08-17 22:00:37 +05:30
#### Tables and data conversion to utf8mb4
Now that you have a persistent MySQL setup, you can safely upgrade tables after setup or upgrade time:
# Convert tables not using ROW_FORMAT DYNAMIC:
mysql> SELECT CONCAT('ALTER TABLE `', TABLE_NAME,'` ROW_FORMAT=DYNAMIC;') AS 'Copy & run these SQL statements:' FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA="gitlabhq_production" AND TABLE_TYPE="BASE TABLE" AND ROW_FORMAT!="Dynamic";
# !! If previous query returned results, copy & run all shown SQL statements
# Convert tables/columns not using utf8mb4/utf8mb4_general_ci as encoding/collation:
mysql > SET foreign_key_checks = 0;
mysql > SELECT CONCAT('ALTER TABLE `', TABLE_NAME,'` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;') AS 'Copy & run these SQL statements:' FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA="gitlabhq_production" AND TABLE_COLLATION != "utf8mb4_general_ci" AND TABLE_TYPE="BASE TABLE";
# !! If previous query returned results, copy & run all shown SQL statements
# Turn foreign key checks back on
mysql > SET foreign_key_checks = 1;
# You can now quit the database session
2014-09-02 18:07:02 +05:30
mysql> \q
2017-08-17 22:00:37 +05:30
Ensure your GitLab database configuration file uses a proper connection encoding and collation:
2019-01-03 12:48:30 +05:30
```sudo -u git -H editor config/database.yml```
2017-08-17 22:00:37 +05:30
production:
adapter: mysql2
encoding: utf8mb4
collation: utf8mb4_general_ci
[Restart your GitLab instance](../administration/restart_gitlab.md).
2019-01-03 12:48:30 +05:30
2016-01-29 22:53:50 +05:30
## MySQL strings limits
After installation or upgrade, remember to run the `add_limits_mysql` Rake task:
2017-08-17 22:00:37 +05:30
**Omnibus GitLab installations**
2019-01-03 12:48:30 +05:30
```
2017-08-17 22:00:37 +05:30
sudo gitlab-rake add_limits_mysql
```
**Installations from source**
2019-01-03 12:48:30 +05:30
```
2017-08-17 22:00:37 +05:30
bundle exec rake add_limits_mysql RAILS_ENV=production
2016-01-29 22:53:50 +05:30
```
The `text` type in MySQL has a different size limit than the `text` type in
PostgreSQL. In MySQL `text` columns are limited to ~65kB, whereas in PostgreSQL
`text` columns are limited up to ~1GB!
The `add_limits_mysql` Rake task converts some important `text` columns in the
GitLab database to `longtext` columns, which can persist values of up to 4GB
(sometimes less if the value contains multibyte characters).
Details can be found in the [PostgreSQL][postgres-text-type] and
2019-01-03 12:48:30 +05:30
[MySQL][mysql-text-types] manuals.
2016-01-29 22:53:50 +05:30
2017-09-10 17:25:29 +05:30
[postgres-text-type]: http://www.postgresql.org/docs/9.2/static/datatype-character.html
2019-01-03 12:48:30 +05:30
[mysql-text-types]: http://dev.mysql.com/doc/refman/5.7/en/string-type-overview.html
2018-03-17 18:26:18 +05:30
[ce-38152]: https://gitlab.com/gitlab-org/gitlab-ce/issues/38152