debian-mirror-gitlab/doc/user/project/settings/import_export.md

381 lines
15 KiB
Markdown
Raw Normal View History

2020-10-24 23:57:45 +05:30
---
2021-01-29 00:20:46 +05:30
stage: Manage
group: Import
2021-02-22 17:27:13 +05:30
info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments"
2020-10-24 23:57:45 +05:30
---
2021-09-04 01:27:46 +05:30
# Project import/export **(FREE)**
2016-08-24 12:49:21 +05:30
2020-06-23 00:09:42 +05:30
> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/3050) in GitLab 8.9.
2020-03-13 15:44:24 +05:30
> - From GitLab 10.0, administrators can disable the project export option on the GitLab instance.
Existing projects running on any GitLab instance or GitLab.com can be exported with all their related
data and be moved into a new GitLab instance.
The **GitLab import/export** button is displayed if the project import option is enabled.
See also:
- [Project import/export API](../../../api/project_import_export.md)
2021-03-11 19:13:27 +05:30
- [Project import/export administration Rake tasks](../../../administration/raketasks/project_import_export.md) **(FREE SELF)**
2020-05-24 23:13:21 +05:30
- [Group import/export](../../group/settings/import_export.md)
2020-04-08 14:13:33 +05:30
- [Group import/export API](../../../api/group_import_export.md)
2020-03-13 15:44:24 +05:30
To set up a project import/export:
2021-09-04 01:27:46 +05:30
1. On the top bar, go to **Menu > Admin > Settings > General > Visibility and access controls**.
1. Scroll to **Import sources**.
1. Enable the desired **Import sources**.
2020-03-13 15:44:24 +05:30
## Important notes
Note the following:
2021-10-27 15:23:28 +05:30
- Before you can import a project, you must export the data first.
See [Export a project and its data](#export-a-project-and-its-data)
2021-01-29 00:20:46 +05:30
for how you can export a project through the UI.
2020-05-24 23:13:21 +05:30
- Imports from a newer version of GitLab are not supported.
The Importing GitLab version must be greater than or equal to the Exporting GitLab version.
2021-06-08 01:23:25 +05:30
- Imports fail unless the import and export GitLab instances are
2020-03-13 15:44:24 +05:30
compatible as described in the [Version history](#version-history).
2021-06-08 01:23:25 +05:30
- Exports are generated in your configured `shared_path`, a temporary shared directory,
2021-04-29 21:17:54 +05:30
and are moved to your configured `uploads_directory`. Every 24 hours, a specific worker deletes these export files.
2020-03-13 15:44:24 +05:30
- Group members are exported as project members, as long as the user has
2021-10-27 15:23:28 +05:30
a maintainer or administrator role in the group where the exported project lives.
2021-09-04 01:27:46 +05:30
- Project members with the [Owner role](../../permissions.md) are imported as Maintainers.
- Imported users can be mapped by their public email on self-managed instances, if an administrative user (not an owner) does the import.
2021-11-11 11:23:49 +05:30
Additionally, the user must be an existing member of the namespace, or the user can be added as a
member of the project for contributions to be mapped.
2020-03-13 15:44:24 +05:30
Otherwise, a supplementary comment is left to mention that the original author and
2021-06-08 01:23:25 +05:30
the MRs, notes, or issues are owned by the importer.
- For project migration imports performed over GitLab.com Groups, preserving author information is
possible through a [professional services engagement](https://about.gitlab.com/services/migration/).
2020-03-13 15:44:24 +05:30
- If an imported project contains merge requests originating from forks,
2021-06-08 01:23:25 +05:30
then new branches associated with such merge requests are created
2021-10-27 15:23:28 +05:30
in a project during the import/export. Thus, the number of branches
2020-03-13 15:44:24 +05:30
in the exported project could be bigger than in the original project.
2021-02-22 17:27:13 +05:30
- Deploy keys allowed to push to protected branches are not exported. Therefore,
2021-10-27 15:23:28 +05:30
you must recreate this association by first enabling these deploy keys in your
2021-02-22 17:27:13 +05:30
imported project and then updating your protected branches accordingly.
2016-08-24 12:49:21 +05:30
2016-09-29 09:46:39 +05:30
## Version history
2021-06-08 01:23:25 +05:30
### 14.0+
In GitLab 14.0, the JSON format is no longer supported for project and group exports. To allow for a
transitional period, you can still import any JSON exports. The new format for imports and exports
is NDJSON.
2020-07-28 23:09:34 +05:30
### 13.0+
2020-05-24 23:13:21 +05:30
Starting with GitLab 13.0, GitLab can import bundles that were exported from a different GitLab deployment.
This ability is limited to two previous GitLab [minor](../../../policy/maintenance.md#versioning)
releases, which is similar to our process for [Security Releases](../../../policy/maintenance.md#security-releases).
For example:
| Current version | Can import bundles exported from |
|-----------------|----------------------------------|
| 13.0 | 13.0, 12.10, 12.9 |
| 13.1 | 13.1, 13.0, 12.10 |
### 12.x
Prior to 13.0 this was a defined compatibility table:
2020-03-13 15:44:24 +05:30
2020-04-08 14:13:33 +05:30
| Exporting GitLab version | Importing GitLab version |
| -------------------------- | -------------------------- |
2020-07-28 23:09:34 +05:30
| 11.7 to 12.10 | 11.7 to 12.10 |
2020-04-08 14:13:33 +05:30
| 11.1 to 11.6 | 11.1 to 11.6 |
| 10.8 to 11.0 | 10.8 to 11.0 |
| 10.4 to 10.7 | 10.4 to 10.7 |
| 10.3 | 10.3 |
| 10.0 to 10.2 | 10.0 to 10.2 |
| 9.4 to 9.6 | 9.4 to 9.6 |
| 9.2 to 9.3 | 9.2 to 9.3 |
| 8.17 to 9.1 | 8.17 to 9.1 |
| 8.13 to 8.16 | 8.13 to 8.16 |
| 8.12 | 8.12 |
| 8.10.3 to 8.11 | 8.10.3 to 8.11 |
| 8.10.0 to 8.10.2 | 8.10.0 to 8.10.2 |
| 8.9.5 to 8.9.11 | 8.9.5 to 8.9.11 |
| 8.9.0 to 8.9.4 | 8.9.0 to 8.9.4 |
2017-08-17 22:00:37 +05:30
2020-03-13 15:44:24 +05:30
Projects can be exported and imported only between versions of GitLab with matching Import/Export versions.
For example, 8.10.3 and 8.11 have the same Import/Export version (0.1.3)
2021-06-08 01:23:25 +05:30
and the exports between them are compatible.
2016-09-29 09:46:39 +05:30
2020-05-24 23:13:21 +05:30
## Between CE and EE
You can export projects from the [Community Edition to the Enterprise Edition](https://about.gitlab.com/install/ce-or-ee/) and vice versa.
This assumes [version history](#version-history) requirements are met.
2021-09-30 23:02:18 +05:30
If you're exporting a project from the Enterprise Edition to the Community Edition, you may lose data that is retained only in the Enterprise Edition. For more information, see [downgrading from EE to CE](../../../index.md).
2020-05-24 23:13:21 +05:30
2016-08-24 12:49:21 +05:30
## Exported contents
2021-06-08 01:23:25 +05:30
The following items are exported:
2016-08-24 12:49:21 +05:30
- Project and wiki repositories
- Project uploads
2020-07-28 23:09:34 +05:30
- Project configuration, excluding integrations
2020-04-22 19:07:51 +05:30
- Issues with comments, merge requests with diffs and comments, labels, milestones, snippets, time tracking,
2016-08-24 12:49:21 +05:30
and other project entities
2020-04-22 19:07:51 +05:30
- Design Management files and data
2018-05-09 12:01:36 +05:30
- LFS objects
2019-10-12 21:52:04 +05:30
- Issue boards
2020-04-08 14:13:33 +05:30
- Pipelines history
2021-02-22 17:27:13 +05:30
- Push Rules
2021-04-17 20:07:23 +05:30
- Awards
2016-08-24 12:49:21 +05:30
2021-06-08 01:23:25 +05:30
The following items are **not** exported:
2016-08-24 12:49:21 +05:30
- Build traces and artifacts
2017-08-17 22:00:37 +05:30
- Container registry images
2021-04-17 20:07:23 +05:30
- CI/CD variables
2021-09-30 23:02:18 +05:30
- Pipeline triggers
2019-02-15 15:39:39 +05:30
- Webhooks
2017-08-17 22:00:37 +05:30
- Any encrypted tokens
2019-09-30 21:07:59 +05:30
- Merge Request Approvers
2021-09-30 23:02:18 +05:30
- Repository size limits
2019-09-30 21:07:59 +05:30
2021-11-18 22:05:49 +05:30
These content rules also apply to creating projects from templates on the
[group](../../group/custom_project_templates.md)
or [instance](../../admin_area/custom_project_templates.md)
levels, because the same export and import mechanisms are used.
2021-02-22 17:27:13 +05:30
NOTE:
2019-09-30 21:07:59 +05:30
For more details on the specific data persisted in a project export, see the
2021-09-04 01:27:46 +05:30
[`import_export.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/import_export/project/import_export.yml) file.
2016-08-24 12:49:21 +05:30
2021-10-27 15:23:28 +05:30
## Export a project and its data
2016-08-24 12:49:21 +05:30
2021-01-29 00:20:46 +05:30
Full project export functionality is limited to project maintainers and owners.
You can configure such functionality through [project settings](index.md):
To export a project and its data, follow these steps:
2019-09-04 21:01:54 +05:30
1. Go to your project's homepage.
2016-08-24 12:49:21 +05:30
2021-12-11 22:18:48 +05:30
1. Select **Settings** in the sidebar.
1. Scroll down and expand the **Advanced** section.
2016-08-24 12:49:21 +05:30
1. Scroll down to find the **Export project** button:
2019-09-30 21:07:59 +05:30
![Export button](img/import_export_export_button.png)
2016-08-24 12:49:21 +05:30
2021-10-27 15:23:28 +05:30
1. After the export is generated, you should receive an email with a link to
2016-08-24 12:49:21 +05:30
download the file:
2019-09-30 21:07:59 +05:30
![Email download link](img/import_export_mail_link.png)
2016-08-24 12:49:21 +05:30
1. Alternatively, you can come back to the project settings and download the
2021-10-27 15:23:28 +05:30
file from there, or generate a new export. After the file is available, the page
2016-08-24 12:49:21 +05:30
should show the **Download export** button:
2019-09-30 21:07:59 +05:30
![Download export](img/import_export_download_export.png)
2016-08-24 12:49:21 +05:30
2021-10-27 15:23:28 +05:30
## Import the project
2021-12-11 22:18:48 +05:30
> Default maximum import file size [changed](https://gitlab.com/gitlab-org/gitlab/-/issues/251106) from 50 MB to unlimited in GitLab 13.8.
2021-10-27 15:23:28 +05:30
WARNING:
Only import projects from sources you trust. If you import a project from an untrusted source, it
may be possible for an attacker to steal your sensitive data.
2016-08-24 12:49:21 +05:30
2019-09-30 21:07:59 +05:30
1. The GitLab project import feature is the first import option when creating a
2021-12-11 22:18:48 +05:30
new project. Select **GitLab export**:
2016-08-24 12:49:21 +05:30
2019-09-30 21:07:59 +05:30
![New project](img/import_export_new_project.png)
2016-08-24 12:49:21 +05:30
2019-09-04 21:01:54 +05:30
1. Enter your project name and URL. Then select the file you exported previously:
2016-08-24 12:49:21 +05:30
2019-09-30 21:07:59 +05:30
![Select file](img/import_export_select_file.png)
2016-08-24 12:49:21 +05:30
2021-12-11 22:18:48 +05:30
1. Select **Import project** to begin importing. Your newly imported project
2021-06-08 01:23:25 +05:30
page appears shortly.
2019-10-12 21:52:04 +05:30
2021-02-22 17:27:13 +05:30
NOTE:
2019-10-12 21:52:04 +05:30
If use of the `Internal` visibility level
2021-09-30 23:02:18 +05:30
[is restricted](../../../public_access/public_access.md#restrict-use-of-public-or-internal-projects),
2019-10-12 21:52:04 +05:30
all imported projects are given the visibility of `Private`.
2020-03-13 15:44:24 +05:30
2021-12-11 22:18:48 +05:30
The maximum import file size can be set by the Administrator, and the default is `0` (unlimited).
As an administrator, you can modify the maximum import file size. To do so, use the `max_import_size` option in the [Application settings API](../../../api/settings.md#change-application-settings) or the [Admin Area UI](../../admin_area/settings/account_and_limit_settings.md).
2020-06-23 00:09:42 +05:30
2020-07-28 23:09:34 +05:30
### Project import status
You can query an import through the [Project import/export API](../../../api/project_import_export.md#import-status).
As described in the API documentation, the query may return an import error or exceptions.
2021-03-11 19:13:27 +05:30
### Import large projects **(FREE SELF)**
2020-07-28 23:09:34 +05:30
If you have a larger project, consider using a Rake task, as described in our [developer documentation](../../../development/import_project.md#importing-via-a-rake-task).
2021-02-22 17:27:13 +05:30
## Rate Limits
To help avoid abuse, by default, users are rate limited to:
2020-03-13 15:44:24 +05:30
2021-02-22 17:27:13 +05:30
| Request Type | Limit |
| ---------------- | ---------------------------------------- |
| Export | 6 projects per minute |
| Download export | 1 download per group per minute |
| Import | 6 projects per minute |
2020-03-13 15:44:24 +05:30
2021-10-27 15:23:28 +05:30
GitLab.com may have [different settings](../../gitlab_com/index.md#importexport) from the defaults.
2021-12-11 22:18:48 +05:30
## Automate group and project import **(PREMIUM)**
For information on automating user, group, and project import API calls, see
[Automate group and project import](../import/index.md#automate-group-and-project-import).
2021-10-27 15:23:28 +05:30
## Troubleshooting
2021-11-18 22:05:49 +05:30
### Project fails to import due to mismatch
If the [shared runners enablement](../../../ci/runners/runners_scope.md#enable-shared-runners)
does not match between the exported project, and the project import, the project fails to import.
Review [issue 276930](https://gitlab.com/gitlab-org/gitlab/-/issues/276930), and either:
- Ensure shared runners are enabled in both the source and destination projects.
2021-12-11 22:18:48 +05:30
- Disable shared runners on the parent group when you import the project.
2021-11-18 22:05:49 +05:30
2021-12-11 22:18:48 +05:30
### Import workarounds for large repositories
2021-10-27 15:23:28 +05:30
[Maximum import size limitations](#import-the-project)
2021-12-11 22:18:48 +05:30
can prevent an import from being successful. If changing the import limits is not possible, you can
try one of the workarounds listed here.
#### Workaround option 1
The following local workflow can be used to temporarily
reduce the repository size for another import attempt:
2021-10-27 15:23:28 +05:30
2021-11-11 11:23:49 +05:30
1. Create a temporary working directory from the export:
2021-10-27 15:23:28 +05:30
```shell
EXPORT=<filename-without-extension>
mkdir "$EXPORT"
tar -xf "$EXPORT".tar.gz --directory="$EXPORT"/
cd "$EXPORT"/
git clone project.bundle
# Prevent interference with recreating an importable file later
mv project.bundle ../"$EXPORT"-original.bundle
mv ../"$EXPORT".tar.gz ../"$EXPORT"-original.tar.gz
2021-11-11 11:23:49 +05:30
git switch --create smaller-tmp-main
2021-10-27 15:23:28 +05:30
```
2021-11-11 11:23:49 +05:30
1. To reduce the repository size, work on this `smaller-tmp-main` branch:
2021-10-27 15:23:28 +05:30
[identify and remove large files](../repository/reducing_the_repo_size_using_git.md)
or [interactively rebase and fixup](../../../topics/git/git_rebase.md#interactive-rebase)
to reduce the number of commits.
```shell
# Reduce the .git/objects/pack/ file size
cd project
git reflog expire --expire=now --all
git gc --prune=now --aggressive
# Prepare recreating an importable file
2021-11-18 22:05:49 +05:30
git bundle create ../project.bundle <default-branch-name>
2021-10-27 15:23:28 +05:30
cd ..
mv project/ ../"$EXPORT"-project
cd ..
# Recreate an importable file
tar -czf "$EXPORT"-smaller.tar.gz --directory="$EXPORT"/ .
```
1. Import this new, smaller file into GitLab.
1. In a full clone of the original repository,
use `git remote set-url origin <new-url> && git push --force --all`
to complete the import.
1. Update the imported repository's
[branch protection rules](../protected_branches.md) and
its [default branch](../repository/branches/default.md), and
2021-11-11 11:23:49 +05:30
delete the temporary, `smaller-tmp-main` branch, and
2021-10-27 15:23:28 +05:30
the local, temporary data.
2021-11-18 22:05:49 +05:30
2021-12-11 22:18:48 +05:30
#### Workaround option 2
Rather than attempting to push all changes at once, this workaround:
- Separates the project import from the Git Repository import
- Incrementally pushes the repository to GitLab
1. Make a local clone of the repository to migrate. In a later step, you push this clone outside of
the project export.
1. Download the export and remove the `project.bundle` (which contains the Git repository):
```shell
tar -czvf new_export.tar.gz --exclude='project.bundle' @old_export.tar.gz
```
1. Import the export without a Git repository. It asks you to confirm to import without a
repository.
1. Save this bash script as a file and run it after adding the appropriate origin.
```shell
#!/bin/sh
# ASSUMPTIONS:
# - The GitLab location is "origin"
# - The default branch is "main"
# - This will attempt to push in chunks of 500MB (dividing the total size by 500MB).
# Decrease this size to push in smaller chunks if you still receive timeouts.
git gc
SIZE=$(git count-objects -v 2> /dev/null | grep size-pack | awk '{print $2}')
# Be conservative... and try to push 2GB at a time
# (given this assumes each commit is the same size - which is wrong)
BATCHES=$(($SIZE / 500000))
TOTAL_COMMITS=$(git rev-list --count HEAD)
if (( BATCHES > TOTAL_COMMITS )); then
BATCHES=$TOTAL_COMMITS
fi
INCREMENTS=$(( ($TOTAL_COMMITS / $BATCHES) - 1 ))
for (( BATCH=BATCHES; BATCH>=1; BATCH-- ))
do
COMMIT_NUM=$(( $BATCH - $INCREMENTS ))
COMMIT_SHA=$(git log -n $COMMIT_NUM --format=format:%H | tail -1)
git push -u origin ${COMMIT_SHA}:refs/heads/main
done
git push -u origin main
git push -u origin -—all
git push -u origin -—tags
```
2021-11-18 22:05:49 +05:30
### Manually execute export steps
Exports sometimes fail without giving enough information to troubleshoot. In these cases, it can be
helpful to [execute the export process manually within rails](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/uncategorized/project-export.md#export-a-project-via-rails-console).
Execute each line individually, rather than pasting the entire block at once, so you can see any
errors each command returns.
```shell
u = User.find_by_username('someuser')
p = Project.find_by_full_path('some/project')
e = Projects::ImportExport::ExportService.new(p,u)
e.send(:version_saver).send(:save)
e.send(:avatar_saver).send(:save)
e.send(:project_tree_saver).send(:save)
e.send(:uploads_saver).send(:save)
e.send(:wiki_repo_saver).send(:save)
e.send(:lfs_saver).send(:save)
e.send(:snippets_repo_saver).send(:save)
e.send(:design_repo_saver).send(:save)
s = Gitlab::ImportExport::Saver.new(exportable: p, shared:p.import_export_shared)
s.send(:compress_and_save)
s.send(:save_upload)
```