debian-mirror-gitlab/doc/topics/git/lfs/index.md

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

291 lines
12 KiB
Markdown
Raw Normal View History

2020-04-22 19:07:51 +05:30
---
2020-10-24 23:57:45 +05:30
stage: Create
group: Source Code
2022-11-25 23:54:43 +05:30
info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments"
2020-10-24 23:57:45 +05:30
type: reference, howto
2020-04-22 19:07:51 +05:30
disqus_identifier: 'https://docs.gitlab.com/ee/workflow/lfs/lfs/index.html'
---
2021-03-11 19:13:27 +05:30
# Git Large File Storage (LFS) **(FREE)**
2020-04-22 19:07:51 +05:30
Managing large files such as audio, video and graphics files has always been one
of the shortcomings of Git. The general recommendation is to not have Git repositories
2022-06-21 17:19:12 +05:30
larger than 1 GB to preserve performance.
2020-04-22 19:07:51 +05:30
![Git LFS tracking status](img/lfs-icon.png)
2021-03-11 19:13:27 +05:30
Files tracked by Git LFS display an icon to indicate if the file is stored as a
blob or an LFS pointer.
2020-04-22 19:07:51 +05:30
## How it works
2022-06-21 17:19:12 +05:30
Git LFS client communicates with the GitLab server over HTTPS. It uses HTTP Basic Authentication
2021-03-11 19:13:27 +05:30
to authorize client requests. After the request is authorized, Git LFS client receives
2020-04-22 19:07:51 +05:30
instructions from where to fetch or where to push the large file.
## GitLab server configuration
Documentation for GitLab instance administrators is under [LFS administration doc](../../../administration/lfs/index.md).
## Requirements
2022-07-23 23:45:48 +05:30
- Git LFS must be [enabled in project settings](../../../user/project/settings/index.md#configure-project-visibility-features-and-permissions).
2022-06-21 17:19:12 +05:30
- [Git LFS client](https://git-lfs.github.com) version 1.0.1 or higher must be installed.
2020-04-22 19:07:51 +05:30
## Known limitations
2021-03-11 19:13:27 +05:30
- Git LFS v1 original API is not supported, because it was deprecated early in LFS
development.
- When SSH is set as a remote, Git LFS objects still go through HTTPS.
2022-06-21 17:19:12 +05:30
- Any Git LFS request asks for HTTPS credentials, so we recommend a good Git
credentials store.
2021-03-11 19:13:27 +05:30
- Git LFS always assumes HTTPS so if you have GitLab server on HTTP you must
[add the URL to Git configuration manually](#troubleshooting).
2021-12-11 22:18:48 +05:30
- [Group wikis](../../../user/project/wiki/group.md) do not support Git LFS.
2020-04-22 19:07:51 +05:30
2023-01-13 00:05:48 +05:30
## How LFS objects affect repository size
When you add an LFS object to a repository, GitLab:
1. Creates an LFS object.
1. Associates the LFS object with the repository.
1. Queues a job to recalculate your project's statistics, including storage size and
LFS object storage. Your LFS object storage is the sum of the size of all LFS objects
associated with the repository.
When your repository is forked, LFS objects from the upstream project are associated
with the fork. When the fork is created, the LFS object storage for the fork is equal
to the storage used by the upstream project. If new LFS objects are added to the fork,
the total object storage changes for the fork, but not the upstream project.
If you create a merge request from the fork back to the upstream project,
any new LFS objects in the fork become associated with the upstream project.
2020-04-22 19:07:51 +05:30
## Using Git LFS
2022-07-16 23:28:13 +05:30
Let's take a look at the workflow for checking large files into your Git
2020-04-22 19:07:51 +05:30
repository with Git LFS. For example, if you want to upload a very large file and
check it into your Git repository:
```shell
2022-08-27 11:52:29 +05:30
git clone git@gitlab.example.com:group/my-sample-project.git
cd my-sample-project
2020-04-22 19:07:51 +05:30
git lfs install # initialize the Git LFS project
git lfs track "*.iso" # select the file extensions that you want to treat as large files
```
2021-03-11 19:13:27 +05:30
After you mark a file extension for tracking as a LFS object you can use
Git as usual without redoing the command to track a file with the same extension:
2020-04-22 19:07:51 +05:30
```shell
cp ~/tmp/debian.iso ./ # copy a large file into the current directory
git add . # add the large file to the project
git commit -am "Added Debian iso" # commit the file meta data
2022-06-21 17:19:12 +05:30
git push origin main # sync the git repo and large file to the GitLab server
2020-04-22 19:07:51 +05:30
```
**Make sure** that `.gitattributes` is tracked by Git. Otherwise Git
2021-03-11 19:13:27 +05:30
LFS doesn't work properly for people cloning the project:
2020-04-22 19:07:51 +05:30
```shell
git add .gitattributes
2022-08-27 11:52:29 +05:30
git commit -am "Added .gitattributes to capture LFS tracking"
git push origin main
2020-04-22 19:07:51 +05:30
```
Cloning the repository works the same as before. Git automatically detects the
LFS-tracked files and clones them via HTTP. If you performed the `git clone`
command with a SSH URL, you have to enter your GitLab credentials for HTTP
authentication.
```shell
2022-08-27 11:52:29 +05:30
git clone git@gitlab.example.com:group/my-sample-project.git
2020-04-22 19:07:51 +05:30
```
If you already cloned the repository and you want to get the latest LFS object
2020-06-23 00:09:42 +05:30
that are on the remote repository, such as for a branch from origin:
2020-04-22 19:07:51 +05:30
```shell
2022-06-21 17:19:12 +05:30
git lfs fetch origin main
2020-04-22 19:07:51 +05:30
```
2021-03-11 19:13:27 +05:30
Make sure your files aren't listed in `.gitignore`, otherwise, they are ignored by Git
and are not pushed to the remote repository.
2020-07-28 23:09:34 +05:30
2021-03-08 18:12:59 +05:30
### Migrate an existing repository to Git LFS
2020-04-22 19:07:51 +05:30
2022-08-27 11:52:29 +05:30
Read the documentation on how to [migrate an existing Git repository with Git LFS](https://github.com/git-lfs/git-lfs/blob/main/docs/man/git-lfs-migrate.adoc).
2020-04-22 19:07:51 +05:30
### Removing objects from LFS
To remove objects from LFS:
2020-06-23 00:09:42 +05:30
1. Use [`git filter-repo`](../../../user/project/repository/reducing_the_repo_size_using_git.md) to remove the objects from the repository.
2020-04-22 19:07:51 +05:30
1. Delete the relevant LFS lines for the objects you have removed from your `.gitattributes` file and commit those changes.
## File Locking
2020-11-24 15:15:51 +05:30
See the documentation on [File Locking](../../../user/project/file_lock.md).
2020-04-22 19:07:51 +05:30
2021-01-03 14:25:43 +05:30
## LFS objects in project archives
2022-03-02 08:16:31 +05:30
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15079) support for including Git LFS blobs inside [project source downloads](../../../user/project/repository/index.md) in GitLab 13.5 [with a flag](../../../administration/feature_flags.md) named `include_lfs_blobs_in_archive`. Disabled by default.
> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/46572) in GitLab 13.6.
> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/62539) in GitLab 14.0. Feature flag `include_lfs_blobs_in_archive` removed.
2021-01-03 14:25:43 +05:30
2022-08-27 11:52:29 +05:30
Prior to GitLab 13.5, [project source downloads](../../../user/project/repository/index.md) would include Git
2021-01-03 14:25:43 +05:30
LFS pointers instead of the actual objects. For example, LFS pointers
look like the following:
```markdown
version https://git-lfs.github.com/spec/v1
oid sha256:3ea5dd307f195f449f0e08234183b82e92c3d5f4cff11c2a6bb014f9e0de12aa
size 177735
```
2021-09-04 01:27:46 +05:30
In GitLab version 13.5 and later, these pointers are converted to the uploaded
LFS object.
2021-01-03 14:25:43 +05:30
Technical details about how this works can be found in the [development documentation for LFS](../../../development/lfs.md#including-lfs-blobs-in-project-archives).
2023-03-04 22:38:38 +05:30
## Related topics
- Blog post: [Getting started with Git LFS](https://about.gitlab.com/blog/2017/01/30/getting-started-with-git-lfs-tutorial/)
- [Git LFS developer information](../../../development/lfs.md)
- [GitLab Git Large File Storage (LFS) Administration](../../../administration/lfs/index.md) for self-managed instances
2021-09-04 01:27:46 +05:30
## Troubleshooting
2021-01-03 14:25:43 +05:30
2022-07-16 23:28:13 +05:30
### Encountered `n` files that should have been pointers, but weren't
2021-01-03 14:25:43 +05:30
2022-07-16 23:28:13 +05:30
This error indicates the files are expected to be tracked by LFS, but
the repository is not tracking them as LFS. This issue can be one
2021-09-04 01:27:46 +05:30
potential reason for this error:
[Files not tracked with LFS when uploaded through the web interface](https://gitlab.com/gitlab-org/gitlab/-/issues/326342#note_586820485)
2021-01-03 14:25:43 +05:30
2021-09-04 01:27:46 +05:30
To resolve the problem, migrate the affected file (or files) and push back to the repository:
2021-01-03 14:25:43 +05:30
2021-09-04 01:27:46 +05:30
1. Migrate the file to LFS:
2021-01-03 14:25:43 +05:30
2021-09-04 01:27:46 +05:30
```shell
git lfs migrate import --yes --no-rewrite "<your-file>"
```
2021-01-03 14:25:43 +05:30
2021-09-04 01:27:46 +05:30
1. Push back to your repository:
```shell
git push
```
2022-01-26 12:08:38 +05:30
1. Optional. Clean up your `.git` folder:
2021-09-04 01:27:46 +05:30
```shell
git reflog expire --expire-unreachable=now --all
git gc --prune=now
```
2020-04-22 19:07:51 +05:30
### error: Repository or object not found
2022-07-16 23:28:13 +05:30
This error can occur for a few reasons, including:
2020-04-22 19:07:51 +05:30
- You don't have permissions to access certain LFS object
Check if you have permissions to push to the project or fetch from the project.
- Project is not allowed to access the LFS object
LFS object you are trying to push to the project or fetch from the project is not
available to the project anymore. Probably the object was removed from the server.
- Local Git repository is using deprecated LFS API
### Invalid status for `<url>` : 501
2021-03-11 19:13:27 +05:30
Git LFS logs the failures into a log file.
2020-04-22 19:07:51 +05:30
To view this log file, while in project directory:
```shell
git lfs logs last
```
If the status `error 501` is shown, it is because:
- Git LFS is not enabled in project settings. Check your project settings and
enable Git LFS.
- Git LFS support is not enabled on the GitLab server. Check with your GitLab
administrator why Git LFS is not enabled on the server. See
[LFS administration documentation](../../../administration/lfs/index.md) for instructions
on how to enable LFS support.
- Git LFS client version is not supported by GitLab server. Check your Git LFS
2020-06-23 00:09:42 +05:30
version with `git lfs version`. Check the Git configuration of the project for traces
of deprecated API with `git lfs -l`. If `batch = false` is set in the configuration,
2020-04-22 19:07:51 +05:30
remove the line and try to update your Git LFS client. Only version 1.0.1 and
newer are supported.
2022-07-23 23:45:48 +05:30
### `getsockopt: connection refused`
2021-03-11 19:13:27 +05:30
If you push an LFS object to a project and receive an error like this,
2020-04-22 19:07:51 +05:30
the LFS client is trying to reach GitLab through HTTPS. However, your GitLab
2021-03-11 19:13:27 +05:30
instance is being served on HTTP:
```plaintext
Post <URL>/info/lfs/objects/batch: dial tcp IP: getsockopt: connection refused
```
2020-04-22 19:07:51 +05:30
2020-05-24 23:13:21 +05:30
This behavior is caused by Git LFS using HTTPS connections by default when a
2020-06-23 00:09:42 +05:30
`lfsurl` is not set in the Git configuration.
2020-04-22 19:07:51 +05:30
2020-06-23 00:09:42 +05:30
To prevent this from happening, set the LFS URL in project Git configuration:
2020-04-22 19:07:51 +05:30
```shell
2022-08-27 11:52:29 +05:30
git config --add lfs.url "http://gitlab.example.com/group/my-sample-project.git/info/lfs"
2020-04-22 19:07:51 +05:30
```
### Credentials are always required when pushing an object
2021-02-22 17:27:13 +05:30
NOTE:
2020-04-22 19:07:51 +05:30
With 8.12 GitLab added LFS support to SSH. The Git LFS communication
still goes over HTTP, but now the SSH client passes the correct credentials
2021-03-11 19:13:27 +05:30
to the Git LFS client. No action is required by the user.
2020-04-22 19:07:51 +05:30
2021-03-11 19:13:27 +05:30
Git LFS authenticates the user with HTTP Basic Authentication on every push for
every object, so user HTTPS credentials are required.
2020-04-22 19:07:51 +05:30
By default, Git has support for remembering the credentials for each repository
2022-07-16 23:28:13 +05:30
you use. To learn more, read the [Git credentials man pages](https://git-scm.com/docs/gitcredentials).
2020-04-22 19:07:51 +05:30
For example, you can tell Git to remember the password for a period of time in
which you expect to push the objects:
```shell
git config --global credential.helper 'cache --timeout=3600'
```
2021-03-11 19:13:27 +05:30
This remembers the credentials for an hour, after which Git operations
2020-04-22 19:07:51 +05:30
require re-authentication.
If you are using OS X you can use `osxkeychain` to store and encrypt your credentials.
For Windows, you can use `wincred` or Microsoft's [Git Credential Manager for Windows](https://github.com/Microsoft/Git-Credential-Manager-for-Windows/releases).
More details about various methods of storing the user credentials can be found
on [Git Credential Storage documentation](https://git-scm.com/book/en/v2/Git-Tools-Credential-Storage).
### LFS objects are missing on push
GitLab checks files to detect LFS pointers on push. If LFS pointers are detected, GitLab tries to verify that those files already exist in LFS on GitLab.
Verify that LFS is installed locally and consider a manual push with `git lfs push --all`.
If you are storing LFS files outside of GitLab you can disable LFS on the project by setting `lfs_enabled: false` with the [projects API](../../../api/projects.md#edit-project).
### Hosting LFS objects externally
2020-05-24 23:13:21 +05:30
It is possible to host LFS objects externally by setting a custom LFS URL with `git config -f .lfsconfig lfs.url https://example.com/<project>.git/info/lfs`.
2020-04-22 19:07:51 +05:30
2022-07-23 23:45:48 +05:30
You might choose to do this if you are using an appliance like a Nexus Repository to store LFS data. If you choose to use an external LFS store,
2021-03-11 19:13:27 +05:30
GitLab can't verify LFS objects. Pushes then fail if you have GitLab LFS support enabled.
2020-04-22 19:07:51 +05:30
2021-03-11 19:13:27 +05:30
To stop push failure, LFS support can be disabled in the [Project settings](../../../user/project/settings/index.md), which also disables GitLab LFS value-adds (Verifying LFS objects, UI integration for LFS).