forgejo-federation/routers/api/packages
Gusted fa37a211fb
[GITEA] Drop sha256-simd in favor of stdlib
- In Go 1.21 the crypto/sha256 [got a massive
improvement](https://go.dev/doc/go1.21#crypto/sha256) by utilizing the
SHA instructions for AMD64 CPUs, which sha256-simd already was doing.
The performance is now on par and I think it's preferable to use the
standard library rather than a package when possible.

```
cpu: AMD Ryzen 5 3600X 6-Core Processor
                │  simd.txt   │               go.txt                │
                │   sec/op    │    sec/op     vs base               │
Hash/8Bytes-12    63.25n ± 1%    73.38n ± 1%  +16.02% (p=0.002 n=6)
Hash/64Bytes-12   98.73n ± 1%   105.30n ± 1%   +6.65% (p=0.002 n=6)
Hash/1K-12        567.2n ± 1%    572.8n ± 1%   +0.99% (p=0.002 n=6)
Hash/8K-12        4.062µ ± 1%    4.062µ ± 1%        ~ (p=0.396 n=6)
Hash/1M-12        512.1µ ± 0%    510.6µ ± 1%        ~ (p=0.485 n=6)
Hash/5M-12        2.556m ± 1%    2.564m ± 0%        ~ (p=0.093 n=6)
Hash/10M-12       5.112m ± 0%    5.127m ± 0%        ~ (p=0.093 n=6)
geomean           13.82µ         14.27µ        +3.28%

                │   simd.txt   │               go.txt                │
                │     B/s      │     B/s       vs base               │
Hash/8Bytes-12    120.6Mi ± 1%   104.0Mi ± 1%  -13.81% (p=0.002 n=6)
Hash/64Bytes-12   618.2Mi ± 1%   579.8Mi ± 1%   -6.22% (p=0.002 n=6)
Hash/1K-12        1.682Gi ± 1%   1.665Gi ± 1%   -0.98% (p=0.002 n=6)
Hash/8K-12        1.878Gi ± 1%   1.878Gi ± 1%        ~ (p=0.310 n=6)
Hash/1M-12        1.907Gi ± 0%   1.913Gi ± 1%        ~ (p=0.485 n=6)
Hash/5M-12        1.911Gi ± 1%   1.904Gi ± 0%        ~ (p=0.093 n=6)
Hash/10M-12       1.910Gi ± 0%   1.905Gi ± 0%        ~ (p=0.093 n=6)
geomean           1.066Gi        1.032Gi        -3.18%
```

(cherry picked from commit abd94ff5b59c86e793fd9bf12187ea6cfd1f3fa1)
(cherry picked from commit 15e81637abf70576a564cf9eecaa9640228afb5b)

Conflicts:
	go.mod
	https://codeberg.org/forgejo/forgejo/pulls/1581
(cherry picked from commit 325d92917f655c999b81b08832ee623d6b669f0f)

Conflicts:
	modules/context/context_cookie.go
	https://codeberg.org/forgejo/forgejo/pulls/1617
(cherry picked from commit 358819e8959886faa171ac16541097500d0a703e)
(cherry picked from commit 362fd7aae17832fa922fa017794bc564ca43060d)
(cherry picked from commit 4f64ee294ee05c93042b6ec68f0a179ec249dab9)
(cherry picked from commit 4bde77f7b13c5f961c141c01b6da1f9eda5ec387)
(cherry picked from commit 1311e30a811675eb623692349e4e808a85aabef6)
(cherry picked from commit 57b69e334c2973118488b9b5dbdc8a2c88135756)
(cherry picked from commit 52dc892fadecf39e89c3c351edc9efb42522257b)
(cherry picked from commit 77f54f4187869c6eabcc837742fd3f908093a76c)
(cherry picked from commit 0d0392f3a510ce3683bb649dee1e65b45dd91354)

Conflicts:
	go.mod
	https://codeberg.org/forgejo/forgejo/pulls/2034
(cherry picked from commit 92798364e8fe3188a2100b54f3adea943f8309e9)
(cherry picked from commit 43d218127752aa9251c4c3ef71b9c060f109dffc)
(cherry picked from commit 45c88b86a35729fc0b2dc6b72bc33caf9f69265f)
(cherry picked from commit a1cd6f4e3a7956773cbc0aef8abb80d17b62eb49)
(cherry picked from commit 01191dc2adf8c57ae448be37e73158005a8ff74d)
(cherry picked from commit 151e07f37e2854ad633f1352fb0ce3cd06f4b2ae)
2024-02-05 16:09:40 +01:00
..
alpine Fix the wrong HTTP response status code for duplicate packages (#27480) 2023-10-10 15:39:58 +02:00
cargo Do not force creation of _cargo-index repo on publish (#27266) 2023-10-24 03:26:38 +00:00
chef [GITEA] Drop sha256-simd in favor of stdlib 2024-02-05 16:09:40 +01:00
composer Fix the wrong HTTP response status code for duplicate packages (#27480) 2023-10-10 15:39:58 +02:00
conan Fix the wrong HTTP response status code for duplicate packages (#27480) 2023-10-10 15:39:58 +02:00
conda Another round of db.DefaultContext refactor (#27103) 2023-09-25 13:17:37 +00:00
container Prevent anonymous container access if RequireSignInView is enabled (#28877) 2024-01-21 16:31:29 +00:00
cran Another round of db.DefaultContext refactor (#27103) 2023-09-25 13:17:37 +00:00
debian Fix the wrong HTTP response status code for duplicate packages (#27480) 2023-10-10 15:39:58 +02:00
generic Another round of db.DefaultContext refactor (#27103) 2023-09-25 13:17:37 +00:00
goproxy Another round of db.DefaultContext refactor (#27103) 2023-09-25 13:17:37 +00:00
helm Another round of db.DefaultContext refactor (#27103) 2023-09-25 13:17:37 +00:00
helper Replace interface{} with any (#25686) 2023-07-04 18:36:08 +00:00
maven [GITEA] Drop sha256-simd in favor of stdlib 2024-02-05 16:09:40 +01:00
npm Fix the wrong HTTP response status code for duplicate packages (#27480) 2023-10-10 15:39:58 +02:00
nuget [GITEA] feat(nuget): basic manifest download 2024-02-05 15:08:04 +01:00
pub Fix the wrong HTTP response status code for duplicate packages (#27480) 2023-10-10 15:39:58 +02:00
pypi Fix the wrong HTTP response status code for duplicate packages (#27480) 2023-10-10 15:39:58 +02:00
rpm Fix some RPM registry flaws (#28782) 2024-01-19 11:37:10 +00:00
rubygems Fix the wrong HTTP response status code for duplicate packages (#27480) 2023-10-10 15:39:58 +02:00
swift Another round of db.DefaultContext refactor (#27103) 2023-09-25 13:17:37 +00:00
vagrant Another round of db.DefaultContext refactor (#27103) 2023-09-25 13:17:37 +00:00
api.go Fix some RPM registry flaws (#28782) 2024-01-19 11:37:10 +00:00
README.md Add package registry architecture overview (#23445) 2023-03-13 18:15:09 -04:00

Gitea Package Registry

This document gives a brief overview how the package registry is organized in code.

Structure

The package registry code is divided into multiple modules to split the functionality and make code reuse possible.

Module Description
models/packages Common methods and models used by all registry types
models/packages/<type> Methods used by specific registry type. There should be no need to use type specific models.
modules/packages Common methods and types used by multiple registry types
modules/packages/<type> Registry type specific methods and types (e.g. metadata extraction of package files)
routers/api/packages Route definitions for all registry types
routers/api/packages/<type> Route implementation for a specific registry type
services/packages Helper methods used by registry types to handle common tasks like package creation and deletion in routers
services/packages/<type> Registry type specific methods used by routers and services

Models

Every package registry implementation uses the same underlaying models:

Model Description
Package The root of a package providing values fixed for every version (e.g. the package name)
PackageVersion A version of a package containing metadata (e.g. the package description)
PackageFile A file of a package describing its content (e.g. file name)
PackageBlob The content of a file (may be shared by multiple files)
PackageProperty Additional properties attached to Package, PackageVersion or PackageFile (e.g. used if metadata is needed for routing)

The following diagram shows the relationship between the models:

Package <1---*> PackageVersion <1---*> PackageFile <*---1> PackageBlob

Adding a new package registry type

Before adding a new package registry type have a look at the existing implementation to get an impression of how it could work. Most registry types offer endpoints to retrieve the metadata, upload and download package files. The upload endpoint is often the heavy part because it must validate the uploaded blob, extract metadata and create the models. The methods to validate and extract the metadata should be added in the modules/packages/<type> package. If the upload is valid the methods in services/packages allow to store the upload and create the corresponding models. It depends if the registry type allows multiple files per package version which method should be called:

  • CreatePackageAndAddFile: error if package version already exists
  • CreatePackageOrAddFileToExisting: error if file already exists
  • AddFileToExistingPackage: error if package version does not exist or file already exists

services/packages also contains helper methods to download a file or to remove a package version. There are no helper methods for metadata endpoints because they are very type specific.