We wanted to be able to use the IAM role provided by the EC2 instance
metadata in order to access S3 via the Minio configuration. To do this,
a new credentials chain is added that will check the following locations
for credentials when an access key is not provided. In priority order,
they are:
1. MINIO_ prefixed environment variables
2. AWS_ prefixed environment variables
3. a minio credentials file
4. an aws credentials file
5. EC2 instance metadata
(cherry picked from commit c0880e7695346997c6a93f05cd01634cb3ad03ee)
Conflicts:
docs/content/administration/config-cheat-sheet.en-us.md
does not exist in Forgejo
(cherry picked from commit f9d75d4705ece5c119f2fd5e2bfbaf630d0bf739)
(cherry picked from commit 64f76f4ab269daa6a584305164732c68be0161c5)
(cherry picked from commit 5d024541551f1be98444923f0fa327a855104725)
[CI] Forgejo Actions workflows
(cherry picked from commit 3ff59b5379ebf761f32875f9d869a1d18f79741c)
(cherry picked from commit 8af826a6f7c70d03079ec68f10230041695017cd)
(cherry picked from commit d7c09d9cc80037a28d488da142ae2a2a99f59ac9)
[CI] use the docker label instead of ubuntu-latest
(cherry picked from commit b6a6470db6c8bae9963e204f9c8c408f309e81e3)
[CI] all tests need compliance before proceeding
(cherry picked from commit b35c496f2c3034164b9bb3a3550f35026adf9372)
(cherry picked from commit 36a4148a8ec654ae9fa5a6925a3b8606b96aebcc)
(cherry picked from commit 7ffcffa653808a284f422fdc31f6ea07874b585d)
(cherry picked from commit 8a246d296e502c181c6ee779d1773c9e78ca7acf)
(cherry picked from commit dd0b6e1826ba7699f967de2c00c1332909a53473)
[CI] Forgejo Actions based release process (squash) MySQL optimization
Refs: https://codeberg.org/forgejo/forgejo/issues/976
(cherry picked from commit b4b8c489e6e9fe36349eced5d4249467b25df2a7)
(cherry picked from commit 1e861db4afdc32549784381a96b16dac508ec1c0)
(cherry picked from commit a6c0e00330ded851c2f524039259f0b723d793e4)
(cherry picked from commit f97b336465fd6be15079e756c9ee3a580f0aa339)
(cherry picked from commit 6d65d5f0d6f798556a8f9e547896be03a5ee2f87)
(cherry picked from commit 79bfbadbed3bf84b96f2c027d119a4aa5c60d5e2)
(cherry picked from commit e86c40a34a97fc568c724ee5f8ee5b2bf46ebf19)
(cherry picked from commit da0c454adbcf1b5dbb95d9d74afaa371e32351a0)
(cherry picked from commit b49d892cda968b22ad6c29caade042690d96f2e2)
[CI] enable minio tests
(cherry picked from commit 4d8f4380311b872061bd20f561424662ecd8d817)
(cherry picked from commit c4eeb0a61ef579c21b0382edd61c62adebe93b28)
[CI] Forgejo Actions based CI for PR & branches (squash) cleanup
(cherry picked from commit 80eb20e84267552c971096693558734f0ec7afdd)
(cherry picked from commit d2ff589858961e4f25bc1d903cdf57df1db4c316)
(cherry picked from commit f6eedecb67fce992605bb1fcdd93c866ec47a9d4)
(cherry picked from commit cf458091e245e1c70944e940a214d5127f304303)
(cherry picked from commit ddd322cb2de165126080b222c1b09b2a91fea182)
(cherry picked from commit f0f5729b6467bea053d532c4c64c54a0fe737c98)
Conflicts:
.github/workflows/pull-db-tests.yml
https://codeberg.org/forgejo/forgejo/pulls/1573
(cherry picked from commit bb347aedd4e5030eea46ce22176a0444619df6b8)
(cherry picked from commit 0f5ebe8c3e78cef6f310ee9097a8413df904c08a)
(cherry picked from commit 9101cb4715ed88c6a3f0ad14443758d9bc9c61ed)
(cherry picked from commit ab118fe4efd96e9bb7134065643f094a379bb192)
[CI] Forgejo Actions based CI for PR & branches (squash) use node:20-bookworm
No longer use the custom test-env image, it is unecessary technical
debt.
Also upgrade to bitnami/minio:2023.8.31 to align with what Gitea tests
(cherry picked from commit d9b77fd2735a52043b4f8f1baaaa2e15073db621)
(cherry picked from commit ef8f099f9ea03d216324ceff348fe643240341dc)
(cherry picked from commit 6d3c675d203ba838de1f0540977f5e7e9f62f055)
(cherry picked from commit fc00ff45f2e810cc2a84bf57cdb2765830e5b07f)
(cherry picked from commit b7f02b9846b8c7087a4827a9c87f48eafd2ba1a2)
(cherry picked from commit d23d86e56728f4a263a0567c14e2dc20407baaff)
(cherry picked from commit e68e65460c40d343a3ada858c0c48d03a6a14bbb)
(cherry picked from commit 96f93a1fdb9f2c6ecbd05ac24c5375f3e0d631df)
(cherry picked from commit 2ff3080018633212075935ec6644654184009a2e)
(cherry picked from commit 8ea00bcff471b877e442f7b78bb46880c9e396f0)
(cherry picked from commit 85be961970b0d374d9816348a8a1b51289f7c787)
(cherry picked from commit 387b279fc33d79be1f0f0b35b32f5c17b0d64c9c)
(cherry picked from commit 219be7aa85fbf2bd5cb9daead3544b27f6200d83)
(cherry picked from commit 929b75ee39cd6d1a462bbd0c20597c357949c953)
(cherry picked from commit dd42a30d6a9e6cad26c9c916d733a0270d94a87e)
Should BucketExists (HeadBucket) fail because of an error related to
the connection rather than the existence of the bucket, no information
is available and the admin is left guessing.
https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadBucket.html
> This action is useful to determine if a bucket exists and you have
> permission to access it. The action returns a 200 OK if the bucket
> exists and you have permission to access it.
>
> If the bucket does not exist or you do not have permission to access
> it, the HEAD request returns a generic 400 Bad Request, 403
> Forbidden or 404 Not Found code. A message body is not included, so
> you cannot determine the exception beyond these error codes.
GetBucketVersioning is used instead and exclusively dedicated to
asserting if using the connection does not return a BadRequest.
If it does the NewMinioStorage logs an error and returns. Otherwise
it keeps going knowing that BucketExists is not going to fail for
reasons unrelated to the existence of the bucket and the permissions
to access it.
(cherry picked from commit d1df4b3bc62e5e61893a923f1c4b58f084eb03af)
Refs: https://codeberg.org/forgejo/forgejo/issues/1338
Previously, `err` was defined above, checked for `err == nil` and used
nowhere else.
Hence, the result of `convertMinioErr` would always be `nil`.
This leads to a NPE further down the line.
That is not intentional, it should convert the error of the most recent
operation, not one of its predecessors.
Found through
https://discord.com/channels/322538954119184384/322538954119184384/1143185780206993550.
For some reason, the permission of the client_id and secret may cannot
create bucket, so now we will check whether bucket does exist first and
then try to create a bucket if it doesn't exist.
Try to fix #25984
Co-authored-by: silverwind <me@silverwind.io>
The MinIO client isn't redirecting to the correct AWS endpoint if a
non-default data center is used.
In my use case I created an AWS bucket at `eu-central-1` region. Because
of the missing region initialization of the client the default
`us-east-1` API endpoint is used returning a `301 Moved Permanently`
response that's not handled properly by MinIO client. This in return
aborts using S3 storage on AWS as the `BucketExists()` call will fail
with the http moved error.
MinIO client trace shows the issue:
```text
---------START-HTTP---------
HEAD / HTTP/1.1
Host: xxxxxxxxxxx-prod-gitea-data.s3.dualstack.us-east-1.amazonaws.com
User-Agent: MinIO (windows; amd64) minio-go/v7.0.61
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20230809/accesspoint.eu-central-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230809T141143Z
HTTP/1.1 301 Moved Permanently
Connection: close
Content-Type: application/xml
Date: Wed, 09 Aug 2023 14:11:43 GMT
Server: AmazonS3
X-Amz-Bucket-Region: eu-central-1
X-Amz-Id-2: UK7wfeYi0HcTcytNvQ3wTAZ5ZP1mOSMnvRZ9Fz4xXzeNsS47NB/KfFx2unFxo3L7XckHpMNPPVo=
X-Amz-Request-Id: S1V2MJV8SZ11GEVN
---------END-HTTP---------
```
Co-authored-by: Heiko Besemann <heiko.besemann@qbeyond.de>
Follow up #22405
Fix #20703
This PR rewrites storage configuration read sequences with some breaks
and tests. It becomes more strict than before and also fixed some
inherit problems.
- Move storage's MinioConfig struct into setting, so after the
configuration loading, the values will be stored into the struct but not
still on some section.
- All storages configurations should be stored on one section,
configuration items cannot be overrided by multiple sections. The
prioioty of configuration is `[attachment]` > `[storage.attachments]` |
`[storage.customized]` > `[storage]` > `default`
- For extra override configuration items, currently are `SERVE_DIRECT`,
`MINIO_BASE_PATH`, `MINIO_BUCKET`, which could be configured in another
section. The prioioty of the override configuration is `[attachment]` >
`[storage.attachments]` > `default`.
- Add more tests for storages configurations.
- Update the storage documentations.
---------
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
minio storage iterator shows different behavior with local fs iterator.
in local fs storage:
``` go
s.IterateObjects("prefix", func(path,obj)
println(path) // show "prefix/xxx.file"
})
```
in minio storage:
```go
s.IterateObjects("prefix", func(path,obj)
println(path) // show "xxx.file"
})
```
I think local fs is correct, minio use wrong `basePath` to trim storage
path prefix.
---------
Co-authored-by: Giteabot <teabot@gitea.io>
Since #23493 has conflicts with latest commits, this PR is my proposal
for fixing #23371
Details are in the comments
And refactor the `modules/options` module, to make it always use
"filepath" to access local files.
Benefits:
* No need to do `util.CleanPath(strings.ReplaceAll(p, "\\", "/"))),
"/")` any more (not only one before)
* The function behaviors are clearly defined
Support to iterator subdirectory in ObjectStorage for
ObjectStorage.Iterator method.
It's required for https://github.com/go-gitea/gitea/pull/22738 to make
artifact files cleanable.
---------
Co-authored-by: Jason Song <i@wolfogre.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
As suggest by Go developers, use `filepath.WalkDir` instead of
`filepath.Walk` because [*Walk is less efficient than WalkDir,
introduced in Go 1.16, which avoids calling `os.Lstat` on every file or
directory visited](https://pkg.go.dev/path/filepath#Walk).
This proposition address that, in a similar way as
https://github.com/go-gitea/gitea/pull/22392 did.
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
The PR #21198 introduced a probable security vulnerability which
resulted in making all storage files be marked as executable.
This PR ensures that these are forcibly marked as non-executable.
Fix #22161
Signed-off-by: Andrew Thornton <art27@cantab.net>
A complement to #21985.
I overlooked it because the name of the switch is `StartServer`, not
`Enabled`. I believe the weird name is a legacy, but renaming is out of
scope.
Change all license headers to comply with REUSE specification.
Fix #16132
Co-authored-by: flynnnnnnnnnn <flynnnnnnnnnn@github>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
Go creates temporary files as 600, but sometimes we want the group to be able to read them (for example,
for another user to back up the storage)
This PR applies the umask to the renamed tmp files in local storage.
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Clean paths when looking in Storage
Ensure paths are clean for minio aswell as local storage.
Use url.Path not RequestURI/EscapedPath in storageHandler.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Apply suggestions from code review
Co-authored-by: Lauris BH <lauris@nix.lv>
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Storage.Iterate provides the path and an open object. On windows using
local storage means that the objects will be locked thus preventing clean
from deleting them.
This PR simply closes the objects early.
Fix #16932
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Retry rename on lock induced failures
Due to external locking on Windows it is possible for an
os.Rename to fail if the files or directories are being
used elsewhere.
This PR simply suggests retrying the rename again similar
to how we handle the os.Remove problems.
Fix #16427
Signed-off-by: Andrew Thornton <art27@cantab.net>
* resolve CI fail
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* _ to unused func options
* rm useless brakets
* rm trifial non used models functions
* rm dead code
* rm dead global vars
* fix routers/api/v1/repo/issue.go
* dont overload import module
This PR fixes several bugs in setting storage
* The default STORAGE_TYPE should be the provided type.
* The Storage config should be passed in to NewStorage as a pointer - otherwise the Mappable interface function MapTo will not be found
* There was a bug in the MapTo function.
Fix #13286
Signed-off-by: Andrew Thornton <art27@cantab.net>
Unfortunately there was a mistake in #13164 which fails to handle
os.PathError wrapping an os.ErrNotExist
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* The `.Use` of storageHandler before setting up the template renderer
causes a panic if there is an error to log.
* The error passed to `ctx.Error` in that case may contain sensitive
information and should not be rendered to the end user. We should
instead log the error and render a simple error message.
* There is no handling of missing avatars and this needs a 404. Minio
errors need to be mapped to standard golang errors such as
os.ErrNotExist.
* There is no logging when storage is set up.
Related #13159
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Avatar support minio
* Support repo avatar minio storage
* Add missing migration
* Fix bug
* Fix test
* Add test for minio store type on avatars and repo avatars; Add documents
* Fix bug
* Fix bug
* Add back missed avatar link method
* refactor codes
* Simplify the codes
* Code improvements
* Fix lint
* Fix test mysql
* Fix test mysql
* Fix test mysql
* Fix settings
* Fix test
* fix test
* Fix bug
* Provide self-registering storage system
Signed-off-by: Andrew Thornton <art27@cantab.net>
* More simplification
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Remove old strings from setting
Signed-off-by: Andrew Thornton <art27@cantab.net>
* oops attachments not attachment
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
