Commit graph

346 commits

Author SHA1 Message Date
wxiaoguang
d98c5db58f
alternative to PR "improve code quality" (#21464)
This PR doesn't require new git version, and can be backported easily.

Co-authored-by: 6543 <6543@obermui.de>
2022-10-15 12:49:26 +02:00
KN4CK3R
0e57ff7eee
Add generic set type (#21408)
This PR adds a generic set type to get rid of maps used as sets.

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2022-10-12 13:18:26 +08:00
zeripath
3ccebf7f40
Stop logging CheckPath returns error: context canceled (#21064)
We should only log CheckPath errors if they are not simply due to
context cancellation - and we should add a little more context to the
error message.

Fix #20709

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-10-10 21:54:30 +01:00
wxiaoguang
c08e42c47e
Refactor parseTreeEntries, speed up tree list (#21368)
Close #20315 (fix the panic when parsing invalid input), Speed up #20231 (use ls-tree without size field)

Introduce ListEntriesRecursiveFast (ls-tree without size) and ListEntriesRecursiveWithSize (ls-tree with size)
2022-10-08 01:20:53 +08:00
wxiaoguang
321964155a
Treat git object mode 40755 as directory (#21195)
Git uses 040000 for tree object, but some users may get 040755 for
unknown reasons

Try to fix #21190
* #21190
2022-09-18 09:31:20 +08:00
wxiaoguang
0ba2f53ca1
Passing command line arguments correctly by string slice (#21168)
Using `append(args, strings.Fields(arg)...)` is dangerous, it may
generate incorrect results.

For example: `arg1 "the dangerous"` will be splitted to 3 arguments:
`arg1`, `"the`, `dangerous"`. In some cases the incorrect arguments may
lead to security problems.
2022-09-14 15:59:52 +02:00
sergemedvid
e07d089be0
Sort branches and tags by date descending (#21136)
This fixes #5709 and #17316 by changing the order of listed branches
and tags to show the ones with latest commits atop.
It's achieved with changing underlying "show-ref" git command with
"for-each-ref" as suggested in https://stackoverflow.com/a/5188364
Also, it's passing format string so the output matches "show-ref"
command output.

close #5709
close #17316
2022-09-14 14:11:24 +02:00
Eng Zer Jun
8b0aaa5f86
test: use T.TempDir to create temporary test directory (#21043)
A testing cleanup. 

This pull request replaces `os.MkdirTemp` with `t.TempDir`. We can use the `T.TempDir` function from the `testing` package to create temporary directory. The directory created by `T.TempDir` is automatically removed when the test and all its subtests complete. 

This saves us at least 2 lines (error check, and cleanup) on every instance, or in some cases adds cleanup that we forgot.

Reference: https://pkg.go.dev/testing#T.TempDir

```go
func TestFoo(t *testing.T) {
	// before
	tmpDir, err := os.MkdirTemp("", "")
	require.NoError(t, err)
	defer os.RemoveAll(tmpDir)

	// now
	tmpDir := t.TempDir()
}
```

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-09-04 16:14:53 +01:00
zeripath
c722a26e7e
Set uploadpack.allowFilter etc on gitea serv to enable partial clones with ssh (#20902)
When setting.Git.DisablePartialClone is set to false then the web server will add filter support to web http. It does this by using`-c` command arguments but this will not work on gitea serv as the upload-pack and receive-pack commands do not support this.
    
Instead we move these options into the .gitconfig instead.

Fix #20400

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-09-04 16:13:23 +01:00
zeripath
e6b3be4608
Add more checks in migration code (#21011)
When migrating add several more important sanity checks:

* SHAs must be SHAs
* Refs must be valid Refs
* URLs must be reasonable

Signed-off-by: Andrew Thornton <art27@cantab.net>

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: techknowlogick <matti@mdranta.net>
2022-09-04 13:47:56 +03:00
John Olheiser
a48d6ba4b4
Go 1.19 format (#20758)
* 1.19 gofumpt

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Change CSV test

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Commit whitespace fixes from @zeripath

Co-authored-by: zeripath <art27@cantab.net>

* Update emoji

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* bump swagger & fix generate-swagger

Signed-off-by: jolheiser <john.olheiser@gmail.com>
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lauris BH <lauris@nix.lv>
2022-08-30 21:15:45 -05:00
zeripath
aa2e473991
Set no-tags in git fetch on compare (#20893)
* Set no-tags in git fetch on compare

In the compare endpoint the git fetch is restricted to a certain branch however,
this does not completely prevent tag acquisition/pollution as git fetch will collect
any tags on that branch.

This causes pollution of the tag namespace and could cause confusion by users.

This PR adds `--no-tags` to the `git fetch` call.

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Update modules/git/repo_compare.go

* Update modules/git/repo_compare.go

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-08-23 14:12:43 -04:00
zeripath
58a4407acb
Remove follow from commits by file (#20765)
The use of `--follow` makes getting these commits very slow on large repositories
as it results in searching the whole commit tree for a blob.

Now as nice as the results of `--follow` are, I am uncertain whether it is really
of sufficient importance to keep around.

Fix #20764

Signed-off-by: Andrew Thornton <art27@cantab.net>

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2022-08-14 21:22:13 -04:00
wxiaoguang
75d96f4a02
Refactor legacy git init (#20376)
* merge `CheckLFSVersion` into `InitFull` (renamed from `InitWithSyncOnce`)
* remove the `Once` during git init, no data-race now
* for doctor sub-commands, `InitFull` should only be called in initialization stage

Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2022-08-09 11:22:24 +08:00
parnic
27789908d8
Use request timeout for git service rpc (#20689)
This enables git.Command's Run to optionally use the given context directly so its deadline will be respected. Otherwise, it falls back to the previous behavior of using the supplied timeout or a default timeout value of 360 seconds.

repo's serviceRPC() calls now use the context's deadline (which is unset/unlimited) instead of the default 6-minute timeout. This means that large repo clones will no longer arbitrarily time out on the upload-pack step, and pushes can take longer than 6 minutes on the receive-pack step.

Fixes #20680

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2022-08-06 16:13:11 +03:00
Gusted
692707f145
Add latest commit's SHA to content response (#20398)
* Add latest commit's SHA to content response

- When requesting the contents of a filepath, add the latest commit's
SHA to the requested file.
- Resolves #12840

* Add swagger

* Fix NPE

* Fix tests

* Hook into LastCommitCache

* Move AddLastCommitCache to a common nogogit and gogit file

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Prevent NPE

Co-authored-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2022-07-30 16:09:04 +08:00
zeripath
a2cfcdb91a
Slightly simplify LastCommitCache (#20444)
The LastCommitCache code is a little complex and there is unnecessary
duplication between the gogit and nogogit variants.

This PR adds the LastCommitCache as a field to the git.Repository and
pre-creates it in the ReferencesGit helpers etc. There has been some
simplification and unification of the variant code.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-07-25 16:39:42 +01:00
wxiaoguang
fee0e4dbea
Remove confusing TrimPrefix(... git.BranchPrefix) (#20369)
Make Repository.GetDefaultBranch return the real branch name, instead of the ref name. Then there is no need to do TrimPrefix for repo.DefaultBranch
2022-07-16 08:10:02 +08:00
silverwind
7d20c8323a
Fix commit status icon when in subdirectory (#20285)
When viewing a subdirectory and the latest commit to that directory in
the table, the commit status icon incorrectly showed the status of the
HEAD commit instead of the latest for that directory.
2022-07-15 14:01:32 +01:00
zeripath
fe09ee564d
Prevent context deadline error propagation in GetCommitsInfo (#20346)
* Prevent context deadline error propagation in GetCommitsInfo

Although `WalkGitLog` tries to test for `context.DeadlineExceededErr`
there is a small chance that the error will propagate to the reader
before it is recognised. This will cause the error to propagate up to
`renderDirectoryFiles` and cause a http status 500.

Here we check that the error passed is a `DeadlineExceededErr` via error.Is

Fix #20329

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-07-14 00:24:29 +02:00
wxiaoguang
496b8e3990
Use git.HOME_PATH for Git HOME directory (#20114)
* Add git.HOME_PATH

* add legacy file check

* Apply suggestions from code review

Co-authored-by: zeripath <art27@cantab.net>

* pass env GNUPGHOME to git command, move the existing .gitconfig to new home, make the fix for 1.17rc more clear.

* set git.HOME_PATH for docker images to default HOME

* Revert "set git.HOME_PATH for docker images to default HOME"

This reverts commit f120101ddc267cef74e4f4b92c783d5fc8e275a1.

* force Gitea to use a stable GNUPGHOME directory

* extra check to ensure only process dir or symlink for legacy files

* refactor variable name

* The legacy dir check (for 1.17-rc1) could be removed with 1.18 release, since users should have upgraded from 1.17-rc to 1.17-stable

* Update modules/git/git.go

Co-authored-by: Steven Kriegler <61625851+justusbunsi@users.noreply.github.com>

* remove initFixGitHome117rc

* Update git.go

* Update docs/content/doc/advanced/config-cheat-sheet.en-us.md

Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Steven Kriegler <61625851+justusbunsi@users.noreply.github.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2022-07-08 16:09:07 +08:00
Wim
cb50375e2b
Add more linters to improve code readability (#19989)
Add nakedret, unconvert, wastedassign, stylecheck and nolintlint linters to improve code readability

- nakedret - https://github.com/alexkohler/nakedret - nakedret is a Go static analysis tool to find naked returns in functions greater than a specified function length.
- unconvert - https://github.com/mdempsky/unconvert - Remove unnecessary type conversions
- wastedassign - https://github.com/sanposhiho/wastedassign -  wastedassign finds wasted assignment statements.
- notlintlint -  Reports ill-formed or insufficient nolint directives
- stylecheck - https://staticcheck.io/docs/checks/#ST - keep style consistent
  - excluded: [ST1003 - Poorly chosen identifier](https://staticcheck.io/docs/checks/#ST1003) and [ST1005 - Incorrectly formatted error string](https://staticcheck.io/docs/checks/#ST1005)
2022-06-20 12:02:49 +02:00
wxiaoguang
05a74e6e22
use quoted regexp instead of git fixed-value (#20029) 2022-06-19 12:56:22 +01:00
zeripath
ea9997a9dd
Add fetch.writeCommitGraph to gitconfig (#20006)
Add fetch.writeCommitGraph to gitconfig to ensure that a commit-graph will be written
on git fetch calls.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-06-17 21:18:35 +01:00
singuliere
a036507204
gitconfig: set safe.directory = * (#19870)
Allow git push to work when networked file systems with mixed
ownership are used with Gitea docker images >= 1.16.6 or Gitea
binaries running alongside git versions published after 04/2022.

There are circumstances independent of Gitea (networked file systems
with various permission systems) by which the git repositories managed
by Gitea may have mixed owners. It is not a behavior that Gitea have
control over nor is it a problem as long as the permissions for Gitea to
operate are correct. Gitea instances have been operating under these
conditions for a number of years.

It is detected as a potential security risk ( see
GHSA-vw2c-22j4-2fh2
) by the most recent git versions. However, Gitea always runs git
commands with a current directory matching the repository on
which it operates. That makes Gitea immune from this security problem
and it is safe to ignore the mixed owner permission check.

This gitconfig modification is done on a file dedicated to the user
exclusively used by Gitea.

Fixes: #19455

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: zeripath <art27@cantab.net>
2022-06-17 06:49:38 +01:00
wxiaoguang
157b405753
Remove legacy git code (ver < 2.0), fine tune markup tests (#19930)
* clean git support for ver < 2.0

* fine tune tests for markup (which requires git module)

* remove unnecessary comments

* try to fix tests

* try test again

* use const for GitVersionRequired instead of var

* try to fix integration test

* Refactor CheckAttributeReader to make a *git.Repository version

* update document for commit signing with Gitea's internal gitconfig

* update document for commit signing with Gitea's internal gitconfig

Co-authored-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2022-06-16 23:47:44 +08:00
Lunny Xiao
ce3dd04c63
Fix some mirror bugs (#18649)
* Fix some mirror bugs

* Remove unnecessary code

* Fix lint

* rename stdard url

* Allow more charactors in git ssh protocol url

* improve the detection

* support ipv6 for git url parse

* Fix bug

* Fix template

* Fix bug

* fix template

* Fix tmpl

* Fix tmpl

* Fix parse ssh with interface

* Rename functions name

Co-authored-by: zeripath <art27@cantab.net>
2022-06-11 21:50:14 +08:00
wxiaoguang
88f2e457d8
Fix data-race problems in git module (quick patch) (#19934)
* Fix data-race problems in git module

* use HomeDir instead of setting.RepoRootPath

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2022-06-11 11:56:27 +08:00
wxiaoguang
a0051634b9
Refactor git module, make Gitea use internal git config (#19732)
* Refactor git module, make Gitea use internal git config, add safe.directory config

* introduce git.InitSimple and git.InitWithConfigSync, make serv cmd use gitconfig

* use HOME instead of GIT_CONFIG_GLOBAL, because git always needs a correct HOME

* fix cmd env in cmd/serv.go

* fine tune error message

* Fix a incorrect test case

* fix configAddNonExist

* fix configAddNonExist logic, add `--fixed-value` flag, add tests

* add configSetNonExist function in case it's needed.

* use configSetNonExist for `user.name` and `user.email`

* add some comments

* Update cmd/serv.go

Co-authored-by: zeripath <art27@cantab.net>

* Update cmd/serv.go

Co-authored-by: zeripath <art27@cantab.net>

* Update modules/git/git.go

Co-authored-by: zeripath <art27@cantab.net>

* Update modules/setting/setting.go

Co-authored-by: zeripath <art27@cantab.net>

* Update modules/git/repo_attribute.go

Co-authored-by: zeripath <art27@cantab.net>

* fix spaces in messages

* use `configSet("core.protectNTFS", ...)` instead of `globalCommandArgs`

* remove GIT_CONFIG_NOSYSTEM, continue to use system's git config

* Update cmd/serv.go

Co-authored-by: zeripath <art27@cantab.net>

* fix merge

* remove code for safe.directory

* separate git.CommonEnvs to CommonGitCmdEnvs and CommonCmdServEnvs

* avoid Golang's data race error

Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2022-06-10 09:57:49 +08:00
Lunny Xiao
dbe415fb91
Only log non ErrNotExist errors in git.GetNote (#19884)
* Fix GetNote

* Only log errors if the error is not ErrNotExist

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Andrew Thornton <art27@cantab.net>
2022-06-07 16:39:50 +08:00
zeripath
1d04e8641d
Set Setpgid on child git processes (#19865)
When Gitea is running as PID 1 git will occassionally orphan child processes leading
to (defunct) processes. This PR simply sets Setpgid to true on these child processes
meaning that these defunct processes will also be correctly reaped.

Fix #19077

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-06-03 15:36:18 +01:00
Lunny Xiao
4344a64107
Allow custom default merge message with .gitea/default_merge_message/<merge_style>_TEMPLATE.md (#18177)
* Allow custom default merge message with .gitea/MERGE_MESSAGE_<merge_style>_TEMPLATE.md

* Some improvements

* Follow some advices

* Fix bug

* Fix bug

* Fix lint

* Fix close comment

* Fix test

* Fix and docs

* Improve codes

* Update docs and remove unnecessary variables

* return error for GetDefaultMergeMessage

* Fix test

* improve code

* ignore unknow unit type

* return error for GetDefaultMergeMessage

* Update services/pull/merge.go

* Some improvements

* Follow some advices

* Fix bug

* Fix lint

* Improve codes

* Update docs and remove unnecessary variables

* return error for GetDefaultMergeMessage

* improve code

* Handle deleted HeadRepo in GetDefaultMergeMessage

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Fix test

* Fix test

Co-authored-by: zeripath <art27@cantab.net>
2022-05-08 20:32:45 +08:00
delvh
5ca224a789
Allow to mark files in a PR as viewed (#19007)
Users can now mark files in PRs as viewed, resulting in them not being shown again by default when they reopen the PR again.
2022-05-07 20:28:10 +02:00
kolaente
59b30f060a
Auto merge pull requests when all checks succeeded via API (#9307)
* Fix indention

Signed-off-by: kolaente <k@knt.li>

* Add option to merge a pr right now without waiting for the checks to succeed

Signed-off-by: kolaente <k@knt.li>

* Fix lint

Signed-off-by: kolaente <k@knt.li>

* Add scheduled pr merge to tables used for testing

Signed-off-by: kolaente <k@knt.li>

* Add status param to make GetPullRequestByHeadBranch reusable

Signed-off-by: kolaente <k@knt.li>

* Move "Merge now" to a seperate button to make the ui clearer

Signed-off-by: kolaente <k@knt.li>

* Update models/scheduled_pull_request_merge.go

Co-authored-by: 赵智超 <1012112796@qq.com>

* Update web_src/js/index.js

Co-authored-by: 赵智超 <1012112796@qq.com>

* Update web_src/js/index.js

Co-authored-by: 赵智超 <1012112796@qq.com>

* Re-add migration after merge

* Fix frontend lint

* Fix version compare

* Add vendored dependencies

* Add basic tets

* Make sure the api route is capable of scheduling PRs for merging

* Fix comparing version

* make vendor

* adopt refactor

* apply suggestion: User -> Doer

* init var once

* Fix Test

* Update templates/repo/issue/view_content/comments.tmpl

* adopt

* nits

* next

* code format

* lint

* use same name schema; rm CreateUnScheduledPRToAutoMergeComment

* API: can not create schedule twice

* Add TestGetBranchNamesForSha

* nits

* new go routine for each pull to merge

* Update models/pull.go

Co-authored-by: a1012112796 <1012112796@qq.com>

* Update models/scheduled_pull_request_merge.go

Co-authored-by: a1012112796 <1012112796@qq.com>

* fix & add renaming sugestions

* Update services/automerge/pull_auto_merge.go

Co-authored-by: a1012112796 <1012112796@qq.com>

* fix conflict relicts

* apply latest refactors

* fix: migration after merge

* Update models/error.go

Co-authored-by: delvh <dev.lh@web.de>

* Update options/locale/locale_en-US.ini

Co-authored-by: delvh <dev.lh@web.de>

* Update options/locale/locale_en-US.ini

Co-authored-by: delvh <dev.lh@web.de>

* adapt latest refactors

* fix test

* use more context

* skip potential edgecases

* document func usage

* GetBranchNamesForSha() -> GetRefsBySha()

* start refactoring

* ajust to new changes

* nit

* docu nit

* the great check move

* move checks for branchprotection into own package

* resolve todo now ...

* move & rename

* unexport if posible

* fix

* check if merge is allowed before merge on scheduled pull

* debugg

* wording

* improve SetDefaults & nits

* NotAllowedToMerge -> DisallowedToMerge

* fix test

* merge files

* use package "errors"

* merge files

* add string names

* other implementation for gogit

* adapt refactor

* more context for models/pull.go

* GetUserRepoPermission use context

* more ctx

* use context for loading pull head/base-repo

* more ctx

* more ctx

* models.LoadIssueCtx()

* models.LoadIssueCtx()

* Handle pull_service.Merge in one DB transaction

* add TODOs

* next

* next

* next

* more ctx

* more ctx

* Start refactoring structure of old pull code ...

* move code into new packages

* shorter names ... and finish **restructure**

* Update models/branches.go

Co-authored-by: zeripath <art27@cantab.net>

* finish UpdateProtectBranch

* more and fix

* update datum

* template: use "svg" helper

* rename prQueue 2 prPatchCheckerQueue

* handle automerge in queue

* lock pull on git&db actions ...

* lock pull on git&db actions ...

* add TODO notes

* the regex

* transaction in tests

* GetRepositoryByIDCtx

* shorter table name and lint fix

* close transaction bevore notify

* Update models/pull.go

* next

* CheckPullMergable check all branch protections!

* Update routers/web/repo/pull.go

* CheckPullMergable check all branch protections!

* Revert "PullService lock via pullID (#19520)" (for now...)

This reverts commit 6cde7c9159a5ea75a10356feb7b8c7ad4c434a9a.

* Update services/pull/check.go

* Use for a repo action one database transaction

* Apply suggestions from code review

* Apply suggestions from code review

Co-authored-by: delvh <dev.lh@web.de>

* Update services/issue/status.go

Co-authored-by: delvh <dev.lh@web.de>

* Update services/issue/status.go

Co-authored-by: delvh <dev.lh@web.de>

* use db.WithTx()

* gofmt

* make pr.GetDefaultMergeMessage() context aware

* make MergePullRequestForm.SetDefaults context aware

* use db.WithTx()

* pull.SetMerged only with context

* fix deadlock in `test-sqlite\#TestAPIBranchProtection`

* dont forget templates

* db.WithTx allow to set the parentCtx

* handle db transaction in service packages but not router

* issue_service.ChangeStatus just had caused another deadlock :/
it has to do something with how notification package is handled

* if we merge a pull in one database transaktion, we get a lock, because merge infoce internal api that cant handle open db sessions to the same repo

* ajust to current master

* Apply suggestions from code review

Co-authored-by: delvh <dev.lh@web.de>

* dont open db transaction in router

* make generate-swagger

* one _success less

* wording nit

* rm

* adapt

* remove not needed test files

* rm less diff & use attr in JS

* ...

* Update services/repository/files/commit.go

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>

* ajust db schema for PullAutoMerge

* skip broken pull refs

* more context in error messages

* remove webUI part for another pull

* remove more WebUI only parts

* API: add CancleAutoMergePR

* Apply suggestions from code review

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>

* fix lint

* Apply suggestions from code review

* cancle -> cancel

Co-authored-by: delvh <dev.lh@web.de>

* change queue identifyer

* fix swagger

* prevent nil issue

* fix and dont drop error

* as per @zeripath

* Update integrations/git_test.go

Co-authored-by: delvh <dev.lh@web.de>

* Update integrations/git_test.go

Co-authored-by: delvh <dev.lh@web.de>

* more declarative integration tests (dedup code)

* use assert.False/True helper

Co-authored-by: 赵智超 <1012112796@qq.com>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2022-05-08 01:05:52 +08:00
6543
92f139d091
Use for a repo action one database transaction (#19576)
... more context

(part of #9307)
2022-05-03 21:46:28 +02:00
wxiaoguang
c18d8d6968
Remove legacy +build: constraint (#19582)
Go 1.17 and later use modern `//go:build` constraints, the old `// +build:` constraints should be removed.
2022-05-02 23:22:45 +08:00
wxiaoguang
509d811243
Upgrade required git version to 2.0 (#19577)
* Upgrade required git version to 2.0

* update document
2022-05-02 20:30:24 +08:00
zeripath
332b2ecd21
Prevent intermittent race in attribute reader close (#19537)
There is a potential rare race possible whereby the c.running channel could
be closed twice. Looking at the code I do not see a need for this c.running
channel and therefore I think we can remove this. (I think the c.running
might have been some attempt to prevent a hang but the use of os.Pipes should
prevent that.)

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: 6543 <6543@obermui.de>
2022-04-28 11:19:33 +08:00
6543
1ebb30e41b
Pass gitRepo down to GetRawDiff, since its used for main repo and wiki (#19461)
as per https://github.com/go-gitea/gitea/pull/19449#issuecomment-1105283931

pass gitRepo down to GetRawDiff, since its used for main repo and wiki
2022-04-25 20:45:18 +02:00
zeripath
0dcc74a8a7
Prevent dangling cat-file calls (goroutine alternative) (#19454)
If an `os/exec.Command` is passed non `*os.File` as an input/output, go
will create `os.Pipe`s and wait for their closure in `cmd.Wait()`.  If
the code following this is responsible for closing `io.Pipe`s or other
handlers then on process death from context cancellation the `Wait` can
hang.

There are two possible solutions:

1. use `os.Pipe` as the input/output as `cmd.Wait` does not wait for these.
2. create a goroutine waiting on the context cancellation that will close the inputs.

This PR provides the second option - which is a simpler change that can
be more easily backported.

Closes #19448

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-04-22 17:20:04 +02:00
zeripath
a7f0ce6207
Add uploadpack.allowAnySHA1InWant to allow --filter=blob:none with older git clients (#19430)
Older git clients need uploadpack.allowAnySHA1InWant if partial cloning is allowed.

Fix #19118

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-04-20 17:52:16 +02:00
wxiaoguang
124b072f0b
Remove git.Command.Run and git.Command.RunInDir* (#19280)
Follows #19266, #8553, Close #18553, now there are only three `Run..(&RunOpts{})` functions.
 * before: `stdout, err := RunInDir(path)`
 * now: `stdout, _, err := RunStdString(&git.RunOpts{Dir:path})`
2022-04-01 10:55:30 +08:00
Peter Gardfjäll
e28cc79c92
Improve sync performance for pull-mirrors (#19125)
This addresses https://github.com/go-gitea/gitea/issues/18352

It aims to improve performance (and resource use) of the `SyncReleasesWithTags` operation for pull-mirrors.

For large repositories with many tags, `SyncReleasesWithTags` can be a costly operation (taking several minutes to complete). The reason is two-fold:
    
1. on sync, every upstream repo tag is compared (for changes) against existing local entries in the release table to ensure that they are up-to-date.
    
2. the procedure for getting _each tag_ involves a series of git operations    
    ```bash
     git show-ref --tags -- v8.2.4477
     git cat-file -t 29ab6ce9f36660cffaad3c8789e71162e5db5d2f
     git cat-file -p 29ab6ce9f36660cffaad3c8789e71162e5db5d2f
     git rev-list --count 29ab6ce9f36660cffaad3c8789e71162e5db5d2f
     ```    

     of which the `git rev-list --count` can be particularly heavy.
    
This PR optimizes performance for pull-mirrors. We utilize the fact that a pull-mirror is always identical to its upstream and rebuild the entire release table on every sync and use a batch `git for-each-ref .. refs/tags` call to retrieve all tags in one go.
    
For large mirror repos, with hundreds of annotated tags, this brings down the duration of the sync operation from several minutes to a few seconds. A few unscientific examples run on my local machine:

- https://github.com/spring-projects/spring-boot (223 tags)
  - before: `0m28,673s`
  - after: `0m2,244s`
- https://github.com/kubernetes/kubernetes (890 tags)
  - before: `8m00s`
  - after: `0m8,520s`
- https://github.com/vim/vim (13954 tags)
  - before: `14m20,383s`
  - after: `0m35,467s`

 

I added a `foreachref` package which contains a flexible way of specifying which reference fields are of interest (`git-for-each-ref(1)`) and to produce a parser for the expected output. These could be reused in other places where `for-each-ref` is used.  I'll add unit tests for those if the overall PR looks promising.
2022-03-31 14:30:40 +02:00
wxiaoguang
b877504b03
Refactor git.Command.Run*, introduce RunWithContextString and RunWithContextBytes (#19266)
This follows 
* https://github.com/go-gitea/gitea/issues/18553

Introduce `RunWithContextString` and `RunWithContextBytes` to help the refactoring. Add related unit tests. They keep the same behavior to save stderr into err.Error() as `RunInXxx` before.

Remove `RunInDirTimeoutPipeline` `RunInDirTimeoutFullPipeline` `RunInDirTimeout` `RunInDirTimeoutEnv`  `RunInDirPipeline`  `RunInDirFullPipeline`  `RunTimeout`, `RunInDirTimeoutEnvPipeline`, `RunInDirTimeoutEnvFullPipeline`, `RunInDirTimeoutEnvFullPipelineFunc`.

Then remaining `RunInDir` `RunInDirBytes` `RunInDirWithEnv` can be easily refactored in next PR with a simple search & replace:
* before: `stdout, err := RunInDir(path)`
* next: `stdout, _, err := RunWithContextString(&git.RunContext{Dir:path})`

Other changes:
1. When `timeout <= 0`, use default. Because `timeout==0` is meaningless and could cause bugs. And now many functions becomes more simple, eg: `GitGcRepos` 9 lines to 1 line. `Fsck` 6 lines to 1 line.
2. Only set defaultCommandExecutionTimeout when the option `setting.Git.Timeout.Default > 0`
2022-03-31 13:56:22 +02:00
wxiaoguang
c83168104b
Use a more general (and faster) method to sanitize URLs with credentials (#19239)
Use a more general method to sanitize URLs with credentials: Simple and intuitive / Faster /  Remove all credentials in all URLs
2022-03-31 10:25:40 +08:00
6543
3e88af898a
Make git.OpenRepository accept Context (#19260)
* OpenRepositoryCtx -> OpenRepository
* OpenRepository -> openRepositoryWithDefaultContext, only for internal usage
2022-03-30 03:13:41 +08:00
zeripath
889a8c268c
Use full output of git show-ref --tags to get tags for PushUpdateAddTag (#19235)
Strangely #19038 appears to relate to an issue whereby a tag appears to
be listed in `git show-ref --tags` but then does not appear when `git
show-ref --tags -- short_name` is called.

As a solution though I propose to stop the second call as it is
unnecessary and only likely to cause problems.

I've also noticed that the tags calls are wildly inefficient and aren't using the common cat-files - so these have been added.

I've also noticed that the git commit-graph is not being written on mirroring - so I've also added writing this to the migration which should improve mirror rendering somewhat. 

Fix #19038

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
2022-03-29 19:12:33 +02:00
Lunny Xiao
c29fbc6d23
Hide sensitive content on admin panel progress monitor (#19218)
Sanitize urls within git process descriptions.

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: Andrew Thornton <art27@cantab.net>
2022-03-27 12:54:09 +01:00
zeripath
41b60d94db
Do not include global arguments in process manager (#19226)
The git command by default adds a number of global arguments. These are not
helpful to be displayed in the process manager and so should be skipped for
default process descriptions.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-03-27 10:09:56 +01:00
zeripath
2d21d2af9e
Make migrations SKIP_TLS_VERIFY apply to git too (#19132)
Make SKIP_TLS_VERIFY apply to git data migrations too through adding the `-c http.sslVerify=false` option to the git clone command.

Fix #18998

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-03-19 14:16:38 +00:00