To avoid deadlock problem, almost database related functions should be
have ctx as the first parameter.
This PR do a refactor for some of these functions.
Fix #25558
Extract from #22743
This PR added a repository's check when creating/deleting branches via
API. Mirror repository and archive repository cannot do that.
Related #14180
Related #25233
Related #22639
Close #19786
Related #12763
This PR will change all the branches retrieve method from reading git
data to read database to reduce git read operations.
- [x] Sync git branches information into database when push git data
- [x] Create a new table `Branch`, merge some columns of `DeletedBranch`
into `Branch` table and drop the table `DeletedBranch`.
- [x] Read `Branch` table when visit `code` -> `branch` page
- [x] Read `Branch` table when list branch names in `code` page dropdown
- [x] Read `Branch` table when list git ref compare page
- [x] Provide a button in admin page to manually sync all branches.
- [x] Sync branches if repository is not empty but database branches are
empty when visiting pages with branches list
- [x] Use `commit_time desc` as the default FindBranch order by to keep
consistent as before and deleted branches will be always at the end.
---------
Co-authored-by: Jason Song <i@wolfogre.com>
#### Added
- API: Create a branch directly from commit on the create branch API
- Added `old_ref_name` parameter to allow creating a new branch from a
specific commit, tag, or branch.
- Deprecated `old_branch_name` parameter in favor of the new
`old_ref_name` parameter.
---------
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
To avoid duplicated load of the same data in an HTTP request, we can set
a context cache to do that. i.e. Some pages may load a user from a
database with the same id in different areas on the same page. But the
code is hidden in two different deep logic. How should we share the
user? As a result of this PR, now if both entry functions accept
`context.Context` as the first parameter and we just need to refactor
`GetUserByID` to reuse the user from the context cache. Then it will not
be loaded twice on an HTTP request.
But of course, sometimes we would like to reload an object from the
database, that's why `RemoveContextData` is also exposed.
The core context cache is here. It defines a new context
```go
type cacheContext struct {
ctx context.Context
data map[any]map[any]any
lock sync.RWMutex
}
var cacheContextKey = struct{}{}
func WithCacheContext(ctx context.Context) context.Context {
return context.WithValue(ctx, cacheContextKey, &cacheContext{
ctx: ctx,
data: make(map[any]map[any]any),
})
}
```
Then you can use the below 4 methods to read/write/del the data within
the same context.
```go
func GetContextData(ctx context.Context, tp, key any) any
func SetContextData(ctx context.Context, tp, key, value any)
func RemoveContextData(ctx context.Context, tp, key any)
func GetWithContextCache[T any](ctx context.Context, cacheGroupKey string, cacheTargetID any, f func() (T, error)) (T, error)
```
Then let's take a look at how `system.GetString` implement it.
```go
func GetSetting(ctx context.Context, key string) (string, error) {
return cache.GetWithContextCache(ctx, contextCacheKey, key, func() (string, error) {
return cache.GetString(genSettingCacheKey(key), func() (string, error) {
res, err := GetSettingNoCache(ctx, key)
if err != nil {
return "", err
}
return res.SettingValue, nil
})
})
}
```
First, it will check if context data include the setting object with the
key. If not, it will query from the global cache which may be memory or
a Redis cache. If not, it will get the object from the database. In the
end, if the object gets from the global cache or database, it will be
set into the context cache.
An object stored in the context cache will only be destroyed after the
context disappeared.
This PR introduce glob match for protected branch name. The separator is
`/` and you can use `*` matching non-separator chars and use `**` across
separator.
It also supports input an exist or non-exist branch name as matching
condition and branch name condition has high priority than glob rule.
Should fix #2529 and #15705
screenshots
<img width="1160" alt="image"
src="https://user-images.githubusercontent.com/81045/205651179-ebb5492a-4ade-4bb4-a13c-965e8c927063.png">
Co-authored-by: zeripath <art27@cantab.net>
After #22362, we can feel free to use transactions without
`db.DefaultContext`.
And there are still lots of models using `db.DefaultContext`, I think we
should refactor them carefully and one by one.
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Change all license headers to comply with REUSE specification.
Fix #16132
Co-authored-by: flynnnnnnnnnn <flynnnnnnnnnn@github>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
This PR adds a context parameter to a bunch of methods. Some helper
`xxxCtx()` methods got replaced with the normal name now.
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Move access and repo permission to models/perm/access
* fix test
* Move some git related files into sub package models/git
* Fix build
* fix git test
* move lfs to sub package
* move more git related functions to models/git
* Move functions sequence
* Some improvements per @KN4CK3R and @delvh
* Don't error when branch's commit doesn't exist
- If one of the branches no longer exists, don't throw an error, it's possible that the branch was destroyed during the process. Simply skip it and disregard it.
- Resolves #19541
* Don't send empty objects
* Use more minimal approach
This PR continues the work in #17125 by progressively ensuring that git
commands run within the request context.
This now means that the if there is a git repo already open in the context it will be used instead of reopening it.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Some refactors related repository model
* Move more methods out of repository
* Move repository into models/repo
* Fix test
* Fix test
* some improvements
* Remove unnecessary function
This PR registers requests with the process manager and manages hierarchy within the processes.
Git repos are then associated with a context, (usually the request's context) - with sub commands using this context as their base context.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Fixes #16381
Note that changes to unprotected files via the web editor still cannot be pushed directly to the protected branch. I could easily add such support for edits and deletes if needed. But for adding, uploading or renaming unprotected files, it is not trivial.
* Extract & Move GetAffectedFiles to modules/git
* Add info about list endpoints to CONTRIBUTING.md
* Let all list endpoints return X-Total-Count header
* Add TODOs for GetCombinedCommitStatusByRef
* Fix models/issue_stopwatch.go
* Rrefactor models.ListDeployKeys
* Introduce helper func and use them for SetLinkHeader related func
* Fix bug when delete branch don't close related PRs
* Merge all deletebranch as one method
* Add missed branch.go
* fix comment
Co-authored-by: Lauris BH <lauris@nix.lv>
Use [chi](https://github.com/go-chi/chi) instead of the forked [macaron](https://gitea.com/macaron/macaron). Since macaron and chi have conflicts with session share, this big PR becomes a have-to thing. According my previous idea, we can replace macaron step by step but I'm wrong. :( Below is a list of big changes on this PR.
- [x] Define `context.ResponseWriter` interface with an implementation `context.Response`.
- [x] Use chi instead of macaron, and also a customize `Route` to wrap chi so that the router usage is similar as before.
- [x] Create different routers for `web`, `api`, `internal` and `install` so that the codes will be more clear and no magic .
- [x] Use https://github.com/unrolled/render instead of macaron's internal render
- [x] Use https://github.com/NYTimes/gziphandler instead of https://gitea.com/macaron/gzip
- [x] Use https://gitea.com/go-chi/session which is a modified version of https://gitea.com/macaron/session and removed `nodb` support since it will not be maintained. **BREAK**
- [x] Use https://gitea.com/go-chi/captcha which is a modified version of https://gitea.com/macaron/captcha
- [x] Use https://gitea.com/go-chi/cache which is a modified version of https://gitea.com/macaron/cache
- [x] Use https://gitea.com/go-chi/binding which is a modified version of https://gitea.com/macaron/binding
- [x] Use https://github.com/go-chi/cors instead of https://gitea.com/macaron/cors
- [x] Dropped https://gitea.com/macaron/i18n and make a new one in `code.gitea.io/gitea/modules/translation`
- [x] Move validation form structs from `code.gitea.io/gitea/modules/auth` to `code.gitea.io/gitea/modules/forms` to avoid dependency cycle.
- [x] Removed macaron log service because it's not need any more. **BREAK**
- [x] All form structs have to be get by `web.GetForm(ctx)` in the route function but not as a function parameter on routes definition.
- [x] Move Git HTTP protocol implementation to use routers directly.
- [x] Fix the problem that chi routes don't support trailing slash but macaron did.
- [x] `/api/v1/swagger` now will be redirect to `/api/swagger` but not render directly so that `APIContext` will not create a html render.
Notices:
- Chi router don't support request with trailing slash
- Integration test `TestUserHeatmap` maybe mysql version related. It's failed on my macOS(mysql 5.7.29 installed via brew) but succeed on CI.
Co-authored-by: 6543 <6543@obermui.de>
* Let Branch and Raw Endpoint return json error if not found
* Revert "RM RepoRefByTypeForAPI and move needed parts into GetRawFile directly"
This reverts commit d826d08577b23765cb3c257e7a861191d1aa9a04.
* more similar to RepoRefByType
* dedub-code
* API should just speak JSON
* nice name
Co-authored-by: zeripath <art27@cantab.net>
* [Enhancement] Allow admin to merge pr with protected file changes
As tilte, show protected message in diff page and merge box.
Signed-off-by: a1012112796 <1012112796@qq.com>
* remove unused ver
* Update options/locale/locale_en-US.ini
Co-authored-by: Cirno the Strongest <1447794+CirnoT@users.noreply.github.com>
* Add TrN
* Apply suggestions from code review
* fix lint
* Update options/locale/locale_en-US.ini
Co-authored-by: zeripath <art27@cantab.net>
* Apply suggestions from code review
* move pr proteced files check to TestPatch
* Call TestPatch when protected branches settings changed
* Apply review suggestion @CirnoT
* move to service @lunny
* slightly restructure routers/private/hook.go
Adds a lot of comments and simplifies the logic
Signed-off-by: Andrew Thornton <art27@cantab.net>
* placate lint
Signed-off-by: Andrew Thornton <art27@cantab.net>
* skip duplicate protected files check
* fix check logic
* slight refactor of TestPatch
Signed-off-by: Andrew Thornton <art27@cantab.net>
* When checking for protected files changes in TestPatch use the temporary repository
Signed-off-by: Andrew Thornton <art27@cantab.net>
* fix introduced issue with hook
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Remove the check on PR index being greater than 0 as it unnecessary
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: techknowlogick <matti@mdranta.net>
Co-authored-by: Cirno the Strongest <1447794+CirnoT@users.noreply.github.com>
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Fix test
* Add no queue for test only
* improve code
* Auto watch whatever branch operation
* Fix lint
* Rename noqueue to immediate
* Remove old PushUpdate function
* Fix tests
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* [FEATURE] [API] Add Endpoint for Branch Creation
Issue: https://github.com/go-gitea/gitea/issues/11376
This commit introduces an API endpoint for branch creation.
The added route is POST /repos/{owner}/{repo}/branches.
A JSON with the name of the new branch and the name of the old branch is
required as parameters.
Signed-off-by: Terence Le Huu Phuong <terence@qwasar.io>
* Put all the logic into CreateBranch and removed CreateRepoBranch
* - Added the error ErrBranchDoesNotExist in error.go
- Made the CreateNewBranch function return an errBranchDoesNotExist error
when the OldBranch does not exist
- Made the CreateBranch API function checks that the repository is not
empty and that branch exists.
* - Added a resetFixtures helper function in integration_test.go to
fine-tune test env resetting
- Added api test for CreateBranch
- Used resetFixture instead of the more general prepareTestEnv in the
repo_branch_test CreateBranch tests
* Moved the resetFixtures call inside the loop for APICreateBranch function
* Put the prepareTestEnv back in repo_branch_test
* fix import order/sort api branch test
Co-authored-by: zeripath <art27@cantab.net>
* use same process as in routers/repo/branch.go/deleteBranch
* make sure default branch can not be deleted
* remove IsDefaultBranch from UI process - it is worth its own pull
* permissions
* add API branch protection endpoint
* lint
* Change to use team names instead of ids.
* Status codes.
* fix
* Fix
* Add new branch protection options (BlockOnRejectedReviews, DismissStaleApprovals, RequireSignedCommits)
* Do xorm query directly
* fix xorm GetUserNamesByIDs
* Add some tests
* Improved GetTeamNamesByID
* http status created for CreateBranchProtection
* Correct status code in integration test
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: zeripath <art27@cantab.net>
* use numbers and not http.Status___ enum
* fix test
* add many missing swagger responses
* code format
* Deletion Sould return 204 ...
* error handling improvements
* if special error type ... then add it to swagger too
* one smal nit
* invalidTopicsError is []string
* valid swagger specification 2.0
- if you add responses swagger can tell you if you do it right 👍
* use ctx.InternalServerError
* Revert "use numbers and not http.Status___ enum"
This reverts commit b1ff386e2418ed6a7f183e756b13277d701278ef.
* use http.Status* enum everywhere
* Use go-git for tree reading and commit info lookup.
Signed-off-by: Filip Navara <navara@emclient.com>
* Use TreeEntry.IsRegular() instead of ObjectType that was removed.
Signed-off-by: Filip Navara <navara@emclient.com>
* Use the treePath to optimize commit info search.
Signed-off-by: Filip Navara <navara@emclient.com>
* Extract the latest commit at treePath along with the other commits.
Signed-off-by: Filip Navara <navara@emclient.com>
* Fix listing commit info for a directory that was created in one commit and never modified after.
Signed-off-by: Filip Navara <navara@emclient.com>
* Avoid nearly all external 'git' invocations when doing directory listing (.editorconfig code path is still hit).
Signed-off-by: Filip Navara <navara@emclient.com>
* Use go-git for reading blobs.
Signed-off-by: Filip Navara <navara@emclient.com>
* Make SHA1 type alias for plumbing.Hash in go-git.
Signed-off-by: Filip Navara <navara@emclient.com>
* Make Signature type alias for object.Signature in go-git.
Signed-off-by: Filip Navara <navara@emclient.com>
* Fix GetCommitsInfo for repository with only one commit.
Signed-off-by: Filip Navara <navara@emclient.com>
* Fix PGP signature verification.
Signed-off-by: Filip Navara <navara@emclient.com>
* Fix issues with walking commit graph across merges.
Signed-off-by: Filip Navara <navara@emclient.com>
* Fix typo in condition.
Signed-off-by: Filip Navara <navara@emclient.com>
* Speed up loading branch list by keeping the repository reference (and thus all the loaded packfile indexes).
Signed-off-by: Filip Navara <navara@emclient.com>
* Fix lising submodules.
Signed-off-by: Filip Navara <navara@emclient.com>
* Fix build
Signed-off-by: Filip Navara <navara@emclient.com>
* Add back commit cache because of name-rev
Signed-off-by: Filip Navara <navara@emclient.com>
* Fix tests
Signed-off-by: Filip Navara <navara@emclient.com>
* Fix code style
* Fix spelling
* Address PR feedback
Signed-off-by: Filip Navara <navara@emclient.com>
* Update vendor module list
Signed-off-by: Filip Navara <navara@emclient.com>
* Fix getting trees by commit id
Signed-off-by: Filip Navara <navara@emclient.com>
* Fix remaining unit test failures
* Fix GetTreeBySHA
* Avoid running `git name-rev` if not necessary
Signed-off-by: Filip Navara <navara@emclient.com>
* Move Branch code to git module
* Clean up GPG signature verification and fix it for tagged commits
* Address PR feedback (import formatting, copyright headers)
* Make blob lookup by SHA working
* Update tests to use public API
* Allow getting content from any type of object through the blob interface
* Change test to actually expect the object content that is in the GIT repository
* Change one more test to actually expect the object content that is in the GIT repository
* Add comments
