Storing the foreign identifier of an imported issue in the database is a prerequisite to implement idempotent migrations or mirror for issues. It is a baby step towards mirroring that introduces a new table.
At the moment when an issue is created by the Gitea uploader, it fails if the issue already exists. The Gitea uploader could be modified so that, instead of failing, it looks up the database to find an existing issue. And if it does it would update the issue instead of creating a new one. However this is not currently possible because an information is missing from the database: the foreign identifier that uniquely represents the issue being migrated is not persisted. With this change, the foreign identifier is stored in the database and the Gitea uploader will then be able to run a query to figure out if a given issue being imported already exists.
The implementation of mirroring for issues, pull requests, releases, etc. can be done in three steps:
1. Store an identifier for the element being mirrored (issue, pull request...) in the database (this is the purpose of these changes)
2. Modify the Gitea uploader to be able to update an existing repository with all it contains (issues, pull request...) instead of failing if it exists
3. Optimize the Gitea uploader to speed up the updates, when possible.
The second step creates code that does not yet exist to enable idempotent migrations with the Gitea uploader. When a migration is done for the first time, the behavior is not changed. But when a migration is done for a repository that already exists, this new code is used to update it.
The third step can use the code created in the second step to optimize and speed up migrations. For instance, when a migration is resumed, an issue that has an update time that is not more recent can be skipped and only newly created issues or updated ones will be updated. Another example of optimization could be that a webhook notifies Gitea when an issue is updated. The code triggered by the webhook would download only this issue and call the code created in the second step to update the issue, as if it was in the process of an idempotent migration.
The ForeignReferences table is added to contain local and foreign ID pairs relative to a given repository. It can later be used for pull requests and other artifacts that can be mirrored. Although the foreign id could be added as a single field in issues or pull requests, it would need to be added to all tables that represent something that can be mirrored. Creating a new table makes for a simpler and more generic design. The drawback is that it requires an extra lookup to obtain the information. However, this extra information is only required during migration or mirroring and does not impact the way Gitea currently works.
The foreign identifier of an issue or pull request is similar to the identifier of an external user, which is stored in reactions, issues, etc. as OriginalPosterID and so on. The representation of a user is however different and the ability of users to link their account to an external user at a later time is also a logic that is different from what is involved in mirroring or migrations. For these reasons, despite some commonalities, it is unclear at this time how the two tables (foreign reference and external user) could be merged together.
The ForeignID field is extracted from the issue migration context so that it can be dumped in files with dump-repo and later restored via restore-repo.
The GetAllComments downloader method is introduced to simplify the implementation and not overload the Context for the purpose of pagination. It also clarifies in which context the comments are paginated and in which context they are not.
The Context interface is no longer useful for the purpose of retrieving the LocalID and ForeignID since they are now both available from the PullRequest and Issue struct. The Reviewable and Commentable interfaces replace and serve the same purpose.
The Context data member of PullRequest and Issue becomes a DownloaderContext to clarify that its purpose is not to support in memory operations while the current downloader is acting but is not otherwise persisted. It is, for instance, used by the GitLab downloader to store the IsMergeRequest boolean and sort out issues.
---
[source](https://lab.forgefriends.org/forgefriends/forgefriends/-/merge_requests/36)
Signed-off-by: Loïc Dachary <loic@dachary.org>
Co-authored-by: Loïc Dachary <loic@dachary.org>
* use go1.18 to build gitea& update min go version to 1.17
* bump in a few more places
* add a few simple tests for isipprivate
* update go.mod
* update URL to https://go.dev/dl/
* golangci-lint
* attempt golangci-lint workaround
* change version
* bump fumpt version
* skip strings.title test
* go mod tidy
* update tests as some aren't private??
* update tests
Unfortunately #18642 does not work because a `*net.OpError` does not implement
the `Is` interface to make `errors.Is` work correctly - thus leading to the
irritating conclusion that a `*net.OpError` is not a `*net.OpError`.
Here we keep the `errors.Is` because presumably this will be fixed at
some point in the golang main source code but also we add a simply type
cast to also check.
Fix #18629
Signed-off-by: Andrew Thornton <art27@cantab.net>
Yet another issue has come up where the logging from SyncMirrors does not provide
enough context. This PR adds more context to these logging events.
Related #19038
Signed-off-by: Andrew Thornton <art27@cantab.net>
Add new feature to delete issues and pulls via API
Co-authored-by: fnetx <git@fralix.ovh>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: Gusted <williamzijl7@hotmail.com>
Co-authored-by: 6543 <6543@obermui.de>
- Add helper method to reduce redundancy
- Expand the scope from displaying days to years
- Reduce irrelevance by not displaying small units (hours, minutes, seconds) when bigger ones apply (years)
This PR adjusts the error returned when there is failure to lock the level db, and
permits a connections to the same leveldb where there is a different connection string.
Reference #18921
Reference #18917
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Don't treat BOM escape sequence as hidden character.
- BOM sequence is a common non-harmfull escape sequence, it shouldn't be
shown as hidden character.
- Follows GitHub's behavior.
- Resolves #18837
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
The service worker causes a lot of issues with JS errors after instance
upgrades while not bringing any real performance gain over regular HTTP
caching.
Disable it by default for this reason. Maybe later we can remove it
completely, as I simply see no benefit in having it.
* Add tests for references with dashes
This commit adds tests for full URLs referencing repos names and user
names containing a dash.
* Extend regex to match URLs to repos/users with dashes
* logs: add the buffer logger to inspect logs during testing
Signed-off-by: Loïc Dachary <loic@dachary.org>
* migrations: add test for importing pull requests in gitea uploader
Signed-off-by: Loïc Dachary <loic@dachary.org>
* for each git.OpenRepositoryCtx, call Close
* Content is expected to return the content of the log
* test for errors before defer
Co-authored-by: Loïc Dachary <loic@dachary.org>
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Repositories missing their directory should not report an error from the stats
indexer.
Close #18847
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
We can't depend on `latest` version of gofumpt because the output will
not be stable across versions. Lock it down to the latest version
released yesterday and run it again.
Currently Gitea will wait for HammerTime or nice shutdown if kill -1 or kill -2
is sent. We should just immediately hammer if there is a second kill.
Signed-off-by: Andrew Thornton <art27@cantab.net>
There is a potential panic due to a mistaken resetting of the length parameter when
multibyte characters go over a read boundary.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Fix display time of milestones
* Move the SecToTime function
From the models/issue_stopwatch.go file to the modules/util package
* Rename the sec_to_time file
* Updated formatting
* Include copyright notice in sec_to_time.go
* Apply PR review suggestions
- Update copyright notice dates to 2022
- Change `1 day 3h 5min 7s` to `1d 3h 5m 7s`
* Rename hrs var and combine conditions
* Update unit tests to match new time pattern
Changed `1min` to `1m`
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
It appears possible that there could be a hang due to unread data from the
repo-attribute command pipes. This PR simply closes these during the defer.
Signed-off-by: Andrew Thornton <art27@cantab.net>
I want to address #17892, where emails notifications are not sent to assignees (issue and PR) and reviewers (PR) when they have the email setting Only email on mention enabled.
From the user experience perspective, when a user gets a issue/PR assigned or a PR review request, he/she would expect to be implicitly mentioned since the assignment or request is personal and targeting a single person only. Thus I see #17892 as a bug. Could we therefore mark this ticket as such?
The changed code just explicitly checks for the EmailNotificationsOnMention setting beside the existing EmailNotificationsEnabled check. Too rude?
@lunny mentioned a mock mail server for tests, is there something ready. How could I make use of it?
#12774 (comment)
Fix #17892
Add number in queue status to the monitor page so that administrators can
assess how much work is left to be done in the queues.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
- Use a better and more curated list of Ciphers and KeyExchanges, these roughly follows OpenSSH's default.
- Remove some cryptography values which were deprecated.
This code adds a simple endpoint to apply patches to repositories and
branches on gitea. This is then used along with the conflicting checking
code in #18004 to provide a basic implementation of cherry-pick revert.
Now because the buttons necessary for cherry-pick and revert have
required us to create a dropdown next to the Browse Source button
I've also implemented Create Branch and Create Tag operations.
Fix #3880
Fix #17986
Signed-off-by: Andrew Thornton <art27@cantab.net>
WebAuthn may cause a security exception if the provided APP_ID is not allowed for the
current origin. Therefore we should reattempt authentication without the appid
extension.
Also we should allow [u2f] as-well as [U2F] sections.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Simplify Boost/Pause logic
#18658 has added a check to see if we need to boost because there is still work to do
however the check is slightly complex and not ideal. There's no point boosting if
the queue is paused or can't scale. Therefore merge the two selects into one and add
a check to p.paused.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* And on resume add a zeroboost if necessary
Signed-off-by: Andrew Thornton <art27@cantab.net>
* simplify
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lauris BH <lauris@nix.lv>
* Restart zero worker if there is still work to do
It is possible for the zero worker to timeout before all the work is finished.
This may mean that work may take a long time to complete because a worker will only
be induced on repushing.
Also ensure that requested count is reset after pulls and push mirror sync requests and add some more trace logging to the queue push.
Fix #18607
Signed-off-by: Andrew Thornton <art27@cantab.net>
* remove unnecessary web context data fields, and unify the i18n/translation related functions to `Locale`
* in development, show an error if a translation key is missing
* remove the unnecessary loops `for _, lang := range translation.AllLangs()` for every request, which improves the performance slightly
* use `ctx.Locale.Language()` instead of `ctx.Data["Lang"].(string)`
* add more comments about how the Locale/LangType fields are used
When a net.OpError occurs during rendering the underlying connection is essentially
dead and therefore attempting to render further data will only cause further errors.
Therefore in serverErrorInternal detect if the passed in error is an OpError and
if so do not attempt any further rendering.
Fix #18629
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Only attempt to flush queue if the underlying worker pool is not finished
There is a possible race whereby a worker pool could be cancelled but yet the
underlying queue is not empty. This will lead to flush-all cycling because it
cannot empty the pool.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Apply suggestions from code review
Co-authored-by: Gusted <williamzijl7@hotmail.com>
Co-authored-by: Gusted <williamzijl7@hotmail.com>
- Switch to use `CryptoRandomBytes` instead of `CryptoRandomString`, OAuth's secrets are copied pasted and don't need to avoid dubious characters etc.
- `CryptoRandomBytes` gives  `CryptoRandomString` gives  possible states.
- Add a prefix, such that code scanners can easily grep these in source code.
- 32 Bytes + prefix
* Collaborator trust model should trust collaborators
There was an unintended regression in #17917 which leads to only
repository admin commits being trusted. This PR restores the old logic.
Fix #18501
Signed-off-by: Andrew Thornton <art27@cantab.net>
* COrrect use `UserID` in `SearchTeams`
- Use `UserID` in the `SearchTeams` function, currently it was useless
to pass such information. Now it does a INNER statement to `team_user`
which obtains UserID -> TeamID data.
- Make OrgID optional.
- Resolves #18484
* Seperate searching specific user
* Add condition back
* Use correct struct type
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* add test coverage for original author conversion during migrations
And create a function to factorize a code snippet that is repeated
five times and would otherwise be more difficult to test and maintain
consistently.
Signed-off-by: Loïc Dachary <loic@dachary.org>
* fix variable scope and int64 formatting
* add missing calls to remapExternalUser and fix misplaced %d
Co-authored-by: Loïc Dachary <loic@dachary.org>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Don't panic & allow shorter sha1
- Don't panic when the full regex isn't matched and allow the usage of a
shorter sha1 being used.
- Resolves #18471
* Update modules/markup/html.go
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Several users run Gitea in situations whereby `bash` is not available.
If the `SCRIPT_TYPE` is not changed this will cause hooks to fail.
A simple test to check if the provided type is on the PATH should be
sufficient to warn them about this problem.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Change some logging levels
* PlainTextWithBytes - 4xx/5xx this should just be TRACE
* notFoundInternal - the "error" here is too noisy and should be DEBUG
* WorkerPool - Worker pool scaling messages are normal and should be DEBUG
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Ensure git tag tests and other create test repos in tmpdir
There are a few places where tests appear to reuse testing repos which
causes random CI failures.
This PR simply changes these tests to ensure that cloning always happens
into new temporary directories.
Fix #18444
* Change log root for integration tests to use the REPO_TEST_DIR
There is a potential race in the drone integration tests whereby test-mysql etc
will start writing to log files causing make test-check fail.
Fix #18077
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Attempt to prevent the deadlock in the QueueDiskChannel Test again
This time we're going to adjust the pause tests to only test the right
flag.
* Only switch off pushback once we know that we are not pushing anything else
* Ensure full redirection occurs
* More nicely handle a closed datachan
* And handle similar problems in queue_channel_test
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Prevent deadlocks in persistable channel pause test
Because of reuse of the old paused/resumed channels in this test there
was a potential for deadlock. This PR ensures that the channels are always
reobtained.
It further adds some control code to detect hangs in future - and it
ensures that the pausing warning is not shown on shutdown.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* do not warn but do pause
Signed-off-by: Andrew Thornton <art27@cantab.net>
- Pass the Global command args into serviceRPC.
- Fixes error with partial cloning.
- Add partial clone test
- Include diff
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Start adding mechanism to return unhandled data
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Create pushback interface
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Add Pausable interface to WorkerPool and Manager
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Implement Pausable and PushBack for the bytefifos
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Implement Pausable and Pushback for ChannelQueues and ChannelUniqueQueues
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Wire in UI for pausing
Signed-off-by: Andrew Thornton <art27@cantab.net>
* add testcases and fix a few issues
Signed-off-by: Andrew Thornton <art27@cantab.net>
* fix build
Signed-off-by: Andrew Thornton <art27@cantab.net>
* prevent "race" in the test
Signed-off-by: Andrew Thornton <art27@cantab.net>
* fix jsoniter mismerge
Signed-off-by: Andrew Thornton <art27@cantab.net>
* fix conflicts
Signed-off-by: Andrew Thornton <art27@cantab.net>
* fix format
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Add warnings for no worker configurations and prevent data-loss with redis/levelqueue
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Use StopTimer
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
- Disable the browser's function to "sniff" for the content-type on the
provided plain text, this will prevent the possible usage of
user-controlled data being sent, which could be malicious.
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Add config option to hide issue events
Adds a config option `HIDE_ISSUE_EVENTS` to hide most issue events (changed labels, milestones, projects...) on the issue detail page.
If this is true, only the following events (comment types) are shown:
* plain comments
* closed/reopned/merged
* reviews
* Make configurable using a list
* Add docs
* Add missing newline
* Fix merge issues
* Allow changes per user settings
* Fix lint
* Rm old docs
* Apply suggestions from code review
* Use bitsets
* Rm comment
* fmt
* Fix lint
* Use variable/constant to provide key
* fmt
* fix lint
* refactor
* Add a prefix for user setting key
* Add license comment
* Add license comment
* Update services/forms/user_form_hidden_comments.go
Co-authored-by: Gusted <williamzijl7@hotmail.com>
* check len == 0
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: Gusted <williamzijl7@hotmail.com>
Co-authored-by: 6543 <6543@obermui.de>
Make router logger more friendly, show the related function name/file/line.
[BREAKING]
This PR substantially changes the logging format of the router logger. If you use this logging for monitoring e.g. fail2ban you will need to update this to match the new format.
This PR continues the work in #17125 by progressively ensuring that git
commands run within the request context.
This now means that the if there is a git repo already open in the context it will be used instead of reopening it.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Migrate from U2F to Webauthn
Co-authored-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
- Don't use `ioutil` package anymore as it doesn't anything special
anymore since Go 1.16:
```
// As of Go 1.16, the same functionality is now provided
// by package io or package os, and those implementations
// should be preferred in new code.
```
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
In #17933 repoAssignment no longer sets the ctx.Repo.Mirror field meaning that
attempting change mirror settings results in an NPE. This PR simply restores this.
Either we should remove this field or, we should set it. At present it seems simplest
to set it instead of going looking in the Data for the value although converting the
context to a bag of things may be the correct approach in the future.
Fix #18204
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Fix #17514
Given the comments I've adjusted this somewhat. The numbers of characters detected are increased and include things like the use of U+300 to make à instead of à and non-breaking spaces.
There is a button which can be used to escape the content to show it.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Gwyneth Morgan <gwymor@tilde.club>
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
It appears that several versions of sendmail require that the mail is sent to them with
LF line endings instead of CRLF endings - which of course they will then convert back
to CRLF line endings to comply with the SMTP standard.
This PR adds another setting SENDMAIL_CONVERT_CRLF which will pass the message writer
through a filter. This will filter out and convert CRLFs to LFs before writing them
out to sendmail.
Fix #18024
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Team permission allow different unit has different permission
* Finish the interface and the logic
* Fix lint
* Fix translation
* align center for table cell content
* Fix fixture
* merge
* Fix test
* Add deprecated
* Improve code
* Add tooltip
* Fix swagger
* Fix newline
* Fix tests
* Fix tests
* Fix test
* Fix test
* Max permission of external wiki and issues should be read
* Move team units with limited max level below units table
* Update label and column names
* Some improvements
* Fix lint
* Some improvements
* Fix template variables
* Add permission docs
* improve doc
* Fix fixture
* Fix bug
* Fix some bug
* fix
* gofumpt
* Integration test for migration (#18124)
integrations: basic test for Gitea {dump,restore}-repo
This is a first step for integration testing of DumpRepository and
RestoreRepository. It:
runs a Gitea server,
dumps a repo via DumpRepository to the filesystem,
restores the repo via RestoreRepository from the filesystem,
dumps the restored repository to the filesystem,
compares the first and second dump and expects them to be identical
The verification is trivial and the goal is to add more tests for each
topic of the dump.
Signed-off-by: Loïc Dachary <loic@dachary.org>
* Team permission allow different unit has different permission
* Finish the interface and the logic
* Fix lint
* Fix translation
* align center for table cell content
* Fix fixture
* merge
* Fix test
* Add deprecated
* Improve code
* Add tooltip
* Fix swagger
* Fix newline
* Fix tests
* Fix tests
* Fix test
* Fix test
* Max permission of external wiki and issues should be read
* Move team units with limited max level below units table
* Update label and column names
* Some improvements
* Fix lint
* Some improvements
* Fix template variables
* Add permission docs
* improve doc
* Fix fixture
* Fix bug
* Fix some bug
* Fix bug
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Aravinth Manivannan <realaravinth@batsense.net>
- The current implementation of `RandomString` doesn't give you a most-possible unique randomness. It gives you 6*`length` instead of the possible 8*`length` bits(or as `length`x bytes) randomness. This is because `RandomString` is being limited to a max value of 63, this in order to represent the random byte as a letter/digit.
- The recommendation of pbkdf2 is to use 64+ bit salt, which the `RandomString` doesn't give with a length of 10, instead of increasing 10 to a higher number, this patch adds a new function called `RandomBytes` which does give you the guarentee of 8*`length` randomness and thus corresponding of `length`x bytes randomness.
- Use hexadecimal to store the bytes value in the database, as mentioned, it doesn't play nice in order to convert it to a string. This will always be a length of 32(with `length` being 16).
- When we detect on `Authenticate`(source: db) that a user has the old format of salt, re-hash the password such that the user will have it's password hashed with increased salt.
Thanks to @zeripath for working out the rouge edges from my first commit 😄.
Co-authored-by: lafriks <lauris@nix.lv>
Co-authored-by: zeripath <art27@cantab.net>
- Include folders for the disk consumption size, they should be included
as they are also saved on the disk :)
- Have a more accurate picture of the size of a repo.
- Mostly they are the size of the file system's block size. E.g. 4Kb on
Linux.
* Add API to get issue/pull comments and events (timeline)
Adds an API to get both comments and events in one endpoint with all required data.
Closes go-gitea/gitea#13250
* Fix swagger
* Don't show code comments (use review api instead)
* fmt
* Fix comment
* Time -> TrackedTime
* Use var directly
* Add logger
* Fix lint
* Fix test
* Add comments
* fmt
* [test] get issue directly by ID
* Update test
* Add description for changed refs
* Fix build issues + lint
* Fix build
* Use string enums
* Update swagger
* Support `page` and `limit` params
* fmt + swagger
* Use global slices
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Handle invalid issues
- When you hover over a issue reference, and the issue doesn't exist, it
will just hang on the loading animation.
- This patch fixes that by showing them the pop-up with a "Error
occured" message.
* Add I18N
* refactor
* fix comment for lint
* fix unit test for i18n
* fix unit test for i18n
* add comments
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
A consequence of forcibly setting the RoutePath to the escaped url is that the
auto routing to endpoints without terminal slashes fails (Causing #18060.) This
failure raises the possibility that forcibly setting the RoutePath causes other
unexpected behaviors too.
Therefore, instead we should simply pre-escape the URL in the process registering
handler. Then the request URL will be properly escaped for all the following calls.
Fix #17938
Fix #18060
Replace #18062
Replace #17997
Signed-off-by: Andrew Thornton <art27@cantab.net>
a custom name, intended to be used when there's a name conflict
- When a fork request results in a name conflict, HTTP 409: Conflict is
returned instead of 500
- API documentation for the above mentioned changes
Signed-off-by: realaravinth <realaravinth@batsense.net>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: zeripath <art27@cantab.net>
If http.Get() returns an error return nil and err before attempting to
use the broken file.
Thanks to walker xiong for spotting this bug.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Git will and can pack references into packfiles and therefore if you write/read the
files directly you will get false results. Instead you should use update-ref and
show-ref. To that end I have created three new functions in git/repo_commit.go that
will do this correctly.
Related #17191
Signed-off-by: Andrew Thornton <art27@cantab.net>
Although #17487 ensured that the table was quoted in the join it missed that the
query part of the check also needed to be quoted.
Fix #17485
Signed-off-by: Andrew Thornton <art27@cantab.net>
There are repeated panics in tests due to TestRepository_GetTag failing
to run properly. This happens when we attempt to reset the internal
repo for a tag which has failed to load. The problem is - the panic that
this is causing is preventing us from finding what the real error is.
This PR simply moves the failure out so we have a chance to see what
really is failing.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Reset Session ID on login
When logging in the SessionID should be reset and the session cleaned up.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* with new session.RegenerateID function
Signed-off-by: Andrew Thornton <art27@cantab.net>
* update go-chi/session
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Ensure that session id is changed after oauth data is set and between account linking pages too
Signed-off-by: Andrew Thornton <art27@cantab.net>
* placate lint
Signed-off-by: Andrew Thornton <art27@cantab.net>
* as per review
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Prevent off-by-one error on comments on newly appended lines
There was a bug in CutDiffAroundLine whereby if a file without a terminal new line
has a patch which appends lines to it and a comment is placed on one of those lines
the comment diff will be a line out of place.
This fixes CutDiffAroundLine to simply ignore the missing terminal newline - however,
we should really improve this rendering to add a marker to say that there was a
previously missing terminal newline.
Fix #17875
Signed-off-by: Andrew Thornton <art27@cantab.net>
Strangely a weird bug was present in the log escaping code whereby any escaped
character would gain 03d - this was due to a mistake in the format string where
it should have read %03o but read instead %o03d. This has led to spurious 03d
trailing characters on these escaped characters!
Signed-off-by: Andrew Thornton <art27@cantab.net>
The current TestPatch conflict code uses a plain git apply which does not properly
account for 3-way merging. However, we can improve things using `git read-tree -m` to
do a three-way merge then follow the algorithm used in merge-one-file. We can also use
`--patience` and/or `--histogram` to generate a nicer diff for applying patches too.
Fix #13679
Fix #6417
Signed-off-by: Andrew Thornton <art27@cantab.net>
This PR contains multiple fixes. The most important of which is:
* Prevent hang in git cat-file if the repository is not a valid repository
Unfortunately it appears that if git cat-file is run in an invalid
repository it will hang until stdin is closed. This will result in
deadlocked /pulls pages and dangling git cat-file calls if a broken
repository is tried to be reviewed or pulls exists for a broken
repository.
Fix #14734
Fix #9271
Fix #16113
Otherwise there are a few small other fixes included which this PR was initially intending to fix:
* Fix panic on partial compares due to missing PullRequestWorkInProgressPrefixes
* Fix links on pulls pages due to regression from #17551 - by making most /issues routes match /pulls too - Fix #17983
* Fix links on feeds pages due to another regression from #17551 but also fix issue with syncing tags - Fix #17943
* Add missing locale entries for oauth group claims
* Prevent NPEs if ColorFormat is called on nil users, repos or teams.
There was an unfortunate regression in #14293 which has led to the double decoding
of url parameter elements if they contain a '%'. This is due to an issue
with the way chi decodes its RoutePath. In detail the problem lies in
mux.go where the routeHTTP path uses the URL.RawPath or even the
URL.Path instead of the escaped path to do routing.
This PR simply forcibly sets the routePath to that of the EscapedPath.
Fix #17938
Signed-off-by: Andrew Thornton <art27@cantab.net>
Save a bit of bandwidth by only requesting 3-times the rendered avatar
size. Factor 4 is only really beneficial on a handful of mobile phones
and I don't think they are the primary device we design for.
Configurability contributed by zeripath.
Fixes: https://github.com/go-gitea/gitea/pull/17422
Fixes: https://github.com/go-gitea/gitea/issues/16287
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Add missing `X-Total-Count` and fix some related bugs
Adds `X-Total-Count` header to APIs that return a list but doesn't have it yet.
Fixed bugs:
* not returned after reporting error (39eb82446c/routers/api/v1/user/star.go (L70))
* crash with index out of bounds, API issue/issueSubscriptions
I also found various endpoints that return lists but do not apply/support pagination yet:
```
/repos/{owner}/{repo}/issues/{index}/labels
/repos/{owner}/{repo}/issues/comments/{id}/reactions
/repos/{owner}/{repo}/branch_protections
/repos/{owner}/{repo}/contents
/repos/{owner}/{repo}/hooks/git
/repos/{owner}/{repo}/issue_templates
/repos/{owner}/{repo}/releases/{id}/assets
/repos/{owner}/{repo}/reviewers
/repos/{owner}/{repo}/teams
/user/emails
/users/{username}/heatmap
```
If this is not expected, an new issue should be opened.
Closes #13043
* fmt
* Update routers/api/v1/repo/issue_subscription.go
Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
* Use FindAndCount
Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
Co-authored-by: 6543 <6543@obermui.de>
* Add setting to OAuth handlers to override local 2FA settings
This PR adds a setting to OAuth and OpenID login sources to allow the source to
override local 2FA requirements.
Fix #13939
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Fix regression from #16544
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Add scopes settings
Signed-off-by: Andrew Thornton <art27@cantab.net>
* fix trace logging in auth_openid
Signed-off-by: Andrew Thornton <art27@cantab.net>
* add required claim options
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Move UpdateExternalUser to externalaccount
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Allow OAuth2/OIDC to set Admin/Restricted status
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Allow use of the same group claim name for the prohibit login value
Signed-off-by: Andrew Thornton <art27@cantab.net>
* fixup! Move UpdateExternalUser to externalaccount
* as per wxiaoguang
Signed-off-by: Andrew Thornton <art27@cantab.net>
* add label back in
Signed-off-by: Andrew Thornton <art27@cantab.net>
* adjust localisation
Signed-off-by: Andrew Thornton <art27@cantab.net>
* placate lint
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Move keys to models/keys
* Rename models/keys -> models/asymkey
* change the missed package name
* Fix package alias
* Fix test
* Fix docs
* Fix test
* Fix test
* merge
* Some refactors related repository model
* Move more methods out of repository
* Move repository into models/repo
* Fix test
* Fix test
* some improvements
* Remove unnecessary function
* Fix a panic in NotifyCreateIssueComment (caused by string truncation)
* more unit tests
* refactor
* fix some edge cases
* use SplitStringAtByteN for comment content
The current implementation of checkBranchName is highly inefficient
involving opening the repository, the listing all of the branch names
checking them individually before then using using opened repo to get
the tags.
This PR avoids this by simply walking the references from show-ref
instead of opening the repository (in the nogogit case).
Signed-off-by: Andrew Thornton <art27@cantab.net>
* allways set a message-id on mails
* Add unit tests for mailer & Message-ID
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Refactor install page (db type)
* set correct default DB HOST for different DB TYPE
* remove legacy TiDB from documents
* unify the usage of DB TYPE, in code we only use "mysql". "MySQL" is only shown to users for friendly name.
* Gitea can use TiDB via MySQL protocol
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Improve install code to avoid low-level mistakes.
If a user tries to do a re-install in a Gitea database, they gets a warning and double check.
When Gitea runs, it never create empty app.ini automatically.
Also some small (related) refactoring:
* Refactor db.InitEngine related logic make it more clean (especially for the install code)
* Move some i18n strings out from setting.go to make the setting.go can be easily maintained.
* Show errors in CLI code if an incorrect app.ini is used.
* APP_DATA_PATH is created when installing, and checked when starting (no empty directory is created any more).
This PR registers requests with the process manager and manages hierarchy within the processes.
Git repos are then associated with a context, (usually the request's context) - with sub commands using this context as their base context.
Signed-off-by: Andrew Thornton <art27@cantab.net>
This PR adds another option to app.ini make co-committed-by and co-authored-by trailers
optional on a per server basis.
Fix #17194
Signed-off-by: Andrew Thornton <art27@cantab.net>
Make relative unix sockets absolute by making them absolute against the AppWorkPath
Fix #17833
## ⚠️ BREAKING ⚠️
Prior to this PR relative unix sockets would have been asserted to be relative to the current working directory that gitea, gitea serv, hook and manager etc were running in. Hooks and Serv would have failed to work properly under this situation so we expect that although this is a technically breaking change the previous situation was already broken.
Signed-off-by: Andrew Thornton <art27@cantab.net>
MIME types can have multiple optional parameters, eg:
video/webm; codecs="w/e codec"; charset="binary"
This commit replaces the usage of regex for getting the "type/subtype"
with mime.ParseMediaType.
- Use the provided `doer` instead of `rel.Publisher`. The code will also
run on edited releases and deleted ones, which isn't necessary done by
`rel.Publisher`.
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* More pleasantly handle broken or missing git repositories
In #17742 it was noted that there a completely invalid git repository underlying a
repo on gitea.com. This happened due to a problem during a migration however, it
is not beyond the realms of possibility that a corruption could occur to another
user.
This PR adds a check to RepoAssignment that will detect if a repository loading has
failed due to an absent git repository. It will then show a page suggesting the user
contacts the administrator or deletes the repository.
Fix #17742
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Update options/locale/locale_en-US.ini
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Remove unnecessary functions of User struct
* Move more database methods out of user struct
* Move more database methods out of user struct
* Fix template failure
* Fix bug
* Remove finished FIXME
* remove unnecessary code
Unfortunately due to a misread on my behalf I missed that git diff only learned
--skip-to in version 2.31.0. Thus this functionality was not working on older versions
of git.
This PR adds a handler that simply allows for us to skip reading the diffs until
we find the correct file to skip to.
Fix #17731
Signed-off-by: Andrew Thornton <art27@cantab.net>
Use hostmacher to replace matchlist.
And we introduce a better DialContext to do a full host/IP check, otherwise the attackers can still bypass the allow/block list by a 302 redirection.
- `.Teams` isn't a field on the User type, thus using the seperate
loaded teams.
- Add a space between `PathEscape` and argument.
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
This PR fixes the builtin avatar generator.
1. The random background color makes some images very dirty. So now we only use white background for avatars.
2. We use left-right mirror avatars to satisfy #14799
3. Fix a small padding error in the algorithm
* Add settings to allow different SMTP envelope from address
Sometimes it may be advisable to hide or alias the from address on an SMTP mail
envelope. This PR adds two new options to the mailer to allow setting of an overriding
from address.
Fix #17477
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Use a standalone struct name for Organization
* recover unnecessary change
* make the code readable
* Fix template failure
* Fix template failure
* Move HasMemberWithUserID to org
* Fix test
* Remove unnecessary user type check
* Fix test
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Prevent double sanitize.
* Use SanitizeReaderToWriter.
At the moment `actualRender` uses `SanitizeReader` to sanitize the output. But `SanitizeReader` gets called in `markup.render` too so the output gets sanitized twice.
I moved the `SanitizeReader` call into `RenderRaw` because this method does not use `markup.render`. I would like to remove the `RenderRaw`/`RenderRawString` methods too because they are only called from tests, the fuzzer and the `/markup/raw` api endpoint. This endpoint is not in use so I think we could remove them. If we really in the future need a method to render markdown without PostProcessing we could achieve this with a more flexible `renderer.NeedPostProcess` method.
* Prevent deadlock in TestPersistableChannelQueue
There is a potential deadlock in TestPersistableChannelQueue due to attempting to
shutdown the test queue before it is ready.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* prevent npe
Signed-off-by: Andrew Thornton <art27@cantab.net>
Use check attribute code to check the assigned language of a file and send that in to
chroma as a hint for the language of the file.
Signed-off-by: Andrew Thornton <art27@cantab.net>
There are multiple places where Gitea does not properly escape URLs that it is building and there are multiple places where it builds urls when there is already a simpler function available to use this.
This is an extensive PR attempting to fix these issues.
1. The first commit in this PR looks through all href, src and links in the Gitea codebase and has attempted to catch all the places where there is potentially incomplete escaping.
2. Whilst doing this we will prefer to use functions that create URLs over recreating them by hand.
3. All uses of strings should be directly escaped - even if they are not currently expected to contain escaping characters. The main benefit to doing this will be that we can consider relaxing the constraints on user names and reponames in future.
4. The next commit looks at escaping in the wiki and re-considers the urls that are used there. Using the improved escaping here wiki files containing '/'. (This implementation will currently still place all of the wiki files the root directory of the repo but this would not be difficult to change.)
5. The title generation in feeds is now properly escaped.
6. EscapePound is no longer needed - urls should be PathEscaped / QueryEscaped as necessary but then re-escaped with Escape when creating html with locales Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Add copy button to markdown code blocks
Done mostly in JS because I think it's better not to try getting buttons
past the markup sanitizer.
* add svg module tests
* fix sanitizer regexp
* remove outdated comment
* vertically center button in issue comments as well
* add comment to css
* fix undefined on view file line copy
* combine animation less files
* Update modules/markup/markdown/markdown.go
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* add test for different sizes
* add cloneNode and add tests for it
* use deep clone
* remove useless optional chaining
* remove the svg node cache
* unify clipboard copy string and i18n
* remove unused var
* remove unused localization
* minor css tweaks to the button
* comment tweak
* remove useless attribute
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
- Partialy resolvess #17596
- Resolves `badCall` errors from go-critic `badCall: suspicious Join on
1 argument`
- When only 1 argument is passed into `filepath.Join`, it won't do
anything special other than `filepath.Clean(...)` will be applied over
it.
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: 6543 <6543@obermui.de>
This PR adds [GitBucket](https://gitbucket.github.io/) as migration source.
Supported:
- Milestones
- Issues
- Pull Requests
- Comments
- Reviews
- Labels
There is no public usable instance so no integration tests added.
* Correctly handle failed migrations
There is a bug in handling failed migrations whereby the migration task gets decoupled
from the migration repository. This leads to a failure of the task to get deleted with
the repository and also leads to the migration failed page resulting in a ISE.
This PR removes the zeroing out of the task id from the migration but also makes
the migration handler tolerate missing tasks much nicer.
Fix #17571
Signed-off-by: Andrew Thornton <art27@cantab.net>
