Commit graph

92 commits

Author SHA1 Message Date
Lunny Xiao
e5723d6556
Move restore repo to internal router and invoke from command to avoid open the same db file or queues files (#15790)
* Move restore repo to internal router and invoke from command to avoid open the same db file or queues files

* Follow @zeripath's review

* set no timeout for resotre repo private request

* make restore repo cancelable
2021-05-10 15:57:45 +08:00
6543
2a42d80d14
migration: github: if rate limit is not enabled, ignore it (#15490) 2021-04-15 15:34:22 +02:00
KN4CK3R
c03e488e14
Add LFS Migration and Mirror (#14726)
* Implemented LFS client.

* Implemented scanning for pointer files.

* Implemented downloading of lfs files.

* Moved model-dependent code into services.

* Removed models dependency. Added TryReadPointerFromBuffer.

* Migrated code from service to module.

* Centralised storage creation.

* Removed dependency from models.

* Moved ContentStore into modules.

* Share structs between server and client.

* Moved method to services.

* Implemented lfs download on clone.

* Implemented LFS sync on clone and mirror update.

* Added form fields.

* Updated templates.

* Fixed condition.

* Use alternate endpoint.

* Added missing methods.

* Fixed typo and make linter happy.

* Detached pointer parser from gogit dependency.

* Fixed TestGetLFSRange test.

* Added context to support cancellation.

* Use ReadFull to probably read more data.

* Removed duplicated code from models.

* Moved scan implementation into pointer_scanner_nogogit.

* Changed method name.

* Added comments.

* Added more/specific log/error messages.

* Embedded lfs.Pointer into models.LFSMetaObject.

* Moved code from models to module.

* Moved code from models to module.

* Moved code from models to module.

* Reduced pointer usage.

* Embedded type.

* Use promoted fields.

* Fixed unexpected eof.

* Added unit tests.

* Implemented migration of local file paths.

* Show an error on invalid LFS endpoints.

* Hide settings if not used.

* Added LFS info to mirror struct.

* Fixed comment.

* Check LFS endpoint.

* Manage LFS settings from mirror page.

* Fixed selector.

* Adjusted selector.

* Added more tests.

* Added local filesystem migration test.

* Fixed typo.

* Reset settings.

* Added special windows path handling.

* Added unit test for HTTPClient.

* Added unit test for BasicTransferAdapter.

* Moved into util package.

* Test if LFS endpoint is allowed.

* Added support for git://

* Just use a static placeholder as the displayed url may be invalid.

* Reverted to original code.

* Added "Advanced Settings".

* Updated wording.

* Added discovery info link.

* Implemented suggestion.

* Fixed missing format parameter.

* Added Pointer.IsValid().

* Always remove model on error.

* Added suggestions.

* Use channel instead of array.

* Update routers/repo/migrate.go

* fmt

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: zeripath <art27@cantab.net>
2021-04-08 18:25:57 -04:00
zeripath
0bb8bd8190
Add size to Save function (#15264)
Fix #15253

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-04-03 18:19:59 +02:00
6543
5f038cd7fe
[Vendor] update gitea-sdk v0.14.0 (#15103)
* upgraded code.gitea.io/sdk/gitea v0.13.2 => v0.14.0

* rm workaround
2021-03-22 18:03:18 +01:00
KN4CK3R
e8ad6c1ff3
Do not convert file path to lowercase (#15023)
* Do not convert file path to lowercase.

* lint

* Check against lowercase hostname.
2021-03-18 13:58:47 +00:00
zeripath
6e423d5573
Ensure validation occurs on clone addresses too (#14994)
* Ensure validation occurs on clone addresses too

Fix #14984

Signed-off-by: Andrew Thornton <art27@cantab.net>

* fix lint

Signed-off-by: Andrew Thornton <art27@cantab.net>

* fix test

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Fix api tests

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-03-15 17:52:11 -04:00
6543
14d8cb7819
Move Workaround for #12675 into it's own function (#14922)
* Move Workatround for #12675 into it's own function

* use more reliable solution (as tea do)
2021-03-08 20:00:19 +01:00
zeripath
eb576269d4
Re-enable import local paths after reversion from #13610 (#14925)
PR #13610 unfortunately disabled importing repositories from local paths.
This PR restores this functionality.

Fix #14700

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-03-08 13:10:17 +00:00
zeripath
3d8b5ad5f3
Fix a couple of CommentAsPatch issues. (#14804)
* CutDiffAroundLine makes the incorrect assumption that `---` and `+++` always represent part of the header of a diff.

This PR adds a flag to its parsing to prevent this problem and adds a streaming parsing technique to CutDiffAroundLine using an io.pipe instead of just sending data to an unbounded buffer.

Fix #14711

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Handle unquoted comment patch files

When making comment patches unfortunately the patch does not always quote the filename
This makes the diff --git header ambiguous again.

This PR finally adds handling for ambiguity in to parse patch

Fix #14812

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Add in testing for no error

There is no way currently for CutDiffAroundLine in this test to cause an
error however, it should still be tested.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-02-27 19:46:14 +01:00
Lunny Xiao
e79dae29cc
Fix repo-restore bug with poster not replaced (#14759)
* Fix restore bug

* Fix restore bug

Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
Co-authored-by: 6543 <6543@obermui.de>
2021-02-22 11:57:41 +08:00
Lunny Xiao
ec06eb112c
Fix github download on migration (#14703)
* Fix github download on migration

* Use Context for Client

Co-authored-by: zeripath <art27@cantab.net>
2021-02-18 01:36:49 +01:00
Lunny Xiao
240fea8c14
Fix rate limit bug when downloading assets on migrating from github (#14564) 2021-02-07 15:56:11 +00:00
6543
81c833d92d
Add support to migrate from gogs (#14342)
Add support to migrate gogs:

  *  issues
  *  comments
  *  labels
  *  milestones
  *  wiki


Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: Andrew Thornton <art27@cantab.net>
2021-01-21 20:33:58 +01:00
Paul Barton
7576e37a65
Add MirrorInterval to the API (#14163)
* Added MirrorInterval to the API

* Remove MirrorInterval from CreateRepository

* Removed Duplicate UpdateMirror Function

* Updated Error Logging

* Update Log Message for is not Mirror

Co-authored-by: 6543 <6543@obermui.de>

* Delete Debug Statement that snuck in

Co-authored-by: zeripath <art27@cantab.net>

* Add Check for If Interval is too small

* Output to API Call

* Add Error Object when time is Less than Min Interval

* Frequency Error Message

Co-authored-by: zeripath <art27@cantab.net>

* Allow Zero Mirror Interval

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: zeripath <art27@cantab.net>
2021-01-02 18:47:47 -05:00
Lunny Xiao
dd08853b10
Dump github/gitlab/gitea repository data to a local directory and restore to gitea (#12244)
* Dump github/gitlab repository data to a local directory

* Fix lint

* Adjust directory structure

* Allow migration special units

* Allow migration ignore release assets

* Fix lint

* Add restore repository

* stage the changes

* Merge

* Fix lint

* Update the interface

* Add some restore methods

* Finish restore

* Add comments

* Fix restore

* Add a token flag

* Fix bug

* Fix test

* Fix test

* Fix bug

* Fix bug

* Fix lint

* Fix restore

* refactor downloader

* fmt

* Fix bug isEnd detection on getIssues

* Refactor maxPerPage

* Remove unused codes

* Remove unused codes

* Fix bug

* Fix restore

* Fix dump

* Uploader should not depend downloader

* use release attachment name but not id

* Fix restore bug

* Fix lint

* Fix restore bug

* Add a method of DownloadFunc for base.Release to make uploader not depend on downloader

* fix Release yml marshal

* Fix trace information

* Fix bug when dump & restore

* Save relative path on yml file

* Fix bug

* Use relative path

* Update docs

* Use git service string but not int

* Recognize clone addr to service type
2020-12-27 11:34:19 +08:00
6543
212fa340cf
Migration: drop to long repo topics (#14152) 2020-12-27 09:23:57 +08:00
zeripath
825efa2605
[API] Add Ref to Issue (#13946)
* Add Ref to api.Issue

This PR adds Ref to api.Issue and adds the backend parts to the migrator.

Fix #13918

Signed-off-by: Andrew Thornton <art27@cantab.net>
2020-12-13 11:34:11 +00:00
6543
2b4a08e962
Migrations: Use Process Manager to create own Context (#13792) 2020-12-02 18:36:06 +00:00
6543
b2435af9be
Add Allow-/Block-List for Migrate & Mirrors (#13610)
* add black list and white list support for migrating repositories

* fix fmt

* fix lint

* fix vendor

* fix modules.txt

* clean diff

* specify log message

* use blocklist/allowlist

* allways use lowercase to match url

* Apply allow/block

* Settings: use existing "migrations" section

* convert domains lower case

* dont store unused value

* Block private addresses for migration by default

* fix lint

* use proposed-upstream func to detect private IP addr

* a nit

* add own error for blocked migration, add tests, imprufe api

* fix test

* fix-if-localhost-is-ipv4

* rename error & error message

* rename setting options

* Apply suggestions from code review

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2020-11-28 19:37:58 -05:00
6543
24b3b2140a
finaly fix gitlab migration with subdir 2.0 (#13646)
* final fix 2.0?

* ignore Approvals for pulls if not found

* CI.restart()

Co-authored-by: Lauris BH <lauris@nix.lv>
2020-11-20 01:18:34 +02:00
6543
ad2a288622
finaly fix gitlab migration with subdir (#13629)
* finaly fix #13535

* add logging
2020-11-19 00:17:56 -05:00
6543
77aa742528
Migration: Gitlab: Support Subdirectory (#13563)
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2020-11-16 21:22:49 +02:00
zeripath
5b2096dd5f
Migrations should not fail for comment reactions (#13352)
An extension to #13444 - where we now ensure that comment reaction failures do not cause migrations failure

Signed-off-by: Andrew Thornton <art27@cantab.net>
2020-10-29 17:41:49 -04:00
zeripath
4b33afcd5b
Migration failure during reaction migration from gitea (#13344)
* Migrating reactions is just not that important

A failure during migrating reactions should not cause failure of
migration.

Signed-off-by: Andrew Thornton <art27@cantab.net>

* When checking issue reactions check the correct permission

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2020-10-28 22:23:31 -04:00
zeripath
7dfb2fc176
Add migrated pulls to pull request task queue (#13331)
* Add migrated pulls to pull request task queue

Fix #13321

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Improve error reports

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2020-10-27 17:34:56 -04:00
Lunny Xiao
d130cd147f
Fix bug isEnd detection on getIssues/getPullRequests (#13299) 2020-10-25 01:11:03 -04:00
6543
81d3942bf0
Migrations: Gitea should not fail just because of no apiConfig return (#13229)
* close #13227

* log it

👍

Co-authored-by: zeripath <art27@cantab.net>

Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2020-10-22 01:46:18 -04:00
6543
49b1948cb1
Gitea 2 Gitea migration (#12657)
* first draft

* update gitea sdk to 9e280adb4da

* adapt feat of updated sdk

* releases now works

* break the Reactions loop

* use convertGiteaLabel

* fix endless loop because paggination is not supported there !!!

* rename gitea local uploader files

* pagination can bite you in the ass

* Version Checks

* lint

* docs

* rename gitea sdk import to miss future conficts

* go-swagger: dont scan the sdk structs

* make sure gitea can shutdown gracefully

* make GetPullRequests and GetIssues similar

* rm useles

* Add Test: started ...

* ... add tests ...

* Add tests and Fixing things

* Workaround missing SHA

* Adapt: Ensure that all migration requests are cancellable
(714ab71ddc)

* LINT: fix misspells in test set

* adapt ListMergeRequestAwardEmoji

* update sdk

* Return error when creating giteadownloader failed

* update sdk

* adapt new sdk

* adopt new features

* check version before err

* adapt: 'migrate service type switch page'

* optimize

* Fix DefaultBranch

* impruve

* handle subPath

* fix test

* Fix ReviewCommentPosition

* test GetReviews

* add DefaultBranch int test set

* rm unused

* Update SDK to v0.13.0

* addopt sdk changes

* found better link

* format template

* Update Docs

* Update Gitea SDK (v0.13.1)
2020-10-14 07:06:00 +03:00
6543
845d0afb33
4 line diff (#13103) 2020-10-12 09:15:15 +08:00
John Olheiser
ec6a35aeb1
Hopefully support GH enterprise (#12863)
Signed-off-by: jolheiser <john.olheiser@gmail.com>

Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-09-21 10:36:51 -04:00
John Olheiser
d9085fe176
Fix anonymous GL migration (#12862)
* Fix anonymous GL migration

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Rely on password instead

Signed-off-by: jolheiser <john.olheiser@gmail.com>

Co-authored-by: zeripath <art27@cantab.net>
2020-09-15 15:32:14 -04:00
6543
6c61f498ea
On Migration respect old DefaultBranch (#12843)
* On Migration respect old DefaultBranch

* add DefaultBranch int test set

Co-authored-by: zeripath <art27@cantab.net>
2020-09-15 22:37:44 +08:00
6543
fd60ebfe14
[API] Migration: Change ServiceType String (#12672)
* use different structs for MigrateRepoOptions on UI and API

* Fix TokenAuth and rename UID to an understandable Name

* fix swagger doc

* simplify & mk redable

* R E F A C T O R:
migration has now internal 3 structs to store its options:
 * the Options for WebUI: modules/auth/repo_form.go
 * the Options for API: modules/structs/repo.go
 * the option struct with after validation for internal prossessing: modules/migrations/base/options.go

* Copyright Header

* Deprecate UID - add RepoOwner

* adopt repo.go -> migrate.go

* add comment about each struct purpose

* lint
2020-09-10 23:29:19 +01:00
6543
e10327168d
GitLab DL: normalizeColor (#12793) 2020-09-10 14:04:30 +01:00
Lunny Xiao
0db9043aa7
return error when create gitlabdownloader (#12790) 2020-09-10 11:27:49 +03:00
6543
0c6a802731
[Vendor] Update xanzy/go-gitlab v0.31.0 => v0.37.0 (#12701)
* update github.com/xanzy/go-gitlab v0.31.0 => v0.37.0

* vendor

* adapt changes

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2020-09-06 11:37:53 -04:00
6543
ad4a644fb3
Migrations: Gitlab Add Reactions Support for Issues & MergeRequests (#12695)
* fix migrations: gitlab_test

* more stable test

* Get Reactions to Pulls

* Reactions for Issues

* Comments are still hard

Co-authored-by: Lauris BH <lauris@nix.lv>
2020-09-03 10:35:17 +03:00
zeripath
714ab71ddc
Ensure that all migration requests are cancellable (#12669)
* Ensure that all migration requests are cancellable

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Use WithContext as RequestWithContext is go 1.14

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2020-09-02 13:49:25 -04:00
John Olheiser
211321fb93
Git migration UX (#12619)
* Initial work

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Implementation

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Fix gitlab and token cloning

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Imports and JS

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Fix test

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Linting

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Generate swagger

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Move mirror toggle and rename options

Signed-off-by: jolheiser <john.olheiser@gmail.com>

Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-08-28 09:36:37 +08:00
Lunny Xiao
62e6c9bc6c
Add a storage layer for attachments (#11387)
* Add a storage layer for attachments

* Fix some bug

* fix test

* Fix copyright head and lint

* Fix bug

* Add setting for minio and flags for migrate-storage

* Add documents

* fix lint

* Add test for minio store type on attachments

* fix test

* fix test

* Apply suggestions from code review

Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>

* Add warning when storage migrated successfully

* Fix drone

* fix test

* rebase

* Fix test

* display the error on console

* Move minio test to amd64 since minio docker don't support arm64

* refactor the codes

* add trace

* Fix test

* remove log on xorm

* Fi download bug

* Add a storage layer for attachments

* Add setting for minio and flags for migrate-storage

* fix lint

* Add test for minio store type on attachments

* Apply suggestions from code review

Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>

* Fix drone

* fix test

* Fix test

* display the error on console

* Move minio test to amd64 since minio docker don't support arm64

* refactor the codes

* add trace

* Fix test

* Add URL function to serve attachments directly from S3/Minio

* Add ability to enable/disable redirection in attachment configuration

* Fix typo

* Add a storage layer for attachments

* Add setting for minio and flags for migrate-storage

* fix lint

* Add test for minio store type on attachments

* Apply suggestions from code review

Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>

* Fix drone

* fix test

* Fix test

* display the error on console

* Move minio test to amd64 since minio docker don't support arm64

* don't change unrelated files

* Fix lint

* Fix build

* update go.mod and go.sum

* Use github.com/minio/minio-go/v6

* Remove unused function

* Upgrade minio to v7 and some other improvements

* fix lint

* Fix go mod

Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
Co-authored-by: Tyler <tystuyfzand@gmail.com>
2020-08-18 12:23:45 +08:00
6543
b3e06523c1
[Vendor] upgrade google/go-github to v32.1.0 (#12361)
* upgrate go-github client to v32.1.0

* migrate
2020-07-31 15:22:34 +01:00
6543
8bdc9795d8
Add name filter to API for GetMilestoneList (#12336)
Adds a name filter to the API for GetMilestoneList

Includes a small refactor: merge GetMilestones and GetMilestonesByRepoID

Close #12260

Needed for https://gitea.com/gitea/go-sdk/issues/383 and https://gitea.com/gitea/tea/pulls/149
2020-07-28 12:30:40 +01:00
Lunny Xiao
1dd3f19ee3
Use google/uuid to instead satori/go.uuid (#11943)
Co-authored-by: Lauris BH <lauris@nix.lv>
2020-06-18 17:18:44 +08:00
Gernot Eger
a3fe9d87f2
Set the base url when migrating from Gitlab using access token or username without password (#11852)
When migrating from gitlab, set the baseUrl in NewGitlabDownloader when using an access token or username without password

Fix #11851
2020-06-11 16:41:01 +01:00
zeripath
9a2e47b23a
Refactor Cron and merge dashboard tasks (#10745)
* Refactor Cron and merge dashboard tasks

* Merge Cron and Dashboard tasks
* Make every cron task report a system notice on completion
* Refactor the creation of these tasks
* Ensure that execution counts of tasks is correct
* Allow cron tasks to be started from the cron page

* golangci-lint fixes

* Enforce that only one task with the same name can be registered

Signed-off-by: Andrew Thornton <art27@cantab.net>

* fix name check

Signed-off-by: Andrew Thornton <art27@cantab.net>

* as per @guillep2k

* as per @lafriks

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Add git.CommandContext variants

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2020-05-16 19:31:38 -04:00
zeripath
c58bc4bf80
Prevent timer leaks in Workerpool and others (#11333)
There is a potential memory leak in `Workerpool` due to the intricacies of
`time.Timer` stopping.

Whenever a `time.Timer` is `Stop`ped its channel must be cleared using a
`select` if the result of the `Stop()` is `false`.

Unfortunately in `Workerpool` these were checked the wrong way round.

However, there were a few other places that were not being checked.

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-05-08 16:46:05 +01:00
6543
5bfb9bc2b6
When migrating from Gitlab map Approvals to approving Reviews (#11147)
When migrating from Gitlab map Gitlab Approvals to approving Reviews

Co-Authored-By: zeripath <art27@cantab.net>
2020-04-20 13:30:46 +01:00
6543
e37e217602
[Fix] RetryDownloader (add GetReviews()) (#11093)
* add GetReviews() to RetryDownloader
2020-04-20 03:23:28 +01:00
6543
82dbb34c9c
Vendor Update: go-gitlab v0.22.1 -> v0.31.0 (#11136)
* vendor update: go-gitlab to v0.31.0

* migrate client init to v0.31.0

* refactor
2020-04-19 21:23:05 +01:00