bench-forgejo/integrations
zeripath 62eb1b0f25 Graceful Queues: Issue Indexing and Tasks (#9363)
* Queue: Add generic graceful queues with settings

* Queue & Setting: Add worker pool implementation

* Queue: Add worker settings

* Queue: Make resizing worker pools

* Queue: Add name variable to queues

* Queue: Add monitoring

* Queue: Improve logging

* Issues: Gracefulise the issues indexer

Remove the old now unused specific queues

* Task: Move to generic queue and gracefulise

* Issues: Standardise the issues indexer queue settings

* Fix test

* Queue: Allow Redis to connect to unix

* Prevent deadlock during early shutdown of issue indexer

* Add MaxWorker settings to queues

* Merge branch 'master' into graceful-queues

* Update modules/indexer/issues/indexer.go

Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com>

* Update modules/indexer/issues/indexer.go

Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com>

* Update modules/queue/queue_channel.go

Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com>

* Update modules/queue/queue_disk.go

* Update modules/queue/queue_disk_channel.go

Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com>

* Rename queue.Description to queue.ManagedQueue as per @guillep2k

* Cancel pool workers when removed

* Remove dependency on queue from setting

* Update modules/queue/queue_redis.go

Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com>

* As per @guillep2k add mutex locks on shutdown/terminate

* move unlocking out of setInternal

* Add warning if number of workers < 0

* Small changes as per @guillep2k

* No redis host specified not found

* Clean up documentation for queues

* Update docs/content/doc/advanced/config-cheat-sheet.en-us.md

* Update modules/indexer/issues/indexer_test.go

* Ensure that persistable channel queue is added to manager

* Rename QUEUE_NAME REDIS_QUEUE_NAME

* Revert "Rename QUEUE_NAME REDIS_QUEUE_NAME"

This reverts commit 1f83b4fc9b9dabda186257b38c265fe7012f90df.

Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: techknowlogick <matti@mdranta.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-01-07 12:23:09 +01:00
..
gitea-repositories-meta Redirect issue if repo has configured external tracker. (#9247) 2019-12-06 23:21:18 -05:00
migration-test Graceful: Xorm, RepoIndexer, Cron and Others (#9282) 2019-12-15 09:51:28 +00:00
api_admin_org_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
api_admin_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
api_branch_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
api_comment_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
api_fork_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
api_gpg_keys_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
api_helper_for_declarative_test.go Sign merges, CRUD, Wiki and Repository initialisation with gpg key (#7631) 2019-10-16 14:42:42 +01:00
api_issue_label_test.go Move sdk structs to modules/structs (#6905) 2019-05-11 18:21:34 +08:00
api_issue_reaction_test.go [API] Fix 9544 | return 200 when reaction already exist (#9550) 2019-12-31 03:21:21 -05:00
api_issue_stopwatch_test.go [API] extend StopWatch (#9196) 2019-12-11 23:23:05 -05:00
api_issue_test.go Fix #9189 - API Allow only specific Colums to be updated on Issue (#9539) 2020-01-01 17:51:10 -05:00
api_issue_tracked_time_test.go [API] Extend times API (#9200) 2019-12-27 20:30:58 +00:00
api_keys_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
api_org_test.go Fixes #7023 - API Org Visibility (#7028) 2019-05-30 13:57:55 -04:00
api_pull_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
api_releases_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
api_repo_edit_test.go Api: advanced settings for repository (external wiki, issue tracker etc.) (#7756) 2019-10-02 17:30:41 +08:00
api_repo_file_create_test.go [API] Extend contents with dates (#9464) 2019-12-24 10:33:52 +08:00
api_repo_file_delete_test.go Fixes #7152 - Allow create/update/delete message to be empty, use default message (#7324) 2019-06-29 16:19:24 +01:00
api_repo_file_helpers.go Move sdk structs to modules/structs (#6905) 2019-05-11 18:21:34 +08:00
api_repo_file_update_test.go File Edit: Author/Committer interchanged [BugFix] (#9297) 2019-12-09 21:11:24 +08:00
api_repo_get_contents_list_test.go Add Close() method to gogitRepository (#8901) 2019-11-13 07:01:19 +00:00
api_repo_get_contents_test.go Add Close() method to gogitRepository (#8901) 2019-11-13 07:01:19 +00:00
api_repo_git_blobs_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
api_repo_git_commits_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
api_repo_git_hook_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
api_repo_git_ref_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
api_repo_git_tags_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
api_repo_git_trees_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
api_repo_lfs_locks_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
api_repo_raw_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
api_repo_tags_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
api_repo_test.go Redirect issue if repo has configured external tracker. (#9247) 2019-12-06 23:21:18 -05:00
api_repo_topic_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
api_team_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
api_team_user_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
api_token_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
api_user_heatmap_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
api_user_orgs_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
api_user_search_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
attachment_test.go Only serve attachments when linked to issue/release and if accessible by user (#9340) 2020-01-05 01:20:08 +02:00
auth_ldap_test.go Graceful: Xorm, RepoIndexer, Cron and Others (#9282) 2019-12-15 09:51:28 +00:00
benchmarks_test.go Missed defer prepareTestEnv (#9285) 2019-12-08 01:14:50 +00:00
branches_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
change_default_branch_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
cmd_keys_test.go AuthorizedKeysCommand should not query db directly (#9371) 2019-12-16 20:49:07 -05:00
cors_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
create_no_session_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
delete_user_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
download_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
editor_test.go Add golangci (#6418) 2019-06-12 15:41:28 -04:00
empty_repo_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
explore_repos_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
git_helper_for_declarative_test.go Fix the intermittent TestGPGGit failures (#9360) 2019-12-15 16:21:16 +00:00
git_test.go Push to create repo (#8419) 2019-12-15 10:49:52 +08:00
gpg_git_test.go Fix the intermittent TestGPGGit failures (#9360) 2019-12-15 16:21:16 +00:00
html_helper.go integration tests: Use t.Helper() (#7654) 2019-07-29 12:15:18 +08:00
integration_test.go Graceful: Xorm, RepoIndexer, Cron and Others (#9282) 2019-12-15 09:51:28 +00:00
issue_test.go Graceful Queues: Issue Indexing and Tasks (#9363) 2020-01-07 12:23:09 +01:00
lfs_getobject_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
links_test.go [BugFix] use default avatar for ghost user (fix 500 error) (#9536) 2019-12-29 16:24:50 +02:00
mssql.ini.tmpl Sign merges, CRUD, Wiki and Repository initialisation with gpg key (#7631) 2019-10-16 14:42:42 +01:00
mysql.ini.tmpl Sign merges, CRUD, Wiki and Repository initialisation with gpg key (#7631) 2019-10-16 14:42:42 +01:00
mysql8.ini.tmpl Sign merges, CRUD, Wiki and Repository initialisation with gpg key (#7631) 2019-10-16 14:42:42 +01:00
nonascii_branches_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
oauth_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
org_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
pgsql.ini.tmpl Sign merges, CRUD, Wiki and Repository initialisation with gpg key (#7631) 2019-10-16 14:42:42 +01:00
private-testing.key Fix the intermittent TestGPGGit failures (#9360) 2019-12-15 16:21:16 +00:00
pull_compare_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
pull_create_test.go Make CI work (#8057) 2019-09-02 15:12:29 -04:00
pull_merge_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
pull_review_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
pull_status_test.go Fix pull creation with empty changes (#7920) 2019-08-20 13:43:00 -04:00
README.md Improve integration tests (#8276) 2019-09-25 14:13:18 +02:00
README_ZH.md Improve integration tests (#8276) 2019-09-25 14:13:18 +02:00
release_test.go tests: add attachement tests integration (#9309) 2019-12-10 19:06:48 +08:00
repo_activity_test.go Remove local clones & make hooks run on merge/edit/upload (#6672) 2019-05-11 11:29:17 -04:00
repo_branch_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
repo_commits_search_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
repo_commits_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
repo_fork_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
repo_generate_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
repo_migrate_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
repo_search_test.go Move code indexer related code to a new package (#9191) 2019-12-08 14:15:35 -05:00
repo_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
repo_watch_test.go Auto-subscribe user to repository when they commit/tag to it (#7657) 2019-11-10 09:22:19 +00:00
repofiles_delete_test.go Add Close() method to gogitRepository (#8901) 2019-11-13 07:01:19 +00:00
repofiles_update_test.go integrations: fix dropped errors in TestCreateOrUpdateRepoFileForUpdateWithFileMove() (#9040) 2019-11-16 18:21:39 +00:00
setting_test.go Add a /user/login landing page option (#9622) 2020-01-06 17:50:44 +01:00
signin_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
signout_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
signup_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
sqlite.ini Sign merges, CRUD, Wiki and Repository initialisation with gpg key (#7631) 2019-10-16 14:42:42 +01:00
ssh_key_test.go Use gitea forked macaron (#7933) 2019-08-23 12:40:29 -04:00
testlogger.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
timetracking_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
user_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
version_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00
xss_test.go Fix "data race" in testlogger (#9159) 2019-11-26 01:21:37 +02:00

Integrations tests

Integration tests can be run with make commands for the appropriate backends, namely:

make test-mysql
make test-pgsql
make test-sqlite

Make sure to perform a clean build before running tests:

make clean build

Run all tests via local drone

drone exec --local --build-event "pull_request"

Run sqlite integrations tests

Start tests

make test-sqlite

Run mysql integrations tests

Setup a mysql database inside docker

docker run -e "MYSQL_DATABASE=test" -e "MYSQL_ALLOW_EMPTY_PASSWORD=yes" -p 3306:3306 --rm --name mysql mysql:5.7 #(just ctrl-c to stop db and clean the container) 

Start tests based on the database container

TEST_MYSQL_HOST=localhost:3306 TEST_MYSQL_DBNAME=test TEST_MYSQL_USERNAME=root TEST_MYSQL_PASSWORD='' make test-mysql

Run pgsql integrations tests

Setup a pgsql database inside docker

docker run -e "POSTGRES_DB=test" -p 5432:5432 --rm --name pgsql postgres:9.5 #(just ctrl-c to stop db and clean the container) 

Start tests based on the database container

TEST_PGSQL_HOST=localhost:5432 TEST_PGSQL_DBNAME=test TEST_PGSQL_USERNAME=postgres TEST_PGSQL_PASSWORD=postgres make test-pgsql

Run mssql integrations tests

Setup a mssql database inside docker

docker run -e "ACCEPT_EULA=Y" -e "MSSQL_PID=Standard" -e "SA_PASSWORD=MwantsaSecurePassword1" -p 1433:1433 --rm --name mssql microsoft/mssql-server-linux:latest #(just ctrl-c to stop db and clean the container) 

Start tests based on the database container

TEST_MSSQL_HOST=localhost:1433 TEST_MSSQL_DBNAME=gitea_test TEST_MSSQL_USERNAME=sa TEST_MSSQL_PASSWORD=MwantsaSecurePassword1 make test-mssql

Running individual tests

Example command to run GPG test:

For sqlite:

make test-sqlite#GPG

For other databases(replace MSSQL to MYSQL, MYSQL8, PGSQL):

TEST_MSSQL_HOST=localhost:1433 TEST_MSSQL_DBNAME=test TEST_MSSQL_USERNAME=sa TEST_MSSQL_PASSWORD=MwantsaSecurePassword1 make test-mssql#GPG