ForgeFlux/ftest
2
0
Fork 0
Code Issues 2 Pull requests 10 Projects Releases Packages Wiki Activity

feat: & fix: define control & result repos, compliance suites, diff test and test sutie

Browse source
This commit is contained in:
Aravinth Manivannan 2023-09-05 19:19:24 +05:30
parent a3e907027f
commit af95c79b3b
Signed by: realaravinth
GPG key ID: AD9F0F08E855ED88
1 changed files with 48 additions and 21 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

69
README.md
View file

@ -2,12 +2,14 @@
## Goals ## Goals
1. Flexible: should be able to define new test suites with ease 1. Flexible: should be able to define new tests with ease
2. Transparent: test results should be transparently available 2. Transparent: test results should be transparently available
3. Reproducible: test results should reproducible 3. Versioned: Evolution of a forge implementation must be easily
4. Ease of deployment observable
5. Ease of use for developers and forge developers 4. Reproducible: test results should reproducible
6. Ease of use for general public 5. Ease of deployment
6. Ease of use for developers and forge developers
7. Ease of use for general public
## Development Notes ## Development Notes
@ -19,17 +21,25 @@ Docker.
### Test suites ### Test suites
Each test suite will be it's own Docker image. This allows for polyglot A test suite will be made of multiple, independently defined tests, so
test suites, which should make it easy to define new test suites. Also, that the same test implementation can be reused between multiple suites.
containerization will offer some level of security when running
untrusted code. Tests suites can live anywhere on the internet, but the have to be
linked in the control repository for them to be available on an ftest
instance. This will enable ftest admins authorize the code that is
run on their servers.
### Test jobs ### Test jobs
A new container of of the test suite image will be Each test will be it's own container image. This allows for polyglot
deployed whenever a new test job is scheduled. The container will be tests, which should make it easy to define new tests. Also,
given a secret associated with the job ID. When the job is complete, the containerization will offer some level of security when running
test suite container should upload the results to the `ftest` server. untrusted code.
A new container of of the test image will be deployed whenever a new
compliance job is scheduled. The container will be given a secret
associated with the job ID. When the job is complete, the test suite
container should upload the results to the `ftest` server.
### Test result schema ### Test result schema
@ -37,15 +47,32 @@ test suite container should upload the results to the `ftest` server.
"tests": [{ "tests": [{
"test_id": string, "test_id": string,
"success": boolean, "success": boolean,
"logs": string, "logs": string, // debug data that the test generates
"raw_logs": string "raw_logs": string // container logs (ex: docker logs <container-name>
}] }]
``` ```
### Scheduling new test jobs ### Compliance job
The test runner will accept new jobs through a Git repository. To Compliance job consist of more than one test suite.
schedule a new test job(for instance when a forge instance is updated),
forge developers can send a patch to the Git repository with the #### Scheduling new compliance test jobs
`docker-compsoe` definition file to spin up their software, and a job
file that enumerates the test suits that must be run against it. The test runner will accept new jobs through a Git repository called the
control repository. To
schedule a new compliance job(for instance when a forge instance is updated),
forge developers can send a patch to the control repository with the
`docker-compsoe` definition file to spin up their software and its
dependencies, and a job
file that enumerates the test suites that must be run against it.
#### Compliance Results
For an implementation to be 100% compliant with a test suite, it will
have to successfully pass all the tests defined in the test suite. A
partial compliance score can be calculated using the same method.
The compliance report will include Compliance score, test logs, and the
control repository commit that triggered job. This data will be used to
create a versioned report in the form of a static site deployed from a
Git repository (results repository).