90 lines
2.9 KiB
Markdown
90 lines
2.9 KiB
Markdown
# ftest: compliance suite for software forge federation
|
|
|
|
## Goals
|
|
|
|
1. Flexible: should be able to define new tests with ease
|
|
2. Transparent: test results should be transparently available
|
|
3. Versioned: Evolution of a forge implementation must be easily
|
|
observable
|
|
4. Reproducible: test results should reproducible
|
|
5. Ease of deployment
|
|
6. Ease of use for developers and forge developers
|
|
7. Ease of use for general public
|
|
|
|
## Development Notes
|
|
|
|
### Create `ftest` Docker subnet before running tests or deploying
|
|
|
|
```bash
|
|
docker network create --attachable -d bridge ftest
|
|
```
|
|
|
|
### `cargo test` is breaking due to `Exception: Error while creating access token: 400...`
|
|
|
|
Run `cargo test -j 1` to run tests serially.
|
|
|
|
|
|
### Test Runner
|
|
|
|
Choice of test runner should not be hard-codded, it should be possible to
|
|
define new runner interfaces. But for now, we are going to go with
|
|
Docker.
|
|
|
|
### Test suites
|
|
|
|
A test suite will be made of multiple, independently defined tests, so
|
|
that the same test implementation can be reused between multiple suites.
|
|
|
|
Tests suites can live anywhere on the internet, but the have to be
|
|
linked in the control repository for them to be available on an ftest
|
|
instance. This will enable ftest admins authorize the code that is
|
|
run on their servers.
|
|
|
|
### Test jobs
|
|
|
|
Each test will be it's own container image. This allows for polyglot
|
|
tests, which should make it easy to define new tests. Also,
|
|
containerization will offer some level of security when running
|
|
untrusted code.
|
|
|
|
A new container of of the test image will be deployed whenever a new
|
|
compliance job is scheduled. The container will be given a secret
|
|
associated with the job ID. When the job is complete, the test suite
|
|
container should upload the results to the `ftest` server.
|
|
|
|
### Test result schema
|
|
|
|
```json
|
|
"tests": [{
|
|
"test_id": string,
|
|
"success": boolean,
|
|
"logs": string, // debug data that the test generates
|
|
"raw_logs": string // container logs (ex: docker logs <container-name>
|
|
}]
|
|
```
|
|
|
|
### Compliance job
|
|
|
|
Compliance job consist of more than one test suite.
|
|
|
|
#### Scheduling new compliance test jobs
|
|
|
|
The test runner will accept new jobs through a Git repository called the
|
|
control repository. To
|
|
schedule a new compliance job(for instance when a forge instance is updated),
|
|
forge developers can send a patch to the control repository with the
|
|
`docker-compsoe` definition file to spin up their software and its
|
|
dependencies, and a job
|
|
file that enumerates the test suites that must be run against it.
|
|
|
|
#### Compliance Results
|
|
|
|
For an implementation to be 100% compliant with a test suite, it will
|
|
have to successfully pass all the tests defined in the test suite. A
|
|
partial compliance score can be calculated using the same method.
|
|
|
|
The compliance report will include Compliance score, test logs, and the
|
|
control repository commit that triggered job. This data will be used to
|
|
create a versioned report in the form of a static site deployed from a
|
|
Git repository (results repository).
|