51 lines
1.5 KiB
Markdown
51 lines
1.5 KiB
Markdown
# ftest: compliance suite for software forge federation
|
|
|
|
## Goals
|
|
|
|
1. Flexible: should be able to define new test suites with ease
|
|
2. Transparent: test results should be transparently available
|
|
3. Reproducible: test results should reproducible
|
|
4. Ease of deployment
|
|
5. Ease of use for developers and forge developers
|
|
6. Ease of use for general public
|
|
|
|
## Development Notes
|
|
|
|
### Test Runner
|
|
|
|
Choice of test runner should not be hard-codded, it should be possible to
|
|
define new runner interfaces. But for now, we are going to go with
|
|
Docker.
|
|
|
|
### Test suites
|
|
|
|
Each test suite will be it's own Docker image. This allows for polyglot
|
|
test suites, which should make it easy to define new test suites. Also,
|
|
containerization will offer some level of security when running
|
|
untrusted code.
|
|
|
|
### Test jobs
|
|
|
|
A new container of of the test suite image will be
|
|
deployed whenever a new test job is scheduled. The container will be
|
|
given a secret associated with the job ID. When the job is complete, the
|
|
test suite container should upload the results to the `ftest` server.
|
|
|
|
### Test result schema
|
|
|
|
```json
|
|
"tests": [{
|
|
"test_id": string,
|
|
"success": boolean,
|
|
"logs": string,
|
|
"raw_logs": string
|
|
}]
|
|
```
|
|
|
|
### Scheduling new test jobs
|
|
|
|
The test runner will accept new jobs through a Git repository. To
|
|
schedule a new test job(for instance when a forge instance is updated),
|
|
forge developers can send a patch to the Git repository with the
|
|
`docker-compsoe` definition file to spin up their software, and a job
|
|
file that enumerates the test suits that must be run against it.
|