ftest: compliance suite for software forge federation

Goals

1. Flexible: should be able to define new test suites with ease
2. Transparent: test results should be transparently available
3. Reproducible: test results should reproducible
4. Ease of deployment
5. Ease of use for developers and forge developers
6. Ease of use for general public

Development Notes

Test Runner

Choice of test runner should not be hard-codded, it should be possible to define new runner interfaces. But for now, we are going to go with Docker.

Test suites

Each test suite will be it's own Docker image. This allows for polyglot test suites, which should make it easy to define new test suites. Also, containerization will offer some level of security when running untrusted code.

Test jobs

A new container of of the test suite image will be deployed whenever a new test job is scheduled. The container will be given a secret associated with the job ID. When the job is complete, the test suite container should upload the results to the ftest server.

Test result schema

"tests": [{
"test_id": string,
"success": boolean,
"logs": string,
"raw_logs": string
}]

Scheduling new test jobs

The test runner will accept new jobs through a Git repository. To schedule a new test job(for instance when a forge instance is updated), forge developers can send a patch to the Git repository with the docker-compsoe definition file to spin up their software, and a job file that enumerates the test suits that must be run against it.