# ftest: compliance suite for software forge federation

## Goals

1. Flexible: should be able to define new test suites with ease
2. Transparent: test results should be transparently available
3. Reproducible: test results should reproducible
4. Ease of deployment
5. Ease of use for developers and forge developers
6. Ease of use for general public

## Development Notes

### Test Runner

Choice of test runner should not be hard-codded, it should be possible to
define new runner interfaces. But for now, we are going to go with
Docker.

### Test suites

Each test suite will be it's own Docker image. This allows for polyglot
test suites, which should make it easy to define new test suites. Also,
containerization will offer some level of security when running
untrusted code.

### Test jobs

A new container of of the test suite image will be
deployed whenever a new test job is scheduled. The container will be
given a secret associated with the job ID. When the job is complete, the
test suite container should upload the results to the `ftest` server.

### Test result schema

```json
"tests": [{
"test_id": string,
"success": boolean,
"logs": string,
"raw_logs": string
}]
```

### Scheduling new test jobs

The test runner will accept new jobs through a Git repository. To
schedule a new test job(for instance when a forge instance is updated),
forge developers can send a patch to the Git repository with the
`docker-compsoe` definition file to spin up their software, and a job
file that enumerates the test suits that must be run against it.