feat: readme with dev notes

README.md

@@ -0,0 +1,51 @@
+# ftest: compliance suite for software forge federation
+
+## Goals
+
+1. Flexible: should be able to define new test suites with ease
+2. Transparent: test results should be transparently available
+3. Reproducible: test results should reproducible
+4. Ease of deployment
+5. Ease of use for developers and forge developers
+6. Ease of use for general public
+
+## Development Notes
+
+### Test Runner
+
+Choice of test runner should not be hard-codded, it should be possible to
+define new runner interfaces. But for now, we are going to go with
+Docker.
+
+### Test suites
+
+Each test suite will be it's own Docker image. This allows for polyglot
+test suites, which should make it easy to define new test suites. Also,
+containerization will offer some level of security when running
+untrusted code.
+
+### Test jobs
+
+A new container of of the test suite image will be
+deployed whenever a new test job is scheduled. The container will be
+given a secret associated with the job ID. When the job is complete, the
+test suite container should upload the results to the `ftest` server.
+
+### Test result schema
+
+```json
+"tests": [{
+"test_id": string,
+"success": boolean,
+"logs": string,
+"raw_logs": string
+}]
+```
+
+### Scheduling new test jobs
+
+The test runner will accept new jobs through a Git repository. To
+schedule a new test job(for instance when a forge instance is updated),
+forge developers can send a patch to the Git repository with the
+`docker-compsoe` definition file to spin up their software, and a job
+file that enumerates the test suits that must be run against it.