feat: & fix: define control & result repos, compliance suites, diff test and test sutie

README.md

@@ -2,12 +2,14 @@
 
 ## Goals
 
-1. Flexible: should be able to define new test suites with ease
+1. Flexible: should be able to define new tests with ease
 2. Transparent: test results should be transparently available
-3. Reproducible: test results should reproducible
-4. Ease of deployment
-5. Ease of use for developers and forge developers
-6. Ease of use for general public
+3. Versioned: Evolution of a forge implementation must be easily
+   observable
+4. Reproducible: test results should reproducible
+5. Ease of deployment
+6. Ease of use for developers and forge developers
+7. Ease of use for general public
 
 ## Development Notes
 
@@ -19,17 +21,25 @@ Docker.
 
 ### Test suites
 
-Each test suite will be it's own Docker image. This allows for polyglot
-test suites, which should make it easy to define new test suites. Also,
-containerization will offer some level of security when running
-untrusted code.
+A test suite will be made of multiple, independently defined tests, so
+that the same test implementation can be reused between multiple suites.
+
+Tests suites can live anywhere on the internet, but the have to be
+linked in the control repository for them to be available on an ftest
+instance. This will enable ftest admins authorize the code that is
+run on their servers.
 
 ### Test jobs
 
-A new container of of the test suite image will be
-deployed whenever a new test job is scheduled. The container will be
-given a secret associated with the job ID. When the job is complete, the
-test suite container should upload the results to the `ftest` server.
+Each test will be it's own container image. This allows for polyglot
+tests, which should make it easy to define new tests. Also,
+containerization will offer some level of security when running
+untrusted code.
+
+A new container of of the test image will be deployed whenever a new
+compliance job is scheduled. The container will be given a secret
+associated with the job ID. When the job is complete, the test suite
+container should upload the results to the `ftest` server.
 
 ### Test result schema
 
@@ -37,15 +47,32 @@ test suite container should upload the results to the `ftest` server.
 "tests": [{
 "test_id": string,
 "success": boolean,
-"logs": string,
-"raw_logs": string
+"logs": string, // debug data that the test generates
+"raw_logs": string // container logs (ex: docker logs <container-name>
 }]
 ```
 
-### Scheduling new test jobs
+### Compliance job
 
-The test runner will accept new jobs through a Git repository. To
-schedule a new test job(for instance when a forge instance is updated),
-forge developers can send a patch to the Git repository with the
-`docker-compsoe` definition file to spin up their software, and a job
-file that enumerates the test suits that must be run against it.
+Compliance job consist of more than one test suite.
+
+#### Scheduling new compliance test jobs
+
+The test runner will accept new jobs through a Git repository called the
+control repository. To
+schedule a new compliance job(for instance when a forge instance is updated),
+forge developers can send a patch to the control repository with the
+`docker-compsoe` definition file to spin up their software and its
+dependencies, and a job
+file that enumerates the test suites that must be run against it.
+
+#### Compliance Results
+
+For an implementation to be 100% compliant with a test suite, it will
+have to successfully pass all the tests defined in the test suite. A
+partial compliance score can be calculated using the same method.
+
+The compliance report will include Compliance score, test logs, and the
+control repository commit that triggered job. This data will be used to
+create a versioned report in the form of a static site deployed from a
+Git repository (results repository).