diff --git a/content/ftest/_index.md b/content/ftest/_index.md new file mode 100644 index 0000000..fb3404b --- /dev/null +++ b/content/ftest/_index.md @@ -0,0 +1,6 @@ ++++ +title = "ftest" +insert_anchor_links = "right" +weight = 1 +sort_by = "weight" ++++ diff --git a/content/ftest/about.md b/content/ftest/about.md new file mode 100644 index 0000000..f493aa6 --- /dev/null +++ b/content/ftest/about.md @@ -0,0 +1,20 @@ ++++ +title = "About ftest" +insert_anchor_links = "right" +weight = 1 ++++ + +- [Source code](https://git.batsense.net/ForgeFlux/ftest) +- Status: Usable +- Description: Compliance checker/test runner for [ActivityPub](https://activitypub.rocks) and by + extension, [ForgeFed](https://forgefed.org) + +`ftest` is a compliance checking tool that is similar to +[matrix-org/sytest](https://github.com/matrix-org/sytest) but for ActivityPub and ForgeFed. + +An instance (will be) is hosted by ForgeFlux, that is run on renewable +energy. + +## Quick links + +- [Add new specimen to ftest](@/ftest/tutorial-new-specimen.md) diff --git a/content/ftest/ftest.toml.md b/content/ftest/ftest.toml.md new file mode 100644 index 0000000..5f5e41d --- /dev/null +++ b/content/ftest/ftest.toml.md @@ -0,0 +1,53 @@ ++++ +title = "ftest.toml" +insert_anchor_links = "right" +weight = 5 ++++ + +## `fest.toml` or Job Manifest + +Job manifest is defined in `ftest.toml` file in the specimen's +subdirectory. + +## ftest.toml contents: + +It can contain information on: + +### 1. Initialization code + +The code that needs to be run to prepare the test environment. ftest +runs Docker containers to initialize test environments. + +#### Example + +[forgejo-init-script](https://git.batsense.net/ForgeFlux/forgejo-init-script) performs the following operations in a sequence: + +1. Install Forgejo (the binary starts with a web installer that accepts + configuration parameters for database, mailer, etc.) +2. Create accounts +3. Create repositories +4. Create issues +5. Create access tokens + +NOTE: Some of these operations are irrelevant, as I (@realaravinth) had nicked them from +another project of mine and was too lazy to modify ;) + +### 2. Test Suite + +A test suite is a collection of tests. For instance, ActivityPub is a +test suite that could include tests for the WebFinger protocol and the +Nodeinfo protocol. + +### 3. Tests + +A test is a docker container that tests the +[specimen](@/ftest/terminology.md#specimen) for a specific protocol or +feature. + +#### Example + +[Nodeinfo test](https://git.batsense.net/nodeinfo-test) is the simplest +test available within ftest at the moment. It tests the specimen's +Nodeinfo implementation against the [Nodeinfo specification](https://nodeinfo.diaspora.software/protocol.html). + +Please see [here](/ftest/tests.md) for developing new tests. diff --git a/content/ftest/quick-links.md b/content/ftest/quick-links.md new file mode 100644 index 0000000..f5ec5c8 --- /dev/null +++ b/content/ftest/quick-links.md @@ -0,0 +1,26 @@ ++++ +title = "Quick Links" +insert_anchor_links = "right" +weight = 3 ++++ + +# Quick Links + +## Repositories + +- [Source code](https://git.batsense.net/ForgeFlux/ftest) +- [Control repository](https://git.batsense.net/ForgeFlux/ftest-control) +- [Results repository](https://git.batsense.net/ForgeFlux/ftest-results) +- [Forgejo-init-script (Initialization code)](https://git.batsense.net/ForgeFlux/forgejo-init-script) + +## Test suites + +- [WebFinger](https://git.batsense.net/ForgeFlux/webfinger-test) +- [Nodeinfo](https://git.batsense.net/ForgeFlux/nodeinfo-test) +- [ActivityPub (WIP)](https://git.batsense.net/ForgeFlux/nodeinfo-test) + +## Container Images + +- [WebFinger](https://hub.docker.com/r/forgeflux/webfinger-test/) +- [Nodeinfo](https://hub.docker.com/r/forgeflux/nodeinfo-test/) +- [ActivityPub](https://hub.docker.com/r/forgeflux/ap-test/) diff --git a/content/ftest/terminology.md b/content/ftest/terminology.md new file mode 100644 index 0000000..065e806 --- /dev/null +++ b/content/ftest/terminology.md @@ -0,0 +1,52 @@ ++++ +title = "Terminology" +insert_anchor_links = "right" +weight = 2 ++++ + +## Specimen + +A specimen is an ActityPub or a ForgeFed instance that has to be tested +on ftest. + +## Job Manifest + +or `ftest.toml` + +Contains information on the test environment, initialization code and +the test suites that has to be run on the [specimen](/ftest/terminology.md#specimen) + +## Control Repository + +- Flagship instance: [git.batsense.net/ForgeFlux/ftest-control](https://forgeflux.org/ForgeFlux/ftest-control) + +A Repository that houses [job manifests](@/ftest/terminology.md#job-manifest), and Infrastructure-as-Code +(docker-compose, currently) to spin up a test environment containing the +[specimen](@/ftest/terminology.md#specimen). + +### Directory structure + +Each specimen has it's own directory in the control repository: + +```bash +23:50 atm@lab ftest-control ±|gotosocial ✗|→ tree + +└── targets + ├── forgejo + │   ├── conf + │   │   └── app.ini + │   ├── docker-compose.yml + │   └── ftest.toml + └── gotosocial + ├── data + │   └── config + │   └── config.yaml + ├── docker-compose.yaml + └── Dockerfile +``` + +### specimen docker-compose.yml + +The control repository _MUST_ contain a `docker-compose.yml` to spin up +the specimen. It may include configuration files in the same directory, +which to docker-compose file can rely on using bind mounts within the container. diff --git a/content/ftest/tests.md b/content/ftest/tests.md new file mode 100644 index 0000000..82dd442 --- /dev/null +++ b/content/ftest/tests.md @@ -0,0 +1,50 @@ ++++ +title = "Tests" +insert_anchor_links = "right" +weight = 6 ++++ + +Tests are Docker containers that test +[specimens](@/ftest/terminology.md#specimen) a very specific protocol +or a feature. They integrate with ftest by accepting configuration +parameters through environment variables and uploading results through a +REST API. + +## Configuration (Environment Variables) + +### Auto-generated + +Some environment variables are automatically generated by ftest + +| Name | Purpose | +| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | +| `FTEST_AUTH` | Authentication token that should be used when uploading results to ftest server. It is ephemeral, and randomly generated for each test. | +| `FTEST_HOST` | The hostname of the ftest server. Results must be uploaded to here. Will be in format http://ftest:9000 | +| `FTEST_TARGET_HOST` | The hostname of the specimen server. Tests must be run against this hostname. Will be in format http://specimen:8080 | + +### User provided + +There's option to fetch custom environment variables also. Please +specify them in your test's documentation clearly. The README file of +the test is recommended. + +## Results API: + +A formal OpenAPI specification for the ftest server is yet to be +created. For now, this command should upload test results to the ftest +server: + +```python +def upload_logs_to_ftest(auth, success: bool, logs: str): + ftest = f"http://ftest.example.org/api/v1/{auth}/results" + payload = {"success": success, "logs": logs} + res = requests.post( + ftest, json=payload, headers={"Origin": "http://example.org"} + ) + if res.status_code == 200: + logger.info("Upload successful") + else: + print(res) + +upload_logs_to_ftest("supersecretauthtoekn_provided_by_ftest", True, "") +``` diff --git a/content/ftest/tutorial-new-specimen.md b/content/ftest/tutorial-new-specimen.md new file mode 100644 index 0000000..0061d58 --- /dev/null +++ b/content/ftest/tutorial-new-specimen.md @@ -0,0 +1,40 @@ ++++ +title = "[Tutorial] Add new specimen" +insert_anchor_links = "right" +weight = 7 ++++ + +## How to add new specimen + +### 1. Fork the [control repository](@/ftest/terminology.md#control-repository). + +``` +git clone https://git.batsense.net/ForgeFlux/ftest-control +``` + +### 2. Create specimen subdirectory + +If [specimen](@/ftest/terminology.md#specimen) doesn't exist within `targets/` directory, then create subdirectory for it: + +```bash +mkdir targets/ +cd targets/ +``` + +### 3. Create docker-compose.yml: + +```bash +touch docker-compose.yml +``` + +Populate docker-compose.yml with instructions to spin up a test +environment for the specimen. The specimen's upstream sources +will include a reference docker-compose file should be fine in most cases. + +### 4. Create job manifest + +Job manifest is defined in `ftest.toml` file in the specimen's +subdirectory. It can contain information on: + +#### Initialization code +needs to be run to prepare the test environment.