Neo

An attempt to re-write a incomplete implementation

Workspaces (components)

This repository is divided into multiple cargo workspaces for faster compilation and better reusability.

Crate	Description
antivirus/libantivirus	Defines traits and utilities to scan uploaded (and processed) website assets
antivirus/libclamav	Implements libantivirus for clamav. Expects clamav to be available via UNIX socket.
archive_files_utils	Utilities to compress & decompress zip and various tarball formats
auth	Authentication module. OIDC Relying party implementation. Implemented & tested against Keycloak.
authz	Object-capability authorization. Provides primitives to define and enroll entities and actors, and grant, delegate & revoke capabilities.
common_utils	utilities to work with CQRS. Might be used by other workspaces.
db_utils	Database utilities to create, and drop databases. Also contains an implementation with the sqlx (postgres) crate.
db_utils/migrations/libmigrator	Database migrations for the dns crate. Bakes in migrations using the SQLx macro. Also contains the general CQRS events table that cqrs-es crate requires.
db_utils/migrations/migrator	CLI tool to run dns crate's migrations. Uses dns/migrations/libmigrator under the hood. Used for development to set up environments.
dns	LibrePages' DNS component. Create zones, and manage records.
dns/libwhois	fetch WHOIS data from IANA > TLD authoritative WHOIS server > Registrar's WHOIS server. Runs on best-effort basis, since some TLDs (like .app don't publish WHOIS records)
dns/dnsserver/libdnsserver	Library defining traits for LibrePages to work with a DNS server
dns/dnsserver/libbind9	bind9 implementation of libdnsserver
dns/dnsserver/libbind9/keystore	Distributed key storage for bind9 DNSSEC keys. Writes to Git repository.
dns/dnsserver/libbind9/libbind9db	Synchronized DNSSEC signing using `pg_advisory_lock on individual zones to ensure only one leader bind9 is signing on that zone.
libgit	Git CLI wrapper to manipulate git repositories
static_site	Manages asset bundles, CDN&TLS
static_site/libobjectstore	ports&adapters to interact with S3, utilities to manage customer-uploaded bundles and resulting deployment bundles.
static_site/webserver/libwebserver	defines ports to interact with webserver for use as LP CDN
static_site/webserver/libcaddy	implements libwebserver ports for Caddy
payments	TODO: payments and refunds
subscriptions	TODO: subscription management

Development Requirements

1. Nightly Rust
2. docker: optional (for dev dependencies like Postures,bind9,garage,keycloak&mailpit)
3. devenv, direnv via the nix-shell: optional, easy environment management
4. git

Note: A new version of Nightly Rust toolchain is released every **cough** every night. We use devenv to pin rustc version. If compiling without devenv, please open issue when compilation fails. Nightly is buggy, so a later version (than pinned) could fail.

Development environment

Workspaces that depend on services have make commands to setup environment. Run make env to download, configure and run service containers using Docker. Run make help for full list of available commands.

Development environment variables

There are various .env.sample fails littered across workspaces. cp .env.sample .env and fill relevant information. Must consolidate .env file someday. devenv and direnv expects them.

Name	Purpose
LIBGIT_TEST_REPO="ssh://git@git.batsense.net/LibrePages/test-libgit.git"	libgit writes to this repository in tests
LIBGIT_TEST_REPO_SSH_KEY=$(realpath ../dns/dnsserver/libbind9/keystore/tests/test)	deploy SSH key
DATABASE_URL="postgres://postgres:password@localhost:5432/postgres"	default database URL
db=librepages-neo-postgres	database container name
keycloak=librepages-keycloak-dev	keycloak container name
keycloak_volume=librepages-keycloak-dev-data	keycloak volume
keycloak_port=8080	default keycloak port
keycloak_image=quay.io/keycloak/keycloak:26.6.3	keycloak container image
KEYCLOAK_ADMIN=admin	default keycloak username
KEYCLOAK_ADMIN_PASSWORD=admin	default keycloak password
KEYCLOAK_URL="http://localhost:8080"	SMTP test server
mailpit=librepages-mailpit-dev	SMTP test server container name
RNDC_BIN=$(realpath tests/bin/rndc.sh)	used in tests, executes rndc from within bind9 container
NAMED_CHECKZONE_BIN=$(realpath tests/bin/named_checkzone.sh)	used in tests, executes named-checkzone from within bind9 container
NAMED_CHECKCONF_BIN=$(realpath tests/bin/named_checkconf.sh)	used in tests, executes named-checkconf from within bind9 containerk
BIND_CONFIG_BASE_DIR=$(realpath /tmp/libbind9-tests/)	bind9 dir; used in tests
BIND_PORT=30053	test bind9 instance port
KEYSORE_TEST_REPO="ssh://git@git.batsense.net/LibrePages/test-keystore.git"	used by libbind9 keystore. Use as is.
KEYSTORE_TEST_REPO_SSH_KEY=$(realpath keystore/tests/test)	Deploy key configured with keystore repo Use as is.
GARAGE_DEFAULT_ACCESS_KEY=""	dev garage instance run locally .generate with openssl rand -hex 32
`GARAGE_DEFAULT_SECRET_KEY=""	dev garage instance run locally .generate with openssl rand -hex 32
GARAGE_DEFAULT_BUCKET="default-bucket"	name of defualt S3 bucket
CLAMD_SOCK_PATH="/tmp/clamav/clamd.sock"	clamd UNIX socket

DB: SQLx offline compilation cache

If making changes to database schema or queries or anything DB-related, run make db.sqlx.offline before committing. Or after :p

DB migrations

Each domain contains all required DB migrations. make env copies it to libmigrator and runs migrations. Even though both exist in tree, don't edit libmigrator/migrations directory manually.