feat-protobuf #1

Merged
realaravinth merged 15 commits from feat-protobuf into master 2023-12-30 14:24:36 +05:30
Owner

This PR swaps out Actix-Web based JSON networking stack with tonic and protobuf. Protobuf implementation was bootstrapped from databend's metasrv package (Copyright 2023, Apache 2-0) with improvements for health monitoring. metasrv wraps JSON with protobuf, so this patch starts from there. Upon benchmarking, we realized that a pure protobuf implementation is 4x faster, so the JSON dependency was removed from the wire format.

The in-memory implementation of dcache and the efficient wire format exposed a bottleneck in the critical section of libmCaptcha. A fix for this will be uploaded in a later PR.

This PR swaps out Actix-Web based JSON networking stack with tonic and protobuf. Protobuf implementation was bootstrapped from databend's [metasrv package (Copyright 2023, Apache 2-0)](https://github.com/datafuselabs/databend/tree/6603392a958ba8593b1f4b01410bebedd484c6a9/metasrv) with improvements for health monitoring. `metasrv` wraps JSON with protobuf, so this patch starts from there. Upon benchmarking, we realized that a pure protobuf implementation is 4x faster, so the JSON dependency was removed from the wire format. The in-memory implementation of dcache and the efficient wire format exposed a bottleneck in the critical section of libmCaptcha. A fix for this will be uploaded in a later PR.
feat: use protobuf for RPC
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed
70ef43b720
feat: add protobuf bench
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed
a2dd2c31f6
chore: cleanup deadcode from REST impl
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed
ci/woodpecker/pr/woodpecker Pipeline failed
ba9694b31f
feat: python grpc test client
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed
ci/woodpecker/pr/woodpecker Pipeline failed
4446cd83bd
feat: use grpc within locust
Some checks failed
ci/woodpecker/pr/woodpecker Pipeline is pending
ci/woodpecker/push/woodpecker Pipeline failed
337f89f25a
feat: batch RPC
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed
a288450721
feat: bench pipeline implementaion
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed
ci/woodpecker/pr/woodpecker Pipeline failed
59b847b740
feat: install protoc
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
ci/woodpecker/pr/woodpecker Pipeline was successful
ci/woodpecker/pull_request_closed/woodpecker Pipeline was successful
01d4c2fce3
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: mCaptcha/dcache#1
No description provided.