feat-protobuf #1

Merged

realaravinth merged 15 commits from feat-protobuf into master 2023-12-30 14:24:36 +05:30

Conversation 0 Commits 15 Files Changed 39 +4189 -1237

realaravinth commented 2023-12-26 15:15:05 +05:30

Owner

Copy Link

This PR swaps out Actix-Web based JSON networking stack with tonic and protobuf. Protobuf implementation was bootstrapped from databend's metasrv package (Copyright 2023, Apache 2-0) with improvements for health monitoring. metasrv wraps JSON with protobuf, so this patch starts from there. Upon benchmarking, we realized that a pure protobuf implementation is 4x faster, so the JSON dependency was removed from the wire format.

The in-memory implementation of dcache and the efficient wire format exposed a bottleneck in the critical section of libmCaptcha. A fix for this will be uploaded in a later PR.

This PR swaps out Actix-Web based JSON networking stack with tonic and protobuf. Protobuf implementation was bootstrapped from databend's [metasrv package (Copyright 2023, Apache 2-0)](https://github.com/datafuselabs/databend/tree/6603392a958ba8593b1f4b01410bebedd484c6a9/metasrv) with improvements for health monitoring. `metasrv` wraps JSON with protobuf, so this patch starts from there. Upon benchmarking, we realized that a pure protobuf implementation is 4x faster, so the JSON dependency was removed from the wire format. The in-memory implementation of dcache and the efficient wire format exposed a bottleneck in the critical section of libmCaptcha. A fix for this will be uploaded in a later PR.

realaravinth self-assigned this 2023-12-26 15:15:05 +05:30

realaravinth added 3 commits 2023-12-26 15:15:05 +05:30

70ef43b720

feat: use protobuf for RPC

a2dd2c31f6

feat: add protobuf bench

ba9694b31f

chore: cleanup deadcode from REST impl

realaravinth referenced this pull request from mCaptcha/2023-NLnet 2023-12-26 18:53:37 +05:30

Benchmark and compare performance against the semi-distributed, Redis-based cache implementation that is currently used #12

realaravinth added 2 commits 2023-12-27 19:48:57 +05:30

6d90790e58

fix: actix actors will only start on actix_rt

4446cd83bd

feat: python grpc test client

realaravinth added 2 commits 2023-12-28 14:24:31 +05:30

e50b7a5751

chore: log learner joins

337f89f25a

feat: use grpc within locust

realaravinth added 4 commits 2023-12-28 14:24:53 +05:30

3ad3d280d9

feat: use protobuf without JSON hack

67051bc187

feat: call individual RPC methods without the JSON hack in test script

69172db518

feat: add nojson results

565ffec3c6

feat: port locust to use GRPC payloads

realaravinth added 3 commits 2023-12-30 14:09:44 +05:30

0f5762536b

debug: pipeline dump

a288450721

feat: batch RPC

59b847b740

feat: bench pipeline implementaion

realaravinth added 1 commit 2023-12-30 14:18:19 +05:30

01d4c2fce3

feat: install protoc

realaravinth merged commit b4469e03d0 into master 2023-12-30 14:24:36 +05:30

realaravinth referenced this issue from a commit 2023-12-30 14:24:36 +05:30

Merge pull request 'feat-protobuf' (#1) from feat-protobuf into master

realaravinth referenced this pull request 2023-12-30 14:44:13 +05:30

Use atomic types to speedup variable difficulty alogirthm #3

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

  

bug

Something is not working

  

duplicate

This issue or pull request already exists

  

enhancement

New feature

  

help wanted

Need some help

  

invalid

Something is wrong

  

question

More information is needed

  

wontfix

This won't be fixed

No Label

bug

duplicate

enhancement

help wanted

invalid

question

wontfix

Milestone

Clear milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

No Assignees

realaravinth

1 Participants

Notifications

Subscribe

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: mCaptcha/dcache#1

No description provided.