feat-protobuf #1

Merged

realaravinth merged 15 commits from feat-protobuf into master 2023-12-30 14:24:36 +05:30

Conversation 0 Commits 15 Files changed 39 +4189 -1237

realaravinth commented 2023-12-26 15:15:05 +05:30

Owner

Copy link

This PR swaps out Actix-Web based JSON networking stack with tonic and protobuf. Protobuf implementation was bootstrapped from databend's metasrv package (Copyright 2023, Apache 2-0) with improvements for health monitoring. metasrv wraps JSON with protobuf, so this patch starts from there. Upon benchmarking, we realized that a pure protobuf implementation is 4x faster, so the JSON dependency was removed from the wire format.

The in-memory implementation of dcache and the efficient wire format exposed a bottleneck in the critical section of libmCaptcha. A fix for this will be uploaded in a later PR.

This PR swaps out Actix-Web based JSON networking stack with tonic and protobuf. Protobuf implementation was bootstrapped from databend's [metasrv package (Copyright 2023, Apache 2-0)](https://github.com/datafuselabs/databend/tree/6603392a958ba8593b1f4b01410bebedd484c6a9/metasrv) with improvements for health monitoring. `metasrv` wraps JSON with protobuf, so this patch starts from there. Upon benchmarking, we realized that a pure protobuf implementation is 4x faster, so the JSON dependency was removed from the wire format. The in-memory implementation of dcache and the efficient wire format exposed a bottleneck in the critical section of libmCaptcha. A fix for this will be uploaded in a later PR.

realaravinth self-assigned this 2023-12-26 15:15:05 +05:30

realaravinth added 3 commits 2023-12-26 15:15:05 +05:30

feat: use protobuf for RPC 70ef43b720

feat: add protobuf bench a2dd2c31f6

chore: cleanup deadcode from REST impl ba9694b31f

realaravinth referenced this pull request from mCaptcha/2023-NLnet 2023-12-26 18:53:37 +05:30

Benchmark and compare performance against the semi-distributed, Redis-based cache implementation that is currently used #12

realaravinth added 2 commits 2023-12-27 19:48:57 +05:30

fix: actix actors will only start on actix_rt 6d90790e58

feat: python grpc test client 4446cd83bd

realaravinth added 2 commits 2023-12-28 14:24:31 +05:30

chore: log learner joins e50b7a5751

feat: use grpc within locust 337f89f25a

realaravinth added 4 commits 2023-12-28 14:24:53 +05:30

feat: use protobuf without JSON hack 3ad3d280d9

feat: call individual RPC methods without the JSON hack in test script 67051bc187

feat: add nojson results 69172db518

feat: port locust to use GRPC payloads 565ffec3c6

realaravinth added 3 commits 2023-12-30 14:09:44 +05:30

debug: pipeline dump 0f5762536b

feat: batch RPC a288450721

feat: bench pipeline implementaion 59b847b740

realaravinth added 1 commit 2023-12-30 14:18:19 +05:30

feat: install protoc 01d4c2fce3

realaravinth merged commit b4469e03d0 into master 2023-12-30 14:24:36 +05:30

realaravinth referenced this pull request from a commit 2023-12-30 14:24:36 +05:30

Merge pull request 'feat-protobuf' (#1) from feat-protobuf into master

realaravinth referenced this pull request 2023-12-30 14:44:13 +05:30

Use atomic types to speedup variable difficulty alogirthm #3

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

  

bug

Something is not working

  

duplicate

This issue or pull request already exists

  

enhancement

New feature

  

help wanted

Need some help

  

invalid

Something is wrong

  

question

More information is needed

  

renovate-bot

  

renovate-security

  

security

  

wontfix

This won't be fixed

No labels

bug

duplicate

enhancement

help wanted

invalid

question

renovate-bot

renovate-security

security

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

realaravinth

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: mCaptcha/dcache#1

No description provided.