realaravinth/forgejo-federation
1
0
Fork 0
Code Issues Pull requests 6 Projects Releases Packages Wiki Activity

[gitea] fix: Elasticsearch: Request Entity Too Large #28117 (#29062)

Browse source 
Fix for gitea putting everything into one request without batching and
sending it to Elasticsearch for indexing as issued in #28117

This issue occured in large repositories while Gitea tries to
index the code using ElasticSearch.

I've applied necessary changes that takes batch length from below config
(app.ini)
```
[queue.code_indexer]
BATCH_LENGTH=<length_int>
```
and batches all requests to Elasticsearch in chunks as configured in the
above config

(cherry picked from commit 5c0fc9087211f01375f208d679a1e6de0685320c)
This commit is contained in:
dark-angel 2024-02-07 14:27:16 +05:30 committed by Earl Warren
parent 75937b5784
commit 00370f17a4
WARNING! Although there is a key with this ID in the database it does not verify this commit! This commit is SUSPICIOUS.
GPG key ID: 0579CB2928A78A00
1 changed files with 11 additions and 5 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

16
modules/indexer/code/elasticsearch/elasticsearch.go
View file

@ -180,11 +180,17 @@ func (b *Indexer) Index(ctx context.Context, repo *repo_model.Repository, sha st
} }
if len(reqs) > 0 { if len(reqs) > 0 {
_, err := b.inner.Client.Bulk(). esBatchSize := 50
Index(b.inner.VersionedIndexName()).
Add(reqs...). for i := 0; i < len(reqs); i += esBatchSize {
Do(ctx) _, err := b.inner.Client.Bulk().
return err Index(b.inner.VersionedIndexName()).
Add(reqs[i:min(i+esBatchSize, len(reqs))]...).
Do(ctx)
if err != nil {
return err
}
}
} }
return nil return nil
} }