colbert-ir/colbertv2.0 (Score)
[](https://colab.research.google.com/github/stanford-futuredata/ColBERT/blob/main/docs/intro2new.ipynb)
Overview
Benchmarks
AskUbuntuDupQuestions
Duplicate question detection from AskUbuntu
Corpus: 6,743 Queries: 360
Quality
ndcg at 10 0.5989
map at 10 0.4425
mrr at 10 0.6798
CMedQAv1-reranking
Quality
ndcg at 10 0.1846
map at 10 0.1367
mrr at 10 0.1955
CMedQAv2-reranking
Quality
ndcg at 10 0.1979
map at 10 0.1478
mrr at 10 0.2017
CQADupstackPhysicsRetrieval
Duplicate question retrieval from StackExchange Physics
Corpus: 38,314 Queries: 1,039
Performance L4 b1 c16
Corpus 33.2K tok/s
Corpus p50 58.6ms
Query 2.9K tok/s
Query p50 58.1ms
CosQA
Code search with natural language queries
Corpus: 6,267 Queries: 500
Performance L4 b1 c16
Corpus 14.7K tok/s
Corpus p50 56.1ms
Query 1.8K tok/s
Query p50 51.8ms
FiQA2018
Financial opinion mining and question answering
Corpus: 57,599 Queries: 648
Performance L4 b1 c16
Corpus 38.0K tok/s
Corpus p50 59.5ms
Query 3.9K tok/s
Query p50 47.3ms
LegalBenchConsumerContractsQA
Question answering on consumer contracts
Corpus: 153 Queries: 396
Performance L4 b1 c16
Corpus 77.2K tok/s
Corpus p50 96.9ms
Query 5.4K tok/s
Query p50 49.0ms
MMarcoReranking
Multilingual MARCO passage reranking (Chinese)
Quality
ndcg at 10 0.0869
map at 10 0.0659
mrr at 10 0.0675
NFCorpus
Biomedical literature search from NutritionFacts.org
Corpus: 3,593 Queries: 323
Performance L4 b1 c16
Corpus 42.8K tok/s
Corpus p50 89.2ms
Query 1.4K tok/s
Query p50 57.1ms
SCIDOCS
Citation prediction, document classification, and recommendation for scientific papers
Corpus: 25,656 Queries: 1,000
Performance L4 b1 c16
Corpus 41.3K tok/s
Corpus p50 66.2ms
Query 3.7K tok/s
Query p50 48.2ms
SciFact
Scientific claim verification using research literature
Corpus: 5,183 Queries: 300
Performance L4 b1 c16
Corpus 53.2K tok/s
Corpus p50 72.8ms
Query 5.1K tok/s
Query p50 51.0ms
StackOverflowQA
Programming question answering from Stack Overflow
Corpus: 19,931 Queries: 1,994
Performance L4 b1 c16
Corpus 40.5K tok/s
Corpus p50 74.0ms
Query 62.4K tok/s
Query p50 75.6ms
T2Reranking
Chinese passage ranking benchmark
Quality
ndcg at 10 0.6533
map at 10 0.4716
mrr at 10 0.6980