BetterEvals

Cheaper, cost-aware, statistically valid A/B evaluation of language models

Author: Kamil Ciosek

Do you want to compare LLMs minimizing the API cost?

Enter BetterEvals: a library that performs statistically valid LLM comparisons in a way that minimizes the dollar cost of evaluation, not just the number of API calls. It does all of the below.

• Take into account the real cost model with three prices (cached input, uncached input, ouput)
• Stop as soon as the comparison result is statistically valid
• Adapt the number of samples per model (cheaper, larger-variance models are evaluated more often)

BetterEvals gives the biggest benefit where there is a large variance in the rewards associated with different model responses.

The statistical methodology ensuring the validity of the stopping rule is based on the excellent paper Estimating means of bounded random variables by betting by Ian Waudby-Smith, Aaditya Ramdas (the authors are not involved with the BetterEvals project).

It cuts evaluation cost by ~20% on the MMLU-Pro benchmark, relative to the naive method.