What Prometheus 2 does
Prometheus 2 is a family of open-source evaluator language models in 7B and 8x7B sizes, released in May 2024, for judging LLM outputs. It supports absolute grading, which assigns a score from 1 to 5, and relative grading, which chooses between two responses labeled A or B. The prometheus-eval Python package uses vLLM for local inference and LiteLLM for API-based models, and includes batch grading methods. Reference answers are optional. The project states that it aims to address fairness, controllability, and affordability. It is licensed under Apache-2.0 and is aimed at teams building internal evaluation pipelines.
How Prometheus 2 compares with its closest alternatives
4 of the 5 closest alternatives have a free plan, as Prometheus 2 does. Only Dydema lists a starting price, $1 a month. All 5 offer an API, as Prometheus 2 does. Prometheus 2 offers a command-line tool. Of the 4 alternatives with platforms on record, 1 offers a command-line tool. Prometheus 2 was founded in 2024; the alternatives were founded between 2016 and 2025, 3 of them earlier. Prometheus 2 has no single headquarters on record; its alternatives with a known headquarters are in the United States.