metadata

title: InferBench
emoji: 🥇
colorFrom: green
colorTo: indigo
sdk: gradio
app_file: dashboard/app.py
build_command: pip install -r requirements.txt
pinned: true
license: apache-2.0
short_description: A cost/quality/speed Leaderboard for Inference Providers!
sdk_version: 5.19.0
tags:
  - leaderboard

InferBench

Evaluate the quality and efficiency of image gen api's.

Installation

Install dependencies

Install dependencies with conda like that:

conda env create -f environment.yml

Install uv

Install uv with pip like that:

uv venv --python 3.12

Then activate the environment:

source .venv/bin/activate

Then install the dependencies with uv:

uv sync --all-groups

Usage

Create .env file with all the credentials you will need.

This is how you can generate the images.

python sample.py replicate draw_bench genai_bench geneval hps parti

This is how you would evaluate the benchmarks once you have all images:

python evaluate.py replicate draw_bench genai_bench geneval hps parti

Leaderboard

The leaderboard is hosted on Hugging Face with gradio.

To run the dashboard locally, you can use the following command:

python dashboard/app.py

To deploy the dashboard to Hugging Face, you can use the following commands:

First, add the remote:

git remote add hf https://huggingface.co/spaces/PrunaAI/InferBench

Then push the changes of your branch to the remote:

git push hf $(git rev-parse --abbrev-ref HEAD):main --force