AISBench Benchmark Tool

🚀 Get Started

  • Tool Installation & Uninstallation
  • Quick Start
  • Dataset Preparation Guide

🧭 Basic Tutorials

  • Supported Evaluation Scenarios
  • Explanation of Evaluation Results
  • Detailed Parameter Description

🔬 Advanced Tutorials

  • Running AISBench with a Custom Configuration File
  • Service-Oriented Steady-State Performance Testing
  • Request Sending Rate (RPS) Distribution Control and Visualization Guide
  • Guide to Multi-Turn Dialogue Evaluation
  • Guide to Using Random Synthetic Datasets
  • Guide to Using Custom Datasets
  • Evaluation Using Judge Model
  • Speculative Decoding Metrics Collection
  • Prefix Cache Dataset Generation, Benchmarking, and Hit-Rate Analysis
  • Response Anomaly Detection

📐 Extended Benchmarks

  • Extended Multimodal Generation Benchmarks
  • agent evaluation benchmark
    • τ²-Bench
    • SWEbench User Guide
    • SWE-Bench Pro User Guide
    • Harbor Terminal-Bench

💪 Best Practices

  • Evaluating the Mathematical Capabilities of DeepSeek-R1-Distill-Qwen-14B Based on NVIDIA A100 Accelerator Card: 100% Paper Reproduction
  • Evaluating DeepSeek-R1’s Mathematical Capabilities Based on Ascend 800I-A2: 100% Paper Reproduction
  • Reproducing Dataset Evaluation Results from Large Language Model (LLM) Papers (Technical Reports) — Taking the GPQA Dataset Used by DeepSeek R1 as an Example

❓ FAQs

  • AISBench FAQ (Frequently Asked Questions)
  • Error Code Description

👨‍💻 Developer Guide

  • Contributing Guide
  • Supporting New Model Backends
  • Supporting New Datasets and Accuracy Evaluators
  • Supporting New Inferencers

📝 Prompt Engineering

  • Prompt Template
  • Meta Template
  • Prompt Overview
  • Retriever

🏷️ Others

  • 🔜 Coming Soon
  • 🤝 Acknowledgments
AISBench Benchmark Tool
  • agent evaluation benchmark
  • View page source

agent evaluation benchmark

  • τ²-Bench
    • τ²-Bench Evaluation Benchmark Introduction
    • Quick Start with τ²-Bench Evaluation in AISBench
    • Continue Evaluation After Interruption
    • Multiple Executions of a Single Case (pass^k)
    • Using the TAU2-mini Sampled Subset
  • SWEbench User Guide
    • 1. Feature Overview
    • 2. Prerequisites
    • 3. Minimal Configuration (Run First, Tune Later)
    • 4. Run Commands
    • 5. How to Read Outputs
    • 6. Common Issues and Troubleshooting (SWEB Error Codes)
    • 7. Advanced Tips (Optional)
  • SWE-Bench Pro User Guide
    • 1. Feature Overview
    • 2. Prerequisites
    • 3. Minimal Configuration (Run First, Tune Later)
    • 4. Run Commands
    • 5. How to Read Outputs
    • 6. Common Issues and Troubleshooting (SWEBP Error Codes)
    • 7. Advanced Tips (Optional)
  • Harbor Terminal-Bench
    • Harbor Introduction
    • Quick Start with Harbor Terminal-Bench 2.0 in AISBench
    • Continue Evaluation After Interruption
    • Multiple Executions of a Single Case (pass@k)
    • Task Configuration (in datasets) - Key Parameters
    • Agent Configuration (in models) - Related Parameters
Previous Next

© Copyright 2025, AISBench AI System Performance Evaluation Benchmark Committee.

Built with Sphinx using a theme provided by Read the Docs.