Inspect AI

# Inspect > Open-source framework for large language model evaluations ## Basics - [Welcome](https://inspect.aisi.org.uk/index.html.md): Welcome and overview of Inspect AI. - [Tutorial](https://inspect.aisi.org.uk/tutorial.html.md): Step-by-step walkthrough of Inspect, from a first eval through agents, analysis, and the broader feature set. - [Options](https://inspect.aisi.org.uk/options.html.md): Covers the various options available for evaluations as well as how to manage model credentials. - [Log Viewer](https://inspect.aisi.org.uk/log-viewer.html.md): How to use Inspect View to develop and debug evaluations, including how to provide additional log metadata and integrate it with Python logging. - [VS Code](https://inspect.aisi.org.uk/vscode.html.md): Using the Inspect VS Code Extension to run, tune, debug, and visualise evaluations. ## Components - [Tasks](https://inspect.aisi.org.uk/tasks.html.md): Tasks bring together datasets, solvers, and scorers to define an evaluation. Strategies for creating flexible, re-usable tasks and for configuring and overriding them at runtime. - [Datasets](https://inspect.aisi.org.uk/datasets.html.md): Datasets provide samples to evaluation tasks. How to adapt various data sources for use with Inspect, including multi-modal data. - [Solvers](https://inspect.aisi.org.uk/solvers.html.md): Solvers encompass prompt engineering and other elicitation strategies. Using built-in solvers and creating your own. - [Scorers](https://inspect.aisi.org.uk/scorers.html.md): Scorers evaluate the work of solvers and aggregate scores into metrics. Overview of the built-in scorers and pointers to custom scorers, metrics, and the scoring workflow. ## Models - [Using Models](https://inspect.aisi.org.uk/models.html.md): Models provide a uniform API for evaluating a variety of large language models and using models within evaluations. - [Providers](https://inspect.aisi.org.uk/providers.html.md): Usage details and available options for the various supported model providers. - [Caching](https://inspect.aisi.org.uk/caching.html.md): Caching enables you to cache model output to reduce API calls, saving both time and expense. - [Concurrency](https://inspect.aisi.org.uk/models-concurrency.html.md): Tuning model API connection limits, adaptive concurrency, and rate-limit handling. - [Compaction](https://inspect.aisi.org.uk/compaction.html.md): Compacting message histories for long-running agents that exceed the context window. - [Fallbacks](https://inspect.aisi.org.uk/fallbacks.html.md): Retrying classifier-refused requests on a fallback model, and how fallbacks are recorded in logs, dataframes, and the viewer. - [Multimodal](https://inspect.aisi.org.uk/multimodal.html.md): How to use images, audio, and video in evaluations. - [Reasoning](https://inspect.aisi.org.uk/reasoning.html.md): Additional options and data available for reasoning models. - [Structured Output](https://inspect.aisi.org.uk/structured.html.md): How to constrain model output to a particular JSON schema. - [Batch Mode](https://inspect.aisi.org.uk/models-batch.html.md): Using batch processing APIs for model inference. ## Scoring - [Standard Scorers](https://inspect.aisi.org.uk/standard-scorers.html.md): The built-in scorers that ship with Inspect (text matching, multiple choice, model grading, math, and perplexity), the metrics they report, and where to go next. - [Custom Scorers](https://inspect.aisi.org.uk/custom-scorers.html.md): Write your own scorers using the Score, Value, and Target types, call models from scorers, and access the sandbox. - [Model Grading](https://inspect.aisi.org.uk/model-graded.html.md): Use another model to grade outputs with model_graded_qa() and model_graded_fact(). Customise templates, instructions, grader models, multi-model voting, and chat history. - [Scoring Metrics](https://inspect.aisi.org.uk/metrics.html.md): Built-in scoring metrics, metric grouping, clustered standard errors, custom metrics, and reducing epoc […gekürzt]