---
name: Toolchain tests
description:
    Instructions for authoring, structuring, and running toolchain tests using
    the file_test infrastructure.
---

# Toolchain tests

<!--
Part of the Carbon Language project, under the Apache License v2.0 with LLVM
Exceptions. See /LICENSE for license information.
SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-->

## Introduction

This skill provides guidelines and patterns for creating and updating tests for
the Carbon toolchain, especially file tests in `toolchain/*/testdata/` (for
example, `toolchain/check/testdata/`).

Toolchain tests evaluate Carbon source files through Lexing, Parsing, Checking,
and optionally Lowering. Output (for example SemIR dumps, Clang errors) is
captured and validated using inline CHECK records.

## Structure and Authoring

### File Layout and Headers

Test files must start with the standard Carbon license, followed by
configuration comments. Separate sections with blank comment lines (`//`).

```carbon
// Part of the Carbon Language project, under the Apache License v2.0 with LLVM
// Exceptions. See /LICENSE for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
// INCLUDE-FILE: toolchain/testing/testdata/min_prelude/...
//
// AUTOUPDATE
```

-   `// AUTOUPDATE` is mandatory for files using CHECK markers.
-   `// TIP:` lines are automatically generated by the autoupdater. You do not
    need to hand-write them. It is harmless to add them, but the script will
    handle it.

### Minimized Preludes

When writing tests entirely unrelated to the Core package, specify a minimal
prelude file using `// INCLUDE-FILE`. Usually, include
`toolchain/testing/testdata/min_prelude/` scripts, such as `int.carbon` or
`primitives.carbon`. This significantly speeds up execution and minimizes STDOUT
noise.

### Split Tests and `[[@TEST_NAME]]`

A single physical file can test multiple scenarios using split constraints:

```carbon
// --- passing_case.carbon
library "[[@TEST_NAME]]";

// ...

// --- fail_bad_case.carbon
library "[[@TEST_NAME]]";
// ...
```

-   Use `library "[[@TEST_NAME]]";` in each split when necessary to prevent name
    conflicts or redefining the default library.
-   Exactly `[[@TEST_NAME]]` (including the brackets) should be used. The test
    infrastructure automatically replaces it with the split's filename minus
    `todo_` and `fail_` prefixes.
-   **Do not put code that is expected to pass and code that is expected to fail
    into the same split.** Validation relies on non-failing splits producing
    absolutely no errors and failing splits producing the correct compiler
    errors independently.

### File Prefixing: `fail_` and `todo_`

Expected failures must be differentiated from unexpected failures (and from
bugs). Include prefixes to name individual split files or the main test:

-   `fail_...`: The test should and does produce compiler errors.
-   `todo_fail_...`: The test should produce errors but currently does not.
-   `fail_todo_...`: The test does produce errors or crashes, but it shouldn't
    (or produces the wrong errors or otherwise misbehaves with errors).
-   `todo_...`: The test has some incorrect behavior, but doesn't produce errors
    currently, and shouldn't.

**Main File Naming**: The main test file (and any split-files) must have a
`fail_` prefix if they have an associated error. **Exception**: The main file
may omit `fail_` if it contains a least one split that has a `fail_` prefix.

Both the `fail_` and `todo_` prefixes are stripped from filename properties like
`[[@TEST_NAME]]`.

### Test Code Comments

-   **No agent thinking:** Do not include comments describing your reasoning or
    "train of thought" (for example, "Wait, but...") inside the test files. Any
    comments left in tests should be concise and describe what the test _itself_
    is validating for human readers.

## SemIR Dumps and Minimizing Output

Limit STDOUT checks to the logic under test. Always use `//@dump-sem-ir-begin`
and `//@dump-sem-ir-end` around the specific declarations/blocks where SemIR
output is desired. Only use these markers and **not**
`--dump-sem-ir-ranges=if-present` or similar extra args—new tests use
`//@dump-sem-ir...` to naturally filter output to the highlighted segments based
on the default behavior.

```carbon
//@dump-sem-ir-begin
fn F(x:? form(ref i32));
//@dump-sem-ir-end
```

## Creating/Updating the Output

AI tools should **never** hand-write or manually touch `// CHECK:STDOUT:` or
`// CHECK:STDERR:` comments.

Write your Carbon test code, headers, and `// AUTOUPDATE` then run the test
updater:

```bash
./toolchain/autoupdate_testdata.py toolchain/PATH/TO/YOUR/TEST.carbon
```

Review the updated test outputs (for example, by way of `git diff`). Ensure
logic paths are correctly tested rather than producing massive boilerplate
blocks.
