Recovering deleted experiment rows with versioned BTQL queries

Applies to:

Plan -
Deployment -

Summary

Issue: Experiment rows were accidentally deleted and need to be restored. Resolution: Use a versioned BTQL query to retrieve rows as they existed before deletion, referencing an identified transaction version (_xact_id), then re-insert them into the experiment.

Resolution Steps

Step 1: Identify a pre-deletion `_xact_id`

Each row in Braintrust is stamped with a _xact_id, a monotonically increasing transaction ID. You need a value from before the deletion occurred. If rows still exist in the experiment, query for the maximum _xact_id:

curl -X POST https://api.braintrust.dev/btql \
  -H "Authorization: Bearer <YOUR_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "SELECT MAX(_xact_id) AS max_xact_id FROM experiment('"'"'<EXPERIMENT_ID>'"'"')"
  }'

Use the max_xact_id value returned as your version in Step 2. If all rows were deleted before capturing a _xact_id, contact Braintrust support so we can help identify the correct version from internal logs.

Step 2: Run a versioned query to recover the rows

Pass the _xact_id as the version parameter to retrieve the experiment as it existed at that transaction:

curl -X POST https://api.braintrust.dev/btql \
  -H "Authorization: Bearer <YOUR_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "SELECT * FROM experiment('"'"'<EXPERIMENT_ID>'"'"') ORDER BY _pagination_key LIMIT 1000",
    "version": "<XACT_ID>"
  }'

If more than 1000 rows were deleted, paginate using the cursor token returned in the response:

curl -X POST https://api.braintrust.dev/btql \
  -H "Authorization: Bearer <YOUR_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "SELECT * FROM experiment('"'"'<EXPERIMENT_ID>'"'"') ORDER BY _pagination_key LIMIT 1000 OFFSET '"'"'<CURSOR_TOKEN>'"'"'",
    "version": "<XACT_ID>"
  }'

Repeat until no cursor is returned in the response.

Step 3: Re-insert the recovered rows

Strip server-managed fields before re-inserting, then POST to the experiment’s insert endpoint:

import requests

HEADERS = {
    "Authorization": "Bearer <YOUR_API_KEY>",
    "Content-Type": "application/json",
}
STRIP_FIELDS = {"experiment_id", "project_id", "_xact_id", "_pagination_key", "audit_data"}

recovered_rows = [...]  # rows from the Step 2 response

rows_to_insert = [
    {k: v for k, v in row.items() if k not in STRIP_FIELDS}
    for row in recovered_rows
]

resp = requests.post(
    "https://api.braintrust.dev/v1/experiment/<EXPERIMENT_ID>/insert",
    headers=HEADERS,
    json={"events": rows_to_insert},
)
resp.raise_for_status()
print(f"Re-inserted {len(rows_to_insert)} rows")

After re-insertion, rows are immediately visible in the Braintrust UI.

Additional Information

Why versioned queries can be slow

Versioned queries disable segment elimination, so the query engine must scan all stored data to reconstruct state at a given transaction. Query time scales with the total storage history of the experiment, not just currently visible rows.

Versioned queries are subject to the standard 30-second BTQL timeout and there is no per-request way to extend it. If the scan times out on a very large experiment, contact Braintrust support for assisted recovery.

Why certain fields must be stripped before re-insertion

The /v1/experiment/{id}/insert endpoint rejects server-assigned fields with a 400 error. Strip these from each row before posting:

Field	Reason
`experiment_id`	Set from the URL path
`project_id`	Set from the URL path
`_xact_id`	Assigned by the server on insert
`_pagination_key`	Assigned by the server on insert
`audit_data`	Read-only metadata

​Summary

​Resolution Steps

​Step 1: Identify a pre-deletion _xact_id

​Step 2: Run a versioned query to recover the rows

​Step 3: Re-insert the recovered rows

​Additional Information

​Why versioned queries can be slow

​Why certain fields must be stripped before re-insertion

Summary

Resolution Steps

Step 1: Identify a pre-deletion `_xact_id`

Step 2: Run a versioned query to recover the rows

Step 3: Re-insert the recovered rows

Additional Information

Why versioned queries can be slow

Why certain fields must be stripped before re-insertion