Cookbook

Inspect a recorded HF training run

Walk through the HF submission flow and inspect the logs, adapter, and verifier results from the completed experiment.

This walkthrough describes the current Submission Lab and the bounded experiment recorded on September 21, 2026. The Arena now provides a bounded train-and-evaluate path for the Google Auto practice task. The examples and screenshots below also preserve the historical hillclimb evidence; that checkpoint-search recipe is distinct from the current single training run. The private lab and artifact links return a Hugging Face 404 page to readers without access. Signing in alone does not grant access. A CPU submission smoke test, a completed GPU job, and a passing task verifier are three different milestones.

Before you start

• Confirm that your Hugging Face account has been granted access to the Submission Lab, the benchflow Jobs namespace, and the linked artifact repositories.
• For command-line inspection, use Python 3.10 or newer and an existing Hugging Face login with those permissions. Keep tokens in the credential store or secret environment; never paste them into recipes, screenshots, or logs.
• For a new environment, follow the environment authoring specification and validate its oracle and unchanged verifier first.
• Check the existing reservation before launching. One active or uncertain run blocks another. Completed runs no longer permanently block new runs; the shared budget must still have sufficient unallocated compute.

The recorded GPU runner uses Qwen/Qwen3.6-27B at revision 6a9e13bd6fc8f0983b9b99948120bc37f49c13e9 with 4-bit NF4 loading on an HF A100 Large. This identifies the recorded configuration; GPU availability must be checked for each new run.

How submission works

Choose SkillsBench in the Arena competition selector. Its current practice protocol is one Google Auto repair task, Qwen3.6-27B LoRA, and three original verifier checks. This measures seen-task fit, not held-out performance.

1. Open the Submission Lab and open the Train tab.
2. Review the model, optimizer steps, learning rate, and LoRA rank. The general GPU recipe accepts 20, 50, or 100 steps; learning rates 0.00002, 0.00005, or 0.0001; and ranks 8, 16, or 32.
3. Select Train & evaluate, review the allocation, and authorize with an HF account that has BenchFlow write or admin access. Submit once and retain the returned job link. The runner trains, saves and reloads the adapter, and executes the original verifier.
4. If the same request already exists, or a run is active or uncertain, inspect it before retrying. API retries must reuse the request ID and unchanged recipe. After a terminal run, a new request can reserve another run within the remaining budget.

The current GPU route is /api/arena/train; /api/arena/recipe previews the configuration and /api/arena/jobs returns receipts and results. Include a stable request ID and reuse it after an uncertain response. The older /api/gpu/submit route remains historical. The separate /api/submit route is a CPU planner smoke test and rejects training mode. A completed CPU test proves submission and artifact plumbing; it does not perform optimizer updates.

The lab reserves a run durably before launch and sets a two-hour HF job timeout. Each new Arena run checks current HF prices and reserves GPU plus verifier compute, capped at $11 per run within the $200 project allowance. A conservative $50 allocation covers earlier work and unresolved ancillary billing; reservations are not invoices and are not automatically refunded. Inspect /api/arena/budget before launching.

To evaluate an existing artifact, use Submissions → Submit model. Register a public or accessible BenchFlow Qwen3.6-27B LoRA safetensors adapter. Its exact revision is pinned. A BenchFlow editor can then select Evaluate to launch the same fixed verifier. The status and verified check count appear in the submissions table.

Follow the job

Open the returned Jobs page under the same authorized account. Scheduling means the GPU has not started training. Running means the process is active; inspect the logs for model loading and optimizer-step output. Completed means the command exited successfully—continue to artifact and verifier checks below.

This read-only example inspects the recorded hillclimb job and the last log lines. It creates no GPU job.

python3 -m venv .venv-hf-inspect
source .venv-hf-inspect/bin/activate
python -m pip install huggingface_hub==1.32.0
python - <<'PY'
from huggingface_hub import HfApi
api = HfApi()  # uses your existing HF login
job_id = "6ab0d98852d0dbd7f1d77a38"
job = api.inspect_job(job_id=job_id, namespace="benchflow")
print(job.status.stage)
for line in api.fetch_job_logs(
    job_id=job_id, namespace="benchflow", tail=30
):
    print(line)
PY

Use the GPU run ledger to connect the reservation, job ID, model artifact, and evaluation report. The hillclimb is an entry in the ledger’s additional runs, not its primary job. New Arena reservations are stored separately in the Arena job ledger. The Space refreshes job status from HF; the receipt alone is not proof that a job is still running or has completed.

Verify the result

1. Training: logs contain optimizer steps and loss, rather than only planner output.
2. Artifact: for new training, open Trained adapter in Results. The dataset folder arena/adapters/<run_id> contains the saved adapter and tokenizer. The result JSON records the configuration; historical runs instead used model repositories.
3. Inference: check the generated repair and evaluation record. Saving an adapter alone does not establish usable inference.
4. Task: the original verifier checks the notes file, patch file, and successful build. Check individual test results and infrastructure status.
5. Cleanup: each verifier sandbox reports termination. Reconcile live job and sandbox state before another experiment.

Inspect a current run without starting compute. Install huggingface_hub and httpx, and use your existing HF login. Set run_id to the receipt returned by your submission. A passing report requires completed status, the requested optimizer steps, adapter reload, the original checks, and confirmed cleanup.

python - <<'PY'
import httpx
from huggingface_hub import get_token
run_id = "arena-8be1f36c6eae"
base = "https://benchflow-posttrain-submission-lab-20260920.hf.space"
response = httpx.get(base + "/api/arena/jobs",
                    headers={"Authorization": "Bearer " + get_token()}, timeout=60)
response.raise_for_status()
run = next(row for row in response.json() if row["run_id"] == run_id)
print("HF status:", run["status"])
print("Result:", run.get("result", {}))
print("Full report:", run["report_url"])
print("Saved adapter:", run.get("adapter_url", "Not saved/reloaded yet"))
PY

New training outputs are already evaluated by their run. To register one separately through Submit model, first publish its adapter files to an accessible HF model repository; that form does not accept dataset paths. The separate recorded hillclimb report uses a different schema, shown below.

The hillclimb report stores the generated proposal, verifier log, per-test results, checkpoint hashes, rollback decisions, and accepted training steps. Read it without launching anything:

python - <<'PY'
import json
from pathlib import Path
from huggingface_hub import hf_hub_download
repo = "benchflow/posttrain-google-auto-hillclimb-20260921"
path = hf_hub_download(repo, "hillclimb-result.json", force_download=True)
report = json.loads(Path(path).read_text())
print("Status:", report["status"])
print("Baseline:", report["baseline"]["passed_tests"], "/ 3")
for candidate in report["rounds"]:
    print(candidate["round"], candidate["update_steps"],
          candidate.get("passed_tests"), candidate["accepted"],
          candidate.get("sandbox_terminated"))
print("Incumbent:", report["incumbent"])
PY

Recorded overfit experiment

This is deliberately a seen-task experiment on a google/auto Java repair. Supervised updates learn a complete successful teacher repair. The original verifier selects checkpoints: accept only a strict increase in passed tests; restore the incumbent on a tie or regression. It is verifier-guided SFT checkpoint search, not RL/GRPO or evidence of held-out generalization.

The search performed 28 optimizer steps in total. The accepted adapter retains 16 steps because the rejected 4- and 8-step candidates were rolled back. The score covers three checks on one task, not three independent tasks or a 41-task benchmark.

To inspect the inputs used by this run, authorized collaborators can open the original reservation and pinned configuration. Its config.data_revision identifies the artifact snapshot. That snapshot contains these source inputs:

evaluation/hillclimb-v1/run.py       # training + original verifier
evaluation/hillclimb-v1/core.py      # extraction, acceptance, rollback policy
evaluation/single-task/example.json # successful teacher repair
evaluation/task_tool_parser.py      # generated tool-call parser
evaluation/google-auto-task/verifier/test_outputs.py
evaluation/google-auto-task/verifier/run_passed.sh

The experiment uses a 4-step initial update, doubles rejected update budgets up to 32, allows at most six candidates, and stops at 3/3. Its training settings are learning rate 0.0002, LoRA rank 32 / alpha 64, dropout 0, seed 42, batch size 1, and a constant learning-rate schedule. The dedicated hillclimb runner has different settings from the general Space recipe selector.

A fresh rerun is currently blocked on a portable launcher and documented preflight. The historical scripts hardcode the benchflow namespace, output repository, shared ledger, and reservation. Simply changing the run ID or invoking the downloaded runner is not a supported rerun path. The historical launcher is intentionally tied to its existing reservation and must not be rerun unchanged. Keep the two-hour GPU timeout, bounded candidate count, per-verifier timeout, and sandbox cleanup. The normal Space submit button does not currently launch this hillclimb recipe.

Before a runnable recipe can be published, maintainers must provide access to the pinned teacher data and verifier image; parameterize every namespace, output, and ledger write; test a fresh unique reservation and image preflight; and reconcile current GPU plus sandbox prices against the remaining $200 cap. This page does not implement those controls. Any eventual rerun should compare its report against the recorded result; do not assume identical scores from a new GPU/software configuration. A successful reproduction requires valid original-verifier evaluations, checkpoint rollback on rejected candidates, an accepted adapter, and terminated sandboxes.

Troubleshooting

The job completed, but nothing trained.

Check whether it was the CPU planner route. Follow the GPU run receipt and look for optimizer-step logs.

The job is scheduling.

No training progress is implied. Check current HF hardware availability and the job state; avoid duplicate submissions while waiting.

Submit is blocked by an existing reservation.

Inspect the linked job and reconcile its artifacts and cost first. The lab uses a durable guard against duplicate spending; do not delete the reservation to bypass it.

A Space, Jobs, or artifact link returns 404 or asks me to log in.

These resources are private. Hugging Face can return 404 rather than a login prompt to an unauthorized reader. Confirm the account and its resource permissions with the maintainer; signing in does not itself grant access. The screenshots remain readable here, but the CLI examples also require access.

The loss fell, but the verifier did not improve.

Loss measures teacher-token prediction. Read the free-generation output, patch application, and original test results; the hillclimb accepts only a verifier improvement.

A verifier timed out or a sandbox failed.

Treat the evaluation as invalid, not as a model score. The runner rejects acceptance and stops on infrastructure failure or unconfirmed sandbox termination.