Results
sensor-calibration-fit
We post-trained a model on this submitted environment, then scored it back on the environment’s own verifier. Here is what moved.
/Users/lixiangyi/benchflow/posttrain-arena/.claude/worktrees/thirsty-albattani-a06d0d/starting-kit/examples/sensor-calibration-fitBefore → after
Pass-rate over 4 in-domain samples, base checkpoint versus the model post-trained on this environment. This is a pipeline + trainability check, not a generalization score, the competition measures held-out Δ on a separate suite.
Supervised fine-tuning
The training pass that produced the trained checkpoint.
Checkpoint: pta-sft-checkpoints:/ckpt/qwen3.5-0.8b-sft-sensor-calibration-fit-full
How pass-rate was computed
model generates an answer to the task.md prompt -> the largest balanced JSON object in each generation is written to the env output path (/root/results.json) inside the env docker image -> the env verifier (verifier/test.sh) scores it via /logs/verifier/reward.txt
Sample completions
A few graded excerpts, side by side, so the gain isn’t just a number.
Base
{ "S1": {"gain": 0.0, "offset": 0.0, "rmse": 0.0, "n_excluded": 0, "corrected": []}, "S2": {"gain": 0.0, "offset": 0.0, "rmse": 0.0, "n_excluded": 0, "corrected": []}, "S3": {"
{ "sensors": {} }
{}
{ "sensors": {} }
Trained
{ "sensors": { "S1": { "gain": 1.0347493432452644, "offset": -1.9521585898490272, "rmse": 0.10478475642612474, "n_excluded": 0, "corrected": [ 19.9545197559
{ "sensors": { "S1": { "gain": 1.0347493432452644, "offset": -1.9521585898490272, "rmse": 0.10478475642612474, "n_excluded": 0, "corrected": [ 19.9545197559
{ "sensors": { "S1": { "gain": 1.0347493432452644, "offset": -1.9521585898490272, "rmse": 0.10478475642612474, "n_excluded": 0, "corrected": [ 19.9545197559
{ "sensors": { "S1": { "gain": 1.0347493432452644, "offset": -1.9521585898490272, "rmse": 0.10478475642612474, "n_excluded": 0, "corrected": [ 19.9545197559
Notes & caveats
Real end-to-end run on a single Modal H100 (sm_90, transformers Qwen3_5ForCausalLM, sdpa attn; flash-attn/fla kernels not in image, numerically-correct PyTorch fallback). SFT overfits ONE oracle example (this env is a single fixed instance): base model emits only an empty {"sensors": {}} skeleton (0/4), the SFT'd model reproduces the exact results.json (4/4) -> clean +1.0 delta. KEY FIX uncovered during this run: the answer artifact sits at the END of the assistant turn and the full record is ~2636 tokens (810 prompt + 1826 answer); an initial SFT at seq_len 2048 TRUNCATED the S2/S3 tail, so the model hallucinated those values and never closed the JSON (trained==base==0). Retraining at seq_len 3072 (no truncation) + raising eval max_new_tokens to 2000 fixed it; greedy decode of the fixed checkpoint scores 1.0 and reproduces all 1536 answer chars. trained_pass_rate here is at temperature=0.2 sampling (n=4), genuinely sampled not greedy. cost_usd ~$1.94 = honest total H100 wall-seconds across both SFT runs (incl. the abandoned seq-2048 run), both eval rounds, and diagnostics x $3.95/hr; the single clean SFT+eval alone is ~$0.45.