┌─ new project ──────────────────────────────────────────────────────────────────────────────────┐ │ Acme Suppliers Customer Support Model │ │ │ │ Acme Suppliers needs a compact customer support model that can answer order status, returns, │ │ shipping, billing, and product availability questions in a clear brand voice. The dataset will │ │ turn policy docs, tone notes, and agent-approved examples into ChatML rows covering routine │ │ support, messy edge cases, and clean handoffs when a human should step in. │ └────────────────────────────────────────────────────────────────────────────────────────────────┘ ┌─ 1/7 interview ────────────────────────────────────────────────────────────────────────────────┐ │ ✓ your idea │ │ ✓ refund-policy.pdf refund policy source - 6 searchable chunks │ │ ✓ tone notes brand voice and support tone notes │ │ · round 1 7 questions │ │ ✓ target model A small 2B-4B LLM that can run on device │ │ ✓ input distribution customer chats with typos, urgency, missing order details │ │ ✓ output contract 2-4 sentence answers with policy, next step, no fluff │ │ ✓ coverage map refunds, shipping, billing, product fit, order status │ │ ✓ negative space legal, medical, and cancellations hand off to humans │ │ ✓ tone persona warm, efficient support voice; apologize once, then solve │ │ ✓ deployment context edge API replies under 200 tokens for chat widgets │ │ · round 2 5 questions │ │ ✓ quality rubric grounded in policy, order-specific, date or next step │ │ ✓ edge cases split shipments, gift cards, chargebacks, marketplace orders │ │ ✓ volume budget 240 rows across topics, edge cases, and holdout probes │ │ ✓ gold examples 3 approved agent replies used as gold examples │ │ ✓ privacy compliance synthetic data only; no real emails, cards, or order IDs │ │ · task definition support chat for policy-backed ecommerce questions │ │ · grounding sources refund policy PDF and tone notes become source chunks │ │ ✓ interview/gold-draft.md gold examples drafted from approved agent replies │ │ ✓ gold examples approved │ │ ✓ understanding confirmed user feedback folded into the plan │ │ ✓ interview/sample-rows.md sample training rows ready for review │ │ ✓ sample rows approved │ │ ✓ interview complete 12 slots answered · 2 derived · 3 gates · 00:41 │ ├────────────────────────────────────────────────────────────────────────────────────────────────┤ │ ✓ interview [████████████████] done │ └────────────────────────────────────────────────────────────────────────────────────────────────┘ ┌─ 2/7 research ─────────────────────────────────────────────────────────────────────────────────┐ │ ✓ models 3 candidates scored: Qwen3.5-9B, Llama-4-Scout-8B, Phi-5-mini │ │ ✓ format Qwen 3.5 uses ChatML; eos <|im_end|>; system prompt supported │ │ ✓ row count Single-domain support skill: 200-500 rows is the researched band │ │ ✓ license Qwen: Apache-2.0. DeepSeek/MiMo outputs usable for training per ToS (recorded) │ │ ✓ model Qwen/Qwen3.5-9B · confirmed by you │ │ ✓ research complete 4 findings · 00:09 │ ├────────────────────────────────────────────────────────────────────────────────────────────────┤ │ ✓ research [████████████████] done │ └────────────────────────────────────────────────────────────────────────────────────────────────┘ ┌─ 3/7 source of truth ──────────────────────────────────────────────────────────────────────────┐ │ ✓ sot/03-dataset-spec.md task · schema · counts · template │ │ ✓ sot/04-coverage-matrix.md 4 topics × 6 scenarios · 240 rows │ │ ✓ sot/05-style-guide.md voice · bounds · formatting │ │ ✓ sot/06-quality-rubric.md 5-dimension 1-5 rubric │ │ ✓ sot/07-negative-policy.md refusals & handoffs │ │ ✓ sot/08-samplers.yaml 6 personas · 4 registers · entity pools │ │ ✓ sot/09-grounding-index.md source docs → chunk ids │ │ ◆ eval/probes.jsonl 18 probes across 18 cells · frozen │ │ ✓ source of truth signed off │ │ ✓ source of truth complete 7 docs · probes frozen · 00:11 │ ├────────────────────────────────────────────────────────────────────────────────────────────────┤ │ ✓ source of truth [████████████████] done │ └────────────────────────────────────────────────────────────────────────────────────────────────┘ ┌─ 4/7 generation ───────────────────────────────────────────────────────────────────────────────┐ │ ✓ preflight 3 provider checks passed │ │ ◆ production gate · post sot 7 checks passed │ │ ✓ judge calibration b001 · agreement 91% │ │ ! i-0001 Assistant opens with 'Certainly!' │ │ ! i-0002 US-only date formats in replies │ │ ✓ batch 01 calibration · 17/20 accepted · $0.45 · 2 issues logged │ │ ✓ judge calibration b002 · agreement 93% │ │ ✓ i-0001 compiled → gate A detector │ │ ✓ batch 02 calibration · 18/20 accepted · $0.45 │ │ ✓ spot check 5/5 approved │ │ ✓ batch 03 46/50 accepted · $0.21 │ │ ✓ batch 04 47/50 accepted · $0.21 │ │ ↻ resumed │ │ ✓ batch 05 48/50 accepted · $0.21 │ │ ✓ batch 06 48/50 accepted · $0.21 │ │ ✓ batch 07 16/18 accepted · $0.21 │ │ ✓ generation complete 240/240 rows · 7 batches · 01:22 │ ├────────────────────────────────────────────────────────────────────────────────────────────────┤ │ ✓ generation [████████████████] done │ └────────────────────────────────────────────────────────────────────────────────────────────────┘ ┌─ 5/7 corpus audit ─────────────────────────────────────────────────────────────────────────────┐ │ ◆ production gate · post calibration 1 checks passed │ │ ✓ probe coverage 12/12 sizeable cells have a frozen probe │ │ ! budget $4.82 of $6.00 (80%) · pauses safely at the cap │ │ ✓ fix spec 3 to fix · 0 to regenerate │ │ ✓ audit round 1 5 flagged → 3 confirmed · 3 fixed │ │ ✓ audit round 2 clean │ │ ✓ audit round 3 clean │ │ ◆ corpus frozen 240 rows · 1e70… │ │ ! i-0003 Auto-enrollment claims in multi-intent replies │ │ ✓ fixed b006r027 logged i-0003 │ │ ✓ similar-row sweep 3 found · 3 fixed │ │ ✓ corpus audit complete 3 rounds · 3 rows fixed · 00:23 │ ├────────────────────────────────────────────────────────────────────────────────────────────────┤ │ ✓ corpus audit [████████████████] done │ └────────────────────────────────────────────────────────────────────────────────────────────────┘ ┌─ 6/7 export ───────────────────────────────────────────────────────────────────────────────────┐ │ ◆ production gate · pre export 3 checks passed │ │ ✓ deliverables/dataset-card.md dataset card │ │ ╭────────────────────────────────────────────────────────────────────────────────────────────╮ │ │ │ deliverables · 3 files · chatml template │ │ │ ├─────────────────────────────────────────────────────────────────────────────────────┬──────┤ │ │ │ file │ rows │ │ │ ├─────────────────────────────────────────────────────────────────────────────────────┼──────┤ │ │ │ deliverables/train.jsonl │ 192 │ │ │ │ deliverables/val.jsonl │ 24 │ │ │ │ deliverables/test.jsonl │ 24 │ │ │ ├─────────────────────────────────────────────────────────────────────────────────────┴──────┤ │ │ │ ✓ probe leakage check — no probes in the training set │ │ │ ╰────────────────────────────────────────────────────────────────────────────────────────────╯ │ │ ✓ export complete 3 files · leakage checked · 00:03 │ ├────────────────────────────────────────────────────────────────────────────────────────────────┤ │ ✓ export [████████████████] done │ └────────────────────────────────────────────────────────────────────────────────────────────────┘ ┌─ 7/7 runbook ──────────────────────────────────────────────────────────────────────────────────┐ │ ✓ deliverables/RUNBOOK.md start with the smoke train │ ├────────────────────────────────────────────────────────────────────────────────────────────────┤ │ ✓ runbook [████████████████] done │ └────────────────────────────────────────────────────────────────────────────────────────────────┘ ╭────────────────────────────────────────────────────────────────────────────────────────────────╮ │ ◆ dataset ready — Acme Suppliers Customer Support Model │ ├────────┬───────────────────────────────────────────────────────────────────────────────────────┤ │ rows │ 240 accepted · chatml template │ │ splits │ train 192 · val 24 · test 24 │ │ audit │ 3 rounds · 3 rows fixed · clean │ │ spend │ $5.62 of $6.00 cap │ │ time │ 03:01 │ ├────────┴───────────────────────────────────────────────────────────────────────────────────────┤ │ → deliverables/RUNBOOK.md · smoke train first │ │ → ft view · every row, and why it passed │ ╰────────────────────────────────────────────────────────────────────────────────────────────────╯