Unified training framework for OpenClaw on Tinker cloud infrastructure. Supports three training methods through a single entry point:

export TINKER_API_KEY="your-tinker-api-key"

# Combined method
python run.py --method combine --model-name Qwen/Qwen3-8B --prm-m 1 --batch-size 16 --w-opd 1.0 --w-rl 1.0 --train-epochs 2

# RL method 
python run.py --method rl --model-name Qwen/Qwen3-8B --prm-m 3 --batch-size 16

# OPD method
python run.py --method opd --model-name Qwen/Qwen3-8B --prm-m 1 --batch-size 16

run.py                 CLI entry point (--method {rl, opd, combine})
├── config.py          Unified TinkerConfig dataclass
├── trainer.py         Training loop: rollout → score → build datums → forward_backward → optim_step
│   ├── rollout.py     RolloutWorker: launches API proxy, feeds prompts, collects completions
│   │   └── api_server.py   OpenAI-compatible proxy with method-specific subclasses
│   ├── scorers.py     PRMScorer / OPDScorer / CombinedScorer
│   └── data_formatter.py   TrainingSample → Tinker Datum conversion

• Trainer (trainer.py): Orchestrates the full loop. Creates two Tinker clients — a LoRA training client (policy model) and a base sampling client (teacher/judge). Handles checkpoint saving at configurable intervals and graceful shutdown.

• RolloutWorker (rollout.py): Spins up a local OpenAI-compatible API server that forwards requests to the Tinker policy model. External environments (OpenClaw tasks) connect to this server. Completed sessions are queued for scoring and training.

• API Server (api_server.py): Base class _BaseServer provides shared infrastructure (Tinker forwarding, auth, streaming, tokenization, record management). Three subclasses handle method-specific logic:

  • OpenClawRLServer — PRM scoring with at-least-one guarantee
  • OpenClawOPDServer — Hint judge + teacher log-probs, drops turns without next_state
  • OpenClawCombineServer — Three-way dispatch (opd+rl / opd-only / rl-only)

• Scorers (scorers.py): Each scorer evaluates completed sessions and produces TrainingSample objects with rewards and optional teacher log-probs.

• Data Formatter (data_formatter.py): Converts TrainingSample batches into Tinker Datum objects for training. RL/OPD use scalar GRPO advantages; Combined computes per-token w_opd * teacher_adv + w_rl * reward.

All parameters can be set via CLI flags or environment variables:

Standard GRPO reinforcement learning with Process Reward Model scoring:

1. Policy model generates responses via the API proxy
2. PRM evaluates each turn by scoring the next state (majority vote over M samples)
3. Rewards: +1 (correct), -1 (incorrect), 0 (uncertain)
4. At-least-one guarantee: If all turns in a session score ≤ 0, the best turn gets reward = +1
5. GRPO advantages (scalar reward broadcast) → Tinker Datum → training step

On-Policy Distillation using hindsight hints and teacher knowledge:

1. Policy model generates responses; environment provides next_state observations
2. Hint judge extracts key information from next_state into a concise hint
3. Teacher model scores the response (with hint context) to get token-level log-probs
4. Advantage = per-token distillation: teacher_lp - student_lp
5. All samples get reward = 1.0 (no explicit reward signal)
6. Optional --eval-mode: also compute PRM eval scores for monitoring

Weighted combination with three-way sample dispatch:

• OPD+RL samples (have both next_state and reward): get both advantage components
• OPD-only samples (next_state but no reward): only teacher distillation advantage
• RL-only samples (reward but no next_state): only scalar reward advantage

Combined advantage per token:

combined_adv_i = w_opd * (teacher_lp_i - student_lp_i) + w_rl * reward

This project uses the Tinker cloud platform for:

• LoRA Training: create_lora_training_client_async(base_model=..., rank=...) for the policy model
• Sampling: create_sampling_client_async(base_model=...) for the teacher/judge model
• Training ops: forward_backward_async() + optim_step_async() per step
• Checkpointing: save_weights_and_get_sampling_client_async() to update the policy sampling client
• Loss functions: Supports ppo, importance_sampling, cispo