Skip to content

Heuristic policy cookbook

Patterns for the opposing side — the side that runs as a heuristic policy rather than a controlled (learned or planned) policy. Each pattern is a recipe; the gallery ships canonical implementations of all four.

In single-agent framing, the opposite side runs cfg.guard_policy or cfg.bandit_policy (depending on which side cfg.controlled_side points to). Multi-agent and POMDP framings ignore the opposite side's field — both sides are driven by the caller.

Pattern: lead-intercept chaser

Predict the opponent's next-step position under a free-drift assumption, then thrust along the line to that predicted point. T3 walks through this in tutorial form; the gallery class is LeadInterceptPursuer.

    import dataclasses

    from orbitalgym import make_pursuit_evasion
    from orbitalgym.policies.heuristic import (
        JitteredPolicy,
        LeadInterceptPursuer,
        OrthogonalEvader,
    )

    cfg = make_pursuit_evasion(seed=0, max_horizon_s=200.0)
    cfg = dataclasses.replace(cfg, bandit_policy=LeadInterceptPursuer(max_dv_mps=0.05, dt=cfg.dt))

Gallery: LeadInterceptPursuer.

Pattern: evasive maneuver

Move perpendicular to the opponent's relative velocity. With near-zero relative velocity the orthogonal subspace is degenerate; the canonical implementation falls back to a fixed reference axis to keep the action finite.

    cfg3 = make_pursuit_evasion(seed=0, max_horizon_s=200.0)
    cfg3 = dataclasses.replace(cfg3, bandit_policy=OrthogonalEvader(max_dv_mps=0.05))

Gallery: OrthogonalEvader.

Pattern: sun-tracker

Maintain Sun-line geometry between the controlled vehicle and the opponent. Useful for Sun-Blocking. The simple version takes the Sun direction in the local RTN frame as a static knob; a dynamic Sun version (re-derived per step from epoch + time) is out of scope for the gallery — see games/sun_blocking.py:_sun_eci_at_mjd for the dynamic approach.

Gallery: SunTrackerBlocker.

Pattern: randomized heuristic

Wrap any deterministic policy with PRNG-keyed Gaussian jitter. Great for population-of-opponents training where you want a smooth distribution of adversaries.

    base = LeadInterceptPursuer(max_dv_mps=0.05, dt=cfg.dt)
    cfg2 = dataclasses.replace(cfg, bandit_policy=JitteredPolicy(base=base, sigma=0.005))

Gallery: JitteredPolicy.

Pitfalls

  • Why the n_vehicles=0 and command_cls=None defaults? The env populates these via dataclasses.replace at construction; hard-coding them defeats portability across scenarios.
  • Why must side identity be static? Side is a Python StrEnum, never a traced JAX array. JAX specialises the trace per-side via the static identity check. See Symmetric core.
  • The __call__ signature is rigid. (self, policy_state, obs, key, t) → (action, next_policy_state). Stateless? Pass None and forward it unchanged.
  • JitteredPolicy requires its base keyword. No default — pass JitteredPolicy(base=..., sigma=...).

See also