Adaptive Stress Testing for the POMDPs.jl ecosystem.

Provides implementation of Adaptive Stress Testing (AST) formulation of MDPs/POMDPs.

Provides virtual interface for the gray-box environment and simulator.

Provides virtual interface for the black-box system under test (SUT).

GrayBox.Simulation

Abstract base type for a gray-box simulation.

GrayBox.Sample

Holds sampled value and log-probability.

GrayBox.Environment

Alias type for a dictionary of gray-box environment distributions.

 e.g., `Environment(:variable_name => Sampleable)`

GrayBox.EnvironmentSample

Alias type for a single environment sample.

e.g., `EnvironmentSample(:variable_name => Sample(value, logprob))`

environment(sim::GrayBox.Simulation)

Return all distributions used in the simulation environment.

transition!(sim::Union{GrayBox.Simulation, GrayBox.Simulation})::Real

Given an input sample::EnvironmentSample, apply transition and return the transition log-probability.

transition!(sim::GrayBox.Simulation)::Real

Apply a transition step, and return the transition log-probability (used with ASTSeedAction).

initialize!(sim::GrayBox.Simulation)

Reset state to its initial state.

evaluate!(sim::GrayBox.Simulation)::Tuple{logprob::Real, miss_distance::Real, isevent::Bool}
evaluate!(sim::GrayBox.Simulation, sample::GrayBox.EnvironmentSample)::Tuple{logprob::Real, miss_distance::Real, isevent::Bool}

Evaluate the SUT given some input seed and current state, returns logprob, miss_distance, and isevent indication. If the sample version is implemented, then ASTSampleAction will be used instead of ASTSeedAction.

distance(sim::GrayBox.Simulation)::Real

Return how close to an event a terminal state was (i.e. some measure of "miss distance" to the event of interest).

isevent(sim::GrayBox.Simulation)::Bool

Return a boolean indicating if the SUT reached a failure event of interest.

isterminal(sim::GrayBox.Simulation)::Bool

Return an indication that the simulation is in a terminal state.

search!(planner)

Search for failures given a planner. Implemented by each solver.

Initialize AST MDP state. Overridden from POMDPs.initialstate interface.

reward(mdp::ASTMDP, logprob::Real, isevent::Bool, isterminal::Bool, miss_distance::Real)::Float64

Reward function for the AST formulation. Defaults to:

R_E       if isterminal and isevent (1)
-d        if isterminal and !isevent (2)
log(p)    otherwise (3)

1) Terminates with event, collect reward_bonus (defaults to 0)
2) Terminates without event, collect negative miss distance
3) Each non-terminal step, accumulate reward correlated with the transition probability

For epsidic reward problems (i.e. rewards only at the end of an episode), set mdp.params.episodic_rewards to get:

(log(p) - d)*R_E    if isterminal and isevent (1)
log(p) - d          if isterminal and !isevent (2)
0                   otherwise (3)

1) Terminates with event, collect transition probability and miss distance with multiplicative reward bonus
2) Terminates without event, collect transitions probability and miss distance
3) Each non-terminal step, no intermediate reward (set `mdp.params.give_intermediate_reward` to use log transition probability)

Generate next state and reward for AST MDP (handles episodic reward problems). Overridden from POMDPs.gen interface.

Determine if AST MDP is in a terminal state. Overridden from POMDPs.isterminal interface.

AST problems are (generally) undiscounted to treat future reward equally. Overridden from POMDPs.discount interface.

Randomly select next action, independent of the state.

Randomly select next action, independent of the state. Overridden from POMDPs.action interface.

Used by the CommonRLInterface to interact with deep RL solvers.

Reset AST simulation to a given state; used by the MCTS DPWSolver as the reset_callback function.

AST.record(::ASTMDP, sym::Symbol, val)

Recard an ASTMetric specified by sym.

Record the best paths from termination leaf node.

Get k-th top path from the recorded top_paths.

Rollout simulation for MCTS; used by the MCTS DPWSolver as the estimate_value function. Custom rollout records action trace once the depth has been reached.

Rollout to only execute SUT at end (p accounts for probabilities generated outside the rollout)

User defined:

feed_gen Function to feed best action, replaces call to gen when feeding

feed_type Indicate when to feed best action. Either at the start of the rollout :start, or mid-rollout :mid

best_callback Callback function to record best miss distance or reward for later feeding during rollout

Play back a given action trace from the initialstate of the MDP.

Follow MCTS optimal path online calling action after each selected state.

Trace up the tree to get all ancestor states.

Trace up the tree to get all ancestor actions.

Trace up the tree and accumulate Q-values.

Trace up the tree to get all ancestor actions and summed Q-values.

Clear data stored in mdp.metrics.

AST.ASTParams

Adaptive Stress Testing specific simulation parameters.

Abstract type for the AST action variants.

AST.ASTSeedAction

Random seed AST action.

AST.ASTSampleAction

Random environment sample as the AST action.

AST.ASTState

State of the AST MDP.

AST.ASTMetrics

Debugging metrics.

AST.ASTMDP

Adaptive Stress Testing MDP problem formulation object.

print_metrics(planner)
print_metrics(mdp::ASTMDP)
print_metrics(metrics::ASTMetrics)

Print failure metrics including:

• iteration of first failure
• number of failures
• failure rate
• highest log-likelihood of failure

Visualize MCTS tree structure for AST MDP.

Stacked figure with metrics over episodes:

- Running miss distance mean
- Minimum miss distance
- Cumulative number of failure events

Stacked figure with distributions:

- Miss distance distribution
- Log-likelihood distribution