R/hyperdrive.R
median_stopping_policy.RdMedian stopping is an early termination policy based on running averages of primary metrics reported by the runs. This policy computes running averages across all training runs and terminates runs whose performance is worse than the median of the running averages. Specifically, a run will be canceled at interval N if its best primary metric reported up to interval N is worse than the median of the running averages for intervals 1:N across all runs.
median_stopping_policy(evaluation_interval = 1L, delay_evaluation = 0L)
| evaluation_interval | An integer of the frequency for applying policy. |
|---|---|
| delay_evaluation | An integer of the number of intervals for which to delay the first evaluation. |
The MedianStoppingPolicy object.
The median stopping policy takes the following optional configuration parameters:
evaluation_interval: Optional. The frequency for applying the policy.
Each time the training script logs the primary metric counts as one
interval.
delay_evaluation: Optional. The number of intervals to delay the
policy evaluation. Use this parameter to avoid premature termination
of training runs. If specified, the policy applies every multiple of
evaluation_interval that is greater than or equal to delay_evaluation.
This policy is inspired from the research publication Google Vizier: A Service for Black-Box Optimization.
If you are looking for a conservative policy that provides savings without
terminating promising jobs, you can use a MedianStoppingPolicy with
evaluation_interval = 1 and delay_evaluation = 5. These are conservative
settings that can provide approximately 25%-35% savings with no loss on
the primary metric (based on our evaluation data).
# In this example, the early termination policy is applied at every # interval starting at evaluation interval 5. A run will be terminated at # interval 5 if its best primary metric is worse than the median of the # running averages over intervals 1:5 across all training runs if (FALSE) { early_termination_policy = median_stopping_policy(evaluation_interval = 1L, delay_evaluation = 5L) }