pg.early_stopping.early_stop_by_rank

Accessible via pg.early_stopping.early_stop_by_rank.

class early_stop_by_rank(step_ranks, metric='reward', maximize=True)

Bases: pg.Functor

Step-wise early stopping policy based on the rank of reward/metric.

Example:

policy = early_stop_by_rank([
  # Stop at step 1 if accuracy is less than top 80% previous trials at
  # this step, enabled when there are at least 5 previous trials reported
  # at this step.
  (1, 0.8, 5),

  # Stop at step 2 if accuracy is less than top 20% previous trials at
  # this step, enabled when there are at least 10 previous trials reported
  # at this step.
  (2, 0.2, 10),

  # Stop at step 3 if accuracy is less than the 3rd best trial at this step,
  # enabled when there are at least 3 previous trials reported at this step.
  (3, 3, 3)
], metric='accuracy')()
Parameters:

  • step_ranks –

    A list of tuple (gating step, rank threshold, trigger histogram size).

    gating step - At which step this rule will be triggered. rank threshold - A float number in range (0, 1) indicating the rank

    percentage or an integer (> 0) indicating the absolute rank as the threshold for early stopping.

    trigger historgram size - The minimal number of historical trials

    repoted at current step for this rule to trigger.

  • metric – Based on which metric the rank will be computed. Use str for metric name or a callable object that takes a measurement object at a given step as input and returns a float value.

  • maximize – If True, reward or metric value below the threshold will be stopped, otherwise trials with values above the threshold will be stopped.

Returns:

A StepWise early stopping policy.