Package: retry

import "../ibm-cos-sdk-go-v2/aws/retry"

Overview

Package retry provides interfaces and implementations for SDK request retry behavior.

Retryer Interface and Implementations

This package defines Retryer interface that is used to either implement custom retry behavior or to extend the existing retry implementations provided by the SDK. This package provides a single retry implementation: Standard.

Standard

Standard is the default retryer implementation used by service clients. The standard retryer is a rate limited retryer that has a configurable max attempts to limit the number of retry attempts when a retryable error occurs. In addition, the retryer uses a configurable token bucket to rate limit the retry attempts across the client, and uses an additional delay policy to limit the time between a requests subsequent attempts.

By default the standard retryer uses the DefaultRetryables slice of IsErrorRetryable types to determine whether a given error is retryable. By default this list of retryables includes the following: - Retrying errors that implement the RetryableError method, and return true. - Connection Errors - Errors that implement a ConnectionError, Temporary, or Timeout method that return true. - Connection Reset Errors. - net.OpErr types that are dialing errors or are temporary. - HTTP Status Codes: 500, 502, 503, and 504. - API Error Codes - RequestTimeout, RequestTimeoutException - Throttling, ThrottlingException, ThrottledException, RequestThrottledException, TooManyRequestsException, RequestThrottled, SlowDown, EC2ThrottledException - ProvisionedThroughputExceededException, RequestLimitExceeded, BandwidthLimitExceeded, LimitExceededException - TransactionInProgressException, PriorRequestNotComplete

The standard retryer will not retry a request in the event if the context associated with the request has been cancelled. Applications must handle this case explicitly if they wish to retry with a different context value.

You can configure the standard retryer implementation to fit your applications by constructing a standard retryer using the NewStandard function, and providing one more functional argument that mutate the StandardOptions structure. StandardOptions provides the ability to modify the token bucket rate limiter, retryable error conditions, and the retry delay policy.

For example to modify the default retry attempts for the standard retryer:

// configure the custom retryer
customRetry := retry.NewStandard(func(o *retry.StandardOptions) {
    o.MaxAttempts = 5
})

// create a service client with the retryer
s3.NewFromConfig(cfg, func(o *s3.Options) {
    o.Retryer = customRetry
})

Utilities

A number of package functions have been provided to easily wrap retryer implementations in an implementation agnostic way. These are:

AddWithErrorCodes      - Provides the ability to add additional API error codes that should be considered retryable
                        in addition to those considered retryable by the provided retryer.

AddWithMaxAttempts     - Provides the ability to set the max number of attempts for retrying a request by wrapping
                         a retryer implementation.

AddWithMaxBackoffDelay - Provides the ability to set the max back off delay that can occur before retrying a
                         request by wrapping a retryer implementation.

The following package functions have been provided to easily satisfy different retry interfaces to further customize a given retryer’s behavior:

BackoffDelayerFunc   - Can be used to wrap a function to satisfy the BackoffDelayer interface. For example,
                       you can use this method to easily create custom back off policies to be used with the
                       standard retryer.

IsErrorRetryableFunc - Can be used to wrap a function to satisfy the IsErrorRetryable interface. For example,
                       this can be used to extend the standard retryer to add additional logic to determine if an
                       error should be retried.

IsErrorTimeoutFunc   - Can be used to wrap a function to satisfy IsErrorTimeout interface. For example,
                       this can be used to extend the standard retryer to add additional logic to determine if an
                        error should be considered a timeout.

Sub-Packages

internal

Constants

const DefaultRequestCost uint = readonly

DefaultRequestCost is the cost of a single request from the adaptive rate limited token bucket.

Value:

1

const DefaultMaxAttempts int = readonly

DefaultMaxAttempts is the maximum of attempts for an API request

Value:

3

const DefaultMaxBackoff time.Duration = readonly

DefaultMaxBackoff is the maximum back off delay between attempts

Value:

20 * time.Second

const DefaultRetryRateTokens uint = readonly

Value:

500

const DefaultRetryCost uint = readonly

Value:

5

const DefaultRetryTimeoutCost uint = readonly

Value:

10

const DefaultNoRetryIncrement uint = readonly

Value:

1

Variables

var DefaultThrottles = writable

DefaultThrottles provides the set of errors considered throttle errors that are checked by default.

Value:

[]IsErrorThrottle{
	ThrottleErrorCode{
		Codes: DefaultThrottleErrorCodes,
	},
}

var DefaultRetryableHTTPStatusCodes = writable

DefaultRetryableHTTPStatusCodes is the default set of HTTP status codes the SDK should consider as retryable errors.

Value:

map[int]struct{}{
	500: {},
	502: {},
	503: {},
	504: {},
}

var DefaultRetryableErrorCodes = writable

DefaultRetryableErrorCodes provides the set of API error codes that should be retried.

Value:

map[string]struct{}{
	"RequestTimeout":          {},
	"RequestTimeoutException": {},
}

var DefaultThrottleErrorCodes = writable

DefaultThrottleErrorCodes provides the set of API error codes that are considered throttle errors.

Value:

map[string]struct{}{
	"Throttling":                             {},
	"ThrottlingException":                    {},
	"ThrottledException":                     {},
	"RequestThrottledException":              {},
	"TooManyRequestsException":               {},
	"ProvisionedThroughputExceededException": {},
	"TransactionInProgressException":         {},
	"RequestLimitExceeded":                   {},
	"BandwidthLimitExceeded":                 {},
	"LimitExceededException":                 {},
	"RequestThrottled":                       {},
	"SlowDown":                               {},
	"PriorRequestNotComplete":                {},
	"EC2ThrottledException":                  {},
}

var DefaultRetryables = writable

DefaultRetryables provides the set of retryable checks that are used by default.

Value:

[]IsErrorRetryable{
	NoRetryCanceledError{},
	RetryableError{},
	RetryableConnectionError{},
	RetryableHTTPStatusCode{
		Codes: DefaultRetryableHTTPStatusCodes,
	},
	RetryableErrorCode{
		Codes: DefaultRetryableErrorCodes,
	},
	RetryableErrorCode{
		Codes: DefaultThrottleErrorCodes,
	},
}

var DefaultTimeouts = writable

DefaultTimeouts provides the set of timeout checks that are used by default.

Value:

[]IsErrorTimeout{
	TimeouterError{},
}

Type Summary

• AdaptiveMode struct

  AdaptiveMode provides an experimental retry strategy that expands on the Standard retry strategy, adding client attempt rate limits.

• AdaptiveModeOptions struct

  AdaptiveModeOptions provides the functional options for configuring the adaptive retry mode, and delay behavior.

• AddRetryMiddlewaresOptions struct

  AddRetryMiddlewaresOptions is the set of options that can be passed to AddRetryMiddlewares for configuring retry associated middleware.

• Attempt struct

  Attempt is a Smithy Finalize middleware that handles retry attempts using the provided Retryer implementation.

• AttemptResult struct

  AttemptResult represents attempt result returned by a single request attempt.

• AttemptResults struct

  AttemptResults represents struct containing metadata returned by all request attempts.

• BackoffDelayerFunc struct

  BackoffDelay returns the delay before attempt to retry a request.

• ExponentialJitterBackoff struct

  ExponentialJitterBackoff provides backoff delays with jitter based on the number of attempts.

• IsErrorRetryableFunc struct

  IsErrorRetryable returns if the error is retryable.

• IsErrorRetryables struct

  IsErrorRetryable returns if the error is retryable if any of the checks in the list return a value other than unknown.

• IsErrorThrottleFunc struct

  IsErrorThrottle returns if the error is a throttle error.

• IsErrorThrottles struct

  IsErrorThrottle returns if the error is a throttle error if any of the checks in the list return a value other than unknown.

• IsErrorTimeoutFunc struct

  IsErrorTimeout returns if the error is retryable.

• IsErrorTimeouts struct

  IsErrorTimeout returns if the error is retryable if any of the checks in the list return a value other than unknown.

• MaxAttemptsError struct

  MaxAttemptsError provides the error when the maximum number of attempts have been exceeded.

• MetricsHeader struct

  ID returns the middleware identifier.

• RetryableConnectionError struct

  IsErrorRetryable returns if the error is caused by and HTTP connection error, and should be retried.

• RetryableErrorCode struct

  RetryableErrorCode determines if an attempt should be retried based on the API error code.

• RetryableHTTPStatusCode struct

  RetryableHTTPStatusCode provides a IsErrorRetryable based on HTTP status codes.

• Standard struct

  Standard is the standard retry pattern for the SDK.

• StandardOptions struct

  StandardOptions provides the functional options for configuring the standard retryable, and delay behavior.

• ThrottleErrorCode struct

  ThrottleErrorCode determines if an attempt should be retried based on the API error code.

• TimeouterError struct

  IsErrorTimeout returns if the error is a timeout error.

Interface Summary

• BackoffDelayer interface

  BackoffDelayer provides the interface for determining the delay to before another request attempt, that previously failed.

• IsErrorRetryable interface

  IsErrorRetryable provides the interface of an implementation to determine if a error as the result of an operation is retryable.

• IsErrorThrottle interface

  IsErrorThrottle provides the interface of an implementation to determine if a error response from an operation is a throttling error.

• IsErrorTimeout interface

  IsErrorTimeout provides the interface of an implementation to determine if a error matches.

• RateLimiter interface

  RateLimiter provides the interface for limiting the rate of attempt retries allowed by the retryer.

Function Summary

• func AddRetryMiddlewares(stack *smithymiddle.Stack, options AddRetryMiddlewaresOptions) error

  AddRetryMiddlewares adds retry middleware to operation middleware stack.

• func AddWithErrorCodes(r aws.Retryer, codes ...string) aws.Retryer

  AddWithErrorCodes returns a Retryer with additional error codes considered for determining if the error should be retried.

• func AddWithMaxAttempts(r aws.Retryer, max int) aws.Retryer

  AddWithMaxAttempts returns a Retryer with MaxAttempts set to the value specified.

• func AddWithMaxBackoffDelay(r aws.Retryer, delay time.Duration) aws.Retryer

  AddWithMaxBackoffDelay returns a retryer wrapping the passed in retryer overriding the RetryDelay behavior for a alternate minimum initial backoff delay.

• func GetAttemptResults(metadata middleware.Metadata) (AttemptResults, bool)

  GetAttemptResults retrieves attempts results from middleware metadata.

• func NewAdaptiveMode(optFns ...func(*AdaptiveModeOptions)) *AdaptiveMode

  NewAdaptiveMode returns an initialized AdaptiveMode retry strategy.

• func NewAttemptMiddleware(retryer aws.Retryer, requestCloner RequestCloner, optFns ...func(*Attempt)) *Attempt

  NewAttemptMiddleware returns a new Attempt retry middleware.

• func NewStandard(fnOpts ...func(*StandardOptions)) *Standard

  NewStandard initializes a standard retry behavior with defaults that can be overridden via functional options.

Type Details

AdaptiveModeOptions struct

AdaptiveModeOptions provides the functional options for configuring the adaptive retry mode, and delay behavior.

Structure Fields:

FailOnNoAttemptTokens bool

If the adaptive token bucket is empty, when an attempt will be made AdaptiveMode will sleep until a token is available. This can occur when attempts fail with throttle errors. Use this option to disable the sleep until token is available, and return error immediately.

RequestCost uint

The cost of an attempt from the AdaptiveMode’s adaptive token bucket.

Throttles []IsErrorThrottle

Set of strategies to determine if the attempt failed due to a throttle error.

It is safe to append to this list in NewAdaptiveMode’s functional options.

StandardOptions []func(*StandardOptions)

Set of options for standard retry mode that AdaptiveMode is built on top of. AdaptiveMode may apply its own defaults to Standard retry mode that are different than the defaults of NewStandard. Use these options to override the default options.

AddRetryMiddlewaresOptions struct

AddRetryMiddlewaresOptions is the set of options that can be passed to AddRetryMiddlewares for configuring retry associated middleware.

Structure Fields:

Retryer aws.Retryer

LogRetryAttempts bool

Enable the logging of retry attempts performed by the SDK. This will include logging retry attempts, unretryable errors, and when max attempts are reached.

AttemptResults struct

AttemptResults represents struct containing metadata returned by all request attempts.

Structure Fields:

Results []AttemptResult

Results is a slice consisting attempt result from all request attempts. Results are stored in order request attempt is made.

StandardOptions struct

StandardOptions provides the functional options for configuring the standard retryable, and delay behavior.

Structure Fields:

MaxAttempts int

Maximum number of attempts that should be made.

MaxBackoff time.Duration

MaxBackoff duration between retried attempts.

Backoff BackoffDelayer

Provides the backoff strategy the retryer will use to determine the delay between retry attempts.

Retryables []IsErrorRetryable

Set of strategies to determine if the attempt should be retried based on the error response received.

It is safe to append to this list in NewStandard’s functional options.

Timeouts []IsErrorTimeout

Set of strategies to determine if the attempt failed due to a timeout error.

It is safe to append to this list in NewStandard’s functional options.

RateLimiter RateLimiter

Provides the rate limiting strategy for rate limiting attempt retries across all attempts the retryer is being used with.

A RateLimiter operates as a token bucket with a set capacity, where attempt failures events consume tokens. A retry attempt that attempts to consume more tokens than what’s available results in operation failure. The default implementation is parameterized as follows: - a capacity of 500 (DefaultRetryRateTokens) - a retry caused by a timeout costs 10 tokens (DefaultRetryCost) - a retry caused by other errors costs 5 tokens (DefaultRetryTimeoutCost) - an operation that succeeds on the 1st attempt adds 1 token (DefaultNoRetryIncrement)

You can disable rate limiting by setting this field to ratelimit.None.

RetryCost uint

The cost to deduct from the RateLimiter’s token bucket per retry.

RetryTimeoutCost uint

The cost to deduct from the RateLimiter’s token bucket per retry caused by timeout error.

NoRetryIncrement uint

The cost to payback to the RateLimiter’s token bucket for successful attempts.

Function Details

func AddRetryMiddlewares(stack *smithymiddle.Stack, options AddRetryMiddlewaresOptions) error

AddRetryMiddlewares adds retry middleware to operation middleware stack

// File 'aws/retry/middleware.go', line 393

func AddRetryMiddlewares(stack *smithymiddle.Stack, options AddRetryMiddlewaresOptions) error {
	attempt := NewAttemptMiddleware(options.Retryer, http.RequestCloner, func(middleware *Attempt) {
		middleware.LogAttempts = options.LogRetryAttempts
	})

	// index retry to before signing, if signing exists
	if err := stack.Finalize.Insert(attempt, "Signing", smithymiddle.Before); err != nil {
		return err
	}

	if err := stack.Finalize.Insert(&MetricsHeader{}, attempt.ID(), smithymiddle.After); err != nil {
		return err
	}
	return nil
}

func AddWithErrorCodes(r aws.Retryer, codes ...string) aws.Retryer

AddWithErrorCodes returns a Retryer with additional error codes considered for determining if the error should be retried.

Examples:

// Wrap a standard retyer and add the types.NoSuchBucketException Amazon S3 error code as retryable
custom := retry.AddWithErrorCodes(retry.NewStandard(), (*types.NoSuchBucketException)(nil).ErrorCode())

fmt.Println(custom.IsErrorRetryable(&types.NoSuchBucketException{}))

// File 'aws/retry/retry.go', line 11

func AddWithErrorCodes(r aws.Retryer, codes ...string) aws.Retryer {
	retryable := &RetryableErrorCode{
		Codes: map[string]struct{}{},
	}
	for _, c := range codes {
		retryable.Codes[c] = struct{}{}
	}

	return &withIsErrorRetryable{
		RetryerV2: wrapAsRetryerV2(r),
		Retryable: retryable,
	}
}

func AddWithMaxAttempts(r aws.Retryer, max int) aws.Retryer

AddWithMaxAttempts returns a Retryer with MaxAttempts set to the value specified.

Examples:

// Wrap a standard retyer and set the max attempts to 5
custom := retry.AddWithMaxAttempts(retry.NewStandard(), 5)

fmt.Println(custom.MaxAttempts())

// File 'aws/retry/retry.go', line 39

func AddWithMaxAttempts(r aws.Retryer, max int) aws.Retryer {
	return &withMaxAttempts{
		RetryerV2: wrapAsRetryerV2(r),
		Max:       max,
	}
}

func AddWithMaxBackoffDelay(r aws.Retryer, delay time.Duration) aws.Retryer

AddWithMaxBackoffDelay returns a retryer wrapping the passed in retryer overriding the RetryDelay behavior for a alternate minimum initial backoff delay.

Examples:

// Wrap a standard retyer and add tthe NoSuchBucket API error code to the list of retryables
custom := retry.AddWithMaxBackoffDelay(retry.NewStandard(), time.Second*5)
_ = custom

// File 'aws/retry/retry.go', line 58

func AddWithMaxBackoffDelay(r aws.Retryer, delay time.Duration) aws.Retryer {
	return &withMaxBackoffDelay{
		RetryerV2: wrapAsRetryerV2(r),
		backoff:   NewExponentialJitterBackoff(delay),
	}
}

func GetAttemptResults(metadata middleware.Metadata) (AttemptResults, bool)

GetAttemptResults retrieves attempts results from middleware metadata.

// File 'aws/retry/metadata.go', line 13

func GetAttemptResults(metadata middleware.Metadata) (AttemptResults, bool) {
	m, ok := metadata.Get(attemptResultsKey{}).(AttemptResults)
	return m, ok
}

func NewAdaptiveMode(optFns ...func(*AdaptiveModeOptions)) *AdaptiveMode

NewAdaptiveMode returns an initialized AdaptiveMode retry strategy.

// File 'aws/retry/adaptive.go', line 70

func NewAdaptiveMode(optFns ...func(*AdaptiveModeOptions)) *AdaptiveMode {
	o := AdaptiveModeOptions{
		RequestCost: DefaultRequestCost,
		Throttles:   append([]IsErrorThrottle{}, DefaultThrottles...),
	}
	for _, fn := range optFns {
		fn(&o)
	}

	return &AdaptiveMode{
		options:   o,
		throttles: IsErrorThrottles(o.Throttles),
		retryer:   NewStandard(o.StandardOptions...),
		rateLimit: newAdaptiveRateLimit(),
	}
}

func NewAttemptMiddleware(retryer aws.Retryer, requestCloner RequestCloner, optFns ...func(*Attempt)) *Attempt

NewAttemptMiddleware returns a new Attempt retry middleware.

// File 'aws/retry/middleware.go', line 54

func NewAttemptMiddleware(retryer aws.Retryer, requestCloner RequestCloner, optFns ...func(*Attempt)) *Attempt {
	m := &Attempt{
		retryer:       wrapAsRetryerV2(retryer),
		requestCloner: requestCloner,
	}
	for _, fn := range optFns {
		fn(m)
	}
	if m.OperationMeter == nil {
		m.OperationMeter = metrics.NopMeterProvider{}.Meter("")
	}

	return m
}

func NewStandard(fnOpts ...func(*StandardOptions)) *Standard

NewStandard initializes a standard retry behavior with defaults that can be overridden via functional options.

// File 'aws/retry/standard.go', line 170

func NewStandard(fnOpts ...func(*StandardOptions)) *Standard {
	o := StandardOptions{
		MaxAttempts: DefaultMaxAttempts,
		MaxBackoff:  DefaultMaxBackoff,
		Retryables:  append([]IsErrorRetryable{}, DefaultRetryables...),
		Timeouts:    append([]IsErrorTimeout{}, DefaultTimeouts...),

		RateLimiter:      ratelimit.NewTokenRateLimit(DefaultRetryRateTokens),
		RetryCost:        DefaultRetryCost,
		RetryTimeoutCost: DefaultRetryTimeoutCost,
		NoRetryIncrement: DefaultNoRetryIncrement,
	}
	for _, fn := range fnOpts {
		fn(&o)
	}
	if o.MaxAttempts <= 0 {
		o.MaxAttempts = DefaultMaxAttempts
	}

	backoff := o.Backoff
	if backoff == nil {
		backoff = NewExponentialJitterBackoff(o.MaxBackoff)
	}

	return &Standard{
		options:   o,
		backoff:   backoff,
		retryable: IsErrorRetryables(o.Retryables),
		timeout:   IsErrorTimeouts(o.Timeouts),
	}
}