Struct: s3manager.Downloader

import "../ibm-cos-sdk-go/service/s3/s3manager"

Overview

The Downloader structure that calls Download(). It is safe to call Download() on this structure for multiple objects and across concurrent goroutines. Mutating the Downloader's properties is not safe to be done concurrently.

Implemented Interfaces

s3crypto.Cipher, s3manageriface.DownloadWithIterator, s3manageriface.DownloaderAPI, s3manager.ReadSeekerWriteTo, s3manager.WriterReadFrom

Structure Field Summary collapse

BufferProvider WriterReadFromProvider

Defines the buffer strategy used when downloading a part.
Concurrency int

The number of goroutines to spin up in parallel when sending parts.
PartSize int64

The size (in bytes) to request from S3 for each part.
RequestOptions []request.Option

List of request options that will be passed down to individual API operation requests made by the downloader.
S3 s3iface.S3API

An S3 client to use when performing downloads.

Method Summary collapse

func (d Downloader) Download(w io.WriterAt, input *s3.GetObjectInput, options ...func(*Downloader)) (n int64, err error)

Download downloads an object in S3 and writes the payload into w using concurrent GET requests.
func (d Downloader) DownloadWithContext(ctx aws.Context, w io.WriterAt, input *s3.GetObjectInput, options ...func(*Downloader)) (n int64, err error)

DownloadWithContext downloads an object in S3 and writes the payload into w using concurrent GET requests.
func (d Downloader) DownloadWithIterator(ctx aws.Context, iter BatchDownloadIterator, opts ...func(*Downloader)) error

DownloadWithIterator will download a batched amount of objects in S3 and writes them to the io.WriterAt specificed in the iterator.

Structure Field Details

BufferProvider WriterReadFromProvider

Defines the buffer strategy used when downloading a part.

If a WriterReadFromProvider is given the Download manager will pass the io.WriterAt of the Download request to the provider and will use the returned WriterReadFrom from the provider as the destination writer when copying from http response body.

Concurrency int

The number of goroutines to spin up in parallel when sending parts. If this is set to zero, the DefaultDownloadConcurrency value will be used.

Concurrency of 1 will download the parts sequentially.

Concurrency is ignored if the Range input parameter is provided.

PartSize int64

The size (in bytes) to request from S3 for each part. The minimum allowed part size is 5MB, and if this value is set to zero, the DefaultDownloadPartSize value will be used.

PartSize is ignored if the Range input parameter is provided.

RequestOptions []request.Option

List of request options that will be passed down to individual API operation requests made by the downloader.

S3 s3iface.S3API

An S3 client to use when performing downloads.

Method Details

func (d Downloader) Download(w io.WriterAt, input s3.GetObjectInput, options ...func(Downloader)) (n int64, err error)

Download downloads an object in S3 and writes the payload into w using concurrent GET requests. The n int64 returned is the size of the object downloaded in bytes.

It is safe to call this method concurrently across goroutines.

The w io.WriterAt can be satisfied by an os.File to do multipart concurrent downloads, or in memory []byte wrapper using aws.WriteAtBuffer.

Specifying a Downloader.Concurrency of 1 will cause the Downloader to download the parts from S3 sequentially.

If the GetObjectInput's Range value is provided that will cause the downloader to perform a single GetObjectInput request for that object's range. This will caused the part size, and concurrency configurations to be ignored.



164
165
166

// File 'service/s3/s3manager/download.go', line 164

func (d Downloader) Download(w io.WriterAt, input *s3.GetObjectInput, options ...func(*Downloader)) (n int64, err error) {
	return d.DownloadWithContext(aws.BackgroundContext(), w, input, options...)
}

func (d Downloader) DownloadWithContext(ctx aws.Context, w io.WriterAt, input s3.GetObjectInput, options ...func(Downloader)) (n int64, err error)

DownloadWithContext downloads an object in S3 and writes the payload into w using concurrent GET requests. The n int64 returned is the size of the object downloaded in bytes.

DownloadWithContext is the same as Download with the additional support for Context input parameters. The Context must not be nil. A nil Context will cause a panic. Use the Context to add deadlining, timeouts, etc. The DownloadWithContext may create sub-contexts for individual underlying requests.

Additional functional options can be provided to configure the individual download. These options are copies of the Downloader instance Download is called from. Modifying the options will not impact the original Downloader instance. Use the WithDownloaderRequestOptions helper function to pass in request options that will be applied to all API operations made with this downloader.

The w io.WriterAt can be satisfied by an os.File to do multipart concurrent downloads, or in memory []byte wrapper using aws.WriteAtBuffer.

Specifying a Downloader.Concurrency of 1 will cause the Downloader to download the parts from S3 sequentially.

It is safe to call this method concurrently across goroutines.

// File 'service/s3/s3manager/download.go', line 195

func (d Downloader) DownloadWithContext(ctx aws.Context, w io.WriterAt, input *s3.GetObjectInput, options ...func(*Downloader)) (n int64, err error) {
	if err := validateSupportedARNType(aws.StringValue(input.Bucket)); err != nil {
		return 0, err
	}

	impl := downloader{w: w, in: input, cfg: d, ctx: ctx}

	for _, option := range options {
		option(&impl.cfg)
	}
	impl.cfg.RequestOptions = append(impl.cfg.RequestOptions, request.WithAppendUserAgent("S3Manager"))

	if s, ok := d.S3.(maxRetrier); ok {
		impl.partBodyMaxRetries = s.MaxRetries()
	}

	impl.totalBytes = -1
	if impl.cfg.Concurrency == 0 {
		impl.cfg.Concurrency = DefaultDownloadConcurrency
	}

	if impl.cfg.PartSize == 0 {
		impl.cfg.PartSize = DefaultDownloadPartSize
	}

	return impl.download()
}

func (d Downloader) DownloadWithIterator(ctx aws.Context, iter BatchDownloadIterator, opts ...func(*Downloader)) error

DownloadWithIterator will download a batched amount of objects in S3 and writes them to the io.WriterAt specificed in the iterator.

Example:

svc := s3manager.NewDownloader(session)

fooFile, err := os.Open("/tmp/foo.file")
if err != nil {
	return err
}

barFile, err := os.Open("/tmp/bar.file")
if err != nil {
	return err
}

objects := []s3manager.BatchDownloadObject {
	{
		Object: &s3.GetObjectInput {
			Bucket: aws.String("bucket"),
			Key: aws.String("foo"),
		},
		Writer: fooFile,
	},
	{
		Object: &s3.GetObjectInput {
			Bucket: aws.String("bucket"),
			Key: aws.String("bar"),
		},
		Writer: barFile,
	},
}

iter := &s3manager.DownloadObjectsIterator{Objects: objects}
if err := svc.DownloadWithIterator(aws.BackgroundContext(), iter); err != nil {
	return err
}

// File 'service/s3/s3manager/download.go', line 261

func (d Downloader) DownloadWithIterator(ctx aws.Context, iter BatchDownloadIterator, opts ...func(*Downloader)) error {
	var errs []Error
	for iter.Next() {
		object := iter.DownloadObject()
		if _, err := d.DownloadWithContext(ctx, object.Writer, object.Object, opts...); err != nil {
			errs = append(errs, newError(err, object.Object.Bucket, object.Object.Key))
		}

		if object.After == nil {
			continue
		}

		if err := object.After(); err != nil {
			errs = append(errs, newError(err, object.Object.Bucket, object.Object.Key))
		}
	}

	if len(errs) > 0 {
		return NewBatchError("BatchedDownloadIncomplete", "some objects have failed to download.", errs)
	}
	return nil
}