Package: s3manager
Overview
Package s3manager provides utilities to upload and download objects from S3 concurrently. Helpful for when working with large objects.
Sub-Packages
Constants
-
const DefaultBatchSize = readonly
DefaultBatchSize is the batch size we initialize when constructing a batch delete client. This value is used when calling DeleteObjects. This represents how many objects to delete per DeleteObjects call.
-
Value:
100 -
const ErrDeleteBatchFailCode = readonly
ErrDeleteBatchFailCode represents an error code which will be returned only when DeleteObjects.Errors has an error that does not contain a code.
-
Value:
"DeleteBatchError" -
const MaxUploadParts = readonly
MaxUploadParts is the maximum allowed number of parts in a multi-part upload on Amazon S3.
-
Value:
10000 -
const MinUploadPartSize int64 = readonly
MinUploadPartSize is the minimum allowed part size when uploading a part to Amazon S3.
-
Value:
1024 * 1024 * 5 -
const DefaultUploadPartSize = readonly
DefaultUploadPartSize is the default part size to buffer chunks of a payload into.
-
Value:
-
const DefaultUploadConcurrency = readonly
DefaultUploadConcurrency is the default number of goroutines to spin up when using Upload().
-
Value:
5 -
const DefaultDownloadPartSize = readonly
DefaultDownloadPartSize is the default range of bytes to get at a time when using Download().
-
Value:
1024 * 1024 * 5 -
const DefaultDownloadConcurrency = readonly
DefaultDownloadConcurrency is the default number of goroutines to spin up when using Download().
-
Value:
5
Type Summary collapse
-
BatchDelete
struct
BatchDelete will use the s3 package's service client to perform a batch delete.
-
BatchDeleteObject
struct
BatchDeleteObject is a wrapper object for calling the batch delete operation.
-
BatchDownloadObject
struct
BatchDownloadObject contains all necessary information to run a batch operation once.
-
BatchError
struct
BatchError will contain the key and bucket of the object that failed to either upload or download.
-
BatchUploadObject
struct
BatchUploadObject contains all necessary information to run a batch operation once.
-
BufferedReadSeeker
struct
BufferedReadSeeker is buffered io.ReadSeeker.
-
BufferedReadSeekerWriteTo
struct
BufferedReadSeekerWriteTo wraps a BufferedReadSeeker with an io.WriteAt implementation.
-
BufferedReadSeekerWriteToPool
struct
BufferedReadSeekerWriteToPool uses a sync.Pool to create and reuse []byte slices for buffering parts in memory.
-
DeleteListIterator
struct
DeleteListIterator is an alternative iterator for the BatchDelete client.
-
DeleteObjectsIterator
struct
DeleteObjectsIterator is an interface that uses the scanner pattern to iterate through a series of objects to be deleted.
-
DownloadObjectsIterator
struct
DownloadObjectsIterator implements the BatchDownloadIterator interface and allows for batched download of objects.
-
Downloader
struct
The Downloader structure that calls Download().
-
Error
struct
Error will contain the original error, bucket, and key of the operation that failed during batch operations.
- Errors struct
-
PooledBufferedReadFromProvider
struct
PooledBufferedReadFromProvider is a WriterReadFromProvider that uses a sync.Pool to manage allocation and reuse of *bufio.Writer structures.
-
UploadInput
struct
UploadInput provides the input parameters for uploading a stream or buffer to an object in an Amazon S3 bucket.
-
UploadObjectsIterator
struct
UploadObjectsIterator implements the BatchUploadIterator interface and allows for batched upload of objects.
-
UploadOutput
struct
UploadOutput represents a response from the Upload() call.
-
Uploader
struct
The Uploader structure that calls Upload().
Interface Summary collapse
-
BatchDeleteIterator
interface
BatchDeleteIterator is an interface that uses the scanner pattern to iterate through what needs to be deleted.
-
BatchDownloadIterator
interface
BatchDownloadIterator is an interface that uses the scanner pattern to iterate through a series of objects to be downloaded.
-
BatchUploadIterator
interface
BatchUploadIterator is an interface that uses the scanner pattern to iterate through what needs to be uploaded.
-
MultiUploadFailure
interface
A MultiUploadFailure wraps a failed S3 multipart upload.
-
ReadSeekerWriteTo
interface
ReadSeekerWriteTo defines an interface implementing io.WriteTo and io.ReadSeeker.
-
ReadSeekerWriteToProvider
interface
ReadSeekerWriteToProvider provides an implementation of io.WriteTo for an io.ReadSeeker.
-
WriterReadFrom
interface
WriterReadFrom defines an interface implementing io.Writer and io.ReaderFrom.
-
WriterReadFromProvider
interface
WriterReadFromProvider provides an implementation of io.ReadFrom for the given io.Writer.
Function Summary collapse
-
func GetBucketRegion(ctx aws.Context, c client.ConfigProvider, bucket, regionHint string, opts ...request.Option) (string, error)
GetBucketRegion will attempt to get the region for a bucket using the regionHint to determine which AWS partition to perform the query on.
-
func GetBucketRegionWithClient(ctx aws.Context, svc s3iface.S3API, bucket string, opts ...request.Option) (string, error)
GetBucketRegionWithClient is the same as GetBucketRegion with the exception that it takes a S3 service client instead of a Session.
-
func NewBatchDelete(c client.ConfigProvider, options ...func(*BatchDelete)) *BatchDelete
NewBatchDelete will return a new delete client that can delete a batched amount of objects.
-
func NewBatchDeleteWithClient(client s3iface.S3API, options ...func(*BatchDelete)) *BatchDelete
NewBatchDeleteWithClient will return a new delete client that can delete a batched amount of objects.
-
func NewBatchError(code, message string, err []Error) awserr.Error
NewBatchError will return a BatchError that satisfies the awserr.Error interface.
-
func NewDeleteListIterator(svc s3iface.S3API, input *s3.ListObjectsInput, opts ...func(*DeleteListIterator)) BatchDeleteIterator
NewDeleteListIterator will return a new DeleteListIterator.
-
func NewDownloader(c client.ConfigProvider, options ...func(*Downloader)) *Downloader
NewDownloader creates a new Downloader instance to downloads objects from S3 in concurrent chunks.
-
func NewDownloaderWithClient(svc s3iface.S3API, options ...func(*Downloader)) *Downloader
NewDownloaderWithClient creates a new Downloader instance to downloads objects from S3 in concurrent chunks.
-
func NewUploader(c client.ConfigProvider, options ...func(*Uploader)) *Uploader
NewUploader creates a new Uploader instance to upload objects to S3.
-
func NewUploaderWithClient(svc s3iface.S3API, options ...func(*Uploader)) *Uploader
NewUploaderWithClient creates a new Uploader instance to upload objects to S3.
-
func WithDownloaderRequestOptions(opts ...request.Option) func(*Downloader)
WithDownloaderRequestOptions appends to the Downloader's API request options.
-
func WithUploaderRequestOptions(opts ...request.Option) func(*Uploader)
WithUploaderRequestOptions appends to the Uploader's API request options.
Type Details
BatchDeleteObject struct
BatchDeleteObject is a wrapper object for calling the batch delete operation.
BatchDownloadObject struct
BatchDownloadObject contains all necessary information to run a batch operation once.
BatchUploadObject struct
BatchUploadObject contains all necessary information to run a batch operation once.
UploadInput struct
UploadInput provides the input parameters for uploading a stream or buffer to an object in an Amazon S3 bucket. This type is similar to the s3 package's PutObjectInput with the exception that the Body member is an io.Reader instead of an io.ReadSeeker.
The ContentMD5 member for pre-computed MD5 checksums will be ignored for multipart uploads. Objects that will be uploaded in a single part, the ContentMD5 will be used.
The Checksum members for pre-computed checksums will be ignored for multipart uploads. Objects that will be uploaded in a single part, will include the checksum member in the request.
UploadOutput struct
UploadOutput represents a response from the Upload() call.
Function Details
func GetBucketRegion(ctx aws.Context, c client.ConfigProvider, bucket, regionHint string, opts ...request.Option) (string, error)
GetBucketRegion will attempt to get the region for a bucket using the regionHint to determine which AWS partition to perform the query on.
The request will not be signed, and will not use your AWS credentials.
A “NotFound” error code will be returned if the bucket does not exist in the AWS partition the regionHint belongs to. If the regionHint parameter is an empty string GetBucketRegion will fallback to the ConfigProvider's region config. If the regionHint is empty, and the ConfigProvider does not have a region value, an error will be returned..
For example to get the region of a bucket which exists in “eu-central-1” you could provide a region hint of “us-west-2”.
sess := session.Must(session.NewSession())
bucket := "my-bucket"
region, err := s3manager.GetBucketRegion(ctx, sess, bucket, "us-west-2")
if err != nil {
if aerr, ok := err.(awserr.Error); ok && aerr.Code() == "NotFound" {
fmt.Fprintf(os.Stderr, "unable to find bucket %s's region not found\n", bucket)
}
return err
}
fmt.Printf("Bucket %s is in %s region\n", bucket, region)
By default the request will be made to the Amazon S3 endpoint using the Path style addressing.
s3.us-west-2.amazonaws.com/bucketname
This is not compatible with Amazon S3's FIPS endpoints. To override this behavior to use Virtual Host style addressing, provide a functional option that will set the Request's Config.S3ForcePathStyle to aws.Bool(false).
region, err := s3manager.GetBucketRegion(ctx, sess, "bucketname", "us-west-2", func(r *request.Request) {
r.S3ForcePathStyle = aws.Bool(false)
})
To configure the GetBucketRegion to make a request via the Amazon S3 FIPS endpoints directly when a FIPS region name is not available, (e.g. fips-us-gov-west-1) set the Config.Endpoint on the Session, or client the utility is called with. The hint region will be ignored if an endpoint URL is configured on the session or client.
sess, err := session.NewSession(&aws.Config{
Endpoint: aws.String("https://s3-fips.us-west-2.amazonaws.com"),
})
region, err := s3manager.GetBucketRegion(context.Background(), sess, "bucketname", "")
62 63 64 65 66 67 68 69 |
// File 'service/s3/s3manager/bucket_region.go', line 62
|
func GetBucketRegionWithClient(ctx aws.Context, svc s3iface.S3API, bucket string, opts ...request.Option) (string, error)
GetBucketRegionWithClient is the same as GetBucketRegion with the exception that it takes a S3 service client instead of a Session. The regionHint is derived from the region the S3 service client was created in.
By default the request will be made to the Amazon S3 endpoint using the Path style addressing.
s3.us-west-2.amazonaws.com/bucketname
This is not compatible with Amazon S3's FIPS endpoints. To override this behavior to use Virtual Host style addressing, provide a functional option that will set the Request's Config.S3ForcePathStyle to aws.Bool(false).
region, err := s3manager.GetBucketRegionWithClient(ctx, client, "bucketname", func(r *request.Request) {
r.S3ForcePathStyle = aws.Bool(false)
})
To configure the GetBucketRegion to make a request via the Amazon S3 FIPS endpoints directly when a FIPS region name is not available, (e.g. fips-us-gov-west-1) set the Config.Endpoint on the Session, or client the utility is called with. The hint region will be ignored if an endpoint URL is configured on the session or client.
region, err := s3manager.GetBucketRegionWithClient(context.Background(),
s3.New(sess, &aws.Config{
Endpoint: aws.String("https://s3-fips.us-west-2.amazonaws.com"),
}),
"bucketname")
See GetBucketRegion for more information.
103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 |
// File 'service/s3/s3manager/bucket_region.go', line 103
|
func NewBatchDelete(c client.ConfigProvider, options ...func(*BatchDelete)) *BatchDelete
NewBatchDelete will return a new delete client that can delete a batched amount of objects.
Example:
batcher := s3manager.NewBatchDelete(sess, size)
objects := []BatchDeleteObject{
{
Object: &s3.DeleteObjectInput {
Key: aws.String("key"),
Bucket: aws.String("bucket"),
},
},
}
if err := batcher.Delete(aws.BackgroundContext(), &s3manager.DeleteObjectsIterator{
Objects: objects,
}); err != nil {
return err
}
257 258 259 260 |
// File 'service/s3/s3manager/batch.go', line 257
|
func NewBatchDeleteWithClient(client s3iface.S3API, options ...func(*BatchDelete)) *BatchDelete
NewBatchDeleteWithClient will return a new delete client that can delete a batched amount of objects.
Example:
batcher := s3manager.NewBatchDeleteWithClient(client, size)
objects := []BatchDeleteObject{
{
Object: &s3.DeleteObjectInput {
Key: aws.String("key"),
Bucket: aws.String("bucket"),
},
},
}
if err := batcher.Delete(aws.BackgroundContext(), &s3manager.DeleteObjectsIterator{
Objects: objects,
}); err != nil {
return err
}
223 224 225 226 227 228 229 230 231 232 233 234 |
// File 'service/s3/s3manager/batch.go', line 223
|
func NewBatchError(code, message string, err []Error) awserr.Error
NewBatchError will return a BatchError that satisfies the awserr.Error interface.
74 75 76 77 78 79 80 |
// File 'service/s3/s3manager/batch.go', line 74
|
func NewDeleteListIterator(svc s3iface.S3API, input *s3.ListObjectsInput, opts ...func(*DeleteListIterator)) BatchDeleteIterator
NewDeleteListIterator will return a new DeleteListIterator.
145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 |
// File 'service/s3/s3manager/batch.go', line 145
|
func NewDownloader(c client.ConfigProvider, options ...func(*Downloader)) *Downloader
NewDownloader creates a new Downloader instance to downloads objects from S3 in concurrent chunks. Pass in additional functional options to customize the downloader behavior. Requires a client.ConfigProvider in order to create a S3 service client. The session.Session satisfies the client.ConfigProvider interface.
Example:
// The session the S3 Downloader will use
sess := session.Must(session.NewSession())
// Create a downloader with the session and default options
downloader := s3manager.NewDownloader(sess)
// Create a downloader with the session and custom options
downloader := s3manager.NewDownloader(sess, func(d *s3manager.Downloader) {
d.PartSize = 64 * 1024 * 1024 // 64MB per part
})
99 100 101 |
// File 'service/s3/s3manager/download.go', line 99
|
func NewDownloaderWithClient(svc s3iface.S3API, options ...func(*Downloader)) *Downloader
NewDownloaderWithClient creates a new Downloader instance to downloads objects from S3 in concurrent chunks. Pass in additional functional options to customize the downloader behavior. Requires a S3 service client to make S3 API calls.
Example:
// The session the S3 Downloader will use
sess := session.Must(session.NewSession())
// The S3 client the S3 Downloader will use
s3Svc := s3.New(sess)
// Create a downloader with the s3 client and default options
downloader := s3manager.NewDownloaderWithClient(s3Svc)
// Create a downloader with the s3 client and custom options
downloader := s3manager.NewDownloaderWithClient(s3Svc, func(d *s3manager.Downloader) {
d.PartSize = 64 * 1024 * 1024 // 64MB per part
})
137 138 139 |
// File 'service/s3/s3manager/download.go', line 137
|
func NewUploader(c client.ConfigProvider, options ...func(*Uploader)) *Uploader
NewUploader creates a new Uploader instance to upload objects to S3. Pass In additional functional options to customize the uploader's behavior. Requires a client.ConfigProvider in order to create a S3 service client. The session.Session satisfies the client.ConfigProvider interface.
Example:
// The session the S3 Uploader will use
sess := session.Must(session.NewSession())
// Create an uploader with the session and default options
uploader := s3manager.NewUploader(sess)
// Create an uploader with the session and custom options
uploader := s3manager.NewUploader(session, func(u *s3manager.Uploader) {
u.PartSize = 64 * 1024 * 1024 // 64MB per part
})
199 200 201 |
// File 'service/s3/s3manager/upload.go', line 199
|
func NewUploaderWithClient(svc s3iface.S3API, options ...func(*Uploader)) *Uploader
NewUploaderWithClient creates a new Uploader instance to upload objects to S3. Pass in additional functional options to customize the uploader's behavior. Requires a S3 service client to make S3 API calls.
Example:
// The session the S3 Uploader will use
sess := session.Must(session.NewSession())
// S3 service client the Upload manager will use.
s3Svc := s3.New(sess)
// Create an uploader with S3 client and default options
uploader := s3manager.NewUploaderWithClient(s3Svc)
// Create an uploader with S3 client and custom options
uploader := s3manager.NewUploaderWithClient(s3Svc, func(u *s3manager.Uploader) {
u.PartSize = 64 * 1024 * 1024 // 64MB per part
})
241 242 243 |
// File 'service/s3/s3manager/upload.go', line 241
|
func WithDownloaderRequestOptions(opts ...request.Option) func(*Downloader)
WithDownloaderRequestOptions appends to the Downloader's API request options.
75 76 77 78 79 |
// File 'service/s3/s3manager/download.go', line 75
|
func WithUploaderRequestOptions(opts ...request.Option) func(*Uploader)
WithUploaderRequestOptions appends to the Uploader's API request options.
116 117 118 119 120 |
// File 'service/s3/s3manager/upload.go', line 116
|