Package com.ibm.cloud.objectstorage.services.s3

Synchronous client for accessing Amazon S3.

See: Description

Interface Summary

Interface	Description
AmazonS3	Provides an interface for accessing the Amazon S3 web service.
AmazonS3EncryptionV2	Interface for an S3 client that performs encryption, V2.
Headers	Common S3 HTTP header values used throughout the Amazon Web Services S3 Java client.
OnFileDelete	A service provider interface (SPI) used to notify the event of a file deletion.

Class Summary

Class	Description
AbstractAmazonS3	An Abstract class that users needs to extend instead of AmazonS3 interface.
AmazonS3Builder<Subclass extends AmazonS3Builder,TypeToBuild extends AmazonS3>
AmazonS3Client	Provides the client for accessing the Amazon S3 web service.
AmazonS3ClientBuilder	Fluent builder for AmazonS3.
AmazonS3URI	A URI wrapper that can parse out information about an S3 URI.
S3ClientOptions	S3 client configuration options such as the request access style.
S3ClientOptions.Builder
S3ResponseMetadata	Extension of ResponseMetadata with Amazon S3 specific data.
UploadObjectObserver	An observer that gets notified of ciphertext file creation for the purpose of pipelined parallel multi-part uploads of encrypted data to S3.

Exception Summary

Exception	Description
KeyWrapException	Key wrapping related security exception.

Package com.ibm.cloud.objectstorage.services.s3 Description

Synchronous client for accessing Amazon S3.

Appendix: Amazon S3 client-side encryption meta information

Introduction

This appendix summarizes the current crypto related meta information associated with an S3 object encrypted using SDK client-side encryption. In SDK's terminology, these meta information are classified as "user metatdata". In general, the meta information is just a map of key/value pairs. Physically they are persisted either as metadata of the encrypted S3 object, or as a separate S3 object called an "instruction file" (which is just a JSON file in plaintext.) For all references to the Cipher Algorithm Names, Modes, and Padding, such as "AES/GCM/NoPadding", please refer to Oracle's Java™ Cryptography Architecture Standard Algorithm Name Documentation for more details.

S3 metadata vs instruction file

Note when a key/value pair is stored as "user metadata" as part of the S3 metadata, the SDK always adds an additional prefix of x-amz-meta- to the key name. In contrast, the SDK doesn't add any prefix when the key names are stored in an instruction file. (The reason has to do with how the http headers are handled.)

For example, x-amz-key would get stored as x-amz-meta-x-amz-key in S3 metadata. In contrast, x-amz-key would simply be stored as is (ie without the prefix) in an instruction file. All the keys described below are considered "user metadata" by the SDK.

S3 1st gen crypto meta information

For non-KMS client side encryption, this crypto meta information is used to support the "Encryption Only" crypto mode (which involves the use of AES/CBC/PKCS5Padding for content encryption, and AES/ECB for encrypting the one-time randomly generated data key). For KMS client-side encryption, the 2nd gen crypto meta information is used.

Scope

This metadata can only be generated when encrypting using the V1 encryption client, AmazonS3EncryptionClient, but can be interpreted (decrypted) by com.ibm.cloud.objectstorage.services.s3.AmazonS3EncryptionClientV2. The format is also known as v1 metadata, but because there are now v1 and v2 clients, the term v1 metadata is confusing.

Encryption: AmazonS3EncryptionClient, crypto mode EncryptionOnly
Decryption: AmazonS3EncryptionClient, crypto mode EncryptionOnly or com.ibm.cloud.objectstorage.services.s3.AmazonS3EncryptionClientV2, crypto mode AuthenticationOnly

Metadata

key	description
x-amz-key	Content encrypting key (cek) in encrypted form, base64 encoded. The cek is randomly generated per S3 object, and is always an AES 256-bit key. The corresponding cipher is always "AES/CBC/PKCS5Padding".
x-amz-iv	Randomly generated IV (per S3 object), base64 encoded
x-amz-matdesc	Customer provided material description in JSON (UTF8) format. Used to identify the client-side master key (ie used to encrypt/wrap the generated content encrypting key).
x-amz-unencrypted-content-length	Unencrypted content length (optional but should be specified whenever possible).

S3 2nd gen crypto meta information

This crypto meta information is used to support both authenticated encryption (which involves the use of AES/GCM/NoPadding for content encryption, and key wrapping for the one-time randomly generated data key), and KMS client-side encryption (which can either be encryption-only or authenticated encryption).

Scope

2nd gen metadata can be generated/encrypted by both the V1 and V2 encryption clients. The format is also known as v2 metadata, but this is confusing since the V1 client also generates 2nd gen metadata in the AuthenticatedEncryption and StrictAuthenticatedEncryption modes.

Encryption: AmazonS3EncryptionClient/ AmazonS3EncryptionClient, crypto modes AuthenticatedEncryption and StrictAuthenticatedEncryption
Decryption: AmazonS3EncryptionClient, all crypto modes, or com.ibm.cloud.objectstorage.services.s3.AmazonS3EncryptionClientV2, crypto modes AuthenticatedEncryption and StrictAuthenticatedEncryption.

Note: While the V1 client can both generate and interpret 2nd gen metadata, it cannot decrypt any object encrypted using the V2 client, because the values of x-amz-wrap-alg differ.

Metadata used the same way by V1 and V2 clients

key	description
x-amz-key-v2	CEK in key wrapped form. This is necessary so that the S3 encryption client that doesn't recognize the v2 format will not mistakenly decrypt S3 object encrypted in v2 format.
x-amz-iv	Randomly generated IV (per S3 object), base64 encoded. (Same as v1.)
x-amz-unencrypted-content-length	Unencrypted content length. (optional but should be specified whenever possible. Same as v1.)
x-amz-tag-len	Tag length (in bits) when AEAD is in use. Only applicable if AEAD is in use. This meta information is absent otherwise, or if KMS is in use. Supported value: "128"

Metadata using V1 client

key	description
x-amz-matdesc	Customer provided material description in JSON format. (Same as v1). Used to identify the client-side master key. For KMS client side encryption, the KMS Customer Master Key ID is stored as part of the material description, x-amz-matdesc, under the key-name "kms_cmk_id".
x-amz-wrap-alg	Key wrapping algorithm used. Supported values: "AESWrap", "RSA/ECB/OAEPWithSHA-256AndMGF1Padding", "kms" No standard key wrapping is used if this meta information is absent Always set to "kms" if KMS is used for client-side encryption
x-amz-cek-alg	Content encryption algorithm used. Supported values: "AES/GCM/NoPadding", "AES/CBC/PKCS5Padding" Default to "AES/CBC/PKCS5Padding" if this key is absent.

Metadata using V2 client

key	description
x-amz-matdesc	Customer provided material description in JSON format. (Same as v1). For KMS client side encryption, the cek algorithm is stored as part of the material description under the key-name aws:x-amz-cek-alg.
x-amz-wrap-alg	Key wrapping algorithm used. Supported values: "AES/GCM/NoPadding" (symmetric default), "RSA-OAEP-SHA1", "RSA-OAEP-SHA1" (asymmetric default), "kms" No standard key wrapping is used if this meta information is absent Always set to "kms" if KMS is used for client-side encryption
x-amz-cek-alg	Content encryption algorithm used. Supported values: "AES/GCM/NoPadding"

KMS Integration for client-side encryption

1. All client-side KMS protected S3 objects are stored in v2 crypto meta information format. However, the key wrapping algorithm, x-amz-wrap-alg is always set to "kms".
2. For V1 clients, the KMS Customer Master Key ID is currently stored as part of the material description, x-amz-matdesc, under the key-name "kms_cmk_id".
3. For V2 clients, the KMS Customer Master Key ID is not stored as part of the material description and must be supplied by the decrypting client.

Sample S3 crypto meta information for KMS client-side encryption

V1 client, CryptoMode: EncryptionOnly

 x-amz-meta-x-amz-key-v2: 
 CiAJ2GqwpJRnt4izujwoalC2YdNRNSMEslG5rw3pEI+J8hKnAQEBAwB4CdhqsKSUZ7eIs7o8KGpQtmHTUTUjBLJRua8N6RCPifIAAAB+MHwGCSqGSIb3DQEHBqBvMG0CAQAwaAYJKoZIhvcNAQcBMB4GCWCGSAFlAwQBLjARBAyD0rzN3KHViWixDVcCARCAOwhZ6xA8rob3Z0mNx3uKFieHKVyIC/PKlchALvG1oVLnf86mcZJNpTXtwMkKkDocVj3Z1zGjZTufFsOH
 x-amz-iv: xqOHh+M6y0UogwG5SHJM3Q==
 x-amz-unencrypted-content-length: 32768
 x-amz-cek-alg: AES/CBC/PKCS5Padding
 x-amz-wrap-alg: kms
 x-amz-matdesc: {"kms_cmk_id":"d3fd2273-4ca0-4da5-b1c2-c89742ec6a26"}
 

V1 client, CryptoMode: AuthenticatedEncryption/StrictAuthenticatedEncryption

 x-amz-key-v2: CiAJ2GqwpJRnt4izujwoalC2YdNRNSMEslG5rw3pEI+J8hKnAQEBAwB4CdhqsKSUZ7eIs7o8KGpQtmHTUTUjBLJRua8N6RCPifIAAAB+MHwGCSqGSIb3DQEHBqBvMG0CAQAwaAYJKoZIhvcNAQcBMB4GCWCGSAFlAwQBLjARBAytFVyW4I6QmwcPK1ECARCAO1VzoM6KxA9DHyNMm/BEbbjle2vLA7IY7f9OkUoSqrdxgKxpNID9QaL/7928m1Djtld0bJpHefjm89C
 x-amz-iv: VZBksyij6DhrUMye
 x-amz-unencrypted-content-length: 32768
 x-amz-cek-alg: AES/GCM/NoPadding
 x-amz-wrap-alg: kms
 x-amz-matdesc: {"kms_cmk_id":"d3fd2273-4ca0-4da5-b1c2-c89742ec6a26"}
 x-amz-tag-len: 128
 

V2 client, CryptoMode: AuthenticatedEncryption/StrictAuthenticatedEncryption

 x-amz-key-v2: CiAJ2GqwpJRnt4izujwoalC2YdNRNSMEslG5rw3pEI+J8hKnAQEBAwB4CdhqsKSUZ7eIs7o8KGpQtmHTUTUjBLJRua8N6RCPifIAAAB+MHwGCSqGSIb3DQEHBqBvMG0CAQAwaAYJKoZIhvcNAQcBMB4GCWCGSAFlAwQBLjARBAytFVyW4I6QmwcPK1ECARCAO1VzoM6KxA9DHyNMm/BEbbjle2vLA7IY7f9OkUoSqrdxgKxpNID9QaL/7928m1Djtld0bJpHefjm89C
 x-amz-iv: VZBksyij6DhrUMye
 x-amz-unencrypted-content-length: 32768
 x-amz-cek-alg: AES/GCM/NoPadding
 x-amz-wrap-alg: kms
 x-amz-matdesc: {"aws:x-amz-cek-alg":"AES/GCM/NoPadding"}
 x-amz-tag-len: 128
 

S3 metadata of an instruction file

The following key is always included as part of the S3 metadata of the instruction file (which is just an S3 object). Since the SDK treats this key as "user metadata", the key name is always prefixed with x-amz-meta- when physically stored.

key	description
x-amz-crypto-instr-file	the presence of this key is used to indicate this S3 object is an instruction file (of an associated client-side encrypted S3 object).