Definition

• Storage service - a backbone service of AWS, provides secure, durable, highly-scalable.

• Easy to use object storage with a simple web service interface

  • Object-based storage

  • A safe place to store your files

  • The data is spread across multiple device and facilities

  • File: 0-5TB

  • Unlimited storage

  • Files saved in buckets

  • Globally (like domains)

Buckets

• Web folder (a flat folder) - containers for objects, root namespace for AWS S3.

• The name must be unique across all AWS accounts (like domains).

Basics

Object-based storage

• Objects are the entities for files stored in S3

• Size: up to 5TB

• Store data as binary

• Object has:

  • keys: a unique identifier (like a file name)

  • data: binary data

  • metadata: information about the object

    • system metadata: auto-generated by S3

    • user metadata (optional): generated by the user

  • versionId: for versioning

  • subresources:

    • ACL: access control list

    • torrent: torrent supported info

Data Consistency

• Data can be accessed by REST API using URL:

  • Not support object locking → request that has the latest timestamp is executed

  • read-after-write consistency for PUTS new objects

  • eventually consistency for PUTS to an existing object or DELETE objects

Access Control

• S3 is secure by default (public access is turned off)

• Access List Controls (ACLs)

  • Logging for buckets

  • Hosting static webs

• Bucket policies

  • IP range

  • AWS account

  • Objects with prefixes

• IAM

• Bucket policies are the recommended access control mechanism

Advanced Features

Prefixes and Delimiters

• S3 uses a flat structure in a bucket

• Using prefixes and delimiters in order to make a file and folder hierarchy

• Not a file system

Storage Classes

Object Lifecycle

• Automated tiering

• Should be used to reduce cost

Encryption

Server-side encryption

• SSE-S3 (AWS-Managed Keys): "check-box-style" encryption AWS handles the management and protection of the key. A new key is issued monthly.

• SSE-KMS (AWS KMS Keys): Handle user-provided keys management and protection, provides auditing → access logging

• SSE-C (Customer-Provided Keys): Encrypt/decrypt data by using user-provided keys

Client-side encryption

• AWS KMS key

• Client-side master key

For maximum simplicity and ease of use, use SSE-S3 or SSE-KMS

Versioning

• Protects data against accidental or malicious deletion

• Once enabled, can not be removed, it can only be suspended

• Bucket level setting

• Save versions of the file by using IDs

MFA Delete

• Additional authentication prevents accidental or malicious deletion

• Can only be enabled by the root account

Pre-Signed URLs

• Use owner security credential to grant time-limited permission to access objects

• Protect against "content scraping."

Multi-part upload

• Used for large file

• 3 steps process

  • Initiation

  • Uploading the parts

  • Completion

• Should be used for objects larger than 100Mbytes

• Must be used for objects larger than 5GB

Cross-Region Replication

• Asynchronously replicate to a target bucket in another region

• Versioning must be turned on

• Must use IAM policy to give Amazon the permission

• Replicate only new object (created after activation), and delete marker is replicated

• Delete a delete marker is not replication between buckets

Events

• Trigger action by S3 events:

  • SNS

  • SQS

  • Lambda

Logging

• Off by default

• Should use with prefixed (/logs, bucketname/logs)

• Information

  • Request user account

  • Bucket name

  • Request time

  • Action (GET, PUT, LIST)

  • Response status, error code

Best practices

• Use for hybrid IT environments and applications (data in on-premise file systems, databases back-ups,...)

• bulk blob for data

• If request rate > 100 requests per second, use a hash prefix

• Static web hosting should use CloudFront as a cache layer

• domain: bucket-name.s3-website-regions.amazonaws.com

Glacier

• Extremely low-cost storage service provides durable, secure, and flexible storage for data archiving and online backup.

• Retrieval time of three to five hours

• 99.9(9)% durability

Archives

• Data is stored in archives

• Up to 40TB and an unlimited number of archives

• Assigned a unique archive ID at the time of creation

• Automatically encrypted and immutable

Vaults

• Vaults are containers for archives

• Each AWS accounts can have up to 1000 vaults

Vault Locks

• Enforce compliance controls with a vault lock policy

• Once locked, the policy can no longer be changed

Data Retrieval

• Up to 5% of data can be stored for free each month

S3 vs. Glacier

Data size| 40TB archive | 5TB object | Identifier|system-generated Ids|friendly key names| Encryption| automatically encrypted|encrypted at rest|

Review questions: 90% → PASSED!

Mindmap