AWS automates Glacier S3 tiering – for a small consideration

Amazon yesterday unleashed a barrage of product updates at the AWS Storage Day. Additional tiering services for Amazon S3 Glacier archive storage was probably the most notable announcement but new features for EBS, EFX, FSx, DataSync, Snow offerings and the Storage Gateway also scrambled to gain our attention.

Glacier, the AWS S3-based object storage service, includes archive and deep archive tiers for infrequently and rarely accessed data. For a fee, AWS will now, monitor your data in Glacier and automatically move cold data into and between the archive and deep archive tiers, and newly-accessed archive data up a tier.

“These new optimisations will reduce the amount of manual work you need to do to archive objects with unpredictable access patterns,” writes Marcia Villalba, a senior developer advocate at AWS.

Archive access tiers

There are two new access tiers: Archive Access with the same performance and pricing as the S3 Glacier storage class; and Deep Archive Access, which has the same performance and pricing as the Deep Archive storage class.

S3 Intelligent-Tiering automatically moves objects that haven’t been accessed for 90 days to the Archive Access tier, and to the Deep Archive Access tier after 180 days. Customers pay about $1 per TB per month in the Deep Archive tier. 

AWS Archive tiering.

Archive access tier objects are retrieved in three-five hours, and Deep Archive access tier object retrieval takes up to 12 hours. The new tiers join the existing Frequent Access and Infrequent Access tiers.

Objects smaller than 128KB are kept in the Frequent Access tier. For each object archived to the Archive Access tier or Deep Archive Access tier, S3 uses 8KB of storage for the name of the object and other metadata (billed at S3 Standard storage rates) and 32 KB of storage for index and related metadata (billed at S3 Glacier and S3 Glacier Deep Archive storage rates). 

To recap, AWS offers:

  • Simple Storage Service (S3) Standard
  • S3 Standard Infrequent Access (IA)
  • S3 Glacier – retrieval within 3 – 5 hours
  • S3 Glacier Deep Archive – retrieval within 12 hours

There are now four access tiers, with automated data movement services, matching these; Frequent Access (S3 Standard), Infrequent Access (S3 Standard Infrequent Access), Archive Access (S3 Glacier), and Deep Archive Access ( S3 Glacier Deep Archive.)

More AWS storage stuff

For the record, we note AWS’s other storage announcements yesterday.

FSx (File Systems) 

  • FSx for Lustre gets storage quotas
  • FSx for Windows File Server gets DNS alias access and file share access from container-based Windows workloads running on ECS,
  • FSX systems can have scheduled, policy-driven backup from AWS Backup

EBS (Elastic Block Store) gets a Cold HDD (sc1) volume type for low-cost magnetic storage for large, sequential, cold-data workloads.

EFS (Elastic File System) – users can create an EFS when using the EC2 Launch Instance Wizard, get the new EFS added to the instance and mounted automatically when the instance is launched.

DataSync

  • DataSync can run fully-automated data transfers between datasets stored in S3, EFS, or FSx for Windows File Server, without deploying DataSync agents
  • DataSync network bandwidth can be adjusted dynamically up or down

Snow family

  • Virtual machine images in raw (disk image) format as AMIs into Snowball Edge Storage Optimized and Snowball Edge Compute Optimised devices,
  • Import Windows 2012 and Windows 2016 virtual machine images and use them to launch instances on Snowball devices.
Snowball Edge device.

Storage Gateway

  • Import Windows 2012 and Windows 2016 virtual machine images and use them to launch instances on Snowball devices,
  • Create a schedule to control the maximum network bandwidth consumed by Tape and Volume Gateways,
  • Uploads to File Gateway can trigger Amazon CloudWatch or Amazon EventBridge notifications and so initiate automated workflows,
  • File Gateway supports access-based limits to ensure users only see the SMB file shares, folders, and files that they have permission to open.

Read an AWS blog by Chief Evangelist Jeff Barr to explore these points in more detail.