Upload to S3 Glacier with Amazon S3 Lifecycle Rules

There is a number of storage classes for every need and budget in Amazon Web Services cloud storage. The cheapest method to store the backup for a long time is Amazon Glacier. However, there are two different ways to upload files to that storage: direct and using the AWS Glacier lifecycle policy. In this article, we will demonstrate why the lifecycle policy method is more practical.

Why Uploading to Amazon S3 Glacier?

Amazon S3 Glacier is the most cost-effective storage solution for the infrequently accessed data Amazon Web Services has to offer. You can store any amounts of data for $0.004 per gigabyte per month, so it proves itself cheaper than the on-premises solutions.

It is also secure: by default, only you can access your data. Additionally, you can use the IAM policies to define the additional circle of users who can access your data.

Further reading How to Configure IAM Users for Amazon S3 Glacier

All the uploaded data is encrypted on the server side using 256-bit Advanced Encryption Standard (AES-256). Besides, you can also set up the client-side encryption on your side.

With that said, Amazon Glacier is one of the best options for the data archival.

FREE WHITEPAPER

Archiving to the Cloud: Cold Storage

How to use cold storage - like Amazon S3 Glacier - cost-effectively and efficiently? Find out in our whitepaper:

What Is Amazon S3 Lifecycle Policies?

AWS lifecycle rules are a set of options aimed to manage the data stored within the S3. It allows to move or to delete the objects after the certain number of days by configuring your own lifecycle rules.

Previously, the lifecycle configuration could only be implemented by creating an XML file, but nowadays it is possible to create it using a pretty intuitive web GUI within the AWS management console.

Upload to Glacier Through S3 Lifecycle Policies

Amazon S3 Glacier offers a few data upload options:

• AWS control web interface.
• AWS (SDK) software or Amazon S3 Glacier API.
• Amazon S3 allows setting lifecycle management rules to archive unused data from Amazon S3 to S3 Glacier.
• Own solutions of AWS (APN) network partners that are already embedded in S3 Glacier.

Let’s consider archiving using AWS Glacier Lifecycle Policy in detail.

The cost of data storage in S3 or S3 Glacier is just a part of the price for these services. An extra pay is charged for requests for data stored in S3 and download traffic. In S3 Glacier, this list also includes the cost of data retrieval and the price also depends on the retrieval speed.

 

The point is that all information in Glacier is stored on reasonably priced tape drives, but this information is copied to more expensive media before its transfer to the user. So, when data is archived directly to Glacier, it takes half a day to index it and 3-5 hours more to make it available for retrieval (if standard retrieval is used). If data is uploaded to S3, all files will be instantly available.

It is possible to store several data versions and manage their lifecycle in S3. Upon expiration, data is deleted or it can be transferred to S3 Glacier. If you set a storage class equal to 0 days, information will be immediately sent to S3 Glacier. It is of use when information is rarely accessed in everyday life but its storage life is limited.

Though, it might seem that uploading data to S3 first and go with it to Glacier afterward might be more expensive, AWS has ensured that this exact scenario leads to no more expenses than direct Glacier upload.

Applying AWS Lifecycle Rules via AWS Management Console

The lifecycle rules can be set up in the AWS management console directly.

Here is our step-by-step guide:

1. Open  your AWS console, click “Services” in the right upper corner and select “S3”
   
2. Select the bucket you want to set up lifecycle rule for and click on it. You will see the following menu on the right to it.
   
3. Click “Management”, and then “ Add lifecycle rule” button.
   
4. Select the name and add the tags/prefixes for your lifecycle rule if necessary.
   
5. Check the versions of data you want to be transferred with the lifecycle rule, then select “Transition to Amazon S3 Glacier after” and specify the required amount of days (note that if you set different setting from "0", the costs of storage and file transfer will be bigger).
   
6. After that comes the “Expiration” step. Since the lifecycle is set up for the bucket in S3, you can just skip it and click “Next”.
   
7. See the “Review” step and click “Save”.
   

Automating AWS Glacier Lifecycle Rules with MSP360 Backup

After setting up the backup plan and cloud storage in S3, you should enable AWS Glacier Lifecycle Policy in MSP360 Backup. Go to Tools and click Lifecycle Policy or use the same parameters in the left pane. For the step-by-step guide please check out this article:

Further reading How to Archive Amazon S3 Data to Glacier with MSP360 Backup

In the dialog box select the source of data that will be transferred to the storage.

As a result of such transfer, the files would first get to S3 and after that would get to Glacier in 60 days. If you set 0 days - they will be archived automatically.

Please keep in mind that if your setting is different from "0", the costs of storage and file transfer will be bigger. AWS will charge additional costs for storing files in S3 storage.

This approach has an advantage over the pure Glacier transfer. With the pure Glacier transfer, you have to wait 3-5 hours for the initial inventory to complete. With the AWS S3 Glacier lifecycle policy set to 0 days, the inventory happens as soon as the files are in the storage.

Summary

The need to use Amazon S3 and Glacier individually or in combination depends on the objectives of each separate project. We prepared a comparison table for you to evaluate expenses for both methods of data archiving:

Amazon Web Services provides a possibility to use 5 GB of S3 storage (unfortunately without Glacier) for FREE for 12 months – this time period is enough to get acquainted with system features. We recommend trying MSP360 Backup available as a free trial for 15 days to assess archiving features in full.

You are welcome to share your experience and ask questions in the comments section below.

Denis G

Denis is a content manager at MSP360. He learned the product and cloud services the hard way - as a support team member. He finds the simplest descriptions to difficult concepts and makes comparisons between services, features, products and other comparisons.

See Full Bio

FREE WHITEPAPER

Smart Guide to Amazon S3 Glacier

• How to avoid the hidden costs of S3 Glacier
• How to actually save on data cloud storage with lifecycle policies
• Why data recovery planning can save you a dime