Amazon S3 buckets protection is a critical security measure when using AWS. There have been numerous bad stories about unprotected S3 buckets, from established contractors like Accenture and Booz Allen Hamilton to huge companies like Verizon Wireless and Time Warner Cable. In this article, you will learn how to enhance the security of S3 buckets with pre-signed S3 URLs, and how to generate them for various cases.
Table of Contents
What Are Pre-Signed S3 URLs and Why Use Them?
Pre-signed URLs are used to provide short-term access to a private object in your S3 bucket. They work by appending an AWS Access Key, expiration time, and Sigv4 signature as query parameters to the S3 object.
There are two common use cases when you may want to use them:
- Simple, occasional sharing of private files.
- Frequent, programmatic access to view or upload a file in an application.
Let's explore each of these use cases below.
Using S3 URLs for File Sharing
Imagine you may want to share a confidential presentation with a business partner, or you want to allow a friend to download a video file you're storing in your S3 bucket. In both situations, you could generate a URL in CloudBerry Explorer for Amazon S3, and share it to allow the recipient short-term access.
There are a couple of different approaches for generating these URLs in an ad-hoc, one-off fashion, including:
- Using the AWS Tools for Powershell.
- Using the AWS CLI.
- Using CloudBerry Explorer.
See below for examples of each of these approaches.
Generating a Pre-Signed S3 URL with AWS Tools for Powershell
If you use the AWS Tools for Powershell, you can use the Get-S3PreSignedURLcmdlet to generate a pre-signed URL in your Powershell.
The syntax is:
Get-S3PreSignedURL -Bucket cloudberry-examples -Key presentation.ppt -Expires 2018-07-13
The cmdlet will return the URL that you can copy and use as needed.
Generating a pre-signed S3 URL with the AWS CLI
To generate it with the AWS CLI, you can simply use the aws s3 presign command.
On a Windows system, the command is:
"C:\Program Files\Amazon\AWSCLI\aws.exe" s3 presign s3://cloudberry-examples/presentation.ppt
On a Mac or Linux system, the command is:
$ aws s3 presign s3://cloudberry-examples/presentation.ppt
The console will return a URL that you can copy and paste to the desired user.
Generating a pre-signed S3 URL with MSP360 Explorer
MSP360 Explorer is the easiest way to generate a one-off URL. Simply follow the steps below.
First, choose the object for which you want to generate a URL, then click the "Web URL" button, as shown in the image below.
Second, choose whether you want an HTTP or HTTPS URL. You should prefer an HTTPS URL as the query string parameters, including the Access Key and signature, will be sent over a secure connection. Select the "Expire URL at a certain date"option to configure the expiration date. Finally, click "Generate", and copy the URL.
Using S3 URLs for Temporary, Automated Access in Your Application Code
The examples shown above are useful for generating a single pre-signed URL that you need for an ad hoc use case. More commonly, you may have an application that needs to generate short-term access to an S3 bucket. For instance, you need this URL when you store files on S3, and users need to download them from your bucket, or, alternatively, upload something.
You can perform both of these operations with the AWS SDKs for any language. Below are examples of how to use Boto 3, the AWS SDK for Python, to generate these URLs in your application code.
Generating a URL for Reading an Object in Your Application Code with Python and Boto3
As mentioned above, you may want to provide temporary read access to an S3 object to a user of your application. The code snippet below shows how you would do it in your application code.
First, we import the boto3 library and construct a client to interact with S3. Then, we generate a URL that will allow the GetObject API call on the object we specify:
import boto3
s3 = boto3.client('s3')
url = s3.generate_presigned_url(
ClientMethod='get_object',
Params={
'Bucket': 'cloudberry-examples',
'Key': 'invoices/user1234/july2018.pdf'
}
)
print(url)
# https://cloudberry-examples.s3.amazonaws.com/invoices/user1234/july2018.pdf?AWSAccessKeyId=AKIALGKOKBY37F5FZF4I&Signature=bPSs8Kcak%2FgjEq
This URL could be sent to a user to view in their browser and receive temporary access.
Generating a URL for uploading an object in your application code with Python and Boto3
You can generate a pre-signed URL that can be used for POST requests. This can be useful for allowing clients to upload large files directly from the browser with limited permissions.
Imagine I want to allow a user to upload a file to my cloudberry-examples bucket with the key name of uploads/image.jpg. In the example below, I use the generate_presigned_post method to construct the Amazon S3 URL and return it to the client. I can even add conditions onto the request, such as ensuring the file size is no larger than 1 MB:
import boto3
s3 = boto3.client('s3')
response = s3.generate_presigned_post(
Bucket='cloudberry-examples',
Key='uploads/image.jpg',
Conditions=[
['content-length-range', 1, 1048579]
]
)
print(response)
{'url': 'https://cloudberry-examples.s3.amazonaws.com/', 'fields': {'key': 'uploads/image.jpg', 'AWSAccessKeyId': 'AKIALGKOKBY37F5FZF4I', 'policy': 'eyJleHBpcmF0aW9uIjogIjIwMTgtMDctMTNUMDI6Mzg6MTBaIiwgImNvbmRpdGlvb
nMiOiBbWyJjb250ZW50LWxlbmd0aC1yYW5nZSIsIDEsIDEwNDg1NzldLCB7Im
J1Y2tldCI6ICJjbG91ZGJlcnJ5LWV4YW1wbGVzIn0sIHsia2V5IjogInVwbG9
hZHMvaW1hZ2UuanBnIn1dfQ==', 'signature': 'ZY7Orehfdzg+ToJJXhYuV/XyK5o='}}
The response will include a URL property, as well as a fields property with a set of key-value pairs. The fields key-value pairs must be sent with the file as part of a multipart/form-data request.
Conclusion
In this article, we learned that pre-signed S3 URLs are an easy way to provide secure access to an object in your S3 buckets. We discussed a few scenarios where you would want to use them, and how to generate these URLs in a variety of methods.
- Introduction to Amazon S3 access tools
- Writing IAM policies
- Mastering Amazon S3 identities
