In this blog post, we will demonstrate an optimization that we added to our "Sync Folders" feature that allows to sync Amazon S3 bucket with local folders. In the previous version of MSP360 S3 Explorer we do request header for each file on s3 to get original date modified. It used to make the user pay extra money to Amazon because of the number of requests (equals to the number of files on s3). If you let’s say have 70 000 files on S3 that you want to sync, every time sync feature runs it would run ahead request 70 000 times.
In version 2.0 we are adding an option "Compare Content", which is ON by default.
The option calculates MD5 checksum for each file on a local computer that has a matching file on s3. Then it compares the MD5 for local file and ETAG (the MD5 checksum calculated by Amazon when the file is uploaded to s3).
If the ETAG is different we are looking for Date Created (the date of file uploading) and compare to local Date Modified. If local date modified is newer than Date Created on s3 we do not do additional requests for this file and upload it to s3 storage.
If local date modified is older than Date Created on s3 we do request for headers for this file and look into Date Modified stored in our header.
This way we can significantly minimize the number of requests to Amazon S3.
- File management in Amazon S3 and S3-compatible storage
- Encryption and compression
- IAM and security management
