Mounting non us-east-1 AWS based S3 bucket into an EC2 instance

The Problem

AWS S3 is a global service for storing block data; however, when using a bucket as a file system is can be confusing how to use a bucket that is not in the us-east-1 region.

Pre-Flight

• AWS Account

The Solution

Using the s3fs utility it is possible to mount a bucket with as little as the following:

s3fs s3://my_bucket_name /path/to/my/mount/point \
-o url=https://s3.amazonaws.com

Looking at the documentation for s3fs via man s3fs we see very detailed documentation. But nothing about non us-east-1 bucket mounting. It took me awhile but after some digging around trials I found that we need to use the REGIONAL HTTPS endpoint for the bucket. Trying to use the generic global HTTPS endpoint would not work.

Combine this knowledge with the fact that the Instance Meta-Data Server (IMDS) contains the instance availability zone we can automate the mounting process in a few simple steps.

# Get IMDS token
IMDS_TOKEN=$(curl -H "X-aws-ec2-metadata-token-ttl-seconds: 15" -X PUT "http://169.254.169.254/latest/api/token")

# Get Instance AZ
EC2_AZ=$(curl -H "X-aws-ec2-metadata-token: $IMDS_TOKEN" -s http://169.254.169.254/latest/meta-data/placement/availability-zone)

# Get Instance Region
EC2_REGION=${EC2_AZ::-1}

# Mount bucket using regional endpoint
s3fs s3://my_bucket_name /path/to/my/mount/point \
-o url=https://s3-${EC2_REGION}.amazonaws.com

Wrap Up

And there we have it. Mounting an AWS S3 bucket based in a region not us-east-1 into a EC2 Linux host.

Additional Resources

• https://github.com/s3fs-fuse/s3fs-fuse/issues/465
• https://cloudkul.com/blog/mounting-s3-bucket-linux-ec2-instance/
• S3 Perf: https://improve.dk/pushing-the-limits-of-amazon-s3-upload-performance/
• Alt FS binaries: http://balamurugan99.blogspot.com/2016/07/mounting-amazon-s3-as-linux-filesystem.html
• https://stackoverflow.com/questions/4249488/find-region-from-within-an-ec2-instance