How to Upload a File to Directory in S3 Bucket Using Boto

How to upload a file to directory in S3 bucket using boto

NOTE: This answer uses boto. See the other answer that uses boto3, which is newer.

Try this...

import boto
import boto.s3
import sys
from boto.s3.key import Key

AWS_ACCESS_KEY_ID = ''
AWS_SECRET_ACCESS_KEY = ''

bucket_name = AWS_ACCESS_KEY_ID.lower() + '-dump'
conn = boto.connect_s3(AWS_ACCESS_KEY_ID,
        AWS_SECRET_ACCESS_KEY)

bucket = conn.create_bucket(bucket_name,
    location=boto.s3.connection.Location.DEFAULT)

testfile = "replace this with an actual filename"
print 'Uploading %s to Amazon S3 bucket %s' % \
   (testfile, bucket_name)

def percent_cb(complete, total):
    sys.stdout.write('.')
    sys.stdout.flush()

k = Key(bucket)
k.key = 'my test file'
k.set_contents_from_filename(testfile,
    cb=percent_cb, num_cb=10)

[UPDATE]
I am not a pythonist, so thanks for the heads up about the import statements.
Also, I'd not recommend placing credentials inside your own source code. If you are running this inside AWS use IAM Credentials with Instance Profiles (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2_instance-profiles.html), and to keep the same behaviour in your Dev/Test environment, use something like Hologram from AdRoll (https://github.com/AdRoll/hologram)

upload a directory to s3 with boto

There is nothing in the boto library itself that would allow you to upload an entire directory. You could write your own code to traverse the directory using os.walk or similar and to upload each individual file using boto.

There is a command line utility in boto called s3put that could handle this or you could use the AWS CLI tool which has a lot of features that allow you to upload entire directories or even sync the S3 bucket with a local directory or vice-versa.

Upload file to S3 folder using python boto

Folders don't really exist in S3. You can prefix the file name (object key) with the something that looks like a folder path.

It's not entirely clear to me what your code is doing with the file paths, but your code needs to be changed to something like this:

for f in files_to_upload:
    key = "my/s3/folder/name/" + f
    client.put_object(Bucket=BUCKET_NAME, Key=key, Body=f)

Note: You weren't passing a Body parameter, so I think your code was just creating empty objects in S3.

uploading file to specific folder in S3 using boto3

You do not need to pass the Key value as an absolute path. The following should work:

upload_file('/tmp/' + filename, '<bucket-name>', 'folder/{}'.format(filename))

Boto - Uploading file to a specific location on Amazon S3

All you should have to do is prepend the virtual directory path to the key name prior to uploading. For example:

key_name = 'my test file'
path = 'images/holiday'
full_key_name = os.path.join(path, key_name)
k = bucket.new_key(full_key_name)
k.set_contents_from_filename(...)

You may have to change that a bit for your application but hopefully that gives you the basic idea.

Uploading Files to AWS S3 Bucket Folder in Python Causes Regex Error

The API call wants the bucket name or ARN. Your bucket name is task-details and your bucket ARN is arn:aws:s3:::task-details.

You use the Key parameter when calling upload_file to specify the object's key, for example archive/cats/persian.png. Note that the S3 object key is not simply the object/file name but also includes the prefix/folder.

---

Related Topics