How to Copy File Across Buckets Using Aws-S3 or Aws-Sdk Gem in Ruby on Rails

How to copy file across buckets using aws-s3 or aws-sdk gem in ruby on rails

Using the right_aws gem:

# With s3 being an S3 object acquired via S3Interface.new
# Copies key1 from bucket b1 to key1_copy in bucket b2:
s3.copy('b1', 'key1', 'b2', 'key1_copy')

the gotcha I ran into is that if you have pics/1234/yourfile.jpg the bucket is only pics and the key is 1234/yourfile.jpg

I got the answer from here: How do I copy files between buckets using s3 from a rails application?

Copying a file inside an S3 bucket (ruby)

Extracted from the documentation, you'll have to use the copy_to. For instance:

s3 = AWS::S3.new

# Upload a file and set server-side encryption.
bucket1 = s3.buckets[source_bucket]
bucket2 = s3.buckets[target_bucket]
obj1 = bucket1.objects[source_key]
obj2 = bucket2.objects[target_key]

obj1.copy_to(obj2)

How do I only copy missing objects between buckets using ruby aws-sdk?

I never worked with that gem, but you code looks like it is possible to receive an array with all items store in a bucket. Load that list for both buckets and determine the missing files with a simple array operations. Should be much faster.

# load file lists (looks up objects in batches of 1000)
source_files  = AWS::S3.new.buckets[prod_bucket].objects.map(&:key)
target_files  = AWS::S3.new.buckets[dev_bucket].objects.map(&:key)

# determine files missing in dev
files_to_copy = source_files - target_files
files_to_copy.each_with_index do |file_name, i|
  puts "Coping #{i}/#{files_to_copy.size}: #{file_name}"

  S3Object.store(file_name, 
                 S3Object.value(file_name, PROD_BUCKET_NAME), 
                 DEV_BUCKET_NAME)
end

# determine files on dev that are not existing on prod
files_to_remove = target_files - source_files
files_to_remove.each_with_index do |file_name, i|
  puts "Removing #{i}/#{files_to_remove.size}: #{file_name}"

  S3Object.delete(file_name, DEV_BUCKET_NAME)
end

How to copy list of public S3 files to private S3 bucket

1) There's is no sdk feature for an 'internal copy' of public S3 objects to ones private S3 bucket.

2) the below source works, which keeps the same S3 directory structure

vid_file = 'http://example.com.s3.amazonaws.com/assets/videos/abc123.mp4'
vid_response = HTTParty.get(vid_file)

if vid_response.code == 200

  uri_path = URI(vid_url).path 
  uri_path.slice!(0) # slice!(0) removes leading slash, otherwise creates an empty s3 folder

  s3 = Aws::S3::Resource.new(region: ENV['AWS_REGION'])
  obj = s3.bucket(ENV['S3_BUCKET']).object(uri_path)
  obj.put(body: vid_response.body) if !obj.exists?
end

Copy file in a bucket to a paperclip attachment with aws-sdk

I'm copying the file using AWS::S3::S3Object#copy_to(target_obj) and I get target obj by calling #s3_object() on the attachment after setting some column values and saving it:

my_model.some_file_file_name = obj.key
my_model.some_file_file_size = obj.content_length
my_model.some_file_content_type = obj.content_type
my_model.some_file_updated_at = obj.last_modified
my_model.save

obj.copy_to(my_model.some_file.s3_object)

---

Related Topics