How to Use Aws Cli to Only Copy Files in S3 Bucket That Match a Given String Pattern

How to use AWS CLI to only copy files in S3 bucket that match a given string pattern

The alternatives that you have listed are the best options because S3 CLI doesn't support regex.

Use of Exclude and Include Filters:

Currently, there is no support for the use of UNIX style wildcards in
a command's path arguments. However, most commands have --exclude
"" and --include "" parameters that can achieve the
desired result. These parameters perform pattern matching to either
exclude or include a particular file or object. The following pattern
symbols are supported.

*: Matches everything
?: Matches any single character
[sequence]: Matches any character in sequence
[!sequence]: Matches any character not in sequence

AWS CLI command to copy specific files from S3 to local or hdfs

You could try to use AWC Cli:

    aws s3 cp s3://bucketname/ ~/ --exclude "*" --include "abc2018-*" --recursive

This command copy only the pattern you want.
Enjoy!

how to get list of files on aws bucket with a specific pattern

Sadly you can't do this with s3 ls. But you could possibly use Exclude and Include Filters along with --dryrun for commands that support filters. For example, for s3 cp:

aws s3 cp s3://my_bucket ./. --recursive --dryrun --exclude "*" --include "my*.txt"

--dryrun (boolean) Displays the operations that would be performed using the specified command without actually running them.

This should print out all the objects that would be normally copied without --dryrun option.

Since you have a large bucket, you can test it out on small bucket just to get the feel of the command as its output is different then from s3 ls.

How to upload files matching a pattern with aws cli to s3

AWS s3 cli doesn't support regex, but there is an exclude and include for s3.
So you should be able to use:

aws s3 cp /var/log/ s3://test/dom0/ --recursive --exclude "*" --include "console.*"

Note the order of the exclude and include, if you switch them around then nothing will be uploaded. You can include more patterns by adding more includes.

AWS-s3 : How to copy files from s3 bucket to another s3 bucket based on filename

I guess your code written in python, you just need to handle the bucket object as string.

my_bucket = s3.Bucket('s3-dev')

for my_bucket_object in my_bucket.objects.all():
    file_name =  my_bucket_object.key.split('/')[-1] # avoid the folder contains the pattern
    if file_name.startswith('abc'):
        print(my_bucket_object.key)

How can I use wildcards to `cp` a group of files with the AWS CLI

To download multiple files from an aws bucket to your current directory, you can use recursive, exclude, and include flags.
The order of the parameters matters.

Example command:

aws s3 cp s3://data/ . --recursive --exclude "*" --include "2016-08*"

For more info on how to use these filters: http://docs.aws.amazon.com/cli/latest/reference/s3/#use-of-exclude-and-include-filters

How to copy subset of files from one S3 bucket folder to another by date

There is no aws-cli command that will do this for you in a single line. If the number of files is relatively small, say a hundred thousands or fewer I think it would be easiest to write a bash script, or use your favourite language's AWS SDK, that lists the first folder, filters on creation date and issues the copy commands.

If the number of files is large you can create an S3 Inventory that will give you a listing of all the files in the bucket, which you can download and generate the copy commands from. This will be cheaper and quicker than listing when there are lots and lots of files.

---

Something like this could be a start, using @jarmod's suggestion about --copy-source-if-modified-since:

for key in $(aws s3api list-objects --bucket my-bucket --prefix folder1/ --query 'Contents[].Key' --output text); do
  relative_key=${key/folder1/folder2}
  aws s3api copy-object --bucket my-bucket --key "$relative_key" --source-object "my-bucket/$key" --copy-source-if-modified-since THE_CUTOFF_DATE
done

It will copy each object individually, and it will be fairly slow if there are lots of objects, but it's at least somewhere to start.

---

Related Topics