Copying One Azure Blob to Another Blob in Azure Storage Client 2.0

Copying one Azure blob to another blob in Azure Storage Client 2.0

Gaurav Mantri has written a series of articles on Azure Storage on version 2.0. I have taken this code extract from his blog post of Storage Client Library 2.0 – Migrating Blob Storage Code for Blob Copy

CloudStorageAccount storageAccount = new CloudStorageAccount(new StorageCredentials(accountName, accountKey), true);
CloudBlobClient cloudBlobClient = storageAccount.CreateCloudBlobClient();
CloudBlobContainer sourceContainer = cloudBlobClient.GetContainerReference(containerName);
CloudBlobContainer targetContainer = cloudBlobClient.GetContainerReference(targetContainerName);
string blobName = "<Blob Name e.g. myblob.txt>";
CloudBlockBlob sourceBlob = sourceContainer.GetBlockBlobReference(blobName);
CloudBlockBlob targetBlob = targetContainer.GetBlockBlobReference(blobName);
targetBlob.StartCopyFromBlob(sourceBlob);

Azure Storage move blob to other container

Here is what worked for me (answer edited after better answer by @Deumber was posted):

    public async Task<CloudBlockBlob> Move(CloudBlockBlob srcBlob, CloudBlobContainer destContainer)
    {
        CloudBlockBlob destBlob;

        if (srcBlob == null)
        {
            throw new Exception("Source blob cannot be null.");
        }

        if (!destContainer.Exists())
        {
            throw new Exception("Destination container does not exist.");
        }

        //Copy source blob to destination container
        string name = srcBlob.Uri.Segments.Last();
        destBlob = destContainer.GetBlockBlobReference(name);
        await destBlob.StartCopyAsync(srcBlob);
        //remove source blob after copy is done.
        srcBlob.Delete();
        return destBlob;
    }

How to copy an Azure File to an Azure Blob?

How to copy an Azure File to an Azure Blob?

We also can use CloudBlockBlob.StartCopy(CloudFile). CloudFile type is also can be accepted by the CloudBlockBlob.StartCopy function.
How to copy CloudFile to blob please refer to document. The following demo code is snippet from the document.

// Parse the connection string for the storage account.
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(
    Microsoft.Azure.CloudConfigurationManager.GetSetting("StorageConnectionString"));

// Create a CloudFileClient object for credentialed access to File storage.
CloudFileClient fileClient = storageAccount.CreateCloudFileClient();

// Create a new file share, if it does not already exist.
CloudFileShare share = fileClient.GetShareReference("sample-share");
share.CreateIfNotExists();

// Create a new file in the root directory.
CloudFile sourceFile = share.GetRootDirectoryReference().GetFileReference("sample-file.txt");
sourceFile.UploadText("A sample file in the root directory.");

// Get a reference to the blob to which the file will be copied.
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
CloudBlobContainer container = blobClient.GetContainerReference("sample-container");
container.CreateIfNotExists();
CloudBlockBlob destBlob = container.GetBlockBlobReference("sample-blob.txt");

// Create a SAS for the file that's valid for 24 hours.
// Note that when you are copying a file to a blob, or a blob to a file, you must use a SAS
// to authenticate access to the source object, even if you are copying within the same
// storage account.
string fileSas = sourceFile.GetSharedAccessSignature(new SharedAccessFilePolicy()
{
    // Only read permissions are required for the source file.
    Permissions = SharedAccessFilePermissions.Read,
    SharedAccessExpiryTime = DateTime.UtcNow.AddHours(24)
});

// Construct the URI to the source file, including the SAS token.
Uri fileSasUri = new Uri(sourceFile.StorageUri.PrimaryUri.ToString() + fileSas);

// Copy the file to the blob.
destBlob.StartCopy(fileSasUri);

Note:

If you are copying a blob to a file, or a file to a blob, you must use a shared access signature (SAS) to authenticate the source object, even if you are copying within the same storage account.

How to clone blob container and contents

Can I copy all of the blobs from live to test without downloading
them? They would need to maintain the same names.

Yes, you can. Copying blob is an asynchronous server-side operation. You simply tell the blob service the blobs to copy & destination details and it will do the job for you. No need to download first and upload them to destination.

How do I work out what it will cost to do this? The live blob storage
is roughly 130GB, but it should just be copying the data within the
same data centre right?

So there are 3 things you need to consider when it comes to costing: 1) Storage costs, 2) transaction costs and 3) data egress costs.

Since the copied blobs will be stored somewhere, they will be consuming storage and you will incur storage costs.

Copy operation will perform some read operations on source blobs and then write operation on destination blobs (to create them), so you will have to incur transaction costs. At very minimum for each blob copy, you can expect 2 transactions - read on source and write on destination (though there can be more transactions).

You incur data egress costs if the destination storage account is not in the same region as your source storage account. As long as both storage accounts are in the same region, you would not incur this.

You can use Azure Storage Pricing Calculator to get an idea about how much it is going to cost you.

I've also found AzCopy which looks promising but it looks like it
would copy the files one by one so I'm worried it would end up taking
a long time and costing a lot.

Blobs are always copied one-by-one. Copying across storage accounts is always async server side operation so you can't really predict how much time it would take for the copy operation to complete but in my experience it is quite fast. If you want to control when the blobs are copied, you would need to download them first and upload them. AzCopy supports this mode as well.

As far as costs are concerned, I think it is a relative term when you say it is going to cost a lot. But in general Azure Storage is very cheap and 130 GB is not a whole lot of data.

Transferring file between two Azure fileshares under the same Azure Storage account

Please try by removing the IP restrictions from your SAS token.

Copy blob operation is a server-side copy where storage service is copying the data.

Putting IP restrictions in your SAS token is preventing the storage service to read the data from the source.

Copying large amount of data on Azure Blob Storage

Please see Copy Blob REST API documentation for more information on this.

In short, when a Copy Blob request is received, it will be processed by the Blob service either synchronously or asynchronously. However, please note that the Blob service copies blobs on a best-effort basis.

When copying from a page blob, the Blob service creates a destination page blob of the source blob’s length, initially containing all zeroes. When copying from a block blob, the Blob service creates a committed blob of zero length before returning from this operation. The final blob will be committed when the copy completes.

Azure Blob Storage - Copy production to development

You will have to write a tiny CopyBlob program that takes the constraints and calls AzCopy with required params to copy over required blobs.

[Update]

You can use Azure Storage Library and StartCopyFromBlob method to copy blob. If you are not using .Net, you can find equivalent in azure storage library written in other languages as well.

This copy blob is asynchronous in nature meaning 'without having to retrieve the content', saves cost and time. More details on Azure Storage Blog.

This answer shows how to copy blob, just for completion.

Copying a Azure Blob and injecting in azure media services gives System.Data.Services.Client.DataServiceQueryException

You need to get the URL for the destinationFileBlob from your destinationLocator.

---

Related Topics