How to Mount Google Bucket as Local Disk on Linux Instance with Full Access Rights

How to mount Google Bucket as local disk on Linux instance with full access rights

I saw your question because I was having exactly the same problem and I also did the same steps as you.
The solution to give user root full control of the mounted cloud folder :

You have to go to your Google Cloude place, search for "Service account" and clic on it.

Then you have to export the key file of your service account (.json)
(I have created a new service account with the Google Cloud Shell consol using this command : gcloud auth application-default login
And then followed the steps when I was prompted by the shell.)

Clic on Create Key and choose JSON

Upload the .JSON keyfile to your linux server.
Then on your Linux server, run this command : gcsfuse -o allow_other --gid 0 --uid 0 --file-mode 777 --dir-mode 777 --key-file /path_to_your_keyFile_that_you_just_uploaded.json nameOfYourBucket /path/to/mount

To find your root user GID & UID, login in root to your server and in terminal type : id -u root for UID & id -g root for GID

Hope I helped, because I've been struggling for long and no resource on internet really helped. Cheers.

GCP: how to access cloud storage bucket from a VM instance

Google Cloud Storage is a object storage API, it is not a filesystem. As a result, it isn't really designed to be "mounted" within a VM. It is designed to be highly durable and scalable to extraordinarily large objects (and large numbers of objects).

Though you can use gcsfuse to mount it as a filesystem, that method has pretty significant drawbacks. For example, it can be expensive in operation count to do even simple operations for a normal filesystem.

Likewise, there are many surprising behaviors that are a result of the fact that it is an object store. For example, you can't edit objects -- they are immutable. To give the illusion of writing to the middle of an object, the object is, in effect, deleted and recreated whenever a call to close() or fsync() happens.

The best way to use GCS is to design your application to use the API (or the S3 compatible API) directly. That way the semantics are well understood by the application, and you can optimize for them to get better performance and control your costs. Thus, to access it from your docker container, ensure your container has a way to authenticate through GCS (either through the credentials on the instance, or by deploying a key for a service account with the necessary permissions to access the bucket), then have the application call the API directly.

Finally, if what you need is actually a filesystem, and not specifically GCS, Google Cloud does offer at least 2 other options if you need a large mountable filesystem that is designed for that specific use case:

• Persistent Disk, which is the baseline filesystem that you get with a VM, but you can mount many of these devices on a single VM. However, you can't mount them read/write to multiple VMs at once -- if you need to mount to multiple VMs, the persistent disk must be read only for all instances they are mounted to.
• Cloud Filestore is a managed service that provides an NFS server in front of a persistent disk. Thus, the filesystem can be mounted read/write and shared across many VMs. However it is significantly more expensive (as of this writing, about $0.20/GB/month vs $0.04/GB/month in us-central1) than PD, and there are minimum size requirements (1TB).

Where Are the Files in My Google Cloud Storage Bucket?

I also had this issue. The answer @jacobsa addressed the same issue for me.

Instead of mounting with

gcsfuse my-bucket /path/to/mount

I used

gcsfuse --implicit-dirs my-bucket /path/to/mount

I also checked that the bucket is setup with the right permissions by adding myself as the Owner in Edit bucket permissions

Is there a way to use a google cloud storage bucket file as a local file?

On a VM or on your workstation, you can use GCSFUSE (on linux/mac only) to mount a GCS bucket in a directory and interact with GCS as a local directory (be careful, you need to keep in mind that the system calls are wrapped in API calls and therefore the performance, and the cost aren't the same as a local storage).

If you use serverless product, App Engine, Cloud Functions or Cloud Run (for now), these product are stateless and you can't mount partition on it. The only way to achieve that is to download the file in the /tmp directory (in memory file system), and then to perform the processing with the locally downloaded file.

Be careful, because it's in memory file system, if you don't clean manually the /tmp directory, you can run with out of memory issue. In addition, when the instance restart or scale up, you will lost the downloaded files; because it's only in memory and not persistent.

Allow Google Cloud Compute Engine Instance to write file to Google Storage Bucket - Python

I understand that you are using gcfuse on your Linux VM Instance to access Google Cloud Storage.

Key file is a Service Account credentials key, that will allow you to initiate Cloud SDK or Client Library as another Service Account. You can download key file from Cloud Console. However, if you are using VM Instance, you are automatically using Compute Engine Default Service Account. You can check it using console command: $ gcloud init.

To configure properly your credentials, please follow the documentation.

Compute Engine Default Service Account, need to have enabled Access Scope Storage > Full. Access Scope is the mechanism that limits access level to Cloud APIs. That can be done during machine creation or when VM Instance is stopped.

Please note that Access Scopes are defined explicitly for the Service Account that you select for VM Instance.

Cloud Storage objects names have requirements. It is strongly recommended avoid using hash symbol "#" in the names of the objects.

---

Related Topics