Gcsfuse Input/Output Error

gcsfuse Input/Output error

It appears from the Insufficient Permission errors in your debug output that gcsfuse doesn't have sufficient permissions to your bucket. Probably it has read-only access.

Be sure to read the credentials documentation for gcsfuse. In particular, if you're using a service account on a GCE VM make sure to set up the VM with the storage-full access scope.

Input/output error when writing to google cloud storage bucket

It appears from the --foreground --debug_fuse output that you're using credentials that aren't allowed to write to the bucket. They are probably read-only (StatObject didn't return a 403, and gcsfuse checks at startup that it can list the bucket).

Try giving the docs about credentials a careful read. In particular, if you're getting credentials automatically on a Google Compute Engine VM, you probably forgot to create it with the storage-full scope.

Google Cloud FUSE with Cloud Run [Errno 5] Input/output error: '/database' but already granted Storage Object Admin access

It's unclear to me why you're looking for /database.

Did you deploy by --update-env-vars=MNT_DIR=/database,BUCKET=...?

The app defaults to MNT_DIR=/mnt/gcs as this is set in gcsfuse.Dockerfile

Where does the [Errno 5] error originate?

I deployed the code as-is from the tutorial and it works for me.

It uses /mnt/gcs (surfaced through the Cloud Run endpoint url +/mnt/gcs.

Files are created upon refresh and these are visible in the GCS bucket.

Update

Here's my end-to-end script:

Q=70354313

BILLING="..." # Your Billing Account
PROJECT="$(whoami)-$(date +%y%m%d)-${Q}" # Or ...
BUCKET="$(whoami)-$(date +%y%m%d)-${Q}" # Or ...
REGION="us-west1" # Or ...

NAME="stackoverflow"

MNT_DIR="/database"

ACCOUNT="stackoverflow"
EMAIL="${ACCOUNT}@${PROJECT}.iam.gserviceaccount.com"

git clone https://github.com/GoogleCloudPlatform/python-docs-samples.git
cd python-docs-samples/run/filesystem/

# Use FUSE not Filestore
rm Dockerfile 
cp gcsfuse.Dockerfile Dockerfile

# GCP stuff
gcloud projects create ${PROJECT}

gcloud beta billing projects link ${PROJECT} \
--billing-account=${BILLING}

SERVICES=(
    "artifactregistry"
    "cloudbuild"
    "run"
)
for SERVICE in ${SERVICES[@]}
do
  gcloud services enable ${SERVICE}.googleapis.com \
  --project=${PROJECT}
done

# Create GCS Bucket
gsutil mb -l ${REGION} -p ${PROJECT} gs://${BUCKET}

# Create Service Account
gcloud iam service-accounts create ${ACCOUNT} \
--project=${PROJECT}

gcloud projects add-iam-policy-binding ${PROJECT} \
--member=serviceAccount:${EMAIL} \
--role=roles/storage.objectAdmin

gcloud beta run deploy ${NAME} \
--source=${PWD} \
--execution-environment=gen2 \
--allow-unauthenticated \
--service-account=${ACCOUNT} \
--update-env-vars=MT_DIR=${MNT_DIR},BUCKET=${BUCKET} \
--region=${REGION} \
--project=${PROJECT} 

Then:

ENDPOINT=$(\
  gcloud run services describe ${NAME} \
  --region=${REGION} \
  --platform=managed \
  --project=${PROJECT} \
  --format="value(status.url)") && echo ${ENDPOINT}

# Curl the Cloud Run service endpoint 5 times
for test in {1..5}
do
  curl \
  --silent \
  --location \
  --output /dev/null \
  --write-out "%{response_code}\n" \
  ${ENDPOINT}
  # Files are only differentiated at minute accuracy
  sleep 60s
done

# Enumerate the GCS Bucket
gsutil ls gs://${BUCKET}

GCSFuse Provided scope(s) are not authroized error

This error might be caused by the allow_other command. It is used to override the access permissions of FUSE.

As a security measure, fuse itself restricts file system access to the user who mounted the file system (cf. fuse.txt). For this reason, gcsfuse by default shows all files as owned by the invoking user. Therefore, you should invoke gcsfuse as the user that will be using the file system, not as the root.

If you know what you are doing, you can override these behaviors with the allow_other mount option supported by fuse and with the --uid and --gid flags supported by gcsfuse. Be careful, this may have security implications!

If you read the description of the command you will find:

allow_other

This option overrides the security measure restricting file access
to the user mounting the filesystem. This option is by default only
allowed to the root, but this restriction can be removed with a
(userspace) configuration option.

I think you are getting an out of scope message either because you have not specified the --uid (user id) and the --gid (group id) flags along with the allow_other command or because you have not set up a userspace yet to properly override root permissions with this command If you do not need to use root access, I would simply remove this part of the command and try again.

Another possible cause I found in a question related to the error 403 was that the bucket or the objects inside of it did not have the correct Access Control system. You mentioned you are using public access Google Cloud Storage, but please consider that your bucket might not have the correct access permissions as well.

OSError input/output error when writing to mounted filesystem

I found the cause. I enabled the debug options as others suggested. Thank you. Unfortunately the logs did not tell me much. Fortunately I had one more VM and I tested there and it worked fine (mount + read/write to bucket). I understood its not the permissions on the bucket side. The VM was created with "Allow default access" which gives readonly to "Storage". I granted read/write to store. One can specify also "Allow full access to all Cloud APIs" although its better not to grant all permissions if you do not need them.

What happens when network connection to GCP is lost?

Based on the repository documentation for gcsfuse, file upload retries are already built into the utility, and they happen when there are problems accessing the storage bucket that is mounted. You are able to modify the maximum backoff for retries by using the --max-retry-sleep flag. This flag controls the maximum time that can be reached between retries, after which retrying stops. The flag accepts an X amount of minutes as input.

This doc page is also relevant if you would like to know more about specific characteristics of gcsfuse.

---

Related Topics