How to Limit File Size on Commit

How to limit file size on commit?

This pre-commit hook will do the file size check:

.git/hooks/pre-commit

#!/bin/sh
hard_limit=$(git config hooks.filesizehardlimit)
soft_limit=$(git config hooks.filesizesoftlimit)
: ${hard_limit:=10000000}
: ${soft_limit:=500000}

list_new_or_modified_files()
{
git diff --staged --name-status|sed -e '/^D/ d; /^D/! s/.\s\+//'
}

unmunge()
{
local result="${1#\"}"
result="${result%\"}"
env echo -e "$result"
}

check_file_size()
{
n=0
while read -r munged_filename
do
f="$(unmunge "$munged_filename")"
h=$(git ls-files -s "$f"|cut -d' ' -f 2)
s=$(git cat-file -s "$h")
if [ "$s" -gt $hard_limit ]
then
env echo -E 1>&2 "ERROR: hard size limit ($hard_limit) exceeded: $munged_filename ($s)"
n=$((n+1))
elif [ "$s" -gt $soft_limit ]
then
env echo -E 1>&2 "WARNING: soft size limit ($soft_limit) exceeded: $munged_filename ($s)"
fi
done

[ $n -eq 0 ]
}

list_new_or_modified_files | check_file_size

Above script must be saved as .git/hooks/pre-commit with execution permissions enabled (chmod +x .git/hooks/pre-commit).

The default soft (warning) and hard (error) size limits are set to 500,000 and 10,000,000 bytes but can be overriden through the hooks.filesizesoftlimit and hooks.filesizehardlimit settings respectively:

$ git config hooks.filesizesoftlimit 100000
$ git config hooks.filesizehardlimit 4000000

How can I limit file size to 50MB in GitLab?

There are few ways this can be configured to prevent large files from being committed:

Push Rules (Premium)

If you have a Premium or Ultimate subscription, you can configure push rules that prevent committing files over a certain size. These are enforced as pre-commit hooks and does not affect files pushed using LFS. You can apply this at the project or group level.

Go to -> settings -> repository -> push rules here you can specify maximum file size (MB) to set this limit

max file size

For self-managed instances, this can also be managed instance-wide under Admin Area -> Push Rules.

Instance limit settings (self-managed only)

Administrators of self-managed GitLab instances can also set limits on an entire GitLab instance. To do this effectively, two settings must be configured in the admin area:

  1. Max push size this limits the size of files that can be pushed using git
  2. For files added through the web UI, the max attachment size setting must also be greater than or equal to the max push size.

Similar to push rules, this does not apply to files added through LFS.

In all cases, file sizes are, of course, also limited by global storage limits for projects, which includes all git storage, LFS, artifacts, etc.

Alternative: client-side hooks

As an alternative to settings enforced by gitlab, users can also configure local git hooks to prevent committing large files. Of course, this requires users to set this up correctly for each repository locally and isn't enforced by GitLab.

How to decrease maximum file size on Git or Github repo?

This cannot be done on GitHub itself (over which you have no control)

You would need to do that locally, with this (for example) pre-commit hook, in order to check the size of the files part of your commit.

But that means it is a local-only workaround, that needs to be applied on each machine you or your colleagues are working on.

That differs from a private Git repo hosting server, where you can easily limit the size on the listener (an HTTPS one like NGiNX for example, with client_max_body_size )

Limiting file size in git repository

As I was struggling with it for a while, even with the description, and I think this is relevant for others too, I thought I'd post an implementation of how what J16 SDiZ described could be implemented.

So, my take on the server-side update hook preventing too big files to be pushed:

#!/bin/bash

# Script to limit the size of a push to git repository.
# Git repo has issues with big pushes, and we shouldn't have a real need for those
#
# eis/02.02.2012

# --- Safety check, should not be run from command line
if [ -z "$GIT_DIR" ]; then
echo "Don't run this script from the command line." >&2
echo " (if you want, you could supply GIT_DIR then run" >&2
echo " $0 <ref> <oldrev> <newrev>)" >&2
exit 1
fi

# Test that tab replacement works, issue in some Solaris envs at least
testvariable=`echo -e "\t" | sed 's/\s//'`
if [ "$testvariable" != "" ]; then
echo "Environment check failed - please contact git hosting." >&2
exit 1
fi


# File size limit is meant to be configured through 'hooks.filesizelimit' setting
filesizelimit=$(git config hooks.filesizelimit)

# If we haven't configured a file size limit, use default value of about 100M
if [ -z "$filesizelimit" ]; then
filesizelimit=100000000
fi

# Reference to incoming checkin can be found at $3
refname=$3

# With this command, we can find information about the file coming in that has biggest size
# We also normalize the line for excess whitespace
biggest_checkin_normalized=$(git ls-tree --full-tree -r -l $refname | sort -k 4 -n -r | head -1 | sed 's/^ *//;s/ *$//;s/\s\{1,\}/ /g' )

# Based on that, we can find what we are interested about
filesize=`echo $biggest_checkin_normalized | cut -d ' ' -f4,4`

# Actual comparison
# To cancel a push, we exit with status code 1
# It is also a good idea to print out some info about the cause of rejection
if [ $filesize -gt $filesizelimit ]; then

# To be more user-friendly, we also look up the name of the offending file
filename=`echo $biggest_checkin_normalized | cut -d ' ' -f5,5`

echo "Error: Too large push attempted." >&2
echo >&2
echo "File size limit is $filesizelimit, and you tried to push file named $filename of size $filesize." >&2
echo "Contact configuration team if you really need to do this." >&2
exit 1
fi

exit 0

Note that it's been commented that this code only checks the latest commit, so this code would need to be tweaked to iterate commits between $2 and $3 and do the check to all of them.

File size limit of 100.00 in previous commit

One way to fix the problem is an interactive rebase (see here for documentation):

git rebase -i Commit3

You'll get an editable text file that looks like this:

pick f0ce53a Add a big pesky zip file and some other stuff
pick Commit2 Misc. changes
pick Commit1 Misc. changes

There will be some helpful commented lines at the bottom of the file listing the various possible commands. The ones we care about are pick, which means the commit should be used as is, and edit, which will stop the rebase at the given commit to allow us to amend it.

f0ce53a is the commit where the zip file was added so we'll change pick to edit on that line, leaving us with:

edit f0ce53a Add a big pesky zip file and some other stuff
pick Commit2 Misc. changes
pick Commit1 Misc. changes

Save the file and quit (vim: :x), and you'll be on the problem commit. Remove the zip file and amend the commit, then continue the rebase:

git rm foo.zip
git commit --amend
git rebase --continue

You've rewritten history as though the zip file was never committed.

Github file size limit changed 6/18/13. Can't push now

As rlb.usa noted, Github has added a file size limit that prevents you from pushing files > 100MB. You tried to remove the file in a new commit and tried to push that. That fails, because you are not just pushing that last commit, but also three others. Those three commits contain versions of cron_log that are 141MB and 126MB in size. They cause your push to fail.

To fix that, you have two options:

  • Run git rebase -i origin/master, set every commit to edit and remove the file in each with git commit --amend.
  • Use the BFG Repo-Cleaner to clean all your history.


Related Topics



Leave a reply



Submit