Linux Command to Replace String in Large File with Another String

Linux command to replace string in LARGE file with another string

sed is a good choice for large files.

sed -i.bak -e 's%C://temp%//home//some//blah%' large_file.sql

It is a good choice because doesn't read the whole file at once to change it. Quoting the manual:

A stream editor is used to perform
basic text transformations on an input
stream (a file or input from a
pipeline). While in some ways similar
to an editor which permits scripted
edits (such as ed), sed works by
making only one pass over the
input(s), and is consequently more
efficient. But it is sed's ability to
filter text in a pipeline which
particularly distinguishes it from
other types of editors.

The relevant manual section is here. A small explanation follows

-i.bak enables in place editing leaving a backup copy with .bak extension

s%foo%bar% uses s, the substitution command, which
substitutes matches of first string
in between the % sign, 'foo', for the second
string, 'bar'. It's usually written as s//
but because your strings have plenty
of slashes, it's more convenient to
change them for something else so you
avoid having to escape them.

Example


vinko@mithril:~$ sed -i.bak -e 's%C://temp%//home//some//blah%' a.txt
vinko@mithril:~$ more a.txt
//home//some//blah
D://temp
//home//some//blah
D://temp
vinko@mithril:~$ more a.txt.bak
C://temp
D://temp
C://temp
D://temp

How to replace a string in multiple files in linux command line


cd /path/to/your/folder
sed -i 's/foo/bar/g' *

Occurrences of "foo" will be replaced with "bar".

On BSD systems like macOS, you need to provide a backup extension like -i '.bak' or else "risk corruption or partial content" per the manpage.

cd /path/to/your/folder
sed -i '.bak' 's/foo/bar/g' *

Linux command to replace string in HUGE file with another string

The problem is that in the event of a replacement, all programs will make a copy of the file with the substitution in place in order to replace the original file ultimately -- they don't want to risk losing the original for obvious reasons.

With perl, you can do this in a one-liner, but that doesn't make it any shorter (well, it probably does compared to vim, since vim preserves history in yet another file, which perl doesn't):

perl -pi -e 's,\bLATIN1\b,UTF-8,g if $. <= 30' thefile

Find and replace text in a 47GB large file

Sed (stream editor for filtering and transforming text) is your friend.

sed -i 's/old text/new text/g' file

Sed performs text transformations in a single pass.

Replacing a String from all the files with the value from text file

Suggesting to feed result from grep command to sed command:

  sed -i "s/replace-me/$(cat $GIRL/abcd3.txt)/g" $( grep -rl 'replace-me' $GIRL )


Find and Replace Inside a Text File from a Bash Command

The easiest way is to use sed (or perl):

sed -i -e 's/abc/XYZ/g' /tmp/file.txt

Which will invoke sed to do an in-place edit due to the -i option. This can be called from bash.

If you really really want to use just bash, then the following can work:

while IFS='' read -r a; do
echo "${a//abc/XYZ}"
done < /tmp/file.txt > /tmp/file.txt.t
mv /tmp/file.txt{.t,}

This loops over each line, doing a substitution, and writing to a temporary file (don't want to clobber the input). The move at the end just moves temporary to the original name. (For robustness and security, the temporary file name should not be static or predictable, but let's not go there.)

For Mac users:

sed -i '' 's/abc/XYZ/g' /tmp/file.txt

(See the comment below why)

Sed replace first occurence in place with big files

You can edit up to the first match using sed:

sed -e '1,/pattern/{s/pattern/replace/;}'

On lines 1 to N-1 (where line N contains the pattern), the substitution does nothing; on line N, it does the real work. Thereafter, you're no longer in the 1,/pattern/ range of lines so there is no further transformation.

Note that this doesn't work if line 1 matches the pattern; it then makes changes in line 1 and the next line that matches the pattern. With GNU sed at least, you can change the 1 to 0 and that works OK.

printf "%s\n" pattern pattern pattern pattern |
sed -e '0,/pattern/{s/pattern/replace/;}'

However, the description says "in the first 100 lines" and while line 1 is in the first 100 lines, that isn't the way you'd normally describe it when it appears on line 1.

You can add a -i option to overwrite the original file once you've tested it. Beware: not all versions of sed support -i and on Mac OS X, the backup suffix is mandatory -i.bak (but can be empty: use -i ''). By contrast, GNU sed has an optional suffix which must be attached to the -i option. Hence, -i.bak works with both GNU and Mac (BSD) sed; other uses of the -i option are specific to the variant of sed you're using.



Related Topics



Leave a reply



Submit