Linux command to replace string in LARGE file with another string
sed is a good choice for large files.
sed -i.bak -e 's%C://temp%//home//some//blah%' large_file.sql
It is a good choice because doesn't read the whole file at once to change it. Quoting the manual:
A stream editor is used to perform
basic text transformations on an input
stream (a file or input from a
pipeline). While in some ways similar
to an editor which permits scripted
edits (such as ed), sed works by
making only one pass over the
input(s), and is consequently more
efficient. But it is sed's ability to
filter text in a pipeline which
particularly distinguishes it from
other types of editors.
The relevant manual section is here. A small explanation follows
-i.bak enables in place editing leaving a backup copy with .bak extension
s%foo%bar% uses s, the substitution command, which
substitutes matches of first string
in between the % sign, 'foo', for the second
string, 'bar'. It's usually written as s//
but because your strings have plenty
of slashes, it's more convenient to
change them for something else so you
avoid having to escape them.
Example
vinko@mithril:~$ sed -i.bak -e 's%C://temp%//home//some//blah%' a.txt
vinko@mithril:~$ more a.txt
//home//some//blah
D://temp
//home//some//blah
D://temp
vinko@mithril:~$ more a.txt.bak
C://temp
D://temp
C://temp
D://temp
How to replace a string in multiple files in linux command line
cd /path/to/your/folder
sed -i 's/foo/bar/g' *
Occurrences of "foo" will be replaced with "bar".
On BSD systems like macOS, you need to provide a backup extension like -i '.bak'
or else "risk corruption or partial content" per the manpage.
cd /path/to/your/folder
sed -i '.bak' 's/foo/bar/g' *
Linux command to replace string in HUGE file with another string
The problem is that in the event of a replacement, all programs will make a copy of the file with the substitution in place in order to replace the original file ultimately -- they don't want to risk losing the original for obvious reasons.
With perl, you can do this in a one-liner, but that doesn't make it any shorter (well, it probably does compared to vim, since vim preserves history in yet another file, which perl doesn't):
perl -pi -e 's,\bLATIN1\b,UTF-8,g if $. <= 30' thefile
Find and replace text in a 47GB large file
Sed (stream editor for filtering and transforming text) is your friend.
sed -i 's/old text/new text/g' file
Sed performs text transformations in a single pass.
Replacing a String from all the files with the value from text file
Suggesting to feed result from grep
command to sed
command:
sed -i "s/replace-me/$(cat $GIRL/abcd3.txt)/g" $( grep -rl 'replace-me' $GIRL )
Find and Replace Inside a Text File from a Bash Command
The easiest way is to use sed (or perl):
sed -i -e 's/abc/XYZ/g' /tmp/file.txt
Which will invoke sed to do an in-place edit due to the -i
option. This can be called from bash.
If you really really want to use just bash, then the following can work:
while IFS='' read -r a; do
echo "${a//abc/XYZ}"
done < /tmp/file.txt > /tmp/file.txt.t
mv /tmp/file.txt{.t,}
This loops over each line, doing a substitution, and writing to a temporary file (don't want to clobber the input). The move at the end just moves temporary to the original name. (For robustness and security, the temporary file name should not be static or predictable, but let's not go there.)
For Mac users:
sed -i '' 's/abc/XYZ/g' /tmp/file.txt
(See the comment below why)
Sed replace first occurence in place with big files
You can edit up to the first match using sed
:
sed -e '1,/pattern/{s/pattern/replace/;}'
On lines 1 to N-1 (where line N contains the pattern), the substitution does nothing; on line N, it does the real work. Thereafter, you're no longer in the 1,/pattern/
range of lines so there is no further transformation.
Note that this doesn't work if line 1 matches the pattern; it then makes changes in line 1 and the next line that matches the pattern. With GNU sed
at least, you can change the 1
to 0
and that works OK.
printf "%s\n" pattern pattern pattern pattern |
sed -e '0,/pattern/{s/pattern/replace/;}'
However, the description says "in the first 100 lines" and while line 1 is in the first 100 lines, that isn't the way you'd normally describe it when it appears on line 1.
You can add a -i
option to overwrite the original file once you've tested it. Beware: not all versions of sed
support -i
and on Mac OS X, the backup suffix is mandatory -i.bak
(but can be empty: use -i ''
). By contrast, GNU sed
has an optional suffix which must be attached to the -i
option. Hence, -i.bak
works with both GNU and Mac (BSD) sed
; other uses of the -i
option are specific to the variant of sed
you're using.
Related Topics
Shared Library Bottleneck on Numa MAChine
What Happens When a Signal Is Received While Already in a Signal Handler
Set Static Ip If Not Obtained from Dhcp (Script)
Jmeter - Could Not Find the Testplan Class
Add Some Specific Time While Using the Linux Command "Date"
Auto Exit Telnet Command Back to Prompt Without Human Intervention ^] Quit Close Exit Code 1
How to Set Up My Linux X Terminal So That Emacs Has Access to 256 Colors
Maven: Bash Mvn Permission Denied
I Can't Execute Command Modprobe Vboxdrv
Shell Script to Copy and Prepend Folder Name to Files from Multiple Subdirectories
Bash Scripting - Read Single Keystroke Including Special Keys Enter and Space
Cannot --Enable-Pcregrep-Libbz2 Because Bzlib.H Was Not Found
Go Http Server Testing Ab VS Wrk So Much Difference in Result
Open O_Creat | O_Excl on Nfs in Linux
How Are Percpu Pointers Implemented in the Linux Kernel
Error: Ld.So: Object 'Libgtk3-Nocsd.So.0' from Ld_Preload Cannot Be Preloaded