Sed - Pass Match to External Command

sed - pass match to external command

if your sed is GNU sed. you can use 'e' to pass matched group to external command/tools within sed command.

an example might be helpful to make it clear:

say, you have a problem:

you have a string "120+20foobar" now you want to get the calculation result of 120+20 part, and replace "oo" to "xx" in "foobar" part.

Note that this example is not for solving the problem above, just for
showing the sed 'e' usage

so you could make 120+20 in the first match group, and rest in 2nd group, then pass two groups to different command/tools and then get the result. like:

   kent$  echo "100+20foobar"|sed -r 's#([0-9+]*)(.*)#echo  \1 \|bc\;echo \2 \| sed "s/oo/xx/g"#ge'
    120
    fxxbar

in this way, you could nest many seds one in another one, till you get lost. :D

How do I push `sed` matches to the shell call in the replacement pattern?

So you are trying to call an external command from inside the replacement pattern of a sed substitution. I dont' think it can be done, the $... inside a pattern just allows you to use an already existent (constant) shell variable.

I'd go with Perl, see the /e option in the search-replace operator (s/.../.../e).

UPDATE: I was wrong, sed plays nicely with the shell, and it allows you do to that. But, then, the backlash in \1 should be escaped. Try instead:

sed "s/^URL=\(.*\)/TITLE=$(curl -s \\1 | head -n 1)/" file.txt

How to find/replace and increment a matched number with sed/awk?

I think finding file isn't the difficult part for you. I therefore just go to the point, to do the +1 calculation. If you have gnu sed, it could be done in this way:

sed -r 's/(.*)(\?cache_version=)([0-9]+)(.*)/echo "\1\2$((\3+1))\4"/ge' file

let's take an example:

kent$  cat test 
ello
barbaz?cache_version=3fooooo
bye

kent$  sed -r 's/(.*)(\?cache_version=)([0-9]+)(.*)/echo "\1\2$((\3+1))\4"/ge' test     
ello                                                                             
barbaz?cache_version=4fooooo
bye

you could add -i option if you like.

edit

/e allows you to pass matched part to external command, and do substitution with the execution result. Gnu sed only.

see this example: external command/tool echo, bc are used

kent$  echo "result:3*3"|sed -r 's/(result:)(.*)/echo \1$(echo "\2"\|bc)/ge'       

gives output:

result:9

you could use other powerful external command, like cut, sed (again), awk...

Match a line and return a previous line before the match containing a pattern

This sed command should extract it:

sed -n '
    /^Id: / {                 # If the line starts with "Id: "
        s///                  # Remove the "Id: "
        h                     # Store what is left in the hold space
    }
    /^  email: '"$email"'/ {  # If the line starts with "  email: " plus the email
        x                     # Swap pattern and hold space
        p                     # Print pattern space
        q                     # Stop processing
    }
' infile

where $email is the shell variable containing the escaped version of test_1@somedomain.com:

raw='test_1@somedomain.com'
email=$(sed 's|[]/.*^$\[]|\\&|g' <<< "$raw")

This escapes the sed special characters .*/^$[]\.

Or, more compact:

sed -n '/^Id: /{s///;h};/^  email: '"$email"'/{x;p;q}' infile

macOS sed requires an extra ; before each closing }.

And yes, it's probably easier with awk /p>

Sed replace with the output of a bash command using capture group as argument

You can use e in GNU sed to pass the substitution string to a shell for evaluation. This way, you can say:

printf "%s %s" "something" "\1"

Where \1 holds a captured group. All together:

$ sed -r 's#match_([0-9]*).*#printf "%s %s" "something" "\1"#e' <<< "match_555 hello"
something 555

This comes handy when you want to perform some shell action with a captured group, like in this case.

So, let's capture the first part of the line, then the part that needs to be encoded and finally the rest. Once this is done, let's print those pieces back with printf triggering the usage of base64 -d against the second slice:

$ sed -r '/^Base64/s#(.*;)([^\&]*)(&.*)# printf "%s%s%s" "\1" $(echo "\2" | base64 -d) "\3";#e' file
someline
someline
Base64Expression stringValue="foo"
someline
Base64Expression stringValue="bar"

Step by step:

sed -r '/^Base64/s#(.*;)([^\&]*)(&.*)# printf "%s%s%s" "\1" $(echo "\2" | base64 -d) "\3";#e' file
#        ^^^^^^^    ^^^  ^^^^^^  ^^^                        ^^^^^^^^^^^^^^^^^^^^^^^^       ^
#           |   first part  |   the rest                encode the 2nd captured group      |
#           |               |                                                              |
#           |           important part                                      execute the command
#           |
# on lines starting with Base64, do...

The idea comes from this superb answer by anubhava on How to change date format in sed?.

Trying to change date string in a file to different format with sed

In your example, you use simple quote to protect the sed expression, that's why you CANNOT use other simple quote inside (there is no escape in this case).

If your sed is GNU sed. you can use 'e' to pass matched group to external command/tools within sed command as explained here.

But it means you call an external command for each match. So this solution can be slow if your text contains a lot of dates.

If you have only one date by line, I suggest the following script :

sed -r 's#([0-9]{1,2})-([0-9]{1,2})-([0-9]{4})#date -d"\1/\2/\3" +"%B %d, %Y"#e'

Multiple mathematical operations on a file containing numbers

For calculations, use awk!

$ awk '{$(NF-1)=sprintf("%.0f", ($(NF-1) + $NF)/2 * 141); NF--}1' file
AJ29 IO_0_VRN_10 10945
AJ30 IO_L1P_T0_100M 12738
AJ31 IO_L1N_T0_100S 14367
AK29 IO_L2P_T0_101M 9376
AL29 IO_L2N_T0_101S 9016

This replaces the penultimate field with the result of (penultimate*last)/2 * 141). To make it round, we use %.0f format as indicated in Awk printf number in width and round it up.

Also, it looks to me that you are piping way too many things: I counted one call to grep and 13 (!) to sed. You can probably use sed -e 'first block' -e 'second block' ... instead.

Explanation

In awk, NF refers to the number of fields on the current line. Since $n refers to the field number n, with $(NF-1) we refer to the penultimate field.

• {...}1 do stuff and then print the resulting line. 1 evaluates as True and anything True triggers awk to perform its default action, which is to print the current line.
• $(NF-1) + $NF)/2 * 141 perform the calculation: `(penultimate + last) / 2 * 141
• {$(NF-1)=sprintf( ... ) assign the result of the previous calculation to the penultimate field. Using sprintf with %.0f we make sure the rounding is performed, as described above.
• {...; NF--} once the calculation is done, we have its result in the penultimate field. To remove the last column, we just say "hey, decrease the number of fields" so that the last one gets "removed".

---

Related Topics