🐧 Linux Notes

Sed - Stream EDitor

Aug 21, 20244 min read

sources

-i update source file

Pattern building

  • \x... hexadecimal caracter.
    • \xC2\xA0 → non breaking space
    • \x0D → ^M
    • \x61 → a
  • .\{5\} 5 caracters

Pattern substitution

Basic substitutions

  • sed "s/foo/bar/" replaces only 1st instance in a line
  • sed -n "s/foo/bar/p" replaces only 1st instance in a line, and only print replaced lines
  • sed "s/foo/bar/4" replaces only the 4th instance in a line
  • sed "s/foo/bar/g" replaces ALL instances in a line
  • sed "s/foo/bar/3g" replaces all instances, starting the replacements with the third Note: several options can be combined, ie sed "s/foo/bar/gi"

trim

  • sed "s/^[ \t]*//" Delete leading whitespace (spaces, tabs) from front of each line
  • sed "s/[ \t]*$//" Delete trailing whitespace (spaces, tabs) from end of each line
  • sed "s/^[ \t]*;s/[ \t]*$" Delete BOTH leading and trailing whitespace from each line

Advanced substitutions

  • sed "/baz/s/foo/bar/g" substitute “foo” with “bar” ONLY for lines which contain “baz”
  • sed "/baz/!s/foo/bar/g" substitute “foo” with “bar” EXCEPT for lines which contain “baz”

Print or delete full line

  • -n suppress automatic printing of pattern space

p = print

  • sed -n "3p" print third line
  • sed -n "$p" print last line
  • sed -n "3,8p" print lines between 3 and 8 (inclusive)
  • sed -n "3,$p" print everything, starting line 3
  • sed -n "/PATTERN/p" print lines matching pattern (grep equivalent)
  • sed -n "/PATTERN/!p" print lines NOT matching pattern (grep -v equivalent)
  • sed -n "/PATTERN/,5p" print lines between pattern and 5th line of the input
  • sed -n "/PATTERN/,+5p" print line matching pattern, and 5 more lines (6 lines in total)
  • sed -n "/PATTERN1/,/PATTERN2/p" prints between tokens (lines matching tokens are printed)

d = delete

  • sed "1d" kill first line
  • sed "3,8!d" print lines between 3 and 8 (inclusive)
  • sed "$!d" remove everything but last line (print last line)
  • sed "/./,$!d" preserve lines between the first non empty one and the end of the file (delete all leading blank lines at top of file)
  • sed "/PATTERN/d" remove line matching pattern (grep -v equivalent)
  • sed "/PATTERN/!d" remove everything BUT line matching pattern (grep equivalent)
  • sed "/PATTERN1/!d; /PATTERN1/!d; /PATTERN1/!d" remove everything BUT line matching ONE or more patterns
  • sed "/token1/,/token2/d" remove between tokens (lines matching tokens are removed)

q = quit

  • sed 10q exits after printing the first 10 lines. head -n 10 equivalent
  • sed q exits after printing first line. head -n 1 equivalent
  • sed "/PATTERN/q" print file content until pattern is found (line matching pattern is printed)

Insert or append a line

i text = Insert line (before)

  • sed "/PATTERN/i pattern found on next line"
  • sed "1i inserted at the beginning"
  • sed "$i inserted before the last line"

a text = Append line (after)

  • sed "/PATTERN/a pattern found just before"
  • sed "1a inserted after the first line"
  • sed "$a inserted at the end"

Using a pattern file

... | sed -f filename.txt

cat filename.txt
s/un/one/
s/deux/two/

Playing with pattern space and hold space

  • h H Copy/append pattern space to hold space.
  • g G Copy/append hold space to pattern space.
  • n N Read/append the next line of input into the pattern space.
  • p Print the current pattern space.
  • x Exchange the contents of the hold and pattern spaces.
  • d Delete pattern space. Start next cycle.
  • D If pattern space contains no newline, start a normal new cycle as if the d command was issued. Otherwise, delete text in the pattern space up to the first newline, and restart cycle with the resultant pattern space, without reading a new line of input.

sed -n "/password/{x;p};h" print the line before a line containing ‘password’

sed -n "/^$/{p;h;};/./{x;/./p;}" delete the last line of each paragraph → so intuitive in order to achieve this result ☺

TODO: to be updated

delete duplicate lines from a sorted file (emulates “uniq”). First

line in a set of duplicate lines is kept, the rest are deleted

sed ’!N; /^\(.*\)\n\1/!P; D’

delete all CONSECUTIVE blank lines from file except the first; also

deletes all blank lines from top and end of file (emulates “cat -s”)

sed ’/./,/^/!d' # method 1, allows 0 blanks at top, 1 at EOF sed '/^/N;/\n$/D’ # method 2, allows 1 blank at top, 0 at EOF

delete all CONSECUTIVE blank lines from file except the first 2:

sed ’/^/N;//D’

delete all trailing blank lines at end of file

sed -e :a -e ’/^\n*/ba’

Sed multiline (ou équivalent avec perl)

cat a
one
two
three
four
five
six
eg244698 ~ $ cat a | perl -0pe 's/two\nthree/yo/'
one
yo
four
five
six
eg244698 ~ $ cat a
one
two
three
four
five
six
eg244698 ~ $ perl -i -0pe 's/two\nthree/yo/' a
eg244698 ~ $ cat a
one
yo
four
five
six

Color

  • inverted red: sed 's#FATAL#\x1b[1;41m&\x1b[0m#’
  • red: sed 's#ERROR#\x1b[1;31m&\x1b[0m#’
  • yellow: sed 's#WARN#\x1b[1;33m&\x1b[0m#’
  • blue: sed 's#INFO#\x1b[1;34m&\x1b[0m#’
  • light blue: sed 's#DEBUG#\x1b[1;36m&\x1b[0m#’
  • green: sed 's#TRACE#\x1b[1;32m&\x1b[0m#’

alias colorlog="sed 's#FATAL#\x1b[1;41m&\x1b[0m#;s#ERROR#\x1b[1;31m&\x1b[0m#;s#WARN#\x1b[1;33m&\x1b[0m#;s#INFO#\x1b[1;34m&\x1b[0m#;s#DEBUG#\x1b[1;36m&\x1b[0m#;s#TRACE#\x1b[1;32m&\x1b[0m#'"