How to delete from a text file, all lines that contain a specific string?

Asked 2023-09-20 20:33:59 View 124,929

How would I use sed to delete all lines in a text file that contain a specific string?

Answers

To remove the line and print the output to standard out:

sed '/pattern to match/d' ./infile

To directly modify the file – does not work with BSD sed:

sed -i '/pattern to match/d' ./infile

Same, but for BSD sed (Mac OS X and FreeBSD) – does not work with GNU sed:

sed -i '' '/pattern to match/d' ./infile

To directly modify the file (and create a backup) – works with BSD and GNU sed:

sed -i.bak '/pattern to match/d' ./infile

Answered   2023-09-20 20:33:59

  • How do I know what version of sed I have? GNU or non GNU? - anyone
  • I dont't know why, but the least command resulted in an empty file :( - anyone
  • With GNU sed 4,2,2, the -i '' doesn't work at all. It then treats the pattern to match as the filename. Just remove the ''. - anyone
  • on ubuntu sed -i.bak "/str/d" ./infile deletes my entire file. - anyone
  • Note that if your pattern will contain forward slashes, you can use an alternate pattern delimiter but must then escape the first one, e.g.: sed -i.bak "\#$pattern_variable_containing_slashes#d" ./infile - anyone

There are many other ways to delete lines with specific string besides sed:

AWK

awk '!/pattern/' file > temp && mv temp file

Ruby (1.9+)

ruby -i.bak -ne 'print if not /test/' file

Perl

perl -ni.bak -e "print unless /pattern/" file

Shell (bash 3.2 and later)

while read -r line
do
  [[ ! $line =~ pattern ]] && echo "$line"
done <file > o
mv o file

GNU grep

grep -v "pattern" file > temp && mv temp file

And of course sed (printing the inverse is faster than actual deletion):

sed -n '/pattern/!p' file

Answered   2023-09-20 20:33:59

  • how to delete a particular line with a pattern and also the line immediately above it? I have a fine with thousands of such lines in between different data. - anyone
  • On OS/X, the shell variation doesn't preserve leading spaces, but the grep -v variation worked well for me. - anyone
  • the sed example have a different behaviour, it only greps! it should be something like sed -n -i '/pattern/!p' file. - anyone
  • The grep version does not work when every line matches the pattern. Better do: grep -v "pattern" file > temp; mv temp file This might apply to some of the other examples depending on the return value. - anyone
  • "printing the inverse is faster than actual deletion" - Not on my machine (2012 MacBook Air, OS X 10.13.2). Create file: seq -f %f 10000000 >foo.txt. sed d: time sed -i '' '/6543210/d' foo.txt real 0m9.294s. sed !p: time sed -i '' -n '/6543210/!p' foo.txt real 0m13.671s. (For smaller files, the difference is larger.) - anyone

You can use sed to replace lines in place in a file. However, it seems to be much slower than using grep for the inverse into a second file and then moving the second file over the original.

e.g.

sed -i '/pattern/d' filename      

or

grep -v "pattern" filename > filename2; mv filename2 filename

The first command takes 3 times longer on my machine anyway.

Answered   2023-09-20 20:33:59

  • Voting up your answer too, just because you tried a performance comparison! - anyone
  • +1 for offering option to overwrite current file with the grep line. - anyone
  • The second 'grep' solution is also better for large files - anyone
  • I'm curious what the performance difference would be if it were sed '/pattern/d' filename > filename2; mv filename2 filename - anyone
  • (using ubuntu's /usr/share/dict/words) grep and mv: 0.010s | sed in place: 0.197s | sed and mv: 0.031s - anyone

The easy way to do it, with GNU sed:

sed --in-place '/some string here/d' yourfile

Answered   2023-09-20 20:33:59

  • A handy tip for others who stumble on this Q&A thread and are new to shell scripting: Short options are fine for one-time uses on the command line, but long options should be preferred in scripts since they're more readable. - anyone
  • +1 for the --in-place flag. I need to test that out on permissions protected files. (have to do some user scrubbing.) - anyone
  • Note that the long option is only available on GNU sed. Mac and BSD users will need to install gsed to do it this way. - anyone
  • Another tip: if your regex doesn't appear to match, try the -r option (or -E, depending on your version). This enables the use of regex metacharacters +, ?, {...} and (...). - anyone
  • This is the correct answer when your disk no have more space and you can't copy the text to another file. This command do what was questioned? - anyone

You may consider using ex (which is a standard Unix command-based editor):

ex +g/match/d -cwq file

where:

  • + executes given Ex command (man ex), same as -c which executes wq (write and quit)
  • g/match/d - Ex command to delete lines with given match, see: Power of g

The above example is a POSIX-compliant method for in-place editing a file as per this post at Unix.SE and POSIX specifications for ex.


The difference with sed is that:

sed is a Stream EDitor, not a file editor.BashFAQ

Unless you enjoy unportable code, I/O overhead and some other bad side effects. So basically some parameters (such as in-place/-i) are non-standard FreeBSD extensions and may not be available on other operating systems.

Answered   2023-09-20 20:33:59

  • that's great... when I do man ex it gives me the man for vim, it seems ex is part of vim... if I understood right that means the pattern syntax for match is vimregex.com which is similar but different to POSIX and PCRE flavours? - anyone
  • :g is POSIX-compliant command with some slight differences. I assume PCRE was based on it. - anyone
  • @kenorb "I/O overhead and some other bad side effects" could you elaborate? AFAIK ex is using a temp file, just like every other sane tool, besides idk using dd - anyone

I was struggling with this on Mac. Plus, I needed to do it using variable replacement.

So I used:

sed -i '' "/$pattern/d" $file

where $file is the file where deletion is needed and $pattern is the pattern to be matched for deletion.

I picked the '' from this comment.

The thing to note here is use of double quotes in "/$pattern/d". Variable won't work when we use single quotes.

Answered   2023-09-20 20:33:59

  • Mac sed requires a parameter after -i, so if you don't want a backup, you still have to add an empty string: -i '' - anyone
  • For shell use sed -i "/$pattern/d" $file . Thank you for your answer. - anyone

You can also use this:

 grep -v 'pattern' filename

Here -v will print only other than your pattern (that means invert match).

Answered   2023-09-20 20:33:59

To get a inplace like result with grep you can do this:

echo "$(grep -v "pattern" filename)" >filename

Answered   2023-09-20 20:33:59

  • This is only good for the bash shell or similar (not tcsh). - anyone

I have made a small benchmark with a file which contains approximately 345 000 lines. The way with grep seems to be around 15 times faster than the sed method in this case.

I have tried both with and without the setting LC_ALL=C, it does not seem change the timings significantly. The search string (CDGA_00004.pdbqt.gz.tar) is somewhere in the middle of the file.

Here are the commands and the timings:

time sed -i "/CDGA_00004.pdbqt.gz.tar/d" /tmp/input.txt

real    0m0.711s
user    0m0.179s
sys     0m0.530s

time perl -ni -e 'print unless /CDGA_00004.pdbqt.gz.tar/' /tmp/input.txt

real    0m0.105s
user    0m0.088s
sys     0m0.016s

time (grep -v CDGA_00004.pdbqt.gz.tar /tmp/input.txt > /tmp/input.tmp; mv /tmp/input.tmp /tmp/input.txt )

real    0m0.046s
user    0m0.014s
sys     0m0.019s

Answered   2023-09-20 20:33:59

  • Which platform are you on? Which versions of sed/perl/grep do you use? - anyone
  • The platform I use is Linux (Gentoo). The sed version is GNU sed v 4.2.2, the perl version perl 5 (I cant tell which revision I used at the time of the test), and grep (GNU) is version 3.0. - anyone

Delete lines from all files that match the match

grep -rl 'text_to_search' . | xargs sed -i '/text_to_search/d'

Answered   2023-09-20 20:33:59

Answered   2023-09-20 20:33:59

perl -i    -nle'/regexp/||print' file1 file2 file3
perl -i.bk -nle'/regexp/||print' file1 file2 file3

The first command edits the file(s) inplace (-i).

The second command does the same thing but keeps a copy or backup of the original file(s) by adding .bk to the file names (.bk can be changed to anything).

Answered   2023-09-20 20:33:59

You can also delete a range of lines in a file. For example to delete stored procedures in a SQL file.

sed '/CREATE PROCEDURE.*/,/END ;/d' sqllines.sql

This will remove all lines between CREATE PROCEDURE and END ;.

I have cleaned up many sql files withe this sed command.

Answered   2023-09-20 20:33:59

echo -e "/thing_to_delete\ndd\033:x\n" | vim file_to_edit.txt

Answered   2023-09-20 20:33:59

I found most of the answers not useful for me, If you use vim I found this very easy and straightforward:

:g/<pattern>/d

Source

Answered   2023-09-20 20:33:59

  • if your comfortable with vim as opposed to sed you can use ex stackoverflow.com/a/33186317 beware that it's slower. What's nice about vim is you can \v<pattern> to avoid backslashitis - anyone

Just in case someone wants to do it for exact matches of strings, you can use the -w flag in grep - w for whole. That is, for example if you want to delete the lines that have number 11, but keep the lines with number 111:

-bash-4.1$ head file
1
11
111

-bash-4.1$ grep -v "11" file
1

-bash-4.1$ grep -w -v "11" file
1
111

It also works with the -f flag if you want to exclude several exact patterns at once. If "blacklist" is a file with several patterns on each line that you want to delete from "file":

grep -w -v -f blacklist file

Answered   2023-09-20 20:33:59

  • A bit misleading. -w, --word-regexp Select only those lines containing matches that form whole words. vs. -x, --line-regexp Select only those matches that exactly match the whole line. For a regular expression pattern, this is like parenthesizing the pattern and then surrounding it with ^ and $. - anyone

to show the treated text in console

cat filename | sed '/text to remove/d' 

to save treated text into a file

cat filename | sed '/text to remove/d' > newfile

to append treated text info an existing file

cat filename | sed '/text to remove/d' >> newfile

to treat already treated text, in this case remove more lines of what has been removed

cat filename | sed '/text to remove/d' | sed '/remove this too/d' | more

the | more will show text in chunks of one page at a time.

Answered   2023-09-20 20:33:59

  • With -i you can edit the file in place: sed -i '/text to remove/d' filename is the same as cat filename | sed '/text to remove/d' - anyone

Curiously enough, the accepted answer does not actually answer the question directly. The question asks about using sed to replace a string, but the answer seems to presuppose knowledge of how to convert an arbitrary string into a regex.

Many programming language libraries have a function to perform such a transformation, e.g.

python: re.escape(STRING)
ruby: Regexp.escape(STRING)
java:  Pattern.quote(STRING)

But how to do it on the command line?

Since this is a sed-oriented question, one approach would be to use sed itself:

sed 's/\([\[/({.*+^$?]\)/\\\1/g'

So given an arbitrary string $STRING we could write something like:

re=$(sed 's/\([\[({.*+^$?]\)/\\\1/g' <<< "$STRING")
sed "/$re/d" FILE

or as a one-liner:

 sed "/$(sed 's/\([\[/({.*+^$?]\)/\\\1/g' <<< "$STRING")/d" 

with variations as described elsewhere on this page.

Answered   2023-09-20 20:33:59

  • Where's the target file path in the last syntax? - anyone
cat filename | grep -v "pattern" > filename.1
mv filename.1 filename

Answered   2023-09-20 20:33:59

  • You're overwriting a file while it's still in use. - anyone
  • @DavorCubranic fixed - anyone

You can use good old ed to edit a file in a similar fashion to the answer that uses ex. The big difference in this case is that ed takes its commands via standard input, not as command line arguments like ex can. When using it in a script, the usual way to accomodate this is to use printf to pipe commands to it:

printf "%s\n" "g/pattern/d" w | ed -s filename

or with a heredoc:

ed -s filename <<EOF
g/pattern/d
w
EOF

Answered   2023-09-20 20:33:59

This solution is for doing the same operation on multiple file.

for file in *.txt; do grep -v "Matching Text" $file > temp_file.txt; mv temp_file.txt $file; done

Answered   2023-09-20 20:33:59