Linux Tips and Tricks

Linux Tips and Tricks

Remove duplicate lines from file using bash and perl

September11

we have a file called dupl and it has some duplicate entires

1
2
3
4
5
6
7
$ cat dupl
39901
39901
39902
39903
39902
39003

we can use sort and uniq command to remove the duplicate entires

1
2
3
4
5
$ sort dupl | uniq
39003
39901
39902
39903

using AWK

1
2
3
4
5
$ awk '{a[$1]}END{for (i in a) print i}' dupl
39901
39902
39903
39003

Using Perl

1
2
3
4
5
$  perl -ne '$s{$_}++||print' dupl
39901
39902
39903
39003
posted under Uncategorized

Email will not be published

Website example

Your Comment:


Recent Comments

    Categories