Re: Eliminating duplicate lines
This worked great. Thanks! The uniq command actually accepts an input and output file. No pipe needed. Arthur Pemberton wrote:
I think the command is `uniq`. You just pipe your text to it. Man page: http://www.computerhope.com/unix/uuniq.htm
Thanks to Ron also. This may be easier to remember since it uses more common commands. Geoffrion, Ron P [IT] wrote:
cat filelist.txt | sort -u > filelist.unique
On Friday 23 December 2005 04:35, Jim Herrmann wrote:
Thanks to Ron also. This may be easier to remember since it uses more common commands.
How is 'sort' and more common than 'uniq'? I really only use the two in combination (uniq -c | sort -n) anyway...
I had never heard of uniq, but sort is quite common. I think that uniq also requires the input to be sorted if you want to get rid of all duplicates. No? However, sort -u won't give you a count like uniq - c will. Good point. Thanks, Jim Luke-Jr wrote:
On Friday 23 December 2005 04:35, Jim Herrmann wrote:
Thanks to Ron also. This may be easier to remember since it uses more common commands.
How is 'sort' and more common than 'uniq'? I really only use the two in combination (uniq -c | sort -n) anyway...
That's backwards. You have to put the *sort* to the left of the *uniq* for it to work, because *uniq* requires that its input already be sorted. The beauty of the Unix philosophy is that *uniq* doesn't have to know about sorting, because *sort* already does. On 12/22/05, Luke-Jr <luke@dashjr.org> wrote:
On Friday 23 December 2005 04:35, Jim Herrmann wrote:
Thanks to Ron also. This may be easier to remember since it uses more common commands.
How is 'sort' and more common than 'uniq'? I really only use the two in combination (uniq -c | sort -n) anyway...
On Friday 23 December 2005 15:23, Monty J. Harder wrote:
On 12/22/05, Luke-Jr <luke@dashjr.org> wrote:
On Friday 23 December 2005 04:35, Jim Herrmann wrote:
Thanks to Ron also. This may be easier to remember since it uses more common commands.
How is 'sort' and more common than 'uniq'? I really only use the two in combination (uniq -c | sort -n) anyway...
That's backwards. You have to put the *sort* to the left of the *uniq* for it to work, because *uniq* requires that its input already be sorted. The beauty of the Unix philosophy is that *uniq* doesn't have to know about sorting, because *sort* already does.
Not quite backwards, read it again-- you're right that it does require presorted content, though.
Oh. I've used that many times as well. Like to make a Top Ten list of the most popular things: *something | sort | uniq -c | sort -nr | head* On 12/23/05, Luke-Jr <luke@dashjr.org> wrote:
On Friday 23 December 2005 15:23, Monty J. Harder wrote:
On 12/22/05, Luke-Jr <luke@dashjr.org> wrote:
On Friday 23 December 2005 04:35, Jim Herrmann wrote:
Thanks to Ron also. This may be easier to remember since it uses more common commands.
How is 'sort' and more common than 'uniq'? I really only use the two in combination (uniq -c | sort -n) anyway...
That's backwards. You have to put the *sort* to the left of the *uniq* for it to work, because *uniq* requires that its input already be sorted. The beauty of the Unix philosophy is that *uniq* doesn't have to know about sorting, because *sort* already does.
Not quite backwards, read it again-- you're right that it does require presorted content, though.
I find that running this command in my Thunderbird Mail Local Folders directory will give me addresses that are being used to send junk mail to me. grep -i "Delivered-To" Junk | sort | uniq -c |sort -nr >spamaddresses.txt I get tons of junk in my postmaster, catch all account, and I'm trying to block some of it before it gets to the postmaster inbox. To do that, I need to know what addresses are being used. This does it. However, I think I'm going to start rejecting any addresses that are not defined to my domain, and thus this is not needed. It has, however, served as a great learning opportunity on the use of sort and uniq. Thanks, Jim Monty J. Harder wrote:
Oh. I've used that many times as well. Like to make a Top Ten list of the most popular things: */something/ | sort | uniq -c | sort -nr | head*
On 12/23/05, *Luke-Jr* <luke@dashjr.org <mailto:luke@dashjr.org>> wrote:
On Friday 23 December 2005 15:23, Monty J. Harder wrote: > On 12/22/05, Luke-Jr < luke@dashjr.org <mailto:luke@dashjr.org>> wrote: > > On Friday 23 December 2005 04:35, Jim Herrmann wrote: > > > Thanks to Ron also. This may be easier to remember since it uses more > > > common commands. > > > > How is 'sort' and more common than 'uniq'? I really only use the two in > > combination (uniq -c | sort -n) anyway... > > That's backwards. You have to put the *sort* to the left of the *uniq* for > it to work, because *uniq* requires that its input already be sorted. The > beauty of the Unix philosophy is that *uniq* doesn't have to know about > sorting, because *sort* already does.
Not quite backwards, read it again-- you're right that it does require presorted content, though.
------------------------------------------------------------------------
_______________________________________________ Kclug mailing list Kclug@kclug.org http://kclug.org/mailman/listinfo/kclug
participants (3)
-
Jim Herrmann -
Luke-Jr -
Monty J. Harder