Puppy Linux Discussion Forum

scsijon

I have a list of 24000+ 'words' in a Text file. It's in a vertical list format.

However, about 40+% are duplicates and i'd like to delete them.

The format of the 'words' are basically anything that can be typed in on a keyboard.

I have used 'return-newline' as the separator.

Has anyone a simple script I can run to 'fix the problem'

thanks

SFR · Joined: 26 Oct 2011 Posts: 1039

Hey Scsijon

To make it clear - the list looks like this:
abc
def
abc
abc
blablabla
zxzx
something
blablabla
...

and should look like this:
abc
def
blablabla
zxzx
something
...

right?

If you don't mind that lines will be also sorted:

vovchik · Joined: 23 Oct 2006 Posts: 1285 Location: Ukraine

Dear guys,

This is pretty easy too:

amigo · Joined: 02 Apr 2007 Posts: 2217

I recently had the same problem and wrote something for it. But, I don't find it right now, so I've re-created it:

tallboy · Joined: 21 Sep 2010 Posts: 438 Location: Oslo, Norway

Sorry, mistake, could not find a way to delete post.

GustavoYz · Joined: 07 Jul 2010 Posts: 895 Location: .ar

scsijon · Posted: Fri 01 Mar 2013, 18:17 Post_subject:

Sorry folks, I wish it was that simple.

I have already sorted the list, that was when I relaized how many duplicates were in it.

consider this:

aaa
aba
ada
ad
aea
aea
aea
agd
agd
ased
ased
ased-ss
ased-ss<p

and on we go.

I want to remove all the duplicates found.

It's what happens when you need to rebuild a crashed component list from backups.

Keef · Joined: 20 Dec 2007 Posts: 618 Location: Staffordshire

scsijon · Posted: Fri 01 Mar 2013, 19:59 Post_subject:

sorry , vovchick, GustavoYz and Keef

yes it does, I must have done a typo the first time I tried it.

thanks all