duplicate lines using for

  • Thread starter Thread starter branigan
  • Start date Start date
B

branigan

I have used the follow for syntax to pull certain text from a file. However,
my question is how can I remove

duplicate lines. An example is below. What I want out is a list of ip's
without duplicates.

for /f "tokens=2,7" %a in (newone.txt) do @echo %a %b

newone.txt follows:

10.29.2.4 01BD
10.29.2.4 01BD
10.29.2.4 01BD
10.29.2.4 01BD
10.55.6.78 01BD
10.54.2.1 01BD
10.55.6.78 01BD
10.8.9.9 01BD
13.2.33.3 01BD
13.2.33.3 01BD

I want:

10.29.2.4 01bd
10.55.6.78 01BD
10.8.9.9 01BD....
 
branigan said:
I have used the follow for syntax to pull certain text from a file.
However, my question is how can I remove

duplicate lines. An example is below. What I want out is a list of ip's
without duplicates.

for /f "tokens=2,7" %a in (newone.txt) do @echo %a %b

newone.txt follows:

10.29.2.4 01BD
10.29.2.4 01BD
10.29.2.4 01BD
10.29.2.4 01BD
10.55.6.78 01BD
10.54.2.1 01BD
10.55.6.78 01BD
10.8.9.9 01BD
13.2.33.3 01BD
13.2.33.3 01BD

I want:

10.29.2.4 01bd
10.55.6.78 01BD
10.8.9.9 01BD....

As a batch file:

[1]@echo off&setlocal enabledelayedexpansion

[2]if exist temp.txt del temp.txt
[3]for /f "tokens=2,7" %%a in (newone.txt) do echo %%a %%b>>temp.txt
[4]set ypl=
[5]for /f "tokens=*" %%i in ('type temp.txt^|sort') do if not x!ypl!==x%%i
set ypl=%%i&echo %%i
[6]if exist temp.txt del temp.txt

Each line begins [number]. Lines will be wrapped in transmission and need to
be rejoined. The [number] at the beginning of each line needs to be removed.

HTH

....Bill
 
I'm uncertain what happened since the basic logic of your script appears
sound but didn't work (at least for me) as posted -

[1]@echo off&setlocal enabledelayedexpansion
[2]if exist temp.txt del temp.txt
[3]for /f "tokens=*" %%a in (newone.txt) do echo %%a>>temp.txt
[4]set ypl=
[5]for /f "tokens=*" %%i in ('type temp.txt^|sort') do if not
x!ypl!==x%%i set ypl=%%i&echo %%i
[6]if exist temp.txt del temp.txt

--
Dean Wells [MVP / Directory Services]
MSEtechnology
[[ Please respond to the Newsgroup only regarding posts ]]
R e m o v e t h e m a s k t o s e n d e m a i l
branigan said:
I have used the follow for syntax to pull certain text from a file.
However, my question is how can I remove

duplicate lines. An example is below. What I want out is a list of
ip's without duplicates.

for /f "tokens=2,7" %a in (newone.txt) do @echo %a %b

newone.txt follows:

10.29.2.4 01BD
10.29.2.4 01BD
10.29.2.4 01BD
10.29.2.4 01BD
10.55.6.78 01BD
10.54.2.1 01BD
10.55.6.78 01BD
10.8.9.9 01BD
13.2.33.3 01BD
13.2.33.3 01BD

I want:

10.29.2.4 01bd
10.55.6.78 01BD
10.8.9.9 01BD....

As a batch file:

[1]@echo off&setlocal enabledelayedexpansion

[2]if exist temp.txt del temp.txt
[3]for /f "tokens=2,7" %%a in (newone.txt) do echo %%a %%b>>temp.txt
[4]set ypl=
[5]for /f "tokens=*" %%i in ('type temp.txt^|sort') do if not
x!ypl!==x%%i set ypl=%%i&echo %%i
[6]if exist temp.txt del temp.txt

Each line begins [number]. Lines will be wrapped in transmission and
need to be rejoined. The [number] at the beginning of each line needs
to be removed.
HTH

...Bill
 
billious said:
branigan said:
I have used the follow for syntax to pull certain text from a file.
However, my question is how can I remove

duplicate lines. An example is below. What I want out is a list of
ip's without duplicates.

for /f "tokens=2,7" %a in (newone.txt) do @echo %a %b

newone.txt follows:

10.29.2.4 01BD
10.29.2.4 01BD
10.29.2.4 01BD
10.29.2.4 01BD
10.55.6.78 01BD
10.54.2.1 01BD
10.55.6.78 01BD
10.8.9.9 01BD
13.2.33.3 01BD
13.2.33.3 01BD

I want:

10.29.2.4 01bd
10.55.6.78 01BD
10.8.9.9 01BD....

As a batch file:

[1]@echo off&setlocal enabledelayedexpansion

[2]if exist temp.txt del temp.txt
[3]for /f "tokens=2,7" %%a in (newone.txt) do echo %%a %%b>>temp.txt
[4]set ypl=
[5]for /f "tokens=*" %%i in ('type temp.txt^|sort') do if not
x!ypl!==x%%i set ypl=%%i&echo %%i
[6]if exist temp.txt del temp.txt

Each line begins [number]. Lines will be wrapped in transmission and
need to be rejoined. The [number] at the beginning of each line needs
to be removed.
HTH

...Bill
Dean Wells said:
I'm uncertain what happened since the basic logic of your script appears
sound but didn't work (at least for me) as posted -

[1]@echo off&setlocal enabledelayedexpansion
[2]if exist temp.txt del temp.txt
[3]for /f "tokens=*" %%a in (newone.txt) do echo %%a>>temp.txt
[4]set ypl=
[5]for /f "tokens=*" %%i in ('type temp.txt^|sort') do if not x!ypl!==x%%i
set ypl=%%i&echo %%i
[6]if exist temp.txt del temp.txt

--
Dean Wells [MVP / Directory Services]
MSEtechnology
[[ Please respond to the Newsgroup only regarding posts ]]
R e m o v e t h e m a s k t o s e n d e m a i l

I'm using XPHSP2 on which it worked perfectly. The only little glitch I can
see might be line 3, where the selected tokens would be required - but the
OP only gave us a list of the data extracted from newone.txt, not the actual
source.

So - what did it not do that it should have done, or whid did it do that it
should not have done?

....Bill
 
branigan said:
my question is how can I remove duplicate lines.

97) I need to remove duplicate entries from the output or a file.
130306 Jul 12 2005 ftp://garbo.uwasa.fi/pc/link/tscmd.zip
tscmd.zip Useful NT/2000/XP script tricks and tips, T.Salmi

All the best, Timo
 
It produced a single line of output only based on the original poster's
sample file.

--
Dean Wells [MVP / Directory Services]
MSEtechnology
[[ Please respond to the Newsgroup only regarding posts ]]
R e m o v e t h e m a s k t o s e n d e m a i l
billious said:
I have used the follow for syntax to pull certain text from a file.
However, my question is how can I remove

duplicate lines. An example is below. What I want out is a list of
ip's without duplicates.

for /f "tokens=2,7" %a in (newone.txt) do @echo %a %b

newone.txt follows:

10.29.2.4 01BD
10.29.2.4 01BD
10.29.2.4 01BD
10.29.2.4 01BD
10.55.6.78 01BD
10.54.2.1 01BD
10.55.6.78 01BD
10.8.9.9 01BD
13.2.33.3 01BD
13.2.33.3 01BD

I want:

10.29.2.4 01bd
10.55.6.78 01BD
10.8.9.9 01BD....



As a batch file:

[1]@echo off&setlocal enabledelayedexpansion

[2]if exist temp.txt del temp.txt
[3]for /f "tokens=2,7" %%a in (newone.txt) do echo %%a %%b>>temp.txt
[4]set ypl=
[5]for /f "tokens=*" %%i in ('type temp.txt^|sort') do if not
x!ypl!==x%%i set ypl=%%i&echo %%i
[6]if exist temp.txt del temp.txt

Each line begins [number]. Lines will be wrapped in transmission and
need to be rejoined. The [number] at the beginning of each line
needs to be removed.
HTH

...Bill
Dean Wells said:
I'm uncertain what happened since the basic logic of your script
appears sound but didn't work (at least for me) as posted -

[1]@echo off&setlocal enabledelayedexpansion
[2]if exist temp.txt del temp.txt
[3]for /f "tokens=*" %%a in (newone.txt) do echo %%a>>temp.txt
[4]set ypl=
[5]for /f "tokens=*" %%i in ('type temp.txt^|sort') do if not
x!ypl!==x%%i set ypl=%%i&echo %%i
[6]if exist temp.txt del temp.txt

--
Dean Wells [MVP / Directory Services]
MSEtechnology
[[ Please respond to the Newsgroup only regarding posts ]]
R e m o v e t h e m a s k t o s e n d e m a i l

I'm using XPHSP2 on which it worked perfectly. The only little glitch
I can see might be line 3, where the selected tokens would be
required - but the OP only gave us a list of the data extracted from
newone.txt, not the actual source.

So - what did it not do that it should have done, or whid did it do
that it should not have done?

...Bill
 
billious said:
billious wrote:
I have used the follow for syntax to pull certain text from a file.
However, my question is how can I remove

duplicate lines. An example is below. What I want out is a list of
ip's without duplicates.

for /f "tokens=2,7" %a in (newone.txt) do @echo %a %b

newone.txt follows:

10.29.2.4 01BD
10.29.2.4 01BD
10.29.2.4 01BD
10.29.2.4 01BD
10.55.6.78 01BD
10.54.2.1 01BD
10.55.6.78 01BD
10.8.9.9 01BD
13.2.33.3 01BD
13.2.33.3 01BD

I want:

10.29.2.4 01bd
10.55.6.78 01BD
10.8.9.9 01BD....



As a batch file:

[1]@echo off&setlocal enabledelayedexpansion

[2]if exist temp.txt del temp.txt
[3]for /f "tokens=2,7" %%a in (newone.txt) do echo %%a %%b>>temp.txt
[4]set ypl=
[5]for /f "tokens=*" %%i in ('type temp.txt^|sort') do if not
x!ypl!==x%%i set ypl=%%i&echo %%i
[6]if exist temp.txt del temp.txt

Each line begins [number]. Lines will be wrapped in transmission and
need to be rejoined. The [number] at the beginning of each line
needs to be removed.
HTH

...Bill
Dean Wells said:
I'm uncertain what happened since the basic logic of your script
appears sound but didn't work (at least for me) as posted -

[1]@echo off&setlocal enabledelayedexpansion
[2]if exist temp.txt del temp.txt
[3]for /f "tokens=*" %%a in (newone.txt) do echo %%a>>temp.txt
[4]set ypl=
[5]for /f "tokens=*" %%i in ('type temp.txt^|sort') do if not
x!ypl!==x%%i set ypl=%%i&echo %%i
[6]if exist temp.txt del temp.txt

--
Dean Wells [MVP / Directory Services]
MSEtechnology
[[ Please respond to the Newsgroup only regarding posts ]]
R e m o v e t h e m a s k t o s e n d e m a i l

I'm using XPHSP2 on which it worked perfectly. The only little glitch
I can see might be line 3, where the selected tokens would be
required - but the OP only gave us a list of the data extracted from
newone.txt, not the actual source.

So - what did it not do that it should have done, or whid did it do
that it should not have done?

...Bill


Dean Wells said:
It produced a single line of output only based on the original poster's
sample file.

--
Dean Wells [MVP / Directory Services]
MSEtechnology
[[ Please respond to the Newsgroup only regarding posts ]]
R e m o v e t h e m a s k t o s e n d e m a i l

I can reproduce that fault by changing ">>" to ">" in line 3. That would
produce a one-line report (the last data line.)

No doubt other changes will produce other effects. Please post your input
file and reveal which single line was produced. A precise copy of your batch
would also be useful.

I tried this on XP - perhaps you are using 2K and that's the difference??

....Bill
 
The modified script was included in my original reply, the input file
was a copy and paste from the original post (no padding or irrelevant
symbols were present), my OS is Windows Professional XP SP2. That
aside, it all appears to be working now and is based entirely on your
original logic.

--
Dean Wells [MVP / Directory Services]
MSEtechnology
[[ Please respond to the Newsgroup only regarding posts ]]
R e m o v e t h e m a s k t o s e n d e m a i l
billious said:
billious wrote:
I have used the follow for syntax to pull certain text from a
file. However, my question is how can I remove

duplicate lines. An example is below. What I want out is a list
of ip's without duplicates.

for /f "tokens=2,7" %a in (newone.txt) do @echo %a %b

newone.txt follows:

10.29.2.4 01BD
10.29.2.4 01BD
10.29.2.4 01BD
10.29.2.4 01BD
10.55.6.78 01BD
10.54.2.1 01BD
10.55.6.78 01BD
10.8.9.9 01BD
13.2.33.3 01BD
13.2.33.3 01BD

I want:

10.29.2.4 01bd
10.55.6.78 01BD
10.8.9.9 01BD....



As a batch file:

[1]@echo off&setlocal enabledelayedexpansion

[2]if exist temp.txt del temp.txt
[3]for /f "tokens=2,7" %%a in (newone.txt) do echo %%a
%%b>>temp.txt [4]set ypl=
[5]for /f "tokens=*" %%i in ('type temp.txt^|sort') do if not
x!ypl!==x%%i set ypl=%%i&echo %%i
[6]if exist temp.txt del temp.txt

Each line begins [number]. Lines will be wrapped in transmission
and need to be rejoined. The [number] at the beginning of each
line needs to be removed.
HTH

...Bill


I'm uncertain what happened since the basic logic of your script
appears sound but didn't work (at least for me) as posted -

[1]@echo off&setlocal enabledelayedexpansion
[2]if exist temp.txt del temp.txt
[3]for /f "tokens=*" %%a in (newone.txt) do echo %%a>>temp.txt
[4]set ypl=
[5]for /f "tokens=*" %%i in ('type temp.txt^|sort') do if not
x!ypl!==x%%i set ypl=%%i&echo %%i
[6]if exist temp.txt del temp.txt

--
Dean Wells [MVP / Directory Services]
MSEtechnology
[[ Please respond to the Newsgroup only regarding posts ]]
R e m o v e t h e m a s k t o s e n d e m a i l


I'm using XPHSP2 on which it worked perfectly. The only little
glitch I can see might be line 3, where the selected tokens would be
required - but the OP only gave us a list of the data extracted from
newone.txt, not the actual source.

So - what did it not do that it should have done, or whid did it do
that it should not have done?

...Bill


Dean Wells said:
It produced a single line of output only based on the original
poster's sample file.

--
Dean Wells [MVP / Directory Services]
MSEtechnology
[[ Please respond to the Newsgroup only regarding posts ]]
R e m o v e t h e m a s k t o s e n d e m a i l

I can reproduce that fault by changing ">>" to ">" in line 3. That
would produce a one-line report (the last data line.)

No doubt other changes will produce other effects. Please post your
input file and reveal which single line was produced. A precise copy
of your batch would also be useful.

I tried this on XP - perhaps you are using 2K and that's the
difference??
...Bill
 
The following list is from the actual output (ie. newone.txt). This is what
I want to trip down.


AT1/IMA3.1 10.55.24.76 Fa0/0 62.7.123.112 06 10BE 01BD
3
AT1/IMA3.1 10.10.131.70 Fa0/0 191.180.209.199 06 09F5 01BD
3
AT1/IMA3.1 10.50.3.210 Fa0/0 14.117.32.228 06 04AF 01BD
3
AT1/IMA3.1 10.55.24.76 Fa0/0 143.54.32.35 06 0D3F 01BD
3
AT1/IMA3.1 10.50.3.210 Fa0/0 133.90.162.144 06 0547 01BD
3
AT1/IMA3.1 10.40.67.240 Fa0/0 200.14.124.213 06 1128 01BD
3
AT1/IMA3.1 10.55.0.252 Fa0/0 163.117.250.141 06 0C13 01BD
3
AT1/IMA3.1 10.10.131.70 Fa0/0 151.165.42.244 06 06B7 01BD
3
AT1/IMA3.1 10.10.131.70 Fa0/0 10.212.232.134 06 0D90 01BD
1
AT1/IMA3.1 10.55.24.76 Fa0/0 182.164.36.8 06 10A0 01BD
3
AT1/IMA3.1 10.40.67.240 Fa0/0 182.241.155.134 06 05F7 01BD
3
AT1/IMA3.1 10.10.131.70 Fa0/0 159.115.92.85 06 0C2A 01BD
3
AT1/IMA3.1 10.10.131.70 Fa0/0 144.195.42.229 06 0DCD 01BD
1
AT1/IMA3.1 10.55.1.247 Fa0/0 59.130.26.170 06 072E 01BD
2
AT1/IMA3.1 10.50.1.232 Fa0/0 84.236.47.210 06 1022 01BD
3
AT1/IMA3.1 10.50.1.232 Fa0/0 223.86.56.116 06 0F68 01BD
3
AT1/IMA3.1 10.50.2.252 Fa0/0 181.86.249.226 06 0DE8 01BD
3
AT1/IMA3.1 10.10.131.70 Fa0/0 158.123.53.55 06 045A 01BD
3
AT1/IMA3.1 10.50.3.210 Fa0/0 199.176.182.28 06 06AD 01BD
1
AT1/IMA3.1 10.10.131.70 Fa0/0 45.16.144.56 06 06F2 01BD
3
AT1/IMA3.1 10.10.131.70 Fa0/0 29.210.37.168 06 0BA4 01BD
3
AT1/IMA3.1 10.50.3.210 Fa0/0 176.183.36.90 06 1261 01BD
3




Dean Wells said:
I'm uncertain what happened since the basic logic of your script appears
sound but didn't work (at least for me) as posted -

[1]@echo off&setlocal enabledelayedexpansion
[2]if exist temp.txt del temp.txt
[3]for /f "tokens=*" %%a in (newone.txt) do echo %%a>>temp.txt
[4]set ypl=
[5]for /f "tokens=*" %%i in ('type temp.txt^|sort') do if not x!ypl!==x%%i
set ypl=%%i&echo %%i
[6]if exist temp.txt del temp.txt

--
Dean Wells [MVP / Directory Services]
MSEtechnology
[[ Please respond to the Newsgroup only regarding posts ]]
R e m o v e t h e m a s k t o s e n d e m a i l
branigan said:
I have used the follow for syntax to pull certain text from a file.
However, my question is how can I remove

duplicate lines. An example is below. What I want out is a list of
ip's without duplicates.

for /f "tokens=2,7" %a in (newone.txt) do @echo %a %b

newone.txt follows:

10.29.2.4 01BD
10.29.2.4 01BD
10.29.2.4 01BD
10.29.2.4 01BD
10.55.6.78 01BD
10.54.2.1 01BD
10.55.6.78 01BD
10.8.9.9 01BD
13.2.33.3 01BD
13.2.33.3 01BD

I want:

10.29.2.4 01bd
10.55.6.78 01BD
10.8.9.9 01BD....

As a batch file:

[1]@echo off&setlocal enabledelayedexpansion

[2]if exist temp.txt del temp.txt
[3]for /f "tokens=2,7" %%a in (newone.txt) do echo %%a %%b>>temp.txt
[4]set ypl=
[5]for /f "tokens=*" %%i in ('type temp.txt^|sort') do if not
x!ypl!==x%%i set ypl=%%i&echo %%i
[6]if exist temp.txt del temp.txt

Each line begins [number]. Lines will be wrapped in transmission and
need to be rejoined. The [number] at the beginning of each line needs
to be removed.
HTH

...Bill
 
[1]@echo off&setlocal enabledelayedexpansion
[2]if exist temp.txt del temp.txt
[3]for /f "tokens=2,7" %%a in (newone.txt) do echo %%a %%b>>temp.txt
[4]set ypl=
[5]for /f "tokens=*" %%i in ('type temp.txt^|sort') do if not x!ypl!==x%%i
set ypl=%%i&echo %%i
[6]if exist temp.txt del temp.txt

Each line begins [number]. Lines will be wrapped in transmission and need to
be rejoined. The [number] at the beginning of each line needs to be removed.

Which is the batch I originally posted, yields:

10.10.131.70 01BD
10.40.67.240 01BD
10.50.1.232 01BD
10.50.2.252 01BD
10.50.3.210 01BD
10.55.0.252 01BD
10.55.1.247 01BD
10.55.24.76 01BD

which appears to be correct.

Does this solve the problem?

....Bill

branigan said:
The following list is from the actual output (ie. newone.txt). This is
what I want to trip down.


AT1/IMA3.1 10.55.24.76 Fa0/0 62.7.123.112 06 10BE 01BD 3
AT1/IMA3.1 10.10.131.70 Fa0/0 191.180.209.199 06 09F5 01BD 3
AT1/IMA3.1 10.50.3.210 Fa0/0 14.117.32.228 06 04AF 01BD 3
AT1/IMA3.1 10.55.24.76 Fa0/0 143.54.32.35 06 0D3F 01BD 3
AT1/IMA3.1 10.50.3.210 Fa0/0 133.90.162.144 06 0547 01BD 3
AT1/IMA3.1 10.40.67.240 Fa0/0 200.14.124.213 06 1128 01BD 3
AT1/IMA3.1 10.55.0.252 Fa0/0 163.117.250.141 06 0C13 01BD 3
AT1/IMA3.1 10.10.131.70 Fa0/0 151.165.42.244 06 06B7 01BD 3
AT1/IMA3.1 10.10.131.70 Fa0/0 10.212.232.134 06 0D90 01BD 1
AT1/IMA3.1 10.55.24.76 Fa0/0 182.164.36.8 06 10A0 01BD 3
AT1/IMA3.1 10.40.67.240 Fa0/0 182.241.155.134 06 05F7 01BD 3
AT1/IMA3.1 10.10.131.70 Fa0/0 159.115.92.85 06 0C2A 01BD 3
AT1/IMA3.1 10.10.131.70 Fa0/0 144.195.42.229 06 0DCD 01BD 1
AT1/IMA3.1 10.55.1.247 Fa0/0 59.130.26.170 06 072E 01BD 2
AT1/IMA3.1 10.50.1.232 Fa0/0 84.236.47.210 06 1022 01BD 3
AT1/IMA3.1 10.50.1.232 Fa0/0 223.86.56.116 06 0F68 01BD 3
AT1/IMA3.1 10.50.2.252 Fa0/0 181.86.249.226 06 0DE8 01BD 3
AT1/IMA3.1 10.10.131.70 Fa0/0 158.123.53.55 06 045A 01BD 3
AT1/IMA3.1 10.50.3.210 Fa0/0 199.176.182.28 06 06AD 01BD 1
AT1/IMA3.1 10.10.131.70 Fa0/0 45.16.144.56 06 06F2 01BD 3
AT1/IMA3.1 10.10.131.70 Fa0/0 29.210.37.168 06 0BA4 01BD 3
AT1/IMA3.1 10.50.3.210 Fa0/0 176.183.36.90 06 1261 01BD 3




Dean Wells said:
I'm uncertain what happened since the basic logic of your script appears
sound but didn't work (at least for me) as posted -

[1]@echo off&setlocal enabledelayedexpansion
[2]if exist temp.txt del temp.txt
[3]for /f "tokens=*" %%a in (newone.txt) do echo %%a>>temp.txt
[4]set ypl=
[5]for /f "tokens=*" %%i in ('type temp.txt^|sort') do if not
x!ypl!==x%%i set ypl=%%i&echo %%i
[6]if exist temp.txt del temp.txt

--
Dean Wells [MVP / Directory Services]
MSEtechnology
[[ Please respond to the Newsgroup only regarding posts ]]
R e m o v e t h e m a s k t o s e n d e m a i l
I have used the follow for syntax to pull certain text from a file.
However, my question is how can I remove

duplicate lines. An example is below. What I want out is a list of
ip's without duplicates.

for /f "tokens=2,7" %a in (newone.txt) do @echo %a %b

newone.txt follows:

10.29.2.4 01BD
10.29.2.4 01BD
10.29.2.4 01BD
10.29.2.4 01BD
10.55.6.78 01BD
10.54.2.1 01BD
10.55.6.78 01BD
10.8.9.9 01BD
13.2.33.3 01BD
13.2.33.3 01BD

I want:

10.29.2.4 01bd
10.55.6.78 01BD
10.8.9.9 01BD....



As a batch file:

[1]@echo off&setlocal enabledelayedexpansion

[2]if exist temp.txt del temp.txt
[3]for /f "tokens=2,7" %%a in (newone.txt) do echo %%a %%b>>temp.txt
[4]set ypl=
[5]for /f "tokens=*" %%i in ('type temp.txt^|sort') do if not
x!ypl!==x%%i set ypl=%%i&echo %%i
[6]if exist temp.txt del temp.txt

Each line begins [number]. Lines will be wrapped in transmission and
need to be rejoined. The [number] at the beginning of each line needs
to be removed.
HTH

...Bill
 
Back
Top