Finding missing and duplicate lines in text files using PowerShell

by Klaus Graefensteiner 7/22/2008 6:17:00 AM

Introduction

Every now and then I need to investigate bugs that get exposed because countable "things" are less or more than expected. In my special case I was dealing with an application that manages subscription handles to memory registers of programmable logic controllers (PLC). These programs are called Data Access Servers. Occasionally we get calls form customers reporting that the number of handles is less, or sometimes more than expected. The first step to debug this situation is to find out, which handles are missing or which ones are duplicates. This blog post describes how PowerShell's Compare-Object cmdlet makes this task, which used to be a pain in the "peep" now a piece of cake.

Topology analysis of planet Jupiter 

Figure 1: Topology analysis of planet Jupiter

Procedure

The PLC memory addresses that clients are subscribing to are maintained in a list data structure. This list can be displayed in an MMC snap-in and it can be exported to a text file.

Step1: Export the item list

Open the Wonderware SMC and navigate to your DA Server node and the to Diagnostics\Client Groups\YourGroupName. Right click and export items into a text file.

Exporting Client Group Items I

Figure 2: Exporting Client Group Items

Item list in Notepad

Figure 3: Item list in Notepad

Step2: Fire up PowerShell

Script for missing items

This script requires two files. One file that got exported with the correct number items. I named it ClientItemsBefore.txt. The other file was exported when items were missing. This is called ClientItemsAfter.txt.

   1: #Load strings from first file
   2: $a = Get-Content -Path $home\desktop\ClientItemsBefore.txt
   3: #Initialize arrays of item names that we are going to compare
   4: $ins1 = ,0
   5: $ins2 = ,0
   6:  
   7: #Exclude headings and start with first data line
   8: #Get only the name of the item
   9: for($i = 1; $i -lt $a.Length; $i++)
  10: {
  11:     $in = ($a[$i].split("`t"))[0]
  12:     $ins1 += $in
  13: }
  14:  
  15: #Load strings from second file
  16: $a = Get-Content -Path $home\desktop\ClientItemsAfter.txt
  17: #Exclude headings start with first data line
  18: for($i = 1; $i -lt $a.Length; $i++)
  19: {
  20:     $in = ($a[$i].split("`t"))[0]
  21:     $ins2 += $in
  22: }
  23: #Find the differences in the two lists
  24: Compare-Object $ins1 $ins2
Script for duplicate items

In this case I exported the items into a file called ClientItemsWithDuplicates.txt.

   1: cls
   2: #Load strings from first file
   3: $a = Get-Content -Path $home\desktop\ClientItemsWithDuplicates.txt
   4: #Initialize arrays of item names that we are going to compare
   5: $ins1 = ,0
   6:  
   7: #Exclude headings start and with first data line
   8: #Get only the name of the item
   9: for($i = 1; $i -lt $a.Length; $i++)
  10: {
  11:     $in = ($a[$i].split("`t"))[0]
  12:     $ins1 += $in
  13: }
  14:  
  15: #Get rid of duplicates
  16: $ins2 = $ins1  | sort -Unique
  17:  
  18: #Compare before and after
  19: Compare-Object $ins1 $ins2

That's it

Before PowerShell I would, in the missing items case, combine the two item export files into one file. Then I would use a LogParser query with a Having clause to find the ones that are not duplicates. If I expected duplicates in one file, then I would use LogParser and search for them with an appropriate Having clause to filter them out. Is PowerShell going to replace the LogParser?

The script and item export files can be downloaded here: FindingMissingItems.zip

kick it on DotNetKicks.com

Be the first to rate this post

  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5

Tags: , , , ,

Debugging | PowerShell | Wonderware

Related posts

Comments

8/24/2008 11:28:27 PM

Tiago Albineli Motta

Insteresting, but too much verbose

Tiago Albineli Motta br

Add comment


(Will show your Gravatar icon)  

  Country flag

[b][/b] - [i][/i] - [u][/u]- [quote][/quote]



Live preview

9/7/2008 5:24:50 AM

Powered by BlogEngine.NET 1.3.0.0
Vanilla Theme by Klaus Graefensteiner

About Klaus Graefensteiner

GRAVATAR icon of Klaus Graefensteiner I enjoy the programming of machines.

E-mail me Send mail
Blogroll as OPML OPML LinkedIn Profile View Klaus Graefensteiner's LinkedIn profile

Calendar

<<  September 2008  >>
MoTuWeThFrSaSu
25262728293031
1234567
891011121314
15161718192021
22232425262728
293012345

View posts in large calendar

Recent comments

Disclaimer

The opinions expressed herein are my own personal opinions and do not represent my employer's view in anyway.

© Copyright 2008

Sign in