These tools are great, but fairly labor-intensive. So in an effort to automate the process and spend more time viewing the files and less time processing them, I recompiled the merging program and wrote a Linux script to do the work.
This processing could be done on a Windows machine using scripts and tools there, but I decided that I'd rather work in Linux for file viewing and processing. This leaves my Windows machines to run UniTrunker and create the logfiles, and my Linux machine to copy that logfiles and do the processing.
There are a few manual tasks that I had to perform initially. These include:
1. Export the system.xml file from UniTrunker and run either msxsl.exe or xslproc to extract group and user information into a .txt file.
2. Set up the folders and file hierarchy in advance
3. Mount the shared file location of the UniTrunker-logfile on the Windows computer
Once these tasks were done, I created a script for each system I was going to export and a cron job to run them each night shortly after midnight after UniTrunker had created and zipped the daily logfiles.
A link to the Linux version of UTC to merge logfiles with user and group data can be found here: UTC.sh file (Note this is not a .sh file, but an executable with no extension. In order for it to be downloaded as a file and not shown in a browser, I added the extension. Once it's downloaded, changed the name to remove the .sh extension.)
An example of the script I created:
utcopyexample.sh
!/bin/bash
datestring=$(date +%Y%m%d)
yesterday=$(date --date="yesterday" +%Y%m%d)
yearmonth=$(date --date="yesterday" +%Y%m)
processfile=~/your/path/here/$yearmonth/SystemName/UTM$yesterday.txt
processpath=/your/path/here/$yearmonth/SystemName/
filestring=UniTrunker-$yesterday.log.Z #Name of logfile created by UniTrunker
filestring2=UT$yesterday.log.Z #Shorten filename for ease
#Copy logfile from Windows machine to Linux machine for unzipping:
cp /mnt/nameofwindows/computerRunning/unitrunker/S00000031/$filestring ~/Documents/UTSort/raw/$filestring2
gunzip /home/parallels/Documents/UTSort/raw/UT$yesterday.log.Z #unzip logfile
#Move unzipped file to log storage location:
mv /home/parallels/Documents/UTSort/raw/UT$yesterday.log /home/parallels/Documents/UTSort/$yearmonth/Davis/UT$yesterday.log
#Run UTC sort/merging program to add user names and group names to logfile in addition to UID and GID numbers
#Note: Prior to running this script, it is necessary to export the System.xml file in UT and then run MSXSL.exe in Windows
#or xsltproc in Linux to create users.txt and a groups.txt files to be used by the UTC sort/merging program
UTC /your/path/here/$yearmonth/SystemName/UT$yesterday.log /your/path/here/users.txt /your/path/here/groups.txt $processfile
#Below are examples of searches/sorts that I do on the logfiles to sort via talkgroups or users or find unknowns.
#Place your talkgroup IDs or search terms in the field after grep and create folders and change filenames to be relevant
#to your local groups and users:
more $processfile | grep ,X | grep -v XP >${processpath}/WXPD/WXPD$yesterday.txt #WX PD
more $processfile | grep 10816 >${processpath}/Ops5/Ops5$yesterday.txt #Davis County Ops 5
more $processfile | grep 10848 >${processpath}/Ops6/Ops6$yesterday.txt #Davis County Ops 6
more $processfile | grep 8544 >${processpath}/BountPD1/BountPD1$yesterday.txt #Bountiful PD 1
more $processfile | grep '9440\|9408' >${processpath}/DavisLaw/DavisLaw$yesterday.txt #Davis Law 1 and 2
more $processfile | grep 9952 >${processpath}/DavisService/DavisService$yesterday.txt #Davis Service
more $processfile | grep 17600 >${processpath}/NRegional/NRegional$yesterday.txt #Northern Regional
more $processfile | grep 18464 >${processpath}/Event2/Event2$yesterday.txt #Event 2
more $processfile | grep ', ,G' >>${processpath}/Unknown/DavisUnknown$yearmonth.txt #unknown users
more $processfile | grep 'G,.*, ,'>>${processpath}/Unknown/DavisUnknownTGID$yearmonth.txt #unknown talkgroups
sort -u -t, -k8,8 ${processpath}/Unknown/SLCUnknownTGID$yearmonth.txt | cut -d, -f8 | sort -n>${processpath}/Unknown/SLCUnknownTGIDSorted$yearmonth.txt #List unknown TGIDs
sort -u -t, -k5,5 ${processpath}/Unknown/SLCUnknownUID$yearmonth.txt | cut -d, -f5 |sort -n>${processpath}/Unknown/SLCUnknownUIDSorted$yearmonth.txt # List unknown UIDs
No comments:
Post a Comment