UniTrunker Automated File Merge

I've put together some tools to analyze UniTrunker data.  The details can be found here: UniTrunker file merge.

These tools are great, but fairly labor-intensive. So in an effort to automate the process and spend more time viewing the files and less time processing them, I recompiled the merging program and wrote a Linux script to do the work.

This processing could be done on a Windows machine using scripts and tools there, but I decided that I'd rather work in Linux for file viewing and processing.  This leaves my Windows machines to run  UniTrunker and create the logfiles, and my Linux machine to copy that logfiles and do the processing.

There are a few manual tasks that I had to perform initially.  These include:

1. Export the system.xml file from UniTrunker and run either msxsl.exe or xslproc to extract group and user information into a .txt file.
2. Set up the folders and file hierarchy in advance
3. Mount the shared file location of the UniTrunker-logfile on the Windows computer

Once these tasks were done, I created a script for each system I was going to export and a cron job to run them each night shortly after midnight after UniTrunker had created and zipped the daily logfiles.

A link to the Linux version of UTC to merge logfiles with user and group data can be found here: UTC.sh file (Note this is not a .sh file, but an executable with no extension. In order for it to be downloaded as a file and not shown in a browser, I added the extension.  Once it's downloaded, changed the name to remove the .sh extension.)

An example of the script I created:
utcopyexample.sh

!/bin/bash
datestring=$(date +%Y%m%d)
yesterday=$(date --date="yesterday" +%Y%m%d)
yearmonth=$(date --date="yesterday" +%Y%m)
processfile=~/your/path/here/$yearmonth/SystemName/UTM$yesterday.txt
processpath=/your/path/here/$yearmonth/SystemName/
filestring=UniTrunker-$yesterday.log.Z #Name of logfile created by UniTrunker
filestring2=UT$yesterday.log.Z #Shorten filename for ease
#Copy logfile from Windows machine to Linux machine for unzipping:
cp /mnt/nameofwindows/computerRunning/unitrunker/S00000031/$filestring ~/Documents/UTSort/raw/$filestring2
gunzip /home/parallels/Documents/UTSort/raw/UT$yesterday.log.Z #unzip logfile
#Move unzipped file to log storage location:
mv /home/parallels/Documents/UTSort/raw/UT$yesterday.log /home/parallels/Documents/UTSort/$yearmonth/Davis/UT$yesterday.log
#Run UTC sort/merging program to add user names and group names to logfile in addition to UID and GID numbers
#Note: Prior to running this script, it is necessary to export the System.xml file in UT and then run MSXSL.exe in Windows
#or xsltproc in Linux to create users.txt and a groups.txt files to be used by the UTC sort/merging program
UTC /your/path/here/$yearmonth/SystemName/UT$yesterday.log /your/path/here/users.txt /your/path/here/groups.txt $processfile
#Below are examples of searches/sorts that I do on the logfiles to sort via talkgroups or users or find unknowns.
#Place your talkgroup IDs or search terms in the field after grep and create folders and change filenames to be relevant
#to your local groups and users:
more $processfile | grep ,X | grep -v XP >${processpath}/WXPD/WXPD$yesterday.txt #WX PD
more $processfile | grep 10816 >${processpath}/Ops5/Ops5$yesterday.txt #Davis County Ops 5
more $processfile | grep 10848 >${processpath}/Ops6/Ops6$yesterday.txt #Davis County Ops 6
more $processfile | grep 8544 >${processpath}/BountPD1/BountPD1$yesterday.txt #Bountiful PD 1
more $processfile | grep '9440\|9408' >${processpath}/DavisLaw/DavisLaw$yesterday.txt #Davis Law 1 and 2
more $processfile | grep 9952 >${processpath}/DavisService/DavisService$yesterday.txt #Davis Service
more $processfile | grep 17600 >${processpath}/NRegional/NRegional$yesterday.txt #Northern Regional
more $processfile | grep 18464 >${processpath}/Event2/Event2$yesterday.txt #Event 2
more $processfile | grep  ', ,G' >>${processpath}/Unknown/DavisUnknown$yearmonth.txt  #unknown users
more $processfile | grep 'G,.*, ,'>>${processpath}/Unknown/DavisUnknownTGID$yearmonth.txt #unknown talkgroups
sort -u -t, -k8,8 ${processpath}/Unknown/SLCUnknownTGID$yearmonth.txt | cut -d, -f8 | sort -n>${processpath}/Unknown/SLCUnknownTGIDSorted$yearmonth.txt #List unknown TGID numbers
sort -u -t, -k5,5 ${processpath}/Unknown/SLCUnknownUID$yearmonth.txt | cut -d, -f5 |sort -n>${processpath}/Unknown/SLCUnknownUIDSorted$yearmonth.txt #List unknown UID numbers

If you're interested in compiling the UTC program yourself, email me and I'll get you the source code.

Virtual Radar

My radio monitoring began as the son of a volunteer fireman, but quickly turned to aviation when I bought my first scanner at around 10 years old. My love of airplanes has led to a career in flying. Strange as it may seem I still am interested in listening to the chatter of airplanes as they pass overhead even though I hear ATC all day at work.

As an addition to my ATC voice monitoring, I was interested in getting a virtual radar receiver that can plot aircraft location and/or altitude using Mode-S and ADS-B data.  For a year I tried a AirNav RadarBox receiver. I enjoyed its ease of use out of the box and the ability to combine outside FAA data in one program. However, the RadarBox is limited in its ability to pass raw data to 3rd party programs such as Planeplotter.

Last month I traded in my RadarBox for a Kinetic Avionics SBS-3 Virtual Radar Receiver (http://www.kinetic.co.uk/sbs-3.php). In addition to the Mode-S reception, the SBS-3 has a couple of other features that are really great! The box has 2 SDR tuners that have a bandwidth of 8 MHz that can be divided into 6 receivers.  Additionally, there is built-in ACARS decoding and for those near the coast a built-in AIS decoder as well.

The box can be connected to a PC via USB, or used stand-alone via ethernet.  Kinetic provides their Basestation software to interface with the SBS-3.  I've found it very robust.  The main difference here between the SBS-3 and RadarBox is the lack of online data updates built into Basestation.  When an aircraft is received, Basestation displays all the data that it receives, but unlike RadarBox, it doesn't go out on the web and download additional aircraft information.

The good thing is there is add-on software that interfaces with Basestation which will download the aircraft information and populate your database.  I use ActiveDisplay (http://www.gatwickaviationsociety.org.uk/AD_home.asp) and decided to purchase a subscription for one year to access the Gatwick Aviation Society aircraft database.  There is a lite version which is free.

My next project is to set up Planeplotter and interface the SBS-3 with it.  One feature I'm interested in is Multilateration.  This gives the ability to approximate the location of non-ADS-B aircraft by using data from other users in the area that can see the aircraft.

Seeing Mode-S and ADS-B data is a fascinating way to enhance aircraft monitoring and add to the awareness of who's flying around.

One Man's Junk is Another Man's Treasure

I lucked out and have found a few scanners being sold as junk by different individuals in the community.  They were old, and needed some cleaning and sprucing up, but to me I was thrilled to get them for a good price.  Even though they don't have all the bells and whistles of a brand new modern scanner, there still is tons of interesting things to be heard on them - trains, planes, ham radio, discriminator audio to name just a few.

The first acquisition was a Regency Digital Flight Scan receiver.  It has a wooden case and is a bit large. I got it home and, after a bit of work to find a motorola to bnc adapter, found that it receives ATC transmissions great.  I have found that modern broadband scanners have a hard time hearing ATC even though they are supposed to be such great receivers.  Filtering the FM broadcast band helps, but they are poor performers. It's nice to be able to have a dedicated ATC receiver.

The next find was a Uniden BC200XLT scanner.  The original owner had no use for it since it didn't do trunking.  I was excited to buy it. After adding a discriminator tap and mounting a jack in the housing (an advantage of having older larger scanners - there's more room to mount an audio jack), it's up and running as an additional analog scanner and discriminator source.

Thunderbirds Practice

We got to watch the Thunderbirds practice today at Hill Air Force Base in preparation for the open house and airshow - Warriors Over the Wasatch - that begins tomorrow.  It was fun to snap some pictures and snag some audio of their practice. 

Thunderbirds Audio

Motorola Maxtrac 5-pin Accessory Jack

My latest hardware project was to add a mono audio jack to the back of my Maxtrac 800 radio.  I have wanted a way to finally be able to record audio from the radios.  I added a couple of crimp pins to the plastic housing and ran wires from Pin 1 (ground) and Pin 4 (external speaker) to a female 1/8" audio jack.  I got the crimp pins from a DB-25 connector kit from Radio Shack. 

UniTrunker Logfile Merging

The program UniTrunker uses XML files to output data pertaining to Systems, Groups, Users, Sites, etc.  It stores call log files in Unicode format with just User numbers and Group numbers, but not text labels associated with them. I wrote a C program to merge the User and Group data with their respective numbers in the call log file.  This is a multiple step process to extract information from the System.xml file into text files for Users, Groups, and Sites; followed by converting those Unicode files to ascii files for handling with the C code; and then running them to created a merged logfile with both User and Group lables appended.  For this I created a batch file to complete the process with 2 inputs from the user during the process.

Prior to running the batch file, all of the needed files need to be placed in the same directory.  These include:

msxsl.exe (which can be downloaded from Microsoft's website)
UTC.exe (the program file which I wrote to perform the merge)
UTMerge.bat (the batch file which calls all the programs and converts files to correct filetypes)
UTxsl.bat(the batch file which calls the msxsl.exe to apply the xsl stylesheets)
UTlog.bat(the batch file which calls UTC.exe program to create a merged log file)
System.xml (this file is exported by going to the systems tab in UniTrunker and exporting the desired system)
users.xsl, groups.xsl, sites.xsl which I have created/modified based on the example from the UT website
UniTrunker-********.LOG file (created daily by UniTrunker and archived to a gzip file)
      -Note this file needs to be unzipped in order for the batch file to work

Once all the files are in the same directory, simply change directories to the location of the files, and then type "utmerge" to call the batch file.

The user will be prompted to enter the name of the .xml file.  Case doesn't matter, but be sure to type it correctly.  After a few seconds of processing the user will be prompted for the date of the logfile to be merged.  After a few more seconds a text file with the merged data will be created in the same directory.

A zip file of the required files can be found here:

UniTrunker Merge Program files

Download and extract the UTMerge.zip file to an accessible location.  Then extract a system.xml file from UniTrunker.  Copy or move a .LOG file from UniTrunker's logfiles to this same directory.  You can go to Run...and type CMD to open a DOS window.  Then change directories to the location of your UTM files and type utm to run the batch file. 

An easier way is to use windows explorer to browse to the directory and double click on UTM.bat. The program will run in a popped-up DOS window.  Carefully type in the information required when prompted, and VOILA, a merged log file will be created!

UPDATE:

I created 2 additional .bat files:UTxsl and UTlog. The UTmerge file can still be used successfully.  However, each time it runs it creates the same users.txt, group.txt, and sites.txt files.  This does no harm, but is redundant and takes time.  I spilt the processes into 2 batch files.  The first to be run should be UTxsl.  This will apply the xsl stylesheet once and create the necessary .txt files.  After these are created, there is no need to run this batch file again unless changes are made in UniTrunker to your system files and subsequently exported.

The second batch file to be run is UTlog.  This simply prompts for the date of the log file you would like to merge with the users and groups files.  It can be run multiple times on logs for different dates without recreating the users.txt and groups.txt files.

UPDATE 2 (February 23, 2013)

After looking for a way to further process the merged log files, I realized the Linux grep command was perfect for looking through logs to isolate individual users or groups or both.  This means I either have to run a grep program on my Windows computer, or move the files to my Linux computer to further process them.  It is a little cumbersome to have to move files to a different platform, but the power to sort the data makes it worth doing.

The power of the the grep command in dissecting the large amount of information in one days' log file is impressive.  For example, after a recent fire department response to a building fire, I sorted the log files based on the fire departments operations channel to see just the calls for the fire separate from the 1000's of other calls on the TRS for the day.

For a better explanation of the grep command see:

http://en.wikipedia.org/wiki/Grep


A sample output from a TG sort using the command:

~$ more UT20130220merged.txt | grep 10848 

20130220210159,2,Call,I,32092,Engine 85,G,10848,Davis Ops 6 - South Davis Metro Fire (Bountiful fire response),488, - 
20130220210242,2,Call,I,32014,Ambulance 83,G,10848,Davis Ops 6 - South Davis Metro Fire (Bountiful fire response),525, - 
20130220210247,2,Call,I,32014,Ambulance 83,G,10848,Davis Ops 6 - South Davis Metro Fire (Bountiful fire response),497, - 
20130220210256,2,Call,I,32031,Engine 85,G,10848,Davis Ops 6 - South Davis Metro Fire (Bountiful fire response),498, - 
20130220210300,2,Call,I,32014,Ambulance 83,G,10848,Davis Ops 6 - South Davis Metro Fire (Bountiful fire response),498, -

Great Utah Shake Out Mock Disaster Audio

I managed to capture some audio from the Davis County EOC during the mock earthquake drill on April 17, 2012 as part of the State of Utah's Great Utah Shakeout Disaster Drill.

Davis County Disaster Talkgroup Audio