Yesterday, I got the following question from a customer:
We have a system that generates files every couple of minutes, they are stored in a folder. Can we send those message contents using MQ visual edit to an mq server as soon as they become available in that folder?
My answer to the customer was:
MQ Visual Edit is a GUI program and does not have any scripting abilities.
We have another program called MQ Batch Toolkit that is a command-line version of MQ Visual Edit (so to speak). You could create a schedule event for every minute or 2 which would list the files of the directory and pass each file to MQ Batch Toolkit for processing. After processing the script can move or delete the file.
The reason I decided to write about this question is because the question and answer seem super simple to implement but the devil is always in the details. The gotcha that some people may not think about is the timing issue when using wildcards for path and file names.
Example script/batch file:
mqbt import -p MQWT1 -q TEST.Q1 -f /app/some/place/*
mv /app/some/place/* /app/some/place/backup/
The script file seems to do the job. First, it will load each file in the directory as a (separate) message to a queue then it will move all files in that directory to a backup directory.
Here’s where support will get that 2:00AM phone call about missing messages. It doesn’t matter the tool/solution you are using, like MQBT, when they are invoked, they will retrieve a directory listing then iterate over that list of files, putting each file as a message to a queue. BUT what happens if 1 or more files arrive after the tool gets the directory listing and before the move command is executed? Well, the tool will not see the file but when the move command is executed, it will see the file and move it.
The other issue is what happens if the program writing the files to the directory is in the middle of writing a file when the script/batch file is started. You don’t want half a file to be put as a message to a queue!!!
Note: What if the script didn’t use a move command but rather a delete command? The evidence of what happened would be gone and of course, people would start claiming that MQ lost their message!!!
Sometimes the simplest solution is what is required. Have the script/batch file processor feed the programs 1 file at a time (the same file).
Solution: The issue with the putting of a partial file can be solved by having the other program write the file to a temporary directory and when it is done writing, move the file to the watched directory.
The examples below will have the script/batch processor retrieve a list of files that have a file extension of “txt” and on the MQPUT, MQBT will set the MQMD Format field as string (i.e. MQSTR) because of the ‘-S’ parameter. For a full list of MQBT parameters, see the MQ Batch Toolkit Installation and Operation manual.
Windows Batch File:
@echo off
setlocal
cd /D C:\Capitalware\MQBT\
for %%f in (C:\app\some\place\*.txt) do (
mqbt insert -p MQWT1 -q TEST.Q1 -S -f %%f
move /Y %%f C:\app\some\place\backup\
)
endlocal
Linux/macOS/Raspberry Pi Script:
#!/bin/sh
PATH="/home/mqm/Capitalware/MQBT:$PATH"
export PATH
cd /home/mqm/Capitalware/MQBT/
for entry in "/app/some/place/"*.txt
do
if [ -f "$entry" ];then
mqbt insert -p MQWT1 -q TEST.Q1 -S -f "$entry"
mv -f "$entry" /app/some/place/backup/
fi
done
Hope that helps.
Regards,
Roger Lacroix
Capitalware Inc.
