Exporting MQ Objects into Individual Files

I posted this item on mqseries.net and I thought I should do a write up here.

Back in the early 2000s, I was a consultant at a customer site and they wanted the nightly dumps of the MQ objects (not messages), 1 line per object and then check those files into a source code repository.

I used SupportPac MS03 to dump the MQ objects but it put all objects in the same file and each object was on many lines. So, I wrote a simple Rexx script to extract each object type to: (1) flatten each object to 1 line and (2) write it their own object file.

SupportPac MS03 has been replaced with the dmpmqcfg program.

I have updated the list of object types in the Rexx script for the modern IBM MQ v9.2 (because IBM has added a lot of new objects in the last 15 years).

Example of running dmpmqcfg against your queue manager called MQA1:

dmpmqcfg -m MQA1 -a > MQA1.all.mqsc

Then you run it against my REXX script called fmt_dmpmqcfg.rex:

/* Format output from dspmqcfg command to a single line for each entry.  */
/* Put each MQ object type into a different file.                        */
/*                                                                       */
/* Example dspmqcfg command:                                             */
/*    dmpmqcfg -m MQA1 -a > MQA1.all.mqsc                                */

trace o
Parse Arg inFN .

if inFN = "" then
do
   Say "Invalid number of parameters."
   Say "i.e."
   Say "     fmt_dmpmqcfg.rex input_file_name"
   Exit
end

types = "QMGR CHANNEL LISTENER NAMELIST PROCESS QALIAS QLOCAL QMODEL QREMOTE SERVICE SUB TOPIC AUTHREC AUTHINFO CHLAUTH COMMINFO"


/* open file */
Call Stream inFN,'C','OPEN READ'

/* Delete previous output and then open it. */
do i=1 to Words(types)
   ptr = Word(types,i)
   outFN = inFN||"."||ptr||".mqsc"
   if (STREAM( outFN, 'C', 'QUERY EXIST' ) <> "") then
      "ERASE  "outFN
   Call Stream outFN,'C','OPEN WRITE'
   counts.ptr = 0
end

total = 0
mqscCmd = ""
do until Lines(inFN) = 0

   inLine = Strip(LineIn(inFN))   /* read next line */

   if (inLine = "") then
      NOP   /* blank line, forget-about-it */
   else if (SubStr(inLine,1,1) = "*") then
      NOP   /* comment line, forget-about-it */
   else
   do
      /* Last attribute in MQSC command? */
      if (Length(inLine) <> LastPos('+',inLine)) then
      do
         mqscCmd = mqscCmd || " " || inLine  /* add last attribute */
         ptr = Word(mqscCmd, 2)

         if (Pos("(", ptr) > 0) then
            ptr = SubStr(ptr, 1, Pos("(", ptr) - 1)

         nothing = LineOut(inFN||"."||ptr||".mqsc", mqscCmd)   /* write to the file  */

         counts.ptr = counts.ptr + 1
         total = total + 1
         mqscCmd = ""
      end
      else
      do
         temp = Strip(SubStr(inLine,1,(LastPos('+',inLine) - 1)))  /* remove '+' */
         if (mqscCmd = "") then
            mqscCmd = temp
         else
            mqscCmd = mqscCmd || " " || temp
      end
   end
end

/* close the files  */
Call Stream inFN,'C','CLOSE'
do i=1 to Words(types)
   ptr = Word(types,i)
   outFN = inFN||"."||ptr||".mqsc"
   Call Stream outFN,'C','CLOSE'
   Say outFN || " contains " || counts.ptr || " MQSC formatted commands."
end

Say ""
Say "Total formatted MQSC commands was:" total

Exit

To run the Rexx script, the command is:

rexx fmt_dmpmqcfg.rex  MQA1.all.mqsc

The Rexx script will output:

MQA1.all.mqsc.QMGR.mqsc contains 1 MQSC formatted commands.
MQA1.all.mqsc.CHANNEL.mqsc contains 43 MQSC formatted commands.
MQA1.all.mqsc.LISTENER.mqsc contains 5 MQSC formatted commands.
MQA1.all.mqsc.NAMELIST.mqsc contains 3 MQSC formatted commands.
MQA1.all.mqsc.PROCESS.mqsc contains 2 MQSC formatted commands.
MQA1.all.mqsc.QALIAS.mqsc contains 4 MQSC formatted commands.
MQA1.all.mqsc.QLOCAL.mqsc contains 111 MQSC formatted commands.
MQA1.all.mqsc.QMODEL.mqsc contains 7 MQSC formatted commands.
MQA1.all.mqsc.QREMOTE.mqsc contains 11 MQSC formatted commands.
MQA1.all.mqsc.SERVICE.mqsc contains 6 MQSC formatted commands.
MQA1.all.mqsc.SUB.mqsc contains 1 MQSC formatted commands.
MQA1.all.mqsc.TOPIC.mqsc contains 8 MQSC formatted commands.
MQA1.all.mqsc.AUTHREC.mqsc contains 438 MQSC formatted commands.
MQA1.all.mqsc.AUTHINFO.mqsc contains 4 MQSC formatted commands.
MQA1.all.mqsc.CHLAUTH.mqsc contains 3 MQSC formatted commands.
MQA1.all.mqsc.COMMINFO.mqsc contains 1 MQSC formatted commands.

Total formatted MQSC commands was: 648

If you setup a job (script/batch file) to do this nightly and then check the output files into a source code repository then you will have a complete history of changes made to your queue manager. Because the Rexx script flattens the data to 1 line per object, your source code repository will show you exactly which objects have changed.

Its a simple hack but well worth implementing.

Regards,
Roger Lacroix
Capitalware Inc.

IBM MQ, IBM MQ Appliance, Linux, macOS (Mac OS X), Open Source, Programming, Rexx, Windows Comments Off on Exporting MQ Objects into Individual Files

IBM MQ Fix Pack 8.0.0.16 Released

IBM has just released Fix Pack 8.0.0.16 for IBM MQ
https://www.ibm.com/support/pages/node/6417257

Regards,
Roger Lacroix
Capitalware Inc.

Fix Packs for MQ, IBM i (OS/400), IBM MQ, Linux, Unix, Windows Comments Off on IBM MQ Fix Pack 8.0.0.16 Released

IBM MQ Message Duplication

IBM Support has a technical note labeled Does MQ have a feature to generate automatically a duplicate message?.

The options listed on that page are to use Pub/Sub or programs qload and dmpmqmsg. You can also use MQ Visual Edit and MQ Batch Toolkit to get messages from a queue and write the message to a file. You can even use the open source programs Message Multiplexer (MMX) and/or Msg2File to do it too.

The problem with these programs is that if there are getters (applications performing MQGET API calls) then those programs will NOT capture all of the messages flowing through the queue.

The Pub/Sub solution is ok IF the user does not care about duplicating the message’s MQMD fields. The instructions on this page labeled MQ: You want to put a message into a queue and you want to generate duplicate messages into other queues has the following warning:

NOTE: Strictly speaking, the messages received by Q1 and Q2 are NOT truly identical. Even though they will have the same payload (text string ‘TEST’) and same characteristics, each message will have its own unique message-id.

Notice that the only differences in this particular example are the last 2 characters from “MsgId” and CorrelId”.

The only true solution to EXACTLY replicate a message and its MQMD fields is Capitalware’s MQ Message Replication (MQMR). MQMR will duplicate/clone messages being written (via MQPUT or MQPUT1 API calls) to an application’s output queue and MQMR will write the exact same messages to ‘n’ target queues (‘n’ can be up to 100). When MQMR replicates a message both the message data and the message’s MQMD structure will be cloned. This means that the fields of the MQMD structure (i.e. PutTime, MessageId, CorrelId, UserId, etc..) will be exactly the same as the original message’s MQMD structure.


MQMR can clone messages being written to any MQ queue type. i.e. local, alias, remote and/or cluster queues (including transmission queues).

Existing applications do not need to be modified or updated nor is it necessary to set up Publish/Subscribe to use MQMR. MQMR is deployed and configured at the queue manager level; hence, no application changes are required.

MQMR includes 2 auxiliary programs to help with the management of the cloned/replicated messages:

  • MQ Queue To SQLite DB (MQ2SDB) program will offload MQ messages to an SQLite database.
  • SQLite DB To MQ Queue (SDB2MQ) program will load SQLite database rows into messages in an MQ queue.

Capitalware offers free trials of MQMR including free support, so if anyone would like to try out MQMR then send the email to support@capitalware.com

Regards,
Roger Lacroix
Capitalware Inc.

Capitalware, IBM i (OS/400), IBM MQ, Linux, MQ Message Replication, Raspberry Pi, Unix, Windows Comments Off on IBM MQ Message Duplication

Interesting Blog Item: IBM Knowledge Center – LATEST!

Morag Hughson of MQGem has a new blog posting called IBM Knowledge Center – LATEST! regarding a new feature in the IBM Knowledge Center.

Instead of providing a link with the MQ version number, you can swap the version number for the word “latest”. This is a cool feature for putting links in a blog posting or forums on the internet. The only “gotcha” is that you have to manually edit the link yourself which of course can lead to typos hence an invalid link.

IBM should add a “Latest Link” link on the button bar on each page next to “Print PDF Help” buttons, so that you don’t have to manually edit the link.

Regards,
Roger Lacroix
Capitalware Inc.

HPE NonStop, IBM i (OS/400), IBM MQ, IBM MQ Appliance, Linux, macOS (Mac OS X), Programming, Security, Unix, Windows, z/OS Comments Off on Interesting Blog Item: IBM Knowledge Center – LATEST!

IBM MQ End of Service Dates

So, what version of IBM MQ (aka WebSphere MQ & MQSeries) are you running? Are you behind in upgrading your release of MQ?

Everybody “should be” running IBM MQ v9.0, v9.1 or v9.2, if you want support from IBM.

In case it has slipped your mind, here is a list of IBM MQ releases and there end of service dates:

Note: IBM v9.1 and v9.2 do not yet have end of service dates.

Regards,
Roger Lacroix
Capitalware Inc.

IBM i (OS/400), IBM MQ, IBM MQ Appliance, Linux, Unix, Windows, z/OS Comments Off on IBM MQ End of Service Dates

Reasons MQSC Commands for an AMQP Channel May Not Work

IBM has posted a support document describing the various reasons why an MQSC command may not work for an AMQP channel.
https://www.ibm.com/support/pages/node/6411487

Regards,
Roger Lacroix
Capitalware Inc.

Education, IBM MQ, IBM MQ Appliance, Linux, Unix, Windows Comments Off on Reasons MQSC Commands for an AMQP Channel May Not Work

Using Loki and Grafana with MQ Logs

Mark Taylor of IBM has posted a blog item on using Loki and Grafana software to parse and display information from MQ logs.
https://marketaylor.synology.me/?p=838

Regards,
Roger Lacroix
Capitalware Inc.

IBM MQ, IBM MQ Appliance, Linux, Unix, Windows Comments Off on Using Loki and Grafana with MQ Logs

Local Transactions Using IBM MQ and the JMS API

Olja Rastic-Dulborough has written an article that introduces the user to the concepts of local transactions using IBM MQ and JMS.
https://developer.ibm.com/components/ibm-mq/articles/an-introduction-to-local-transactions-using-mq-and-jms/

Regards,
Roger Lacroix
Capitalware Inc.

Education, HPE NonStop, IBM i (OS/400), IBM MQ, IBM MQ Appliance, Java, JMS, Linux, macOS (Mac OS X), Programming, Raspberry Pi, Unix, Windows, z/OS Comments Off on Local Transactions Using IBM MQ and the JMS API

Join Microsoft and the Java developer community at Devnexus 2021

Microsoft is sponsoring and participating in Devnexus 2021, you can read more about it here:
https://techcommunity.microsoft.com/t5/apps-on-azure/join-microsoft-and-the-java-developer-community-at-devnexus-2021/ba-p/2114247?WT.mc_id=appsonazure-twitter-shboyer

Microsoft is excited to be sponsoring and participating in Devnexus 2021, and you might be wondering—what is Microsoft doing with Java? Quite a bit actually! We’ve been busy releasing new features and functionality to support Java apps on Azure, contributing to open-source projects, and improving our developer tools experience. Read on to find out more about Java at Microsoft and a get a preview of our sessions at Devnexus this year.

Java on Azure

Regards,
Roger Lacroix
Capitalware Inc.

Education, Java, JMS, Programming, Windows Comments Off on Join Microsoft and the Java developer community at Devnexus 2021

Part 2: Customer Question Regarding Automation with MQ Visual Edit

So, continuing with the customer, after they successfully tested MQ Batch Toolkit on Windows with a scheduled Windows batch script, they had a new question:

how can we be sure that the file.txt are send in the proper order ( chronological) from the 1 to last ( by time)

So, here are the batch/scripts files from this blog posting updated to handle getting a list of files by sorted date.

Windows Batch File:

@echo off
setlocal

cd /D C:\Capitalware\MQBT\

for /F %%f in ( 'dir C:\app\some\place\*.txt  /A-D-H /O:D /B' ) do (
  mqbt insert -p MQWT1 -q TEST.Q1 -S -f %%f
  move /Y %%f C:\app\some\place\backup\
)

endlocal

Notes:
– The “/A-D-H” parameter says do not include directories or hidden files.
– The “/O:D” parameter says sort by date.
– The “/B” parameter says only output the file name.

Linux/macOS/Raspberry Pi Script:

#!/bin/sh

PATH="/home/mqm/Capitalware/MQBT:$PATH"
export PATH

cd /home/mqm/Capitalware/MQBT/

ls -tr /app/some/place/*.txt | while read entry
do
  if [ -f "$entry" ];then
     mqbt insert -p MQWT1 -q TEST.Q1 -S -f "$entry"
     mv -f "$entry" /app/some/place/backup/
  fi
done

Notes:
– The “-tr” parameter of the ls command says sort files by time oldest first.

Hope that helps.

Regards,
Roger Lacroix
Capitalware Inc.

Capitalware, IBM MQ, IBM MQ Appliance, Linux, macOS (Mac OS X), MQ Batch Toolkit, MQ Visual Edit, Programming, Raspberry Pi, Windows Comments Off on Part 2: Customer Question Regarding Automation with MQ Visual Edit