I am always looking to tune Capitalware’s applications . After reviewing MQ Auditor, it occurred to me that I could save 8-12 ms (milliseconds) per audit record that MQ Auditor writes to the audit files. Now, 8-12 ms per audit record may not sound like a lot but if your queue manager is processing a million or more API calls per hour, milliseconds do add up.
So, I came up with a design for a new companion application for MQ Auditor called Audit Queue Off Load (AQOL). To use the new AQOL application, the user will activate the audit queue feature within MQ Auditor. Hence, all audit data will be written to the audit queue and will be processed by the new AQOL application.
Of course, everyone’s first thought is that “won’t putting all of the audit messages to a queue slow things down?”. Normally, I would say yes, but there are 2 reasons why in this case the answer is no.
- The messages with the audit data are non-persistent messages. Hence, they are not written to MQ’s log files. Also, if there is a consumer waiting to get a message then the messages are not written to the queue file either. Hence, the message data will only reside in memory.
- I designed the AQOL application to be a “massively parallel” application. The application is broken into 2 components: front-end (MQ Getter) and back-end (File Q) threads. The MQ Getter threads (default of 8 ) are used to get messages off the audit queue as fast as possible and place them on the internal queues. The File Q threads (1 per open file) processes the internal queue and writes the audit data to a particular file.
Internal Object Relationships:
- The relationship between the MQ queue (CAPITALWARE.AUDIT.QUEUE) and the front-end MQ Getter threads is one-to-many.
- The relationship between the front-end MQ Getter threads and the internal queues is many-to-one.
- The relationship between the internal queues and the File Q threads is one-to-one.
- The relationship between the File Q threads and the actual audit files is one-to-one.
On low volume queue managers, 8 MQ Getter threads may be enough but for busy queue managers, the user may need 50 or 75 MQ Getter threads. The idea is to never allow messages to pile up on the MQ audit queue.
The File Q threads are totally controlled by the number of processes and threads that are connecting to the queue manager. If you have 100 processes connecting to the queue manager, then the AQOL application will have 100 File Q threads running to write the audit data to the 100 files. If, on a very busy queue manager, you have 1000 processes connecting to the queue manager, then the AQOL application will have 1000 File Q threads running to write the audit data to the 1000 files.
There is a timer thread within the AQOL application that is used to place an “inactivity message” on each internal queue. The inactivity message causes each File Q thread to check if there has been any file activity within the last 20 minutes (user changeable). If there has not been any file activity within the last 20 minutes, then the audit file is closed and archived.
The AQOL application has been designed to be used via the MQ Service, so that AQOL will automatically start and stop when the queue manager starts and stops.
As I said, the new AQOL application is a “massively parallel” application. 🙂
Does anyone want to beta test it?
Regards,
Roger Lacroix
Capitalware Inc.
