[Bernstein09] Section 4.8. Queuing Products and Standards

4.8. Queuing Products and Standards
Avariety of queue manager products are available. One of the originalimplementations was in IBM’s IMS TP monitor, where queued TP was thedefault behavior. Queuing is integrated with many other transactionalmiddleware products, such as Oracle’s WebLogic and JBoss Messaging. Itis also integrated in Oracle Database, called Oracle Streams AQ, and inWindows, called Microsoft Message Queue (MSMQ). Some vendors offerqueuing in independent products, such as TIBCO’s Enterprise MessageService, Progress’ SonicMQ, Apache ActiveMQ, and IBM’s Websphere MQ (MQ= Message Queuing). A consortium sponsored by JP Morgan Chase hasproposed a messaging standard, called Advanced Message QueuingProtocol. We briefly describe WebSphere MQ and Oracle Streams AQ hereas two examples of such products.
IBM’s Websphere MQ
IBM promotes WebSphere MQ[1]as an integration solution among its various operating system and TPenvironments and those of other vendors. It has a proprietary API,called Message Queuing Interface (MQI), a Java Messaging Service (JMS)API, and a non-Java equivalent of JMS. It can be used by applicationsrunning under IBM’s transactional middleware such as WebSphereApplication Server and CICS Transaction Server, and on any operatingsystem supported by WebSphere MQ, including IBM AIX, i5/OS, OS/400, andz/OS, as well as HP-UX, Linux, Sun Solaris, and Microsoft Windows.
The information in this section is based on WebSphere MQ V6.0.
The WebSphere MQ queue manageraccepts input from an application via the JMS API or MQI verbs. Themain verbs are MQPUT to enqueue a message and MQGET to dequeue amessage. A named queue can support multiple concurrent enqueuers anddequeuers.
Toprocess an MQPUT, the queue manager starts a transaction if theapplication is not already executing one and places the message in thequeue. The operation is committed along with the rest of thetransaction (which can be the normal exit from the application) or canoptionally run in its own transaction as described in the subsection onnontransactional queuing in this section. The enqueued message consistsof application data and the message context,including a variety of parameters, such as a system-generated messageID, a flag indicating whether the message is persistent, a messagepriority, the name of the destination queue when forwarding, the nameof the reply queue (if any), message type (datagram, request, reply,report), correlation id (to link a reply to a request), priority,expiry time, application-defined format type, code page identifiers(for language localization), context information (to identify the userand application that generated the message), and report options—whetherthe recipient should confirm on arrival (when it’s enqueued), ondelivery (when it’s dequeued), on expiration (if the expiry time isexceeded), on positive action (the application successfully servicedit), on negative action (the application was unable to service it), oron exception.
Amessage that is oversized for the queue manager or application can bedecomposed into smaller segments. Moreover, several messages can beassigned to a group, which allows the application to correlateindependent messages, such as those that arrive from different sourcesbut must be processed by the same application.
Anapplication can request that MQI operations participate in atransaction. Otherwise, by default, each individual MQPUT or MQGETexecutes outside a transaction, meaning that the operation completesimmediately whether or not the application is executing a transaction.
WebSphereMQ offers several transaction management options for applications thatare running within a transaction. If the only transactional operationsare MQI operations, then the transaction can be managed as a localtransaction by MQ. If the transaction needs to access othertransactional resources, then MQ can play the role of a resourcemanager under an external transaction manager, such as the JavaTransaction API in Java EE. If no external transaction manager ispresent, then on non-mainframe platforms MQ’s XA-capable transactionmanager can coordinate the transactions across MQ and databases.
Likemany queuing products, WebSphere MQ offers the ability to enqueuepersistent and nonpersistent messages in the same queue. Nonpersistentmessages are more efficient but less reliable. They do not incurlogging overhead and normally are handled in main memory, without beingwritten to disk. Both types of messages obey transaction semantics.However, a persistent message is delivered exactly once, whereas anonpersistent message is delivered at most once; that is, once (in theabsence of failures) or not at all (if there is a failure).
Queueforwarding is handled by another component, which is much like anordinary client that does MQGET from one queue manager and MQPUT toanother, though it does have special access to the log for its sequencenumber management. So if MQPUT has a destination queue name that mapsto a remote queue, this component forwards the message asynchronouslyand transactionally, using an intermediate node if necessary, to thesystem on which the remote queue exists (seeFigure 4.12).The queue forwarding component uses a transaction that’s internal to MQto coordinate updates to the source and target queues.
Figure 4.12. WebSphere MQ Architecture. Messages can be forwarded transparently between queue managers running on different platforms.

Anapplication issues an MQGET to dequeue a message. The queue managerstarts a transaction upon receipt of an MQGET verb, dequeues themessage from the message queue, and upon the commit from theapplication, physically removes the message from the queue. If thetransaction aborts, the queue manager returns the message to the queue.MQGET supports a blocking option, which blocks the caller if the queueis empty and awakens it when a message is available or a timeoutexpires. It also supports a signaling option, where the caller cancontinue executing and is notified when the desired message arrives.Messages can be retrieved in order or by searching for a given ID orkey. The queue can also be browsed, to examine the messages of thequeue without deleting them. WebSphere MQ also includes a dispatcherthat triggers the execution of an application when the first messagearrives on a queue, whenever a new message arrives, or when the queuelength reaches a predefined threshold.
WebSphereMQ supports multiple named queues per queue manager. Each queue managerhas the following components: a connection manager for managing theconnections between the application and the queues; a message managerfor remote communications; a data manager to manage the physicalstorage of the linked lists comprising the queue; a lock manager forlocking queues and messages; a buffer manager for caching data,ordering writes, and flushing data to disk; a recovery manager forkeeping track of active transactions in the event of a failure/restart;and a log manager for handling the recovery log. The component namesdiffer slightly in different products.
Features of most message queuing systems that support JMS are similar to WebSphereMQ.
Oracle Streams AQ
Unlikemost queue management products, which are independent middlewarecomponents, Oracle Streams AQ is a queuing facility that is built intoOracle Database.[2]It is built on top of Oracle Streams, which enables the propagation andmanagement of information in data streams, either within a database orfrom one database to another. AQ can be accessed from most popularprogramming languages via APIs for PL/SQL, Oracle Call Interface,Oracle Objects for OLE, and extended versions of JDBC and JMS thatprovide access to Oracle-specific features such as those in AQ. It alsooffers web-based access via SOAP through the AQ XML Servlet.
The information in this section is based on Oracle 11 g.
InOracle Streams AQ queues are mapped to a table that can be accessedusing the standard types of queuing operations, such as enqueue anddequeue. Since queued messages are stored in a table, they can also beaccessed by SQL queries.
Oracle Streams AQ is a complete queuing system offering most of the capabilities described inSection 4.5.The enqueue operation takes a queue name, payload, message properties,and enqueue options as input and returns a message ID. Messageproperties and enqueue options control the behavior of the enqueueoperation. For example, using message properties, the sender cancontrol the earliest time when a message is consumed, whethera message is volatile or persistent, the retry threshold after which apoison message is added to an exception queue, whether the operation istransactional, and priority ordering.
Theentire history of information about a message is maintained along withthe message itself. This serves as proof of the sending and receipt ofmessages and can be used for nonrepudiation of the sender and receiver.The history includes the name of the agent and database that performedthe enqueue or dequeue operation and the time and transaction ID of theoperation. After the message is propagated to the destination queue, itstill includes the message ID of the source message so that the sourceand destination messages and their histories can be correlated.Stronger nonrepudiation can be achieved by storing the digitalsignature of the sender and receiver.
The following are some additional features worth highlighting:
A message can be enqueued with an explicit set of recipients, which overrides the list of subscribers to the queue.
A caller can batch multiple items in an enqueue or dequeue operation, which is less expensive than enqueuing or dequeuing the items one by one.
A consumer can dequeue a message without deleting it from the queue based on the queue’s retention policy. The first dequeue runs as a select query, which returns a snapshot of the messages to be dequeued. Subsequent dequeues within the same transaction are performed on the same snapshot without issuing a new select.
A sender can split a complex message into a message group, which the consumer can process atomically.
A caller can listen to multiple queues, waiting for a message to arrive. If the listen operation returns successfully, then the caller must issue a dequeue to retrieve the message.
A caller can dequeue a message without retrieving the message’s content. This is useful for deleting a large message whose content is irrelevant.