[openamq-dev] Named & Persistent Queue Status

[Martin Sustrik]

Hi Matthew,

Before implementing anything you should be a bit more detailed about 
what kind of functionality you expect to achieve using MQ series-syle 
persistent queues.

I expect that you don't trust the messages being held in the memory and 
would like the whole thing to be more reliable. However, it's important 
to know what kind of failures you would like messages to survive. 100% 
reliability is impossible anyway, so you should focus on the problems 
you expect to experience yourself:

1. OpenAMQ crash. Very unlikely. We are running OpenAMQ in production 
for over a year now, passing hundreds of millions of messages per day 
and haven't seen a single crash so far AFAIK.

2. Computer hardware failure. Persisting messages may help here, unless 
the failing component is persistent storage itself. Anyway, messages 
would be recoverable by moving the HD (or whatever) to the different 
box, thus creating message latency of several hours.

3. Power outage. Yes, persistent queues would certainly help here. The 
question is how often do you experience power outages, whether you have 
UPS etc. and whether implementing persistent queues is worth of the trouble.

4. Network failure. This kind of failure is the most common, however, it 
has nothing to do with persistent queues. Instead, recovering from 
network failure requires intricate system of message acknowledging, 
window control, replay on reconnection etc. Also, if you have two 
different network paths from node A to node B you may want to use SCTP 
instead of TCP as an underlying protocol to make use of its multihoming 
(network failover) feature. Actually, we have the code for this, but it 
haven't been made public so far.

Martin

Matthew Alton wrote:
> Hmm...  I must say that I am intrigued if bewildered.
> 
> Tell you what.  If you would be so kind as to provide me with a precise 
> definition of this technique, I will write a library that implements 
> it.  This will be an entirely independent project executed by a crusty 
> old crowgeezer who has long ago forsworn the use of anything more 
> sophisticated than vi, cc, sed, awk, m4, and make.  The library will be 
> licensed LGPL or BSD or whatever.  You could certainly distribute it 
> with OpenAMQ proper if you wish.  I fear that this arrangement is to be 
> preferred to my patching the OpenAMQ code itself owing to my great 
> reluctance to attempt model-oriented programming, whatever that might 
> be.  Twenty-two years of corporate C hacking has instilled in me an 
> abject terror of new coding methodologies.  Anything sufficiently 
> buzzword-laden and visually impressive may be sold to corporate 
> management for any price, my friends -- a law of nature terrifying in 
> its implications.  The stories I could tell...
> 
> Where was I?  Oh, yes.  I will write a library.
> 
> I will also create a detailed tutorial on the use of the library.  This 
> will be copyrighted using some thoroughly permissive species of Creative 
> Commons license, so folks will be free to create alternative versions of 
> the tutorial to suit varying requirements and tastes.  I am informed 
> that many individuals, particularly those whose mother tongue is not 
> English, find my writing style to be abstruse and overwrought.  My 
> manager, for example, is known to print out my lengthier emails and read 
> them with extensive recourse to a dictionary while perched upon the 
> commode.  Managers think best during their constitutionals.  The sudden 
> relief of pressure on their cognitive apparatus is known to bring on a 
> surge of comprehension which, alas, is but mild and brief.  The use to 
> which my printed emails are put afterward is left as an exercise to the 
> reader.
> 
> What was I talking about?  Yes, the library.  If you would be so very 
> good as to provide me with a definitive description of the method of 
> implementing persistent queues using OpenAMQ, I will write a library and 
> a tutorial.
> 
> Tell the truth, now.  Did you use child psychology to induce me to agree 
> to this project?  I'm highly susceptible.  All my manager has to do is 
> to loudly complain that a certain technical feat is completely 
> impossible...
> 
> Peace.
> 
> 
>     On Nov 18, 2007 7:03 PM, Matthew Alton < simplicissimus at gmail.com
>     <mailto:simplicissimus at gmail.com>> wrote:
> 
>      > Interesting.  To be clear, this method of achieving persistence is
>      > necessarily synchronous, yes?  What we are discussing is not
>     equivalent to a
>      > persistent queue a la MQ Series.  The notion of "email for
>     programs" is out,
>      > correct?
> 
>     No, it's not synchronous as such.  The request-ack dialog delivers one
>     message to a destination.  If you want to do a full request + response
>     dialog, you can either combine the response and the ack, or you can do
>     request-ack plus response-ack as four messages.  It sounds like extra
>     overhead but these messages are cheap.
> 
>     To some extent it is equivalent to creating a reliable message queue
>     at the sender side and a reliable ack queue at the recipient side.  If
>     we have this code in a framework (an API layer sitting above WireAPI)
>     then it's effectively free for applications.
> 
>     What you cannot do easily is 1-to-N distribution, i.e. transaction
>     sharing, with reliability, using this model.
> 
>     If you were to add persistent queues to OpenAMQ, you would get a more
>     classic reliability model, with transactions.  The protocol supports
>     this  We did not implement it for a couple of reasons, mainly we found
>     the end-to-end reliability model easier and more robust.
> 
>     -Pieter
>     _______________________________________________
>     openamq-dev mailing list
>     openamq-dev at lists.openamq.org <mailto:openamq-dev at lists.openamq.org>
>     http://lists.openamq.org/mailman/listinfo/openamq-dev
> 
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> openamq-dev mailing list
> openamq-dev at lists.openamq.org
> http://lists.openamq.org/mailman/listinfo/openamq-dev