[openamq-dev] Named & Persistent Queue Status

[Matthew Alton]

Interesting.  To be clear, this method of achieving persistence is
necessarily synchronous, yes?  What we are discussing is not equivalent to a
persistent queue *a la* MQ Series.  The notion of "email for programs" is
out, correct?

At this point you may reasonably pound your fists on the table and scream,
"OpenAMQ is not MQ Series, you philistine!!"  It's quite alright.  I'll
wait...

Okay, now that's over.  Let's discuss how I can distort OpenAMQ into MQ
Series.  I really truly do need asynchronous persistent queues in my work.
I have a whole lobe of my brain dedicated to the "fire & forget" paradigm.
I really truly desperately want to use OpenAMQ for my purposes.  My trouble
at present is that I can't see how the "request + ack" method gets me
anywhere beyond simple TCP/IP socket programming.  My sender client can't
"forget" about the message until receipt is confirmed by the receiver
client.  Ergo they must be communicating synchronously.  TCP/IP gets me that
much without even requiring an ack operation on the part of either process.

How does the naive method of implementing persistent queues on the server
imply egregious complexity?  I do not claim that it does not imply the
complexity, I just don't yet understand how the complexity logically follows
from the technique.  The semantics of the truly persistent server-maintained
queue seem, to my innocent sensibilities, to consist entirely in saving
messages to reliable storage for later retrieval.  I venture this notion
with the disclaimer that I am only three chapters into my transaction
processing principles book.  For all I know, chapter four may lay bare the
whole issue.

I believe that there exists a demand for a true open source MQ Series
replacement.  It may be that OpenAMQ is the right answer.  I hope so.  But
at present, OpenAMQ seems to require at best a rather drastic shift in
thinking and a corresponding change in methodology from the status quo.  I
believe that persistent, named, server-maintained queues would go a long way
toward facilitating adoption of OpenAMQ in the marketplace.  I don't believe
that their addition to OpenAMQ would complicate the code at all.  The trick
seems to me to lie in the correct division of duties.  Assume that storage
and network are reliable and you're done.  These assumptions are inherent in
software itself.  They're already made.  A healthy recognition of their
ineluctable nature provides the refreshing simplicity that we crave.

As to the objection that persistent queues would harm the performance of the
system, yes, absolutely they would.  So long as we are synchronously
flushing I/O caches to disks, the laws of physics dictate that we take a
performance hit.  This is a consideration for the application designer,
however, and need not effect the OpenAMQ system itself.  The OpenAMQ system
should be designed so as to incur only such performance degradation as is
utterly unavoidable.  If a non-persistent queue is defined on the same
instance as persistent ones, it should not suffer any performance
degradation at all.  Network and CPU should not wait on disks.

I would love to help with the implementation of persistent queues in
OpenAMQ, but, alas, I know nothing of model-oriented development.  I am at
present still kicking the tires and scratching my head.  I really want this
thing to work somehow.

I'll stay at it.

Peace.


On Nov 17, 2007 5:05 AM, Pieter Hintjens <ph at imatix.com> wrote:

> On Nov 16, 2007 10:55 PM, Matthew Alton <simplicissimus at gmail.com> wrote:
>
> >   I have a question for you about current and planned features in
> > OpenAMQ.  My research has turned up conflicting information on whether
> > classical persistent queues are being implemented.
>
> The web page is somewhat optimistic.  We have focussed, and will
> continue to focus, on making the software as simple and reliable as
> possible, and add functionality only when there are users demanding
> (and hopefully willing to pay) for it.
>
> There is a simple way to do persistent messaging, and this should
> probably be documented as a tutorial.  This is how we do large-scale
> reliable messaging over OpenAMQ in deployments that need it.
>
> First, we accept that the broker can in theory crash or disappear and
> lose messages.  We construct our reliable messages as a request + ack
> pair, from sender to recipient.  That is, one message to the
> recipient, and a response back.  Senders store their outgoing messages
> in a databaseand if they do not receive a response back within a short
> time, they resend the request.  Recipients, if they get the same
> request again, just resend the response but don't do the work again.
>
> This should probably be built as a simple reusable framework around
> wireapi, using sqlite or something.
>
> The advantages of this model, over putting reliability inside the
> broker, is that it works across any kind of AMQP broker, across
> OpenAMQ clusters, federations, etc.
>
> The disadvantage is that it needs extra libraries - which we have not
> yet made - to be really easy to use.
>
> The approach is very different from that used by MQ series, where the
> queues themselves are persistent and reliable, and the brokers are big
> and fat and complex.
>
> >   Also, I have access to FreeBSD/SPARC64, AIX and HP-UX machines with
> > C compilers.  Would it be helpful if I were to try to build OpenAMQ on
> > these machines?
>
> Yes, the more systems we test on, the better.  Afaik people have
> tested on AIX but not on HP-UX.
>
> -Pieter
> _______________________________________________
> openamq-dev mailing list
> openamq-dev at lists.openamq.org
> http://lists.openamq.org/mailman/listinfo/openamq-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.openamq.org/pipermail/openamq-dev/attachments/20071118/1cbc527f/attachment.htm