[openamq-dev] amq_server on AIX with amq.topic exchange crashes after 60K+ messages

[Dmitri Tsyganov]

Yes, this seems to be memory. Our reader isn't reading fast enough from
queue and amq_server memory usage rapidly grows till process dies. I
will have to look into how to speedup our reader and how to configure
amq_server to be more sustainable to this kind of situations. 

What is strange is that our reader does not do much, it just sits in a
loop and waits for content from the queue to arrive. Then it copies a
content and destroys it. I wonder why it is so much slower then the
writer. Things much better when I have reader running on the same box,
maybe it is network...

Thanks,

Dmitri

On Tue, 2007-09-11 at 12:07 +0200, Martin Sustrik wrote:
> Dmitri,
> 
> This looks like a memory issue.
> 
> What happens is:
> 
> tcb->method_in->content = amq_content_basic_new ();
> if (amq_content_basic_record_header (
>     (amq_content_basic_t *) tcb->method_in->content,
>     tcb->bucket_in)) ...
> 
> It looks like the first statement amq_content_basic_new (constructor)
> returns NULL (not enough memory available).
> 
> Then amq_content_basic_record_headers is called which asserts whether
> first argument is NULL aborting the whole process.
> 
> Can you run the broker in debugger and check out if this is the case?
> 
> Martin
> 
> On 10/9/2007, "Dmitri Tsyganov" <dmitri.tsyganov at dowjones.com> wrote:
> 
> >Hello,
> >
> >I am running release multi-threaded version of amq_server (v 1.2c3) on
> >IBM AIX 5.
> >
> >I am using a client (running on Linux) to post messages to a topic
> >exchange (amq.topic) with a routing key that looks like this:
> >
> >	US.NASDAQ.STOCK.msft (last part - symbol - varies from message to
> >message)
> >
> >I use another client application (also running on Linux), which
> >subscribes to topic exchange and receives messages.
> >
> >Sender client send messages as fast as it can and receiver tries to read
> >them as fast as it can.
> >
> >After successfully transmitting about 60,000 +- messages, amq_server on
> >AIX crashes. It crashes consistently in the same spot.
> >
> >Here is a result of error message I see on amq_server console:
> >
> >Assertion failed: self, file  amq_content_basic.c, line 674
> >IOT/Abort trap (core dumped)
> >
> >And here is core dump:
> >
> >IOT/Abort trap in pthread_kill at 0xd0128470 ($t9)
> >0xd0128470 (pthread_kill+0x88) 80410014        lwz   r2,0x14(r1)
> >(dbx) where
> >pthread_kill(??, ??) at 0xd0128470
> >_p_raise(??) at 0xd0127f08
> >raise.raise(??) at 0xd0315b60
> >abort() at 0xd03450b0
> >__assert_c99(??, ??, ??, ??) at 0xd03af1e0
> >amq_content_basic_record_header() at 0x1008fbc8
> >action_expect_content_basic_header() at 0x100caa30
> >amq_server_agent_manager() at 0x100d75f0
> >s_execute() at 0x10055dc4
> >dummy_worker(opaque = 0x203c51e0), line 138 in "thread.c"
> >(dbx)
> >
> >Could somebody take a look at this and have an idea what may be wrong
> >there? I didn't notice any excessive memory usage on the server side.
> >
> >Thank you,
> >
> >Dmitri
> >
> >_______________________________________________
> >openamq-dev mailing list
> >openamq-dev at lists.openamq.org
> >http://lists.openamq.org/mailman/listinfo/openamq-dev
> _______________________________________________
> openamq-dev mailing list
> openamq-dev at lists.openamq.org
> http://lists.openamq.org/mailman/listinfo/openamq-dev