[openamq-dev] Slow Consumer

[Martin Sustrik]

Malinga,

Slow consumer problem is unavoidable. If the rate messages are produced 
in is greater than rate they are consumed in, the messages have to be 
queued before delivery. Note that this is not dependent on overall speed 
of the consumer. If consumer is able to consumer 100M messages a second 
and producer produces 101M a second, you still have the same problem.

Queueing messages means holding them in the memory. If there is a lot of 
them (and if producer is faster than consumer, the amount will grow 
without limit) the memory gets used out and the broker eventually starts 
swapping thus making the whole process extremely slow.

To prevent this situation there are 2 possible solutions:

1. Use queue limits (drop messages when queue is full, or kill the slow 
consumer connection).
2. Implement message throttling in your producer - meaning that you 
won't send more messages than consumers are able to consume.

Martin

Malinga wrote:
> Hi experts,
>     I have a consumer and recever program for OpenAMQ, they sending 
> message (150 bytes) and receiving.
> If we send 10000 messages thats working fine but, when i increase the 
> number of messages to 10000000
> (10 M), and running the two sets of consumer and producer, (so it will 
> be 20 M messages with two publishers and two consumers for two different 
> queues). So then after couple of minuets amq server consoles says,
>  slowing consumer and it takes up lots of memory. So the consumers take 
> soooo long to receiver the messages.
> 
>     I know in all the messaging systems they have this slow consumer 
> issue. But my point is its just the basic
>  consumer and receiver program just do nothing bu the send and receive. 
> so can you tell me what the problem
>  would be (if there is a issue) ?. Since we have high volume data 
> transfer we migh get those issues in the later stages of the product.
> 
> -Malinga
> 
> ------------------------------------------------------------------------
> Catch up on fall's hot new shows 
> <http://us.rd.yahoo.com/tv/mail/tagline/falltv/evt=47093/*http://tv.yahoo.com/collections/3658 
>  > on Yahoo! TV. Watch previews, get listings, and more!
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> openamq-dev mailing list
> openamq-dev at lists.openamq.org
> http://lists.openamq.org/mailman/listinfo/openamq-dev