Archive for December, 2010

links for 2010-12-09

ActiveMQ – not what it used to be?

I’ve posted a quick update to this here.

When I started this job a couple years ago, we were mildly invested in ActiveMQ with a single queue for some mission critical work. Since I’d had good experience with ActiveMQ 4.x in previous jobs, I didn’t hesitate to push forward and suggest we invest more in ActiveMQ and architect solutions around it’s usage.  After all, I had experienced flawless performance and up time with 4.x.

Forward 2 years: We now have 30 or so topics and queues with moderate usage and it’s incapable of staying up more than a few minutes when configured as a network of brokers.

We’ve gone so far as paying the big bucks to get consulting from the folks at FuseSource. Unfortunately, the suggestions and help have been … uh … expensive with no results at all.

The setup

The setup is relatively straightforward:

  • 3 brokers in different cities in different countries which need to work as a network of brokers.
  • Latency can be high (80ms)
  • Java clients using the client libraries over the tcp protocol
  • C++ clients using the ActiveCPP library
  • A few dozen topics and queues
  • No big messages (5MB max with the average being closer to a few hundred k)
  • Moderate traffic (a few hundred thousand messages per day overall)

Everything performs well enough in a single broker setup, but that’s un-acceptable for us.

What was done

Pretty much everything :)

Our expert had us upgrade to the latest version (5.4.2) with horrendous results; the KahaDB would continuously corrupt. Let me say this about trying to fix a basic problem with a minor upgrade: It’s bullshit. If version 5.4.1 can’t do something as straightforward as we’re trying to do, some suggestion about upgrading a point release isn’t going to do anything.(edit: The language was harsh. As it turns out, the minor upgrade was to adress another issue and not the network of brokers issue). And don’t be mistaken; our usage is basic and reasonable and definitely falls into the advertised set of features touted by ActiveMQ.

What’s next

We’re throwing all our weight behind JBoss HornetQ right now. The team is pretty psyched because they’re all fed up and disillusioned with the ActiveMQ product. Obviously, the migration path will not be easy, but at least we should no more be prisoner to a buggy product. (edit: Again, harsh: I’d say ‘prisoner to a product that doesn’t work in our configuration’)