Thursday, May 5, 2011

Benchmarking HornetQ 2.2.2 Paging Mode

We've been using HornetQ 2.1.2 for some time now where I work and I was thrilled to find that several of the problems I have had were addressed in the newest release. One of the biggest production issues for me was the drop in performance when entering paging mode.

If you're adding messages to HornetQ faster than you can consume them, at some point it wont be able to keep them all in memory at a the same time. I chose to let HornetQ page them to disk so nothing get's dropped. Although HornetQ's paging performance is the fastest on the market, it's still a big performance hit which for me maxed out the Disk IO. It was at that point I realized I'd under-spec'ed my server needs because my speed tests were based on not paging. My fault, not HornetQ's.

Clebert informed me that the newest version of HornetQ has an optimization where it doesn't have to sync to disk for every single message that gets paged. I wrote a quick & dirty benchmark so that I could compare paging performance of 2.1.2 vs 2.2.2 and see if I could justify the upgrade. Unfortunately I can't share the code b/c it's intertwined with our app.

The upgrade was easy. All the config changes were additive as far as I can tell, so I was able to drop my old configs on top of the new version. Since I'm instantiating the connection factory directly in code, there were some slight changes in the constructors for those. I changed 10 lines of code and was ready to rock. Since my benchmark has no consumers and I have HornetQ set with a small heap, the server quickly goes into paging mode.

Old version, 10 producer threads, no consumers, 100K messages
6:05 = 277/sec

New version, same deal
1:08 = 1470/sec

That's more than a 5 fold increase. I'm pretty thrilled to get that much more mileage out of each HornetQ server in that situation. This wasn't the most scientific test in the world, but I just needed to get a ballpark and I figured I'd share my numbers. If you've got any numbers to share, please post a comment!

For those interested, these were persisted byte messages, each under 1k on 1 large EC2 instance.

No comments:

Post a Comment