Orion v. Tomcat microbenchmark

Last night, Gerald Quakenbush (of Mastermind Security Group) and I had a discussion about security. He mentioned FTD's security woes from a few years ago (2003), reported on ABC and a few other major news outlets, based on their use of a predictable shopping cart ID, and that got me thinking about J2EE container security as a whole (sorry, bad pun.)

Gerald mentioned that WebSphere was generally okay in terms of session id generation, as was Tomcat, although he said - without going into any detail - that WebLogic had been known to have some issues in the past. (Note that he was talking historically, so this shouldn't be taken as an indictment of anyone's current product suite.)

Since I use Orion personally, when Gerald mentioned session hijacking, I instantly wondered how Orion managed. Gerald and I then cranked up Paros to see what Orion sent across the wire for sessions.

Lo and behold, session ids in Orion are twelve characters, and in the few tests we ran, we saw identifiers like this: AAAALFLAPBPA and, subsequently, AAAMGLDGPBPA. Note the close representation, in particular the beginning ("AAA") and ending ("PBPA") sequences.

So I resolved to write a microbenchmark, to submit 100,000 requests to Orion, to try to determine if there was a repeatable or predictable period to the sessionids. I wrote a JSP file that did nothing more than invalidate the current session, so that a client would, if hitting that JSP over and over again, simply get a multitude of session ids.

Then I wrote a client that started up one hundred threads that simply hammered the server, with a small (30-80ms) delay between requests. Due to the nature of the test, I chose to use HTTP 1.0, as opposed to HTTP 1.1, which would have been far faster.

I was surprised by the results. Most importantly, I did not get a repetition in Orion's sessionid generation. I'm dissatisfied with the session identifiers, but the period certainly exceeds 100,000 sessions, which is a good start.

However, Orion's average response time of 326ms worried me, as I've done benchmarks on Orion before that performed far, far better.

So the next thing to do was, of course, to crank up Tomcat 5.5. After a bit of tweaking (to match the memory settings I used on Orion), I reran the test: Tomcat's average response time (over 100,000 requests) was 441ms, with the server settling into a pattern of repeated full GCs starting at about halfway through the test.

At no point was thread contention an issue, although it looks like Tomcat's sessions and their invalidation code are very heavy.

I also found that I really, really like the J2SE 5.0 VM. The language changes... well, they're okay, even convenient in places, but I can take or leave them. The VM changes - in particular, the monitoring tools - are incredibly useful (although I should note that I did not have monitoring on when running the tests.)

To be sure, this is a microbenchmark, a very specific one - one whose exact characteristics are incredibly unlikely in the real world, and tomcat's numbers are negatively affected by my decision not to allocate a larger maximum heap to it. I'm not sure by how much yet.

What's more, the original thrust of the test - to check session ID generation and period - has not been concluded. Orion did not generate duplicates, although it generated near duplicates, as shown before - and the IDs are not specifically predictable. I don't know that that's enough, and I'll be running a longer test - Gerald suggested a million hits, and my personal paranoia suggests a larger number - to find out what the distribution is.