Brent Boyer has written a comprehensive set of articles on benchmarking Java code.
Robust Java benchmarking, Part 1: Issues, Understand the pitfalls of benchmarking Java code
Robust Java benchmarking, Part 2: Statistics and solutions, Introducing a ready-to-run software benchmarking framework