Looking through my Referrers I stumbled over this Javalobby thread.

<junitreport> is supposed to work with JDK 1.5 (or Java 5 if you like) in Ant 1.6.2, there has been no change after that in CVS. The very first post in the thread says the task failed with

BUILD FAILED
java.lang.OutOfMemoryError: Java heap space
so the problem is insufficient heap space, not Ant's task. Try to give it more memory, using ANT_OPTS, for example. And come over to user@ant.apache.org ;-)

Java 1.5 comes with XSLTC as its default XSLT processor and maybe this one needs a whole lot of memory, even more than Xalan-J 2? Stephane reported quite a few problems with XSLTC and recommends using a recent version of Xalan-J 2 using the endorsed standards mechanism instead in the Ant bug report:

About XSLTC, I did a quick hack and I'm having a look but...it looks like I need to upgrade to 2GB of memory before getting some results from it. The memory usage is absolutely massive.

Running w/ JDK 1.5B1 (xalan 2.5.2) it jumps straight to 420MB and and I'm transforming a 32.3KB xml...I'm running out of memory (only have a 512MB laptop) about 60s later while it takes less than a second to perform the transformation with JDK 1.4 and endorsed Xalan 2.6.0 (w and w/o xsltc)

path: /en/Apache/Ant | #