« Spring is here (again) | Main | Your Money or Your Life? »

July 30, 2004

Engine strip-down and rebuild

It all started so innocently.... bug #69319 "JDK 1.4 assertions cause VerifyError". VerifyErrors indicate one of two things - either a serious bug, or misuse of the -XnoWeave option (use -Xreweavable instead). This one was a real bug.

The test case was a pure Java program, and it looked unlikely to me that it would be a bug in AspectJ itself, so I ran the test case against the Eclipse 3.0 M6 compiler (which is the version inside AspectJ 1.2). Sure enough, it failed in the same way. Trying again with the Eclipse 3.0 final compiler, the test passed. So, the only way to fix this bug was to tackle a task I've been putting off for a couple of weeks - and upgrade the Java compiler inside AspectJ.

For those of you who aren't familiar with AspectJ's internals (I guess that's most of you...), the AspectJ compiler is built as an extension of the Eclipse JDT Java compiler. That gives us a good, robust, high-quality (ahem, apart from this bug.....) Java compiler as as starting point and avoids the AspectJ project team wasting precious resources building yet another Java compiler implementation. The Java compiler sources that we take from org.eclipse.jdt.core are in a carefully managed part of the AspectJ source tree. Moving to a new version of the compiler is analagous to doing a full strip-down and rebuild of a car engine....

I started by bringing across a copy of the Eclipse 3.0 final Java compiler sources. These are checked into the main tree, and then work proceeds in a branch (it makes subsequent merges easier when we come to upgrade the compiler again). Starting from the beginning - with the grammar files and parser, and slowly building up, I added back in all of the extension points that AspectJ exploits in order to add in its extra behaviour. We try and keep the changes in the actual Java compiler source as minimal and as clean as possible - while it's by no means as simple as a couple of well-defined interfaces, it's clean as it can be in the circumstances given that we stretch the compiler beyond its original design goals. With the base work done, the AspectJ compiler extensions can be layered in on top - carefully checking at each stage that assumptions which held for previous versions of the compiler are still true in the new version, and updating the code in places where the Java compiler interfaces have moved.

At this point in the process, the full AspectJ test suite will run (but probably not pass!). The final piece of the puzzle is to work your way through the failing tests (thankfully, the AspectJ test suite gives us pretty good coverage after several years of development), figuring out what's changed and why, and any implications that may have.

With all the tests passing, I checked the results back into the tree - major upgrade completed. This represents a big chunk of the work I wanted to do for a 1.2.1 release. I was so pleased with myself I forget to run the test case that started all this off in the first place - bug 69319. I only remembered as I sat down to write this blog entry. I just added it to the test suite and ran it. It passes, phew :)

Posted by adrian at July 30, 2004 07:04 PM [permalink]

Comments

Adrian,

maybe I'm mising some dependencies, but is it not possible to use the previous version of aj to add the aspectj hooks into the new (binary) version of the compiler?

it would be a nice dogfood exercise if it was possible. maybe the changes are aspectual in nature and hence not a good idea.

thoughts?
Ken.

Posted by: Ken Horn at July 31, 2004 01:47 PM

err, I meant NOT aspectual in nature...

Posted by: Ken Horn at July 31, 2004 01:48 PM

Ah, this reminds me of when I was at Sun working on the Prism parallel debugger. Whenever the dbx (scalar) debugger team would upgrade its stabs code to the latest compilers, we'd have to remerge in all of the changes in a manner not to different from what you're doing. (It was nice to pick up the bigfixes "for free.")

Given that both projects are under the Eclipse umbrella, is there any chance of getting your hooks putback into the mainline JDT compiler? Our goal for Prism was to eventually give them a test suite against what they had, and for them to notify us if anything broke (to let us know ahead of time of major changes). It never got that far, but the more communication we had between the two teams, the better things went.

Posted by: Macneil Shonle at July 31, 2004 07:48 PM

The question about using aspects to help make the extensions to the core compiler is a good one. I inherited a historical decision not to use Aspectj in the writing of AspectJ itself. This decision was made way back, 0.8 release or before, when the language was young and the PARC team didn't want to influence its design too heavily towards compiler writing. The original reasons have long since gone, but the decision was never changed. There are quite a few places (surprise!) in the AspectJ codebase where I'd like to use AspectJ to improve the modularity. I hope to start phasing in the use of AspectJ in some of the compiler modules over the course of this next phase of development. In this point case, using aspects would help to ease the compiler integration, but not alleviate the need entirely for some changes in the JDT core.

Posted by: Adrian at August 6, 2004 07:49 AM

In answer to Macneil's question... yes, the goal is that the extension points we need in the JDT compiler will eventually make their way back into the JDT codebase itself, so that over time we have to do less and less. We are very careful to ensure the changes we make in the JDT layer introduce no dependencies on AspectJ classes so that this will be possible.

Posted by: Adrian at August 6, 2004 07:52 AM

Post a comment