« January 2005 | Main | April 2005 »

March 22, 2005

Hacking with Harrop...

(Rob Harrop that is, of Spring fame...)

Off the back of the AOSD 2005 library meeting, Rob Harrop and I managed to spend a happy few hours pair programming - made all the more interesting for Rob by the fact that I use a dvorak keyboard layout and occassionally forgot to switch it back to qwerty before giving him the keyboard....

By the time we were done, we could configure any aspect type (whatever the instantiation model) from Spring simply by using the @Bean annotation and defining a beans.xml file. In fact, we could configure any type with the @Bean annotation. We made it so that in the simplest case there is no need for the programmer to explicitly initialise spring - aspects do that for you when first detecting a type with the @Bean annotation, all you have to do is have one or more beans.xml files on the classpath. Oh, and did I mention that any type (including aspects) configured this way can be auto-exported as MBeans and managed via JMX? (no user code involved). This represents a pretty neat solution for configuring and managing aspects.

As ever, I'll sketch out the basics of the implementation for you here...

I'll use as an example an abstraction of the common session factory / session / worker pattern that crops up again and again. A worker class performs work that takes place in a session. Sessions are created by the session factory. Here's my Worker class:

public class Worker { public static void main(String[] args) { Worker worker1 = new Worker(); Worker worker2 = new Worker(); worker1.doWork(); worker2.doWork(); worker1.doWork(); } public void doWork() { Session session = SessionManager.aspectOf().getSession(); System.out.println("Doing work in " + session); doWorkHelper1(); doWorkHelper2(); } private void doWorkHelper1() { Session session = SessionManager.aspectOf().getSession(); System.out.println("Doing work in helper 1 for " + session); } private void doWorkHelper2() { Session session = SessionManager.aspectOf().getSession(); System.out.println("Doing work in helper 2 for " + session); } }

Worker has a convenience main method that creates two workers and asks them to do work. Note that the doWorkHelper1() and doWorkHelper2() methods are called within the cflow of the doWork method. In each method we print out the current session (from the SessionManager aspect). The SessionManager aspect is a percflow aspect that creates a new aspect for each control flow through doWork....

@Bean("SessionManager") public aspect SessionManager percflow(session()) { private Session session; private SessionFactory factory; /** * SessionFactory to use is configured by DI */ public void setSessionFactory(SessionFactory factory) { this.factory = factory; } /** * Make the current session available to anyone who needs it */ public Session getSession() { return session; } /** * A new SessionManager (and session) begins whenever * we execute doWork */ pointcut session() : execution(* doWork(..)); /** * Create a session using the factory that was passed to * us when we were configured */ before() : session() { session = factory.beginSession(); } /** * Close the session at the end of the session() control * flow. */ after() : session() { session.close(); } }

Note that the SessionManager aspect is annotated with @Bean, and has a setSessionFactory(..) method that is to be called by an IoC container to pass it the session factory to use.

Here's the Spring beans.xml file for the project. Note that this is 100% standard Spring, no funny stuff.

<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE beans PUBLIC "-//SPRING//DTD BEAN//EN" "http://www.springframework.org/dtd/spring-beans.dtd"> <beans> <bean name="SessionManager" class="org.aspectprogrammer.test.SessionManager"> <property name="sessionFactory"> <ref bean="SessionFactory"/> </property> </bean> <bean name="SessionFactory" class="org.aspectprogrammer.test.SessionFactory"> </bean> </beans>

Just compile this with the Spring AspectJ library on your aspectpath and run main:

Doing work in session #1001 Doing work in helper 1 for session #1001 Doing work in helper 2 for session #1001 Doing work in session #1002 Doing work in helper 1 for session #1002 Doing work in helper 2 for session #1002 Doing work in session #1003 Doing work in helper 1 for session #1003 Doing work in helper 2 for session #1003

We created 3 SessionManager aspect instances, one per control flow, and each was correctly configured by Spring. In true Spring style, note that we did this without using one single Spring API in our program

To get this to work, we used a couple of library aspects as follows. The abstract BeanConfigurator aspect will configure (and validate) any object whose type is annotated with @Bean:

/** * Configure and validate any object whose type is annotated with @Bean. * Validation occurs for any type implementing the ValidateConfiguration * interface. To be subclassed for each IoC mechanism you want to * support (eg. Spring, HiveMind, Pico, JDK, ...). */ public abstract aspect BeanConfigurator { /** * We only validate @Beans... */ declare warning : staticinitialization(ValidateConfiguration+) && !staticinitialization(@Bean *) && !staticinitialization(ValidateConfiguration) : "Implementors of ValidateConfiguration must have the @Bean annotation"; /** * The creation of a new bean (an object with the @Bean annotation) */ pointcut beanCreation(Bean beanAnnotation,Object beanInstance) : initialization((@Bean *).new(..)) && @this(beanAnnotation) && this(beanInstance); /** * All beans should be configured after construction. */ after(Bean beanAnnotation, Object beanInstance) returning : beanCreation(beanAnnotation,beanInstance) { configureBean(beanInstance,getBeanName(beanAnnotation,beanInstance)); } /** * If a bean implements the ValidateConfiguration interface, then call * validate() on it once it has been configured (this advice runs after the * configuration advice). */ after(Bean beanAnnotation, ValidateConfiguration beanInstance) returning : beanCreation(beanAnnotation,Object) && this(beanInstance) { beanInstance.validate(); } /** * The bean name is either the value given in the annotation (@Bean("MyBean") ), * or the name of the type if no value is given (@Bean ). */ private String getBeanName(Bean beanAnnotation, Object beanInstance) { String beanName = beanAnnotation.value(); if (beanName.equals("")) beanName = beanInstance.getClass().getName(); return beanName; } /** * To be overriden by sub-aspects. Configure the bean instance using the given * bean name. */ protected abstract void configureBean(Object bean,String beanName); }

This abstract aspect has been designed to allow you to plug-in any IoC container you choose. We chose Spring ;). So here's the concrete Spring sub-aspect:

/** * Configuration of any @Bean using Spring */ public aspect SpringBeanConfigurator extends BeanConfigurator implements ApplicationContextAware { private ConfigurableListableBeanFactory beanFactory; /** * DI the Spring application context in which this aspect should configure beans. */ public void setApplicationContext(ApplicationContext ctx) { if(!(ctx instanceof ConfigurableApplicationContext)) { throw new AspectConfigurationException( "ApplicationContext [" + ctx + "] does not implement ConfigurableApplicationContext."); } beanFactory = ((ConfigurableApplicationContext)ctx).getBeanFactory(); } /** * Implementation of configureBean from the super-aspect */ protected void configureBean(Object bean,String beanName) { if (beanFactory != null && beanFactory.containsBeanDefinition(beanName)) { beanFactory.applyBeanPropertyValues(bean,beanName); } } }

The final piece of the puzzle is how Spring gets bootstrapped. We use an aspect for that too. Here's the abstract Spring initialization aspect - it's abstract because we don't know what strategy you want to use for the creation of an application context.

/** * Implements lazy initialization of Spring based on first use of an * @Bean type. */ public abstract aspect AbstractSpringInitializer { /** * we only want to create the application context once */ static boolean initialized = false; /** * the first time we load a type with the @Bean annotation */ pointcut firstTimeLoadingofBeanAnnotation() : staticinitialization(@Bean *) && if(!initialized); /** * Create the Spring application context, delegate creation strategy to * concrete sub-aspects. */ after() returning : firstTimeLoadingofBeanAnnotation() { ApplicationContext context = createApplicationContext(); SpringBeanConfigurator.aspectOf().setApplicationContext(context); } /** * Sub-aspects override this method to provide a concrete strategy for * application context creation. */ protected abstract ApplicationContext createApplicationContext(); }

For my program above, I used a DefaultSpringInitializer that configures the application context based on one or more beans.xml files available on the classpath

/** * A default implementation of the AbstractSpringInitializer aspect. This will configure * the Spring application context using the set of beans.xml files available in the classpath * (what you want for many applications). Configuration files can be overriden by specifying * the org.aspectj.ajlib.configuration.spring.configLocations property (useful for testing etc.). */ public aspect DefaultSpringInitializer extends AbstractSpringInitializer { public static final String CONFIG_LOCATIONS = "org.aspectj.ajlib.configuration.spring.configLocations"; private static final String DEFAULT_LOCATIONS_PATTERN = "classpath*:/**/beans.xml"; protected ApplicationContext createApplicationContext() { String configFilePattern = null; try { configFilePattern = System.getProperty(CONFIG_LOCATIONS); if (configFilePattern == null) configFilePattern = DEFAULT_LOCATIONS_PATTERN; } catch(SecurityException secEx) { configFilePattern = DEFAULT_LOCATIONS_PATTERN; } return new ClassPathXmlApplicationContext( StringUtils.commaDelimitedListToStringArray(configFilePattern)); } }

The result of all this, is that with a few very simple building blocks you can configure aspects (and any other type) simply by writing a standard beans.xml file and using the @Bean annotation - no Spring APIs required in your code at all. There's more coming out of the joint Spring / AspectJ development effort, but I hope this whets your appetite for now.

Posted by adrian at 03:47 PM [permalink] | Comments (2)

Aspect library discussion at AOSD 2005

We spent a 1.5 days with a group of AOP experts at AOSD 2005 looking at what form an aspect library for AspectJ might take and what should be in it. I copied down the contents of the whiteboards that had a rough taxonomy of possible aspects. Here it is, unfiltered and in its raw state. A lot of work still remains to be done to end up with an actual list of aspects that make sense...

If you have examples of aspects that you'd like to see in an aspect library and they're not on this list, please let us know (a comment on this entry would be fine...

J2SE aspects

  • debugging
  • enforcement
  • heuristics??
  • standard pointcuts
  • "helper" aspects (I can't remember what these were now)
  • Security
    • JAAS LoginContext mgt
    • doPriviliged
    • sealed object
    • signed object
    • guarded object
  • Caching
    • simple caching
    • JCS
  • Pooling
    • resource management
    • eager/lazy
  • MVC
  • Concurrency (util.concurrent)
  • Design patterns (with care)
  • EDA / asynchronous
  • logging/tracing
  • failure handling
  • IoC
    • Spring
    • HiveMind
    • Pico
    • JDK
  • DBC
  • Profiling
  • ui
    • swing
    • eclipse?
    • ...
  • remoting
  • join point sequencing
  • cflow across threads?
  • testing aspects

J2EE aspects

  • Spring ... anything that makes sense when working with spring
  • security
  • tx
  • concurrency
  • ejb
  • caching
  • pooling
  • validation / modification checking
  • persistence
    • exception mapping
    • hibernate
    • jdo
    • toplink
    • ...
  • webservices
    • beehive / annotations
    • grid
  • jms
    • eha bindings
    • signing / sealing
  • standard pointcuts
  • indexing
  • jmx
  • error handling
    • retry
    • detection
    • recovery

We also discussed the use of the various pattern language books (Enterprise integration patterns, Enterprise patterns, POSA, ... as a source of both potential aspects and also a pattern language to give the library a coherency it otherwise might not have.

Posted by adrian at 02:49 PM [permalink] | Comments (0)

March 21, 2005

Per-instance aspects

A lot of people have asked for the code I showed in the AspectJ BOF at the AOSD conference last week for implementing per-instance aspect associations. The basic problem is to associate aspect behaviour with object instances on an individual basis. JBoss often cite their caching scheme as a use case for this - objects are associated with the caching aspect when they are added to the cache, but not before. For this reason, I'll use a (very simple) caching aspect for the example in this entry.

Imagine some class, Worker that produces a cachable result:

public class Worker { private static int counter = 100; @Caching.CachableResult public Integer doWork() { return new Integer(counter++); } }

Note that every time doWork is called on a Worker instance the value of a single static counter is returned and then incremented. The doWork method is annotated with the @Caching.CachableResult annotation indicating that its result may be cached.

What we'd like is a way to associate individual worker objects with a caching aspect, so that we can cache their results. Here's a test case from my test suite that should give you the idea:

public void testInstanceCaching() { Worker w1 = new Worker(); Worker w2 = new Worker(); assertEquals(new Integer(100), w1.doWork()); assertEquals(new Integer(101), w2.doWork()); assertEquals(new Integer(102), w1.doWork()); w1.associateAspect(Caching.class); assertEquals(new Integer(103),w1.doWork()); // cache miss assertEquals(new Integer(103),w1.doWork()); // cache hit assertEquals(new Integer(104),w2.doWork()); // no caching on w2 w2.associateAspect(Caching.class); assertEquals(new Integer(105),w2.doWork()); // cache miss assertEquals(new Integer(105),w2.doWork()); // cache hit assertEquals(new Integer(103),w1.doWork()); // cache hit w1.disassociateAspect(Caching.class); assertEquals(new Integer(106),w1.doWork()); // no more caching for w1 assertEquals(new Integer(105),w2.doWork()); // w2 still cached w1.associateAspect(Caching.class); assertEquals(new Integer(103),w1.doWork()); // caching enabled again }

Note the calls to associateAspect and disassociateAspect being made on instances of the Worker class. In JBoss you "prepare" a class for per-instance advising (weave hooks into all of the class's join points of interest) when it is loaded. In the scheme I'm implementing here, you do the following:

aspect DeclareWorkerInstanceAdvisable { declare parents : Worker implements PerInstanceAdvisable; }

The PerInstanceAdvisable interface is very simple (and something we should include in the AspectJ library):

public interface PerInstanceAdvisable { public void associateAspect(Class aspectType); public void disassociateAspect(Class aspectType); static aspect DefaultImplementation { public void PerInstanceAdvisable.associateAspect(Class aspectType) {} public void PerInstanceAdvisable.disassociateAspect(Class aspectType) {} } }

Note the trick of declaring a default implementation of an interface as a static inner aspect of the interface itself. I personally think this is a very elegant idiom.

So far so good - now it's time to look at the actual caching aspect implementation. First off the PerInstanceAssociationAspect super-aspect that can serve as the base class for any aspect participating in per-instance associations.

/** * Super-aspect for all aspects implementing per-instance associations. One aspect * instance will be created for each associated object. */ public abstract aspect PerInstanceAssociationAspect perthis(aspectAssociation()) { /** * This flag controls whether or not this instance of the aspect is * "active". An aspect becomes active when it is associated with * an instance, an inactive if it is disassociated again. */ protected boolean enabled = true; /** * Sub-aspects override this pointcut to match if the Class argument * at an aspectAssocation() join point matches their type. The * overriden version should look something like this: * protected pointcut isThisAspect(Class aspectType) : * args(aspectType) && if (aspectType.equals(MyAspect.class)); * where "MyAspect" is the type of the sub-aspect. */ protected abstract pointcut isThisAspect(Class aspectType); /** * A call to associateAspect with our aspect type passed as the * argument */ pointcut aspectAssociation() : execution(* PerInstanceAdvisable.associateAspect(Class)) && isThisAspect(Class); /** * A call to disassociateAspect. The implicit scoping given by the perthis * clause means that this only matches at join points where the object * executing the disassociateAspect operation is the the object with * which this aspect instance is associated. */ pointcut aspectDisassociation() : execution(* PerInstanceAdvisable.disassociateAspect(Class)); /** * Enable this aspect whenever it is associated. The implicit scoping * from the perthis clause restricts this advice to the object instance * with which we are associated only. */ after() returning : aspectAssociation() { enabled = true; } /** * Disable this aspect whenever it is disassociated. The implicit scoping * from the perthis clause restricts this advice to the object instance * with which we are associated only. */ after() returning : aspectDisassociation() { enabled = false; } }

I've ajdoc'd the aspect so hopefully it's self-explanatory from the comments.

Finally, here's the Caching sub-aspect that I promised right at the start of this article.

/** * A Caching aspect that caches the results of @CachableResult * operations, on a per-instance basis. PerInstanceAdvisable objects * may be associated with the cache by calling obj.associateAspect(Caching.class). * They may be disassociated from the cache by calling * obj.disassociateAspect(Caching.class). */ public aspect Caching extends PerInstanceAssociationAspect { /** * The guard that ensures advice never excutes for an object that * has previously been associated with the cache (causing this * aspect instance to be created), but has subsequently been * disassociated and not re-associated. */ private static boolean enabledFor(Object cachee) { return hasAspect(cachee) && aspectOf(cachee).enabled; } /** * The concrete definition of isThisAspect from the super-aspect. * Matches when the user tries to associate an aspect of type * Caching .class */ protected pointcut isThisAspect(Class aspectType) : args(aspectType) && if (aspectType.equals(Caching.class)); // ----- // the remainder of this aspect is particular to caching... /** * Marker annotation to be used by operations that produce * Cachable results; */ public @interface CachableResult {} private Cache cache = new Cache(); /** * A cached operation is the execution of any method that has * the @CachableResult annotation and an enabled caching * aspect. */ pointcut cachedOperation(Object cachee) : execution(@CachableResult * *(..)) && this(cachee) && if (enabledFor(cachee)); /** * Basic caching scheme... use the cached result if we * have one, otherwise proceed with the operation and * cache the result for next time. */ Object around(Object obj ) : cachedOperation(obj) { Object result = null; if (cache.hasResultFor(thisJoinPointStaticPart)) { // cache hit result = cache.getResultFor(thisJoinPointStaticPart); } else { // cache miss result = proceed(obj); cache.addResult(thisJoinPointStaticPart,result); } return result; } /** * Trivial Cache implementation using a Map. Note that JoinPoint.StaticPart makes * a stable key for caching the results of operations at join points. */ private static class Cache { Map cachedResults = new HashMap(); public boolean hasResultFor(JoinPoint.StaticPart jpsp) { return cachedResults.containsKey(jpsp); } public Object getResultFor(JoinPoint.StaticPart jpsp) { return cachedResults.get(jpsp); } public void addResult(JoinPoint.StaticPart jpsp, Object result) { cachedResults.put(jpsp,result); } } }

A weakness of this scheme as opposed to the JBoss implementation is that aspects intended to be used on a per-instance basis (such as the caching aspect above) have to be written with that intention in mind (whereas I believe that in JBoss you can associate any aspect with instances of a pre-prepared class). On the plus side, from my understanding of the differing implementations, this solution should perform better than the JBoss implementation.

Posted by adrian at 04:07 PM [permalink] | Comments (1)

March 19, 2005

Sam Pullara on VM-level AOP (AOSD 2005)

Sam's talk was a survey of different weaving times and strategies, concluding that VM based weaving is where we ultimately want to be (no disagreement with the audience on that point). However, it will take a long time to get there, and a lot of the issues are not technical...

Sam also gave a live demo of the new @AspectJ support running in IntelliJ having obtained a copy of the code from Alex. He's the very first person outside of the team to play with it, and his initial impressions were very favourable.

The current state of the art supports weaving at compile time, class load time, runtime hotswap, and runtime using OO methods. Weaving is done using either bytecode modification or dynamic proxies.

Compile-time weaving

Pros

  • predictable runtime performance
  • ide friendly
  • bounded set of classes and effects
  • most compatible with current vms

Cons

  • must have classes available
  • cannot dynamically change aspects
  • can break the license of some sw
  • large up-front cost

Class load-time weaving

Pros

  • aspect decision deferred to deployment
  • can be used with most hot deploy systems
  • can apply to code unavailable at compile time

Cons

  • startup time can be arbitrarily large
  • some software incompatibilities

Hotswap weaving

Pros

  • similar to class-load time...
  • can be changed without reloading class
  • deployment time configuration

Cons

  • startup time
  • some JVM incompatibility
  • requires native code in pre Java 5 vms
  • complicated command lines
  • need to run in debug mode prior to JDK 5 - 10-30% performance degradation, complete retest required.

Runtime OO methods

Pros

  • very dynamic
  • easy to understand for OO programmers
  • implementation is simpler

Cons

  • often slow due to reflection
  • supports many fewer pointcut types
  • startup time is affected (proxy generation) - this can be significant (cited 200% increase in one example)

Bytecode modification weaving

Pros

  • can be implemented on virtually any VM
  • mostly invisible to end user

Cons

  • can change the shape of the class
  • some side-effects like serialization are exposed
  • typically only one agent/weaver is possible
  • class level only

Dynamic proxy weaving

Pros

  • standard java
  • very simple

Cons

  • recursion is not easy
  • often inefficient
  • proxies need to be regenerated on each run

Issues with SOTA

  • current AOP implementations are like working with generics implementations prior to the standardisation in java 5
  • debugging of current systems often much more difficult than plain Java (eg. stack traces)
  • without standard support, portability suffers between vms and platforms
  • many target users will be unwilling to depend on a language feature that isn't "supported"

VM level AOP

  • is not really weaving any more since no bytecode modification is needed
  • debugging would be simplified as the debug apis would support aop
  • performance:- could be tied into the vm optimizer more efficiently
  • memory usage would be better since double analysis and storage would not be required
  • works much better with reflection code

The JRockit team have a prototype of VM-level weaving. The SteamLoom project does VM weaving based on the Jikes RVM too.

Posted by adrian at 03:40 AM [permalink] | Comments (3)

March 18, 2005

Ron Bodkin on Aspects and Security (AOSD 2005)

I missed the beginning of this talk, so here's the story from half-way through.

Ron is showing an aspect for JAAS authentication. It uses around advice to set up a JAAS LoginContext around secured request operations. Making the point that more web containers are now starting to provide built in support for this. Role-based authorization is straightforward too - just use before advice to get and check the Permission before executing an authorized method.

Motivating example - editing employee data. A manager can only read and edit sensitive data for their own employees (it's not a general permission to say "can I read employee data or not"). We need data-driven authorization. Ron is starting off by showing how you would implement this with a standard java proxy-based solution. Requires custom factories and invocation handlers. Better than scattering the code all throughout the app, but it's a primitive form. Using aspects you can code this in a simpler more reusable way. Define a pointcut for a sensitive operation. Use before advice that takes advantage of exposed context.

Now Ron is adding in security auditing - you certainly want to audit failed authorization checks for example. Uses after returning advice to record a successful authorization, and after throwing advice to record a failed one.

Next example shows how annotations are useful in working with security and aspects. For example: pointcut p3pDataAccessField(Field field) : get((* @P3P) *.*) && ... (P3P is a standard for data privacy, defines about 17 types of sensitive data).

Next example is an aspect that filters out sensitive results (rows) from a database query. Using around advice to do this. Adds a where clause constraint to the query before it executes.

Domain-specific tools are really important here. it's not the programmer that we want to be making security decisions. Security experts should do this - and showing those experts a bunch of pointcuts won't work. Need something on top.

Final example: UI filtering requirements. You should not give people links or options to do things that they are not authorized to perform, show them fields that they are not allowed to see etc. Basic strategy: advice finds unauthorized field display. servlet filter removes complete context. Deployment options are precompiled JSPs, or configure containers JSP compiler to us ajc.

Posted by adrian at 11:54 PM [permalink] | Comments (0)

Dave Thomas' keynote at AOSD 2005

Dave Thomas' keynote

Transitioning AOSD from Research Park to Main Street

Dave was the founder and CEO of OTI (not Pragmatic Dave). Currently CEO of Bederra Research Labs.

One of the most important things you can do is fail fast and fail often. Nervous of the black and white world in which technology is painted. We need dialogue, not just propaganda - we should listen to each other and to critics.

complexity of application development today

We live in a very complex world. There is latent technical complexity, accidental complexity. Open source from the OS to the application allows every customer application to be "unique". This creats an immense tool and services opportunity to make an even bigger layer cake :). (Dave is a big fan of Spring btw).

Latent complexity... so many things you have to know in order to be able to do something (objects, components, interfaces, patterns, AOP, MOF, UML, Java/C#, XML, Web Services, Scripting, Messaging, ...

New acronym: TMS (too much stuff) its PMS but for engineers ;)

There is gratuitous technical complexity for simple problems. See e.g. http://www.jot.fm/issues/issue_2003_09/column1 All a customer wants to do is query a database and get some results... but they have to deal with objects, persistence frameworks, ... and so on.

Accidental complexity is the real killer. Fragile, complicated, redundant: multiple technologies with 100s of APIs. Requires extensive expert level experience with platform technologies. Normal person just trying to do things cannot keep up. We have complex and bug ridden libraries: The Deplorable State of Class Libraries

We have a skills gap. Java programmers can be had really cheap - every college produces them and Sun certifies them (so they must be good, right?). But the skill level has been dramatically lowered over time. They don't understand comp sci or basic engineering. (So how will they understand aspects?). Once met someone with a masters in SW Eng. who had never written any code!!

challenges of application development 2010

Need to be business driven versus being technology driven. Based on sound and deep understanding of the business. Complexity mandates inter-disciplinary collaborative definition and development driven by business experts. First time applications vs traditional process improvement. The ultimate pair programming is a programmer plus a business expert. We need support for high performance teams. In 2010 there will be a bunch of legacy platforms (J2EE, .NET, ...).

Dave is not a fan of MDD. We need development in real-time, execution in real-time, massive amounts of data, ... Are our languages and tools up to it?

Experiences of selling new technology

Sell concept and benefits relating to previous/current practice - avoid the language/tool debate. Illustrate with an example which makes sense to them. (Nice anecdote, selling Smalltalk to execs "can't we name it something better - maybe Bigtalk??"). Address the critics concerns before they ask. Explain that tools help productivity and quality. Aside... what if the guys who did J2EE had looked at something like CICS - which already had separation of concerns and the developers only had to write a small piece of business logic. Tuxedo and MTS were the next generation of ideas built on CICS.

Every talk they (OTI) gave started and concluded with this message "this is exactly the same as you have been doing all along, except that it is completely different!". Respect what people are doing and explain that you understand their problems. You have to understand the languages of the people that you are talking too. Technical cultures (eg. SAP) are much more diverse than human cultures!

Early adopters pull the ideas and tools and move aggressively ahead. Will inflate your most conservative claims. Will alienate others inside their company. Position yourself between the early adopter and those who fear/oppose them. Keep the advocate with you, but make sure that you talk to the real customer too.

The later adopters have a long list of questions they just need to have answered. Often have very different skills and much less software engineering experience than any AOSD attendee can imagine. Want standards, customer references, case studies and insurance of major vendors/ISVs. Make sure you always operate with honesty, integrity, and modest claims. Dave : we actualy persuaded ourselves that people could program in Smalltalk - they can't! Gurus can, but the man on the street can't.

AOSD Works!

Encourage separation of concerns as a best practice. Encourage role/subject modelling. Refactor to improve maintainability of large complex products, eg. middleware. Refactor to improve usability of large compelx products, eg. middleware. Manage variability of product line assembly/deployment. (Dave really is not a UML fan at all...).

J2EE desparately needs an industry standard aspect library to provide a simpler programming model. Java desparately needs a better model than J2EE.

AOSD Analysis and Design

Revive an understanding of role modelling and interfaces. Provide an implementation of use case extensions. Requires a revision to UML and associated tool chain. What does A/D mean in the presence of aspect libraries?? What does it mean with respect to architecture? It was shown yesterday that aspects can encode architectural constraints - we need to get that message out more (AMC - this is a ref. to the talk that Gregor and I gave yesterday).

AOSD Software Engineering

Who, where, when should AOP be applied? What does unit testing, acceptance, regression testing mean in the context of aspects? How can we avoid the problems we found with before, after, doesNotUnderstand etc. in large Lisp and Smalltalk programs? How does one debug at the level of the abstract? Dave is now talking about how positive the dialog at the "Adopting AOP" talk was yesterday - it took the OO community a long time to get to that stage. Can powerful tools really fix a bad platform - maybe we need to build a good platform with AOP rather than fix J2EE?

AOSD Speculation

When do we need an aspect language versus an aspect transformer vs an aspect runtime? Is there a useful subset of AOSD that can be implemented via existing OO mechanisms? (Eg Envy/Developer, AspectJ2EE). Are there standards of practice and convergence of key concepts? Are some problems better solved using non-OO languages/techniques - eg. dependencies. These are all interesting questions for us to consider. Teaching people just UML and Java is just not enough.

We need to teach things like computational reflection back into CS programs. Read SICP, The Art of the MetaObject Protocol. People who have read and studied these books (and ones like them) are always the leaders of new ideas. Functional programming -> design patterns. Meta Object Protocols are the big ideas behind AOP/SEP. Separation of Concerns. Could microsoft pheonix be the eclipse for code generation?

Some predictions... with the usual caveats. AOSD will at a minimum be a master craftsman tool for systems programmers. It isn't clear that AOP will be "outside" for application developers? AOP will likely find success in sw product engineering vs application dev - in the same way that OO CBSE has. Aspect libraries should increase the use, and also reduce the need for aspect developers in fawour of aspect users. AOP Analysis and Design will find a slow adoption, and may get stuck behind UML 2. Teaching AOSD concepts - separation of concerns, role modelling, and meta object protocols may find a resurgence. AOP will be a challenge to position in undergrad curriculum.

Language neutral tools will find better acceptances than new languages. @lt;yuck @wins /yuck> AOP in every language is easy but not interesting - eg. aspect javascript. AOSD tools will fill current gaps eg. true OO->AOP refactoring. AOP load/deployment time weaving will become the preferred implementation. AOP OS and VMs will remain a research topic for at least another generation of commercial VMs. Hopefully by 2010...

AOSD Research

What formal machinery can be usefully brought to bare on AOP? Does the holy grail of a declarative reflective language philosophized by Brian Smith really exisst? AOSD suggests we may be closer to true meta programming? program composition of fragments of requiremens, design, code into programs eg. hyperspaces, feature oriented programming, beta. How many people have a basic understanding of monads? (only 1 or 2 hands). "This is a problem folks" - you need to go look at what they are doing. Do classes really matter??? We have a simple abstraction... but maybe it's wrong. The model given to you in physics 101, but the simple abstraction is wrong. To do more powerful stuff in physics you need to understand the deeper stuff. Keep AOSD as the name of the conference, and encourage and welcome a broader community to come. The great tool that Dave wants is one that easily allows you to take a large program and discover and refactor into aspects. Will be done by teams of specialists initially. Please do not chain graduate students to eclipse, java, c#, solely to claim that your research results have industrial relevance. Get the ideas out, and then you can go through the industrial phase. Buffer the industrial noise - find ways to do simple quick demos.

Q & A

Q. What about the relationships between people and programming languages? Different people work in different styles. A. Dave is a big fan of DSLs. E.g. functional programmers tend to have strong mathematical backgrounds. However, we shouldn't be ignorant of the benefits of the thought processes and ideas in different programming language cultures. Would like to see a cross-language family culture.

Q. We can study what's working in large systems and generalize. We can also research at a very small scale according to principles and then hypothesize that they scale up. Which is better? Any advice? A. Success of many people attributed to "split brains" :- enjoy the elegance of a good theorem and a clean solution, but also like seeing them worked out in "scruffy" systems. Eric Myer at Microsoft has had a big impact on Dave Thomas' thinking recently. Don't look at the code... look at the people who build it. SW is a craft, design is a craft. Schools of architecture and design don't have "go to class...", they have lots of studio time etc.

Q. I liked the cut the BS aspect of your talk. The reality of our life is that we have to get money to pay the bills. How do we reconcile the economic constraints with the message to go after great science? A. Solve the problem that the customer wants solved with the technology they want it solved in, but do it using tools you built using whatever technology is the most appropriate. E.g IBM process that said you had to have a complete functional spec before you could do a prototype. So they built the prototype and from that created the spec! This was a much better strategy than fighting the process. They had tools that created what was needed. Find ways to play within the needs and culture of the customer, it's not hard...

Q. What is Dave's vision of the sw industry once all the bad technologies have been simplified? A. There is a whole other parallel universe for application development - very many people who are not really bothered about J2EE etc.. J2EE will only be relevant as a legacy technology in 15 years. The world will move to those people who understand the application domain. Innovation happens at the edge.

Posted by adrian at 11:52 PM [permalink] | Comments (0)

March 17, 2005

Mik Kersten : Comparing AOP tools

The last session of yesterday was Mik Kersten giving a comparison of the different AOP tools and their strengths and weaknesses. My battery had given out by then so no live blogging for me. However, the talk was based on Mik's excellent articles on developerWorks so even better than my random jottings, you can read the whole thing online at:

These are well worth checking out if you haven't done so already...

I should have also said yesterday that much of the material from Ramnivas' talk on metadata is also written up as a developerWorks article as part of the " AOP@Work" series.

Posted by adrian at 01:27 PM [permalink] | Comments (0)

Nick Lesiecki : Experiences using AspectJ at VMS

My notes from Nick's talk at AOSD this afternoon...

Aspects at VMS

Going to assume that the audience don't need to be sold on aspects, will give the warts-and-all story. Aspects first applied to VMS' Adbase project - J2EE integration across 5 separated data repositories. Started with EJB, JMS, JSPs, Struts, moved to Hibernate, Tapestry, Spring. Started in Jan. 2004. 4-7 programmers with mixed level of experience. 3 programmers new to Java and OO. Nick was the only one who knew J2EE, and the only one who had any exposure to AOP.

They followed the "standard approach" to adopting AOP, scaling their use of aspects over time. First step was applying aspects to a third-party library - binary weaving of an existing jar (Apache's JSTL). Original exception stack trace from the application was no longer propagated by the framework, so added an ErrorLogger to capture exceptions and log them to the console. Found limitations at this point: no around/after/cflow on handler join points - which prevents an exception chainging solution. Ended up using percflow aspects to "guess" what the underlying exception might have been. Aspect stored the last handled exception, solution allowed exception to bubble up all the way up the stack.

Benefit was a drastic improvement in error diagnosis. Could have been done by modifying source of that one open source project... but have now applied aspects to several projects - better reuse.

Next aspect introduced was "basket pricing". Since so many different operations can affect the state of the shopping cart (about 12 places over 3 classes) it was clearly a cross-cutting concern. Employed a strategy using a basketChange pointcut to mark the basket dirty, and then the basket is "cleaned" before any operation that needs the price. Aspect handled nested changes. Worked in collaboration with an interface called Chargeable (with default implementations provided by Aspectj ITDs). Were able to exclude changes that happened during the reprice operation itself by using cflow. They used separate objects delegated to by the aspect to handle most of the complex logic - easier for incremental compilation and for unit testing. Also allows DI of different pricers using something like Spring. Could reason about and evolve the pricing behaviour in isolation. Gives you the ability to go back and do things you would never do if you hadn't encapsulated the tangled concern in an aspect in the first place. They refactored the strategy four times to get the best - would never have done this if it had been tangled.

Removing the pricing logic clarified things in the system. The downside is that the aspect must be maintained as pricing operations change (eg. a brand new pricing operation is added). This must be captured by the aspect - so you have to update the aspect in sync with the changes. Obliviousness can lead to forgetfulness. We have to master the art of robust pointcuts (see Adrian's blog - thanks nick :) ). Some judgement and experience is needed to do this. We need to move from the team being "oblivious" to being "aware" of the aspect. Tool support helps here. People need to be aware of what aspects are active in the system.

Question is, will metadata help? Could have @AffectsPricing annotation? This may be better than an enumeration in a pointcut. Gives you more of a clue in the tools than a simple advice marker does. Conveys more meaning in a visual examination of the source code

Final aspect to be discussed: management of relationships between persistent aspects in the source code. Eg maintaining child and parent associations. About 90+ places in the code dealing with this. Also in the middle of switching from EJB to Hibernate - lets try out AspectJ for this. Outline of solution: detect calls that update a relationship. Use marker interface plus stable naming pattern (set*). With annotations in Hibernate 3.0 we will get more stable signatures to match on. The AOP part was fairly simple, on setting the relationship in one direction, it propagated the change in the other direction. Hibernate piece was more complex... but this isn't a Hibernate talk. Built their own proxy support to wrap individual objcets. Generics will help with typing of collections and join point matching.

For relationship severing use the hibernate Lifecycle interface. Used declare parents to add this to all persistent objects. Lifecycle interface then gave jps for severing relationships. Used a percflow aspect to hold state for objects deleted in a cascaded delete. This allowed severing of each link only once.

Relationships during construction were more tricky. New hibernate objects automatically registered with hibernate session via advice. All of the objects implement equals and hashcode based on an id. Ended up reciprocating relationships to objects that were not yet fully formed. Decided to defer relationship propagation until after ids were assigned by Hibernate. Used an aspect to do this. Uses worker object pattern (Ramnivas). Allowed constructor code to be written "naturally", but with deferred creation of relationship updates.

Benefits: bidirectional relationships managed in 93 places. Deletions result in automatic discontinuation of relationships - no dangling references. Neither behaviour can be forgotten - prevents subtle bugs. Getting the whole solution right took about 2 pws. The aspects form a kind of 'mini framework' but without the usual burden of a framework. The existence of libraries will really boost AOP adoption. Help people leverage others domain knowledge as well as investment in AOP.

Q. it took 2pws to remove 100 lines of code - was it worth it in the end? A. "not sure" in this case - but with a library aspect it definitely would be. From an architecture standpoint Nicks opinion is that it definitely was worth it - the POJOs are all a lot simpler: smaller, easier to understand, easier to write. So may pay for itself over the lifetime of the system - too early to tell yet.

Part 2: Adoption/Impact - a look at some of the issues involved in adopting AOP into a development process.

As a whole the experience was positive. Basic concepts were reported as easy to grasp, but mastering the nuances took time. About 6 calendar months elapsed, 5-10% of effort devoted to practical learning of AOP. Using pairing sessions to develop aspects initially, developers moved quickly from observers-> participants -> initiators in AOP pairing sessions.

Found that not all developers needed to be AOP experts (but must have some familiarity). Some negative reaction, certain amount of FUD, training burden as staff rotated through the project. The fear relates to "unknown forces tampering with code" - "we've got an aspect in place that causes .equals() to return false from any class whose name begins with P..." cartoon. The less orthogonal the concern, the greater the awareness necessary. Saw the "aspects as a red herring" phenomenon - programmers immediately suspecting an aspect of causing a problem even when in was a simple problem in their code. Counter forces are general familiarity with AOP, and staff awareness of aspects in the system. (Audience comment... if you've showed that slide 15 years ago people would have said exactly the same thing about objects).

AOP meshes well with agile processes. Incremental adoption: pilot projcts, limited scope aspects, soak time. "Don't jump into the deep-end". New concepts take time to work their way through your unconscious. VMS are test-infected, and their AOP adoption benefited greatly. There is a myth that aspects cannot be unit tested - VMS experience is that this is completely untrue, they have successfully unit tested all of their aspects. Verify aspects are working correctly before applied to code. Can detect when aspects have unintended side effects. Can detect when an aspect fails to advise an important join point. Tests also encourage good aspect design, and serve as documentation for the aspect.

Pairing is a great way to transfer knowledge. Have an expert in earshot for AOP related problems. Refactoring: practices for AO code similar to those of OO code - small steps. Have both added and removed aspects with refactoring.

Tools wanted: join point shadow diffs - check that pointcuts are equivalent. Especially temporal, to detect new/unintended matches after a compile (AMC note - we have this in the labs, will be released in AJDT real soon now).

The risk of overadoption.... XP shuns needless complexity. Pairing kept things sane.

The largest obstacles were practical. Tools critical to understanding an AO program. Used visualizer plus markers plus debugger. Problems are that too many advice markers can reduce effectiveness (eg. a tracing aspect puts markers everywhere and theses get in the way). Views only updated incrementally in the latest milestone drop - was a problem in the the last releases. Compilation times were the biggest issue. Full compile takes 40s, incremental takes 7s - but changing an aspect forces a full recompile. [AMC Note - recent updates to AJDT have roughly halved compilation time on VMS' project.] But XP development makes heavy use of incremental changes/testing. "The benefits realized by AOP over OO do not outweigh the drawbacks of productivity loss and are an impediment to TDD." said one senior programmer in VMS. This issue is seen as the biggest barrier to adoption in VMS. Ron Bodkin - "yeah but think how long it would take if you had to redo it by hand". Eclipse sets the bar high, AJDT is getting close, but not there yet. Biggest issue is support for the simple refactorings like method rename when you have a pointcut referring to the method. Basic refactoring support plus faster compilation times are the biggest issues. (AMC note - these are both high priority items in the AJDT and AspectJ plans).

Incompatibilies: bytecodes produced by AspectJ exposed eg. bug in JRockit VM. But AspectJ always gets the blame as the new kid on the block. Makes team want to kick out AspectJ, not JRockit for example. (Workaround implemented in AspectJ a while back btw).

VMS considers its adoption of AOP successful - improved resilience to change in code base, easier to change crosscutting things (even possible to change some things that weren't before

Q. what would happen if Nick left? Would the enthusiasm continue at VMS? A. Hard to gauge... Nick has been the driving force.

Posted by adrian at 01:14 AM [permalink] | Comments (0)

March 16, 2005

AOP with Metadata: principles and practices (Ramnivas Laddad)

Why do we need AOP and metadata? Today we have signature based pointcuts. These exploit inherent information associated with the signature. This works well in many cases. However, metadata based pointcuts also enable us to leverage metadata at matched join points. Useful for transaction management, security etc.

Ramnivas is showing transaction management implemented without aspects at all - inlined class to JTA. Now he is showing how aspects can remove all of the tx logic from the business class. The first implementation uses around advice to "transactedOps()". The around advice has the familiar try-catch-finally block. But how do you write the pointcut? One solution is to enumerate all of the matching signatures - but how do we know what they will be?

Current solution:- create an abstract aspect and make the transactedOps pointcut abstract - now you can define it in (eg.) a Banking specific aspect. It would look like:

public pointcut transactedOps() : execution(* Bank.credit(..)) || ...;

With annotations we can write:

pointcut transactedOps() : @annotation(Transactional);

(AMC note ... I would have added && execution(* *(..)) to the above to limit matching to method execution join points.)

And now you annotate each method that needs tx support with the @Transactional annotation.

If you need to add security (say) you just add another annotation: @Authorized. Ramnivas is making a nice analogy: annotations are playing the role of additional method "signatures" in alternate dimensions (the @Transactional annotation is the method signature in the transaction dimension, the @Authorized annotation is the method signature in the security dimension and so on. Recall that annotations can have values etc. Rather than calling them "dimensions" I might call them "domains" - so a method has a signature in the business domain, the transaction domain etc..

When you think of it like this, the Account class looks as follows in the transaction domain (dimension):

public class Account { @Transactional * credit(..) @Transactional * debit(..) .. }

Onto best practices...

Metadata is an easy way to capture crosscutting concerns - just sprinkle some annotations around. It limits the collaboration between aspects and classes to just annotations. Downside is that collaboration from classes is needed. It's clearly nonsense to use annotations on every method for say logging :).

Guideline: don't use annotations when you can do without them. Eg. you want to capture all RMI operations. You can write a simple, stable pointcut: execution(* Remote+.*(..) throws RemoteException) - no need for an annotation here. Likewise you can write a stable pointcut for all thread-safe calls to the Swing library - you don't need annotations for this either.

Not sure if Ramnivas is going to mention this, but there are also expressibility limitations with annotations - you can't say for example "all calls to x within y" since you can't annotate calls (you can say "all calls to a method with an annotation X made within y" - but that's a slightly different thing). There are other examples too...

A second guideline from Ramnivas: employ aspect inheritance to defer pointcut specification to a sub-aspect. Now write one pointcut per "subsystem" this will be much easier than trying to write a single global pointcut that works across the whole system.

A third guideline - make use of existing annotation types - eg. EJB

A fourth guideline - if you create annotation types, use names that describe a property of the annotated element, not an instruction to some processor. Eg. @ReadOnly, not @ReadLock, use @Idempotent, not @RetryOnFailure, and so on.

A final guideline - wisdom comes with experience: start with something and refactor as you go along.

Ramnivas is now discussing whether metadata-fortified AOP harms "obliviousness". His argument is that it does not if the guidelines are followed.

The final flourish in Ramnivas' talk. You could have an aspect that matches on annotated methods... if you don't want to annotate them individually, you can use an inner aspect (participant pattern) to declare them...

Posted by adrian at 10:19 PM [permalink] | Comments (2)

Adopting AOP, and AspectJ 5 talks

It's harder to write an objective review of the next two talks I attended since I was giving them! Gregor (Kiczales) and I did a double act on "Adopting AOP". The talk was structured around the Adopting AOP talks that Gregor and I have done in multiple venues (the material for those talks was jointly developed and we continue to share enhancements and updates). Rather than just give the standard adopting talk though, we gave a running commentary on it - what works well, which areas are weaker and need more development etc. There was a good exchange of ideas with the audience which is exactly what we were hoping to provoke. The goal of the talk was that we could all leave with a better story to tell about the adoption of aop within an organisation. There was an interesting discussion at the end on the role that aspect libraries will play in driving adoption. I think getting to a "standard" aspect library that ships with AspectJ will be a huge boost since it shortens the time-to-value in getting started with AOP and helps the end-user avoid reinventing the wheel in common cases.

The second talk was "AspectJ 5" which I did jointly with Jonas Boner. We had only 30 minutes for this talk - giving a comprehensive overview of AspectJ 5 in only 30 minutes is an impossible task so we cherry-picked some of the highlights. We talked briefy about the merger and then got straight down to details. I covered some of the changes in AspectJ 5 to deal with the new Java 5 language features, including a short discussion on how AspectJ 5 will deal with parameterized types in join point matching. Jonas reviewed the @AspectJ (annotation-based development) style coming in AspectJ 5. If you haven't seen it before, it looks like this:

@Aspect public class NoopAspect { @Pointcut("execution(void Math.add(..))") void addMethods() {} @Before("addMethods") public void noop() { System.out.println("In advice"); } }

Which is exactly the same as:

public aspect NoopAspect { pointcut addMethods() : execution(void Math.add(..)); before() : addMethods() { System.out.println("In advice"); } }

We closed out with a discussion of the load-time weaving enhancements coming in AspectJ 5 and the support for AspectJ 5 in AJDT.

Posted by adrian at 10:17 PM [permalink] | Comments (0)

Grady Booch's keynote at AOSD

Grady Booch has just taken the platform to deliver his keynote, "The Complexity of Programming Models".

The opening question is : "how many lines of code are written each year around the world?". A. about 35 billion lines of source code are written each year. This is based on about 15 million or so software professionals world-wide, of which current estimation is about 30% actually cut code. (The percentage of professionals cutting code has steadily declined :( ). Each developer contributes about 7,000 loc a year. Over history, we have about a trillion lines of code written so far!

The general trend is that things are getting more complex... A given system means different things to many different stakeholders. What "simplicity" is for each stakeholder also varies. Simplicity has different points of view.

In the presence of essential complexity, establishing simplicity in one part of a system requires trading off complexity in another.

The best technology is one that is invisible... the task of the development team is to engineer the illusion of simplicity. In languages we see the tradeoffs in history. Control structures are a tradeoff between primitiveness and convenience. Garbage collection is a tradeoff between expliciteness and abstraction. VB and Smalltalk are tradeoffs between performance of development and performance of execution. Beans, aspects, services are a tradeoff between packaging for design versus packaging for development versus packaging for deployment...

A programming model specifies the semantic universe within which the developer labours and is defined by the languages, platforms, tools, etc. they work with.

For example, think about a web-centric programming model. There are a set of languages commonly used (HTML, CSS, XML, RSS, Java, ...), a variety of platforms (LAMP, J2EE) etc., a set of best practices, and a set of tools. These things combine to make the programming model for web services. And this is not simple... Question for introducing new things like aspects - are we making things simpler, or are we adding complexity. Other programming model examples are game development, high-performance computing, command and control, AI, domain-specific frameworks,... Check out Grady's Handbook of Software Architecture at http://www.booch.com/architecture.

A system is shaped by a myriad of design decisions by different stakeholders that work to balance the forces swirling around the system.

E.g. in civil architecture there are forces of compression and tension. "Any time you depart from established practice, make ten times the effort, ten times the investigation...".

In software some of the forces are: schedule, mission, cost, legal, ethical/moral, dependability, quality, performance, functionality, resilience, production, context, compatibility, resources. These are grouped in categories of business, values, operations, development, and environment.

Software is inherently complex due to complexity of the problem domain, difficulty of managing the dev. process, fluidity of software, and the fundamental challenges of discrete systems.

Software has limits, even if you don't think so. E.g. a space system that needed very accurate synchronized time - but the requirement violate the fundamental laws of relativity.

In the dev. process we face challenges of teams, multiple languages, platforms, tools, etc., scalability (size up, speed up, scale up, scale out http://www.intelligententerprise.com/db_area/archives/1999/991602/scalable.jhtml).

Discrete systems have non-continuous behaviour, combinatorial explosion of states, corruption from unexpected external evennts, lack of mathematical tools and intellectual capacity to model the behaviour of large discrete systems.

We can master essential complexity, but we can never make it go away.

How do we measure complexity? In biological systems we have a variety of techniques (http://www.carleton.ca/~hmasum/complex.html). The Kolmogorov measure says "what's the simplest structure that will produce this behaviour?". One very simple measure - how many words does it take to write down a description of the behaviours? Kolmogorov applied to software: "the relative size of a program capable of generating a given string".

If we don't know how to measure complexity, it is reasonable to suggest that we don't know how to measure simplicity.

Beauty/elegance is one thing we recognise. Elegance means simplicity and less new code, an elegant solution solves the whole problem.

Triggers of complexity include significant interactions, high number of parts and degrees of freedom, nonlinearity, broken symmetry, and nonholonomic constraints (localized transient anarchy). [No, I don't know what nonholonomic constraints are...].

As systems evolve, objects that we once considered complex become the primitive objects upon which more complex systems are built. Hierarchic systems are usually composed of only a few different kinds of subsystems in various combinations and arrangements... Systems can be decomposable (independent parts) or nearly decomposable (most sw components). In biology, the most resilient systems have the loosest coupling. See Simon, The Organisation of Complex Systems. Interesting finding: overall flexibilty increases by having fewer components but greater composability.

A complex system that works is invariably fonud to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work.

Self-organising systems are interesting to study. For example pattern formating in slime molds, flocking of birds, schooling of fish, wall building by ants... All of these have underlying simple rules. Can we build complex systems from simple rules, and let the complex behaviour emerge?

As complexity increases, the increase in functionality and understandability is exponential. We reach a barrier where we just cannot understand the system anymore.

Fundamentals never go out of style.

(or how to mitigate complexity)

Aspects are not a fad - there is something deep and profound about them, but we cannot ignore the fundamentals so always hold them in mind as you do dev/research. Fundamentals are:

  • crisp abstractions
  • clear separation of concerns
  • balanced distribution of responsibilities
  • simplicity via common abstractions and mechanisms

Abstraction... all abstractions are context dependent. All non-trivial abstractions are to some degree leaky (Joel on Software). There is no such thing as a perfect abstraction.

Gabriel: Worse is Better. Don't strive for "perfection" good enough is better. Both implementation and interface must be simple, although it is more important for the implementation to be simple.

The entire history of sw enginerering is one of rising levels of abstraction.

are you on that path?

To attack complexity, stick to the fundamentals, relax a constraint, make assumptions. It's better to be simple than to handle every possible case.

Architecture is the sweet spot for aspects... (interesting to see where this bit of the talk is going to go). Mature physical systems have stable architectures which have grown over long periods of time. In software-intensive system we haven't achieved this. Architecting software is different. Kent Beck: "Software architecture is what software architects do". Other definitions: http://www.sei.edu/architecture. Architecture establishes the context for design and implementation. Changing architectural decisions causes significant ripple effects. IEEE 1471-2000 defines software architecture. An architectural style is the classification of a system's architecture according to those with similar patterns. A pattern is a common solution to a common problem. An architecture is the result of weaving multiple views of the system from the perspective of different stakeholders. There are five fundamental views of a software systems (4+1 model): logical, implementation, process, deployment, and use case views. Not all systems require all views.

Logical view focuses on functionality, key abstractions, mechanisms, separation of concerns and distribution of responsiblities. This is where the aspect community is currently focusing. Challenge to the audience: what could aspects do in the other "views"?

Process view focuses on performance, scalability, throughput. Fruitful work for aspects here? Nothing happening yet.

Implementation view focuses on configuration management - the components used to assemble and release the physical system.Deployment view focuses on distribution, communication, provisioning. Use case view describes the behaviour of the system as seen by its end users and other external stakeholders. The logical view and the implementation view are tightly coupled.

Biological systems have crosscutting concerns: reproduction, protein creation, metabolism, cellular proliferation, heme synthesis, heat production,.. (and many more on Grady's slides). Concerns are not isomorphic to structure (there's no piece in the body that is the "temparature regulation" piece). In biological systems, these aspects evolved simultaneously and interdependently at each level of abstraction. What we have today with aspects is "cool stuff", but we can do even better than that when we look across the other architectural views. Some structures and behaviours crosscut components: security, concurrency, caching, persistent.

The role of AOSD: remember the fundamentals - don't make it more complicated. The current sweet spot for aspects involves elements of each of the fundamentals: especially with regard to building crisp abstractions and the separation of concerns for roles relative to packaging. This impacts primarily the interplay of the logical view and the use case view. Q. What are the semantics of an aspect? Tell me something that is an aspect, and something that isn't an aspect, and how we can classify the two.

The current pragmatic focus is upon transformation tools that focus on already visiblie artifacts. Tha harder focus - the most disruptive yet potentially the most valuable- is upon transformation tools that focus on deep semantic representations and then the creation of these traditional artifacts by reflection. Ie. source code as a pretty-printed side-effect, not a central object.

Make some simplifying assumptions by picking a single platform and see what you can do...

Closing thoughts: sw is fundamentally ,wickedly hard and it's not going to get any better. But the world doesn't need more technology - we need less, the best technology is invisible.

Posted by adrian at 04:49 PM [permalink] | Comments (0)

AOSD 2005 Conference has started...

I'm here in Chicago for the AOSD 2005 conference. It's a gathering of many of the key influencers, practioners, and researchers in the field. Last night was a great example. We had an impromptu BoF organised by Nick Leseicki and Eric Bodden on aspect instance lifecycles (per-xxx models). About 15 people showed up and we were able to have a really good discussion on topic. We bounced around all sorts of ideas, culminating in the following realisations:

  • You can use inner aspects of a perxxx aspect if you need to match join points that fall outside of the implicit scope of the perclause (see my earlier post on @Singleton for an example of this technique).
  • The combination of a perxxx model plus an ability to easily enable/disable the advice in an aspect instance covered every use case that participants raised. We discussed ways to make runtime enable/disable of aspect instances as easy as possible - including direct language support.

I really enjoyed the high bandwidth discussion and a chance to interact with some of the most sophisticated aspect-oriented programmers in the world. This really sums up what this conference is all about.

I'll try to keep some blogging going through the event so that those of you who can't be here can catch a glimpse of what's going on...

Right now I'm listening to "demo madness" - 3 minute overviews of all of the demos at the conference. Looks like there's some great stuff going on. It's amazing to see how many people are building (quite sophisticated) Eclipse-based tools for AOP. Eclipse really does seem to have won the IDE battle for AOP hands down.

Posted by adrian at 02:59 PM [permalink] | Comments (0)

March 09, 2005

Event Driven Architecture

At the recent TSS Symposium (a really great event btw - you should go if you ever get the chance), I had the pleasure of attending Gregor Hohpe's talk on event-driven architectures. He gave a very pragmatic approach to building event driven systems, only taking on as much complexity as you need.

There's an obvious connection between AOP and event-driven systems (join points are events that occur during the runtime execution of a program...). So I'm sitting at the back in Gregor's session, and I get to thinking, "how easy could I make this using AOP?" And then I'm scribbling on the notes page of the handouts. Fortunately it was the last session of the day, so I went back to do some hacking and get the idea out of my system. What follows is the result of that hour of playing around - which if you were at TSS and attending my "Adopting AspectJ" talk is also what I showed briefly during that session.

My first decision was to build an @DSL to support the event driven style. My @DSL has three annotations:

@Retention(RetentionPolicy.RUNTIME) public @interface RaisesEvent { String value(); }
@Retention(RetentionPolicy.RUNTIME) @Target(ElementType.METHOD) public @interface OnEvent { String value(); }
public @interface EventSubscriber { // marker }

Both fields and methods can be annotated with @RaisesEvent("eventName"). An event is published when an annotated field is updated, and after returning from an annotated method. Types that wish to receive event notifications are annotated with the @EventSubscriber annotation. Methods within such types can be annotated with @OnEvent("eventName") and will be invoked whenever a matching event is published. (I chose a simple name based matching scheme, other more complex schemes are clearly possible).

Here's a simple class that creates events:

class Producer { public Producer() {} @RaisesEvent("price-update") private int price; @RaisesEvent("calculation-complete") public int calculate(int x, int y) { return x*y; } public void setPrice(int price) { this.price = price; } }

And here's a class that subscribes to these events:

@EventSubscriber static class Recipient { // exposed public fields for testing public int calculationResult = 0; public int price = 0; @OnEvent("calculation-complete") void onCalculationCompletion(MethodBasedEvent event) { System.out.println("calculation-complete, result = " + event.getResult()); calculationResult = (Integer)event.getResult(); } @OnEvent("price-update") void onPriceUpdate(FieldBasedEvent event){ System.out.println("price-update, new price = " + event.getValue()); price = (Integer) event.getValue(); } }

The semantics I gave this system is that every instance of an @EventSubscriber class receives each event that it has an @OnEvent annotation for. The Producer and Recipient classes were defined as inner classes inside my test case, which passes as follows:

public class EventDispatcherTest extends TestCase { static class Recipient { ... static class Producer { ... public void testMethodEvent() { Recipient r1 = new Recipient(); Recipient r2 = new Recipient(); Producer p = new Producer(); int z = p.calculate(6,7); assertEquals("should have set result on r1", 6*7,r1.calculationResult); assertEquals("should have set result on r2", 6*7,r2.calculationResult); } public void testFieldSetEvent() { Recipient r1 = new Recipient(); Recipient r2 = new Recipient(); Producer p = new Producer(); p.setPrice(10); assertEquals("should have set price on r1",10,r1.price); assertEquals("should have set price on r2",10,r2.price); } @Override protected void setUp() throws Exception { super.setUp(); // reset "static" state outside of test class... SubscriberTracker.aspectOf(Recipient.class).removeAll(); } }

So how does the implementation look?

The implementation consists of two aspects, one to keep track of subscriber instances, and one to handle the publish/subscribe logic. There is also a mini hierarchy of Event interfaces and implementations (Event is the supertype of MethodBasedEvent and FieldBasedEvent). These Event interfaces hold information such as the new field value for field events, and the arguments and return value for method events.

Let's look at the SubscriberTracker aspect first.

import org.aspectprogrammer.eda.annotation.EventSubscriber; import java.util.Map; import java.util.WeakHashMap; import java.util.Set; /** * we need to keep track of all the instances of an event subscriber * so that we can dispatch events to them */ public aspect SubscriberTracker pertypewithin(@EventSubscriber *) { // use WeakHashMap for auto-garbage collection of keys private Map subscribers = new WeakHashMap(); /** * after returning from the initialization of any new object * (of the type that I am pertypewithin...), add it to the map. */ after(Object o) returning() : initialization(new(..)) && this(o) { subscribers.put(o,true); } public Set getSubscribers() { return subscribers.keySet(); } // for testing public void removeAll() { subscribers.clear(); } }

This aspect simply keeps track of all live instances of each type that has an @EventSubscriber annotation (see the pertypewithin clause). We need this so that we can dispatch events to them.

The EventDispatcher aspect is a little more complex, but not much more. Let's look at the aspect in parts. Here's the portion to do with event generation:

public aspect EventDispatcher { /** * the execution of any method with the @RaisesEvent annotation. * The annotation itself is exposed by the pointcut. */ pointcut eventGeneratingExecutionJP(RaisesEvent event) : execution(@RaisesEvent * *(..)) && @annotation(event); /** * Build a MethodBasedEvent from thisJoinPoint, and publish it */ after(RaisesEvent event) returning(Object ret) : eventGeneratingExecutionJP(event) { MethodBasedEventImpl methodEvent = new MethodBasedEventImpl(thisJoinPoint.getThis(), thisJoinPoint.getArgs(), ret); publish(event.value(),methodEvent); } /** * the set (update) of any field with the @RaisesEvent annotation * The annotation and the new value are exposed by the pointcut */ pointcut eventGeneratingSetJP(RaisesEvent event, Object fieldValue) : set(@RaisesEvent * *) && @annotation(event) && args(fieldValue); /** * Build a field event from thisJoinPoint and publish it */ after(RaisesEvent event, Object fieldValue) returning : eventGeneratingSetJP(event,fieldValue) { FieldBasedEventImpl fieldSetEvent = new FieldBasedEventImpl(thisJoinPoint.getTarget(), fieldValue); publish(event.value(),fieldSetEvent); } ...

This should all be fairly straightforward. Notice how easy it is to expose and use annotation values in advice.

Here's the portion of the aspect concerned with subscription. Whenever we load (static initialization) a type with the @EventSubscriber annotation we find all its methods with the @OnEvent annotation and keep them in a Map for dispatching events when they are published.

public aspect EventDispatcher { ... private Map> subscriptions = new HashMap>(); after() returning : staticinitialization(@EventSubscriber *) { subscribe(thisJoinPoint.getSignature().getDeclaringType()); } ... private void subscribe(Class clazz) { Method[] methods = clazz.getDeclaredMethods(); for(Method method : methods) { OnEvent onEvent = method.getAnnotation(OnEvent.class); if (onEvent != null) { Class[] parameterTypes = method.getParameterTypes(); if ((parameterTypes.length != 1) || (!Event.class.isAssignableFrom(parameterTypes[0]))) { throw new RuntimeException( "Signature of @OnEvent method must be V(Event): found " + method); } List subscribers = subscriptions.get(onEvent.value()); if (subscribers == null) subscribers = new ArrayList(); subscribers.add(method); subscriptions.put(onEvent.value(),subscribers); } } }

Notice that methods annotated with @OnEvent are expected to have a certain signature (a single parameter that is of type Event or a subtype of Event). We'll come back to that thought later.

Now that you've seen the subscribe helper method, it's as good a time as any to look at what publish does:

private void publish(String eventName, Event event) { List methods = subscriptions.get(eventName); if (methods == null) return; for( Method method : methods ) { Set subscribers = SubscriberTracker.aspectOf( method.getDeclaringClass()).getSubscribers(); for (Object subscriber : subscribers) { try { method.invoke(subscriber,new Object[]{event}); } catch (Exception ex) { throw new RuntimeException( "Unable to publish event to subscriber " + method + " : " + ex); } } } }

Note that whilst I've implemented a simple synchronous event dispatch it would be easy to change the implementation of publish to perform parallel asynchronous notifications using util.concurrent (or indeed to use any other mechanism to connect publishers and subscribers). The beauty of the scheme is that this change is completely encapsulated in the aspect.

Here's the final piece of the puzzle. We're building a mini @DSL that has basic rules associated with its usage: @OnEvent methods should only occur in types that have the @EventSubscriber annotation, and @OnEvent should only take a single parameter of type Event+, as described earlier. We can do better than a runtime check for this. We can actually extend the compiler to understand the rules of our @DSL. Here's the final piece of the puzzle:

declare warning : execution(@OnEvent * *(..)) && !within(@EventSubscriber *) : "@OnEvent methods must be declared in a type " + "with the @EventSubscriber annotation"; declare warning : execution(@OnEvent * *(!Event+)) : "@OnEvent methods must take a single argumement of type " + "Event (or a subtype of Event)";

Now if anyone using our aspect inadvertantly breaks one of these rules, the compiler will tell them straightaway.

All in all, a pretty powerful little system for such a small amount of code...

Posted by adrian at 07:46 PM [permalink] | Comments (1)

March 08, 2005

Ramnivas on annotations, and @Singleton

I was going to write up the @RaisesEvent stuff I showed at TSSS last week after seeing Gregor Hohpe's talk. Maybe I'll get to that tonight. However, I just saw Ramnivas Laddad's blog entry on annotations and couldn't resist interrupting the intended schedule to knock out a quick prototype.

Ramnivas suggests using an APT (annotation processing tool) to support an annotation @Singleton:

@Retention(RetentionPolicy.RUNTIME) public @interface Singleton { public SingletonKind value() default SingletonKind.Lazy; } public enum SingletonKind { Lazy, // create the singleton instance upon first use Eager; // create the singleton instance as soon as possible }

(I added the runtime retention annotation, the rest of the code is Ramnivas').

So the question is of course, can you implement this in AspectJ 5 without needing to write a custom APT?

Turns out the answer is yes :) Here's my aspect:

public aspect SingletonManager pertypewithin(@Singleton *) { Object _instance = null; /** use a static inner aspect as we need to advise call jps * that happen outside of the pertypewithin type... * (and the implicit && within(T) would exclude them) */ static aspect SingletonManagerHelper { pointcut singletonCreation() : call((@Singleton *).new(..)); Object around() : singletonCreation() { Class clazz = thisJoinPoint.getSignature().getDeclaringType(); SingletonManager mgr = SingletonManager.aspectOf(clazz); if (mgr._instance == null) { mgr._instance = proceed(); } return mgr._instance; } } // Eager instantiation support after(Singleton singleton) returning : staticinitialization(@Singleton *) && @annotation(singleton) { if (singleton.value() == SingletonKind.Eager) { try { _instance = thisJoinPoint .getSignature().getDeclaringType().newInstance(); } catch (IllegalAccessException illEx) { ; // nothing we can do - default cons not visible } catch (InstantiationException instEx) { ; // nothing we can do } } } }

I have two simple classes:

@Singleton(SingletonKind.Eager) public class EagerSingleton { static public boolean instanceCreated = false; public EagerSingleton() { instanceCreated = true; } } @Singleton(SingletonKind.Lazy) public class LazySingleton { }

These classes are used by my (passing) test cases:

public class TestSingletonSupport extends TestCase { public void testLazySingleton() { LazySingleton l1 = new LazySingleton(); LazySingleton l2 = new LazySingleton(); assertEquals("Should be a singleton",l1,l2); } public void testEagerSingleton() { assertTrue("EagerSingleton already created", EagerSingleton.instanceCreated); } }

I think this is the first time I've ever used an inner aspect of an aspect! Happy hacking....

Posted by adrian at 05:11 PM [permalink] | Comments (2)

March 05, 2005

The New Holy Trinity

No, this isn't a religious post. It's about an approach to developing applications. I'm not the only one advocating this approach, and I'm not the first to write about some of the ideas here. What I want to do in this post though is put all the pieces together in one place and boil the approach down to its essence to give you a simple way of thinking about it. So here it is:

Dependency injection, annotations, and aspects are the three foundations on which the next generation of world-class POJO-based applications should be built.

At the centre of an application designed according to this principle are plain-old Java objects (POJOs). Each POJO should do one thing and one thing only. Each POJO should also know one thing and one thing only - how to do what it is that it does. This is the 'secret' that the the POJO hides from the rest of the application. So for example, a BankAccount POJO should know about the internal state of a bank account, and the operations that it supports. It should not know anything about the way any services it needs are created or deployed. It should not know anything about the authentication or authorization requirements that may be placed on it in a particular deployment scenario - or worse still, anything about the implementation of authenticaton and authorization services in the environment in which it is deployed. It should not know anything about account activity monitoring to detect potentially fradulent use. And so on.

I've written about this notion of a 1:1 mapping between distinct concepts in the design and constructs (types) in the implementation before in "AOP without the buzzwords" (this essay was also updated and expanded to form the introduction for our book, Eclipse AspectJ).

To use some old software engineering terms, what's at the centre of applications designed according to these principles are highly cohesive POJOs. These POJOs are very loosely coupled to the rest of the application. In particular, if you look at the three sides of the triangle in the illustration, you'll see that dependency injection ensures the POJO remains unaware of any of the details of how its dependencies are instantiated and configured. DI shields the POJO from this information. Aspects ensure the POJO remains unaware of any crosscutting concerns in the application (such as the account activity monitoring concern in the example above). Aspects are what enable the POJO to focus on doing just one thing and to do it well. Annotations provide additional metadata about the POJO that facilitate easy integration with external services (such as transaction services). They ensure that the POJO remains unaware of any of the details of such services. Annotations coupled with aspects can do much more than this though - they also support the creation of domain specific abstractions that easily bring back to the programmer Abelman and Sussman's often missing third tool in the war against software complexity: the creation of "little languages" (see my post "A Beautiful Language").

This leads me to a really important point about this trinity of DI, aspects, and annotations. The three techniques are mutually self-reinforcing, the whole is much greater than the sum of the parts.

Dependency injection can be used to configure aspects. This is important since when you're building an application using the 1:1 principle, some of the POJOs in the centre of the triangle will themselves be aspects (in a language like AspectJ, this is perfectly natural). Anecdotal evidence from a number of projects across a number of organizations suggests that maybe about 10% of your "POJOs" will be aspects. Dependency injection can also be guided by annotations. You see an example of this in the EJB 3.0 specification (you may recall my own thoughts on the suitability of the chosen annotation name in that spec "What's wrong with @Inject?", but that's not material to this discussion). Completing the circle, aspects can also be used to implement dependency injection scenarios. In general you're best leaving this job to an IoC container, but if you have DI requirements that go beyond the wiring of beans as they are instantiated by the container, aspects are a great solution. To give a couple of examples, in the Eclipse AspectJ book we use aspects to implement a 1:1 design for POJO persistence using Hibernate. We use dependency injection on a per-request basis to inject the Session object that DAOs use for communicating with Hibernate. A second example is the use of an annotation @SpringConfigured(beanName="XXX") on POJOs. It's easy to write an aspect (configured by DI to be BeanFactoryAware in Spring terms) that uses Spring to autowire any instance of an @SpringConfigured object when it is instantiated. This lets you separate instantiation and configuration (normally an IoC container like Spring does both for you) so that you can place instantiation under programmatic control, but still get configuration via DI.

Let's look at another really important way in which these techniques work together. Annotations on their own give you a way of specifying additional metadata for program elements. This is useful when you have tools or services that can understand those annotations and do something with them. Annotations coupled with aspects is a whole other ball game altogether. Aspects let you use annotations to easily implement declarative domain-specific languages (DSLs) as part of your application (or aspect library). This is a really big deal. Aspects can do this since they let you associate behaviour with join points relating to annotated elements. To give an example I've written about recently ("Making Concurrency a Little Bit Easier"), you can easily write a set of annotations that form a declarative DSL for concurrent applications (the domain of this DSL is concurrency). Another example I'll write about in my next post: after attending Gregor Hohpe's talk at TSS yesterday on event-driven architectures I quickly implemented an annotatation-based DSL ("@DSL") for simple event-driven applications. It took me less than an hour. That's pretty powerful (you can judge for yourself when you read the post). Annotations plus aspects let you write these @DSLs.

When you're using @DSLs like this in the construction of your application, you're going to want an aspect language that has rich support for join point matching based on annotations. There's a surprising amount to this, and you can see how AspectJ 5 handles it by reading the "Annotations" chapter of the AspectJ 5 Developer's Notebook.

There's another way in which the combination of aspects and annotations are self-reinforcing too. If you are using an annotation-driven tool or service for a particular deployment of your application, you may need to annotate your POJOs with annotations specific to that tool or service. If you don't want to tie your POJOs to that annotation set (see "When is a POJO not a POJO?") you can use the AspectJ 5 support for "declare annotation" to keep the annotations separate and maintain a loose coupling. You might also want to use declare annotation if a particular tool or service requires you to spread information you consider to be configuration information around your source code (in the form of annotations). Here an aspect using declare annotation can at least get that all back in one place for you.

If this all sounds a bit circular (DI of aspects, aspects to perform DI, annotations to control aspect behaviour, aspects to declare annotations,...) that's because it's meant too. The sides of the triangle all support each other and can be used together in many different ways to meet the needs of your particular situation. The whole is much greater than the sum of the parts.

In summary, I believe that dependency injection, annotations, and aspects are the three foundations on which the next generation of world-class POJO-based applications should be built. At the heart of such applications are POJOs that do one thing only and do it well. The triangle illustration provides a simple visual mnemonic to help you remember the principles. An application constructed according to these principles could be said to follow a "triangle design". You can even easily communicate your design intentions in good old ascii:

        /o\
	 -
	

Glossary

1:1 principle
A way of structuring a software system such that there is a 1:1 correspondence between concepts in the design and implementation constructs.
@DSL
A declarative domain-specific language implemented using a combination of annotations and aspects.
Triangle design
An approach to structuring a software system in which POJOs written according to the 1:1 principle are supported by dependency injection, annotations, and aspects.

References

Disclaimer

As with any post on this site, this post represents my own personal opinions and not necessarily those of my employer.

Posted by adrian at 04:04 PM [permalink] | Comments (8)