Tuesday, December 6, 2011

Dynamic Groovy Edges and Regression Isolation

In JEE application architecture, components are grouped into functional layers.  In a typical multi-tier enterprise web application, there are N layers, N-1 layer junctions, and (N-1) x 2 layer edges.  In the diagram below, for example, there are three layers, two layer junctions, and four layer edges.

Designing and implementing interfaces between application layers can be painful, especially if you don't have a span of control or even influence design and/or implementation.  Interfaces are notorious change zones.  Even with well-defined interface contracts, development teams are routinely confronted with interface changes that require quick turnaround times, compounded by development, test, and deployment overhead.

If we examine why we have interfaces in the first place, we soon realize that we need interoperability (data sharing) between different systems.  It would be desirable to design systems with intrinsic interoperability, but that is a lofty goal that is seldom realized outside of SOA boundaries.  No, we have different datasets in different systems.  So we have interfaces.

It is not only the data that is different, the pace with which different systems and layers evolve is also different.  Junctions are by default, separations between different layers changing at different paces.  For example, the web presentation layer changes rapidly, as the services layer more slowly and the data layer even slower still.  In architecture, we design with "pace layering" in mind, so that we can provision the appropriate interfaces to allow for change, for evolution, at different paces.

If the reader is interested in "pace layering", here is one of the best descriptions of Pace Layering, from Donna Maurer, that I have ever read:
Complex systems can be decomposed into multiple layers, where the layers change at different rates. The “fast layers” learn, absorb shocks and get attention; the “slow layers” remember, constrain and have power. One of the implications of this model is that information architects can do what they have always done–slow, deep, rich work; while tagging can spin madly on the surface. If it is worth keeping, it will seep down into the lower layers.
I agree with Isaac Asimov:  "The only constant is change..."  And, it is difficult to design for constant change.  You just can't code for every event that could possibly happen.  Applications would be woefully bloated and impossible to maintain.  So, we must embrace change, and plan the best methods for reacting to change.  In JEE applications, the best method is the one that introduces the least amount of change to the overall application.  We must also avoid regression loads, during change, if possible.

Lately our teams have been looking at creative ways to use existing Java technologies to create more nimble application interface edges.  The first step was to ask ourselves, what Java tool allows us to quickly change code without having to redeploy our applications?

My first thought was JRebel from Zero Turnaround.  Using JRebel, developers can "instantly see any code change made to an app without redeploying."  However, unless I am mistaken, JRebel is primarily for non-production use.  There may be valid reasons to use JRebel in production systems.  However, I have never worked on a project where that type of JVM manipulation was allowed in production, especially triggered from developer desktops.  There is usually so much bureaucracy and change control processes to tangle with.

Redeploying an application contained in an EAR or WAR to production is the choice of last resort.  On top of the overhead of following the proper change control processes (ITIL anyone), there is the large regression analysis and testing that needs to happen.  It would be great if we could isolate the most dynamic areas of our application (interfaces), so that we could change them in reaction to external changes beyond our control.  It would also be desirable to make these changes in an isolated manner so as not to require an EAR or WAR redeployment.

To solve this dilemma, the initial knee-jerk reaction is to write so-called "smarter" interfaces to account for known possible changes.  That approach will only get you so far.  Nay, the solution to this muddle is to step back and unlearn what we thought we learned earlier in our careers. Actually, the approach that I am getting ready to espouse is similar to the Dale Carnegie approach to handling problems.  The one thing that stuck with me from my class on Human Relations was to consider the worst scenario, accept that scenario if you could not prevent it, and then plan on a mitigation strategy.

So, the worst case scenario is that our interfaces will change; systems will evolve at different paces and our junctions will be affected.  Accept that which we cannot change.  Now, how do we make this situation better?  Well, do what we already know how to do:  write code to satisfy the change.  For JEE applications, this means writing Java...or does it?  What if we could write code that would execute as Java byte code in the JVM, but would not require us to redeploy our entire application.  That would be Groovy.  Yes, the pun was intended.

No matter what you think about the Groovy language, it does offer a solution to our dilemma, provided that we also include certain other "standard" components.  First, the solution is seen below.  we are talking about creating dynamic groovy edges.  With "Groovy Edges" our interfaces are now bracketed by dynamic code components that can actually be isolated from the rest of the application stack.
What is it about Groovy that best suits it for this task?
  1. Groovy can be written as a script and parsed at run-time and compiled into Java byte-code and then loaded by the Groovy class loader.
  2. Groovy scripts can contain Groovy and Java syntax in the script file (*.groovy).
  3. Groovy can use the same Java APIs that are used by traditional Java code.
OK, so how do we write Groovy to be dynamic?  Well, you might first jump to conclusions and recommend Compile-Time Meta-programming with AST Transformations or the Expando Meta Class.  While you might use these methods, the actual solution is even less technical provided the right infrastructure is in place, as seen below.

The solution starts with a JRockit JVM.  Since we will be parsing and compiling Groovy scripts at run-time to execute in the JVM, there will be some impact to class-loading resources.  Unfortunately, for this solution, Hotspot is not adequate, unless of course we define huge heaps with huge permgens in a 64 bit environment.  However, the Java 7 JVM shows promise with how it manages class-loading resources.

Next we need Spring; that should be no issue as many JEE applications use Spring.  Spring allows us to leverage Spring Bean Factory configurations to point to where our "Groovy Beans" scripts are stored.  For this solution, we will store these Groovy scripts outside of the JEE EAR (or WAR) files to preclude the need for deployments when the Groovy changes, and reduce application-wide regression testing and analysis.

 <!-- insert the following for debugging - refresh-check-delay="1000" -->  
      <lang:groovy id="GroovyInterfaceEdge" script-source="${groovyInterfaceEdge}" refresh-check-delay="1000" >  
        <lang:property name="key" value="value"/>  
         </lang:groovy>  

In the code snippet above, the SpringBean "GroovyEdgeObject" is loaded from a URL that is resolved by the Spring EL in the "script-source" attribute.  The "refresh-check-delay" attribute is set to refresh the object in Spring's IOC container every second if changes in the script are detected between object method calls.  With this approach we could store the remote scripts in a CMS or even Amazon Simple Storage Service (S3).  The key point is that the Groovy script is stored outside of our typical regression boundary.  And, Spring will take care of updating the IOC container when changes are detected in the script.

There is additional magic happening with the Spring EL value in the "script-source" attribute above.  This argument, "${groovyInterfaceEdge}", is known as a property placeholder.  Spring uses the "PropertyPlaceholderConfigurer" to load a properties file and then get values from that properties file that match the placeholder keys in bean definitions.

 <!-- Read service related properties from the classpath -->  
    <bean id="services.propertyConfigurer"  
              class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">  
              <property name="location">  
                   <value>classpath:services.properties</value>  
              </property>  
         </bean>   

In the code just above, the services.properties file is loaded from the classpath and the key/value pairs are made available to bean definition attributes using Spring EL placeholders.  So, "${groovyInterfaceEdge}" is resolved to a value in the services.properties file found on the application classpath.  In this example, the "${groovyInterfaceEdge}" would resolve to a URI that can be accessed via the application to load the Groovy script stored in the remote script storage facility. 

In this example, the Groovy script is accessible via a web URI.  Depending on how the CMS is configured in your environment, it may not provide this URI access to all resources.  In this case, we would build and configure a "Content Proxy" that could use the API supplied by the CMS vendor to get at the script on the behalf of the Spring Bean Factory.  When a content proxy is used, the URI resolved by the Spring EL placeholder and PropertyPlaceholderConfigurer API would be an endpoint.

 package com.ironworks.restwebservice.content;  
 import java.io.InputStream;  
 import java.text.SimpleDateFormat;  
 import javax.servlet.http.HttpServletResponse;  
 import javax.ws.rs.GET;  
 import javax.ws.rs.Path;  
 import javax.ws.rs.PathParam;  
 import javax.ws.rs.Produces;  
 import javax.ws.rs.core.Context;  
 import javax.ws.rs.core.MediaType;  
 import javax.ws.rs.core.UriInfo;  
 import org.apache.commons.logging.Log;  
 import org.apache.commons.logging.LogFactory;  
 import com.bea.content.AuthenticationException;  
 import com.bea.content.AuthorizationException;  
 import com.bea.content.ContentContext;  
 import com.bea.content.NoSuchPropertyException;  
 import com.bea.content.Node;  
 import com.bea.content.Property;  
 import com.bea.content.RepositoryException;  
 import com.bea.content.expression.Search;  
 import com.bea.content.federated.ContentManagerFactory;  
 import com.bea.content.federated.INodeManager;  
 import com.bea.content.federated.ISearchManager;  
 import com.bea.content.paging.ContentListKeys;  
 import com.bea.content.paging.ISortableFilterablePagedList;  
 //NEXT 999 LINES SUPPRESS LineLength  
 @Path("/contentProxy")  
 public class ContentProxy {  
       @Context  
       UriInfo uriInfo;  
       private static final Log log = LogFactory.getLog(ContentProxy.class);  
       private static SimpleDateFormat dateFormat = new SimpleDateFormat("EEE, dd MMM yyy HH:mm:ss z");  
      @GET  
      @Path("/{directory}/{nodename}")  
      @Produces(MediaType.TEXT_HTML)  
      public final InputStream contentLookup(  
                @PathParam("directory") final String directory,  
                @PathParam("nodename") final String nodename,  
                @Context HttpServletResponse response) {  
           final Search search =  
          new Search("/WLP Repository/" + directory, -1, "",  
              " cm_nodeName= '" + nodename + "'");  
        final ISearchManager searchManager =  
         ContentManagerFactory.getSearchManager();  
        final INodeManager nodeManager = ContentManagerFactory.getNodeManager();  
        final ContentContext cc = new ContentContext();  
        cc.setParameter(ContentListKeys.BATCH_LOADING_SIZE_KEY, -1);  
        ISortableFilterablePagedList<Node> results = null;  
        try {  
         results = searchManager.search(cc, search);  
        } catch (Exception anything) {  
         // fatal. We cannot handle it.  
         throw new RuntimeException(anything);  
        }  
        for (Node oneNode : results) {  
         try {  
          if (oneNode.getProperty("copy").getType() == Property.BINARY) {  
               response.setHeader("Last-Modified", dateFormat.format(oneNode.getModifiedDate().getTime()));  
           return nodeManager.getStream(new ContentContext(),  
             oneNode.getId(), "copy");  
          }  
         } catch (AuthorizationException e) {  
          e.printStackTrace();  
         } catch (AuthenticationException e) {  
          e.printStackTrace();  
         } catch (NoSuchPropertyException e) {  
          e.printStackTrace();  
         } catch (RepositoryException e) {  
          e.printStackTrace();  
         }  
        }    
        return null;  
       }  
      }  

The code just above is an example of just such a content proxy, though mileage may vary depending on what flavor of CMS is used.  This class, along with the Jersey REST API creates a RESTful web service endpoint that will take in content proxy requests and return the script file from the CMS.  The REST approach is well suited for this type of service, where we simply want to parse the URI components to get at the parameters of the service call.

 groovyInterfaceEdge=http://<HOST>/restServices/contentProxy/scripts/groovyInterfaceEdge  

Putting it altogether, the services.properties file entry above shows how we would configure the URI to call the RESTful content proxy endpoint.  This URI is then returned to the Spring Bean Factory when the Groovy Bean is asked for by the application.  The content proxy returns the Groovy script file (*.groovy) to the application where the Groovy class loader takes care of loading into the JVM for use by the application.

With this solution, we have used convention and well known frameworks to build a sustainable and dynamic interface edge framework.  However, this solution opens up many other possibilities for our applications.  There may be other areas where we would like to load Spring Beans that are dynamic, so that we can change key behavior of volatile areas within our application without having to incur the load or redeployment and major regression testing.

Thursday, December 1, 2011

Richmond Times Dispatch - 2011 state employee salary database - AGAIN!

I received this email from the RTD:
The Times-Dispatch and TimesDispatch.com will publish our annual report on state employee salaries this Sunday, Dec. 4. This year's searchable database includes salaries for all 104,554 state employees, including names and positions for all employees who earn above the state's average of $52,547.
They did the same thing last year.  Here is how I felt last year when I was a state employee caught up in this database:


On LinkedIn.com you refer to your company as “the leading provider of high-quality news…in central Virginia.”  I think at best that claim is suspect, and at worst, you are providing false panoply of episodic capabilities to the public, to hide the depth of your desperation to remain a viable news source.  The Richmond Times-Dispatch has a few redeeming qualities, but taken in aggregate you offer very little to your customers that they could not more freely obtain from a multitude of other online outlets.  There are times when publishing FOIA information is warranted; however, this is not one of them, at least not in its current form.  It is especially unwarranted when the purpose is to fuel a negative agenda.  In economically challenging times, Federal and State employees are easy targets for efficiency and effectiveness ratings.  However, I submit that there are many employees that work very hard and that would and could make more money should they move to the private sector and leave public service behind.  Your irresponsible action in publishing the salaries of state employees, SEARCHABLE BY NAME, smacks of arrogance and sensationalism and is bereft of any dignity or integrity that your newspaper claims to still retain.  As far as I am concerned, my name in the context of my agency and position is “Personally Identifiable Information” and as such should not be used. 

The damage of your actions far outweighs the value delivered to your readers and the general public.  There is absolutely no constructive reasoning that justifies listing employees names in your database.  From your actions it is clear that you completely disregarded the negative impact that such a database would have on state employees.  These same state employees are also your customers.  With this contemptible action, your organization is morally and ethically bankrupt, and you owe your readers an apology for wasting their time with such twaddle.

I would be very interested to know the salaries of each of your employees making over $50K, as well as their job titles. And of course, I would need the names of said employees to more easily target my public disdain for your obviously poor value proposition and the gross mismanagement of your financial resources.  If that makes no sense to you, then you now realize the situation in which thousands of state employees find themselves, without redress.  I am sure that your employees would not mind; after all, your customers are paying you for your information services.  Do they not have the right to get the best value for their money?  Are not the salaries of your employees just more information about which someone, somewhere would want to read?  Is that not how you choose to publish these days?  Of course you need not answer these rhetorical questions; your actions speak volumes about your motivations.
Just because you can, should not be a license to pursue a course of action that has no real probative or news value and genuinely provides a divisive tool for someone’s personal agenda.  Your misguided exploit is completely irresponsible and in my opinion an abuse of said state employees, your customers, and the spirit of the Freedom of Information Act.  It is also an abuse of your power and a misuse of your first amendment rights.  Furthermore, your actions underpin my long held belief that nothing useful comes from your disgraceful and at times, Yellow-Journalistic tabloid.  You may have temporarily boosted your circulation or at best your web site traffic, but you have undoubtedly isolated yet another large customer segment and undermined the public’s trust.  Moreover, if more of your readers were affected by this debauched deed you would undoubtedly apologize or even redact said folly.  Indeed, you misunderstand, in entirety, the meaning of cultivating long-term customer value and loyalty.  This entire episode has been a tactless and colossal waste of our time and attention.  You may be the press, but you are also a business.  I think you should cease acting as the self-anointed herald of worthless hokum and actually cover creditable news that makes our lives better and provides at least a modicum of value to your customers.  However, if you are disinclined to act appropriately, one can only hope that your readers recognize how shallow, foul, and jaded you have become and move on to better outlets of information.  For myself, I am almost giddy with the thought that I do not have a subscription to your worthless paper.  I now find it less than suitable to line the bottom of my dog’s crate!

Thursday, November 10, 2011

JEE Developers: Embrace the Polyglot in You

Are you a polyglot programmer?  The term almost has a guttural pronunciation.  In short, a polyglot person has the ability to speak multiple languages.  So, a polyglot programmer is one who can write code in multiple languages.  Most programmers I know are polyglots out of necessity or career evolution.  I started out with BASIC, then moved to LabVIEW, HTML, Visual Basic and even Lotus Notes, settling into Java and its cohorts later on.  Along the way I wrote C and C++ when the need arose, as well as the suite of UNIX/Linux scripting languages (Perl, Korn, Python, etc.).

Most of us have started out in one language only to switch later to more modern, and/or more expressive languages.  Occasionally we may digress, if the situation calls for older skills, but in general we have moved forward, building on previous knowledge.  I can hear some of you saying right now that true expertise takes years to build.  I can agree with that, to a degree.  It should take years to become an expert in some languages or technologies.  However, expertise is a relative term, and it should not take decades.  Furthermore, most projects don't staff all experts; there is room for intermediate levels as well.

There are those that have stayed with older languages, for whatever reasons; comfort not being the least of these.  However, it seems to be harder for those individuals to adapt to modern needs, and that makes them less desirable to recruiters.  To remain relevant, we need to stretch outside our comfort zones.  Stories are legion these days of folks searching for work, only to find out that their skills are simply outdated.  When I hire, I try to hire for organizational fit first and then skills second.  However, in this economy, recruiters are buyers in a buyer's market.  Some would say, that without the correct buzzwords on resumes or applications, otherwise competent resources are getting overlooked and ignored.  I would say that the skills of these programmers have become less relevant.  As one of my friends would say, "They may have placed themselves in a position to be unlucky."  If your skills are no longer relevant to the strategy of your company or its competitive advantage, you may be unlucky sooner than you would like.

I say embrace the polyglot way.  A good software architect chooses the right systems or layers for the right processing.  Polyglots are not only able to understand the code, they frequently write the code in those different systems or layers.  For example, in Oracle, if you need to process large data sets in a iterative and procedural manner, use PL/SQL.  PL/SQL also gives you better application management and performance, if done correctly.  Do high-level field translation and validation in the HTML form or view layer with JavaScript to enhance the UX.  Do more in-depth data validation in the business layer for data quality and security reasons.

As a software developer you can ill afford to dedicate your focus on one language while ignoring the rest.  Especially if your language (or framework) has complementary cohorts that are used routinely in application implementation.  If you call yourself a JEE developer, then you already have a lot of Java APIs to master.  On top of the JEE technologies, you should also know Spring and Hibernate, intimately, including Spring MVC and EL as well as HQL.  Groovy is also a good language to know especially if your web application needs dynamic integration, rapid prototyping, and/or can leverage ECMS.

You'd better know SQL, and if you work on Oracle projects, know PL/SQL.  If you are writing JEE web applications, know HTML, CSS, and JavaScript.  I am not saying that backend developers have to be frontend experts, but one should know what goes on over the fence from them.  In fact, for all networked applications, know the protocols you would be using (TCP/IP, HTTP/HTTPS, etc).

If your applications are deployed in a UNIX/Linux environment, know scripting languages.  For view-layer integration, know AJAX and JSON.  Finally, almost nothing is possible without XML.  Do you know what it means to be well-formed and valid?  Do you know DTDs and XSDs?  Could you translate XML to something else with XSL/XSLT?  If you are not developing some sort of web service or other integration with XML you most likely have XML configuration files in your application.

In short don't limit yourself when applying for positions or choosing projects because you only focus on Java or .Net.  Know the cohort technologies as well.  You don't need to be an expert in all things, just as an IT generalist is also in low demand.  You need to balance depth with width.  Focus on core technologies, and then be familiar with complementary technologies.  Develop more depth in the non-core technologies when you have sufficient comfort level in the core.  This is not fake-it-to-you-make-it either.  From time to time you will have the opportunity to be a temporary expert in more ancillary technologies or languages or frameworks.  You may even develop a temporary monopoly in new technologies or approaches.  That is where real value is added to your organization.  Even if you can get projects with your older skills, will you be happy with those projects?

If you know enough about the side technologies to spin-up quickly, that should be enough for most projects.  Scope the landscape for relevant technologies and then choose wisely when looking to enhance your capabilities.  In my experience, it is easier to keep current in these technologies if they are complementary, and if you have the chance to use them for real deliverables.  Don't place yourself in a position to be "unlucky".  Be cognizant of the relationship between your skills and the strategy of your organization and the direction in which you wish your career to proceed.  Embracing the polyglot approach can differentiate you from your contemporaries, making you more relevant to your organization and the job market.

Friday, October 7, 2011

PMI CVC PMP 2011 Fall Workshop - Framework Talk

I will be presenting on October 15 for the PMI Central Va Chapter during their Fall 2011 PMI Certification Workshop.  My topic will be the Project Management Framework.  My talk goes from 10:00 AM to 11:15 AM and touches on items found in sections 1 & 2, chapters 1, 2, & 3, of the PMBOK, version 4.

I have been presenting this topic for the PMI CVC for about a year.  In my session I will be discussing these topics and more:
  • Projects, Portfolios, and Program
  • Process Groups, Knowledge Areas, and Processes
  • PMO
  • Project Life Cycle vs. Product Life Cycle
  • Stakeholder Management
  • Organizational Structure
Ever wondered how PMI keeps the PMP exam current and relevant?  This year we have also added information of the PMI Role Delineation Study and the Crosswalk.

This workshop is a great way to come up to speed for the PMP exam as well as gain valuable study tips from fellow project managers that have already passed the exam.  The workshop is also a great opportunity to gain the PDUs needed to maintain existing PMP certifications.  Best of all, attendees receive copies of all the slides presented at the workshop as well as other resources to help them study for the exam.

Monday, September 26, 2011

Greed is a Terrible Thing - Goodbye to the SCEA

Industry pundits have speculated on Oracle's ability to be stewards for Java.  Since when does a good technology steward prevent adoption of said technology?  Can you say greedy-failure?

I tried to register for the Oracle Certified Master, Java EE 5 Enterprise Architect (formerly SCEA) certification today.  After the initial sticker shock of $900 for the cert, I soon realized that Oracle has increased the price by making it mandatory to take one of their training classes in order to gain this certification. I guess they do this for several of their other certifications. That is another several thousand or so.  Mind the reader, they are not requiring that you take all the courses that would teach you about the different technology concerns in the architecture certification.  So, where is the value of making candidates take a class if they can choose the simple "Web Component Development with Servlets & JSPs" or "Java Programming Language"?  What architect cannot write servlets and JSPs, or not understand their positioning?  What architect does not know the difference between HashMap and a Hashtable?

In my opinion, this is a blatant act of greed and a splendid display of stupidity.  Oracle (and its mindless sycophants) can certainly argue that this move strengthens the value of this certification.  Utter twaddle-speak says I!  Pricing a certification to make it unreachable does not make it more valuable to the industry in aggregate; it just makes it unreachable.  This benefits no one, but Oracle and those already certified. 

I (along with several contemporaries) believe this will severely limit the certification to those of us not already engaged in austerity measures (self imposed or not), brought on by the poor economy.  Are those the boundaries that Oracle was actually hoping for?  With this callous, and extremely short-sighted folly, they have effectively isolated a segment of Java architects not backed financially by an organization willing to pay the ridiculous expense.  They have created a class of Java technologists based solely on monetary funds available to the technologist.  Way to go Oracle!

Instead of making the exam more valuable by making it harder to pass (currently you need a 57% to pass the "multiple-guess" exam, part 1 of the exam trilogy) , they seem to be trying to make it more valuable by placing it out of reach, monetarily.  Technology is supposed to facilitate the crossing of economic boundaries, not put in place those of its own making.  This myopic move will only further dilute the viability of Java as an enterprise solution.  There is already a split within the Java community, many of us embracing the simplicity and reuse of the multiple technologies (Spring, Hibernate, Freemarker, Groovy, etc.) not directly espoused by the EE spec and leaving the complexities of EJBs and JSF behind.  I mean, I can certainly specify and write EJB 3.x and JSF; I just choose not to in most cases.  And do not even get me started on Oracle's WebLogic platform.

Large corporations seem to have a knack for destroying technologies with "kinetic stupidity".  IBM and Lotus Notes/Domino anyone?  However, isolating an entire segment of your long-time supporters just to make a few more dollars is just a colossal failure.  They had an opportunity here to move in a positive direction.  In true monolithic fashion, they took the opportunity to miss that opportunity.  It seems that they chose to increase their bottom-line at the expense of the community that has bolstered Java for the last decade or so.

Tuesday, July 12, 2011

JSR 303 – Patterns and Anti-patterns with Hibernate and Spring Web Flow

According to the Java Community Process (JCP), Java Specification Request (JSR) 303 defines “…a meta-data model and API for JavaBean validation based on annotations…”  Simply put, JSR 303 specifies a set of predefined annotations and a facility to create custom annotations that make it easier for Java Bean designers to define how their model objects will be validated at run time.  Similar to and often used with JPA annotations, Javax Bean Validation annotations provide an easy way to apply common validation patterns driven by annotated constraints.  These constraint annotations can specify default messages for constraint violations, or the messages can be resolved at runtime using the Java properties file facility.

In general, Java developers regularly apply design patterns when building applications or components.  During the last decade we have embraced annotated programming as a means of applying common patterns, and Javax Bean Validation annotations follow common patterns that reduce the variability in model validation.  Listing 1 is a snippet from an example Customer bean with both JPA and Javax Bean Validation annotations.

Listing 1
import java.util.Date;

import javax.persistence.Column;
import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.Id;
import javax.validation.constraints.Future;
import javax.validation.constraints.NotNull;
import javax.validation.constraints.Pattern;
import javax.validation.constraints.Size;

@Entity(name = "CUSTOMERS")
public class Customer {

    @Id
    @GeneratedValue(strategy = GenerationType.TABLE)
    private long id;

    @NotNull(message = "{Customer.firstName.NotNull}")
    @Size(min = 1, max = 30, message = "{Customer.firstName.Size}")
    @Column(name = "FIRST_NAME")
    private String firstName;

    @NotNull(message = "{Customer.lastName.NotNull}")
    @Size(min = 1, max = 30, message = "{Customer.lastName.Size}")
    @Column(name = "LAST_NAME")
    private String lastName;

    @NotNull(message = "{Customer.email.NotNull}")
    @Column(name = "EMAIL")
    @Pattern(regexp = "^[\\w-]+(\\.[\\w-]+)*@([a-z0-9-]+(\\.[a-z0-9-]+)*?\\.[a-z]{2,6}|(\\d{1,3}\\.){3}\\d{1,3})(:\\d{4})?$", message = "{Customer.email.Pattern}")
    private String email;

    @NotNull(message = "{Customer.phone.NotNull}")
    @Column(name = "PHONE_NUMBER")
    @Pattern(regexp = "^\\(?(\\d{3})\\)?[- ]?(\\d{3})[- ]?(\\d{4})$|^(\\d{3})[\\.](\\d{3})[\\.](\\d{4})$", message = "{Customer.phone.Pattern}")
    private String phone;

    @NotNull(message = "{Customer.lastActivityDate.NotNull}")
    @Future(message = "{Customer.lastActivityDate.Future}")
    @Column(name = "ACTIVITY_DATE")
    private Date lastActivityDate;

In Listing 1, the validation annotations are easy to understand.  The firstsName and lastName fields have constraints on them imposed by annotations that do not allow null values and that constrain values to a certain size.   The interesting thing about these @NotNull and @Size annotations is that they retrieve their respective constraint violation messages using a message key, {Customer.firstName.Size}, instead of defining a literal message string.  The message arguments of Javax Bean Validation annotations can take a literal string as a default constraint violation message, or a key that points to a property value in a Java properties file that is in the root classpath of the application.  Externalizing these string messages aligns the JSR 303 approach with other annotated programming and message processing techniques used by Java developers today.

The additional constraint annotation seen in the example in Listing 1 is @Pattern.  This annotation, shown in its simplest form, takes both message and regexp arguments.  The regexp argument is a Java string regular expression pattern that is applied as a matching constraint.   In my testing, I tried supplying this value via String Enum arguments and by looking it up from a properties file, much the same way messages are resolved.  This would not work as I kept getting the error, “The value for annotation attribute Pattern.regexp must be a constant expression.”  However, I was able to use a static String constant.  This seems to violate the convention of externalizing Strings to properties files; perhaps this will change in the near future.

Beyond size, null, and pattern constraints, JSR 303 has several other predefined constraint annotations that are listed in Figure 1.  One need only consult the API documentation to discover their usage.

Figure 1
Reference Implementations
Like many of the later JSR initiatives, the reference implementation (RI) for JSR 303 is done by non-Oracle, open source development.  In particular, JSR 303 is led by RedHat and the RI is based on Hibernate’s Validator 4.x.  A second implementation that also passed the Technology Compatibility Kit (TCK) is provided by the Apache Software Foundation.  In short this means that along with the Java Bean Validation API JAR from Oracle, one must also use one of these other implementations.  For this example I chose the RI from Hibernate.  Figure 2 shows the libraries, highlighted in yellow, required to use Javax Bean Validation.  Additional logging libraries are required by the Hibernate implementation.

Figure 2

 
With the Hibernate Validator implementation, there are several additional constraint annotations provided, see Figure 3.  You may notice the @Email and @URL annotations that provide constraints for well-formed email addresses and URLs, respectively.  Of course these are not part of the RI and are considered extensions, albeit very handy extensions.  Listing 2 is an example of what the email field would look like annotated by the @Email annotation.

Figure 3

Listing 2
@NotNull(message = "{Customer.email.NotNull}")
    @Column(name = "EMAIL")
    @Email(message = "{Customer.email.Email}")
    private String email;

Spring Web Flow Validation
The makers of Spring and Spring Web Flow have recognized the need for uniform Bean (model) Validation and have provided the ability to integrate JSR 303 Bean Validation into their containers.  This integration combines the best of industry standard Spring Web Flow with the functionality of JSR 303.  According to Spring Source, “Model validation is driven by constraints specified against a model object.”   For this validation Spring Web Flow embraces two methodologies for validation:  JSR 303 and Validation by Convention. 

One of the issues with these two validation techniques is that they are not mutually exclusive.  If JSR 303 validation is enable along with Spring’s Validation by Convention, duplicate error messages could be the result.  The approach that I recommend is to use validation by convention and setup the validation methods to call JSR 303 validation on model objects as needed.

Validation by Convention using JSR 303
Validation by Convention makes it simple to map validation to Spring Web Flows.  Listing 3 is a snippet from a Spring Web Flow definition, defining the enterCustomerDetails view-state that is bound to the Customer model object.

Listing 3
<view-state id="enterCustomerDetails" model="customer">
        <binder>
            <binding property="firstName" />
            <binding property="lastName" />
            <binding property="email" />
            <binding property="phone" />
            <binding property="lastActivityDate" />
        </binder>
        <transition on="proceed" to="reviewCustomerData" />
        <transition on="cancel" to="cancel" bind="false" />
    </view-state>


Using Validation by Convention we follow the Spring Web Flow pattern of ${Model}Validator and create the CustomerValidator class to handle validation calls from Spring Web Flow.  Inside this class, we must write methods that match the pattern validate${view-state} to link the validation routines to the corresponding Web flow view-state.  Listing 4 is a an example of the CustomerValidator with the validateEnterCustomerDetails() validation method.

Listing 4
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;

import org.springframework.binding.message.MessageBuilder;
import org.springframework.binding.message.MessageContext;
import org.springframework.binding.validation.ValidationContext;

import org.springframework.stereotype.Component;

@Component
public class CustomerValidator {

    public void validateEnterCustomerDetails(Customer customer,
            ValidationContext context) {
        Map<String, List<String>> propertyMap = new LinkedHashMap<String, List<String>>();
        boolean valid = ModelValidator.validateModelProperties(customer,
                propertyMap);

        if (!valid) {
            MessageContext messages = context.getMessageContext();
            for (Entry<String, List<String>> entry : propertyMap.entrySet()) {
                String key = entry.getKey();
                List<String> values = entry.getValue();
                if (null != key && !key.isEmpty() && null != values
                        && null != values.get(0) && !values.get(0).isEmpty()) {
                    messages.addMessage(new MessageBuilder().error()
                            .source(key).defaultText(values.get(0)).build());
                }
            }
        }
    }
}


In Listing 4, the validateEnterCustomerDetails() method is called by Spring Web Flow when a view-state transition occurs.  This method in turns calls the custom class/method ModelValidator.validateModelProperties() method and passes the model object and a map of bean properties to be validated in the model object.  This technique allows us to use the provided conventions of Spring Web Flow with the annotated constraints of JSR 303 Javax Bean Validation.

Listing 5, is the source for the ModelValidator class that does the heavy lifting and works with the bean validation for each model bean.  The idea here is that each Validation by Convention method, matching a Web Flow view-state, would make a call to the ModelValidator, passing in the desired properties to validate.

Listing 5
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;

import javax.validation.ConstraintViolation;
import javax.validation.Validation;
import javax.validation.Validator;
import javax.validation.ValidatorFactory;

public class ModelValidator {
    private static ValidatorFactory factory = Validation.buildDefaultValidatorFactory();
    private static Validator validator = factory.getValidator();

    public static boolean validateModelProperties(AbstractModel model, Map<String, List<String>> messages) {
    boolean isValid = true;

    if (null == messages) {
        messages = new LinkedHashMap<String, List<String>>();
    }

    Set<ConstraintViolation<AbstractModel>> constraintViolations = null;

    for (String key : messages.keySet()) {
        constraintViolations = validator.validateProperty(model, key);

        if (constraintViolations.size() > 0) {
        isValid = false;
        List<String> values = new ArrayList<String>();
        for (ConstraintViolation<AbstractModel> violation : constraintViolations) {
            values.add(violation.getMessage());
        }
        messages.put(key, values);
        }
    }

    return isValid;
    }
}


JSR 303 Custom Constraints
Along with the built-in constraint annotations provided by JSR 303 and Hibernate Validator, JSR 303 provides the facility to write your own custom constraints.  Custom constraints are needed when you want to apply additional logic to your model validation.  This logic would not be possible in the supplied constraints.  For example, if you had a Reservation model object and you needed to validate the check-in and check-out dates, applying the logic that the check-out date should always be after the check-in date, you would need to write a custom constraint.  With the supplied annotations, you can add constraints to force the dates to not be null and to be in the future (compared to today's date), but there is no supplied annotation that would perform the logic necessary for date comparison.

The thing to keep in mind is that creating custom constraints is a 2 step process.  These steps can be done in any order, but we will start with creating the constraint annotation first.  Listing 6 is an example of a custom constraint annotation.  This annotation is applied to the model class, similarly to the @Entity JPA annotation.  What should be noticed here is the validatedBy argument; its purpose is to link the constraint annotation to the implementation class, in this case ReservationDateRangeImpl.  In other words, ReservationDateRangeImpl will perform the actual validation.

Listing 6
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

import javax.validation.Constraint;
import javax.validation.Payload;

@Constraint(validatedBy = ReservationDateRangeImpl.class)
@Target({ ElementType.TYPE, ElementType.METHOD, ElementType.FIELD, ElementType.CONSTRUCTOR, ElementType.PARAMETER })
@Retention(RetentionPolicy.RUNTIME)
public @interface ReservationDateRange {

    String message() default "The check-out date must be after the check-in date.";

    Class[] groups() default {};

    Class[] payload() default {};

}


Listing 7 is the implementation class ReservationDateRangeImpl.  So when the model is validated, @ReservationDateRange will also be applied as a constraint.

Listing 7
import javax.validation.ConstraintValidator;
import javax.validation.ConstraintValidatorContext;

public class ReservationDateRangeImpl implements
        ConstraintValidator<ReservationDateRange, Reservation> {

    public void initialize(reservationDateRange reservationDateRange) {
    }

    public boolean isValid(Reservation reservation,
            ConstraintValidatorContext context) {
        if ((reservation.getCheckInDate() != null)
                && (reservation.getCheckOutDate() != null)
                && reservation.getCheckOutDate().before(
                        reservation.getCheckInDate())) {
            return false;
        }
        return true;
    }
}


A Word About Groups
Looking back at Listing 6, you might notice the line that refers to a Class[] array called groups.  Annotations can be organized into groups; moreover, it is possible for each annotation to belong to several groups, including the DEFAULT group.  With groups, you can specify what combination of constraints (supplied or custom) that you wish to apply during manual validation.  The detractor here is that each group is represented by a Java Class.  To date, I have used interfaces as my group classes.

Issues with Custom Constraints - Separation of Concerns or Anemic Domain
Before you go off and write your own custom constraint annotations, take a moment to reflect upon the purpose and intended behavior of your domain or model objects.  Albeit easy to do, writing custom constraint annotations to enforce business logic may not be the right call.  Adding business logic to domain or model objects is generally considered to be a bad idea since it violates the paradigm of "Separation of Concerns" and mixes the business logic layers of an application with the domain or model layer of the application.  I guess this really depends on what side you agree with in the "Anemic Domain" anti-pattern debate.

I have my beliefs, but I am not advocating in either direction.  One could argue that the Reservation object is incomplete if we cannot constrain the behavior of the check-in/out date fields.  It could also be argued that we have already violated the tenets of proper design by combining persistence in our domain objects, by adding JPA annotations to them.  I look at it from a pure OOP perspective which combines data and behavior in each object.  From there, I apply the "what makes the best sense" paradigm when designing my applications and using these techniques.

Sunday, July 10, 2011

Picking up old and new instruments: Java M(MX)Beans and Agents

Even though JMX and MBeans have been around for over a decade, there are a lot of Java developers that still have never written or used them. They may have used them with JEE application servers, even VisualVM and JConsole, but for the most part they have been relegated to major Java applications.  I submit that they are just as useful when you encounter smaller Java applications that need remote and graceful control.

Remotely and Gracefully
Remote control means accessing MBeans outside of the JVM that they reside in.  That's simple enough.  Graceful control is more esoteric.  What's graceful to one developer may not be so graceful to another or to the end user.  Suffice it to say that to me graceful means being able to control a Java program, including shutting it down and changing its processing, without introducing instability or uncontrolled results and possible data loss.  Is this type of control only important to larger Java applications?

The Example
The scenario:  You have a batch process that is automated via a Java program.  This program could be executing as part of a Windows Scheduled Task or a Unix/Linux CRON job.  The Java program processes file resources on a scheduled basis using multiple threads.  You need to introduce a method to interrupt processing in a deterministic manner so that there is no ambiguity around which files have been processed.  Part of your program already monitors threads as observables.  So, this is the optimum component to signal to the threads that it is time to stop, after a file is processed successfully and before the next file process is started.

There are several ways that this can be accomplished, not the least of which is a signal file that your threaded app reads between each file that is processes.  However, for deterministic control MBeans are a better choice.  And they are very easy to write.  Listing 1 shows how we would implement a simple MBean to control processing.  The ProgramControlMBean interface specifies our shutdown() method, and the ProgramControl implements this interface.  The MBean also has to register with the MBean server to be accessed and used as an MBean.  Listing 1 contains a private register() method to accomplish registration.  If you have one bean to register in your program, I would let the bean register itself.  For multiple MBeans, I might use a different component within the program to register all the beans.

Listing1
public interface ProgramControlMBean {
    public String shutdown();
}



public class ProgramControl implements ProgramControlMBean {
     public String shutdown() {
          //implement shutdown method
     }
     ...
     
     private void register() {
            try {
                 // Register as MBean with MBean Server
                 MBeanServer server = ManagementFactory.getPlatformMBeanServer();

                 ObjectName name = new ObjectName(
                    "PACKAGE_NAME_HERE:type=ProgramControl");

                 server.registerMBean(this, name);
            } catch (Throwable t) {
                 log.error("ProgramControl could not register with MBean server.");
            }

       }
}


With this combination we now have an MBean that will be available in the same JVM that our program runs, hosted by that JVM's JMX server.  If we have done things correctly we are able to access this MBean using the JMX API from within our program.  We are almost done.  We now need to be able to remotely call the MBean to shutdown our program.  This means accessing the JMX server that hosts this MBean from outside the JVM in which it runs.  The folks that created the JMX API also came up with an easy way to access the JMX server.

Listing 2 shows the JVM arguments that are used to start a Java Agent to allow remote access to MBean running in the JVM's JMX server.  Remote programs can access the JMX server via port 3334, authentication and SSL are not required.

Listing 2
-Dcom.sun.management.jmxremote.port=3334
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false


With these JVM arguments, we can easily start a Java Agent when we start the Java program.  Simply put, Java Agents are programs that are usually independent of other programs (including other agents), but loaded in the same JVM.  They have specific goals that are outside of the main program, like data collection or troubleshooting.

Custom Java Agent for Remote JMX/MBean Access
The Java Agent is necessary since the JMX/MBean server is not accessible by resources outside of the local JVM.  With the JVM arguments, it is very easy to configure an agent to allow us to connect to the target JMX server.  We could also write our own Java Agent as seen in Listing 3.

Listing 3
import java.io.IOException;
import java.lang.management.ManagementFactory;
import java.net.InetAddress;
import java.rmi.registry.LocateRegistry;
import java.util.HashMap;

import javax.management.MBeanServer;
import javax.management.remote.JMXConnectorServer;
import javax.management.remote.JMXConnectorServerFactory;
import javax.management.remote.JMXServiceURL;

public class RemoteJmxAgent {

    private RemoteJmxAgent() {
    }

    public static void premain(String agentArgs) throws IOException {
        System.setProperty("java.rmi.server.randomIDs", "true");

        // Start an RMI registry on port 3334
        final int port = Integer.parseInt(System.getProperty(
                "us.va.state.vwc.print.agent.port", "3334"));
        LocateRegistry.createRegistry(port);

        // Get handle to JMX MBean server
        MBeanServer server = ManagementFactory.getPlatformMBeanServer();

        // Environment map.
        HashMap env = new HashMap();

        //Setup RMI connector server
        final String hostname = InetAddress.getLocalHost().getHostName();
        JMXServiceURL url = new JMXServiceURL("service:jmx:rmi://" + hostname
                + ":" + port + "/jndi/rmi://" + hostname + ":" + port
                + "/jmxrmi");
        

        System.out.println("JMX URL::" + url);

        JMXConnectorServer connectorServer = JMXConnectorServerFactory
                .newJMXConnectorServer(url, env, beanServer);

        // Start server
        System.out.println("RMI connector server started on port " + port);

        connectorServer.start();
    }
}


The agent in Listing 3 starts an RMI connector server that will allow remote RMI clients (VisualVM, JConsole, etc.) to connect the JMX MBean server that runs in the JVM that the agent attaches to.  What you should immediately notice is the main() method has been replaced with a premain() method.  That's it, at least at the class definition level.  Getting agents to load when the JVM starts is a little more involved.  First there is the JVM arguments seen in Listing 4; agents must be packaged in their own JAR.

Listing 4
-javaagent agent.jar

And then there is the manifest file entry (Listing 5) that identifies the agent class in the JAR:

Listing 5
Premain-Class: RemoteJmxAgent

If one considers the complexity of writing the RemoteJmxAgent agent class and then adding authentication and transport level security, it just makes more sense that we would use the provided JVM agent configuration as seen in Listing 2.

MXBeans
As of Java 1.6, it is recommended that developers write MXBeans instead of MBeans; moreover, MXBeans are only available in 1.6 and beyond.  MXBeans are very similar to MBeans.  Instead of implementing an interface name ${CLASS_NAME}MBean the MXBean interface is ${CLASS_NAME}MXBean.  They are also registered with the MBean server the same way MBeans are.

The main difference between MBeans and MXBeans is that MXBeans, unlike MBeans, use Java Open Types.  The MXBean implementation maps custom objects returned from MXBeans to a CommonDataSupport type.  This means that the clients using the remote MXBean will not have to have concrete implementations of the classes defined as return types from the MXBean.  There are restrictions on the types returned from MXBeans, but this is still easier than having to remotely implement custom classes like we had to in old days of RMI.  Before using MXBeans, I would be familiar with the specification.

Conclusion
MBeans, MXBeans, and Agents are available to Java developers to help them instrument their Java applications.  I have shown a few examples of these powerful technologies; we must learn to walk before we can run.

Software Estimation - Fibonacci Meets PERT

In my previous post I described an organic process by which I solicited feedback from team members.  The feedback was a Fibonacci sequence number that was used to roughly estimate the complexity and/or work effort on a JIRA task.  This approach is only effective when you have a good team that you can rely on and trust to do the right thing.  This number is just an abstract level indicator of the effort that our development team estimates/recommends for a particular task.

Alone, the Fibonacci number is useless to the business users or project managers.  This is our number.  It carries with it our insights, based on our knowledge, and our experience.  To be useful to others, we must convert it to some metric that they can use.  Enter PERT.

If you are looking for an exhaustive instruction on estimating with PERT, look elsewhere.  I use the typical PERT formulas with a few tweaks here and there.  In the past I have used several multipliers:  Developer Skill Level, Conflicts, New Technology, Code Re-use, etc.  I now have found that the team discussion with Fibonacci output eliminates the need for most of these discrete multipliers.  If you don't trust your team's number, you can always change it.  However, be prepared to communicate to the team as to why you usurped their authority.

For conflicts, I use the industry standard of 0.7. That is to say that we only get 70% of our day to develop.  The rest is consumed with production support, administrative duties and other tasks that conflict with our development efforts. Most of the time, developers discount this or ignore it all together.  I use 1.3 as a multiplier to calculate the expanded time due to a 0.3 reduction in productivity.

An example spreadsheet is seen below.  Obviously as we approach the higher Fibonacci numbers the estimates approach a point at which they are no longer realistic.  This is not a one-size-fits-all approach.  In fact, this method requires tuning over time to better fit your teams velocity and accuracy.

There are multiple levers that can be used to adjust the PERT output of column "E".  This is part of the tuning process.

The main idea is that you remove the complexity of the feature estimates from your development team, when you are still forced to estimate and execute your project in a  waterfall fashion with reasonable predictability and reproducibility.

Wednesday, May 25, 2011

Software Estimation - An Organic Approach

In previous posts I have described the extreme ends of the software estimating spectrum.  Neither end is desirable.  First, let me say that I don't like estimating, at least not the estimating that puts stress on developers to meet unrealistic time-lines.  I like the idea of continually delivering value to customers [Product Owners] based on business priority.  Trying to predict software delivery is risky business.  To mitigate that risk we as professionals adopt Agile beliefs and methodologies.  That is just not always possible.

If I have to estimate for waterfall SDLC, then I believe that good estimating is the result of weighing several factors, not the least of which is experience.  This is why junior developers generally do not make good estimators; they simply lack the requisite experience upon which to base their decisions.  However, if you must estimate, juniors should be included in estimation processes so that they can learn the skill.

If we reflect on the purpose of estimation, we realize that we are simply forecasting a future outcome, based on information available to us now.  The information at our disposal is a an amalgam of data, experience, technology knowledge, code familiarity, application familiarity, business process knowledge, etc.  The list goes on.  The secret sauce to estimating is how we apply these data, information, and knowledge to execute tasks to a prescribed standard.  Knowing what is relevant and what is not also helps in constructing a clear and accurate estimate that will stand up to scrutiny.  Finally, good estimators can articulate why they made certain decisions that led to the estimate.

Truly accurate software estimates start with truly accurate analysis.  That is to say that before a developer begins an estimate, he or she should understand (as much as possible) the functional and non-functional requirements of the work to be accomplished.  The degree to which the analysis is done [correctly] will directly contribute to estimate accuracy.  During the estimation process, developers match requirements to code artifacts and apply proper weighting based on complexity, etc.  Without a good understanding of the requirements, this step is not possible; moreover, estimators may apply a ambiguity factor to the overall estimate when requirements themselves are ambiguous.   This comes dangerously close to sand-bagging or artistry if not properly monitored by senior developers/estimators.

One of the important factors of accurately assessing and estimating work is understanding that one should reach out to his or her team for their input and support.  No one person on a software team knows everything about the task, or the software components (real or imagined) that are involved.  I like to start with the abstract approach of estimating stories and assigning story points that typical Scrum projects use.  I like to get my developers together and have them assign story points to each of our JIRA tickets that have been prioritized by the business.  Even if we are not full-blown Scrum and we do not have a defined "Product Owner" role, we still start with a list of business priorities. 

Once we know the list of prioritized JIRAs each developer picks a number from the Fibonacci sequence, starting at 1 and ending at 144 and assigns it to the JIRA, based on his or her knowledge of the task, code, application, experience, etc.  For this purpose there is no need to go any higher than 144, and using the Fibonacci sequence helps to eliminate the ambiguity of more linear scales, like 1-10 or 1-100.  This approach of assigning more abstract values to the initial estimate and using the input of multiple developers facilitates discussions among team members, about the estimates for each JIRA.  More often than not, the true nature of the work needed is discovered through the discussions and the correct Fibonacci number is finally assigned to the JIRA.  Of course we would take this even further if we were a Scrum shop, but that is not the case right now.

The key to getting the correct Fibonacci number is to ask the team to qualify there answers with what level of developer would be needed. This should fall out of the discussion when certain members of the team have different estimates.  I try to convince my team that we are estimating for intermediate to advanced developers.  This eliminates the need later to take into consideration the level of the developer assigned to the task.

In my next post I will show how the JIRA/Fibonacci number is used further in a PERT analysis for more Waterfall-like estimates.

Tuesday, May 24, 2011

Are you an upbeat artist or a sand bag engineer?

In my experience, many software developers do not know how to properly estimate a development task, regardless of how familiar they are with code streams, constituent technologies, or business processes.  We do not learn this in college, at least not in undergrad.  I place most developers into two categories, artists or sandbag engineers. 

The artists paint "rosey" pictures framed by hyper-positive and unrealistic delivery dates.  These developers mean well, and I would go as far as to say that they are actively trying to counteract the notion that all IT is slow.  However, the main problem is that these guys do not consider the nonproductive time that they have on a daily basis.  They do not consider activities that distract them from the estimated development task, including task switching to other development tasks.  In general, their estimates are nonscientific and based on their recall of previous experiences and anecdotal evidence.  In fact, these developers mostly do not realize that they "cherry-pick" the good experiences from the last development efforts and do not appropriately weigh the bad.

At the end of the day, artists have agreed upon, unrealistic deadlines that require herculean efforts to meet.  Sometimes they become temporary heroes to their managers or customers.  However, as I have said before, hero worship is not a long term, sustainable strategy.  And stories are legion of developers taking shortcuts when they ran out of delivery time.  Sooner or later the facade decays and these developers are exposed for the poor estimators they really are.  They are not bad people, just bad at estimating.

The sandbag engineers (a.k.a. sand-baggers) are not any better.  These developers pad their estimates from the beginning, regardless of the task size or complexity.  Some of them suffer from "Kirk/Scotty" syndrome.  They think that if they say it will take longer than it actually should, they will look good when they deliver way early.  This of course is a fallacy when examined by the astute manager.  Coming in way early is almost as bad is coming in way late.  The estimates are still wrong. 

Sandbagging is insidious.  It slowly undermines the trust and faith that managers and customers have in their developers.  There is nothing wrong with delivering early when the scope of a project is elaborated to the point of reduced effort.  However, more often than not, sandbagging is just a front-loaded risk mitigation strategy that is not grounded in scientific methodology or data.

In following posts, I will describe how I have performed estimation in the past, based on experience, scientific methods, and data.  It's certainly not a perfect process, but it places me on the spectrum between the artists and sand-baggers and my estimates are more accurate.

Center for Teaching Excellence - Institute on Teaching & Learning

Last week I attended a 5 day workshop at VCU's CTE (Center for Teaching Excellence).  To say that it was an enlightening experience would be understating its immediate impact on me.  The ideas and techniques discussed in our sessions and espoused by the guest speakers demonstrated to me that I have a lot to learn about teaching and I need to change the way that I am reaching and engaging students, or not reaching them, as the case may be.

Of particular interest to me are the techniques for formative assessment.  I now realize that my techniques for evaluation are mostly summative.  When teaching CIS topics I guess it is easier to employ summative techniques.  What I immediately realized that I was missing was the proper feedback from my students.  The summative evaluations provide no feedback beyond the measure of how well students memorize.  I need to know if I am reaching them and keeping them engaged beyond the typical 8-10 minute attention span.

Should I teach in the Fall I plan to utilize more active learning techniques with particular collaborative learning activities.  I plan to use "Minute Papers" and JiTT to provide the feedback needed to direct my class and my students' learning towards higher levels.  My goal is to introduce meta-cognitive sessions by the Fall 2011 or Spring 2012.

Friday, May 20, 2011

Richmond Java Users Group (RJUG) Meeting - 05/18/11

Wednesday night I attended an RJUG meeting.  I am on the steering community for this group, but I don't usually have the time to attend regular meetings.  However, this meeting covered topics that I really agreed with.  As an Enterprise/Application Architect, I have over time shifted my thinking to simpler design:  simpler to accomplish, simpler to maintain, and simpler to understand.

The meeting was sponsored by Ironworks.  The topic was Blending Best of Breed Technologies and focused on integrating several Open Source tools and technologies:  Core Spring and Spring MVC, Groovy, FreeMarker, Liferay, and Alfresco CMS.  The presenter, Dennis Sharpe, did a great job presenting an overview of the individual technologies from a high level.  However, the presentation really shined when he demonstrated and communicated the integration techniques for these tools and how they were using them in production.  Of particular interest was how they used Groovy to shift certain highly dynamic code to the Freemarker templates stored in the Alfresco CMS.  I will not attempt to recreate the meeting in this entry.  If Ironworks makes the slides available later, I will post a link.

Thursday, March 17, 2011

PMI CVC Certification Workshop - Spring 2011 (Take Two)

I will be presenting on May 7 for the PMI Central Va Chapter during their second Spring 2011 PMI Certification Workshop.  My topic will be the Project Management Framework.  My talk goes from 10:15 to 11:15.

Wednesday, March 9, 2011

Taxing and Permitting - Architectural Levers

As architects, what levers can we pull to align potentially risky technology decisions with timely technology delivery?

Architects are on the hook for planning the portfolio of technologies that our businesses and customers will use to perform their activities.  To be successful and to remain competitive, our customers must be able to rely on technology that is aligned with business needs and leverage said technologies in a timely, flexible, and cost effective fashion.  And they (our customers) cannot afford to spend time understanding the technologies or how they are implemented.

Often competing with timeliness and flexibility is the need to develop sustainable and extensible architectures.  In fact, if done correctly, these architectures will result in the ability for our technologists to be more timely and flexible.  A mature architecture simplifies (at least helps) the process of IT service delivery and technology adoption and implementation.  It is the immature architecture or architecture in development or transformation (larval stages) that can actually hinder progress and delivery.

There are times when the business just cannot wait to have IT do it right; instead, they just need us to do it right-now.  It is at these times that architects and architecture are discounted the most.  It is also here that we routinely introduce technical debt.  Sooner or later, we (IT, IS, pick one) do something right-now that leads to an unsustainable platform that can no longer be incrementally extended.  At that point we have tech-sand.  At a minimum, we now have to replace, rewire, or rewrite major components or subsystems to continue to meet our customers' needs.  Our customers may also have to exercise their own process workarounds until we can remedy the situation.

I submit that as long as IT does not control the business' decisions this cycle is inevitable.  We all have legacy systems and code-bases that are not as flexible as we would like them to be.  How did they get that way?  Was it a conscious decision that led to this situation?  Was it a series of uninformed decisions that led to disconnected results?  Did we plan for how we would get out of this technical debt?  Regardless of architecture maturity (or immaturity as the case may be) organizations should plan for how they will overcome bad architecture decisions and potentially self-destructive behavior.  The approach that I like is Taxing and Permitting.

If businesses are going to make decisions that disregard sound and established architectural guidelines, then we should also plan for how we will fix the unsoundness that we introduce.  Yes, there should be a architecture tax.  I heard about this concept in a webinar given by Data Blueprint's Peter Aiken.  This was a great session on the importance of Data and Information Architecture.  Data Blueprint (and Peter) have several webinars scheduled for the coming weeks.

This architecture tax would be part of the building permit process that I spoke of in September 2010.  New projects that require IT services should go through architecture reviews.  During these reviews, potential solutions would present architecture building permits to the architecture committee.  To facilitate this process, presenters would follow guidelines for completing building permits.  Depending on the size and complexity of the needed IT solution, the building permit process would have different levels of required business approval and architecture scrutiny.  The building permit process would also prescribe technologies and methodologies that should be used for solution types.  In this way, architecture would simplify the process by reducing the potentially bad choices that technologists could make, while providing a directed approach to architecture approval.

At the end of the day, IT architects are charged with many tasks, but these two seem most important to the scope of this problem:
1.  Make it simpler for IT to deliver faster
2.  Keep the supply of junk in our solutions to a minimum

If the business wants to override architecture guidelines, they should be made to relinquish a portion of the cash flows that they plan to reap as part of the project that led to this decision.  Considering that the majority of most IT budgets are used for maintenance and support, this architecture tax could be used to offset the technical debt incurred by a bad technical decision.  The building permit process would prescribe the tax needed to gain architectural approval for a known substandard solution.  This tax could be realized in a weighted risk or technical debt increase to the cost of capital or hurdle rate that the project must ultimately satisfy to be approved.  Just as some organizations ratchet up their costs for capital based on risk and project type, these hurdles could also be increased based on estimated cost of technical debt remediation.  At this point, architectural approval becomes a gate that the project must make it through to survive.  During the phase gate process, the technical debt and tax get documented as part of the architecture review process.

Of course the entire organization must agree to this type of cash flow weighting, and financial departments would have to understand how to account for and accrue these "taxes" so that they can be used at the discretion of IT to support such poorly designed solutions.  At the end of the day, IT must prepare their slice of the business budget to be able to accommodate potential rework introduced by business decisions that opposed architectural effectiveness and contributed to technical debt and even tech-sand.

Tuesday, March 8, 2011

PMI CVC Certification Workshop - Spring 2011

I will be presenting this weekend for the PMI Central Va Chapter during their semi-annual PMI Certification Workshop.  My topic will be the Project Management Framework.  My talk goes from 10:15 to 11:15.

Sunday, January 23, 2011

What is wayfinding?

According to Wikipedia, wayfinding is "all of the ways in which people and animals orient themselves in physical space and navigate from place to place."  Dictionary.com defines it as, "signs, maps, and other graphic or audible methods used to convey location and directions to travelers."  An example of wayfinding could be "you-are-here" style maps in malls or amusement parks, augmented by color-coded trails to help you navigate.  If you have ever been to Disney World then you have seen great wayfinding in action.

One of my favorite examples was in the admin building of White Oak Semiconductor.  In this building, hallways were painted specific colors to indicate their purpose.  One color indicated that the hallway was used to transition from the outer corridors into the cubicle areas, while other colors indicated that the hallways led to stairwells to transition between floors.  White Oak also adopted a convention to locate restrooms in the hallways that transitioned from common areas or corridors to cubicle work areas.  These simple yet functional wayfinding techniques were easily adopted by the building inhabitants and eased navigation throughout the facility.

What's missing in both the definitions above is virtual wayfinding in the context of web site or application navigation.  To me, wayfinding is a key component (if not the component) driving usability engineering.  Unfortunately, wayfinding is missing in many web designs.  It really is as simple as adopting conventions that are intuitive for users to understand and retain.  Some wayfinding techniques are common to many successful applications (such as bread crumbs) while other techniques are specific to the application and users that the application serves.  However, without wayfinding, users not forced use your application will not return to your application or web site.  This leads to development waste and lost revenue.

To me, there are many ways to build wayfinding into your web site, but understanding your user base is the first stop.  As a designer or architect, you need to understand the whys of your application.  Using well-defined and universally accepted internet navigation standards should be your next stop.  Personalization is also a good idea.  The reason while web portals are so successful is that they allow users to customize their return experiences while utilizing standard techniques for navigation and information presentation.  Regardless of whatever techniques you decide on you should document those standards so that future changes to your web site also follow the same standards, until such a time as they become outdated and counter intuitive.

Wayfinding is meant to help answer questions for your customers/users while they use your web site to get what they need and give you what you want.  Some of these questions are:
1. Where am I, am I there yet?
2. Am I on the right site, in the right place, for what I need or want to do?
3. Where do I go next, or how do I get back to where I was?
4. Is this best choice for me?
5. Did I already see this?

While on your web site, users need to find what they want while also avoiding what they do not want.  If you do not engineer that ability into your web site then you are doomed to mediocre results.

Lead, or get out of the way!

Recently I ran into an ambiguity while planning a lesson for a course as part of an MBA curriculum.  I soon realized that I had more questions about this upcoming assignment than I did answers.  The assignment was to have my students create a 3-sigma control chart to chart the statistical samples/observations of processes, and make decisions based on the charted data.  I have taken descriptive stats, and I retained enough to handle teaching this topic.  However, the observations were stratified into multiple defect types and it looked like I might have to teach multivariate analysis.

Since this was the first time I taught this assignment, I needed more clarification on the direction of this topic.  The course had strict course guidelines and grading rubrics, but this particular assignment was unclear to me.  Since there was a course lead identified for the course, I contacted him.  I spent 30 minutes carefully crafting my question to him and my request for help.  I tried to imply as best as possible that I was not punting on this assignment, but I genuinely needed help.  I contacted him only after I spent considerable time attempting a conclusion on my own.

My email request to him was a couple of paragraphs.  His response was a one liner, "...read the syllabus."  I had read the syllabus.  I am not an idiot, and I know how busy professors can be.  I would not have contacted him if I didn't have to.

If he is the course lead, then he should lead, or give up the responsibility to others more capable of providing leadership.  Leading implies coaching and mentoring.  It implies directing, encouraging, and facilitating.  He had the directing part down.  A manager does that as well.  Maybe we should call him a course manager instead.

Anybody who knows me knows that I do not believe in trying to motivate others, even though many leadership texts espouse motivation as a key leadership skill.  Instead of motivation, I believe in encouraging and facilitating others who are already motivated.  I was motivated to develop the lesson plan and be able to teach the topic so that the my students would succeed in stats assignment.  I was not looking for a handout, just a little more guidance.  Since this was an MBA course, multivariate analysis was not completely out of the question.  He chose to direct me to a source that I had already exhausted, one that I had already indicated to him was not enough for me to understand the assignment.

It was another missed opportunity to lead, to encourage, to facilitate learning by a peer.  I immediately wondered if he interacted with his students in the same indifferent manner.  I am not long out of school and I know what it is like to have a professor that just does the minimum to get by.  The students suffer.  Sure, some take the initiative to bridge the gap on their own, but value is lost in the pedagogical exchange.  I felt bad for his students, but in a whimsical manner, I felt that his under achievement would make the rest of of look better.  I decided to teach the course as I interpreted the course guides and ask forgiveness when my choices diverged from the implied direction.

So, I submit, if you hold yourself up as a leader to others, then be one.  Be there when others need you, and understand the additional duties that are implied by the position and inferred by the ones you are charged to lead.  Or...just get out of way and let others do the job as it needs to be done.