Wednesday, October 31, 2012

Java-based BDD with Cucumber

Recently we have been looking at creating tighter collaboration between our Developers, Testers, and Business Analysts (BAs).  They all have the common project-level goal of delivering high quality, but they each fill different roles in the development cycle.  The key to successful collaboration is the evolution of business requirements to stories to features to executable behavioral specifications in the form of unit tests.

Our projects are primarily Java-based (with some Groovy thrown in for fun), so we have been looking primarily at tools for Behavioral Driven Development (BDD) in Java.  To that end, we first looked at Spock.  Spock is very powerful and integrated well with Groovy.  This makes it very expressive and easy for our developers to learn, and quickly leverage.  There in lies the issue.  Spock, at least in our experience, is very developer-centric.  Feature specifications written in Groovy with Spock Domain Specific Language (DSL) are done by the developer, mostly without the BA or tester.

One can argue that pairing developers with testers and BAs would solve the developer centricity of Spock.  I have read the arguments for pair programming with regards to less defects overcoming combined resources.  However, while pair programming would help developers build the specs in Spock, it does not erase the fact that Spock is mostly for developers.

Developer centricity was always my argument against Test Driven Development (TDD).  While I understand the ideas behind TDD, it seemed to me that it was always lacking that collaboration, key input, from the folks capturing and testing the business requirements.  Writing JUnit tests to meet a requirement, then writing code to pass the JUnit tests is good, but still decoupled from the requirements and true behaviors of the system under development.

With BDD, the BAs can build the User Stories.  From these stories, the testers and developers can distill the features.  Testers can also derive the behavioral specifications that should be the foundations for testing the behaviors of features in the context of user stories.  Finally, it would be very nice if the testers, after agreeing on a standard DSL, could help the developers generate unit tests in an automated fashion, thereby reducing variability from the behavioral specification.  Enter Cucumber.

Cucumber was originally written in Ruby.  I am not a Ruby developer, so I never used the tool.  I heard really good things about it and how successful BDD was with it.  Finally, this year, Aslak Hellesoy ported Cucumber to Java (cucumber-jvm).  I was aware of Cuke4Duke, but I was just not eager to use JRuby. So when I heard about Cucumber in Java, I started my research.  What follows is my initial research and impression of the tool and measure of utility for BDD in Java.

Getting Started
To get started with Cucumber, I highly recommend the book, "The Cucumber Book: Behaviour-Driven Development for Testers and Developers".  Though it is written for Ruby development, its coverage of Cucumber concepts and Gherkin are well worth the read.  The Gherkin examples in the book, work in the Java port of cucumber.

For the uninitiated, Gherkin is the line-based language that Cucumber uses to define behaviors in the form of features, scenarios, and steps.  Gherkin files are feature files with the ".feature" file extension.  One of the first conventions to learn is that only one feature can be specified in a feature file.  The next is the Gherkin syntax.

Feature: Password Manager
  Scenario: Change password
    Given User is logged in
    And User is on edit profile page
    When User presses Edit Password button
    And User enters "value" for new password and repeats "value" for new password confirmation
    And User presses "Change password"
    Then User should see "Password changed"
In this example, the Password Manager feature contains the Change Password scenario.  This scenario is comprised of 6 steps arranged in typical Given, When, and Then Gherkin syntax.

Converting Gherkin to Java
After analysts or testers create the Gherkin feature definitions, the files can be used to generate the Java unit tests that get execute via JUnit.  The first step is to add the cucumber-jvm libraries to your Java project.  Below are the Maven dependencies from my project.

<dependency>
   <groupId>info.cukes</groupId>
   <artifactId>cucumber-core</artifactId>
   <version>1.0.14</version>
  </dependency>
  <dependency>
   <groupId>info.cukes</groupId>
   <artifactId>cucumber-junit</artifactId>
   <version>1.0.14</version>
  </dependency>
  <dependency>
   <groupId>info.cukes</groupId>
   <artifactId>cucumber-java</artifactId>
   <version>1.0.14</version>
  </dependency>
  <dependency>
   <groupId>info.cukes</groupId>
   <artifactId>cucumber-groovy</artifactId>
   <version>1.0.14</version>
  </dependency>
  <dependency>
   <groupId>junit</groupId>
   <artifactId>junit</artifactId>
   <version>4.10</version>
  </dependency>

Next, a unit test runner class is created to hook Cucumber into JUnit.  Below is an example class.

package com.icfi.cuke;

import org.junit.runner.RunWith;

import cucumber.junit.Cucumber;

@RunWith(Cucumber.class)
public class Test {
}

The Test.java file executes as a JUnit test case using the Cucumber.class.  When the the class is executed as a JUnit test case, cucumber-jvm reads the feature files in the same package as the test, or as specified by the @Cucumber.Options annotation.  During execution cucumber-jvm examines the behavioral specifications in the feature files and tries to match them Java test files. 

When first executed, typically there is no matching Java code for the test steps.  At this point, cucumber-jvm outputs what it thinks the Java and Ruby method stubs should be, based on the steps defined in the feature files.  Below is the output (Ruby code removed) from the first run with no matching methods.

@Given("^User is logged in$")
public void User_is_logged_in() throws Throwable {
    // Express the Regexp above with the code you wish you had
    throw new PendingException();
}

@Given("^User is on edit profile page$")
public void User_is_on_edit_profile_page() throws Throwable {
    // Express the Regexp above with the code you wish you had
    throw new PendingException();
}

@When("^User presses Edit Password button$")
public void User_presses_Edit_Password_button() throws Throwable {
    // Express the Regexp above with the code you wish you had
    throw new PendingException();
}

@When("^User enters \"([^\"]*)\" for new password and repeats \"([^\"]*)\" for new password confirmation$")
public void User_enters_for_new_password_and_repeats_for_new_password_confirmation(String arg1, String arg2) throws Throwable {
    // Express the Regexp above with the code you wish you had
    throw new PendingException();
}

@When("^User presses \"([^\"]*)\"$")
public void User_presses(String arg1) throws Throwable {
    // Express the Regexp above with the code you wish you had
    throw new PendingException();
}

@Then("^User should see \"([^\"]*)\"$")
public void User_should_see(String arg1) throws Throwable {
    // Express the Regexp above with the code you wish you had
    throw new PendingException();
}
At this point the reader gets the idea that Cucumber matches step methods to steps within the the feature files via Regular Expressions (RegEx).  In fact, the JUnit output indicates that the step definitions defined in the Gherkin feature file(s) have yet to be implemented. Below is the JUnit output indicating that the step definitions were not executed.  This also points out that the JUnit execution with the Cucumber Class uses the Features, Scenarios, and Steps defined in the features files, and looks for those implementations in the Java unit test code.



Writing the Java Unit Tests
To get started with the Java implementation of the Cucumber unit tests, we only need to paste the recommended code from above.  Once pasted, we execute the Test.java to pick up the Cucumber unit tests.  Only one executes and throws the PendingException, seen below. These are the stubs that we now have to implement.

cucumber.runtime.PendingException: TODO: implement me
 at com.icfi.cuke.PasswordManagerTest.User_is_logged_in(PasswordManagerTest.java:13)
 at ?.Given User is logged in(com\icfi\cuke\PasswordManager.feature:3)

At this point it is important to realize that Cucumber does not enforce how we implement the unit tests.  We could write any number of assertions that would pass or fail.  Cucumber only tries to link the steps defined in the features files with RegEx matches to annotated methods in the unit test classes.  Cucumber also enforces that the steps are executed in the order that they are defined.  The RegEx matches can be spread across multiple unit test classes as well.  Cucumber does not care where the tests actually are, just that they actually are there and in the right order.  Cucumber will arrange the multiple java unit test files in the correct order, if need be.

Matching Steps
The key to getting Gherkin steps to match Java test methods is using the correct statements, within your features files, that convey the correct RegEx in your Java annotations.  It is possible to use two different statements that actually result in the same RegEx annotations in Java.  For example, given the feature file below:

Feature: Password Manager
  Scenario: Change password
    Given User is logged in
    And User is on edit profile page
    When User presses Edit Password button
    And User enters "value" for new password and repeats "value" for new password confirmation
    And User presses "Validate password"
    And User presses "Change password"
    Then User should see "Password changed"
The step containing the strings "Validate password" and "Change password" would result in the same RegEx pattern: "^User presses \"([^\"]*)\"$"

However, Cucumber would only recommend this method once, for two reasons.  First, even though the Cucumber feature file has two similar statements, with the same RegEx pattern result, only one corresponding Java method is needed.  Cucumber will execute the Java method twice, to satisfy the defined steps.  The second reason is that if the Java method is actually written twice (same RegEx pattern and different Java method signature for Java compile reasons), an exception is thrown by the cucumber-jvm runtime, see below.

cucumber.runtime.DuplicateStepDefinitionException: Duplicate step definitions in 
com.icfi.cuke.PasswordManagerTest.User_presses(String) in 
file:/C:/UserData/Workspaces/TOMCAT/Cucumber/target/test-classes/ and 
com.icfi.cuke.PasswordManagerTest.User_presses1(String) in 
file:/C:/UserData/Workspaces/TOMCAT/Cucumber/target/test-classes/

So, even if the defined feature steps cause the same RegEx pattern, Cucumber is smart enough to reuse the same Java method.

Implementing Tests - The Rigor-Gap Remains
Obviously, Cucumber cannot reach into the test case and force the Java developer to write proper unit tests.  In fact, since Cucumber is underpinned by JUnit, the same rigor gaps can occur.  Cucumber will check that defined steps are covered with tests by matching steps to Java method RegEx annotations, and JUnit will check for assertions and other exceptions thrown during execution.  However, this does not force the Java developer to actually write a valid test.  That rigor is still up to the Java developer and he/she that reviews his/her code.  Unit test coverage of code can also be enforced with Sonar and other static code analysis tools.

Completing the Cycle
With Cucumber in your teams' toolboxes, they will be better at completing the collaboration cycle to succeed with BDD.  Cucumber allows your BA's and Testers's to write the Gherkin feature files that can be used to help the Java developer stub-out the Java behavioral specifications in the form of JUnit tests.  While the rigor-gap still exists for actually implementing the unit tests, this is partially overcome by using static code analysis.