Complete CI/CD guide for Java OpenSource projects covering artifact releases

After having successfully been able to create my first Java open source project, a client for PagerDuty events, and integrate it completely with a continuos integration service (Travis CI) I wanted to share my experiences and the challenges I went through during this journey. The aim of this post is not only to ease the ramp-up effort needed to get someone started in the OpenSource world and build an end to end continuous integrated pipeline but also and more importantly be aware of the different challenges that arise when working towards that goal.
The requirements for this project were as follows:
  • Use a distributed version control and source code management (SCM) - GitHub.
  • Support automated continuous integration when merging code into master branch:
    • Compile, build, install and execution of the tests.
    • Release project artifacts outside the immediate development team by doing the following:
      • Prepare for release: Update POM removing SNAPSHOT from the version tag, create a commit and push changes (adding a git release tag) to SCM for tracking purposes.
      • Upload artifact to OSS Sonatype staging repository and promote artifact to OSS Sonatype release repository.
      • Prepare for development iteration: Update POM pumping up the version and adding '-SNAPSHOT' to it, create a commit and push changes to SCM for tracking purposes.
  • Support automated continuous integration for Pull Requests covering:
    • Compile, build, install and execution of the tests.
    • Update Pull Request with build information (success/failure)
Here's a diagram showing the workflow after everything is completed and integrated with the different services:

In order to achieve the above, we will split up the work in the following sections:

Simple Java Application

A simple hello world Java application has been developed using Maven as the build automation tool.  Maven will not only manage all the dependencies of the application but also it will allow us, with the help of few plugins, to publish our component to Maven Central. The source code of the application can be found at:
For the sake of time I will not cover how to set up a Java Maven application from scratch and it will be assumed that the reader has some knowledge about how Maven works. 
The hello-world-cicd application will be used as a reference for the various examples from now onwards.

Deployment to Central Repository

As part of the goals defined at the beginning of the post, we wanted to be able to publish our new open source library to Maven Central (Central Repository) thus it could be leveraged by other people. In order to be able to publish a component to the Central Repository you have to go through an approved repository hosting such as Apache Software Foundation (for apache projects), FuseSource Forge (focus on FUSE related projects), Nuiton.org or the easiest and preferred way is to use Sonatype. Sonatype explained briefly is an Open Source Software Repository Hosting (OSSRH) primarily used by open source project owners to publish their artifacts to the Central Repository. We will use Sonatype to upload our artifacts to Maven Central.


First and foremost, we need an account to be able to publish artifacts to Sonatype. Follow the instructions on the initial set up to get an account. If you don't own a domain (e,g: google.com) it is also common to use a GitHub domain:

github.com/yourusername -> com.github.yourusername (this would be the groupId)

The domain is tightly coupled with the groupId value specified in the artifact's POM. More info about groupId and domains here.

Once the account has been created, we can proceed with the appropriate updates of the artifact's POM. 

Sufficient Metadata

Sonatype requires that the artifact's Project Object Model file (POM) meets certain criteria before it can be published. Hence, we need to make sure that the following bits and bobs are covered.
  • Project Coordinates (GAV):
    • Group Id: Top level
    • Artifact Id: Unique name of the component
    • Version: Version of the component (if the string contains '-SNAPSHOT' it mean that it is a development version)
  • Packaging: How the component will be shipped (jar, war, etc)
  • Project Name: Component's name
  • Description: Brief description regarding what the project is about
  • Url: Component's project website (e,g: https://github.com/dikhan/hello-world-cicd)
  • License Information: License used to distribute the component developed
  • Developer Information: Usually contains information about the owner of the repository and the contributors.
  • Source Control Management system (SCM): Specifies where the source code is hosted. There are multiple providers of such service (svn, GitHub, bitbucket, etc) and examples for each of them can be found in the following link.
Here is how the pom should look like:
4.0.0
    com.github.dikhan
    hello-world-cicd
    1.0-SNAPSHOT
    jar

    hello-world-cicd
    Simple Java Hello Word project to showcase how to do CI/CD with OpenSource projects
    https://github.com/dikhan/hello-world-cicd
    2016
    

    
        
            The MIT License
            https://github.com/dikhan/pagerduty-client/master/LICENSE
            repo
        
    

    
        
            Daniel I. Khan Ramiro
            di.khan.r@gmail.com
            GMT
            https://github.com/dikhan
            
                Administrator
            
        
    

    
    
    

    
        scm:git:https://github.com/dikhan/hello-world-cicd.git
        scm:git:https://github.com/dikhan/hello-world-cicd.git
        https://github.com/dikhan/hello-world-cicd.git
    
The above fields must be populated; otherwise the promotion of the artifact from staging to release will not be allowed. 

Supply Javadoc and Sources

In order to comply with the minimum quality requirements from Central Repository, an application packaged other than pom (in the case of hello-world-cicd .jar) should be shipped with both the sources and javadoc as well as the final jar. This can be achieved by using the following plugins which will generate two files (hello-world-cicd-1.0-SNAPSHOT-sources.jar and hello-world-cicd-1.0-SNAPSHOT-javadoc.jar) when running mvn package.

        
            release
            
                
                    
                    
                        org.apache.maven.plugins
                        maven-source-plugin
                        2.4
                        
                            
                                attach-sources
                                
                                    jar-no-fork
                                
                            
                        
                    
                    
                        org.apache.maven.plugins
                        maven-javadoc-plugin
                        2.10.3
                        
                            
                                attach-javadocs
                                
                                    jar
                                
                            
                        
                    
                
            
        
    
You may have notice that the two plugins are included inside a profile which is active by default. The reason for this will be explained later when covering the preparation of the scripts needed to automate the build inside Travis CI. Running the following command will output three .jar files:
mvn clean package
  • hello-world-cicd-1.0-SNAPSHOT.jar - Bundle prepare for execution
  • hello-world-cicd-1.0-SNAPSHOT-javadoc.jar - Bundle containing java documentation
  • hello-world-cicd-1.0-SNAPSHOT-sources.jar - Bundle containing source code

Sign files with GPG/PGP

Files uploaded to the Central Repository need to be signed appropriately. As part of the deployment verification process the OSSRH will check the signature of the various jar files uploaded (compiled code, source, javadoc) by looking at a set of  public KeyServers (e,g: pgp.mit.edu) to make sure that the public key is not revoked. This ensures the authenticity of the library as only the owner of the component should be able to sign and publish the component.
In order to get this done we need to go through a multiple step process as described below.

Generate signing key

Firstly, we need to create a signing certificate which we can use to sign our code. Below are the commands needed:
$ gpg --gen-key
  • Type of key: Select option 4 RSA (sign only)
  • Key size= 4096
  • Key valid for=20y
Now you will be asked to to input some details which is how you can identify your key. Moreover, this info can also be used by the clients of your libraries to confirm that they were signed by you.
  • Real Name: Provide your real name
  • Email Address: Email address to identify the key
  • Comment: Some comment
Enter a passphrase to protect your key and you should be good to go. Bare in mind that this type of key can only be used for signing purposes and not encryption.

Finding and publish the key

After having successfully created the signing key, we can proceed and publish the public key to few public KeyServer. As mentioned before, the public key will be used by SonaType to verify that the artifacts uploaded are signed appropriately and the certificate is not revoked:
Run the following command to find the newly created key:
$ gpg --list-keys
and something similar to the below should be displayed:
pub   4096R/$keyid 2015-05-29 [expires: +20y]
uid                Your Real Name (some cool comment) 
$keyid is the key identifier. 
It's time to submit the key to MIT server: 
$ gpg --send-keys --keyserver pgp.mit.edu $keyid

Signing libraries test

We will use maven-gpg-plugin to automate the signing process, and it will be included in the release profile which will be used eventually by Travis CI when deploying the code to the Central Repository. Include the following plugin to the release profile along with the maven-source-plugin and maven-javadoc-plugin:
.
    
        org.apache.maven.plugins
        maven-gpg-plugin
        1.5
        
            
                sign-artifacts
                verify
                
                    sign
                
            
        
    
In order to be able to run the above plugin we need to let maven know about what certificate should be used to sign the code. We do that by creating a simple maven settings.xml file like the following:
.


    
        
            ossrh
            
                true
            
            
                gpg
                ${env.GPG_KEY_NAME}
                ${env.GPG_PASSPHRASE}
            
        
    

As can be seen, here we provide values about the type of executable, keyname and passphrase. Let's give it a shot and confirm that we can package our project running the release profile that in turn will make us of the sign plugin which would feed from this settings file to figure out the actual values. Maven supports environmental variables in the settings file and the prefix env. tells maven to get the variable name (e,g: GPG_KEY_NAME) from the environment variables.
GPG_KEY_NAME=YOUR_KEY_ID GPG_PASSPHRASE=YOUR_PWD mvn -Prelease install --settings settings.xml
Adding --settings to the above command ensures that we are not be using any other default maven settings file (e,g: ~/.m2) and therefore avoid inconsistencies. After running the above command we should now see one extra file with extension .asc per every .jar created. The .asc files contain the signature of each file:
  • hello-world-cicd-1.0-SNAPSHOT.jar.asc - Signature of the bundle prepared for execution
  • hello-world-cicd-1.0-SNAPSHOT-javadoc.jar.asc - Signature of the bundle containing java documentation
  • hello-world-cicd-1.0-SNAPSHOT-sources.jar.asc - Signature of the bundle  containing source code
  • hello-world-cicd-1.0-SNAPSHOT.pom.asc - Signature of the pom file

Deployment to Maven Central

In order to be able deploy our artifacts to Maven Central via Sonatype and avoid repetitive manual work we will be using maven-release-plugin. The plugin will help us automate the release process handling the following steps for us:
  • Prepare: build, test, release version update, commit, tag, next snapshot version update, commit
  • Perform: export a release from SCM, run the deploy goal
The pom needs to be updated once again to include the following details.

Plugin information

This time it will not be included in the profile created for release purposes but rather to the root build tag.
.
    
        
            
                org.apache.maven.plugins
                maven-compiler-plugin
                3.6.0
                
                    1.8
                    1.8
                
            

            
            
                org.apache.maven.plugins
                maven-release-plugin
                2.5.3
                
                    ossrh::default::https://oss.sonatype.org/content/repositories/snapshots
                    true
                    false
                    release
                    deploy
                    
                        **/pom.xml
                    
                
             
         
    

Source Control Management

In order to be able to communicate with the SCM (GitHub in this example) the plugin has to know the connection details, developer connection and lastly the url of the repo. For this example, I will be using https protocol as a way to connect to the repo since it will simplify the authentication process in the examples provided below.
.
    
        scm:git:https://github.com/dikhan/hello-world-cicd.git
        scm:git:https://github.com/dikhan/hello-world-cicd.git
        https://github.com/dikhan/hello-world-cicd.git
    

Distribution Management

Similar to above, the plugin needs to also know about where to publish when distributing the artifact. Thus, it needs to know what URL should be used for snapshots as well as for final release versions.
.
    
    
        
            ossrh
            https://oss.sonatype.org/content/repositories/snapshots
        
        
            ossrh
            https://oss.sonatype.org/service/local/staging/deploy/maven2
        
    
At this point we have got the settings.xml file completed with all the needed details and here's a link to a version of the file for reference purposes.
Now it's time to test out the plugin to confirm that it works and it does what it's expected. Please note that the execution of these commands will perform direct changes on both the remote repository and Sonatype. However, it is recommended to still test this out as it will be easier to debug or troubleshoot issues that come up instead of doing it in the CI tool.

Deploy snapshot to staging repository

GPG_KEY_NAME=YOUR_KEY_ID GPG_PASSPHRASE=YOUR_PWD OSSRH_JIRA_USERNAME=USERNAME OSSRH_JIRA_PASSWORD=PWD mvn org.apache.maven.plugins:maven-release-plugin:2.2.1:stage release:stage -DconnectionUrl=scm:git:https://github.com/dikhan/hello-world-cicd.git -DstagingRepository=ossrh::default::https://oss.sonatype.org/content/repositories/snapshots --settings settings.xml
Alternately, the connectionUrl and stagingRepository params can also be configured in the build plugin configuration (maven-release-plugin) along with other configuration. The final set up would look like this:
                org.apache.maven.plugins
                maven-compiler-plugin
                3.6.0
                
                    1.8
                    1.8
                
            

            
            
                org.apache.maven.plugins
                maven-release-plugin
                2.5.3
                
                    scm:git:https://github.com/dikhan/hello-world-cicd.git
                    ossrh::default::https://oss.sonatype.org/content/repositories/snapshots
                    true
                    false
                    release
                    deploy
                    
                        travis-ci/*.sh
                        **/*.java
                        **/*.md
                        **/pom.xml
                        .travis.yml
                    
With the above in the pom we no longer need the params to be passed in the mvn release:stage command:
GPG_KEY_NAME=YOUR_KEY_ID GPG_PASSPHRASE=YOUR_PWD OSSRH_JIRA_USERNAME=USERNAME OSSRH_JIRA_PASSWORD=PWD mvn org.apache.maven.plugins:maven-release-plugin:2.2.1:stage --settings settings.xml
To confirm the upload, the following link should display the new snapshot (1.0.0-SNAPSHOT) just uploaded:
https://oss.sonatype.org/content/repositories/snapshots/com/github/dikhan/hello-world-cicd/

Deploy to release repository

    • Prepare release running the following command (replacing the orange highlighted variables with your details):
GPG_KEY_NAME=YOUR_KEY_ID GPG_PASSPHRASE=YOUR_PWD mvn release:clean release:prepare -Dusername=GITHUB_USERNAME -Dpassword=GITHUB_PWD --settings settings.xml
This command will do the following behind the scenes:
  1. Update the version of the project (and sub-projects) in the pom file removing the snapshot suffix and therefore making it a release version.
  2. Execute tests
  3. Commit and push changes on the pom to the SCM
  4. Create release tag and push it to SCM
  5. Update the version of the project (and sub-projects) in the pom file pumping up the version and adding snapshot as suffix.
  6.  Commit and push changes on the pom to the SCM
    • Perform the release running the following command (replacing the orange highlighted variables with your details):
OSSRH_JIRA_USERNAME=USERNAME OSSRH_JIRA_PASSWORD=PWD mvn release:perform --settings settings.xml
  1. Checkout release tag created previously from SCM
  2. Build and deploy release code to OSSRH (Sonatype)
If you want to clean up the changes in the repo, you can run the following commands and that would bring the code to the state it was before running the above:
  • Remove the two commits created previously from the release plug-in. Remove the lines of commits that you would like to get rid of (e,g: the last two commits) and then save. That will remove the commits 'deleted' from the commit history. Finally we force push to remote to update the repo to match the local changes.
$ git rebase -i HEAD~3
$ git push origin master -f
  • Remove the tag created:
$ git tag -l
$ git tag -d hello-world-cicd-1.0 
$ git push origin :refs/tags/hello-world-cicd-1.0

Travis Integration

At this point, we should have confirmed that we are able to deploy the artifacts to Maven Central using a manual approach. However, this is not an effective way of working and the upmost main goal is to automate as many steps as possible reducing manual intervention making the process of releasing our artifacts less error prone. By doing this, we will also expedite the software development cycle greatly.
Travis-CI is the selected tool to help us mange our Continuous Integration and Continuous Deployment pipeline (CI/CD). The reason why I picked Travis-CI is mainly because it offers a neat simple user interface (UI) to manage the builds, it has GitHub integration and it is free for open source projects. For the sake of brevity, I will not cover Travis-CI initial set-up as there are plenty of tutorials out there already and the Travis-CI documentation is also quite useful.
Without further ado let's dive right into the configuration needed to get our while pipeline automated.

Travis-CI configuration:

Firstly we need to enable Travis-CI to monitor our repository so it is able to trigger a new build anytime a code change is detected.

In the general settings we make sure builds are triggers only in the below specific cases:


Travis yaml file:

Travis-CI requires a descriptive yaml file in order to understand what would be the different steps of the build pipeline. Create a file called '.travis.yml' on the root of your project.
The file will look something like this:
language: java
jdk:
  - oraclejdk8
env:
  global:
  - secure: DkYtFwdk9GB...
  - secure: Gmcb4bcqNRR...
  - secure: had/oI16ksj...
  - secure: GkCNJFw3lE7..
  - secure: gOOO4MU762D..
  - secure: Ulxzv9BYD6S...
  - secure: AjbAWVgAaTv...
branches:
  only:
  - master
install:
  - mvn clean install
  - export TRAVIS_COMMIT_DESCRIPTION=`git log -n 1`
script:
  - chmod +x ./travis-ci/before-deploy.sh ./travis-ci/deploy.sh || travis_terminate 1;
  - ./travis-ci/before-deploy.sh;
  - ./travis-ci/deploy.sh;
cache:
  directories:
  - ~/.m2/repository
Let's describe the most relevant sections:
  • env -> global: This section is reserved to provide values for the environment variables that we used in previous steps. Since we do not want this data to be in clean text, we will be using Travis CLI to encrypt this data. Run the following commands:
$ travis encrypt OSSRH_JIRA_USERNAME=[YOUR_JIRA_USERNAME]
The above would result into the following output:
Please add the following to your .travis.yml file:

  secure: "tEmX0Mgtm4dN..."

Pro Tip: You can add it automatically by running with --add.
Copy the entire secure line and add it to .travis.yml file under env: -> global: section.
As suggested in the tip, travis encrypt can also be executed adding -a to the command and that will add the new encrypted secure into the yml file automatically.
Replicate the same steps for the remaining env variables (used by maven-release-plugin) that need to be encrypted:
$ travis encrypt OSSRH_JIRA_PASSWORD=[YOUR_JIRA_PASSWORD] -a
$ travis encrypt GPG_KEY_NAME=[YOUR_GPG_KEY_NAME] -a
$ travis encrypt GPG_PASSPHRASE=[YOUR_GPG_PASSPHRASE] -a
$ travis encrypt GITHUB_USERNAME=[YOUR_GITHUB_USERNAME] -a
$ travis encrypt GITHUB_PASSWORD=[YOUR_GITHUB_PASSWORD] -a
$ travis encrypt GITHUB_EMAIL=[YOUR_GITHUB_EMAIL] -a
  • branches -> only: This make Travis ignore any branch other than master. Since in the general configuration we enabled 'Build Pull Requests', Travis will only build merges on Master as well as GitHub Pull Requests. 
Please note that if this option is not defined Travis-CI will build any branch pushed to the repo which is not expected and would create a lot of garbage builds in the history.
  • install: This stage is mostly used to prepare the execution environment and install any dependencies needed. In our case, we are just saving the commit description  in an env variable (TRAVIS_COMMIT_DESCRIPTION) which will be used later to prevent travis from building the automatic commits (prepare dev and prepare release) pushed into master created by the maven-release-plugin.
  • script: In this section all the steps that make the continuous deployment possible will be executed. Three different scripts have been created within the 'travis-ci' folder as described below. Note that before executing any script execution permissions need to be granted:
    • ./travis-ci/before-deploy.sh
This script will be preparing the GPG environment so subsequent scripts can make use of the GPG keys to sign files. Hence, first we need to export (to travis-ci/keys folder) both the public and private keys generated previously and encrypt the output file using file travis encryption command.
$ gpg --export --armor your@email.com > travis-ci/keys/codesigning.asc
$ gpg --export-secret-keys --armor your@email.com >> travis-ci/keys/codesigning.asc
Now we need to log in Travis-CI and encrypt the file containing the keys.
$ travis login
...
Successfully logged in as YOUR_USERNAME!
After logging in successfully, the following command will generate an openssl line which needs to be imported into our before-deploy.sh script
travis encrypt-file travis-ci/keys/codesigning.asc travis-ci/keys/codesigning.asc.enc
...
    openssl aes-256-cbc -K $encrypted_XXXX_key -iv $encrypted_YYYY_iv -in codesigning.asc.enc -out travis-ci/keys/codesigning.asc -d
...
The script before-deploy.sh should look like (don't forget to replace the openssl line with the one generated above):
#!/bin/bash
set -ev
# This script will be used by Travis CI, thus it's able to decrypt the cert and sign the artifacts appropriately
echo $TRAVIS_BRANCH;
echo $TRAVIS_COMMIT_DESCRIPTION;
echo $TRAVIS_EVENT_TYPE;
if [[ "$TRAVIS_COMMIT_DESCRIPTION" != *"maven-release-plugin"* ]];then
    if [ "$TRAVIS_BRANCH" == "master" ];then
        # IMPORTANT! REPLACE THE OPEN SSL LINE BELOW WITH THE LINE GENERATED ABOVE
        openssl aes-256-cbc -K $encrypted_75d76ac7d458_key -iv $encrypted_75d76ac7d458_iv -in travis-ci/keys/codesigning.asc.enc -out travis-ci/keys/signingkey.asc -d
        gpg --yes --batch --fast-import travis-ci/keys/signingkey.asc  || { echo $0: mvn failed; exit 1; }
    fi
else
  echo "before-deploy: Not running gpg commands due to maven-release-plugin auto commit - this is just for preparation for dev/releases";
fi
Make sure --fast-import value points to the location where the unencrypted key is.
Note that the first thing the script checks is whether the commit description contains the string maven-release-plugin. This is done to prevent infinite loops when the plugin pushes and merges the two commits needed to prepare the release as well as the new dev snapshot version. Without this check Travis-CI will understand that there is a change in master an would attempt to deploy the new version to maven central one time after another.
  • ./travis-ci/deploy.sh
This script as the name describes itself will be used for the deployment of the artifact to Maven Central. When the code is merged to master it is assumed that that specific version is ready for release and therefore the code executed in this case will perform a release to the release Sonatype repository.
Below is the final version of the script:
#!/bin/bash
set -ev
# This script will be used by Travis CI and will deploy the project to maven, making sure to use the sign and
# build-extras profiles and any settings in our settings file.
echo $TRAVIS_BRANCH;
echo $TRAVIS_COMMIT_DESCRIPTION;
echo $TRAVIS_EVENT_TYPE;
echo $TRAVIS_PULL_REQUEST_BRANCH;
echo $TRAVIS_PULL_REQUEST;
if [[ "$TRAVIS_COMMIT_DESCRIPTION" != *"maven-release-plugin"* ]];then
    if [ "$TRAVIS_BRANCH" == "master" ] && [ "$TRAVIS_PULL_REQUEST_BRANCH" == "" ];then
        git config --global user.email $GITHUB_EMAIL
        git config --global user.name $GITHUB_USERNAME

        git remote set-head origin $TRAVIS_BRANCH
        git show-ref --head
        git symbolic-ref HEAD refs/heads/$TRAVIS_BRANCH
        git symbolic-ref HEAD

        mvn --batch-mode release:clean release:prepare -Dusername=$GITHUB_USERNAME -Dpassword=$GITHUB_PASSWORD
        mvn release:perform --settings travis-ci/settings.xml
    fi
else
  echo "deploy: Not running deploy mvn commands due to maven-release-plugin auto commit - this is just for preparation for dev/releases";
fi
As you may have noticed, there is some pre-configuration needed before deploying the artifact. This is due to the fact that the plugin expects HEAD ref to be defined and point to the branch being built which can be master or a pull request branch. Otherwise, the following error will be thrown:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-release-plugin:
2.5.3:prepare (default-cli) on project hello-world-cicd: An error is occurred 
in the checkin process: Exception while executing SCM command. Detecting the 
current branch failed: fatal: ref HEAD is not a symbolic ref -> [Help 1]
Additionally, git global config needs to be provided to avoid the following error:
fatal: empty ident name issues

Confirming the pipeline

With all the previous bits and bobs in place, we are ready to confirm the CI/CD pipeline. For that, we will create a new branch, submit a PR in GitHub and await for Travis-CI to provide info about the build. While Travis-CI is building the new code GitHub's PR will display the following:

Once the build is finished, Travis-CI will send the result to GitHub:
Now if we merge the Pull Request, Travis-CI will realize about a change in master and will automatically trigger a new build to perform this time the deployment of the artifact version as well as prepare the next version of development.

Pulling the last changes from the repository should show that the pom version has increased (1.1-SNAPSHOT) and that a new release tag has also been created.

$ git tag -l
hello-world-cicd-1.0

GitHub repository configuration

Before wrapping up the tutorial, there is one last thing that needs to be configured. GitHub offers the ability to protect branches and since we do not want everybody to be able merge into master we are going to enforce some restrictions. This can be configured by selecting master branch inside the Branch section within the repository's settings panel.
The next page will allow us to set up more rules specific to the protected branch. In this case I am selecting PullRequests to be required before being able to merge into master (unless you are the admin of the repo). In addition, we also enforce build checks coming from Travis-CI so any given PR would have to honor the status build.

With this last bit in place we can consider the tutorial finished :)

I hope you found the guide helpful. Thanks for reading.

References

One of my major goals for this tutorial was to help the readers to avoid having to go to various tutorials in order to get this process up and running. However, I would have never been able to create this guide without the help of the following readings.
Thanks for reading. Please do not hesitate to provide feedback. 

Java Patterns Worth Knowing

Design patterns describe the best practises and strategies to follow when trying to solve a generic design problem. These patterns were created by experienced object-oriented programmers to address commonly occurring problems and have been applied countless times in all sorts of contexts and situations proving effectively their value. Furthermore, it is worth pointing out that being these patterns language-independent they can be used in any Object Oriented programming language.

Design patterns are classified and grouped in four different categories which are:

  • Creational patters (delegation): These patterns offer the flexibility to decide who is responsible for the object creation and how it will be created, which object will be created and when the creation will take place. Basically, these patterns provide an abstract way to define the object instantiation:
    • Structural patters (aggregation): Focus upon the composition of classes and objects to build larger structures and ease the identification of relationships between entities.
      • Composite
      • Decorator
      • Proxy
      • Facade
      • Behavioural patters (consultation): Concern the communication between objects increasing the flexibility in controlling how this communication is performed.
        • Mediator
        • Chain of Responsibility
        • Observer
        • State
        • Strategy
        • Architectural design patters (architecture level): Patterns related to the different application tiers.
          • Model-View-Controller
          • Service-Oriented architecture
        • Concurrency patterns: Define a way to deal with the multi-threader programming paradigm.
          • Double-checked locked'
          • Monitor object
          • Thread pool
          • Join
          • Lock
        In this write-up I have compiled various examples that fall into the different categories and will be showing how to implement them using the object-oriented programming language Java. The patterns shown in this article are mostly covered in the popular book —Design Patterns: Elements of Reusable Object-Oriented written by  Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides or also known as "Gang of Four" (GoF) book.

        Lastly before we delve into the various patterns, I would like to clarify that designs patterns are not ready-to-use solutions as in libraries that you could use in your program but rather serve as a template solution which needs to be prepared and accommodated to a given context.

        Factory Pattern

        As the name itself indicates,  a factory is used to produce multiple instances of certain type of object(s) on demand. Further, the main responsibility of a factory is to decide which class(es) will be instantiated to create the required object(s) and how the creation of the object(s) will be performed.

        Let's consider the following example - a pastry factory (PastryFactory) should be able to produce various kinds of specific baked products - ConcreteBakedProducts (palmiers, biscuits, profiterole, etc). However, the bakery (BakeryClient) which is the client of the factory, might not want to know about how the pastries are implemented (created) but rather just order them and receive a concrete instantiation of the product requested in the form of its interface - Pastry. The following figure shows the UML diagram of the example described previously:


        Based upon the above diagram we went ahead and created the corresponding Java implementation. For the sake of concreteness not all the possible product classes that would be contained under ConcreteBakedProducts will be implemented; instead we will stick with two simple examples - Palmiers and Biscuits. Clicking on the latter links, you will notice that both classes implement the interface Pastry shown below and provide implementation for the interface methods. This is one of the key points of the factory pattern, as this abstraction will allow the BakeryClient deal with all the different kinds of pastries by just knowing that they are Pastry(s).

        package com.creacodetive.patterns.factory.model;
        
        /**
         * Interface for Pastry products
         * 
         * @author dikhanr
         *
         */
        public interface Pastry {
        
         public String pastryName();
         public String description();
         public String details();
         
        }
        

        As you will see further down, the PastryFactory does not return concrete implementations of pastries such as Palmiers or Biscuits but the actual interface. The createNewPastry(..) method analyses the requested pastryType and returns a Pastry object to the client. Please note that even though we are returning Pastry object instances, this does not mean that we no longer have the concrete implementation (biscuit, palmier). The concrete instance is under-covered in the Pastry object returned.

        package com.creacodetive.patterns.utils;
        import com.creacodetive.patterns.factory.model.Biscuit;
        import com.creacodetive.patterns.factory.model.Palmier;
        import com.creacodetive.patterns.factory.model.Pastry;
        
        /**
         * Factory in charge of producing the different instances of pastries
         * 
         * @author Daniel Isaac Khan Ramiro
         *
         */
        public class PastryFactory {
        
         public static Pastry createNewPastry(String pastryType)
         {
          
          Pastry pastry = null;
          switch(pastryType.toUpperCase()){
           case "PALMIER": 
            pastry = new Palmier();
            break;
           case "BISCUIT":
            pastry = new Biscuit();
            break;    
          }
          return pastry;
         }
         
        }
        

        Thanks to the aforementioned abstraction, the Bakery can easily displayStock() by just calling methods that are part of the contract between the Pastry interface and the concrete classes - pastry.details().

        package com.creacodetive.patterns.factory.api;
        import java.util.ArrayList;
        import java.util.Iterator;
        import java.util.List;
        
        import com.creacodetive.patterns.factory.model.Pastry;
        import com.creacodetive.patterns.utils.PastryFactory;
        
        /**
         * Client that would use the factory to get concrete instances of Pastry
         * 
         * @author Daniel Isaac Khan Ramiro
         *
         */
        public class BakeryClient {
        
         private List<pastry> pastriesList;
         
         public BakeryClient()
         {
          setPastriesList(new ArrayList<pastry>());
         }
         
         public void orderNewPastry(String pastryType)
         {
          Pastry pastry = PastryFactory.createNewPastry(pastryType);
          if(pastry != null)
           pastriesList.add(pastry);
         }
         
         public void displayStock()
         {
          Iterator<pastry> itr = pastriesList.iterator();
          while(itr.hasNext()) {
           Pastry pastry = itr.next();
           System.out.println(pastry.details());
          } 
         }
        
         public List<pastry> getPastriesList() {
          return pastriesList;
         }
        
         public void setPastriesList(List<pastry> pastriesList) {
          this.pastriesList = pastriesList;
         }
        }
        

        Finally, just to make sure that the program behaves as expected, let's run the following simple test client and confirm that the factory generates sweetie and tasty palmiers and biscuits.

        package com.creacodetive.patterns.factory.api;
        public class Test {
        
         public static void main(String[] args)
         {
          BakeryClient bc = new BakeryClient();
          
          System.out.println("");
          System.out.println("Buying a palmier...");
          bc.orderNewPastry("palmier");
          System.out.println("Displaying Bakery's stock...");
          bc.displayStock();
          System.out.println("");
          
          System.out.println("Buying a biscuit...");
          bc.orderNewPastry("biscuit");
          System.out.println("Displaying Bakery's stock...");
          bc.displayStock();
          
         }
         
        }
        

        Console Output:


        For your reference, here are couple examples of factory patterns used within Java EE APIs:
        • java.util.Calendar - getInstance()
        • java.sql.Connection interface - createStatement()
        • java.rmi.server.RmiClientSocketFactory interface - createSocket()

        Singleton Pattern

        The main goal of this pattern is to ensure that only one instance of the class is created. This is achieved by providing just one point of access to the class. In order to be able to control this, we need to force the constructor of the class to be private and expose a public method - normally called getInstance() - that will handle the single instantiation of the class internally and return it as requested.

        Additionally, in order to come up with a good singleton implementation we need to ensure that issues such as synchronization, serialization and potential JVM issues are fully covered.

        Let's now look at how we could solve the synchronization part:

        There are many different ways to create a single instance of the class. However,  these methods differ from each other in the approach selected when initializing the class. Let's go one by one an analyze them:

        - Eager initialization: The instance of the class is created automatically when the class is loaded into the JVM regardless of whether any other class requires it.

        package com.creacodetive.patterns.singleton;
        
        public final class EagerInitSingleton {
         private static EagerInitSingleton instance = new EagerInitSingleton();
        
         /**
          * Note the private access modifier: This forces the client to call the
          * getInstance() method in order to create the instance of the class
          */
         private EagerInitSingleton() {}
        
         /**
          * Method in charge of returning the actual instance of the singleton class.
          * 
          * @return
          */
         public static EagerInitSingleton getInstance() {
          return instance;
         }
        }
        

        Drawback: Object is created in memory even though it may not be needed.

        - Lazy initialization: The instantiation of the class is delayed until any other class asks explicitly for an instance of the singleton class. This approach performs better memory wise compare to the previous example as the object will not be created in memory until it is really needed.

        package com.creacodetive.patterns.singleton;
        
        public final class LazyInitSingleton {
        
         /**
          * Declaring the variable volatile to ensure that the reference of the instance
          * is not returned until the object is constructed.
          */
         private static volatile LazyInitSingleton instance = null;
        
         /**
          * Note the private access modifier: This forces the client to call the
          * getInstance() method in order to create the instance of the class
          */
         private LazyInitSingleton() {}
        
         public static LazyInitSingleton getInstance() {
          synchronized (LazyInitSingleton.class) {
           if (instance == null)
            instance = new LazyInitSingleton();
          }
          return instance;
         }
        }
        

        Note that the synchronized section wraps the check of (instance == null). Otherwise, if it had been placed right after the check we might have faced synchronization issues. The reason is due to the fact that two threads can potentially arrive at the check point at the same time and then wait for the other thread to finish the instantiation. Thereafter, once the second thread is eligible to enter the synchronized code the class has been already instantiated. Nonetheless, the remaining thread will still instantiate the class again resulting into the creation of two instances. You could also think about having a double check within the synchronized statement that checks for whether the class has been already instantiated. However, in my opinion this would be extra code that is not quite needed as there are other approaches that are cleaner code wise.
         
        - Static block initialization: Considering that static blocks are executing while the class loads but before the constructor of the class is called we can certainly use this in our benefit to create the single instance of the class.

        package com.creacodetive.patterns.singleton;
        
        public class StaticInitSingleton {
        
         private static final StaticInitSingleton instance;
        
         // Static block initialization - executed during the loading of the class and
         // even before the constructor is called.
         static {
          try {
           instance = new StaticInitSingleton();
          } catch (Exception e) {
           throw new RuntimeException("Uffff, i was not expecting this!", e);
          }
         }
        
         /**
          * Note the private access modifier: This forces the client to call the
          * getInstance() method in order to create the instance of the class
          */
         private StaticInitSingleton() {}
        
         public static StaticInitSingleton getInstance() {
          return instance;
         }
        }
        

        However, as with the first example, the instance of the singleton class will take place before any class explicitly demands it. Therefore, what can we do in order to avoid it? Let's take a peek at the following snipped of code:

        package com.creacodetive.patterns.singleton;
        
        public class StaticInnerInitSingleton {
        
            private StaticInnerInitSingleton() {}
         
            private static class InnerLazyInit {
                private static final StaticInnerInitSingleton instance = new StaticInnerInitSingleton();
            }
         
            public static StaticInnerInitSingleton getInstance() {
                return InnerLazyInit.instance;
            }
         
        }
        

        As you can see, there are many options which you can choose from. However, I would probably stick with the last approach - Initialization-on-demand holder idiom - as it enables a thread safe and an initialization with great performance.

        Having covered all possible initialization methods and also the scenarios where we might face synchronization issues we are now prepared to move to the next step and tackle issues that might arise regarding serialization/de-serialization.

        When deserializing an object Java internally always tries to create a new instance of that class. Assuming that our Singleton class implements the Serializable interface, we could ran into an issue when the object is being deserialized if the readResolve() method is not returning the single instance but rather a new instance. Thus, in order to avoid this scenario, we just need to implement the aforementioned method by adding the following code into our Singleton class:

        protected Object readResolve() {
                return instance;
            }
        

        Lastly, we need to let the compiler know that the singleton class implemented is unique thus we will prevent possible exceptions thrown when having the class changing in between the serialization and de-serialization. And, how can we do this? By adding just one line of code, the "Serial Version UId" as follows:

        private static final long serialVersionUID = 1L;
        

        In conclusion, depending upon what kind of demands your application has you might want to consider one approach or the other. However, I would highly recommend that before throwing yourself into the pool and start implementing the singleton pattern you make sure that important aspects such as the initialization, synchronization and serialization are taken into consideration specially when dealing with high performance systems. Having said this and just to be on the safe track with all the above scenarios completely covered, I would recommend using the following snipped when implementing the Singleton Pattern:

        package com.creacodetive.patterns.singleton;
        
        import java.io.Serializable;
        
        public class SingletonPattern implements Serializable {
        
         private static final long serialVersionUID = -3283340506858300161L;
        
         private SingletonPattern() {}
        
         private static class InnerSingletonHolder {
          public static final SingletonPattern instance = new SingletonPattern();
         }
        
         public static SingletonPattern getInstance() {
          return InnerSingletonHolder.instance;
         }
        
         protected Object readResolve() {
          return getInstance();
         }
        
        }
        For your reference, here are couple examples of singleton patterns used within Java EE APIs:
        • java.lang.Runtime - getRuntime()
        • java.awt.Desktop - getDesktop()

        Facade Pattern

        This pattern tackles the problem that exist when the functionality of a complex system needs to be exposed as a simplified interface to certain client. Encapsulating the complexity of the subsystem in a single interface greatly reduces the learning curve needed to understand the system. Furthermore, it also makes the platform more decoupled providing more independence between the potential clients that might use the interface and the subsystem. 

        A clear example of a real usage of this pattern is SLF4, which stands for "Simple Logging Facade for Java". Having your application using this abstraction for the logging ensures that any other client making use of your framework will be able to plug-in any logging framework (java.util.logging, logback, log4j) that comply with the SLF4J's specification.

        In the following pattern we are also making use of this pattern thus stay tuned.

        Strategy Pattern

        This pattern falls into the 'Behavioural Patters' category enabling the algorithm's behaviour to be selected in runtime. For instance, consider a task that will be in charge of compressing videos into various formats such as .mkv, .avi and .mp4. The family of different algorithms in this case will be the diverse compressing formats that the compressor tool will support.


        Due to the fact that all the video formatters implement the same interface - CompressStrategy - the client tool, which is the VideoCompressor, can easily interchange them and apply the corresponding compression. Let's take a peek now at the following implementation of the above diagram:

        First we define the interface that will be implemented by the different video formatters.

        package com.creacodetive.patterns.strategy;
        
        public interface CompressStrategy {
        
         public void compress(Video file);
         
        }
        

        After having defined the interface the appropriate different compression methods need to be implemented:

        package com.creacodetive.patterns.strategy;
        
        public class MKV implements CompressStrategy {
        
         @Override
         public void compress(Video file) {
          // Compress video using MKV algorithms
         }
        
        }
        

        package com.creacodetive.patterns.strategy;
        
        public class MP4 implements CompressStrategy {
        
         @Override
         public void compress(Video file) {
          // Compress video using MP4 algorithms
         }
        }
        

        package com.creacodetive.patterns.strategy;
        
        public class AVI implements CompressStrategy {
        
         @Override
         public void compress(Video file) {
          // Compress video using AVI algorithms
         }
        }
        

        Finally, in the client class below - VideoCompressor - you will see that there is a method setCompressionStrategy that enables the reconfiguration of the compression strategy to be applied upon the video. This way, we are able to perform different format algorithms on runtime and the client doesn't need to know anything about the actual compression implementations only caring about what the CompressStrategy interface exposes.

        package com.creacodetive.patterns.strategy;
        
        public class VideoCompressor {
        
         private CompressionStrategy strategy;
        
         public void setCompressionStrategy(CompressionStrategy strategy) {
          this.strategy = strategy;
         }
        
         public void formatVideo(Video video){
          strategy.compress(video);
         }
        
         public static void main(String[] args)
         {
          //Video to be formatted/compressed
          Video video = new Video();
          
          VideoCompressor videoCompressor = new VideoCompressor();
          videoCompressor.setCompressionStrategy(new MKV());
          videoCompressor.formatVideo(video);
          
          videoCompressor.setCompressionStrategy(new AVI());
          videoCompressor.formatVideo(video);
          
          videoCompressor.setCompressionStrategy(new MP4());
          videoCompressor.formatVideo(video); 
         } 
        }
        

        And this will be the output of the program:



        For your reference, here are couple examples of singleton patterns used within Java EE APIs:

        • java.utils.Collections - sort(List list, Comparator c)

        Thanks for reading and please don't hesitate to leave any comments.

        REST Java Architecture

        In this article I am going to walk you through the implementation of a complete client/server RESTful Java web application that fully conforms to the Representational State Transfer (REST) constraints. Restful web services are well suited for the creation of APIs to be exposed to the internet for later consumptions. These APIs provide access to the resources and functionality exposed by the web application and can be accessed by the Unified Resource Identifiers (URIs). RESTful architecture is designed to use a stateless communication protocol (HTTP) which makes the application more scalable both horizontally and vertically, load balanced and easier to deal with upon fail-overs.

        What makes RESTful applications to be simple, lightweight and fast?

        - Resource identification through URI: Global addressing space for resource and service discovery.
        -Uniform interface: Resources are manipulated using a set of operations (CRUD - Create/Read/Update/Delete) determined by the standard HTTP methods (GET, POST, PUT, DELETE) to which the resource is responding.
        - Self-descriptive messages: Content is decoupled in a way that it can be accessed in a variety of formats such as HTML, XML, JSON, plain text, etc.
        - Stateful interaction through links.
        Bearing the above summary regarding REST in mind, let’s get into more detail explaining all the components that will be part of the architecture:

        - RESTEasy: RESTful framework fully complaint with the JAX-RS specification and therefore it ensures that our web services will follow the REST architectural pattern. This will allow us to create the different services that will make our application reachable from the internet.
        - Google Guice: Dependency injection framework that helps our app to be more reusable, testable and maintainable. Moreover, with this in place, we ensure that all the amount of workload needed for complicated tasks such as the inversion of control and dependency injection are properly addressed and performed.
        - MyBatis: Persistance framework in charge of managing and handling the Object relational model (ORM).
        - SLF4J/Logback: Logback will be the framework chosen for all the logging tasks. Since it implements SLF4J, which is the logging facade, we can readily switch to other logging frameworks such as log4j making our app’s logging pretty flexible.

        Figure 1: Diagram showing the software stack used for the architecture.

        Now that we have a more clear view on the general picture, let’s get the ball rolling setting up the project. In order to have an organized build procedure and all the dependencies that our project will rely on we will be using Maven as the build automation tool. Given that, let’s start off by configuring all the Maven projects (different modules) that will compose our architecture.