Deploying to Maven Central

I've been working on developing a vCard parsing library, called ez-vcard, and have decided to upload it to the Maven Central repository. This is the main code repository that all projects configured with Maven use by default. By uploading to Maven Central, developers can more easily use your library with their own projects.

This blog post documents the steps I had to take to do this. Official instructions can be found here, which is where I got most of this information. It's a fairly complex process, so make sure you take your time and don't rush yourself! Your project is going to be released to the world, so make sure you do it right!

1. Prepare the POM file

First, you have to make sure that your POM file is ready.

a. The groupId of your project must be under a domain that you control. If your project is hosted by code hosting service like Sourceforge, then you can prefix the groupId with the hosting service's domain. For example:

• Sourceforge: net.sf.projectName
• Google Code: com.googlecode.projectName
• Github: com.github.projectName

b. The POM must contain the following information:

• <modelVersion>
• <groupId>
• <artifactId>
• <version>
• <packaging>
• <name>
• <description>
• <url>
• <licenses>
• <scm><url>
• <scm><connection>
• <developers>

c. It also must reference the "oss-parent" parent POM if you want to use the special Maven goals to deploy your project (explained in step 4 below).

<parent>
  <groupId>org.sonatype.oss</groupId>
  <artifactId>oss-parent</artifactId>
  <version>7</version>
</parent>

d. Also note that usage of <repository>s and <pluginRepository>s inside of your POM is strongly discouraged. All of your project's dependencies should exist inside of Maven Central.

See the POM of the ez-vcard project for an example of a well-formed POM file.

2. Submit a Sonatype JIRA ticket

You will need to create an account on the Sonatype JIRA website and then submit a JIRA ticket so that your project can be reviewed. Someone will verify that your groupId is valid and that your POM has all the required information. It takes approximately 2 business days to process your request (my ticket was approved the same day I submitted it).

For instructions on how to create a JIRA account and fill out a JIRA ticket, see the Sonatype OSS Maven Repository Usage Guide.

3. Create a public key

While you're waiting for your ticket to be approved, you can generate a public key, which will be used to sign all files that you upload to Maven Central. File signatures are required in order to deploy to Maven Central. A file's signature is stored as a plain text file with the ".asc" extension. They are used to verify whether or not the file was uploaded by the real author.

The public key can be created using a tool called GPG. Most Linux distributions come with this tool pre-installed. If you're on a Windows or Mac computer, you'll have to download it separately (see this page for instructions).

Generate the key

A key can be generated using the following command:

> gpg --gen-key

The command will ask you for the following information (the supplied default is fine for many of these steps):

1. Key type
2. Key size - A high key size means the key will be harder to crack, but it will take longer to generate the key and longer to verify signed files. A size of 2048 is good.
3. Expiration date - The key can be set to never expire, but for extra security, an expiration date can be set. If an expiration date is set, you'll have to regenerate your key once it has expired.
4. Your name and email - This information will be used to label your public key in the public key database.
5. A comment - This can be left blank. It is an optional component of the string that is created from your name and email.
6. Key password - This is optional, but strongly recommended. You will need to enter this every time you sign a file (i.e. every time you deploy to Central).

Afterward, the tool will start collecting data from various activities that are going on inside your computer, such as keyboard and mouse activity. It uses this random data to build a random number called a seed. This seed will be used to kick start a random number generator, which is used to generate your key. It takes a little bit of time, so be patient. Open a text editor and start typing, or just do normal work on your computer. This will speed up the seed generation process.

Distribute the key

The next step is to upload your public key to the Internet. People can then download your key and use it to verify the signatures that you've uploaded with your project files.

First, get the name of your public key. You will need this to run the command that distributes the key. For example, in the console snippet below, the name of the public key is "ED69FC1F".

> gpg --list-keys
pub   2048R/ED69FC1F 2012-09-28
uid                  John Doe <jdoe@hotmail.com>
sub   2048R/E8A6EAD8 2012-09-28

Then, distribute your public key to the Internet with this command:

> gpg --keyserver hkp://pool.sks-keyservers.net/ --send-keys ED69FC1F

There are many key servers on the Internet, but the one above is what I think Maven Central requires or recommends that you to use.

For more information, see How To Generate PGP Signatures With Maven.

4. Deploy to Central

Once your JIRA ticket from step 2 has been approved, you can upload your project to a staging repository, where it will be released to Maven Central. Remember that, once you deploy a release to Central, that version of your library is set in stone. You cannot re-release your library unless you deploy a new version!!

Prepare the project

Before you start, make sure that you do the following:

a. Add your Sonatype JIRA credentials to your Maven settings file (located at "~/.m2/settings.xml").

<settings>
  ...
  <servers>
    <server>
      <id>sonatype-nexus-snapshots</id>
      <username>your-jira-id</username>
      <password>your-jira-pwd</password>
    </server>
    <server>
      <id>sonatype-nexus-staging</id>
      <username>your-jira-id</username>
      <password>your-jira-pwd</password>
    </server>
  </servers>
  ...
</settings>

b. Add "-SNAPSHOT" to the end of the version in the POM. Even though you are deploying a release version, the project's version in the POM file must end in "-SNAPSHOT". Maven will automatically remove this when it builds and deploys your project.

c. Commit all changes to source control. The working copy of your project must have zero uncommitted changes.

Build the project

Next, run these commands to get your project ready for uploading:

> mvn release:clean
> mvn release:prepare -Dusername=SCM_USERNAME -Dpassword=SCM_PASSWORD

The "release:prepare" goal asks you for the following:

1. The release version of your project. For example, if the version in your POM is set to "0.4.1-SNAPSHOT", the release version should be "0.4.1".
2. The name of the SVN tag to create. The goal will automatically create an SVN tag (or equivalent object if using a different SCM) for the release, which is why you need to provide your SCM credentials in the Maven command.
3. The new version to assign to the project after it has been deployed. For example, if you are deploying "0.4.1", you might want the new development version to be "0.4.2-SNAPSHOT".

It then performs the following operations:

1. Does a clean build of the project.
2. Asks for your GPG key password to sign the built files.
3. Changes the version and SCM URLs in your POM to reflect the release version, then commits these changes to your source control system.
4. Creates a SVN tag (or equivalent) for the release version.
5. Changes the version and SCM URLs in your POM to reflect the new development version that was entered above, then commits these changes to your source control system.

Upload the project

The next step is to upload your project to a staging repository. The staging repository gives you one last chance to confirm that your project is in good shape and ready to be released to the world. It also performs automated checks on your project to make sure it meets all the requirements.

> mvn release:perform

This command will:

1. Checkout the SVN tag (or equivalent) that was created with the "release:prepare" goal.
2. Build the checked-out files.
3. Ask for your GPG key password to sign the built files.
4. Upload everything to a staging repository (not Maven Central yet).

Release

Now that your project is in the staging repository, you can release it to the world!

1. Open the Nexus UI by visiting https://oss.sonatype.org/. Login with the JIRA credentials you created in step 2.
2. Click on "Staging Repositories" in the menu on the left.
3. Find the staging repository for your project. Select it, then click the "Close" button. You will be asked to enter a comment describing your action. You can enter something like "Release of version 0.4.1". Closing the repository will perform some automated checks on your project. It makes sure that all files are signed, that your POM has all the required information, and that your project has source code and Javadoc JARs.
4. If there is a problem with your project, you can click "Drop" to delete the staging repository so you can correct the mistakes, re-build, and re-stage your project.
5. Once you've confirmed that your project is in good shape, click "Release". If the "Release" button is disabled, it means it is still performing some automated checks on your project. Wait a few seconds, then click the "Refresh" button. The "Release" should become enabled (if it is not, wait a few more seconds, then click "Refresh" again). After clicking "Release", you'll be asked again to enter a comment.
6. Since this is the first time you are deploying your project to Central, your project must be manually reviewed to make sure everything is OK. Add a comment to the JIRA ticket that you created in step 2, saying that you have released the project. If everything is OK, then the Maven folks will configure your project to sync with Maven Central. It will appear there within 2 hours. All subsequent versions you release will be automatically synced with Central and this last step will not be necessary. It will take approximately 4 hours to appear on search.maven.org.

Congratulations! Your project is now part of the Maven community!

For more information, see the Sonatype OSS Maven Repository Usage Guide.

vCards

When you meet someone for the first time, whether it be a friend or a business partner, how do you exchange contact information? Maybe you send an email to each other to share your email address. Maybe you text or call each other on your cell phones to share your cell phone number. Maybe you write your number on a piece of paper. There are many ways to do this, but each of these ways is error-prone. What if you forget to include a vital piece of information, like the spelling of your last name or the URL of your website?

The idea behind the vCard standard is to provide an easy and hassle-free way for people to share their contact information electronically. A vCard is basically an electronic business card--it contains information like your name, mailing address, email address, telephone number, website, and a picture of yourself. So, when you meet someone new, instead sending them an email with your phone number, address, website, and "darn it what else do they need to know", you can just email them your vCard.

Pretty much all email clients including GMail, Outlook, and Mail can import vCards into their address books. And since many email clients also allow you to export your contacts as a vCard, the vCard standard can act as a sort of data transmission format if you want to switch email clients. You can export all your contacts as a vCard, and then import the vCard into the other email client.

Developer's Overview

From a software engineering perspective, a vCard is just a plain text file (there is also a less popular XML format). It consists of a number of "properties", each of which has zero or more "parameters" and exactly one "value". Each property goes on its own line. Long lines are usually "folded", which means that they are split up into multiple lines. The folded lines all start with a whitespace character to show that they're part of the same line. Binary data, like photos, are encoded in base64.

Here's what my vCard looks like.

BEGIN:VCARD
VERSION:4.0
KIND:individual
SOURCE:http://mikeangstadt.name/mike-angstadt.vcf
FN:Michael Angstadt
N:Angstadt;Michael;;Mr;
NICKNAME:Mike
PHOTO;VALUE=uri:data:image/jpeg;base64,/9j/4AAQSkZJRgABAQEAYABgAAD/4Q
 ZgAASUkqAAgAAAAEABoBBQABAAAAPgAAABsBBQABAAAARgAAACgBAwABAAAAAgAAADEB
 AATgAAAAAAAABgAAAAAQAAAGAAAAABAAAAUGFpbnQuTkVUIHYzLjUuMTAA/9sAQwACAQ
 [more base64 data]
REV;VALUE=timestamp:20120818T155230Z
EMAIL;TYPE=home:mike.angstadt@gmail.com
TEL;TYPE=cell;VALUE=uri:tel:+1 555-555-1234
TEL;TYPE=home;VALUE=uri:tel:+1 555-555-9876
URL;TYPE=home:http://mikeangstadt.name
URL;TYPE=work:http://code.google.com/p/ez-vcard
TZ;VALUE=text:America/New_York
GEO;VALUE=uri:geo:39.95,75.1667
CATEGORIES:Java software engineer,vCard expert,Nice guy
UID:urn:uuid:dd418720-c754-4631-a869-db89d02b831b
LANG:en-US
X-GENERATOR:EZ vCard v0.2.1-SNAPSHOT http://code.google.com/p/ez-vcard
END:VCARD

As you can see, it contains my name, email address, website, and other data. The telephone information is fake, but I wanted to include it just to show what telephone data looks like. The PHOTO property contains a profile picture of myself. Because this property value is so large, it has been folded into multiple lines. The PHOTO property also contains a parameter, "VALUE", which states that the property value is a URI. All vCards must start with "BEGIN:VCARD" and end with "END:VCARD". vCards must use the \r\n newline character sequence.

There are three different vCard versions: 2.1, 3.0, 4.0. Versions 3.0 and 4.0 are RFC standards, defined in RFC 2426 and RFC 6350 respectively. Versions 2.1 and 3.0 are very similar, but version 4.0 is significantly different from the previous versions. It adds many new properties and parameters, and redefines how the values of many existing properties should be encoded.

EZ-vCard

EZ-vCard is an open source Java library that I wrote that reads and creates vCards. It supports all versions of the vCard standard. My goal was to design an API that was as easy to use as possible. For example, here's how to create a vCard file with some basic information:

VCard vcard = new VCard();

vcard.setFormattedName(new FormattedNameType("Barak Obama"));

EmailType email = new EmailType("barak.obama@whitehouse.gov");
email.addType(EmailTypeParameter.WORK);
vcard.addEmail(email);

email = new EmailType("superdude22@hotmail.com");
email.addType(EmailTypeParameter.HOME);
vcard.addEmail(email);

TelephoneType tel = new TelephoneType("(555) 123-5672");
tel.addType(TelephoneTypeParameter.CELL);
vcard.addTelephoneNumber(tel);

File file = new File("obama.vcf");
vcard.write(file);

It's also very easy to read a vCard file using EZ-vCard:

File file = new File("obama.vcf");
VCard vcard = VCard.parse(file);

Nothing says "I'm professional" like a vCard! Create your own vCard today!

JUnit and Temporary Files

Oftentimes, an application interacts with the file system by reading from or creating files and directories. This functionality should, of course, be unit tested to ensure that it works as expected. Also, the unit tests should be self-contained, meaning that any files it reads from or creates should be located within the project itself and not at some location like "C:\unit-test-files". In addition, these temporary files and directories should be cleaned up when the test is done running because, well, they're temporary. And they definitely should not be commited to version control.

You could just throw the files in a location that you know is temporary, like the "target" directory if you use Maven or your operating system's temporary file directory. The problem with this is that if the files are not deleted between test runs, then it could skew your test results. No matter where you put them, they have to be cleaned up when the test is finished running.

You could write the cleanup code yourself OR you could use JUnit's TemporaryFolder class. This class takes care of cleaning up these files and directories after each test finishes running. It will always clean up the files, whether the test passes, fails, or throws an exception. It creates the temp folder when a test starts and deletes the temp folder when the test finishes. It does this for every test method in the class.

Under the covers, TemporaryFolder uses the File.createTempFile() method to create the directory, so it's storing the directory in your operating system's temp directory. It also assigns a unique name to the directory, so if the JVM crashes and TemporaryFolder does NOT clean up your files, the results of your next test run will not be skewed by the files from the previous run.

Here's a code sample demonstrating how the TemporaryFolder class works.

import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TemporaryFolder;

class FileTest {
  @Rule
  public TemporaryFolder temp = new TemporaryFolder();

  @Test
  public void basicTest() throws IOException {
    //the temporary folder is created before this test method runs

    File fileWithoutName = temp.newFile();
    File fileWithName = temp.newFile("myfile.txt");

    File dirWithoutName = temp.newFolder();
    File dirWithName = temp.newFolder("myfolder");

    File fileInsideCreatedDir = new File(dirWithName, "myfile2.txt");

    //the temporary folder is deleted when this test method finishes
  }
}

A class-level instance of TemporaryFolder is created and tagged with the @Rule annotation. This annotation instructs the class to create the temporary folder before a test runs and delete the temporary folder after the test finishes. This field MUST be "public".

The newFile() method creates a new file within the temporary directory. If a file name is not passed into the method, then it will generate a random file name.

The newFolder() method creates a new directory within the temporary directory. As with newFile(), if a name is not passed into the method, then it will generate a random name for the directory.

Note: The zero-argument versions of newFile() and newFolder() were added fairly recently to the API. If you get compilation errors trying to use these, update your JUnit library to the latest version (4.10 at the time of this writing).

You can, of course, create files and directories within a directory that is created with newFolder(). Just pass the File object that was returned by newFolder() into the first argument of the File constructor (as demonstrated with the fileInsideCreatedDir object).

This is a wonderful little gem that takes the pain out of unit testing file system code. I wish I had known about it sooner.

Java 7 Changes

Java 7 was released about a year ago and is the latest version of the Java language. It includes some useful tweaks to the language as well as some improvements to the API. Some of the changes are described in detail below.

Warning: Long blog post ahead! >.<

1. Language Enhancements

Diamond Operator

When using generics in previous versions of Java, you always had to define the generic types twice--once in the variable definition and once in the class constructor call.

Map<Integer, String> map = new HashMap<Integer, String>();

In Java 7, the generic types in the constructor call no longer have to be repeated:

Map<Integer, String> map = new HashMap<>();

Strings in Switch Statements

Before, only integers and characters were allowed to be used in switch statements. In Java 7, strings can be used as well.

switch ("two"){
  case "one":
    ...
    break;
  case "two":
    ...
    break;
  case "three";
    ...
    break;
  default:
    ...
}

Note that the String.equals() method is used behind the scenes to do the string comparison, which means that the comparison is case-sensitive.

The try-with-resource statement

All Java programmers know how important it is to close resources when they are done being used (such as input and output streams). This is best done using a try-catch-finally block.

InputStream in = null;
OutputStream out = null;
try{
  in = ...;
  out = ...;
  ...
} catch (IOException e){
  ...
} finally{
  if (in != null){
      try{
        in.close();
      } catch (IOException e){}
  }
  if (out != null){
      try{
        out.close();
      } catch (IOException e){}
  }
}

Java 7 introduces a special try block, called try-with-resources, that automatically closes the resources for you. This reduces boiler-plate code, making your code shorter, easier to read, and easier to maintain. It also helps to eliminate bugs, since you no longer have to remember to close the resource.

try (InputStream in = ...; OutputStream out = ...){
  ...
} catch (IOException e){
  ...
}

In order to support this, the class must implement the AutoCloseable interface. However, classes that implement Closeable can also be auto-closed. This is because the Closeable interface was modified to extend AutoCloseable.

Multi-exception catch blocks

Previously, only one exception could be caught per catch block. This would sometimes lead to duplicated code:

try {
  ...
} catch (SQLException e) {
  System.out.println(e.getMessage());
  throw e;
} catch (IOException e) {
  System.out.println(e.getMessage());
  throw e;
}

Java 7 lets you put multiple exceptions in a single catch block, thus reducing code duplication:

try {
  ...
} catch (SQLException | IOException e) {
  System.out.println(e.getMessage());
  throw e;
}

Underscores in numeric literals

Sometimes, you have to hard-code a number in your Java code. If the number is long, it can be hard to read.

int i = 1000000;

How long does it take you to read this number? All those digits makes my eyes hurt. Is it ten million? One million? One hundred thousand? In Java 7, underscores can be added to the number to make it more readable.

int i = 1_000_000;

Binary literals

In Java 7, you can hard-code binary values. A binary value starts with "0b" (or "0B") and is followed by a sequence of "0"s and "1"s.

int one = 0b001;
int two = 0b010;
int six = 0b110;

2. New File System API - NIO 2.0

Java 7 adds a revamped file system API called NIO 2.0. Basically, you have a Path class that represents a path on the filesystem and a Files class that allows you to perform operations on an instance of Path, like deleting or copying a file.

Java IO and NIO 2.0 comparison

To give you a feel for the changes, here are two code samples that compare the APIs of the original IO framework and the new NIO 2.0 framework.

Java IO

import java.io.*;

public class JavaIO{
  public static void main(String args[]) throws Exception {
    //define a file
    File file = new File("/home/michael/my-file.txt");

    //create readers/writers
    BufferedReader bufReader = new BufferedReader(new FileReader(file));
    BufferedWriter bufWriter = new BufferedWriter(new FileWriter(file));

    //read file data into memory
    InputStream in = null;
    byte data[];
    try{
      in = new FileInputStream(file);
      ByteArrayOutputStream out = new ByteArrayOutputStream();
      byte buffer[] = new byte[4096];
      int read;
      while ((read = in.read(buffer)) != -1){
        out.write(buffer, 0, read);
      }
      data = out.toByteArray();
    } finally{
      if (in != null){
        in.close();
      }
    }

    //write a string to a file
    PrintWriter writer = null;
    try{
      writer = new PrintWriter(file);
      writer.print("the data");
    } finally{
      if (writer != null){
        writer.close();
      }
    }

    //copy a file
    File target = new File("/home/michael/copy.txt");
    InputStream in2 = null;
    OutputStream out2 = null;
    try{
      in2 = new FileInputStream(file);
      out2 = new FileOutputStream(target);
      byte buffer[] = new byte[4096];
      int read;
      while ( (read = in2.read(buffer)) != -1){
        out2.write(buffer, 0, read);
      }
    } finally {
      if (in2 != null){
        in2.close();
      }
      if (out2 != null){
        out2.close();
      }
    }

    //delete a file
    boolean success = file.delete();
    if (!success){
      //the file couldn't be deleted...we don't know why!!
    }

    //create a directory
    File newDir = new File("/home/michael/mydir");
    success = newDir.mkdir();
    if (!success){
      //directory couldn't be created...we don't know why!!
    }
  }
}

NIO 2.0

import java.io.*;
import java.nio.file.*;
import java.nio.charset.*;

public class JavaNIO{
  public static void main(String args[]) throws Exception {
    //define a file
    Path file = Paths.get("/home/michael/my-file.txt");

    //create readers/writers
    BufferedReader reader = Files.newBufferedReader(file, Charset.defaultCharset());
    BufferedWriter writer = Files.newBufferedWriter(file, Charset.defaultCharset());

    //read file data into memory
    byte data[] = Files.readAllBytes(file);

    //write a string to a file
    Files.write(file, "the data".getBytes());

    //copy a file
    Path target = Paths.get("/home/michael/copy.txt");
    Files.copy(file, target);

    //delete a file
    //throws various exceptions depending on what went wrong
    Files.delete(file);

    //create a directory
    //throws various exceptions depending on what went wrong
    Path newDir = Paths.get("/home/michael/mydir");
    Files.createDirectory(newDir);
  }
}

As you can see, NIO 2.0 adds many convenience methods that remove the need to write a lot of boilerplate code. It also has more fine-grained error handling (for example, when deleting a file and creating a directory).

Directory monitoring

Also added to NIO 2.0 is the ability to monitor directories for changes. This allows your application to immediately respond to events such as files being deleted, modified, or renamed.

WatchService watchService = FileSystems.getDefault().newWatchService();
Path dir = Paths.get("/home/michael");
dir.register(watchService,
  StandardWatchEventKinds.ENTRY_CREATE,
  StandardWatchEventKinds.ENTRY_DELETE,
  StandardWatchEventKinds.ENTRY_MODIFY
);

while(true){
  WatchKey key = watchService.take();
  for (WatchEvent<?> event : key.pollEvents()){
    ...
  }
  key.reset();
}

Because the WatchSevice.take() method blocks while waiting for the next event, you should consider running this code in a separate thread.

Creating new File Systems (ZIP files)

Another feature in NIO 2.0 is the ability to create new file systems. One purpose for this interacting with ZIP files. NIO 2.0 basically treats a ZIP file as if it were a flash drive or another hard drive on your computer. You read, write, copy, and delete files on the ZIP file system as if they were ordinary files on a hard drive. The example below shows how create a ZIP file and add a file to it.

Map<String, String> env = new HashMap<>(); 
env.put("create", "true");
URI uri = URI.create("jar:file:/home/michael/my-zip.zip");

try (FileSystem zip = FileSystems.newFileSystem(uri, env)) {
  Path externalFilePath = Paths.get("my-file.txt");
  Path zipFilePath = zip.getPath("/my-file.txt");          
  Files.copy(externalFilePath, zipFilePath);
}

3. Fork/Join Framework

New fork/join classes were added to Java's Concurrency framework. They make it easier to divide a single task into many smaller tasks so the task can be completed using multi-threading. It uses a worker-stealing technique in which a worker thread will "steal" work from another worker thread if it finishes its work before the other. This helps to reduce the total amount of time it takes for a task to complete. Here's an example of how you would use fork/join to calculate the sum of an array of integers.

import java.util.concurrent.*;

public class ForkJoinDemo{
  public static void main(String args[]){
    ForkJoinPool pool = new ForkJoinPool();
    SumTask task = new SumTask(new int[]{0,1,2,3,4,5});
    Integer result = pool.invoke(task);
    System.out.println(result);
  }

  private static class SumTask extends RecursiveTask<Integer> {
    private final int[] numbers;
    private final int start, end;

    public SumTask(int[] numbers){
      this(numbers, 0, numbers.length - 1);
    }

    public SumTask(int[] numbers, int start, int end){
      this.numbers = numbers;
      this.start = start;
      this.end = end;
    }

    @Override
    public Integer compute(){
      if (start == end){
        return numbers[start];
      }
      if (end-start == 1){
        return numbers[start] + numbers[end];
      }
      int mid = (end+start)/2;
      SumTask t1 = new SumTask(numbers, start, mid);
      t1.fork();
      SumTask t2 = new SumTask(numbers, mid+1, end);
      return t2.compute() + t1.join();
    }
  } 
}

First, a ForkJoinPool object is created. Then, a task (SumTask) is instantiated and passed to the ForkJoinPool.invoke() method, which blocks until the task is complete. By default, ForkJoinPool will create one thread for each core in your computer.

4. Misc

Throwable.getSuppressed()

Java 7 adds a method called getSuppressed() to the Throwable class. This method allows the programmer to get any exceptions that were suppressed by the thrown exception.

An exception can be suppressed in a try-finally block or in the new try-with-resources block. In a try-finally block, if an exception is thrown from both the try and finally blocks, the exception thrown from the try block will be suppressed and the exception thrown from the finally block will be returned.

BufferedReader reader = null;
try{
  reader = ...;
  reader.readLine(); //IOException thrown (**suppressed**)
} finally {
  reader.close(); //IOException thrown
}

In Java 7's new try-with-resources block, it is the opposite--the exception thrown when it closes resource is suppressed, while the exception thrown from the try block is returned:

try (BufferedReader reader = ...){
  reader.readLine(); //IOException thrown
}
//IOException thrown when "reader" is closed (**suppressed**)

In previous Java versions, there was no way to get this suppressed exception. But in Java 7, calling the new Throwable.getSuppressed() method will return an array of all suppressed exceptions.

Shaped and Translucent Windows in Swing

You can now create non-square, transparent windows in Swing. Examples can be found in the Java Swing tutorial.

---

These are just some of the changes that were added to Java 7. Out of all the changes I've mentioned, my favorite is the file monitoring functionality in the NIO 2.0 framework. This is completely new functionality that didn't exist before in previous Java versions. What's your favorite?? Leave a note in the comments below.

The complete Java 7 release notes can be found on Oracle's website.

HTTP 304

The Bing homepage is a lot prettier looking than Google's. It has a slick, high-resolution background image that changes every week or so. But an image like this takes time to download and it increases the load time of the page. What if you're doing a research project and you have to visit Bing several times a day? Does your browser have to download this image over and over again?

The answer is no. When your browser downloads an image for the first time, it saves it to a cache on the hard drive. A If-Modified-Since header is then added to all subsequent requests for the image and it contains the time that the browser last downloaded the image. The server looks at this time and compares it with the time that the image was last modified on the server. If the image hasn't been modified since then, it returns a HTTP 304 response with an empty body (the image data is left out). The browser sees this status code and knows that it's OK to use the cached version of the image. This means that the image does not have to be downloaded again and the page loads more quickly. If the image has been modified since the browser last downloaded it, then a normal HTTP 200 response is returned containing the image data.

---

Using the Bing.com background image mentioned above as an example, let's try using curl to test this out. First, we'll send a request without the If-Modified-Since header. This should return a normal HTTP 200 response with the image in the response body.

Note: Curl sends the response body to stdout--because we're not interested in the actual image data, we'll just direct it to /dev/null to throw it away. The --verbose argument displays the request and response headers.

curl --verbose "http://www.bing.com/az/hprichbg?p=rb%2fTimothyGrassPollen_EN-US8441009544_1366x768.jpg" > /dev/null

Request headers:

GET /az/hprichbg?p=rb%2fTimothyGrassPollen_EN-US8441009544_1366x768.jpg HTTP/1.1
User-Agent: curl/7.21.6 (i686-pc-linux-gnu)
Host: www.bing.com

Response headers:

HTTP/1.1 200 OK
Content-Type: image/jpeg
Last-Modified: Fri, 08 Jun 2012 09:37:14 GMT
Content-Length: 155974

As expected, an HTTP 200 response was returned containing a JPEG image that's about 155KB in size. The Last-Modified header shows the date that the image was last modified on the server.

---

Now let's try sending a request that will cause a HTTP 304 response to be returned. As shown in the Last-Modified header from the response above, the image was last modifed on the morning of June 8. Let's pretend that we last downloaded the image on June 9. Because the image hasn't changed since we've downloaded it, we know that we have the most recent image, so an HTTP 304 response should be returned.

curl --verbose --header "If-Modified-Since: Sat, 09 Jun 2012 09:37:14 GMT" "http://www.bing.com/az/hprichbg?p=rb%2fTimothyGrassPollen_EN-US8441009544_1366x768.jpg" > /dev/null

Request headers:

GET /az/hprichbg?p=rb%2fTimothyGrassPollen_EN-US8441009544_1366x768.jpg HTTP/1.1
User-Agent: curl/7.21.6 (i686-pc-linux-gnu)
Host: www.bing.com
If-Modified-Since: Sat, 09 Jun 2012 09:37:14 GMT

Response headers:

HTTP/1.1 304 Not Modified
Content-Type: image/jpeg
Last-Modified: Fri, 08 Jun 2012 09:37:14 GMT

An HTTP 304 response was returned with an empty body (as shown by the lack of a Content-Length header) as expected.

---

Let's play pretend one more time and say that we last downloaded the image on June 7. The image was last updated on June 8, so this means that we have an outdated copy of the image and we need to download a fresh copy.

curl --verbose --header "If-Modified-Since: Thu, 07 Jun 2012 09:37:14 GMT" "http://www.bing.com/az/hprichbg?p=rb%2fTimothyGrassPollen_EN-US8441009544_1366x768.jpg" > /dev/null

Request headers:

GET /az/hprichbg?p=rb%2fTimothyGrassPollen_EN-US8441009544_1366x768.jpg HTTP/1.1
User-Agent: curl/7.21.6 (i686-pc-linux-gnu)
Host: www.bing.com
If-Modified-Since: Thu, 07 Jun 2012 09:37:14 GMT

Response headers:

HTTP/1.1 200 OK
Content-Type: image/jpeg
Last-Modified: Fri, 08 Jun 2012 09:37:14 GMT
Content-Length: 155974

As shown in the response, the server detected the fact that our copy was out of date and sent us a HTTP 200 response with the image data in it.

---

So as you can see, without this caching mechanism, the web would be much slower. Your browser would have to download everything from scratch every time a page is loaded. But with caching, your browser can pull images from the cache without having to download them again.

Summary of "Your Mouse is a Database" - May 2012 CACM

In a typical web application, when you make an Ajax call to a web service, you have to wait for the entire response to be received before you can act on it. You cannot process the data as it's coming in off the wire. The article "Your Mouse is a Databse" in the May 2012 issue of CACM by Erik Meijer explores a programming API called Reactive Extensions that can asynchronously stream large or infinite amounts of data to an application (think of a stream of Twitter tweets--new tweets are always being created, so there's never an end to the data). Two key words here are "asynchronous" and "stream". Asynchronous means that the data is handled in a separate thread so the application's UI does not lock up. And stream means that the data is acted on as it's being received.

The term "data" is used very broadly--it doesn't have to be data from a web service over the Internet. The data can also be UI input from the user such mouse movements or typing characters into a text box. The data is push-based because it is sent to the consumer instead of the consumer having to explicitly request it.

To help explain the concept, Meijer proposes modifying Java's Future interface (an interface used to check on the status of threads that are running in the background) to handle such data.

interface Future<T> {
  Closable get(Observer<T> callback);
}

He slims the Future<T> interface down to just one method. The get method allows a consumer to subscribe to the data stream. The return value for the method is an instance of Closeable, which allows the programmer to cancel that particular subscription if she wishes.

The Observer<T> parameter processes data from the stream asynchronously. Meijer bases the Observer<T> interface on GWT's AsyncCallback interface by giving it onFailure() and onSuccess() methods. onFailure() is called if there's an error retrieving data from the stream. onSuccess() is called when (or if) the stream ends. The third method, onNext() defines how to handle each data item from the stream (like saving it to a database or displaying it on the screen).

interface Observer<T> {
  void onFailure(Throwable t);
  void onSuccess();
  void onNext(T value);
}

Meijer then describes a number of query opeators in the Reactive Extensions library that can be used to filter this streaming data. For instance, the "where" operator lets the programmer specify the criteria for whether or not a data item should be processed. If the data item does not meet the criteria, then it is discarded. Another operator is "throttle", which prevents too much data from being processed too quickly. For example, if the throttle value is set to 2 seconds and 10 data items are pushed within a 2 second time span, it will only process the most recent message within that 2 second time span. The other 9 messages will be ignored. These operators can be chained together to give the programmer strong control over how to filter a stream.

This idea of streaming push-based based data can help developers design more memory-efficient applications. Tt can also help developers filter the data from larger streams, like Twitter for example, so as to not overwhelm the application with data it doesn't need.

Summary of "Idempotence Is Not a Medical Condition" - CACM, May 2012

I wanted to read the article "Idempotence Is Not a Medical Condition" by Pat Helland in the May 2012 issue of CACM because it smelled like an article that was full of big words and fuzzy architecture abstractions. I learned some things from the article (and I really like the title), but it's mostly full of FUD. The jist of the article is: "Are you SURE that the messages you send over the network are delivered successfully? Are you REALLY sure? I mean, are you SUPER DUPER sure? After all, how can we be certain of anything in this crazy world?" By the way, SQL Server Broker eliminates this uncertainty for you.

---

In this article, Helland says that distributed systems today are largely composed of a collection of off-the-shelf software and cloud-based Internet services. This means that they lack a centrally-enforced communication policy. So, extra care must be taken to ensure that each message is delivered and that no message is lost, even if one of the sub-systems goes down or restarts.

This contrasts with the traditional notion of a distributed system: a cluster of tightly-coupled computers in a lab somewhere, which are specifically programmed to talk only to each other. Communication is simpler in this situation because each sub-system is essentially identical and also physically present in the same location.

Every sub-system has some sort of "plumbing" which sends and receives messages (e.g. a TCP network stack). If the plumbing is able to do things like handle duplicate messages and resend lost messages, then the system's application layer does not have to be programmed to ensure reliable communication. But if the plumbing cannot do these things, then the application layer is responsible.

A message should only be marked as "consumed" if the action that the receiving application performs on the message is successful. For example, if an application tries to save a message it receives to a database, but the database transaction fails, then the message should not be considered "consumed" because the operation that the message invoked (a database call) was not successful. The message should sit in a queue somewhere until the application is able to successfully process it or it should respond to the client saying that it failed the operation.

Helland also states that messages should be idempotent, meaning that if the same message is sent multiple times, the effect it has on the server should be the same as if just one message was sent. The term is often used in the context of HTTP--GET requests are defined as being idempotent, while POST requests are not.

Messages can have a preference for one of two types of behavior: "history" or "currency". History means that the messages must be delivered in order. If message N+1 arrives, but message N has not yet arrived, then it must wait until it receives message N in order to deliver message N+1 to the application. Downloading a file requires this type of communication because all of the file's data must be delivered in order or else you'll get a corrupt file. Currency means that getting the most recent message is what matters. If some messages are skipped or lost, that's OK. Getting the most recent price of a stock is one such example of this behavior type.

Helland says that the greatest moment of uncertainty during the request/response cycle is right after the request is sent and the application is waiting for a response. The assumption is that the message has been received and is being processed, but you don't know this for sure. For all you know, the receiving end could have decided to ignore the request and not send a response. It's for this reason that timeouts are needed. If the requester has not received a response within the timeout period, then it will assume the request was not processed and will stop waiting for a response.

However, there is one thing that you can know for certain, Helland says. "Each message is guaranteed to be delivered zero or more times!". Finally, something I can rely on!

Helland says that despite TCP's robustness and wide-spread use, it cannot fully be trusted to deliver messages reliably. He says that TCP "offers no guarantees once the connection is terminated or one of the processes completes or fails." I don't understand what he's trying to say. Of course it offers no guarantees once the connection is terminated. You can't send any messages over a terminated connection. And of course it offers no guarantees once one of the processes completes or fails. At that point, the TCP conversation is over.

He then goes on to say that "challenges arise when longer-lived participants [such as HTTP requests] are involved." He says that when a persistent HTTP connection is needed, the TCP connection is usually kept alive, but there's no guarantee that this will happen. Because of this, the HTTP request may have to be sent multiple times.

Helland then goes into an in-depth discussion about idempotency. He says that, technically, no idempotent request is truly idempotent because every request has some lasting effect on the server. For instance, most servers keep an access log of every request that was received. Making five identical requests, even if they are idempotent, will add five entries to the log file. Also, the performance of the server is impacted every time it receives a request. The more requests it receives, the more its performance will degrade. However, these side-effects are not related to how the actual application logic behaves, which is the true context in which the term "idempotent" should be used.

Helland states that state-ful communication is more difficult than state-less communication because all previously sent messages must be taken into consideration when processing the current message.

He also points out that there's no way of knowing whether the server that receives a request is doing the actual work to fulfill that request. The server could be forwarding the request to another server to do the actual work.

Helland says that if a dialog between two servers breaks apart in the middle of the conversation, both ends must be able to cleanly recover from this failure.

If a service is load-balanced across multiple servers, then state-ful information must be stored in such a way so that a client's state is not lost. One way to do this is to store the state information on a designated server that the other servers have access to. That way, requests from the same client can be handled by any server in the cluster.

Alternatively, the client could be assigned to a designated server in the cluster by a load-balancer when it makes its first request. This server will then be responsible for maintaining the state information for this client and handling all of the client's requests. The first request that the client makes must be idempotent because if the request is received successfully, but the response from the server is lost, then the client will assume that the request was lost and try making the request again. When the request is sent for the second time, the load-balancer may assign the request to a different server. If the request is not idempotent, then the request will be applied twice, thus tainting the server's data. Once the client receives a server response to its first request, it now knows which server in the cluster it should communicate with, which means that all subsequent messages don't have to be idempotent because there's no risk of sending the same request to multiple servers.

Helland says there are three ways to make this first client request idempotent. (1) You can send basically an empty message to the server, such as a TCP SYN message, (2) you can perform a read-only operation, or (3) you have the server queue a non-idempotent operation which will only be executed by the server once the connection has been confirmed. Approach (1) is the simplest, but approaches (2) and (3) can be seen as more efficient because they are performing a useful operation. Or, in Helland's words: "allowing the application to do useful work with the round-trip is cool."

The last point Helland makes is that the last message of a conversation cannot be guaranteed. This is because, if you were to send a response to the last message stating that you have received it, then it wouldn't be the last message! Therefore, applications must be designed so that it is not important whether the last message is received or not.