Saturday, August 18, 2012

vCards

When you meet someone for the first time, whether it be a friend or a business partner, how do you exchange contact information? Maybe you send an email to each other to share your email address. Maybe you text or call each other on your cell phones to share your cell phone number. Maybe you write your number on a piece of paper. There are many ways to do this, but each of these ways is error-prone. What if you forget to include a vital piece of information, like the spelling of your last name or the URL of your website?

The idea behind the vCard standard is to provide an easy and hassle-free way for people to share their contact information electronically. A vCard is basically an electronic business card--it contains information like your name, mailing address, email address, telephone number, website, and a picture of yourself. So, when you meet someone new, instead sending them an email with your phone number, address, website, and "darn it what else do they need to know", you can just email them your vCard.

Pretty much all email clients including GMail, Outlook, and Mail can import vCards into their address books. And since many email clients also allow you to export your contacts as a vCard, the vCard standard can act as a sort of data transmission format if you want to switch email clients. You can export all your contacts as a vCard, and then import the vCard into the other email client.

Developer's Overview

From a software engineering perspective, a vCard is just a plain text file (there is also a less popular XML format). It consists of a number of "properties", each of which has zero or more "parameters" and exactly one "value". Each property goes on its own line. Long lines are usually "folded", which means that they are split up into multiple lines. The folded lines all start with a whitespace character to show that they're part of the same line. Binary data, like photos, are encoded in base64.

Here's what my vCard looks like.

BEGIN:VCARD
VERSION:4.0
KIND:individual
SOURCE:http://mikeangstadt.name/mike-angstadt.vcf
FN:Michael Angstadt
N:Angstadt;Michael;;Mr;
NICKNAME:Mike
PHOTO;VALUE=uri:data:image/jpeg;base64,/9j/4AAQSkZJRgABAQEAYABgAAD/4Q
 ZgAASUkqAAgAAAAEABoBBQABAAAAPgAAABsBBQABAAAARgAAACgBAwABAAAAAgAAADEB
 AATgAAAAAAAABgAAAAAQAAAGAAAAABAAAAUGFpbnQuTkVUIHYzLjUuMTAA/9sAQwACAQ
 [more base64 data]
REV;VALUE=timestamp:20120818T155230Z
EMAIL;TYPE=home:mike.angstadt@gmail.com
TEL;TYPE=cell;VALUE=uri:tel:+1 555-555-1234
TEL;TYPE=home;VALUE=uri:tel:+1 555-555-9876
URL;TYPE=home:http://mikeangstadt.name
URL;TYPE=work:http://code.google.com/p/ez-vcard
TZ;VALUE=text:America/New_York
GEO;VALUE=uri:geo:39.95,75.1667
CATEGORIES:Java software engineer,vCard expert,Nice guy
UID:urn:uuid:dd418720-c754-4631-a869-db89d02b831b
LANG:en-US
X-GENERATOR:EZ vCard v0.2.1-SNAPSHOT http://code.google.com/p/ez-vcard
END:VCARD

As you can see, it contains my name, email address, website, and other data. The telephone information is fake, but I wanted to include it just to show what telephone data looks like. The PHOTO property contains a profile picture of myself. Because this property value is so large, it has been folded into multiple lines. The PHOTO property also contains a parameter, "VALUE", which states that the property value is a URI. All vCards must start with "BEGIN:VCARD" and end with "END:VCARD". vCards must use the \r\n newline character sequence.

There are three different vCard versions: 2.1, 3.0, 4.0. Versions 3.0 and 4.0 are RFC standards, defined in RFC 2426 and RFC 6350 respectively. Versions 2.1 and 3.0 are very similar, but version 4.0 is significantly different from the previous versions. It adds many new properties and parameters, and redefines how the values of many existing properties should be encoded.

EZ-vCard

EZ-vCard is an open source Java library that I wrote that reads and creates vCards. It supports all versions of the vCard standard. My goal was to design an API that was as easy to use as possible. For example, here's how to create a vCard file with some basic information:

VCard vcard = new VCard();

vcard.setFormattedName(new FormattedNameType("Barak Obama"));

EmailType email = new EmailType("barak.obama@whitehouse.gov");
email.addType(EmailTypeParameter.WORK);
vcard.addEmail(email);

email = new EmailType("superdude22@hotmail.com");
email.addType(EmailTypeParameter.HOME);
vcard.addEmail(email);

TelephoneType tel = new TelephoneType("(555) 123-5672");
tel.addType(TelephoneTypeParameter.CELL);
vcard.addTelephoneNumber(tel);

File file = new File("obama.vcf");
vcard.write(file);

It's also very easy to read a vCard file using EZ-vCard:

File file = new File("obama.vcf");
VCard vcard = VCard.parse(file);

Nothing says "I'm professional" like a vCard! Create your own vCard today!

Sunday, August 12, 2012

JUnit and Temporary Files

Oftentimes, an application interacts with the file system by reading from or creating files and directories. This functionality should, of course, be unit tested to ensure that it works as expected. Also, the unit tests should be self-contained, meaning that any files it reads from or creates should be located within the project itself and not at some location like "C:\unit-test-files". In addition, these temporary files and directories should be cleaned up when the test is done running because, well, they're temporary. And they definitely should not be commited to version control.

You could just throw the files in a location that you know is temporary, like the "target" directory if you use Maven or your operating system's temporary file directory. The problem with this is that if the files are not deleted between test runs, then it could skew your test results. No matter where you put them, they have to be cleaned up when the test is finished running.

You could write the cleanup code yourself OR you could use JUnit's TemporaryFolder class. This class takes care of cleaning up these files and directories after each test finishes running. It will always clean up the files, whether the test passes, fails, or throws an exception. It creates the temp folder when a test starts and deletes the temp folder when the test finishes. It does this for every test method in the class.

Under the covers, TemporaryFolder uses the File.createTempFile() method to create the directory, so it's storing the directory in your operating system's temp directory. It also assigns a unique name to the directory, so if the JVM crashes and TemporaryFolder does NOT clean up your files, the results of your next test run will not be skewed by the files from the previous run.

Here's a code sample demonstrating how the TemporaryFolder class works.

import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TemporaryFolder;

class FileTest {
  @Rule
  public TemporaryFolder temp = new TemporaryFolder();

  @Test
  public void basicTest() throws IOException {
    //the temporary folder is created before this test method runs

    File fileWithoutName = temp.newFile();
    File fileWithName = temp.newFile("myfile.txt");

    File dirWithoutName = temp.newFolder();
    File dirWithName = temp.newFolder("myfolder");

    File fileInsideCreatedDir = new File(dirWithName, "myfile2.txt");

    //the temporary folder is deleted when this test method finishes
  }
}

A class-level instance of TemporaryFolder is created and tagged with the @Rule annotation. This annotation instructs the class to create the temporary folder before a test runs and delete the temporary folder after the test finishes. This field MUST be "public".

The newFile() method creates a new file within the temporary directory. If a file name is not passed into the method, then it will generate a random file name.

The newFolder() method creates a new directory within the temporary directory. As with newFile(), if a name is not passed into the method, then it will generate a random name for the directory.

Note: The zero-argument versions of newFile() and newFolder() were added fairly recently to the API. If you get compilation errors trying to use these, update your JUnit library to the latest version (4.10 at the time of this writing).

You can, of course, create files and directories within a directory that is created with newFolder(). Just pass the File object that was returned by newFolder() into the first argument of the File constructor (as demonstrated with the fileInsideCreatedDir object).

This is a wonderful little gem that takes the pain out of unit testing file system code. I wish I had known about it sooner.

Saturday, June 23, 2012

Java 7 Changes

Java 7 was released about a year ago and is the latest version of the Java language. It includes some useful tweaks to the language as well as some improvements to the API. Some of the changes are described in detail below.

Warning: Long blog post ahead! >.<

1. Language Enhancements

Diamond Operator

When using generics in previous versions of Java, you always had to define the generic types twice--once in the variable definition and once in the class constructor call.

Map<Integer, String> map = new HashMap<Integer, String>();

In Java 7, the generic types in the constructor call no longer have to be repeated:

Map<Integer, String> map = new HashMap<>();

Strings in Switch Statements

Before, only integers and characters were allowed to be used in switch statements. In Java 7, strings can be used as well.

switch ("two"){
  case "one":
    ...
    break;
  case "two":
    ...
    break;
  case "three";
    ...
    break;
  default:
    ...
}

Note that the String.equals() method is used behind the scenes to do the string comparison, which means that the comparison is case-sensitive.

The try-with-resource statement

All Java programmers know how important it is to close resources when they are done being used (such as input and output streams). This is best done using a try-catch-finally block.

InputStream in = null;
OutputStream out = null;
try{
  in = ...;
  out = ...;
  ...
} catch (IOException e){
  ...
} finally{
  if (in != null){
      try{
        in.close();
      } catch (IOException e){}
  }
  if (out != null){
      try{
        out.close();
      } catch (IOException e){}
  }
}

Java 7 introduces a special try block, called try-with-resources, that automatically closes the resources for you. This reduces boiler-plate code, making your code shorter, easier to read, and easier to maintain. It also helps to eliminate bugs, since you no longer have to remember to close the resource.

try (InputStream in = ...; OutputStream out = ...){
  ...
} catch (IOException e){
  ...
}

In order to support this, the class must implement the AutoCloseable interface. However, classes that implement Closeable can also be auto-closed. This is because the Closeable interface was modified to extend AutoCloseable.

Multi-exception catch blocks

Previously, only one exception could be caught per catch block. This would sometimes lead to duplicated code:

try {
  ...
} catch (SQLException e) {
  System.out.println(e.getMessage());
  throw e;
} catch (IOException e) {
  System.out.println(e.getMessage());
  throw e;
}

Java 7 lets you put multiple exceptions in a single catch block, thus reducing code duplication:

try {
  ...
} catch (SQLException | IOException e) {
  System.out.println(e.getMessage());
  throw e;
}

Underscores in numeric literals

Sometimes, you have to hard-code a number in your Java code. If the number is long, it can be hard to read.

int i = 1000000;

How long does it take you to read this number? All those digits makes my eyes hurt. Is it ten million? One million? One hundred thousand? In Java 7, underscores can be added to the number to make it more readable.

int i = 1_000_000;

Binary literals

In Java 7, you can hard-code binary values. A binary value starts with "0b" (or "0B") and is followed by a sequence of "0"s and "1"s.

int one = 0b001;
int two = 0b010;
int six = 0b110;

2. New File System API - NIO 2.0

Java 7 adds a revamped file system API called NIO 2.0. Basically, you have a Path class that represents a path on the filesystem and a Files class that allows you to perform operations on an instance of Path, like deleting or copying a file.

Java IO and NIO 2.0 comparison

To give you a feel for the changes, here are two code samples that compare the APIs of the original IO framework and the new NIO 2.0 framework.

Java IO

import java.io.*;

public class JavaIO{
  public static void main(String args[]) throws Exception {
    //define a file
    File file = new File("/home/michael/my-file.txt");

    //create readers/writers
    BufferedReader bufReader = new BufferedReader(new FileReader(file));
    BufferedWriter bufWriter = new BufferedWriter(new FileWriter(file));

    //read file data into memory
    InputStream in = null;
    byte data[];
    try{
      in = new FileInputStream(file);
      ByteArrayOutputStream out = new ByteArrayOutputStream();
      byte buffer[] = new byte[4096];
      int read;
      while ((read = in.read(buffer)) != -1){
        out.write(buffer, 0, read);
      }
      data = out.toByteArray();
    } finally{
      if (in != null){
        in.close();
      }
    }

    //write a string to a file
    PrintWriter writer = null;
    try{
      writer = new PrintWriter(file);
      writer.print("the data");
    } finally{
      if (writer != null){
        writer.close();
      }
    }

    //copy a file
    File target = new File("/home/michael/copy.txt");
    InputStream in2 = null;
    OutputStream out2 = null;
    try{
      in2 = new FileInputStream(file);
      out2 = new FileOutputStream(target);
      byte buffer[] = new byte[4096];
      int read;
      while ( (read = in2.read(buffer)) != -1){
        out2.write(buffer, 0, read);
      }
    } finally {
      if (in2 != null){
        in2.close();
      }
      if (out2 != null){
        out2.close();
      }
    }

    //delete a file
    boolean success = file.delete();
    if (!success){
      //the file couldn't be deleted...we don't know why!!
    }

    //create a directory
    File newDir = new File("/home/michael/mydir");
    success = newDir.mkdir();
    if (!success){
      //directory couldn't be created...we don't know why!!
    }
  }
}

NIO 2.0

import java.io.*;
import java.nio.file.*;
import java.nio.charset.*;

public class JavaNIO{
  public static void main(String args[]) throws Exception {
    //define a file
    Path file = Paths.get("/home/michael/my-file.txt");

    //create readers/writers
    BufferedReader reader = Files.newBufferedReader(file, Charset.defaultCharset());
    BufferedWriter writer = Files.newBufferedWriter(file, Charset.defaultCharset());

    //read file data into memory
    byte data[] = Files.readAllBytes(file);

    //write a string to a file
    Files.write(file, "the data".getBytes());

    //copy a file
    Path target = Paths.get("/home/michael/copy.txt");
    Files.copy(file, target);

    //delete a file
    //throws various exceptions depending on what went wrong
    Files.delete(file);

    //create a directory
    //throws various exceptions depending on what went wrong
    Path newDir = Paths.get("/home/michael/mydir");
    Files.createDirectory(newDir);
  }
}

As you can see, NIO 2.0 adds many convenience methods that remove the need to write a lot of boilerplate code. It also has more fine-grained error handling (for example, when deleting a file and creating a directory).

Directory monitoring

Also added to NIO 2.0 is the ability to monitor directories for changes. This allows your application to immediately respond to events such as files being deleted, modified, or renamed.

WatchService watchService = FileSystems.getDefault().newWatchService();
Path dir = Paths.get("/home/michael");
dir.register(watchService,
  StandardWatchEventKinds.ENTRY_CREATE,
  StandardWatchEventKinds.ENTRY_DELETE,
  StandardWatchEventKinds.ENTRY_MODIFY
);

while(true){
  WatchKey key = watchService.take();
  for (WatchEvent<?> event : key.pollEvents()){
    ...
  }
  key.reset();
}

Because the WatchSevice.take() method blocks while waiting for the next event, you should consider running this code in a separate thread.

Creating new File Systems (ZIP files)

Another feature in NIO 2.0 is the ability to create new file systems. One purpose for this interacting with ZIP files. NIO 2.0 basically treats a ZIP file as if it were a flash drive or another hard drive on your computer. You read, write, copy, and delete files on the ZIP file system as if they were ordinary files on a hard drive. The example below shows how create a ZIP file and add a file to it.

Map<String, String> env = new HashMap<>(); 
env.put("create", "true");
URI uri = URI.create("jar:file:/home/michael/my-zip.zip");

try (FileSystem zip = FileSystems.newFileSystem(uri, env)) {
  Path externalFilePath = Paths.get("my-file.txt");
  Path zipFilePath = zip.getPath("/my-file.txt");          
  Files.copy(externalFilePath, zipFilePath);
}

3. Fork/Join Framework

New fork/join classes were added to Java's Concurrency framework. They make it easier to divide a single task into many smaller tasks so the task can be completed using multi-threading. It uses a worker-stealing technique in which a worker thread will "steal" work from another worker thread if it finishes its work before the other. This helps to reduce the total amount of time it takes for a task to complete. Here's an example of how you would use fork/join to calculate the sum of an array of integers.

import java.util.concurrent.*;

public class ForkJoinDemo{
  public static void main(String args[]){
    ForkJoinPool pool = new ForkJoinPool();
    SumTask task = new SumTask(new int[]{0,1,2,3,4,5});
    Integer result = pool.invoke(task);
    System.out.println(result);
  }

  private static class SumTask extends RecursiveTask<Integer> {
    private final int[] numbers;
    private final int start, end;

    public SumTask(int[] numbers){
      this(numbers, 0, numbers.length - 1);
    }

    public SumTask(int[] numbers, int start, int end){
      this.numbers = numbers;
      this.start = start;
      this.end = end;
    }

    @Override
    public Integer compute(){
      if (start == end){
        return numbers[start];
      }
      if (end-start == 1){
        return numbers[start] + numbers[end];
      }
      int mid = (end+start)/2;
      SumTask t1 = new SumTask(numbers, start, mid);
      t1.fork();
      SumTask t2 = new SumTask(numbers, mid+1, end);
      return t2.compute() + t1.join();
    }
  } 
}

First, a ForkJoinPool object is created. Then, a task (SumTask) is instantiated and passed to the ForkJoinPool.invoke() method, which blocks until the task is complete. By default, ForkJoinPool will create one thread for each core in your computer.

4. Misc

Throwable.getSuppressed()

Java 7 adds a method called getSuppressed() to the Throwable class. This method allows the programmer to get any exceptions that were suppressed by the thrown exception.

An exception can be suppressed in a try-finally block or in the new try-with-resources block. In a try-finally block, if an exception is thrown from both the try and finally blocks, the exception thrown from the try block will be suppressed and the exception thrown from the finally block will be returned.

BufferedReader reader = null;
try{
  reader = ...;
  reader.readLine(); //IOException thrown (**suppressed**)
} finally {
  reader.close(); //IOException thrown
}

In Java 7's new try-with-resources block, it is the opposite--the exception thrown when it closes resource is suppressed, while the exception thrown from the try block is returned:

try (BufferedReader reader = ...){
  reader.readLine(); //IOException thrown
}
//IOException thrown when "reader" is closed (**suppressed**)

In previous Java versions, there was no way to get this suppressed exception. But in Java 7, calling the new Throwable.getSuppressed() method will return an array of all suppressed exceptions.

Shaped and Translucent Windows in Swing

You can now create non-square, transparent windows in Swing. Examples can be found in the Java Swing tutorial.


These are just some of the changes that were added to Java 7. Out of all the changes I've mentioned, my favorite is the file monitoring functionality in the NIO 2.0 framework. This is completely new functionality that didn't exist before in previous Java versions. What's your favorite?? Leave a note in the comments below.

The complete Java 7 release notes can be found on Oracle's website.

Saturday, June 9, 2012

HTTP 304

The Bing homepage is a lot prettier looking than Google's. It has a slick, high-resolution background image that changes every week or so. But an image like this takes time to download and it increases the load time of the page. What if you're doing a research project and you have to visit Bing several times a day? Does your browser have to download this image over and over again?

The answer is no. When your browser downloads an image for the first time, it saves it to a cache on the hard drive. A If-Modified-Since header is then added to all subsequent requests for the image and it contains the time that the browser last downloaded the image. The server looks at this time and compares it with the time that the image was last modified on the server. If the image hasn't been modified since then, it returns a HTTP 304 response with an empty body (the image data is left out). The browser sees this status code and knows that it's OK to use the cached version of the image. This means that the image does not have to be downloaded again and the page loads more quickly. If the image has been modified since the browser last downloaded it, then a normal HTTP 200 response is returned containing the image data.


Using the Bing.com background image mentioned above as an example, let's try using curl to test this out. First, we'll send a request without the If-Modified-Since header. This should return a normal HTTP 200 response with the image in the response body.

Note: Curl sends the response body to stdout--because we're not interested in the actual image data, we'll just direct it to /dev/null to throw it away. The --verbose argument displays the request and response headers.

curl --verbose "http://www.bing.com/az/hprichbg?p=rb%2fTimothyGrassPollen_EN-US8441009544_1366x768.jpg" > /dev/null

Request headers:

GET /az/hprichbg?p=rb%2fTimothyGrassPollen_EN-US8441009544_1366x768.jpg HTTP/1.1
User-Agent: curl/7.21.6 (i686-pc-linux-gnu)
Host: www.bing.com

Response headers:

HTTP/1.1 200 OK
Content-Type: image/jpeg
Last-Modified: Fri, 08 Jun 2012 09:37:14 GMT
Content-Length: 155974

As expected, an HTTP 200 response was returned containing a JPEG image that's about 155KB in size. The Last-Modified header shows the date that the image was last modified on the server.


Now let's try sending a request that will cause a HTTP 304 response to be returned. As shown in the Last-Modified header from the response above, the image was last modifed on the morning of June 8. Let's pretend that we last downloaded the image on June 9. Because the image hasn't changed since we've downloaded it, we know that we have the most recent image, so an HTTP 304 response should be returned.

curl --verbose --header "If-Modified-Since: Sat, 09 Jun 2012 09:37:14 GMT" "http://www.bing.com/az/hprichbg?p=rb%2fTimothyGrassPollen_EN-US8441009544_1366x768.jpg" > /dev/null

Request headers:

GET /az/hprichbg?p=rb%2fTimothyGrassPollen_EN-US8441009544_1366x768.jpg HTTP/1.1
User-Agent: curl/7.21.6 (i686-pc-linux-gnu)
Host: www.bing.com
If-Modified-Since: Sat, 09 Jun 2012 09:37:14 GMT

Response headers:

HTTP/1.1 304 Not Modified
Content-Type: image/jpeg
Last-Modified: Fri, 08 Jun 2012 09:37:14 GMT

An HTTP 304 response was returned with an empty body (as shown by the lack of a Content-Length header) as expected.


Let's play pretend one more time and say that we last downloaded the image on June 7. The image was last updated on June 8, so this means that we have an outdated copy of the image and we need to download a fresh copy.

curl --verbose --header "If-Modified-Since: Thu, 07 Jun 2012 09:37:14 GMT" "http://www.bing.com/az/hprichbg?p=rb%2fTimothyGrassPollen_EN-US8441009544_1366x768.jpg" > /dev/null

Request headers:

GET /az/hprichbg?p=rb%2fTimothyGrassPollen_EN-US8441009544_1366x768.jpg HTTP/1.1
User-Agent: curl/7.21.6 (i686-pc-linux-gnu)
Host: www.bing.com
If-Modified-Since: Thu, 07 Jun 2012 09:37:14 GMT

Response headers:

HTTP/1.1 200 OK
Content-Type: image/jpeg
Last-Modified: Fri, 08 Jun 2012 09:37:14 GMT
Content-Length: 155974

As shown in the response, the server detected the fact that our copy was out of date and sent us a HTTP 200 response with the image data in it.


So as you can see, without this caching mechanism, the web would be much slower. Your browser would have to download everything from scratch every time a page is loaded. But with caching, your browser can pull images from the cache without having to download them again.

Saturday, May 26, 2012

Summary of "Your Mouse is a Database" - May 2012 CACM

In a typical web application, when you make an Ajax call to a web service, you have to wait for the entire response to be received before you can act on it. You cannot process the data as it's coming in off the wire. The article "Your Mouse is a Databse" in the May 2012 issue of CACM by Erik Meijer explores a programming API called Reactive Extensions that can asynchronously stream large or infinite amounts of data to an application (think of a stream of Twitter tweets--new tweets are always being created, so there's never an end to the data). Two key words here are "asynchronous" and "stream". Asynchronous means that the data is handled in a separate thread so the application's UI does not lock up. And stream means that the data is acted on as it's being received.

The term "data" is used very broadly--it doesn't have to be data from a web service over the Internet. The data can also be UI input from the user such mouse movements or typing characters into a text box. The data is push-based because it is sent to the consumer instead of the consumer having to explicitly request it.

To help explain the concept, Meijer proposes modifying Java's Future interface (an interface used to check on the status of threads that are running in the background) to handle such data.

interface Future<T> {
  Closable get(Observer<T> callback);
}

He slims the Future<T> interface down to just one method. The get method allows a consumer to subscribe to the data stream. The return value for the method is an instance of Closeable, which allows the programmer to cancel that particular subscription if she wishes.

The Observer<T> parameter processes data from the stream asynchronously. Meijer bases the Observer<T> interface on GWT's AsyncCallback interface by giving it onFailure() and onSuccess() methods. onFailure() is called if there's an error retrieving data from the stream. onSuccess() is called when (or if) the stream ends. The third method, onNext() defines how to handle each data item from the stream (like saving it to a database or displaying it on the screen).

interface Observer<T> {
  void onFailure(Throwable t);
  void onSuccess();
  void onNext(T value);
}

Meijer then describes a number of query opeators in the Reactive Extensions library that can be used to filter this streaming data. For instance, the "where" operator lets the programmer specify the criteria for whether or not a data item should be processed. If the data item does not meet the criteria, then it is discarded. Another operator is "throttle", which prevents too much data from being processed too quickly. For example, if the throttle value is set to 2 seconds and 10 data items are pushed within a 2 second time span, it will only process the most recent message within that 2 second time span. The other 9 messages will be ignored. These operators can be chained together to give the programmer strong control over how to filter a stream.

This idea of streaming push-based based data can help developers design more memory-efficient applications. Tt can also help developers filter the data from larger streams, like Twitter for example, so as to not overwhelm the application with data it doesn't need.

Sunday, May 20, 2012

Summary of "Idempotence Is Not a Medical Condition" - CACM, May 2012

I wanted to read the article "Idempotence Is Not a Medical Condition" by Pat Helland in the May 2012 issue of CACM because it smelled like an article that was full of big words and fuzzy architecture abstractions. I learned some things from the article (and I really like the title), but it's mostly full of FUD. The jist of the article is: "Are you SURE that the messages you send over the network are delivered successfully? Are you REALLY sure? I mean, are you SUPER DUPER sure? After all, how can we be certain of anything in this crazy world?" By the way, SQL Server Broker eliminates this uncertainty for you.


In this article, Helland says that distributed systems today are largely composed of a collection of off-the-shelf software and cloud-based Internet services. This means that they lack a centrally-enforced communication policy. So, extra care must be taken to ensure that each message is delivered and that no message is lost, even if one of the sub-systems goes down or restarts.

This contrasts with the traditional notion of a distributed system: a cluster of tightly-coupled computers in a lab somewhere, which are specifically programmed to talk only to each other. Communication is simpler in this situation because each sub-system is essentially identical and also physically present in the same location.

Every sub-system has some sort of "plumbing" which sends and receives messages (e.g. a TCP network stack). If the plumbing is able to do things like handle duplicate messages and resend lost messages, then the system's application layer does not have to be programmed to ensure reliable communication. But if the plumbing cannot do these things, then the application layer is responsible.

A message should only be marked as "consumed" if the action that the receiving application performs on the message is successful. For example, if an application tries to save a message it receives to a database, but the database transaction fails, then the message should not be considered "consumed" because the operation that the message invoked (a database call) was not successful. The message should sit in a queue somewhere until the application is able to successfully process it or it should respond to the client saying that it failed the operation.

Helland also states that messages should be idempotent, meaning that if the same message is sent multiple times, the effect it has on the server should be the same as if just one message was sent. The term is often used in the context of HTTP--GET requests are defined as being idempotent, while POST requests are not.

Messages can have a preference for one of two types of behavior: "history" or "currency". History means that the messages must be delivered in order. If message N+1 arrives, but message N has not yet arrived, then it must wait until it receives message N in order to deliver message N+1 to the application. Downloading a file requires this type of communication because all of the file's data must be delivered in order or else you'll get a corrupt file. Currency means that getting the most recent message is what matters. If some messages are skipped or lost, that's OK. Getting the most recent price of a stock is one such example of this behavior type.

Helland says that the greatest moment of uncertainty during the request/response cycle is right after the request is sent and the application is waiting for a response. The assumption is that the message has been received and is being processed, but you don't know this for sure. For all you know, the receiving end could have decided to ignore the request and not send a response. It's for this reason that timeouts are needed. If the requester has not received a response within the timeout period, then it will assume the request was not processed and will stop waiting for a response.

However, there is one thing that you can know for certain, Helland says. "Each message is guaranteed to be delivered zero or more times!". Finally, something I can rely on!

Helland says that despite TCP's robustness and wide-spread use, it cannot fully be trusted to deliver messages reliably. He says that TCP "offers no guarantees once the connection is terminated or one of the processes completes or fails." I don't understand what he's trying to say. Of course it offers no guarantees once the connection is terminated. You can't send any messages over a terminated connection. And of course it offers no guarantees once one of the processes completes or fails. At that point, the TCP conversation is over.

He then goes on to say that "challenges arise when longer-lived participants [such as HTTP requests] are involved." He says that when a persistent HTTP connection is needed, the TCP connection is usually kept alive, but there's no guarantee that this will happen. Because of this, the HTTP request may have to be sent multiple times.

Helland then goes into an in-depth discussion about idempotency. He says that, technically, no idempotent request is truly idempotent because every request has some lasting effect on the server. For instance, most servers keep an access log of every request that was received. Making five identical requests, even if they are idempotent, will add five entries to the log file. Also, the performance of the server is impacted every time it receives a request. The more requests it receives, the more its performance will degrade. However, these side-effects are not related to how the actual application logic behaves, which is the true context in which the term "idempotent" should be used.

Helland states that state-ful communication is more difficult than state-less communication because all previously sent messages must be taken into consideration when processing the current message.

He also points out that there's no way of knowing whether the server that receives a request is doing the actual work to fulfill that request. The server could be forwarding the request to another server to do the actual work.

Helland says that if a dialog between two servers breaks apart in the middle of the conversation, both ends must be able to cleanly recover from this failure.

If a service is load-balanced across multiple servers, then state-ful information must be stored in such a way so that a client's state is not lost. One way to do this is to store the state information on a designated server that the other servers have access to. That way, requests from the same client can be handled by any server in the cluster.

Alternatively, the client could be assigned to a designated server in the cluster by a load-balancer when it makes its first request. This server will then be responsible for maintaining the state information for this client and handling all of the client's requests. The first request that the client makes must be idempotent because if the request is received successfully, but the response from the server is lost, then the client will assume that the request was lost and try making the request again. When the request is sent for the second time, the load-balancer may assign the request to a different server. If the request is not idempotent, then the request will be applied twice, thus tainting the server's data. Once the client receives a server response to its first request, it now knows which server in the cluster it should communicate with, which means that all subsequent messages don't have to be idempotent because there's no risk of sending the same request to multiple servers.

Helland says there are three ways to make this first client request idempotent. (1) You can send basically an empty message to the server, such as a TCP SYN message, (2) you can perform a read-only operation, or (3) you have the server queue a non-idempotent operation which will only be executed by the server once the connection has been confirmed. Approach (1) is the simplest, but approaches (2) and (3) can be seen as more efficient because they are performing a useful operation. Or, in Helland's words: "allowing the application to do useful work with the round-trip is cool."

The last point Helland makes is that the last message of a conversation cannot be guaranteed. This is because, if you were to send a response to the last message stating that you have received it, then it wouldn't be the last message! Therefore, applications must be designed so that it is not important whether the last message is received or not.

Friday, May 18, 2012

Database Migration Scripts

I recently added database migration functionality to my Sleet SMTP project. This means that, if I release a new version of the application that includes a change to the database schema, the existing databases of deployed applications will be migrated to the new schema automatically. Before, you would have had to wipe the database completely or apply the schema changes manually, so this is a big improvement.

The way it works is as follows. I created a table in the database whose sole purpose is to store the schema version of the database. This is just an integer that starts at "1" and increments every time the schema changes. The source code also contains a version number, which is the schema version that the source code is programmed to use. When Sleet starts up, it compares the version number in the database with the version number in the source code to determine if the schema is out of date.

If the schema is out of date, it runs a series of migration scripts. Each migration script contains the SQL code necessary to migrate the database from one version to the next. For example, if the latest database schema version is "4", then the application will contain three migration scripts: 1-to-2, 2-to-3, and 3-to-4. By chaining these scripts together, the database schema can be updated no matter what version it currently is. For example, if the schema version of my database is "2", it will first execute the 2-to-3 migration script and then execute the 3-to-4 migration script. If it's "3", then it will just execute the 3-to-4 script. If it's "1", then it will execute all of them. All of this is done within a database transaction, so if something goes wrong during the migration process, the database will be restored to its previous state.

The psuedo-code below shows how this is done in code.

//connect to the database
Connection db = ...
db.setAutoCommit(false);

int schemaVersion = 4;
int curSchemaVersion = //"SELECT db_schema_version FROM sleet"
if (curSchemaVersion < schemaVersion) {
  //schema is outdated, run the migration script(s)
  Statment statement = db.createStatement();
  while (curSchemaVersion < schemaVersion) {
    String script = "migrate-" + curVersion + "-" + (curVersion + 1) + ".sql";
    SQLStatementReader in = new SQLStatementReader(new InputStreamReader(getClass().getResourceAsStream(script)));
    String sql;
    while ((sql = in.readStatement()) != null) {
      statement.execute(sql);
    }
    curSchemaVersion++;
  }

  //update the version number in the database
  //"UPDATE sleet SET db_schema_version = [schemaVersion]"

  //commit the transaction
  db.commit();
}