Sunday, March 14, 2010

Working with Scala and Maven

In order to get Maven to properly recognize Scala code, there are are a number of steps you must take. Included below are pom.xml samples, along with explanations.

1. Name your source directories properly:
<project><build>

<sourceDirectory>src/main/scala</sourceDirectory>
<testSourceDirectory>src/test/scala</testSourceDirectory>

</build></project>

2. Add the scala-tools.org repositories.

These are required in order to download the necessary Scala dependencies (see step #3):
<project>
<repositories>
<repository>
<id>scala-tools.org</id>
<name>Scala-tools Maven2 Repository</name>
<url>http://scala-tools.org/repo-releases</url>
</repository>
</repositories>

<pluginRepositories>
<pluginRepository>
<id>scala-tools.org</id>
<name>Scala-tools Maven2 Repository</name>
<url>http://scala-tools.org/repo-releases</url>
</pluginRepository>
</pluginRepositories>
</project>

3. Add the appropriate dependencies:
<project><dependencies>

<!--
This contains the scala compiler, which Maven will use to compile your code and run your unit tests.
-->
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.7.7</version>
</dependency>

<!--
This is Scala's unit testing framework. It's also possible to use JUnit, but when in Rome, right?
-->
<dependency>
<groupId>org.scalatest</groupId>
<artifactId>scalatest</artifactId>
<version>1.0</version>
<scope>test</scope>
</dependency>

<!--
If you want to use scalatest, unfortunately you also need to include JUnit as a dependency (see step #5).
-->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.8.1</version>
<scope>test</scope>
</dependency>

<dependencies></project>

4. Add the scala-tools plugin:
<project><build><plugins>

<plugin>
<groupId>org.scala-tools</groupId>
<artifactId>maven-scala-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
</plugin>

</plugins></build></projects>

5. Add the @RunWith annotation to each unit test.

If you are using scalatest as your unit testing framework, you must trick Maven into thinking that your tests are JUnit tests. Otherwise, Maven will not run your tests:
import org.junit.runner.RunWith
import org.scalatest.junit.JUnitRunner
import org.scalatest.FunSuite

@RunWith(classOf[JUnitRunner])
class MyScalaTest extends FunSuite{
//...
}

Tip: If you're using Eclipse, you can right click on one of these unit tests and select "Run As > JUnit Test" to manually run the test.

Wednesday, March 10, 2010

XPath and Java

XPath is a domain specific language which is used to extract data from an XML document. It's supported by many different general purpose languages like C#, PHP, and Java. Take the following XML document for example:

<library>
<book>
<language>en</language>
<title>Solaris</title>
<author>Stanislaw Lem</author>
</book>
<book>
<language>fr</language>
<title>Le Petit Prince</title>
<author>Antoine de Saint-Exupéry</author>
</book>
<book>
<language>en</language>
<title>Dune</title>
<author>Frank Herbert</author>
</book>
</library>

The following XPath query retrieves the titles of all English-language books:

/library/book[language='en']/title

Java makes working with XPath (and XML in general) kind of complicated, since many different classes are involved. First, the XML document must be loaded into a DOM (Document Object Model).

StreamSource source = new StreamSource(new File("books.xml"));
DOMResult result = new DOMResult();
Transformer transformer = TransformerFactory.newInstance().newTransformer();
transformer.transform(source, result);
Node documentRoot = result.getNode();

This means that the XML text is read into memory and organized into a tree of nodes where each tag is an element node. The top element node would be the <library> element, which would have three child element nodes (<book>), and so on.



To demonstrate the power of XPath, this is what Java code might look like if XPath did not exist. The programmer would have to manually iterate through the entire DOM to get what she needed.

List<String> englishTitles = new ArrayList<String>();
Node books = documentRoot.getFirstChild();
if (books.getNodeName().equals("library")){
for (int i = 0; i < books.getChildNodes().getLength(); ++i){
Node book = books.getChildNodes().item(i);
if (book.getNodeName().equals("book")){
boolean english = false;
String title = null;
for (int j = 0; j < book.getChildNodes().getLength(); ++j){
Node bookChild = book.getChildNodes().item(j);
if (bookChild.getNodeName().equals("language") &&
bookChild.getTextContent().equals("en")){
english = true;
}
if (bookChild.getNodeName().equals("title")){
title = bookChild.getTextContent();
}
}
if (english && title != null){
englishTitles.add(title);
}
}
}
}
for (String title : englishTitles){
System.out.println(title);
}

As you can see, this is very very tedious and error prone! XPath is the better solution by far:

XPath xpath = XPathFactory.newInstance().newXPath();
NodeList nodeList = (NodeList)xpath.evaluate("/library/book[language='en']/title",
documentRoot, XPathConstants.NODESET);
for (int i = 0; i < nodeList.getLength(); ++i){
Node node = nodeList.item(i);
System.out.println(node.getTextContent());
}

Namespaces

XML documents often use namespaces. These are sort of like Java packages--they group related elements together and prevent name collisions from occurring. Let's say that each <language> element belonged to a namespace:

<book>
<language xmlns="http://translate.google.com">en</language>
<title>Solaris</title>
<author>Stanislaw Lem</author>
</book>

Note: While namespaces technically can be anything (like "abc123" for example), they should be globally unique. There's no way to enforce this, so the convention is to use a URI belonging to the person or company creating the namespace. For example, if Oracle wants to use a namespace, they can be fairly certain that no one else in the entire world is using one starting with "http://www.oracle.com".


To make Java aware of namespaces,a NamespaceContext object must be created and added to the XPath object.

XPath xpath = XPathFactory.newInstance().newXPath();
xpath.setNamespaceContext(new NamespaceContext() {
public String getNamespaceURI(String prefix) {
if ("tr".equals(prefix)){
return "http://translate.google.com";
}
return null;
}
public Iterator getPrefixes(String uri) {
return null;
}
public String getPrefix(String uri) {
return null;
}
});

This will assign the prefix "tr" to the namespace "http://translate.google.com". The prefix can be anything, but the namespace must match the one in the XML document.

NodeList nodeList = (NodeList)xpath.evaluate("/library/book[tr:language='en']/title",
documentRoot, XPathConstants.NODESET);
for (int i = 0; i < nodeList.getLength(); ++i){
Node node = nodeList.item(i);
System.out.println(node.getTextContent());
}

To learn more about XPath, you can visit w3schools.com.

Tuesday, March 9, 2010

Dropbox


I just found out about this cool service called Dropbox, which I thought I'd write about. I read a blurb about it in the March issue of Linux Journal.

Dropbox is a free service that lets you store files online. That sounds pretty boring. But it's more than just a free FTP server. What happens is, you install a small app on your computer, which creates a special "dropbox" folder in your home directory (or in My Documents if you're using Windows). The app monitors this folder for any changes and syncs these changes with the Dropbox server. For example, when you copy a file (Word document, MP3, whatever) to this folder, it will immediately upload it to your Dropbox account. If you delete a file, it will delete it from your account.

But the really cool thing is that you can connect multiple computers to your account by installing the Dropbox application on each one (there are Windows, Mac, Linux, and iPhone versions). So if you add a file to your Dropbox folder on your desktop PC, it will immediately download to your MacBook laptop, Linux Netbook, and whatever else.

Thursday, October 22, 2009

An Amazon Affiliate Tutorial

Ok, so I'm a little behind the times. Amazon's Affiliate Program has been around for a while--since 1996 to be exact. Everyone and their grandma must have an account by now, making millions off sales of "The Secret" that they linked to on their blog (this is actually the secret that the book talks about). But incase you happen to be Amish, here's a short tutorial on the whole process of signing up for an account and building affiliate links.

1. Visit https://affiliate-program.amazon.com/ and click the "Join now for FREE!" button. It requires that you create an Amazon account first if you don't already have one.



2. It then asks you for your contact information and has some questions about the website/blog where you'll be putting your affiliate links.



3. And that's all the information they need to get your account set up. You don't even have to submit your payment information (although, you'll want to do this eventually...after all, making a few bucks is why you're here, right?). As you can see in the screenshot, Amazon assigns you a unique "Associates ID". This important identifier will be stuffed into all the affiliate URLs you create so that Amazon knows who give money to when a sale is made. To start making links, click the "Learn More" link in the "Product Links" box.



4. It's really easy to create affiliate links. It just takes a few clicks of the mouse and some copy and pasting. First, search for the product you want to link to. I chose a book I recently read.



5. Next, find the product in the search results and click the "Get Link" button.



6. This will take you to the page that lets you build your link. There's a lot of options here you can play with, so take some time to explore this page on your own. Once you're done crafting the link to your liking, copy the HTML code in the "step 3" box and paste it on your site. It's no secret--it's that simple!

Tuesday, October 6, 2009

Using Magpie to parse Blogger feeds

I recently redesigned my website and one thing I changed was putting my blog on the front page instead of just linking to it. But what's cool (at least, according to my own geeky tastes) is that my blog is hosted on Blogger--not on my website! In this post, I'm going to walk through how I did this.

1. Blogger setup
First, you need to make sure your Blogger feeds are configured to include the full content of each blog post (as opposed to just the first paragraph or whatever). I think this is the default setting, but to make sure, to go the Settings page and click on the "Site Feed" tab. The "Allow Blog Feeds" option should be set to "Full".



2. Atom feed URL
Now, you must get the URL of your blog's Atom feed. Go to your blog's homepage and view the source (in Firefox, this is under "View > Page Source"). Look for a "link" tag that looks like the one below, and grab the value of its "href" attribute.
<link rel="alternate" type="application/atom+xml" title="mangstacular - Atom" href="http://mangstacular.blogspot.com/feeds/posts/default" />

Note: In this tutorial, I use the Atom feed. The RSS feed also has all the same information--it's just arranged differently.


3. Magpie setup
Download Magpie. This is a wonderfully easy-to-use RSS/Atom parser written in PHP which we'll use to parse the Atom feed.

You'll want to configure Magpie to cache the Atom feed so that it only downloads it when your blog changes in some way (i.e. when there's a new post or a new comment). This way, your website won't have to download the feed from Blogger every time someone visits your page. Open the "rss_fetch.inc" file and add these two lines somewhere at the top:
define('MAGPIE_CACHE_ON', true);
define('MAGPIE_CACHE_DIR', 'cache');

This will turn on caching and instruct Magpie to save the cached Atom file in the directory you specify.

Note: Be sure that the permissions of the cache directory allow your web server to write to it. The way you do this varies from server to server, based on the user that your PHP process runs under, but here are the commands that I had to run:
chmod 775 cache
chgrp web cache

What you do NOT want to do is set the folder to be globally writable. While this would work, it would also allow anyone on the Internet to write to that directory--not a good thing.


4. Parse the Feed
Now you're ready to write the code that fetches and parses the feed. Simply calling Magpie's "fetch_rss" function will parse the feed and return all the data in an associative array (despite "rss" being in the function name, it will also parse Atom feeds). No need to deal with any XML. Below is some sample code.

Note: There are a couple quirks, which may be Blogger-specific--be sure to read the code comments.

require_once('magpierss/rss_fetch.inc');

$atom = fetch_rss('http://mangstacular.blogspot.com/feeds/posts/default');
foreach ($atom->items as $item){
//var_dump($item); //see all the stuff that's in each item

$date = date('F j, Y', strtotime($item['published']));
$title = $item['title'];
$content = $item['atom_content']; //no need to run html_entity_decode() or anything
$url = $item['link'];

//because there are two <link> tags whose "rel" attributes are the same...
//...it stuffs the "href" attributes from both tags into this one string...
//...so you must extract the URL you want...
//...which in my case is the URL to the comments page
$commentsUrl = substr($item['link_replies'], strpos($item['link_replies'], 'https'));

$numberOfComments = $item['thr']['total'];

//generate HTML for the entry
//...
}


And now it looks like you're running fancy Wordpress software! ;)

Tuesday, September 29, 2009

Google Maps and its Traffic overlay

Ever since I started my new job, I've gotten into the habit of turning on the morning news while getting ready for work in the morning. They do reports on rush-hour traffic and this morning, there was one whopper of an accident. A tractor-trailer had slide-lined three cars, completely blocking the flow of traffic in that direction. It was on a road I take everyday to work, but I wasn't sure exactly where on the road it was, so didn't know if it would impact my commute.



Then, I remembered that Google Maps has a live traffic feature that overlays the amount of traffic congestion on the map. I could see where the congestion was and figure out if I needed to find an alternate route. Oddly enough, it reported all roads as being more or less clear, which I knew was not the case. The roadside webcams (another Google Maps overlay) also didn't show any signs of an accident.

It turns out that the data just isn't very "live" after all. Not only was the traffic overlay not reporting the current traffic situation, but the webcam images weren't current either--they were anywhere from 30-60 minutes old. Clicking on the image takes you to the website that operates the webcams, and only going there can you get an up-to-the-minute view of traffic.

So it turned out that my drive wasn't affected. And I learned that, while the traffic overlay on Google Maps is great for seeing general traffic trends, it's not so good for live traffic updates.

Sunday, August 16, 2009

Using phpDocumentor with Eclipse



I just finished making improvements to a PHP/MySQL-powered photo gallery website I had written during college (it kind of felt good to see how much my coding skills have improved since then). Besides moving all the SQL into a DAO, cleaning up the UI, and learning how to do file uploads in PHP, I went through all the code and formally documented all the functions and classes. I decided to use phpDocumentor because the format it requires the comments to be in is nearly identical to Java's Javadoc, which I was already familiar with.

I wanted to set up an Ant target that I could run that would generate the HTML documentation. It was pretty straight-forward--just create a target using the Exec task to run a console command. However, even though I could call the "phpdoc" command straight from the command line, Ant had problems with that. I think this is because "phpdoc" is not an executable, but a PHP script. At any rate, it worked fine once I found the script that the command was linked to and used the "php" command to run the script.

<target name="phpdocs" description="Generates phpDocumentor docs">
<delete dir="${doc.dir}" />
<exec executable="php">
<arg value="/usr/local/PEAR/PhpDocumentor/phpDocumentor/phpdoc.inc" />
<arg value="-f" />
<arg value="*.php" />
<arg value="-t" />
<arg value="${doc.dir}" />
<arg value="-ti" />
<arg value="Photo Gallery Documentation" />
</exec>
</target>