Web Scraping with node.js and Cheerio

I recently gave a talk at the BayNode Meetup, about my experiences web scraping for dog agility trials using node.js and the cheerio module.  The results are used for my website, www.nextq.info.

You can find the slides as Google Docs here:  Web scraping with cheerio.  Enjoy!

Groovy-Like XML for Java. Simple and Sane.

Parsing and navigating through XML in Java is a pain.  The org.w3c.dom.* classes are numerous, messy, and "old style", with no Collections, no Generics, no varargs.  XPath helps a lot with the navigation part, but is still a bit complex and messy.

Groovy, with XMLParser and XMLSlurper and their associated classes, makes this amazingly, dramatically easier.  Simple and Sane.  For example, Making Java Groovy Chapter 2 has an example to parse the Google geocoder XML data to retrieve latitude and longitude.  Below is the essentials of the code.  The full code, which is not much longer, is on GitHub here.

String url = 'http://maps.google.com/maps/api/geocode/xml?' + somemore...
def response = new XmlSlurper().parse(url)
stadium.latitude = response.result[0].geometry.location.lat.toDouble()
stadium.longitude = response.result[0].geometry.location.lng.toDouble()

The parsing is trivial, and navigating to the data (location.lat or location.lng) is also simple, following the familiar dot notation.

Can you do something anything like this in pure Java?  Not quite.  So I wrote a small library, xen, to mimic much of how Groovy does things.  The full Geocoder.java code is here, snippet below:

String url = BASE + URLEncoder.encode(address);
Xen response = new XenParser().parse(url);

Option 1: XPath slash style, 1 based indices
latLng[0] = response.toDouble("result[1]/geometry/location/lat");
latLng[1] = response.one("result[1]/geometry/location/lng").toDouble();

Option 2: Groovy dot style, 0 based indices
latLng[0] = response.toDouble(".result[0].geometry.location.lat");
latLng[1] = response.one(".result[0].geometry.location.lng").toDouble();

Pretty close, eh?

The main difference is that we can't use the dot notation directly from an object, but we can use a very similar slash notation based upon XPath syntax. If you use XPath notation, one major difference from Groovy is that array indices in W3C XPath are 1-based, not 0-based.  Therefore note that we access the 1st element of result, not the 0th.  However, if the "path" starts with a . and a letter, as in the final example, the path is treated as a Groovy / "dot notation" style, with 0-based indices.

So, if you want to greatly simplify parsing and navigating through XML, and/or you love how Groovy does things, please check out my (very beta!) xen library which allows you to do it in Java.  Currently it is compiled vs. Java 6 but I think it should be fine in Java 5.  So if you need to support some Android device, or can't or don't want to integrate Groovy into your Java projects, this could be very useful.

Xen library
JavaDocs
README

The README discusses various design decisions, particularly, how my design converged upon many aspects of the Groovy design.   More discussion will appear in later posts.  And, be warned, this is still a very early version, 0.0.2, so there are probably bugs, some mistakes, and upcoming API changes.

Node for Java Programmers

At a recent BayNode Meetup, I gave a 15 minute presentation on "Node for Java Programmers".  Mainly notes on common things I did wrong coming from the Java world, and ideas or idioms to deal with them.

I got some good feedback and positive responses, and recently edited the presentation.

Here is a link to it.   (on Google Docs).

Coding by Convention is Great

... except when it isn't.

"Coding by Convention" (a.k.a. "Convention over Configuration") attempts to simplify programming by telling the programmer the preferred way to name or organize things.  It often saves a lot of time and hassle.  Without it, you write extra configuration files, typically in XML.  Spring and J2EE used to require way too much configuration, with lots of stupid redundancies, something like "When I say Foo bean I mean you to use a Foo.class, when they go to myCompany.com/order/books use the com.myCompany.order.books servlet".  On the other hand there can be too much convention - I've never used Maven, but hear that it is particularly dictatorial and hard to modify.

I'm developing a lot in JavaScript / node.js lately, and wanted the ability to save my data as either csv files or iCal (.ics) files.  Searching the NPM registry finds several candidates.

In csv files, the header line contains the name for each column.  If using convention, this would be taken from the property key.  And the property would map directly to the data in the following lines.  Sometimes you would want to change this.  Can you, or is it all done by convention?  As I understand their documentation:

to-csv      convention only
fast-csv    allows for transformation, but over an entire row
json-2-csv  convention only
json-csv    allows for flexible transformations

I ended up using json-csv, though one drawback of it's power is that it takes more work to use.

On the iCalendar side, the question is how to setup or create the complex VEVENT information.  One could use properties named DTSTART, UID, etc.  But it's extremely unlikely that your object has properties with those unusual and capitalized names, and the correct values.  Plus, DTSTART has a complex format with a possible "DATE-TIME" option and a time format of YYYYMMDDTHHMMSSZ.

cozy-iCal       builds VEvent objects programmatically
ical-generator  convention (uses .start .end for DTSTART DTEND)
icalevent       convention (also uses .start, .end etc)
icsjs           builds programmatically
icalendar       builds VEvent objects programmatically

So, it's a mishmash. And convention, while simple and convenient, doesn't always do the trick.  For example, in my CSV data I'd like to include the distance from the user's location.  This is obviously not even a field in the data, since it is calculated on the fly per user from the respective latitudes and longitudes.  The CSV should not include the latitude and longitude - not very useful to the end user.  My start and end dates are also not fields, they are stored in an array.  So I definitely can't use convention.

Unless...

The obvious work-around is to create a new temporary object, that meets the required convention, from the fields and data in the original, "real" object.  In many cases, you would just wrote custom code to do this, especially if speed is a concern.  There are also some modules that vaguely do this.  (Did I miss some???)  But they are pretty limited.   For example object-adapter can only copy values from a source object (renaming the fields), not apply any functions.

So, I wrote my own general-purpose module, remodeler.  For convenience there are copyKeys() and excludeKeys() methods for properties you simply want to copy as-is or ignore.  For the fancy stuff, you provide key / value pairs.  The key is the new property name, and the value is a "transformation".

If the transformation is a String, it means to copy the value from the old object, using the string as property name. For example, "UID", "uid" would mean to create a UID property by copying the previous uid property.

If the transformation is a function, it will be called via function(oldObject, key) and the result used as the new value.  In practice, the key argument is often ignored.  For example,

"SUMMARY", function(o,k) { return 'name:' + o.name + ' date:' + o.date[0]; }

would mean to create a new SUMMARY property by concatenating two existing values.

In many cases, you are still better off writing your own custom code.  On further thought, I'm not sure how useful my module will be, since in JavaScript, it is just so easy to go

var newthing = {
   newkey: oldObject.oldKey,
   ...
}

Other times you can follow the conventions, or use the programmatic interface.  However, if you want a quick way to "remodel" your domain object, this module might meet your needs.  Let me know what you think.  I think I'm going to try using this with ical-generator.

I'm a Web Magnate

Well, one can dream.  :-)

I've spent the past few months developing a web site / app using a bunch of new (to me) stuff: node.js, Express, mongoDB, along with HTML5, CSS3, and a little JQuery for a cool map using jVectorMap.  All with JavaScript of course. Learned a lot along the way, and now feel somewhat competent with most of the technology.  It's even hooked into Google Maps, Google Analytics, and AddThis for social networking.

Check it out here at nextq.info.  What is it?  It is a search service to find upcoming dog agility trials.  My main hobby.

For hosting, I looked into a few cloud services such as Heroku and Nodejitsu, but decided to use Modulus.io.  So far I've been extremely pleased.

• You get one month free, thereafter the "typical" $15 a month.
• Their dashboard is cool-looking and, more importantly, fairly useful.
• Creating and hooking up a small Mongo Database is simple, even a newbie like me can do it.
• Uploading your code is simple, even a newbie like me can do it.
• Linking their site to my domain name server was fairly straightforward, even a... (you get the idea).
• On a crash it will send you an email (easy to configure) or text, then restarts.

There were a couple of minor gotchas.  I forgot to route both nextq.info and www.nextq.info.  It took me a little while to learn about the .modulusignore file that tells the upload to ignore unnecessary or test files.  (Though it is smart enough to ignore your node_modules folder.)

So far so good.  I especially enjoyed making up agility-centric custom 404 (and 400 and 500) pages.  I'm finding that jVectorMap is a little flaky on some devices, so there is also a much simpler, somewhat "mobile friendly" search page.  But the main parts seem to work fine and some of my agility friends are already using it.

Yet Another Example Demo Node.js / Express Project

I'm working on an app that finds upcoming events and locations, and wanted a quick link bring up the weather for that city.  Which takes a little work.  What I came up with make a very nice little node.js / express "example" or "demo" program.

Thought I'd try it as a Google Docs presentation.

Source code is on GitHub.

I'll likely add some more commentary here in the near future.

From Node.js back to Java, Part 1: EventEmitter and Callbacks

There were a lot of features of Node.js and JavaScript that I liked.  Why not bring them to Java?  That's what I'm working on with Nava, "Bringing good ideas from node.js into Java".  So far, I have implemented a fair amount.  The code can be compiled against Java 6, Java 7, and Android 2.2, and presumably later versions of Android.

The emit package is a pretty direct port of a node.js EventEmitter to Java, and can be used in a very similar manner as in node.js.  The main difference is that the event handler cannot be a closure, (cause we don't have closures!) it must implement a simple interface, Emit.IListener, with one method, handleEvent().  Producers of events would subclass or delegate to Emitter, much like the node class would extends or delegate to EventEmitter.  This code essentially replaces Java's EventListenerList and associated classes and interfaces.  There are several major advantages:

1. The code does not depend on Swing, so it could be used in Android. 
2. The "events" that get fired need not extend EventObject.  They could be Strings or voids.  However, in many cases, subclassing EventObject makes sense.
3. Generics are used to obviate the need for a lot of boilerplate, such as declaring a bunch of Listener subclasses and their respective delegation methods addFooListener(), removeFooListener(), etc...

There are also a few disadvantages, noted in the documentation. The main one is that everything is more loosely typed, as you'd expect as this is based upon JavaScript.  Underneath, there are unchecked casts.

The callback package is a rough port of JavaScript / node.js style callbacks.  Your callback "closures" must implement the Callback interface, or extend from AbstractCallback.  Callbacks can be "chained together", either manually using setNextCallback(), or programatically with the utility method Callbacks.chainUp().

You could then "submit" the callback directly by calling the first one, but that would happen synchronously, in-line, and, other than for initial testing, probably isn't worth it.  The more "node-like" technique would be to create a CallbackExecutor (probably application-wide) and use submitCallback(), passing the first callback and it's data.  Your thread will then continue, just like in node, with no need to await the completion (good) and no knowledge of the result (sometimes annoying).

For example, if you had a Reader that read a File into a String, and a Counter that counted words, the skeleton Java code would be:

      Reader reader = new Reader();

      Counter counter = new Counter();

      Callbacks.chainUp(reader, counter );

      CallbackExecutor cex = new CallbackExecutor(2,1);

      cex.submitCallback(reader, theFile);

      // your code then moves on (or just stops...)

As opposed to actual JavaScript/Node, which would look something like:

      fs.readFile(theFile, function(err, data) {

         if (err) throw err;

         // normally the counting code would be in-line here

         // but for modularity similar to the Java

         countWords(data);

       });

       // your code then moves on (or just stops...)

There's more, but that's a start for today.  Check out the readme file, javadocs, and JUnit tests for more information.