Do we need frameworks for REST?

Many REST frameworks such as Restlets and simple web are gaining popularity of late. There's also a new JSR to give Java developers a new API to build RESTful web services. It's quite natural that as more people "get" REST they look for ways to simplify building their next REST web application / service.

And while I admittedly have to yet fully "get" REST, whatever I have got so far is certainly not by using frameworks that let me implement REST by writing POJOs but rather by extensive use of the POW (Plain Old Web) and POGS (Plain Old Google Search). And applying the principles I learned by using POS (Plain Old Servlets).

This is not a criticism of these frameworks. May be I still haven't understood the role of these frameworks. And may be once I do understand their goodness I myself will start using them.

But my question is this - why are servlets not good enough? Sure they have their limitations. But rather than have yet another framework or a brand new API, why not have a JSR to fix the servlets and JSPs themselves? (A good start would be to not enable sessions for JSPs by default.) After all isn't it the convenience of using frameworks galore that has kept the larger community from understanding the goodness of HTTP? Isn't it the same convenience that has made it a common practice to use (bloated) sessions?

You have to wet your feet to tread the waters. You have to get your hands dirty in HTTP to implement REST.

Legacy operations vs. REST resources

Stefan Tilkov and Sean Gillies respond to my previous post about modelling operations in REST. There's also an interesting discussion on this on the rest-discuss yahoo group.

First of all, I think my example wasn't a good one. While the operations I had in mind were operations alright, they weren't state changing ones. I realize that my example definitely implies changing state. And I myself would advocate a PUT or at worst a POST in that case. The case I am making is for operations that don't change state but are like queries on a given resource.

I agree with both Stefan and Sean. "Resources, not Objects" as Sean puts it. In the REST world, a person does not walk but s/he reaches a location. And if I were designing a REST API for a new system I would most certainly use that approach.

But if I have existing APIs or SOAP web services such that the verbs talk and walk were firmly instilled in the verbiage of my user community, it might be a difficult proposition for me to suddenly introduce a new vocabulary for the same set of operations to my users. The user community sees them as operations and not in terms of the resulting resources (words and location). Legacy wins over technical correctness.

RESTful URLs for non-CRUD operations

A common way of designing a REST system is to recognize the resources and associate a logical hierarchy of URLs to them and perform the traditional CRUD operations by using the well-known HTTP methods (GET, POST, PUT and DELETE).

However, many systems have operations which don't quite fit into the CRUD paradigm. Say, I have a repository of persons. And say I have an id associated with every person which allows me to have nice URLs to each person as such:

http://example.com/persons/1

This URL obviously GETs me the details pertaining to person 1. Now what if I wanted this person to talk and walk. I can think of 3 approaches to designing this scenario:

1. Operations as resources: Which means I can have URLs such as:
   

   http://example.com/persons/1/talk
   http://example.com/persons/1/walk

   However, walk and talk are obviously not resources and designing it this way might be considered unRESTful(?).



2. Operation as a query parameter: This leads to URLs such as:
   

   http://example.com/persons/1?operation=talk
   http://example.com/persons/1?operation=walk

   A drawback of this approach comes to the fore if you need other parameters to perform an operation. So, for instance, if you had to specify what to talk or where to walk. You'll end up with URLs such as these:
   

   http://example.com/persons/1?operation=talk&what=Hello
   http://example.com/persons/1?operation=walk&where=North

   As you can see, with this approach you end up having resources with overloaded operations and parameters. And you have to be aware of these combinations yourself and also explain them to your users.



3. Matrix URIs: Specify the operations using matrix-like URIs. With this, your URLs look as such:
   

   http://example.com/persons/1;talk
   http://example.com/persons/1;walk

   And you can specify the operation parameters using the traditional query string:
   

   http://example.com/persons/1;talk?what=Hello
   http://example.com/persons/1;walk?where=North

   With this approach you have 3 distinct ways of representing 3 different things - slashes (/) for hierarchical resources, semi-colons(;) for operations and query strings (?param1=val1&param2=val2) for parameters.

Although I like the clarity of #3, I haven't seen this approach used all that much. Which makes me reluctant using it myself. Are there any web systems out there that use this approach? Are there any drawbacks to this approach which is why it is not widely employed? Are there any other approaches to designing URLs for operations?

Many questions. Any answers?

SOAP over DCOM in the ArcGIS ADF explained

The ArcGIS Server Java ADF supports accessing the ArcGIS Server over the internet (http) as well as locally (dcom). Internet access uses SOAP. Local access can work with the server objects directly over DCOM or you can also issue SOAP calls over DCOM.

For local access, the ADF gives preference to SOAP over DCOM while it accesses the ArcObjects directly over DCOM only when the functionality is not available thru SOAP. There are 2 primary reasons why SOAP / DCOM is preferred to ArcObjects / DCOM:

• Performance
• Code reuse

Performance:

When you work with ArcObjects / DCOM the server gives you access to remote ArcObjects proxies (such as IMapDescription, ILayerDescription, etc.). Every method call on these proxies is a remote method call. So methods such IMapDescription.getName() and ILayerDescription.getID() are both remote method calls.

On the other hand, when you work with SOAP / DCOM, only the methods defined in the WSDLs are remote calls. Once you have made those remote calls, you get access to objects which are local value objects (such as MapDescription, LayerDescription, etc.). So methods such as MapDescription.getName() and ILayerDescription.getLayerID() are both local method calls.

As you can infer, ArcObjects / DCOM elicits more "chattiness" with the server than SOAP / DCOM. And the reduced number of remote calls in case of SOAP / DCOM obviously translates to better performance over the lifetime of your application.

Code reuse:

If you look at various functionalities supported by the ADF such as AGSMapFunctionality, AGSGeocodeFunctionality, etc., the same functionality classes are used for both internet and local access. This was possible because these functionalities were implemented by using the SOAP interface to the server. The transport is HTTP in case of internet access and DCOM in case of local access but the code remains the same allowing us to reuse the same functionality implementation in both cases.

Capabilities such as the EditingTask which are not available with SOAP have obviously been implemented by using ArcObjects / DCOM.

Bottom line:

In summary, if your functionality can be implemented by using the SOAP interface you should use it. The richness of ArcObjects / DCOM is of course always available to you in cases where SOAP does not suffice.

What is REST?

There is a wealth of material out there on REST but very few that actually explain them succinctly enough for you to, well, pitch them to your manager in the elevator. Looks like someone has tried to do that and done a very good job at it:

Link: REST: the quick pitch

With REST, every piece of information has its own URL.

I'll use some of David's material myself and highlight the key REST concepts as bullet points:

• [Of course] Everything is a URL: And what does that mean? Immediately all your information is readily accessible to everyone. It is cache-able, bookmark-able, search-able, link-able - basically it's intrinsically web enabled.


• Think resources: With REST it helps if you design your system as a repository of resources. Not services. Not as a data provider - but resources.


• URLs matter: You might argue that if it's machines that are calling into my REST resources, how does the niceness of URLs matter? Well, given that URLs are representations of resources and representations can be human readable text formats or browser readable html; your REST URLs are no longer just a privilege of machines. So URLs matter. Avoid query parameters as much as possible. You have a better chance of being indexed by search engines if you avoid 'em. Your implementation becomes easier. Refactoring is smoother.


• POSTs are ok: In the ideal world all HTTP clients and servers would allow PUT and DELETE. But the world doesn't come to a standstill without these methods. Many have done just fine using POST and so would you.


• Requesting content type in URLs is also ok: Again, in the ideal world, clients and servers could do content negotiation. And again, many have done just fine by specifying the format in the URL path or as a query parameter and so would you.


• Consider JSON: JSON is simple. Parsing JSON is simpler. You don't even need to parse it if you are consuming it in a browser. You still want to serve XML given the huge support for it but JSON support is spreading every day and you'll benefit if you're a part of it.


• Use HTTP as a platform: HTTP is not just a protocol. It's a platform. It already provides services such as caching, security (of course more could be done here), standardized status codes - benefit from them.

Is that all to it? Hardly. There's literally a whole science behind it. But that will do for now.

Enough already... I need some REST

Like many others, inspired by Pete Lacey's S Stands for Simple, late last year I began to look into REST and by extension into HTTP, status codes, web caching, et al. In a nutshell I went back to the basics and discovered the wealth that I had turned a blind eye to what with the latest and greatest frameworks "abstracting out" what constitutes the web from me.

Suddenly the stateless nature of HTTP transformed from being a limitation to a virtue. The status codes weren't just numbers but a means of communication (in some cases even the lack of it - 304, anyone?). Caching wasn't something I needed to build but something I needed to learn how to use (ETag, If-None-Match, Cache control headers, what not). Ditto with security. URLs ceased being just names - they are a language.

It took all of that for me to realize that it's not the next WS* standard that will help me develop the next state-of-the-art web service but it's the existing goodness in HTTP, it's what makes the web work today, it's what brought you to this page and what enabled me to publish this page to the world.

Having relatively recently discovered REST I find it simple and natural. Simple is good. Natural is good. It uses the existing web / HTTP infrastructure not merely as a protocol but as a platform. And it fits into this Web 2.0 thingy to a tee: Issue AJAX requests(Actually it's more like AJAX without the X). Receive JSON responses. There's your secret sauce to building mashups.

This is not to say that I suddenly shunt everything that is SOAP and just do REST all the way. Far from it. SOAP has served me very well and I like it and I'll continue to use it. Something that lets me use pure Java / .NET while working with a piece of software half a world away from me is too precious to be ignored.

I believe that SOAP and REST are not contradictory but complementary. They have their own usages and users and they will coexist. And I'll continue to use them both as per my application needs. Horses for courses

I rest my case.

LinkedHashMap to implement LRU caches

I must be living under a rock to have not noticed this before. The LinkedHashMap has a 3 argument constructor. The last argument, if true, orders the map according to the access order. It also defines a protected removeEldestEntry() method which returns false by default. One can override this method to return true in certain situations, in which case the map will remove its eldest entry.

The implementation of an LRUMap will look something like this:

public class LRUMap<K,V> extends LinkedHashMap<K,V> {

 int maxLimit;

 public LRUMap(int maxLimit) {
   super(maxLimit * 10/7 + 1, 0.7f, true);
   this.maxLimit = maxLimit;
 }

 @Override
 protected boolean removeEldestEntry(Map.Entry<K,V> eldest) {
   return size() > maxLimit;
 }

}

The removeEldestEntry() method is invoked by the put*() methods. With the above implementation, if the size of the map exceeds the max limit, the eldest entry is removed.

Short and sweet. Albeit some 2 years too late for my unenlightened self.

ArcGIS Server coding practices

The ESRI dev summit was a great experience. Meeting fellow developers is always good - you talk the same language, exchange ideas, receive feedback, discuss how things can be made better,... I could go on and on.

Ok, so I was asked quite a few server related questions at the summit: What should I keep in mind while coding against the server? Any do's and dont's? Any particular classes / methods I should read more about? etc... So here I'll talk about certain points you should keep in mind while working with the ArcGIS Server, particularly the MapServer:

• Use the description objects: Sure the MapServer gives you access to the IMap but you don't have to go there. A lot can be accomplished by using the IMapDescription, ILayerDescription, et al.


• Release the ServerContext: In a pooled environment you are sharing the server object with other users. So it's your responsibility to release the context once you have performed your set of operations. For web applications, this translates to releasing the context after every request. Of course, if you are using the ADF, the ADF does that for you so you need not worry. But even then it's good to keep this in mind.


• Do not reference server objects after the context has been released: Do you continue to work with the Statement object once the JDBC Connection has been closed? NO. Why? Because a Statement can be executed only while the Connection is live. Similarly, you should not reference ANY server objects once you have released the IServerContext. If you want to persist the state of any server object, you should use the saveObject() method to get a serialized string representation of the object before releasing the context. You can once again rehydrate the object by using the loadObject() method once you have regained access to the context.


• In case you had forgotten - use the description objects: Did you know you could add serializable custom graphics to the IMapDescription object?


• Use methods exposed by the MapServer: Look at the javadoc for the MapServer and for all the interfaces that it implements. It can do a lot more than you might think - you can export maps, layouts, do queries, identifies, get feature info, handle SOAP requests, and a whole lot more.


• Create server objects on the server: You don't create RMI objects on the client. You don't create EJBs on the client. Similarly, you don't create ArcGIS server objects on the client. ALWAYS use the IServerContext.createObject() method to create all and any ArcObject in a server application within your server context at the server.


• BTW, use the description objects: Did you know you could change layer visibilities, select features, even set definition expressions with the ILayerDescription object?

This by no means is an exhaustive list but something which might help you when working with the server. Oh yeah, in parting, in case you had missed: use the description objects!

Using the context control in a multiple page webapp

There have been many forum questions about the correct usage of the context control in a multiple page web application. I will be the first one to admit that this is more of a bug in the ADF than an incorrect usage on the part of the user. Ok, so the short end of it is this question: How do I access the same context across multiple pages of an ADF web application?

Typically, the context is specified on a JSP using the context tag:

<ags:context id="myCtx" resource="myServerObject@myHost"/>

Now, in an ideal world, if you have multiple pages in your webapp, you should have to use the context control in the exact same manner on all other pages and it should work fine. Unfortunately, that's not the case at 9.1. Currently, the way the context tag works is that if you specify the resource attribute of the tag, the first time that the page is accessed, it actually creates a new context. What this means is that if you use the context tag by specifying the resource attribute on different pages of the same webapp, it gives an impression that the context is "resetting" itself - i.e. the current state of your application is lost and the application reverts to the original state of the map server object.

A workaround to this problem is that on subsequent pages of your webapp, you should not specify the resource attribute of the context tag and ensure that the id is same as the id that you specified on the main page:

<ags:context id="myCtx"/>
<!-- resource attribute is not used and id is same as the id of the context on the main page -->

If the resource attribute is not specified, the context tag does not create a new context but instead tries to find an existing context with the same id as specified in the id attribute. So as long as you have the correct id specified, the context that you created on the main page will be accessed on all subsequent pages of the webapp.

This is admittedly confusing and we have addressed it at 9.2 so that the context control can be used in a consistent manner across all pages of the webapp.

Exception handling in web frameworks

There are many good articles out there pertaining to good exception handling practices. Here I'll list some that I've learned over the course of working on web applications and as it pertains to web frameworks.

1. Chain Exceptions:
   Sure one should never lose sight of the destination, but with exceptions one should never lose sight of the source - the all important root cause, the crux of the problem. While you can catch exceptions and throw custom exceptions, never lose the exception that you caught. When creating custom exception classes, always include a constructor that takes the cause as an argument:
   

   public CustomException(String message, Throwable cause) {
     super(message, cause);
   }

   
   Or if you are using an exception which does not have such a constructor, you can always use the initCause() method:
   

   catch(AnException e) {
     IllegalStateException ise = new IllegalStateException("Something went wrong.");
     ise.initCause(e);
     throw ise;
   }

   
   


2. Throw framework specific exceptions:
   The root cause tells you what went wrong but you also want to know where it went wrong. This is especially critical in web frameworks where the framework makes callbacks, does dependency injection and other IoC stuff to call into app specific custom client code. When your client comes to you with a support call or posts the problem on the forums the first thing you want to know is where in your stack did it fail. Debugging and fixing the problem becomes a lot easier once you have narrowed it down to your courtyard. Framework exceptions are best implemented as a hierarchy of custom runtime exceptions. The root of this hierarchy could be a RuntimeException named MyFrameworkException and it could have any number of sub-classes such as MyFrameworkModelException, MyFrameworkControllerException, MyFrameworkViewException, etc. depending on how you have categorized different pieces of your framework.


3. Log, log, log:
   Where exception messages don’t tell much, debug messages logged from important places will. What comments are to source code, log messages are to runtime. The standard JDK logging API allows you to log messages at multiple levels including INFO, CONFIG, FINE, FINER, et al. Logging messages at various levels will not only provide answers to the hows, whens and wheres of the problem but will also go a long way in helping your customers understand the underpinnings of your framework. And more often than not that is all they need to fix the problem themselves!

We are constantly trying to improve the exception handling and logging in the ArcGIS ADF and you can be assured that we are employing these principles as well.