HTTP Content Optimization

You have your business engine all setup. Your processing algorithms have been optimized to the hilt. Your data model is as scalable as any. Any now you are publishing your data to the WWW. Well, the good news is you can still do more - all with plain old HTTP.

Herein I list 3 simple ways you can leverage HTTP to help you better serve your content:

1. Cache-Control headers
2. ETag + If-None-Match
3. gzip

1. Cache-Control headers

Cache-Control is byfar is the most widely used of all http headers and for good reason. You generate your response and you set a Cache-Control response header with a validity period. The clients, the intermediaries and the web infrastructure at large all work overtime for you caching your content until the time that you have tagged it valid.

This of course works best for static resources or for such dynamic resources whose validity you can reasonably predict before hand.

2. ETag + If-None-Match

This is one of the most powerful but unfortunately, a not-frequently used technique. So your content is such that you can't reasonably predict its validity period. Which means #1 doesn't work for you. Your next best buddy is ETags.

This is how it works: You generate your content and you set the http header ETag (entity tag). The ETag represents the state of your resource. Even if one bit of the resource content changes, so does its ETag. You can think of ETag as a simple hash of your content. Ok so you have set the ETag and sent the response. Now the next time the client tries to access the same URL, it will send an If-None-Match request header and it will be set to the same value as that of your ETag. Now you can either regenerate the content or you may have it cached on your server, if the ETag of your content matches the If-None-Match it implies that the content has not changed. The client has indicated to you thru the If-None-Match that it already has this content. So what do you do - send nothing! Yes - simply set the response status to HTTP 304 (Not Modified) and the size of the content that you send this time is exactly 0. At a minimum, you gain in saved bandwidth (read performance) but if you have cached the content on your server, you also gain from saved computation.

3. gzip

With #1 and #2 you benefit by not having to resend your content in certain situations. But even when you can't get away from having to send content, you can still gain in bandwidth by simply compressing it. But of course you want to compress content only for clients that you know can decompress it and even then you need to tell the client that the content you sent is compressed and it needs to decompress it.

No hassle - http makes it fairly straightforward. If the client understands gzip, it sends an Accept-Encoding request header with the string gzip in it. If you (the server) read this header and find the string gzip, you gzip your content and set a response header Content-Encoding to gzip. This tells the client that the content is gzipped and it must decompress it before providing it to the user.

It's normal for gzip to compress text content in upwards of 70% and given how easy it is to compress content you should be compressing your content right about now.

Note that #2 + #3 put together have problems in IE 6 and you might have to take that into account.

But all in all optimizing the delivery of your content with http is simple yet powerful and your web application can only benefit from it.

IE 6: gzip + ETag != If-None-Match

ETag + If-None-Match give you the benefit of not having to send unmodified content repeatedly (HTTP 304). IE6 handles this well.

Content-Encoding with gzip gives you the benefit of compressing the content that you send. IE6 handles this well as well.

So gzip + ETag / If-None-Match should give you the combined benefit of sending compressed content when you must and not sending content at all if it's not modified. Well, as you might have already guessed, IE6 does not handle this well. If your content is gzipped and you send an ETag header as well, IE6 does not send an If-None-Match on subsequent requests. Which of course means that you can't leverage HTTP 304.

So if you are servicing IE6 clients beware that it supports either compression or ETags but not both.

Thankfully, this has been fixed in IE7. Firefox of course just works.

More Resources == Scalable

Link: Sam Ruby on Google Maps

Of course, the rest of the iceberg was that Google had simply tiled the Earth. In so doing, they converted a single web service (call me with a bunch of information, and I will provide you with a custom result) into a large number of individually addressable, cacheable, and scalable web resources.

That's it. More the resources more is the opportunity to use Cache-Control headers, to use ETags, to distribute and load-balance the system.

In the same article, Sam also talks about how the web is not a service but a space. And in today's world adding more "space" will scale your system manifold than implementing a state-of-the-art service with the most optimal algorithm. Processor speeds have flattened. Today it's about dual-cores, quad-cores, (your-budget)-cores. The more cores your program can use to get the job done, the more scalable will your system be.

As Brian Goetz puts it: "Tasks must be amenable to parallelization". Parallelization comes for free with every resource you add. So add more resources and see your system scale.

Do we need frameworks for REST?

Many REST frameworks such as Restlets and simple web are gaining popularity of late. There's also a new JSR to give Java developers a new API to build RESTful web services. It's quite natural that as more people "get" REST they look for ways to simplify building their next REST web application / service.

And while I admittedly have to yet fully "get" REST, whatever I have got so far is certainly not by using frameworks that let me implement REST by writing POJOs but rather by extensive use of the POW (Plain Old Web) and POGS (Plain Old Google Search). And applying the principles I learned by using POS (Plain Old Servlets).

This is not a criticism of these frameworks. May be I still haven't understood the role of these frameworks. And may be once I do understand their goodness I myself will start using them.

But my question is this - why are servlets not good enough? Sure they have their limitations. But rather than have yet another framework or a brand new API, why not have a JSR to fix the servlets and JSPs themselves? (A good start would be to not enable sessions for JSPs by default.) After all isn't it the convenience of using frameworks galore that has kept the larger community from understanding the goodness of HTTP? Isn't it the same convenience that has made it a common practice to use (bloated) sessions?

You have to wet your feet to tread the waters. You have to get your hands dirty in HTTP to implement REST.

What is REST?

There is a wealth of material out there on REST but very few that actually explain them succinctly enough for you to, well, pitch them to your manager in the elevator. Looks like someone has tried to do that and done a very good job at it:

Link: REST: the quick pitch

With REST, every piece of information has its own URL.

I'll use some of David's material myself and highlight the key REST concepts as bullet points:

• [Of course] Everything is a URL: And what does that mean? Immediately all your information is readily accessible to everyone. It is cache-able, bookmark-able, search-able, link-able - basically it's intrinsically web enabled.


• Think resources: With REST it helps if you design your system as a repository of resources. Not services. Not as a data provider - but resources.


• URLs matter: You might argue that if it's machines that are calling into my REST resources, how does the niceness of URLs matter? Well, given that URLs are representations of resources and representations can be human readable text formats or browser readable html; your REST URLs are no longer just a privilege of machines. So URLs matter. Avoid query parameters as much as possible. You have a better chance of being indexed by search engines if you avoid 'em. Your implementation becomes easier. Refactoring is smoother.


• POSTs are ok: In the ideal world all HTTP clients and servers would allow PUT and DELETE. But the world doesn't come to a standstill without these methods. Many have done just fine using POST and so would you.


• Requesting content type in URLs is also ok: Again, in the ideal world, clients and servers could do content negotiation. And again, many have done just fine by specifying the format in the URL path or as a query parameter and so would you.


• Consider JSON: JSON is simple. Parsing JSON is simpler. You don't even need to parse it if you are consuming it in a browser. You still want to serve XML given the huge support for it but JSON support is spreading every day and you'll benefit if you're a part of it.


• Use HTTP as a platform: HTTP is not just a protocol. It's a platform. It already provides services such as caching, security (of course more could be done here), standardized status codes - benefit from them.

Is that all to it? Hardly. There's literally a whole science behind it. But that will do for now.

Enough already... I need some REST

Like many others, inspired by Pete Lacey's S Stands for Simple, late last year I began to look into REST and by extension into HTTP, status codes, web caching, et al. In a nutshell I went back to the basics and discovered the wealth that I had turned a blind eye to what with the latest and greatest frameworks "abstracting out" what constitutes the web from me.

Suddenly the stateless nature of HTTP transformed from being a limitation to a virtue. The status codes weren't just numbers but a means of communication (in some cases even the lack of it - 304, anyone?). Caching wasn't something I needed to build but something I needed to learn how to use (ETag, If-None-Match, Cache control headers, what not). Ditto with security. URLs ceased being just names - they are a language.

It took all of that for me to realize that it's not the next WS* standard that will help me develop the next state-of-the-art web service but it's the existing goodness in HTTP, it's what makes the web work today, it's what brought you to this page and what enabled me to publish this page to the world.

Having relatively recently discovered REST I find it simple and natural. Simple is good. Natural is good. It uses the existing web / HTTP infrastructure not merely as a protocol but as a platform. And it fits into this Web 2.0 thingy to a tee: Issue AJAX requests(Actually it's more like AJAX without the X). Receive JSON responses. There's your secret sauce to building mashups.

This is not to say that I suddenly shunt everything that is SOAP and just do REST all the way. Far from it. SOAP has served me very well and I like it and I'll continue to use it. Something that lets me use pure Java / .NET while working with a piece of software half a world away from me is too precious to be ignored.

I believe that SOAP and REST are not contradictory but complementary. They have their own usages and users and they will coexist. And I'll continue to use them both as per my application needs. Horses for courses

I rest my case.

Exception handling in web frameworks

There are many good articles out there pertaining to good exception handling practices. Here I'll list some that I've learned over the course of working on web applications and as it pertains to web frameworks.

1. Chain Exceptions:
   Sure one should never lose sight of the destination, but with exceptions one should never lose sight of the source - the all important root cause, the crux of the problem. While you can catch exceptions and throw custom exceptions, never lose the exception that you caught. When creating custom exception classes, always include a constructor that takes the cause as an argument:
   

   public CustomException(String message, Throwable cause) {
     super(message, cause);
   }

   
   Or if you are using an exception which does not have such a constructor, you can always use the initCause() method:
   

   catch(AnException e) {
     IllegalStateException ise = new IllegalStateException("Something went wrong.");
     ise.initCause(e);
     throw ise;
   }

   
   


2. Throw framework specific exceptions:
   The root cause tells you what went wrong but you also want to know where it went wrong. This is especially critical in web frameworks where the framework makes callbacks, does dependency injection and other IoC stuff to call into app specific custom client code. When your client comes to you with a support call or posts the problem on the forums the first thing you want to know is where in your stack did it fail. Debugging and fixing the problem becomes a lot easier once you have narrowed it down to your courtyard. Framework exceptions are best implemented as a hierarchy of custom runtime exceptions. The root of this hierarchy could be a RuntimeException named MyFrameworkException and it could have any number of sub-classes such as MyFrameworkModelException, MyFrameworkControllerException, MyFrameworkViewException, etc. depending on how you have categorized different pieces of your framework.


3. Log, log, log:
   Where exception messages don’t tell much, debug messages logged from important places will. What comments are to source code, log messages are to runtime. The standard JDK logging API allows you to log messages at multiple levels including INFO, CONFIG, FINE, FINER, et al. Logging messages at various levels will not only provide answers to the hows, whens and wheres of the problem but will also go a long way in helping your customers understand the underpinnings of your framework. And more often than not that is all they need to fix the problem themselves!

We are constantly trying to improve the exception handling and logging in the ArcGIS ADF and you can be assured that we are employing these principles as well.