Wednesday, June 6, 2012

How to use Apache HttpClient 4 library securely connect to RESTful Web Service

In this blog, we will show to how to securely connect to RESTful web services, using Apache HttpClient 4 and HTTPS and basic authentication through username and password

Self-signed Certificate Support

Apache HttpClient 4 has changed quite a bit from version 3 and has much better self-signed certificate support. In order to support self-signed certificates, we need to create a new class that implements TrustStrategy called TrustSelfSignedStrategy and our TrustSelfSignedStrategy will trust any certificates and just return true. This class is just for illustration purpose and shouldn't be used in production.

   protected static class TrustSelfSignedStrategy implements TrustStrategy  
   {  
     @Override  
     public boolean isTrusted(X509Certificate[] arg0, String arg1) throws CertificateException  
     {  
       return true;  
     }  
   }  

The following code shows how to create a ClientConnectionManager object using the above TrustSelfSignedStrategy.
   protected ClientConnectionManager enableSelfSignedCerts() throws Exception  
   {  
     TrustStrategy trustStrategy = new TrustSelfSignedStrategy();  
     X509HostnameVerifier hostnameVerifier = new AllowAllHostnameVerifier();  
     SSLSocketFactory sslSf = new SSLSocketFactory(trustStrategy, hostnameVerifier);  
     Scheme https = new Scheme("https", 443, sslSf);  
     SchemeRegistry schemeRegistry = new SchemeRegistry();  
     schemeRegistry.register(https);  
     ClientConnectionManager connection = new PoolingClientConnectionManager(schemeRegistry);  
     return connection;  
   }  

Preemptive Basic Authentication with Username and Password

Next we will show how to use preemptive basic authentication using username and password. In web services world we must use preemptive basic authentication, since there is no web client to ask back to and prompt user for authentication credentials.

       String urlString = PING_IDENTITY_SERVER_URL + TOKEN_AUTH + URLEncoder.encode(TEST_TOKEN, "UTF-8");  
       URL url = new URL(urlString);  
       // support self-signed certificates  
       DefaultHttpClient httpClient = new DefaultHttpClient(enableSelfSignedCerts());  
       // add username/password for BASIC authentication  
       httpClient.getCredentialsProvider().setCredentials(new AuthScope(url.getHost(), url.getPort()),  
           new UsernamePasswordCredentials("user", "secret"));  
       // Create AuthCache instance  
       // Add AuthCache to the execution context  
       AuthCache authCache = new BasicAuthCache();  
       BasicScheme basicAuth = new BasicScheme();  
       authCache.put(new HttpHost(url.getHost(), url.getPort(), url.getProtocol()), basicAuth);  
       BasicHttpContext localcontext = new BasicHttpContext();  
       localcontext.setAttribute(ClientContext.AUTH_CACHE, authCache);  
       //  
       HttpGet getRequest = new HttpGet(urlString);  
       getRequest.setHeader("Content-Type", "application/json");  
       // call HTTP GET with authentication information  
       HttpResponse response = httpClient.execute(getRequest, localcontext);  
       if (response.getStatusLine().getStatusCode() != 200)  
       {  
         throw new RuntimeException("Failed : HTTP error code : " + response.getStatusLine().getStatusCode());  
       }  
       BufferedReader br = new BufferedReader(new InputStreamReader((response.getEntity().getContent()))); 

Troubleshooting Tips

If getting a 401 error, make sure that preemptive authentication is used and username and password is correct.

Saturday, June 2, 2012

Secure Jersey with OAuth2, Open Authentication Framework

Overview

Our platform must be secure. After some initial investigation, we decided to go with OAuth2, the next generation of OAuth protocol. The OAuth protocol enables websites or applications (Consumers) to access Protected Resources from a web service (Service Provider) via an API, without requiring Users to disclose their Service Provider credentials to the Consumers. More generally, OAuth creates a freely-implementable and generic methodology for API authentication.


Securing Jersey with OAuth2

We looked at implementing OAuth2 support as Tomcat security realm, servlet filter, or Tomcat valve. In the end, we decided to go with Tomcat valve for the following reasons,
  • Since we are implementing web services and not web applications, there is no standard way of caching Tomcat session between requests. This makes session based security realm irrelevant.
  • Servlet filer can do almost exactly the same thing as a Tomcat Valve, except for a servlet filter is deployed at web application level. This means we have to deploy this servlet for every web application we deploy. Not as convenient as a Tomcat valve
  • On the other hand, Tomcat valve is Tomcat specific. If we want a portable solution, we will have to stick with Servlet filter. Luckily we are sticking with Tomcat for now and there is very little effort if we need to switch to a Servlet filter based implementation

OAuth Tomcat Valve Class

Here is the example Valve class,
 package jersey.oauth;  
 import java.io.IOException;  
 import javax.servlet.ServletException;  
 import javax.servlet.http.HttpServletResponse;  
 import org.apache.catalina.connector.Request;  
 import org.apache.catalina.connector.Response;  
 import org.apache.catalina.valves.ValveBase;  
 import com.sun.security.auth.UserPrincipal;  
 public class OAuthValve extends ValveBase  
 {  
     protected String identityServerURL;  
     public String getIdentityServerURL() {  
         return identityServerURL;  
     }  
     public void setIdentityServerURL(String identityServerURL) {  
         this.identityServerURL = identityServerURL;  
     }  
     @Override  
     public void invoke(Request request, Response response) throws IOException,  
             ServletException {  
         if (request.getMethod().equals("OPTIONS"))  
             getNext().invoke(request, response);  
         else  
         {  
 //            response.sendError(HttpServletResponse.SC_FORBIDDEN);  
             String authentication = request.getHeader("authentication");  
             if (authentication == null)  
             {  
                 authentication = request.getParameter("access_token");  
             }  
             else  
             {  
                     String[] tokens = authentication.split(" ");  
                 if (tokens.length >= 2 && tokens[0].equalsIgnoreCase("Bearer"))  
                 {  
                     authentication = tokens[1];  
                 }  
                 else  
                 {  
                     authentication = null;  
                 }  
             }  
             if (authentication == null)  
                 response.sendError(HttpServletResponse.SC_UNAUTHORIZED);  
             else  
             {  
                 // TODO call identity server, passing on the access token
                 // Set return principal 
                 request.setUserPrincipal(new UserPrincipal("name"));  
                 getNext().invoke(request, response);  
             }  
         }  
     }  
 }  


Here is the sample Host block of Tomcat server.xml file,
 <Host appBase="webapps" autoDeploy="true" name="localhost" unpackWARs="true" xmlNamespaceAware="false" xmlValidation="false">  
     <!-- SingleSignOn valve, share authentication between web applications  
        Documentation at: /docs/config/valve.html -->  
     <!--  
     <Valve className="org.apache.catalina.authenticator.SingleSignOn" />  
     -->  
     <!-- Access log processes all example.  
        Documentation at: /docs/config/valve.html -->  
     <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" pattern="common" prefix="localhost_access_log." resolveHosts="false" suffix=".txt"/>  
         <Valve className="jersey.oauth.OAuthValve" identityServerURL="localhost"/>  
    <Context docBase="JerseyCors" path="/jersey" reloadable="true" source="org.eclipse.jst.jee.server:JerseyCors"/></Host>  
Deploy, start Tomcat and test.

Friday, June 1, 2012

Enable Cross Origin Resource Sharing for Jersey

Overview

In this blog, we will talk about how to enable and configure CORS support for Jersey, and more importantly, how to trouble shoot if CORS is not working properly.

As mentioned in the previous blog, we were disappointed to find out Apache CXF CORS support did not work and were pleasantly surprised on how easy CORS filter has been to setup and configure. We have tested CORS filter against Jersey, RESTeasy, and Apache CXF and it worked for every single one of them.

Enable CORS Support

CORS filter is implemented as a Servlet that must be enabled and configured at the web app's level, inside web.xml file.
Here is a sample web.xml file,
<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://java.sun.com/xml/ns/javaee" xmlns:web="http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd" id="WebApp_ID" version="2.5">
  <servlet>
    <servlet-name>Jersey Root REST Service</servlet-name>
    <servlet-class>com.sun.jersey.spi.container.servlet.ServletContainer</servlet-class>
    <init-param>
      <param-name>com.sun.jersey.config.property.packages</param-name>
      <param-value>jersey.cors</param-value>
    </init-param>
 <init-param>
  <param-name>com.sun.jersey.api.json.POJOMappingFeature</param-name>
  <param-value>true</param-value>
 </init-param>
 <load-on-startup>1</load-on-startup>
  </servlet>
  <servlet-mapping>
    <servlet-name>Jersey Root REST Service</servlet-name>
    <url-pattern>/*</url-pattern>
  </servlet-mapping>
  <filter>
  <filter-name>CORS</filter-name>
  <filter-class>com.thetransactioncompany.cors.CORSFilter</filter-class>
  
  <!-- Note: All parameters are options, if ommitted CORS Filter
       will fall back to the respective default values.
    -->
  <init-param>
   <param-name>cors.allowGenericHttpRequests</param-name>
   <param-value>true</param-value>
  </init-param>
  
  <init-param>
   <param-name>cors.allowOrigin</param-name>
   <param-value>*</param-value>
  </init-param>
  
  <init-param>
   <param-name>cors.supportedMethods</param-name>
   <param-value>GET, HEAD, POST, OPTIONS, PUT, DELETE</param-value>
  </init-param>
  
  <init-param>
   <param-name>cors.supportedHeaders</param-name>
   <param-value>Content-Type, X-Requested-With, Accept, Authentication</param-value>
  </init-param>
  
  <init-param>
   <param-name>cors.exposedHeaders</param-name>
   <param-value>X-Test-1, X-Test-2</param-value>
  </init-param>
  
  <init-param>
   <param-name>cors.supportsCredentials</param-name>
   <param-value>true</param-value>
  </init-param>
  
  <init-param>
   <param-name>cors.maxAge</param-name>
   <param-value>3600</param-value>
  </init-param>

 </filter>

 <filter-mapping>
  <!-- CORS Filter mapping -->
  <filter-name>CORS</filter-name>
  <url-pattern>/*</url-pattern>
 </filter-mapping>
  
</web-app>

Configuring CORS Filter

The default configuration values are good for everything except for the following two fields, 
  • cors.supportedMethods
  • cors.supportedHeaders
These two fields must be checked if CORS filter is not working as one expected.

cors.supportedMethods specifies a list of supported methods and the default value is GET, HEAD, and POST only. We recommend listing all HTTP methods as supported methods.

cors.supportedHeaders lists the set of supported header fields. This set must be expanded if more headers are passed in unexpected.  We recommend listing as many headers as possible.

Testing and Troubleshooting

CORS support can be tested through javascript and here is an example,
<html> 
<head> 
<title>Cors Example</title> 
<script type="text/javascript" src="https://ajax.googleapis.com/ajax/libs/jquery/1.7.1/jquery.min.js"></script>
<script type="text/javascript" src="log4javascript.js"></script>
<script> 
var hello = JSON.stringify({"greeting":"Hello","name":"jersey"});
//alert(hello);
$(document).ready(function() {

 //alert('before ajax call');
 $.ajax({
  headers: {
   Authentication : 'Bearer access_token'
  },
  
  //this is the php file that processes the data and send mail
  //url: "http://localhost:8080/cxf-hello-cors/rest/annotatedGet/hello", 
  //url: "http://localhost:8080/cxf-hello-cors/service1/time", 
  // url: "http://localhost:8080/resteasy/tutorial/helloworld",
  url: "http://localhost:8080/jersey/hello",
  
  contentType: "application/json",

  //GET method is used
  type: 'DELETE',
  
  //pass the data         
  dataType: 'json',   
   
  //data: JSON.stringify(hello), 
  data: hello,
  
  //Do not cache the page
  cache: false,
   
  //success
  success: function (html) {  
   //alert(html); 
   document.getElementById("cors").innerHTML = "Echo: " + html.greeting + "," + html.name; 
           
  } ,
  error:function (data, status) {
   alert(data);
   alert(status);
     }      
 });
     
 });
</script>
</head> 
<body> 

<h1>This is the CORS test page</h1>

<p>Hello, <div id="cors"/>

</body> 
</html>

Troubleshooting CORS

We use a combination of Tomcat access log, Firefox Firebug, and Jersey client to troubleshoot CORS support.
CORS relies on header to relay cross origin  resource sharing information back to the browser and CORS-supported browser will enforce CORS based on these header fields. When CORS is not working as expected, the majority of the errors happen when Web Services do not pass back the appropriate headers due to permission related issues, like supported headers or supported methods. The best place to look for this type of information is in Tomcat's access log.
Here are some sample entries from the access log,

127.0.0.1 - - [31/May/2012:15:40:42 -0400] "GET /jersey/hello HTTP/1.1" 401 -
127.0.0.1 - name [31/May/2012:15:42:27 -0400] "GET /jersey/hello HTTP/1.1" 200 36
127.0.0.1 - - [31/May/2012:15:42:39 -0400] "GET /jersey/hello HTTP/1.1" 401 -
127.0.0.1 - - [31/May/2012:15:44:18 -0400] "GET /jersey/hello HTTP/1.1" 401 -
127.0.0.1 - - [31/May/2012:15:45:45 -0400] "GET /jersey/hello HTTP/1.1" 401 -
127.0.0.1 - - [31/May/2012:15:46:38 -0400] "GET / HTTP/1.1" 401 -
127.0.0.1 - - [31/May/2012:15:46:52 -0400] "GET /jersey/hello HTTP/1.1" 401 -
0:0:0:0:0:0:0:1%0 - - [31/May/2012:15:47:02 -0400] "OPTIONS /jersey/hello HTTP/1.1" 403 94
127.0.0.1 - - [31/May/2012:15:48:06 -0400] "GET / HTTP/1.1" 401 -
0:0:0:0:0:0:0:1%0 - - [31/May/2012:15:51:23 -0400] "OPTIONS /jersey/hello HTTP/1.1" 200 -
0:0:0:0:0:0:0:1%0 - name [31/May/2012:15:51:23 -0400] "DELETE /jersey/hello HTTP/1.1" 200 36
127.0.0.1 - name [31/May/2012:16:01:12 -0400] "GET /jersey/hello HTTP/1.1" 200 36 
Each entry represents an access from the client. The last three entries represent the following, request URI, HTTP status code, return content length. If CORS is not working as expected, check the following,
  • Make sure there is an entry in the access log that corresponds to the request
  • Make sure HTTP status code is correct. If HTTP status code is 403, check CORS filter's
    supported methods and supported headers to make sure that both settings are configured
    properly
If HTTP code is 200 but CORS is still not working, turn Firebug on and examine the request
pay special attention to response headers,
Debugging CORS response with Firebug


Make sure the set of Access-Control-* headers present in response.

Implement RESTful Web Services using Jersey

Jersey Overview

Jersey is the open source, production quality, JAX-RS (JSR 311) Reference Implementation for building RESTful Web services. Jersey is very lightweight and can be deployed in Web application containers like Tomcat, Jetty, Glassfish.

In this blog, we will provide a comprehensive tutorial on how to create a RESTful web services using Jersey, including,
  • Support for HTTP method, GET, PUT, POST, and DELETE
  • Support for various content types including TEXT, XML, and HTML
  • JSON support

Develop RESTful Web Services using Eclipse

We will demonstrate how to develop web services in Jersey using Eclipse. in the following steps,
  1. Create a new Dynamic Web Project in Eclipse
  2. Import Jersey related libraries
  3. Configure web.xml to support Jersey
  4. Create a Java class that implements Web Services
  5. Deploy web project in Tomcat
  6. Start Tomcat and test

Configure Tomcat to enable access log logging

We need to enable Tomcat to log accesses for easy debugging. Uncomment the following block in Tomcat's server.xml file,
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" pattern="common" prefix="localhost_access_log." resolveHosts="false" suffix=".txt"/>

Configure web.xml to support Jersey

Here is a sample web.xml file, which includes JSON support and auto scanning of Java package jersey.cors. Replace package name "jersey.cors" with one's own package.

<?xml version="1.0" encoding="UTF-8"?>

<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://java.sun.com/xml/ns/javaee" xmlns:web="http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd" id="WebApp_ID" version="2.5">
  <servlet>
    <servlet-name>Jersey Root REST Service</servlet-name>
    <servlet-class>com.sun.jersey.spi.container.servlet.ServletContainer</servlet-class>
    <init-param>
      <param-name>com.sun.jersey.config.property.packages</param-name>
      <param-value>jersey.cors</param-value>
    </init-param>
 <init-param>
  <param-name>com.sun.jersey.api.json.POJOMappingFeature</param-name>
  <param-value>true</param-value>
 </init-param>
 <load-on-startup>1</load-on-startup>
  </servlet>
  <servlet-mapping>
    <servlet-name>Jersey Root REST Service</servlet-name>
    <url-pattern>/*</url-pattern>
  </servlet-mapping>
</web-app>

Create a Java Class that Implements Web Services


package jersey.cors;

import javax.ws.rs.DELETE;
import javax.ws.rs.GET;
import javax.ws.rs.POST;
import javax.ws.rs.PUT;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;

// POJO, no interface no extends

// The class registers its methods for the HTTP GET request using the @GET annotation. 
// Using the @Produces annotation, it defines that it can deliver several MIME types,
// text, XML and HTML. 

// The browser requests per default the HTML MIME type.

//Sets the path to base URL + /hello
@Path("/hello")
public class HelloWorldRest {

 @GET
 @Path("/{param}/")
 public String getMsg(@PathParam("param") String msg) {
 
  String output = "Jersey say : " + msg;
 
  return output;
 
 }
 @GET
 @Path("/world")
 public String getFixedMsg(String msg) {
 
  String output = "Jersey say : fixed path" + msg;
 
  return output;
 
 }
 // This method is called if TEXT_PLAIN is request
 @GET
 //@Path("helloworld")
 @Produces(MediaType.TEXT_PLAIN)
 public String sayPlainTextHello() {
  System.out.println("sayPlain");
  return "Hello Jersey";
 }

 // This method is called if XML is request
 @GET
 //@Path("helloworld")
 @Produces(MediaType.TEXT_XML)
 public String sayXMLHello() {
  System.out.println("sayXML");
  return "" + " Hello Jersey" + "";
 }

 // This method is called if HTML is request
 @GET
 //@Path("helloworld")
 @Produces(MediaType.TEXT_HTML)
 public String sayHtmlHello() {
  System.out.println("sayHTML");
  return "<html> " + "<title>" + "Hello Jersey" + "</title>"
    + "<body><h1>" + "Hello Jersey" + "</body></h1>" + "</html> ";
 }
 
 // This method is called if JSON is request
 @GET
 //@Path("helloworld")
 @Produces(MediaType.APPLICATION_JSON)
 public Hello sayJsonHello() {
  return new Hello("Hello", "Jersey");
 }
        @PUT
 @Produces(MediaType.APPLICATION_JSON)
 public Hello updateHello(Hello hello)
 {
  System.out.println("put");
  return hello;
 }
 @POST
 @Produces(MediaType.APPLICATION_JSON)
 public Hello createHello(Hello hello)
 {
  System.out.println("post");
  return hello;
 }
 @DELETE
 @Produces(MediaType.APPLICATION_JSON)
 public Hello deleteHello(Hello hello)
 {
  System.out.println("post");
  return hello;
 } 
 @DELETE
 @Produces(MediaType.TEXT_PLAIN)
 public String deleteHello(String hello)
 {
  System.out.println("post");
  return hello;
 } 
}
Here is the Hello class, which serves as JSON transport object.
package jersey.cors;

import javax.xml.bind.annotation.XmlRootElement;

@XmlRootElement
public class Hello {
 String greeting;
 String name;
 public Hello()
 {
  
 }
 public Hello(String greeting, String name)
 {
  this.greeting = greeting;
  this.name = name;
 }
 public String getGreeting() {
  return greeting;
 }
 public void setGreeting(String greeting) {
  this.greeting = greeting;
 }
 public String getName() {
  return name;
 }
 public void setName(String name) {
  this.name = name;
 }
 
} 

Testing Web Services

After web service project is deployed in Tomcat, we should be able to start Tomcat with no error.
Generally, Web Services can be tested in the following manner,
  • HTTP GET method can be tested directly through a browser
  • HTTP GET, PUT, DELETE, and POST can be tested through Java script
  • HTTP GET and POST can be tested through Jersey client. (Due the limitation in Java URLConnection class, it can be challenging to test HTTP PUT and DELETE using Jersey client). 

Javascript Example

 Here is an Javascript HTTP DELETE example, with JSON support,
<html> 
<head> 
<title>Cors Example</title> 
<script type="text/javascript" src="https://ajax.googleapis.com/ajax/libs/jquery/1.7.1/jquery.min.js"></script>
<script type="text/javascript" src="log4javascript.js"></script>
<script> 
var hello = JSON.stringify({"greeting":"Hello","name":"jersey"});
//alert(hello);
$(document).ready(function() {

 //alert('before ajax call');
 $.ajax({
  headers: {
   Authentication : 'Bearer access_token'
  },
  
  //this is the php file that processes the data and send mail
  //url: "http://localhost:8080/cxf-hello-cors/rest/annotatedGet/hello", 
  //url: "http://localhost:8080/cxf-hello-cors/service1/time", 
  // url: "http://localhost:8080/resteasy/tutorial/helloworld",
  url: "http://localhost:8080/jersey/hello",
  
  contentType: "application/json",

  //GET method is used
  type: 'DELETE',
  
  //pass the data         
  dataType: 'json',   
   
  //data: JSON.stringify(hello), 
  data: hello,
  
  //Do not cache the page
  cache: false,
   
  //success
  success: function (html) {  
   //alert(html); 
   document.getElementById("cors").innerHTML = "Echo: " + html.greeting + "," + html.name; 
           
  } ,
  error:function (data, status) {
   alert(data);
   alert(status);
     }      
 });
     
 });
</script>
</head> 
<body> 

<h1>This is the CORS test page</h1>

<p>Hello, <div id="cors"/>

</body> 
</html>

Jersey Client Example 

Here is a HTTP GET Jersey client example,
package jersey.cors.client;

import com.sun.jersey.api.client.Client;
import com.sun.jersey.api.client.ClientResponse;
import com.sun.jersey.api.client.WebResource;
 
public class JerseyClient {
 
  public static void main(String[] args) {
 try {
 
  Client client = Client.create();
 
  WebResource webResource = client
     .resource("http://localhost:8080/jersey/hello");
  
  ClientResponse response = webResource.accept("application/json")
                   .get(ClientResponse.class);
 
  if (response.getStatus() != 200) {
     throw new RuntimeException("Failed : HTTP error code : "
   + response.getStatus());
  }
 
  String output = response.getEntity(String.class);
 
  System.out.println("Output from Server .... \n");
  System.out.println(output);
 
   } catch (Exception e) {
 
  e.printStackTrace();
 
   }
 
 }
}



 

 

Thursday, May 31, 2012

Build secure, cross-origin resource sharing RESTful web services using Jersey, CORS filter, and OAuth

Overview

My company is taking on the challeng of building out next generation platform based on secure RESTful web services, which must be secured through OAuth and support Cross-Origin Resource Sharing (CORS).

Web Services can be implemented as SOAP and WSDL based or RESTful based. Our new platform will be implemented as RESTful based web service.

Our new platform also needs to support Cross Origin Resource Sharing, which allows third party Javascript to call our platform without being restricted to the same domain. Traditionally the browser enfoces that an AJAX can only call back to the same domain. For CORS supported browser, however, the browser can grant cross domain access if proper access is returned by the back end web services.

Our platform must be secure. After some initial investigation, we decided to go with OAuth2, the next generation of OAuth protocol. The OAuth protocol enables websites or applications (Consumers) to access Protected Resources from a web service (Service Provider) via an API, without requiring Users to disclose their Service Provider credentials to the Consumers. More generally, OAuth creates a freely-implementable and generic methodology for API authentication.

After some research and prototyping, we settle on the following set of technologies to address our particular challenges;
  • Use Jersey to implement RESTful web services
  • Use CORS filter to implement CORS support
  • Implement and deploy on Tomcat OAuth valve class for OAuth support
In this blog series, we will illustrate the following,

RESTful Web Services Framework

The initial candidates consist of three JAX-RS compliant RESTful Web Services framework,


RESTful Framework Description Experience
Jersey Jersey is the JAX-RS reference
implementation from Oralce. One
chooses between CDDL 1.1 license
or GPL 2 with CPE license
Easy to develop and configure.
Has good JSON support.
No external dependency, although
does offer Spring integration
JBoss RESTeasy RESTeasy is maintained by Jobs
and uses Apache 2.0 license
Easy to develop with a lot of
JBoss dependencies
Apache CXF Apache CXF offers both WSDL-based
and RESTful web services and uses
Apache 2.0 license
Easy to develop, heavy
framework with a lot of
dependencies and support for
JAX-WS

We prototyped in all three frameworks and they are all relatively easy to develop and configure. We decided to go with Jersey for the following reasons,

  • Reference implementation
  • Small set of libraries
  • No external dependency
  • Can easily switch if Jersey does not meet our needs in the future

Migrate to another framework

Since all three frameworks are JAX-RS compliant, the implementation classes use JAX-RS standard annotation and there is no direct dependency on the framework. If we need to swamp out Jersey with another framework, we just need to configure Tomcat web application's web.xml with framework specific configuration and deploy framework specific libraries to the lib directory.

CORS Support

We prototyped CORS support based on the following frameworks, CXF CORS support and CORS filter. We couldn't get CXF CORS to even compile, as the class structure has changed from the sample code. Once we fixed the compilation issue, we still couldn't get CORS to work through AJAX.

CORS filter, on the other hand, was a pleasure to work with. In a span of half a day, we quickly set up and integrated CORS filter with Jersey, RESTeasy, and CXF. The only drawback about CORS filter is that it is not as flexible as CXF CORS because CORS filter can only be configured at the global level, while CXF CORS can be configured at each class level, if one can get CXF CORS to work.

So, I decide to go with CORS filter as the one and only working solution.

OAuth Support

All our RESTful web services must be secured with OAuth support. There are potentially a few options on how to implement this and in the end we decided to implement this a Tomcat Valve instead of using a security realm or a servlet filter. The big advantage of implementing OAuth as a Valve is the flexibility of deploying the Valve at the Host level or at the web application level, while servlet filter can only be deployed at the web application level. More on this topic in a later blog.




Tuesday, March 6, 2012

SCTE-130 vs IAB VAST (part II)

In the previous blog, I expanded on the major differences between SCTE-130 and IAB VAST.
In this blog, I will continue examine the rest of the major differences between SCTE-130 and IAB VAST.
In the fourth and final installment, I will discuss the upcoming convergence of advertising decision making between the online and cable deployment and how it presents an compelling business opportunity to bridge the gap between SCTE-130 and IAB VAST.

I will discuss the key different between two standards in the following areas,
  • Type and Interactivity with Ads 
  • Complexity 
  • Protocol Format 
  • When to Play Ads 
  • Deployment, Serving Ads, and Tracking Playback events 
  • Targeting and Addressability 
  • Report gathering to improve addressability 
  • Measurement and Operational Efficiency Best Practices 
In this blog, I will discuss the last five areas.

Deployment, Serving Ads and Tracking Playback Events

Ad Ingest
Unlike the online world, where videos can be served from CDN and consumed by video player directly, an ad must be ingested, transcoded, moved to CDN or onto the video server’s local storage, before the ad can be streamed to setup boxes.

Since ADS is an independent component, ADS will not play an ad until the ADS is sure that particular ad has been ingested and moved to storage where it can be streamed by video servers.

When an ad has been fully ingested, the ingest process will notify all subscribing ADSs through CIS notification interface that this particular ad can be placed during an ad break.

Response
IAB VAST response contains direct links to videos, banners, and overlays, which can be downloaded and played by the video player directly.

SCTE-130 response, on the other hand, contains asset and provider ids, information that will be used to identify the actual ingested ad video files to play by the video server.

Deployment
The Ad server is typically deployed on a publicly accessible network and videos and images are deployed on a public accessible CDN.

In SCTE-130 world, most components will be deployed in MSO’s central sites (session manager, SIS, CIS, POIS, ADS), and some (ADM, video server, splicer) will be deployed on the edge sites.

Serving Ads and Tracking Playback Events
IAB VAST
In the online world, ads are hosted on a CDN, streamed by either a media server or Apache server. The video player is responsible for sending playback events to the Ad server at the appropriate event (video start play, first quartile, midpoint, third quartile, and complete) in real time.

SCTE-130
In the cable world, ads can be hosted on a CDN or shared storage and streamed by a video server. The playback tracking events are handled differently for VOD and for linear Ad insertion.

VOD
For VOD insertion, setup box communicates user’s activities (pause, fast forward, etc) to the streaming video server. Video server collects user’s activities for the entire playlist (content and Ads) and uploads the entire playback events to ADM at the end.

LINEAR
For linear ad insertion, ADM collects playback events from the splicer and the streaming video server and the combined results will determine the success or failure of the ad insertion. The playback event is then sent to ADS.

Targeting and Addressability
SCTE-130 provides the following mechanism for targeting and addressability,
  • Subscriber Information System (SIS), a standard interface that exposes subscriber’s demographic and geographic information 
  • A standard way of passing any targeting-related information from ADM to ADS 
IAB VAST does not define such standard and relies on each Ad server to define its own Ad tags.

Report gathering to improve addressability
Report gathering and analysis is relative straightforward, as the video player will report playback tracking information back to the Ad server directly. The Ad server can mine this data and improve addressability based on user's interaction with the ad.

Report gathering for SCTE-130, on the other hand, can be challenging because of the inherited complexity and multiple systems involved. Video server will report back playback information to ADS through ADM for billing purpose but this information really needs to go back to Subscriber Information System (SIS) in order to improve future addressability.

Measurements and Operational Efficiency Best Practices

IAB offers set of comprehensive guidelines on measurements and operational efficiency best practices.
Digital Video

IAB VAST publishes the following guideline for digital video advertising,
  • Digital Video In-Stream Ad Metrics Definitions 
  • Digital Video Ad Measurement Guidelines 
  • Digital Video Ad Format Guidelines & Best Practices
Measurement Guidelines
IAB publishes measurement guidelines for the following,
  • Ad Impression 
  • Ad Campaign 
  • Ad Verification 
  • Audience Reach 
  • Click 
  • Digital Video Ad 
  • Mobile Web Advertising 
  • Rich Media 
  • Rich Internet Application
Operational Efficiency Best Practices
IAB publishes the following operational efficiency best practices,
  • Ad Load Performance 
  • Rich Media Ads in Asynchronous Ad Environment 
  • Billing Methods 
  • Impression Exchange Solutions 
  • Interactive Advertising Workflow 
  • Interactive Campaign Setup 
  • Revenue Cycle 

Sunday, March 4, 2012

SCTE-130 vs. IAB VAST (Part I)

In the previous blog, I gave a brief introduction to SCTE-130 and IAB VAST. In this blog and next, I will expand on the major differences between SCTE-130 and IAB VAST. In the fourth and final installment, I will discuss the upcoming convergence of advertising decision making between the online and cable deployment and how it presents an interesting business opportunity to bridge the gap between SCTE-130 and IAB VAST.
I will discuss the key different between two standards in the following areas,
  • Type and Interactivity with Ads
  • Complexity
  • Protocol Format
  • When to Play Ads
  • Deployment, Serving Ads, and Tracking Playback events
  • Targeting and Addressability
  • Report gathering to improve addressability
  • Measurement and Operational Efficiency Best Practices
In this blog, I will discuss the first three areas.

Type and Interactivity with Ads

SCTE-130 standard is video centric and relies on EBIF standard to provide interactivity with ads through setup box.
IAB VAST standard supports video, interactive ads, banners, and overlays. IAB defines video player’s interactivity with ads through IAB VPAID standard.

Complexity

IAB VAST

The IAB VAST standard defines the correspondence between a video player and Ad servers. Here are the components in an IAB VAST deployment,
·         Millions of online video players
·         Ad servers
·         CDN that hosts videos, images, Flash files, Silverlight files
·         Media servers or Apache servers

SCTE-130

On the other hand, SCTE-130 standard consists of the following core components,
·         Ad Management Service (ADM)
·         Ad Decision Service (ADS)
·         Placement Opportunity Information System (POIS)
·         Content Information System (CIS)
·         Subscriber Information System (SIS)
Each core components can be developed and deployed independently, and most times they are from different vendors.
To add to the complexity, these core components need to communicate with the following external systems to carry out the Ad insertion,

Systems
Description
SCTE-130 Core Components
Session Manager
Session Manager is responsible for setting up video sessions on behalf of the setup box
ADM
Video Server
Video Server streams/broadcast videos to setup box
ADM
Splicer
A piece of equipment that can perform frame-accurate ad splicing into network broadcast streams
ADM

All these systems must work together to carry an ad insertion event and this is the challenge SCTE-130 attempts to address. This is why SCTE-130 is such a complicated and overarching standard.

Protocol Format

Both SCTE-130 and IAB VAST use XML based protocol. However, IAB VAST does not define a standard request message, only suggested Ad tags that describes duration, player format, height, width, bandwidth, and supported downloading method. SCTE-130, on the other hand, uses XML as both request and response messages.

When to Play Ads

SCTE-130

In cable deployment environment for VOD content, the session manager will request the entire playlist with Ads filled in from ADM during the session setup time. A SCTE-130 playlist response looks like the following, assuming VOD content has no embedded Ads in it,

  1. Play pre-roll Ad video1 (30 seconds)
  2. Play pre-roll Ad video2 (30 seconds)
  3. Play VOD content 0-30 minutes
  4. Play mid-roll Ad video3 (30 seconds)
  5. Play mid-roll Ad video4 (30 seconds)
  6.  Play VOD content 30-60 minutes (end of the content)
  7.  Play post-roll Ad video5 (30 seconds)
  8. Play post-roll Ad video6 (30 seconds)
Clearly SCTE-130 response contains information on when to play ads. When the session manager receives the playlist, the session manager will hand over the playlist to the video server, which will stream the content and ads to the subscriber.

IAB VAST

IAB VAST standard does not contain information on when to play ads so it is up to the player or a different standard (MAST) to decide when to play the returned ads. Typically the VAST response is played right away by the player. In contrast to cable deployment scenario, where the entire playlist is constructed at the beginning of the playback, the online video player must be aware of each ad break in the VOD or linear content and makes IAB VAST request and plays VAST response at the appropriate ad break time.