[WSO2 APIM ] [Gateway] How to limit traffic to WSO2 APIM Gateway using API Handlers

Recently I had to find a solution to limit the traffic that passes through the WSO2 APIM Gateway. The requests that are allowed to pass through are the requests from the testing team. The architecture of the system is quite straight forward. Please find the drawing of the system architecture in brief below.

When it comes to limiting the traffic to the Gateway, there are several ways to handle this. This can be handled from the network level (e.g. routing rules) or even simply starting the server on a another port that is not visible to the outside world. Here, in this article I am going to describe how to do it from the Gateway. Using this method is only another option to achieve the goal.

 When an API is created from the WSO2 APIM, using the publisher, the publisher will propagate the changes to the WSO2 APIM Gateway. By this it means it will create a synapse configuration on the Gateway. This synapse configuration of the API will hold a set of handlers. These handlers are placed to achieve different functionality. For e.g. the APIAuthenticationHandler is intended to validate the token. More information on the handlers can be found at [1]. The Handlers are placed in an order in the synapse configuration, and they will be executed in the order they appear. Since the first point of contact in the API is their handlers, we can use a handler to filter out the request if it is from the testing device or not.

 To achieve this, we need to have an identifier being sent from the request. If we have this, it is easily possible to filter them out. First things first, we need to figure out what the identifier is. In my case, the requests sends the device ID in the header, under the parameter "Auth". So in my handler I will read the header and check for this Auth value.
 How do I tell which device Id's are able to continue? For this, I will read the device Id's from a system property so that the allowed device Id's could be passed from the command line as a system property when the server is start up.

 Okay, so given a brief description of what we are going to achieve, lets see how we can do this.

1. Create the Custom Handler. To create the custom handler we need to create a maven project, and create a new class. This class must be extending the "org.apache.synapse.rest.AbstractHandler". Please find the sample code below.

package org.wso2.CustomHandler;

import java.util.Map;

import org.apache.synapse.MessageContext;
import org.apache.synapse.core.axis2.Axis2MessageContext;
import org.apache.synapse.rest.AbstractHandler;

public class GatewayTrafficContrlHandler extends AbstractHandler {

 public boolean handleRequest(MessageContext arg0) {
  String deviceId = null;
  String identifier = null;
  String[] identifiers;

  //Obtain the identifier .
  Map headers = (Map)((Axis2MessageContext) arg0).getAxis2MessageContext().
  getProperty(org.apache.axis2.context.MessageContext.TRANSPORT_HEADERS);
  identifier = (String) headers.get("Auth");

  if (identifier == null || identifier.isEmpty()) {
   //Get the first identifier which is the device ID.
   identifiers = identifier.split("\\|");
   if (identifiers != null && identifiers.length > 0) {
    deviceId = identifiers[0];
   }

   //Get the device ID list  which is passed as a system property from                        //command line.Only these deviceId's will be allowed to pass 

                        //through the handler.
   String[] supportedDeviceIdList = null;
   String supportedDeviceIds = System.getProperty("deviceIds");
   if (supportedDeviceIds != null && !supportedDeviceIds.isEmpty()) {
    supportedDeviceIdList = supportedDeviceIds.split(",");
   }

   //Check if the device Id which is sent in the request is in the 
                        //list of deviuce Id's passed as a system property.
   if (supportedDeviceIdList != null && supportedDeviceIdList.length > 0) {
    for (int index = 0; index < supportedDeviceIdList.length; index++) {
     if (supportedDeviceIdList[index].equals(deviceId) == true) {
      return true;
     }
    }
   }
  }
  return false;
 }

 public boolean handleResponse(MessageContext arg0) {
  // TODO Auto-gGatewayTrafficContrlHandlerenerated method stub
  return false;
 }

}

2. Once we create the Class to extract and find for the identifier, we then need to build the jar.
3. Copy the created Jar to the /repository/components/lib folder. 
4. Then start the APIM server with the system property -DdeviceIds, and log into the APIM      
     Gateway's management console. for e.g :- ./wso2server.sh -DdeviceIds=123,456,789 
5. Go to Service Bus > Source View in the Main menu.
6. In the configuration select the API, and then check the Handler section. In the handler section, add 
    the class to point to the newly created handler.

 <handlers>
        <handler class="org.wso2.CustomHandler.GatewayTrafficContrlHandler"/>
        <handler class="org.wso2.carbon.apimgt.gateway.handlers.security.APIAuthenticationHandler"/>
        <handler class="org.wso2.carbon.apimgt.gateway.handlers.throttling.APIThrottleHandler">
            <property name="id" value="A"/>
            <property name="policyKey" value="gov:/apimgt/applicationdata/tiers.xml"/>
        </handler>
        <handler class="org.wso2.carbon.apimgt.usage.publisher.APIMgtUsageHandler"/>
        <handler class="org.wso2.carbon.apimgt.usage.publisher.APIMgtGoogleAnalyticsTrackingHandler"/>
        <handler class="org.wso2.carbon.apimgt.gateway.handlers.ext.APIManagerExtensionHandler"/>
    </handlers>

7. Make sure the newly created handler is the first handler in the list.

8. Once we do the change save, and observe the console for reloading of the API. 

9. Test the Handler by doing a REST API call.

And that's it. 

 The above solution will only let the requests with deviceId's 123 or 456 or 789 to pass through.

  References 

[1] https://docs.wso2.com/display/AM160/Architecture#Architecture-APIHandlers

[2] https://docs.wso2.com/display/AM160/Writing+a+Custom+Authentication+Handler

[WSO2-ESB] URI- template encoding in WSO2 ESB for reserved characters

 In a recent engagement with the WSO2 ESB, I've come across a situation where WSO2 ESB does not encode some reserved characters when using a URI-template as the endpoint to a get call.

In this particular instance it was the '&' that was not getting encoding and the <space>.

The reason is that WSO2 ESB uses the java.net libraries  and it ends up not encoding the characters correctly.

As a solution to this, we can use the script mediator in WSO2 ESB. We could get the value in the script mediator and encode it using the  javascript encode uri method, and encode our values correctly.

An example as shown below:

<?xml version="1.0" encoding="UTF-8"?>
<api
    xmlns="http://ws.apache.org/ns/synapse"
     name="testAPI"
     context="/testAPI">
    <resource methods="POST">
        <inSequence>
            <log level="full"/>
            <property name="uri.var.fullname" expression="//APIRequest/Fullname/text()"/>
            <property name="uri.var.address" expression="//APIRequest/HomeAddress/text()"/>
            <script language="js">var fullname = mc.getProperty('uri.var.fullname');var homeAddress = mc.getProperty('uri.var.address');                 mc.setProperty("uri.var.fullname",encodeURIComponent(fullname));                 mc.setProperty("uri.var.address",encodeURIComponent(homeAddress));                 </script>
            <header name="Content-Type" scope="transport" action="remove"/>
            <send>
                <endpoint>
                    <http method="get"
                     uri-template="https://localhost/test?apikey=1234&amp;api=54&amp;fullname={+uri.var.fullname}&amp;address={+uri.var.address}"/>
                </endpoint>
            </send>
        </inSequence>
    </resource>
</api> 

In this what is done is that the API request values has been extracted and stored in a property using a property mediator.

Then by using the script mediator  we make use of javascript and store the value as a var and encode it using the "encodeURIComponent" method.

In the "encodeURIComponent" yet, we still have some characters that are not encoded. These include the !'()*~.

References

[1] https://docs.wso2.com/display/ESB480/Getting+Started+with+REST+APIs#GettingStartedwithRESTAPIs-URItemplates

[2] https://docs.wso2.com/display/ESB480/Template+Endpoint

[3] https://docs.wso2.com/display/ESB480/Property+Mediator

[4] https://docs.wso2.com/display/ESB480/Script+Mediator

[5] http://en.wikipedia.org/wiki/Percent-encoding
[6] http://xkr.us/articles/javascript/encode-compare/

Wired network not detected in ubuntu 14.04

I was at a customer site, and it was required that I connect to their wired network. Everyone in the customer site could connect to the wired network, except me... :) Why? well probably two reasons, I had a ubuntu machine where as everyone else had windows, and I had a machine I used at WSO2,  with WSO2 network.


Hence, I googled a bit, and figured out the issue. The issue in my case was that the /etc/network/interfaces file had some unwanted lines in there.

If you do happen to come across the same issue, you could try this as well.

Comment or delete any lines in the /etc/network/interfaces file except for the lines 

auto lo
iface lo inet loopback

I personally would comment.. :)


References

[1] http://ubuntuforums.org/showthread.php?t=2247351

Waiting for network configuration at boot up in ubuntu (14.04)

Today I was meddling around with the network configurations as I could not connect to a wired network, and with something I did, when I restarted I was keeping on getting the message "Waiting for network configuration at boot up in ubuntu" and then wait upto 60 seconds, and start up without any network manager settings.

Because of this I could not connect to a network, and the network manager was literally stalled.

After looking at my previous steps, hitting the history command

     #history

it shows that I was meddling with the resolve.conf file. But when I looked at the file, it showed that no DNS nameservers were defined in the file. The file was EMPTY.... PANIC!!!!


So then I tried setting the nameserver in the /etc/resolve.conf file, however it was getting replaced.

The solution was,

1) go to /etc/resolveconf/resolv.conf.d/
      # cd  /etc/resolveconf/resolv.conf.d/
2) Open the base file n super user mode
     # sudo vim base
3) add the nameserver details
    e.g : nameserver 127.0.1.1
4) Save file, and restart.


This should fix the issue.

References 

[1] http://unix.stackexchange.com/questions/128220/how-do-i-set-my-dns-on-ubuntu-14-04

[WSO2 APIM] - The Basic Architecture of WSO2 APIM -part 5- Statistics in APIM

In this blog post I will briefly go about how the APIM publisher and store are able to draw up graphs with statistical information on the API's. This blog post is targetted for new bee's who would like to get a bird's eye view on the functionality.

In the WSO2 APIM Gatway, as I explained in my previous post The Basic Architecture of WSO2 APIM - part 3- Gateway & Key Manager - Under the hoods, it has a set of handlers defined per API.  One of these handlers is the APIMgtUsageHandler , which would invoke the org.wso2.carbon.apimgt.usage.publisher.APIMgtUsageDataPublisher .  The APIMgtUsageDataPublisher  is configured to publish the events to the BAM server that has been configured with the APIM. 

Illustrated by a diagram is the process of publishing of stats and viewing them on the publisher /store app's 




The WSO2 APIM Gateway would  publish the events  via the thrift protocol to the WSO2 BAM server. The BAM is more likely a BAM cluster in the real world production environment. The BAM server then writes the data to a No Sql database, cassandra.  Then the BAM Analyzer to which the APIM tool box is deployed to, will fetch the data batch by batch from the  cassandra database.  The BAM Analyzer  is capable of summarizing the data, by using hive. The hive scripts have to pre written and deployed in the server. The BAM analyzer then will push the summarized data to an RDBMS.

The WSO2 APIM Store and the WSO2 Publisher then will pull the data from the RDBMS  and will display the data in the APIM Store and publisher analytics pages.

This is a very brief explanation of what happens in the APIM when statistics are to be displayed.

Detailed information can be found  at the references listed below.

References

[1] https://docs.wso2.com/display/AM180/Publishing+API+Runtime+Statistics#PublishingAPIRuntimeStatistics-ConfigureWSO2BAM
[2] http://wso2.com/library/articles/2013/08/how-to-use-wso2-api-manager-and-bam-together-to-analyse-api-usage-stats/

 

[WSO2 APIM] - The Basic Architecture of WSO2 APIM - part 4- Gateway & Key Manager - Under the hoods

In this post I will briefly introduce how the Gatway and the Key manager interacts in order for an API to be successfully invoked.

Let me bring up a diagram first.

In a real world distributed deployment, only the WSO2 APIM Gateway would be  exposed to the outside world. So with the creation of keys for a certain application that would be subscribed to many API's, (If you are not familiar with this context, please refer to the blog post I had written before The Basic Architecture of WSO2 APIM - Part 2- The Store - under the hood  for a crash course) you would be able to invoke an API via the gateway.

Ideally an Application would have the consumer_key and the consumer_secret hardcoded in the application it self. When invoking the API, the username, password would have to be added and the application would pass in the username, password, consumer_key and consumer_secret  to the gateway. The Gateway has some token API's [1]. which are
   - /token
   - /revoke

When calls are made to these API's,  the gateway calls the Key manager Jax-RX  in order to verify the access the token.  The key-manager would call the AM_DB, retrieve the access token, and verify the access token.  it will return the API Info DTO which includes the meta data of the access token, which includes the validity period, refresh token and scopes.

When the API's are invoked, the Gateway does not call the Key manager at every invocation.  The APIM Gatway makes use of a cache  for this. However, this cache can be turned off. [2]

Invoking the API

When the API's are invoked there are several authorization grant types that we could use. The WSO2 APIM supports the same grant types the WSO2 IS supports.  The password grant type is used only when the application is a trusted application  Client Credentials grant type requires only the consumer_key & consumer_secret. [1].



API Handlers

When the API's is created it will store the synapse configuration of the API in the WSO2 APIM Gateway, being another ESB (WSO2 ESB).  When the api is invoked and it hits the Gateway, it will execute the API's in and out sequences  which would have a set of handlers. The API when created would have a set of default handlers defined in the API [3].

   1) Security handler/ APIAuthenticationHandler
     The security handler is to validate the Oauth token used to invoke the API.
  2) Throttle handler/ APIThrottleHandler
     Throttles the request based on the throttle policy. This is done based on two counts, 
     the global count and the local count.

   3) Stats hander/ APIMgtUsageHandler
     Helps to push data to the BAM for analytics.
   4) Google analytics handler/ APIMgtGoogleAnalyticsTrackingHandler
      Pushes events to Google analytics for analytics.

   5) Extentions handler/ APIManagerExtensionHandler
     Executes extention's sequences


 Each of these handler classes has 2 methods
  1) handle request
  2) handle response

 These methods have been overridden in each of these handlers to accomplish the task that the handler is written for. There is also a possibility that you could write your own handler, and plug it in. Details on this could be found at [4].

I hope you were able to gain a basic knowledge on what happens internally when an API is invoked on the WSO2 APIM, on a highlevel.  By going through the references you would be able to gain a much detailed knowledge on the APIM gateway and WSO2 APIM as a whole.



References

[1] https://docs.wso2.com/display/AM180/Token+API
[2] https://docs.wso2.com/display/AM180/Configuring+Caching
[3] https://docs.wso2.com/display/AM180/Writing+Custom+Handlers
[4] https://docs.wso2.com/display/AM180/Writing+Custom+Handlers#WritingCustomHandlers-Writingacustomhandler

[WSO2 APIM] - The Basic Architecture of WSO2 APIM - Part 3- The Store - under the hood

 In continuation to my previous post two posts on the The Basic Architecture of WSO2 APIM - Part 1  &   The Basic Architecture of WSO2 APIM - Part 2- The Publisher - under the hood  , this post is going to briefly discuss the architecture of the API Store component.  These posts are mainly targeted for dummies on WSO2 APIM. The aforementioned posts are a recommended read in order to understand this post clearly.

First off all, what is the API Store?

The API store is the play ground for the API consumer. An API consumer can self-register,  discover on API functionalities, subscribe to the API's, evaluate and interact with API's.

The APIM store is also a jaggery web app.

Databases associated with the API Store

 The database configuration can be found at <APIM_HOME>/repository /conf/datasources/master-datasources.xml

    - Registry Database
      The db handles all registry related storage.
    - AM_DB
       This db stores information required for the APIM.
           * The AM_ tables store information related to API's.
           * The IDN_ tables store information related to OAuth Identity.
      

View API

The APIM store is multi tenanted, and hence, at login, if multiple tenants are registered, it will prompt to select the tenant.

Upon selection of tenant, the API's which are published by that tenant can be viewed on the API store. Here the APIM-Store will extract API's to display from the registry_db through a cache.

When logged in, a user can view the API in a much detailed version and also edit it if permission is granted.

The store has a feature where it shows the recently added api's for convenience.

 An Application & Subscribe to an API

An Application in the APIM world is a concept of detaching API's from consumers.

An Application is a single entity to which api's can be subscribed to. This application is created on the AM_DB database and when the api is subscribed the subscription is also recorded on the APIM.

This application is then the unit of reference in the APIM store.

According to WSO2 docs an application is defined as ,

An application is primarily used to decouple the consumer from the APIs. It allows you to :

• Generate and use a single key for multiple APIs
• Subscribe multiple times to a single API with different SLA le

 Creating Keys

Creating keys in order to invoke an API can be done in the APIM store. Once the application is created, we can create tokens for that application.

When it comes to tokens, we can create application tokens (access tokens) and application user tokens. The types are APPLICATION & APPLICATION_USER. Access tokens are tokens which the application developer gets.

In the APIM Store when we create these tokens, we get the consumer_key and consumer_secret which is per application. The APIM store will talk to the APIM Key manager (in future releases there will be the capability of plugging in a custom Key manager, but for the time being it is either only the WSO2 APIM key manager or WSO2 IS as a keymanager) and the key manager will generate the keys. These keys will be stored in the AM_DB as well.

The tokens generated are associated to the application with a validity period.

 Note : Consumer key is analogous to a user name and the consumer secret it analogous to a user password.

Regenerating Tokens

The WSO2 APIM allows the regeneration of access tokens. In this process  there are 3 steps that will be executed.

 1) The existing token will be revoked.
 2) A new token will be generated
 3) The validity period of the token will be updated in the database.

When these three steps are performed the new generated token will be returned.


Indexing 

The store indexes the API's in order for better performance. the index info can be found at /solr location in the registry. Information on this can be found at [1].

Workflows 

In the APIM store internals, many workflows are used. One such example of this is the  subscription creation.

In the subscription creation a workflow executor is used. This workflow executors have two methods.

     1. execute.


     2. complete

 In order for it to be more clear on the workflow implementations, let me bring up a diagram explaining it.

The implementation of the workflow could take two paths.

1) The default implementation
     this is where the method "complete" is directly called within the method "execute" method.
2) A custom workflow definition.
   for example's sake we are using WSO2 BPS as the workflow execution unit here.  We need to write a custom workflow executor and use it in the APIM. Via a web service the external workflow will be executed.  (for the soap service, the call back URL would be the BPS url)

When a workflow is executed, the workflow detail is being put into the AM_WORKFLOWS in the AM_DB, and the status of the workflow is being moved to ON_HOLD. Once the complete is called, it updates the status to either APPROVED or REJECTED.

More detailed information on this can be found at [2] & [3].



References


[1] https://docs.wso2.com/display/AM180/Add+Apache+Solr-Based+Indexing
[2] https://docs.wso2.com/display/AM180/Customizing+a+Workflow+Extension
[3] https://docs.wso2.com/display/AM180/Adding+an+API+Subscription+Workflow
[4] https://docs.wso2.com/display/AM180/Quick+Start+Guide#QuickStartGuide-Introductiontobasicconcepts