Wednesday, July 15, 2015

[WSO2 APIM ] [Gateway] How to limit traffic to WSO2 APIM Gateway using API Handlers

Recently I had to find a solution to limit the traffic that passes through the WSO2 APIM Gateway. The requests that are allowed to pass through are the requests from the testing team. The architecture of the system is quite straight forward. Please find the drawing of the system architecture in brief below.
When it comes to limiting the traffic to the Gateway, there are several ways to handle this. This can be handled from the network level (e.g. routing rules) or even simply starting the server on a another port that is not visible to the outside world. Here, in this article I am going to describe how to do it from the Gateway. Using this method is only another option to achieve the goal.

 When an API is created from the WSO2 APIM, using the publisher, the publisher will propagate the changes to the WSO2 APIM Gateway. By this it means it will create a synapse configuration on the Gateway. This synapse configuration of the API will hold a set of handlers. These handlers are placed to achieve different functionality. For e.g. the APIAuthenticationHandler is intended to validate the token. More information on the handlers can be found at [1]. The Handlers are placed in an order in the synapse configuration, and they will be executed in the order they appear. Since the first point of contact in the API is their handlers, we can use a handler to filter out the request if it is from the testing device or not.

 To achieve this, we need to have an identifier being sent from the request. If we have this, it is easily possible to filter them out. First things first, we need to figure out what the identifier is. In my case, the requests sends the device ID in the header, under the parameter "Auth". So in my handler I will read the header and check for this Auth value.
 How do I tell which device Id's are able to continue? For this, I will read the device Id's from a system property so that the allowed device Id's could be passed from the command line as a system property when the server is start up.

 Okay, so given a brief description of what we are going to achieve, lets see how we can do this.

1. Create the Custom Handler. To create the custom handler we need to create a maven project, and create a new class. This class must be extending the "". Please find the sample code below.

package org.wso2.CustomHandler;

import java.util.Map;

import org.apache.synapse.MessageContext;
import org.apache.synapse.core.axis2.Axis2MessageContext;

public class GatewayTrafficContrlHandler extends AbstractHandler {

 public boolean handleRequest(MessageContext arg0) {
  String deviceId = null;
  String identifier = null;
  String[] identifiers;

  //Obtain the identifier .
  Map headers = (Map)((Axis2MessageContext) arg0).getAxis2MessageContext().
  identifier = (String) headers.get("Auth");

  if (identifier == null || identifier.isEmpty()) {
   //Get the first identifier which is the device ID.
   identifiers = identifier.split("\\|");
   if (identifiers != null && identifiers.length > 0) {
    deviceId = identifiers[0];

   //Get the device ID list  which is passed as a system property from                        //command line.Only these deviceId's will be allowed to pass 
                        //through the handler.
   String[] supportedDeviceIdList = null;
   String supportedDeviceIds = System.getProperty("deviceIds");
   if (supportedDeviceIds != null && !supportedDeviceIds.isEmpty()) {
    supportedDeviceIdList = supportedDeviceIds.split(",");

   //Check if the device Id which is sent in the request is in the 
                        //list of deviuce Id's passed as a system property.
   if (supportedDeviceIdList != null && supportedDeviceIdList.length > 0) {
    for (int index = 0; index < supportedDeviceIdList.length; index++) {
     if (supportedDeviceIdList[index].equals(deviceId) == true) {
      return true;
  return false;

 public boolean handleResponse(MessageContext arg0) {
  // TODO Auto-gGatewayTrafficContrlHandlerenerated method stub
  return false;


2. Once we create the Class to extract and find for the identifier, we then need to build the jar.
3. Copy the created Jar to the /repository/components/lib folder. 
4. Then start the APIM server with the system property -DdeviceIds, and log into the APIM      
     Gateway's management console. for e.g :- ./ -DdeviceIds=123,456,789 
5. Go to Service Bus > Source View in the Main menu.
6. In the configuration select the API, and then check the Handler section. In the handler section, add 
    the class to point to the newly created handler.

        <handler class="org.wso2.CustomHandler.GatewayTrafficContrlHandler"/>
        <handler class=""/>
        <handler class="org.wso2.carbon.apimgt.gateway.handlers.throttling.APIThrottleHandler">
            <property name="id" value="A"/>
            <property name="policyKey" value="gov:/apimgt/applicationdata/tiers.xml"/>
        <handler class="org.wso2.carbon.apimgt.usage.publisher.APIMgtUsageHandler"/>
        <handler class="org.wso2.carbon.apimgt.usage.publisher.APIMgtGoogleAnalyticsTrackingHandler"/>
        <handler class="org.wso2.carbon.apimgt.gateway.handlers.ext.APIManagerExtensionHandler"/>

7. Make sure the newly created handler is the first handler in the list.
8. Once we do the change save, and observe the console for reloading of the API. 
9. Test the Handler by doing a REST API call.

And that's it. 

 The above solution will only let the requests with deviceId's 123 or 456 or 789 to pass through.



Tuesday, May 19, 2015

[WSO2-ESB] URI- template encoding in WSO2 ESB for reserved characters

 In a recent engagement with the WSO2 ESB, I've come across a situation where WSO2 ESB does not encode some reserved characters when using a URI-template as the endpoint to a get call.

In this particular instance it was the '&' that was not getting encoding and the <space>.

The reason is that WSO2 ESB uses the libraries  and it ends up not encoding the characters correctly.

As a solution to this, we can use the script mediator in WSO2 ESB. We could get the value in the script mediator and encode it using the  javascript encode uri method, and encode our values correctly.

An example as shown below:

<?xml version="1.0" encoding="UTF-8"?>
    <resource methods="POST">
            <log level="full"/>
            <property name="uri.var.fullname" expression="//APIRequest/Fullname/text()"/>
            <property name="uri.var.address" expression="//APIRequest/HomeAddress/text()"/>
            <script language="js">var fullname = mc.getProperty('uri.var.fullname');var homeAddress = mc.getProperty('uri.var.address');                 mc.setProperty("uri.var.fullname",encodeURIComponent(fullname));                 mc.setProperty("uri.var.address",encodeURIComponent(homeAddress));                 </script>
            <header name="Content-Type" scope="transport" action="remove"/>
                    <http method="get"
In this what is done is that the API request values has been extracted and stored in a property using a property mediator.

Then by using the script mediator  we make use of javascript and store the value as a var and encode it using the "encodeURIComponent" method.

In the "encodeURIComponent" yet, we still have some characters that are not encoded. These include the !'()*~.


Tuesday, April 21, 2015

Wired network not detected in ubuntu 14.04

I was at a customer site, and it was required that I connect to their wired network. Everyone in the customer site could connect to the wired network, except me... :) Why? well probably two reasons, I had a ubuntu machine where as everyone else had windows, and I had a machine I used at WSO2,  with WSO2 network.

Hence, I googled a bit, and figured out the issue. The issue in my case was that the /etc/network/interfaces file had some unwanted lines in there.

If you do happen to come across the same issue, you could try this as well.

Comment or delete any lines in the /etc/network/interfaces file except for the lines 

auto lo
iface lo inet loopback

I personally would comment.. :)



Waiting for network configuration at boot up in ubuntu (14.04)

Today I was meddling around with the network configurations as I could not connect to a wired network, and with something I did, when I restarted I was keeping on getting the message "Waiting for network configuration at boot up in ubuntu" and then wait upto 60 seconds, and start up without any network manager settings.

Because of this I could not connect to a network, and the network manager was literally stalled.

After looking at my previous steps, hitting the history command


it shows that I was meddling with the resolve.conf file. But when I looked at the file, it showed that no DNS nameservers were defined in the file. The file was EMPTY.... PANIC!!!!

So then I tried setting the nameserver in the /etc/resolve.conf file, however it was getting replaced.

The solution was,

1) go to /etc/resolveconf/resolv.conf.d/
      # cd  /etc/resolveconf/resolv.conf.d/
2) Open the base file n super user mode
     # sudo vim base
3) add the nameserver details
    e.g : nameserver
4) Save file, and restart.

This should fix the issue.



Monday, April 20, 2015

[WSO2 APIM] - The Basic Architecture of WSO2 APIM -part 5- Statistics in APIM

In this blog post I will briefly go about how the APIM publisher and store are able to draw up graphs with statistical information on the API's. This blog post is targetted for new bee's who would like to get a bird's eye view on the functionality.

In the WSO2 APIM Gatway, as I explained in my previous post The Basic Architecture of WSO2 APIM - part 3- Gateway & Key Manager - Under the hoods, it has a set of handlers defined per API.  One of these handlers is the APIMgtUsageHandler , which would invoke the org.wso2.carbon.apimgt.usage.publisher.APIMgtUsageDataPublisher .  The APIMgtUsageDataPublisher  is configured to publish the events to the BAM server that has been configured with the APIM. 

Illustrated by a diagram is the process of publishing of stats and viewing them on the publisher /store app's 

The WSO2 APIM Gateway would  publish the events  via the thrift protocol to the WSO2 BAM server. The BAM is more likely a BAM cluster in the real world production environment. The BAM server then writes the data to a No Sql database, cassandra.  Then the BAM Analyzer to which the APIM tool box is deployed to, will fetch the data batch by batch from the  cassandra database.  The BAM Analyzer  is capable of summarizing the data, by using hive. The hive scripts have to pre written and deployed in the server. The BAM analyzer then will push the summarized data to an RDBMS.

The WSO2 APIM Store and the WSO2 Publisher then will pull the data from the RDBMS  and will display the data in the APIM Store and publisher analytics pages.

This is a very brief explanation of what happens in the APIM when statistics are to be displayed.

Detailed information can be found  at the references listed below.




[WSO2 APIM] - The Basic Architecture of WSO2 APIM - part 4- Gateway & Key Manager - Under the hoods

In this post I will briefly introduce how the Gatway and the Key manager interacts in order for an API to be successfully invoked.

Let me bring up a diagram first.

In a real world distributed deployment, only the WSO2 APIM Gateway would be  exposed to the outside world. So with the creation of keys for a certain application that would be subscribed to many API's, (If you are not familiar with this context, please refer to the blog post I had written before The Basic Architecture of WSO2 APIM - Part 2- The Store - under the hood  for a crash course) you would be able to invoke an API via the gateway.

Ideally an Application would have the consumer_key and the consumer_secret hardcoded in the application it self. When invoking the API, the username, password would have to be added and the application would pass in the username, password, consumer_key and consumer_secret  to the gateway. The Gateway has some token API's [1]. which are
   - /token
   - /revoke

When calls are made to these API's,  the gateway calls the Key manager Jax-RX  in order to verify the access the token.  The key-manager would call the AM_DB, retrieve the access token, and verify the access token.  it will return the API Info DTO which includes the meta data of the access token, which includes the validity period, refresh token and scopes.

When the API's are invoked, the Gateway does not call the Key manager at every invocation.  The APIM Gatway makes use of a cache  for this. However, this cache can be turned off. [2]

Invoking the API

When the API's are invoked there are several authorization grant types that we could use. The WSO2 APIM supports the same grant types the WSO2 IS supports.  The password grant type is used only when the application is a trusted application  Client Credentials grant type requires only the consumer_key & consumer_secret. [1].

API Handlers

When the API's is created it will store the synapse configuration of the API in the WSO2 APIM Gateway, being another ESB (WSO2 ESB).  When the api is invoked and it hits the Gateway, it will execute the API's in and out sequences  which would have a set of handlers. The API when created would have a set of default handlers defined in the API [3].

   1) Security handler/ APIAuthenticationHandler
     The security handler is to validate the Oauth token used to invoke the API.
  2) Throttle handler/ APIThrottleHandler
     Throttles the request based on the throttle policy. This is done based on two counts, 
     the global count and the local count.

   3) Stats hander/ APIMgtUsageHandler
     Helps to push data to the BAM for analytics.
   4) Google analytics handler/ APIMgtGoogleAnalyticsTrackingHandler
      Pushes events to Google analytics for analytics.

   5) Extentions handler/ APIManagerExtensionHandler
     Executes extention's sequences

 Each of these handler classes has 2 methods
  1) handle request
  2) handle response

 These methods have been overridden in each of these handlers to accomplish the task that the handler is written for. There is also a possibility that you could write your own handler, and plug it in. Details on this could be found at [4].

I hope you were able to gain a basic knowledge on what happens internally when an API is invoked on the WSO2 APIM, on a highlevel.  By going through the references you would be able to gain a much detailed knowledge on the APIM gateway and WSO2 APIM as a whole.



Friday, April 17, 2015

[WSO2 APIM] - The Basic Architecture of WSO2 APIM - Part 3- The Store - under the hood

 In continuation to my previous post two posts on the The Basic Architecture of WSO2 APIM - Part 1  &   The Basic Architecture of WSO2 APIM - Part 2- The Publisher - under the hood  , this post is going to briefly discuss the architecture of the API Store component.  These posts are mainly targeted for dummies on WSO2 APIM. The aforementioned posts are a recommended read in order to understand this post clearly.

First off all, what is the API Store?

The API store is the play ground for the API consumer. An API consumer can self-register,  discover on API functionalities, subscribe to the API's, evaluate and interact with API's.

The APIM store is also a jaggery web app.

Databases associated with the API Store

 The database configuration can be found at <APIM_HOME>/repository /conf/datasources/master-datasources.xml

    - Registry Database
      The db handles all registry related storage.
    - AM_DB
       This db stores information required for the APIM.
           * The AM_ tables store information related to API's.
           * The IDN_ tables store information related to OAuth Identity.

View API

The APIM store is multi tenanted, and hence, at login, if multiple tenants are registered, it will prompt to select the tenant.

Upon selection of tenant, the API's which are published by that tenant can be viewed on the API store. Here the APIM-Store will extract API's to display from the registry_db through a cache.

When logged in, a user can view the API in a much detailed version and also edit it if permission is granted.

The store has a feature where it shows the recently added api's for convenience.

 An Application & Subscribe to an API

An Application in the APIM world is a concept of detaching API's from consumers.

An Application is a single entity to which api's can be subscribed to. This application is created on the AM_DB database and when the api is subscribed the subscription is also recorded on the APIM.

This application is then the unit of reference in the APIM store.

According to WSO2 docs an application is defined as ,

An application is primarily used to decouple the consumer from the APIs. It allows you to :
  • Generate and use a single key for multiple APIs
  • Subscribe multiple times to a single API with different SLA le
 Creating Keys

Creating keys in order to invoke an API can be done in the APIM store. Once the application is created, we can create tokens for that application.

When it comes to tokens, we can create application tokens (access tokens) and application user tokens. The types are APPLICATION & APPLICATION_USER. Access tokens are tokens which the application developer gets.

In the APIM Store when we create these tokens, we get the consumer_key and consumer_secret which is per application. The APIM store will talk to the APIM Key manager (in future releases there will be the capability of plugging in a custom Key manager, but for the time being it is either only the WSO2 APIM key manager or WSO2 IS as a keymanager) and the key manager will generate the keys. These keys will be stored in the AM_DB as well.

The tokens generated are associated to the application with a validity period.

 Note : Consumer key is analogous to a user name and the consumer secret it analogous to a user password.

Regenerating Tokens

The WSO2 APIM allows the regeneration of access tokens. In this process  there are 3 steps that will be executed.

 1) The existing token will be revoked.
 2) A new token will be generated
 3) The validity period of the token will be updated in the database.

When these three steps are performed the new generated token will be returned.


The store indexes the API's in order for better performance. the index info can be found at /solr location in the registry. Information on this can be found at [1].


In the APIM store internals, many workflows are used. One such example of this is the  subscription creation.

In the subscription creation a workflow executor is used. This workflow executors have two methods.

     1. execute.

     2. complete

 In order for it to be more clear on the workflow implementations, let me bring up a diagram explaining it.

The implementation of the workflow could take two paths.

1) The default implementation
     this is where the method "complete" is directly called within the method "execute" method.
2) A custom workflow definition.
   for example's sake we are using WSO2 BPS as the workflow execution unit here.  We need to write a custom workflow executor and use it in the APIM. Via a web service the external workflow will be executed.  (for the soap service, the call back URL would be the BPS url)

When a workflow is executed, the workflow detail is being put into the AM_WORKFLOWS in the AM_DB, and the status of the workflow is being moved to ON_HOLD. Once the complete is called, it updates the status to either APPROVED or REJECTED.

More detailed information on this can be found at [2] & [3].



Unable to get the hostName or IP address of the server when starting WSO2 Products

Today I faced an issue when a wso2 server that needed restarting did not start up, after shutting down due to the error

{org.apache.synapse.ServerConfigurationInformation} -  Unable to get the hostName or IP address of the server {org.apache.synapse.ServerConfigurationInformation} <hostname> : <hostname> : Name or service not known
at org.apache.synapse.ServerConfigurationInformation.initServerHostAndIP(
at org.apache.synapse.ServerConfigurationInformation.<init>(
at org.apache.synapse.ServerConfigurationInformationFactory.createServerConfigurationInformation(
at org.wso2.carbon.mediation.initializer.ServiceBusInitializer.initESB(
at org.wso2.carbon.mediation.initializer.ServiceBusInitializer.activate(

The issue in my case was that it was not a wso2 bug. It is an issue with the machine.

The OS I was using was ubuntu. The issue was that the hostname was not correctly set.

The easy way  - temporary fix

1) Run the following command. (assuming the hostname you want to set is abc001)

                    #    hostname abc001

2) To verify the hostname just run the following command

                           # hostname

it should output the hostname that we set as the output.

Proper fix

1) We need to fix this properly. In that case, we need to change the /sysconfig/network file.

open this file as sudo or super user, and change the hostname in that file to be the hostname you require.


2) Then you need to update the etc/hosts file, to reflect this change. you should map the entry


                      <local-ip> abc001

3) After that you need to restart the network service.

                   #service network restart
                   #/etc/init.d/network restart

4) Then verify the hostname has been set by running the command

                    # hostname



Wednesday, February 25, 2015

[WSO2 APIM] - The Basic Architecture of WSO2 APIM - Part 2- The Publisher - under the hood

 As a continuation of my previous post  The Basic Architecture of WSO2 APIM - Part 1, this post will concentrate on the publisher module, and its major functions. It is recommended that you read the aforementioned post in order to clearly understand the concepts described in this post.

The Publisher is the starting point of an API life cycle. An API is created at the API Publisher by an API publishers. The Publisher is also a major component in the WSO2 APIM and was introduced in my earlier post.

To start off with let me bring up the basic outline of the Publisher in a graphical mode.

 An API Publisher will ideally interact with the API Publisher module to create and manage an API. The API Publisher is a jaggery web app which provides and easy to work with GUI.

Databases associated with the API Publisher

 The database configuration can be found at <APIM_HOME>/repository /conf/datasources/master-datasources.xml

    - Registry Database
      The db handles all registry related storage.
    - AM_DB
       This db stores information required for the APIM.
       This db stores information such as user permissions.

Logging in 

The end user will log into the API Publisher by entering the username and password in the GUI provided by the web app. Once the credentials are entered the API Publisher will validate it against the authentication manager set in the api-manager.xml found at repository/conf. It is defined under the tag <AuthManager>. This is the server which the Publisher and the store will point to for authentication. By default it will be the embedded LDAP which is the user store which is being pointed by localhost which is started on the APIM server it self. In a typical production deployment this could either be WSO2 IS, or any other IDP (identity provider).

When the end user logs into the API Publisher in a SSO configured environment, it will send a SAML request to the Authentication Server. The Authentication server will process the authentication and send a signed response. The API Publisher then will validate this with the key stores.

Once the authentication is completed it checks for authorization in the UM_DB by checking for publisher permissions. Once the authentication and the authorization succeeds it will succeed in allowing the user to access the API Publisher.

Creating an API

There are two ways of creating an API.
   1. By Filling out the form in the API Publisher web app
   2. By Importing a Swagger doc.

When the API Publsher creates an API, the API information is stored in the Registry Database. This is specifically stored at the governance space of the registry database.Together with the API the swagger document related to the created API is also stored in the Registry database.

When the API is created a synapse config of the API is also stored at the API Gateway. Since the API Gateway is a WSO2 ESB, the API is also capable of having sequences defined in the API definition.

         </custom in sequence>
          </custom out sequence>

In parallel to this, when the APi is created, the AM_DB is also updated. A reference to the API is stored in this database tables. Since we have all the information of the API in the REG_DB, we only store a subset of the API information in the AM_DB, which is required for AM functionality.

When an API is created it is by default in the "created" state.

I hope this gives you a clear understanding on what happens in the publisher end, under the hoods.

In my next few blog posts I am planning to discuss on the  rest of the modules, API Store, and  API Gateway. these will cover the Key manager as well.


Wednesday, February 18, 2015

[WSO2 APIM] - The Basic Architecture of WSO2 APIM - Part 1

After a long time I decided to write a blog post, and this time about APIM. Mainly because my new team is the WSO2 APIM . Today I received a training from Nuwan Dias (WSO2 APIM team) on the basic architecture of APIM, and that training inspired me to come up with this article.

To start off with, yes I did have a prior knowledge on the WSO2 APIM and have worked on it on various instances. But the training I received today cleared many doubts and managed to organize my knowledge on the WSO2 APIM very clearly. Hence, this article is focused for dummies, to gather a basic idea on the WSO2 APIM architecture.

So, before I start there are some main components of WSO2 APIM which I will introduce briefly.

 1) The API Publisher
 2) The API Store
 3) The API Gateway
 4) The Key Manager

The Publisher

This is the component that is responsible for the work flows associated with API creation. This is where an API developer will have most of the work to do. API Publisher is a jaggery application (Jaggery is a framework to write webapps and HTTP-focused web services for all aspects of the application: front-end, communication, Server-side logic and persistence in pure Javascript). The publisher allows the creation of the API by allowing the developer to fill in a form. This form consists of many details such as, the API name, version ,throttling level etc.

There are basically two ways of creating an API.
  •  Filling out the form in create API  on the WSO2 APIM.
  •  By importing a Swagger document (Swagger is a simple yet powerful representation of your RESTful AP) to WSO2 APIM.      
API's created in this component initially is of the Created State. An API is associated with many states.

 The Store

The Store is the component that exposes the published API's. The API's which are created in the publisher needs to be pushed to published state in order for the user to view the API's in the API Store. The API store is responsible for maintaining the API access. Whilst the API Store is responsible for many other activities, I would like to believe that it's main responsibility is to allow users interact with the API's and  subscribe to them.

How the subscription and access is maintained is that the Store will show API's in a tenant domain, when the domain is selected upon navigating to the Store by a user.As mentioned before, these API's are only API's which are in the published state. For a user to access this API, the user requires to create an Application in the API Store. In a practical scenario, let's say there's an API to get weather information, and in my application I need to get the data for a functionality I intend to deliver. Then I would create an application, which would be an OAuth application, in the APIM Store. For a API I can subscribe using this created application which would maintain the subscriptions to API's. when this is done, I can create a token. This token will be a consumer secret and consumer key pair, which are in the world of OAuth terms for something similar to username/password. With this  the store also provides me an access token, which is a APPLICATION type token. With these tokens an application can access the subscribed API and obtain the service.

The API store, just like the Publisher is a Jaggery web application.

The Gateway

The gateway is the access point for a external user to obtain the services offered by the API.

When the Publisher publishes the API a synapse config (Apache Synapse is a lightweight and high-performance Enterprise Service Bus (ESB)) of the API is sent to the Gateway. In a typical production deployment this is the component that will be exposed. The gateway its self is an ESB, and hence sequences can be enforced on these API's. The API when invoked would be running this synapse config, and this config would have a set of handlers which are being called in sequence.These handlers are:
  •  Security handler
  •  Throtling handler
  •  Stats Handler
  •  Google analytics Handler (disaled by default)
  •  Extentions Handler
These handlers are required to run in sequence because each handler populates the message context in the synapse and passes it on, which is required by the next handler in sequence. The gateway has endpoints for token and revoke which are proxies for the actual endpoint which is embedded into the keymanager component.

The Keymanager

The Keymanager is the component which handles the authorization/authentication part. The Keymanager can be configured to be the WSO2 IS as well. The API manager Key manager handles the security related aspects and issues and verifies keys, to the store and from the gateway.

There is a JAX-RS web app embedded into the key manager which has the implementations for token endpoints and revoking tokens. These are the endpoints to which the gateway calls upon requesting tokens.

With these four components being introduced, I will next discuss on how a API is published, how it is involved in the store, gatway and keymanaer in my next blog post.

Till then, I hope this gives you a heads up on the API Manager Basic architecture and hope you read about it more. If Interested I have noted references below for your use.