Use cqlsh terminal to browse cassandra | WSO2 ES 1.0.0

We have used WSO2-BAM’s event stream architecture to store data on Cassandra for social aspects of the WSO2 Enterprise Store.

I have to do few direct Cassandra querying while adding a new feature to the social application within ES distribution. Find following steps I followed in this activity.

1. Download Apache Cassandra 1.1.3 from here[1].
2. Unzip and browse apache-cassandra-1.1.3/bin directory using terminal.
3. Give following command to get the cqlsh terminal. Since I'm using default configuration in my local deployment I'm using admin as the both username and password, localhost as the hostname and 9160 as the port.

./cqlsh localhost 9160 -u admin -p admin

udara@thinkPad:~/Downloads/apache-cassandra-1.1.3/bin$ ./cqlsh localhost 9160 -u admin -p admin
Connected to Test Cluster at localhost:9160.
[cqlsh 2.2.0 | Cassandra 1.1.3 | CQL spec 2.0.0 | Thrift protocol 19.32.0]
Use HELP for help.
cqlsh>

4. Select key-space, in my deployment "EVENT_KS".

cqlsh> use EVENT_KS;

5. Select data from the column family "org_wso2_social_activity".

cqlsh:EVENT_KS> SELECT * FROM org_wso2_social_activity;

To learn more CQL follow this official document[2].

[1]. http://archive.apache.org/dist/cassandra/1.1.3/
[2]. http://cassandra.apache.org/doc/cql/CQL.html

PostgreSQL- Basics

I'm compiling this post to create a reference which any postgreSQL-newbie (like myself) can use to get the basic things right !

You need a postgreSQL server up and running before moving further.

sudo apt-get install postgresql

1. Update postgres user password.In a terminal run following and give the password when prompted.

sudo -u postgres psql postgres
\password postgres

2.  Create Database with the name wso2bam. In a terminal run,

sudo -u postgres createdb wso2bam

3.  Connect to psql- the terminal based frontend,

sudo -u postgres psql postgres

 after you run the above command you will get a terminal with postgres=#.

root@thinkPad:~# sudo -u postgres psql postgres
psql (8.4.19)
Type "help" for help.

postgres=#

Note -: Here onwards I'm going to execute all commands within this psql terminal.

4. List down databases(similar to MySQL show databases.

\list

5. Select database wso2bam.

\c wso2bam

6. List tables within wso2bam current database.

\dt

If you are not interested in remembering and typing all these commands, pgAdmin3 might make your life easy. You can install this tool by running,

sudo apt-get install pgadmin3

You can use following official documentation to try out basic queries[1].

Please note that I have tested and verified above steps on a Ubuntu 12.10 machine, using  both PostgreSQL 8.4 and 9.1 servers.

[1]. http://www.postgresql.org/docs/8.4/static/tutorial-select.html

Install PostgreSQL 8.X on Ubuntu 12.10

If you try to install PostgreSQL on ubuntu using apt-get install postgresql command it will automatically fetch and setup PostgreSQL 9.X.

If following methods are not any option to install PostgreSQL you may continue this blog post.

1. Build using the source code.
2. Downloading deb package and install.

Lets configure our system by updating source list so we can simply install  PostgreSQL using apt-get install postgresql-8.X :)

First create pgdg.list file inside /etc/apt/sources.list.d/ directory with the following content.

deb http://apt.postgresql.org/pub/repos/apt/ precise-pgdg main

Next import sigining key for this repository by executing following command in terminal.

wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -

You are done, simply run "sudo apt-get update" before installing
PostgreSQL 8.X.

Set up WSO2 BAM with Cassandra cluster

First blog post for year 2014, in this post I'm going to set up a single WSO2-BAM node to work with a cluster of three Cassandra nodes. To better understand the setup please refer following diagram.

I have 3VM's with Ubuntu up and running inside my virtual box, VM1 with IP address 192.168.56.101, VM2 & VM3 with 192.168.56.102,192.168.56.103.

Lets download and set up our Cassandra cluster first. You can find latest version of Cassandra from here[1]. Note that I'm going to use Cassandra 1.1.3 for this exercise. Lets assume we have apache-cassandra-1.1.3 directory within /home/udara/cassandra/ directory on all three nodes.

First im going to configure Cassandra inside node1. Lets browse apache-cassandra-1.1.3/conf/ directory and open cassandra.yaml in a text editor. Make sure to update following parameters according to your environment. Before updating we need to generate tokens for our Cassandra ring using the tokengentool here[2].

cluster_name: Test Cluster
initial_token: 0
seed_provider:        
- seeds: "192.168.56.101" 
listen_address: 192.168.56.101
rpc_address: 192.168.56.101
rpc_port: 9160

Update cassandra.yaml within our other nodes as well(VM2 and VM3).

cluster_name: Test Cluster
initial_token: 56713727820156410577229101238628035242
seed_provider:        
- seeds: "192.168.56.101" 
listen_address: 192.168.56.102
rpc_address: 192.168.56.102
rpc_port: 9160

cluster_name: Test Cluster
initial_token: 113427455640312821154458202477256070485
seed_provider:        
- seeds: "192.168.56.101" 
listen_address: 192.168.56.103
rpc_address: 192.168.56.103
rpc_port: 9160

Now we are good to init our cluster, just run bin/cassandra -f within  cassandra-1.1.3 directory in all three nodes. But in my setup I have to fulfill some other requirements as well.

1. Update JVM_OPTS="$JVM_OPTS -Xss280k" within conf/cassandra-env.sh.
2. Create /var/log/cassandra/system.log file & /var/lib/cassandra/ directories.

Lets check the status of our Cassandra cluster. Fulfill this task I'm going to use NodeTool by default shipped with Cassandra.

Run /nodetool -h 192.168.56.101 ring within apache-cassandra-1.1.3/bin directory,  and you will see something similar to this.

node1@node1-VirtualBox:~/apache-cassandra-1.1.3/bin$ ./nodetool -h 192.168.56.101 ring
Address         DC          Rack        Status State   Load            Effective-Ownership Token                                      
                                                                                                                                            6148914691236517205                        
192.168.56.101  datacenter1 rack1       Up     Normal  838.62 KB       100.00%             0                                          
192.168.56.102  datacenter1 rack1       Up     Normal  815.04 KB       100.00%             3074457345618258602                        
192.168.56.103  datacenter1 rack1       Up     Normal  823.84 KB       100.00%             6148914691236517205


Lets configure of BAM node inside VM1 now, download WSO2-BAM latest version from here[3]. I'm going to use WSO2 BAM 2.4.0 latest at the moment.

Update <BAM_HOME>/repository/conf/etc/cassandra-component.xml with the following content.Configure nodes correctly according to your environment.

<Cassandra>
    <Cluster>
       <Name>Test Cluster</Name>
       <DefaultPort>9160</DefaultPort>
       <Nodes>192.168.56.101:9160,192.168.56.102:9160,192.168.56.103:9160</Nodes>
       <AutoDiscovery disable="false" delay="1000"/>
    </Cluster>
</Cassandra>

Update <BAM_HOME>/repository/conf/advanced/streamdefn.xml with the following content.

<StreamDefinition>
    <NodeId>1</NodeId>

    <keySpaceName>EVENT_KS</keySpaceName>
    <eventIndexKeySpaceName>EVENT_INDEX_KS</eventIndexKeySpaceName>

    <ReplicationFactor>3</ReplicationFactor>
    <ReadConsistencyLevel>QUORUM</ReadConsistencyLevel>
    <WriteConsistencyLevel>QUORUM</WriteConsistencyLevel>
    <StrategyClass>org.apache.cassandra.locator.SimpleStrategy</StrategyClass>
</StreamDefinition>

For load-balancing purpose you can define WSO2BAM_CASSANDRA_DATASOURCE datasource as, check for comma separated JDBC URLs.

<datasource>
            <name>WSO2BAM_CASSANDRA_DATASOURCE</name>
            <description>The datasource used for Cassandra data</description>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:cassandra://192.168.56.101:9160/EVENT_KS,jdbc:cassandra://192.168.56.102:9160/EVENT_KS,jdbc:cassandra://192.168.56.103:9160/EVENT_KS</url>
                    <username>admin</username>
                    <password>admin</password>
                </configuration>
            </definition>
</datasource>

Make sure to update hostname localhost with correct IP address, 192.168.56.101 in my setup inside <BAM_HOME>/repository/conf/datasources/master-datasources.xml.

Lets start WSO2 BAM now. Since we need to point BAM to the external Cassandra cluster rather than using embedded Cassandra, run following

sh wso2server.sh -Ddisable.cassandra.server.startup=true

This disable.cassandra.server.startup=true parameter will disable embedded Cassandra during carbon server startup.

Lets run few samples available here[4] and see our setup works properly,

[1] http://cassandra.apache.org/download/
[2] http://www.datastax.com/docs/1.1/initialize/token_generation
[3] http://wso2.com/products/business-activity-monitor/
[4] http://docs.wso2.org/display/BAM240/KPI+Monitoring+Sample

UES-Dashboard with WebSockets enable Gadgets

With the use of Websockets capability of Jaggery, you can create Websockets enabled gadgets within UES.

1. Create new directory for our gadget, name it dummy-chat.
2. Create your gadget XML file with the following content,

    <?xml version="1.0" encoding="UTF-8" ?>
    <Module>
        <ModulePrefs title="WS Dummy chat application"
                    height="350"
                    description="This is a dummy sample gadget to showcase Web socket capability"
                    tags="chat">
           <Require feature="dynamic-height"/>
        </ModulePrefs>
        <Content type="html">
           <![CDATA[
           <script src="/portal/gadgets/dummy-chat/js/dummy-chat.js" type="text/javascript"></script>
           <div id="gadget-wrapper">
                   <div id="msg-content"></div>
                   <div id="gadget-area-div">
                   <textarea rows="4" cols="50" id="msg"></textarea><br/>
                   <button type="button" onclick="send()">Send Me!</button>
                   </div>
           </div>
           ]]>
        </Content>
    </Module>

Name this XML file dummy-chat.xml and save the content inside our gadget directory.

3. Create directory to store JavaScript files of our gadget, name it js.
4. Create dummy-chat.js file with following content inside js directory.

    var url,ws;
    window.onload = function WindowLoad(event) {
       url = 'wss://localhost:9443/ws-chat/server.jag';
       ws = new WebSocket(url);
       //event handler for the message event in the case of text frames
       ws.onopen = function() {
       console.log("web Socket onopen. ");
       };

       ws.onmessage = function(event) {
       console.log("web Socket Onmessage from Server. " + event.data);
       var reply = document.getElementById('msg-content');
       reply.innerHTML = reply.innerHTML + '<br/>' + event.data;
       };

       ws.onclose = function() {
       console.log("web Socket onclose. ");

       };
    }

    //send msg to the server
    function send(){
       var msg = document.getElementById('msg');
       ws.send(msg.value);
       console.log("Client message "+msg.value);
       msg.value = '';
    }

5. If you wish to have eye-catching thumbnail and banner for your gadget, you can store them inside dummy-chat directory.Find our directory structure here,

.
├── banner.jpg
├── dummy-chat.xml
├── js
│   └── dummy-chat.js
└── thumbnail.jpg

We are done with the Websocket client implementation. Client application here provides a text field and button to send messages, and provides a place where messages from the server printed.

Lets move to the server implementation, This dummy server acknowledges all messages and will print the same message with some dummy text.

I’m using the same server implementation from jaggey Websocket doc[1] with a small modification.

1. Create a new jaggery app and name it ws-chat.

2. Create server.jag with following content inside ws-chat directory.

    <%
    var log = new Log();
    webSocket.ontext = function (data) {
        log.info('Client Sent : ' + data);
        var ws = this;
        setTimeout(function () {
           var currentDate = new Date();
           ws.send("message received !!");
           ws.send(currentDate+" : "+data);
        }, 1000);
    };
    
    webSocket.onbinary = function (stream) {
        log.info('Client Streamed : ' + stream.toString());

    };
    %>

Server will send “message received !!”  and the received message with timestamp to the client.

Lets move to the deployment of this gadget client and the server,

1. Copy ws-chat jaggery application and host it inside <UES_HOME>/repository/deployment/server/jaggeryapps/
2. Move dummy-chat gadget to <UES_HOME>/repository/deployment/server/jaggeryapps/portal/gadgets/
3. Start UES instance by running ./wso2server.sh from <UES_HOME>/bin directory.

You can verify availability of the new gadget by browsing, http://localhost:9763/store/assets/gadget/
Lets bookmark this gadget and create our dashboard from dummy-chat gadget.Sign-in to the portal from http://localhost:9763/portal and follow creating a dashboard guide from here[2].
Browse https://localhost:9443/<DASHBOARD_NAME>/ while logged into the UES and you can use dummy-chat gadget within the dashboard.

[1] http://jaggeryjs.org/apidocs/websocket.jag
[2] http://docs.wso2.org/display/UES100/Creating+a+Dashboard

Overcome "class path is too long" error while executing WSO2 ciper tool in windows

The cipher tool is there to encrypt and decrypt simple texts and by default bundled with all WSO2 products. You can simply use this tool by running either the shell script or batch script inside <PRODUCT_HOME>/bin directory according to your environment.

I'm not going to explain in detail about this tool, but you can find out more information from here[1].

In this blog post I'm going to explain a workaround to get rid of the error "The class path is too long for the windows shell to handle it" when you try to run ciphertools.bat -Dconfigure" command within windows environment.

Before continue further just take a back-up of the <PRODUCT_HOME>/bin/cipertools.bat file.

What you have to do is update line no 73 to 77 with,

call ant -buildfile "%CARBON_HOME%\bin\build.xml" -q
set CARBON_CLASSPATH=.\conf
FOR %%c in ("%CARBON_HOME%\lib\*.jar") DO set CARBON_CLASSPATH=!CARBON_CLASSPATH!;".\lib\%%~nc%%~xc"
FOR %%C in ("%CARBON_HOME%\repository\lib\*.jar") DO set CARBON_CLASSPATH=!CARBON_CLASSPATH!;".\repository\lib\%%~nC%%~xC"

This snippet will only add dependent jar files to the class_path and you can get rid of the above mentioned issue.


[1] http://docs.wso2.org/display/Carbon420/Encrypting+and+Decrypting+Simple+Texts

Access WSO2 Carbon admin Webservice using SoapUI

For the purpose of writing this blog post I'm going to use WSO2 GREG, SoapUI 4.5.1 on a linux environment.

In my previous blog post I explained how to explore a WSO2 Carbon admin Webservice, here I'm going to access an admin web service by providing authentication details.

I assume you are in the 3rd step of my previous example and able to list all admin services using OSGI console command listAdminServices.

If you able to access https://localhost:9443/services/UserAdmin?wsdl you are good to move further in this post.

1. Open new SoapUI project, lets name it UserAdmin provide above URL as the initial WSDL and press ok.

You will see all available SOAP requests under "UserAdmin" webservice.

2. Lets run getAllRolesNames, expand it and click on the Request1 link.

You will get the SOAP request generated by SoapUI with the use of provided WSDL.

3. Click on the Aut button under the request tab select authorization type as preemptive.

4. Provide admin username and password of your carbon instance.

5. Send SOAP request.

If you have done everything right you will get the list of available roles in your carbon server.

Further I will explain how to use the already generated Set-Cookie: JSESSIONID to authenticate next requests rather than providing username & password again.

Lets view raw response in our previous step, by pressing raw tab inside response window.You will see HTTP header details,

HTTP/1.1 200 OK
Set-Cookie: JSESSIONID=D9A45BE1CCD7616F88FDF4E0FF85C213; Path=/; Secure; HttpOnly
Content-Type: text/xml;charset=UTF-8
Transfer-Encoding: chunked
Date: Tue, 03 Dec 2013 10:40:15 GMT
Server: WSO2 Carbon Server

I'm going to send a getRolesOfUser SOAP request with the use of available Cookie: JSESSIONID=D9A45BE1CCD7616F88FDF4E0FF85C213.

Click Headers tab under the request window and enter "Header" name as Cookie and "Value" as JSESSIONID=D9A45BE1CCD7616F88FDF4E0FF85C213.
Refer following screen-shot,

Here I'm trying to list down all available roles for the given user, in my case it's me "udara" ;) .

You will see some similar response,