Publish data to cassandra from jaggery.js using WSO2 BAM DataPublisher

To achieve the subject, I used kpi-definition sample app[1] with WSO2 BAM 2.3.0.

First I managed to install few BAM features within WSO2 enterprise-store[2].
You can get those features with the enterprise-store social branch, or install by yourself using following.

Update <enterprise_store_home>/modules/p2-profile-gen/pom.xml with BAM-cassandra feature.

Define following <featureArtifactDef> under <featureArtifacts> ,

org.wso2.carbon:org.wso2.carbon.cassandra.explorer.feature                                    org.wso2.carbon:org.wso2.carbon.cassandra.feature
org.wso2.carbon:org.wso2.carbon.bam.cassandra.data.archive.feature
org.wso2.carbon:org.wso2.carbon.databridge.datapublisher.feature
org.wso2.carbon:org.wso2.carbon.databridge.datareceiver.feature                                    org.wso2.carbon:org.wso2.carbon.databridge.cassandra.feature
org.wso2.carbon:org.wso2.carbon.module.mgt.server.feature                                    org.wso2.carbon:org.wso2.carbon.databridge.streamdefn.cassandra.feature

Define following features under <features> ,

org.wso2.carbon.cassandra.explorer.feature.group
org.wso2.carbon.bam.cassandra.data.archive.feature.group
org.wso2.carbon.cassandra.feature.group
org.wso2.carbon.databridge.datapublisher.feature.group
org.wso2.carbon.databridge.datareceiver.feature.group
org.wso2.carbon.databridge.cassandra.feature.group

Run mvn clean install from within your <enterprise_store_home>.
If have defined above features with correct versions you will get the BUILD SUCCESS with all newly added features.

Lets start the server,
cd modules/distribution/target/
unzip wso2store-1.0.0.zip
sh wso2store-1.0.0/bin/wso2server.sh

Keep in mind if you do have your own Cassandra server up and running make sure you stop that before above step.

Next lets write the same above kpi-definition Java sample in JavaScript, and save it as Stream.js

var host = "localhost";
var url = "tcp://" + host + ":" + "7611";
var username = "admin";
var password = "admin";

var PHONE_RETAIL_STREAM = "org.wso2.bam.phone.retail.store.kpi";
var VERSION = "1.0.0";

var Agent = org.wso2.carbon.databridge.agent.thrift.Agent;
var AgentConfiguration = org.wso2.carbon.databridge.agent.thrift.conf.AgentConfiguration;
var DataPublisher = org.wso2.carbon.databridge.agent.thrift.DataPublisher;
var Event = org.wso2.carbon.databridge.commons.Event;

agentConfiguration = new AgentConfiguration();
agent = new Agent(agentConfiguration);

dataPublisher = new DataPublisher(url, username, password, agent);
var streamId = null;
  try {
     streamId = dataPublisher.findStream(PHONE_RETAIL_STREAM, VERSION);

      } catch (e) {
            streamId = dataPublisher.defineStream("{" +
                                                  "  'name':'" + PHONE_RETAIL_STREAM + "'," +
                                                  "  'version':'" + VERSION + "'," +
                                                  "  'nickName': 'Phone_Retail_Shop'," +
                                                  "  'description': 'Phone Sales'," +
                                                  "  'metaData':[" +
                                                  "          {'name':'clientType','type':'STRING'}" +
                                                  "  ]," +
                                                  "  'payloadData':[" +
                                                  "          {'name':'brand','type':'STRING'}," +
                                                  "          {'name':'quantity','type':'INT'}," +
                                                  "          {'name':'total','type':'INT'}," +
                                                  "          {'name':'user','type':'STRING'}" +
                                                  "  ]" +
                                                  "}");
        }

           
        if(streamId){
            for (var i = 0; i < 100; i++) {
                publishEvents(dataPublisher, streamId, i);
                print("Events published : " + (i + 1)+"<br/>");
            }
            dataPublisher.stop();
        }

    function  publishEvents(dataPublisher, streamId, i){
       var quantity = 1000;
       var ext_str = new java.lang.String("external");
       var nokia_str = new java.lang.String("Nokia");
       var total_int = new java.lang.Integer(123);
       var quantity_int = new java.lang.Integer(12300);
       var user_str = new java.lang.String("UdaraR");
       eventOne = new Event(streamId, Date.now(),[ext_str], null,[nokia_str,total_int, quantity_int, user_str]);
       dataPublisher.publish(eventOne);
    }

Now you can use this within the jaggery script,

var Stream = require("Stream.js");

 

[1] http://docs.wso2.org/display/BAM230/KPI+Monitoring+Sample
[2] https://github.com/wso2/enterprise-store

How to secure API invocation inside an iframe with WSO2 API Manager

First you need to get started with the WSO2 API Manager, I'm using API Manager version 1.4.0 in this post.You can find API Manager latest version from here[1] and you can find installation guide for your platform here[2].

As I'm going to use a sample which is shipped with API Manager 1.4.0 by default, follow this guide[3] in order to set up the Samples.

Now I need the API Access-Token before invoking the API within iframe.

You can login to API store (https://localhost:9443/store) with the following credentials,username: subscriber1/Password: subscriber1 ,find YoutubeFeeds API under the APIs tab and subscribe to the API.

Click on the generate button within My Subscriptions tab. Now you will see something similar to this,

 After obtaining the application key you can test sample API invocation using curl command, E9Cx0Iyu20sOAZXLE5ltvkfWgWMa is the Access Token here.


 curl -H "Authorization :Bearer E9Cx0Iyu20sOAZXLE5ltvkfWgWMa" http://localhost:8280/youtube/1.0.0/most_popular

You will see results from the YouTube.

Now lets move to the iframe part.

I'm going to use LAMPP in order to host my sample html content.

Lets create a simple html file index.html under my /var/www/oauth_iframe/ directory and and set iframe content using following ajax request,

$.ajax({
    type: "GET",
    url: "http://localhost:8280/youtube/1.0.0/most_popular",
    beforeSend: function(xhr, settings){
    xhr.setRequestHeader('Authorization', 'Bearer E9Cx0Iyu20sOAZXLE5ltvkfWgWMa');},    success: function(data){
        $("#iframe").attr('src',"data:text/html;charset=utf-8," + escape(data));
    }
});

If you need to restrict your API invocation from other domains,  
you can set your domain in allowed domains, this will ensure API content access only through the given domain.

[1] http://wso2.com/products/api-manager
[2] http://docs.wso2.org/display/AM140/Installing+the+Product
[3] http://docs.wso2.org/display/AM140/Setting+up+the+Samples

how to free up /boot partition

My /boot partition went full and I'm not able to do any system upgrades
due to this.My following upgrade commands end up with failure messages.

sudo apt-get upgrade | sudo apt-get dist-upgrade

After reading some articles on http://askubuntu.com, I found this command to remove all kernels and headers, excluding current running kernel.

sudo apt-get purge $(dpkg -l linux-{image,headers}-"[0-9]*" | awk '/ii/{print $2}' | grep -ve "$(uname -r | sed -r 's/-[a-z]+//')")

zf2 howto GridFS

GridFS is the mongoDB standard for storing and retrieving files.You can find out more information about GridFS here.

This is how I managed to store and retrieve image using zf2, doctrine-odm and mongoDB.

You can setup a Document,

<?php
namespace Course\Document;

use Doctrine\ODM\MongoDB\Mapping\Annotations as ODM;

/** @ODM\Document(collection="course") */
class Course
{
    /** @ODM\Id */
    private $id;

    /** @ODM\File */
    private $thumb;


    public function getId() {
        return $this->id;
    }

    public function getThumb()
    {
        return $this->thumb;
    }

    public function setId($id) {
        $this->id = $id;
    }

     public function setThumb($thumb)
    {
        $this->thumb = $thumb;
    }

    public function exchangeArray($data)
    {
        $this->id               = (isset($data['id']))     ? $data['id']     : null;
        $this->thumb            = (isset($data['thumb'])) ? $data['thumb'] : null;

    }

   public function getArrayCopy()
    {
        return get_object_vars($this);
    }

    public function getObjectAsArray()
    {
        $array = array(
         'thumb' =>$this->thumb,
                      );
        return $array;
    }

}

Then zf2 form with file upload input field, within your function __construct you can add,

$this->setAttribute('method', 'post');
$this->setAttribute('enctype','multipart/form-data');

$this->add(array(
            'name' => 'id',
            'attributes' => array(
                'type'  => 'hidden',
            ),
        ));
$this->add(array(
            'name' => 'thumb',
            'attributes' => array(
                'type'  => 'file',
            ),
            'options' => array(
                'label' => 'Thumbnail',
            ),
        ));

Then create the form,

<?php
$title = 'upload Image';
$this->headTitle($title);
?>
<h1><?php echo $this->escapeHtml($title); ?></h1>
<?php
$form = $this->form;
$form->setAttribute('action', $this->url('image', array('action' => 'add')));
$form->prepare();

echo $this->form()->openTag($form);
echo $this->formRow($form->get('thumb'));
echo $this->formSubmit($form->get('submit'));
echo $this->form()->closeTag();
?>

and finally update your controller with,

$request = $this->getRequest();
        if ($request->isPost()) {

            $data = array_merge_recursive(
            $this->getRequest()->getPost()->toArray(),
            $this->getRequest()->getFiles()->toArray()
             );


     $file_src   = '/tmp/'.$data['thumb']['name'];
         move_uploaded_file($data['thumb']['tmp_name'], $file_src);


    $dm = $this->getServiceLocator()->get('doctrine.documentmanager.odm_default');
    $course = new Course();
    $course->setThumb($file_src);
    $dm->persist($course);
        $dm->flush();

    }

To retrieve image, update controller with

$dm = $this->getServiceLocator()->get('doctrine.documentmanager.odm_default');
        $courses = $dm->createQueryBuilder('Course\Document\Course')
                ->getQuery()
                ->execute();

        return new ViewModel(array(
           'courses' => $courses,
        ));

And your view,

<table class="table">
<tr>
    <th>thumb</th>
    <th>&nbsp;</th>
</tr>
<?php foreach ($courses as $course) : ?>
<tr>
    <td><?php $i= base64_encode($course->getThumb()->getBytes());echo "<img src=\"data:image/jpg;base64,{$i}\">";?></td>
</tr>
<?php endforeach; ?>
</table>

 Make sure you set the correct header image type in order to render the image.

further info :
Storing Files with MongoGridFS
ZF2 form collections

ZF2 skeleton with mongoDB | using doctrine odm

1. git clone the ZF2 Skeleton Application

    git clone git://github.com/zendframework/ZendSkeletonApplication.git

2. cd into the ZendSkeletonApplication directory and open the composer.json in your favourite text editor.

    {
        "name": "zendframework/skeleton-application",
        "description": "Skeleton Application for ZF2",
        "license": "BSD-3-Clause",
        "keywords": [
            "framework",
            "zf2"
        ],
        "homepage": "http://framework.zend.com/",
        "minimum-stability": "dev",
        "require": {
            "php": ">=5.3.3",
            "zendframework/zendframework": "2.*",
            "doctrine/doctrine-mongo-odm-module": "dev-master"
        }
    }

3. Update your composer.json and run

php composer.phar self-update
php composer.phar install

4. If you are success with installing all dependencies using composer follow the official guide here to set up virtual-host and update hosts file.

5. If you are unable to update those dependencies and found something similar to this,

Check your composer.json for any mistakes, or follow my previous blog post.

6. Add the doctrine modules to your config file, config/application.config.php. Update the modules array similar to this, here im adding zend module named course as well.

        'modules' => array(
            'Application',
            'Course',
            'DoctrineModule',
            'DoctrineMongoODMModule',
        ),

7.  Copy the doctrine-odm config file to your config directory, and update according to your environment.This is where you set your server hosts, ports, username, and passwords etc.

    cp vendor/doctrine/doctrine-mongo-odm-module/config/module.doctrine-mongo-odm.local.php.dist config/module.doctrine-mongo-odm.local.php

 and update,

8. You can try the course module I have created here with course add and retrieve course data.

Installing doctrine ODM through composer

I tried to install doctrine/doctrine-mongo-odm-module through composer with ZF2, but I keep getting an error stating that PHP mango extension is missing from my computer.This is the require section of my composer.json file,

"require": {
        "php": ">=5.3.3",
        "zendframework/zendframework": "2.*",
        "doctrine/doctrine-mongo-odm-module": "dev-master"
    }

 Im getting this after I run  "php composer.phar install"

Correct version of MongoDB driver shows up in phpinfo() as well.

And THE PROBLEM is, Composer runs on PHP-CLI I do have two php.ini files.

/etc/php5/apache2/php.ini
/etc/php5/cli/php.ini

But I only updated MongoDB extension "extension=mongo.so" in /etc/php5/apache2/php.ini.Composer in the other hand is looking for mongo extension in the php.ini located in the /etc/php5/cli/php.ini directory. So by adding mongo extension in /etc/php5/cli/php.ini  file and apache restart solved my problem.

Event driven, asynchronous callbacks and Node.js

Node's approach of code execution isn't unique, but back end execution model is different from environments like PHP or Java.

Node.js core provides methods to reading content of a file in the file-system. fs module provides variety of file system methods.The easiest way to read the entire contents of a file is as follows.

fs = require('fs');
fs.readFile('/home/udara/docs/users.txt');
console.log("Hello Users !!");

Let's assume that the time taken to read users.txt is really long, JavaScript interpreter of Node.js first has to complete reading the users.txt file, and then only it execute the console.log() function. Same as in PHP if this code would be part of a webpage, the user have to wait until it read whole file at once and load the page.

Unlike PHP, Node.js there is only one single process. If there is something happen like above example this will effect the whole process and each and every user would be affected. But in PHP only user who request that web page feel slow page load, but other users requesting other pages would not be affected. 

So where is asynchronous of Node.js ????

fs = require('fs');
fs.readFile('/home/udara/docs/users.txt', function (error,data) {
console.log(data);
});
console.log("Hello Users !!");

Here, without expecting fs.readFile() to directly return a file content to us we pass second parameter an anonymous function.Now Node.js can handle fileread request asynchronously.Only when fileread is done then it will execute anonymous function that was passed to fs.readFile().

Without waiting for the fileread output system will out put Hello Users !! to the console.