Kappa Architecture implementation using Apache Flink, Kafka and Cassandra

Refer to the code here - https://github.com/tuhingupta/kappa-streaming

What is Kappa Architecture?

Kappa Architecture is a software architecture pattern. Rather than using a relational DB like SQL or a key-value store like Cassandra, the canonical data store in a Kappa Architecture system is an append-only immutable log. From the log, data is streamed through a computational system and fed into auxiliary stores for serving.

Source: Oreilly site

It follows a Command Query Responsibility Segregation pattern.

Client Writes

Client sends a stream of data, which could be 1 record or hundred thousands of records as a byte stream of json, csv or any other data format.

Server

Is a non-blocking reactive server written to accept bytes of data and parse/process bytes into records and put them into an immutable append log system like Kafka. 

Kafka

Kafka is an immutable append-only log based messaging system, where the server will write the records received as byte stream. This acts as the log from where different processes can read data and generate client specific views.

Processors

Various in-memory, stream processing frameworks like Flink, Storm, Samza can be used to process these logs to generate client specific views. These processors would be consistently churning data available on topics in Kafka and generating updated data.

Client Reads

Client now reads from the many materialized views that were created by processing the immutable append log.

Sample Use cases:

Log processing

Docker Image for JBoss EAP

Docker Image for JBoss EAP 

Following post explains how to create Docker Image for JBoss EAP and expose its management and web 8080 port for access from host machine or other applications.

Before proceeding with this article you should have Docker and Docker Machine installed on your machine.

You can also refer to my previous post on how to create simple java post.

Dockerfile

Create a similar Docker file. This file is available at following Github repo - tuhingupta/docker-jboss-eap.

If you look at the dockerfile, you will notice:

at line 1, we start with FROM key word. This tells Docker what image to use as base image.

at line 2, we specify MAINTAINER that tells the name, email of the person maintaining the image

at line 4, WORKDIR specifies the working directory in the image.

at line 5, RUN keyword runs whatever command proceeds it.

at line 6, copy jboss-eap zip file to /usr/software folder in the image

at line 9, unzip jbos-eap zip

at line 15, setup user for jboss admin

at line 18, set JAVA_OPT . We need to bind address and management console address so that EAP will bind on all IP addresses for our container.

at line 21, EXPOSE command is used to expose ports from the container.

at line 26, ENTRYPOINT command tells what command will run when the container starts.

Save this file as Dockerfile in a folder that also contains jboss-eap-6.2.4.zip file.

Building the container 

Now you build the image using the following command:

$ docker build --rm -f Dockerfile -t jboss-eap .

Once the image is built, it is ready to be run.

Run docker image

Run the docker image

$ docker run -it -p 0.0.0.0:9990:9990 -p 0.0.0.0:8080:8080 --name jboss-serv -d jboss-eap

you will notice that we used -p option. 

-p option publishes a container's port(s) to the host.

This option publishes the ip that we bind in line 18 of Dockerfile so that it can be accessed from host by using localhost.

Accessing JBoss URLs

Now you can access JBoss management console using 

http://localhost:9990/management

Multi-platform Hybrid Mobile App Development

Looking for a hybrid mobile app using ionic, cordova, angular.js.
Refer to a sample restaurant app developed using Ionic, Cordova and Angular JS
The backend data is provided using json-server.

Refer to the project on my Github site - https://github.com/tuhingupta/ionic-cordova

Post you comments/questions on github or on this blog for more information about the application or how to develop a similar mobile app.


Reference Docs:
http://ionicframework.com/docs/components/

Amazon EC2 - Install node application on EC2 instance.

Before working on this blog you must have the EC2 environment setup. Refer to my older blogs for the same.

• Setup AWS EC2 instance - http://javaredhot.blogspot.com/2016/02/amazon-ec2-creating-and-setting-up.html
• Setup Git, Node.js and NPM on EC2 - http://javaredhot.blogspot.com/2016/02/amazon-aws-installing-git-nodejs-and.html

Get your project

Use git to download your node.js application. 

I have my application on Github - https://github.com/tuhingupta/gitwebhooks

$ git clone https://github.com/tuhingupta/gitwebhooks.git

$ cd gitwebhooks

Configure EC2 to listen inbound traffic at 8080

Somethings need to be configured in your application so that it can be accessed over internet.

Have your node.js application run server over port 8080

In my application (gitwebhooks) above I have set it in app.js

app.set('port', process.env.PORT || 8080);

Now configure your EC2 instance to listen inbound traffic at port 8080

Setup your security groups in EC2 like image below:

Now route HTTP traffic on port 80 to redirect to port 8080

$ sudo iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to 8080

Access your application from internet

Now you can access your application over internet using your public DNS.

I can access my application (a Node.js Express REST API) using:

http://ec2-{my_ec2_id}.us-west-2.compute.amazonaws.com/api/name

Amazon AWS - Installing Git, Node.js and npm on EC2 instance

Before we setup Node.js on AWS, you should have AWS EC2 instance setup and you should be able to login using Putty or Terminal.

Refer to my previous blog on how to setup EC2 instance -
 http://javaredhot.blogspot.com/2016/02/amazon-ec2-creating-and-setting-up.html

Now login to EC2 instance

Install Git

$ sudo yum install git

This will ask for your confirmation and then install git.

Install Node.js

To install node and npm following steps need to be performed.

To compile node we need gcc, make and git to import node source code:

sudo yum install gcc-c++ make
sudo yum install openssl-devel
sudo yum install git

Cloning node.js source code:

git clone https://github.com/nodejs/node.git

Compile and install node.js

cd node
./configure 
make
sudo make install
Add user´s directory to BIN Paths (node binaries location)
sudo su
nano /etc/ec2-user


Inside the editor scroll to where you see the line: 
Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin

Append the value :/usr/local/bin

In some cases node does not install at /usr/local/bin and there is no soft link to the actual location. So in that case you can create a soft link from actual node location to /usr/local/bin

$ cd /usr/local/bin

$ sudo ln -s  /home/ec2-user/node/out/Release/node node
$ sudo ln -s /home/ec2-user/node/out/lib/node_modules/npm/bin/npm npm
$ sudo ln -s /home/ec2-user/node/out/lib/node_modules node_modules

Install npm

git clone https://github.com/npm/npm.git
cd npm
sudo make install

Test if node is working:

node

References:

https://gist.github.com/isaacs/579814
http://iconof.com/blog/how-to-install-setup-node-js-on-amazon-aws-ec2-complete-guide/

Amazon EC2 - Creating and setting up an instance on AWS

Setup AWS account

Set up AWS account at - https://aws.amazon.com/

Launch EC2 instance

On AWS console, 

• Go to AWS services dropdown and select EC2 instance
• Create and launch EC2 instance

Select Container and configure

• I selected Amazon Linux
• Using the configuration available as Free Tier.

Once setup & review is complete you are now provision your instance. And it will be ready to launch

Login to the instance

You can use Mac terminal or Putty on Windows to connect AWS instance.

Store pem file downloaded when creating instance. Make sure that the permissions are restrictive. Hence use 

$ chmod 400 aws.pem

Now use ssh command to connect to instance

ssh -i /path/my-key-pair.pem ec2-user@ec2-198-51-100-1.compute-1.amazonaws.com

For more details on how to ssh, read here - http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AccessingInstancesLinux.html

Docker - Creating simple java image

This is my "Hello World" of creating my own docker image for running a Java program in Java 7 environment.

You can find information about Docker here. Developers need to understand concepts like Docker, docker-machine and how to write a Dockerfile to be able to create own docker images.

Create Dockerfile 

This is how a Dockerfile for our project would look like:

FROM java:7

MAINTAINER tuhin.gupta@gmail.com

COPY ./src/main/java /usr/src/myjavaapp

WORKDIR /usr/src/myjavaapp

RUN javac com/tuhin/example/Main.java 

CMD ["java", "com.tuhin.example.Main"]

see details of Dockerfile syntax here

In our project we are using following instructions:

FROM - tell docker to pull java 7 image

MAINTAINER - instruction allows to set the author field

COPY - instruction is to copy the java source code to docker image file system

WORKDIR - sets the work directory in docker image

RUN - runs javac at the specified location

CMD - this instruction can run any command that can be run from command prompt.

Java code in Main.java is:

package com.tuhin.example;

/**

 * @author Tuhin Gupta

 *

 */

public class Main {

public static void main(String[] args) {

System.out.println("This is the main class.");

}

}

Build Docker Image

Once the Dockerfile is created, now we need to build the docker image for java application

$ docker build -f <path-to-docker-context>/docker/Dockerfile -t myjavaapp <path-to-docker-context>

Above command will build the docker image.

-f : is used to specify the path to docker file within docker context. Dockerfile must be in docker context.

-t : Repository name (and optionally a tag) for the image

This command will build the docker image and you can view your recently created image using following command:

$ docker images

REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE

myjavaapp                 latest              45a3aacd5bd0          1 hours ago                590.1 MB

Run your Docker Image

To run the newly created image and the java program inside the image, use following command:

$ docker run -t --rm --name test-myjavaapp myjavaapp 

This will run myjavaapp image and Main.java class within it.

Save your Docker Image as tar file for sharing

Using following command, you can save a docker image as tar file and share between developers.

$ docker save -o <path-to-save>/myjavaapp.tar <image id>

<image id>: image id from docker images command

Refer to my Github site - https://github.com/tuhingupta/docker-simple-java and download the Dockerfile.

Docker inside corporate VPN Network

If you are trying to create a docker machine and use docker within a corporate VPN network it might not work straight out of an example. 

Following steps would help. When we are creating docker machine, we would need to specify proxy information so that docker can then connect to Docker Hub.

Create docker machine as follows:

$docker-machine create --driver=virtualbox --virtualbox-memory=4096 --virtualbox-cpu-count=2 \

--engine-env HTTP_PROXY=http://proxy.yourorg.com:8989 \

--engine-env HTTPS_PROXY=http://proxy.yourorg.com:8989 \

--engine-env NO_PROXY=localhost,127.0.0.1,localaddress,.localdomain.com,.local,127.0.0.0/8 \

office




Then go to the docker machine VM on VirtualBox and you would need to do port forwarding.

1. Go to the VM on virtual box machine and go to Settings>Network
2. For Adapter1, open Port Forwarding and add port forwarding information. (You can get your machine IP and port information using ` $docker-machine env office` command )








































Once port forwarding information is saved, on terminal run following commands




$ eval "$(docker-machine env office)” $ export DOCKER_HOST=tcp://127.0.0.1:2376 --disable TLS security if you get certificate error when you do $ docker ps $ alias docker="docker --tlsverify=false" Now commands like $ docker ps and $ docker search <image> would work. If you are working outside of office network, offline at home, then you can create a docker machine without proxy information. Use this machine when offline. $docker-machine create --driver=virtualbox --virtualbox-memory=4096 --virtualbox-cpu-count=2 \ default