Play around with Microservices

Knoldus Blogs

What is microservice architecture

Microservice architecture is a method of developing software applications as a suite of independently deployable, modular services in which each service runs a unique process and communicates through a well-defined, lightweight mechanism to serve a business goal.

In other words, it is an architectural style that structures an application as a collection of loosely coupled services, which implement business capabilities.

Why to choose Microservice ?

  • Independent Deployment : Simple and modular services are easy to deploy, and since they are autonomous, are less likely to cause system failures when they go wrong.
  • Strong Module Boundaries: Micro-services forces modular structure, which is particularly important for larger projects.
  • Technology Diversity: With microservices you can mix multiple languages (like java, scala etc), development frameworks and data-storage technologies(postgreSQL , elasticsearch, mongoDb etc). You are not restricted to use only single technology.
  • Easily Replaceable: Nowadays many diverse services communicate with each…

View original post 603 more words

Advertisements

Deep Dive into Spark Cluster Managers

Knoldus Blogs

This blog aims to dig into the different Cluster Management modes in which you can run your spark application.

Spark applications run as independent sets of processes on a cluster, coordinated by the SparkContextobject in your main program which is called the Driver Program. Specifically, to run on a cluster, the SparkContext can connect to several types of Cluster Managers, which allocate resources across applications. Once the connection is established, Spark acquires executors on the nodes in the cluster to run its processes, do some computation and store data for your application. Next, it sends your application code (defined by JAR or Python files passed to SparkContext) to the executors. Finally, SparkContext sends tasks to the executors to run.

                    spark1.jpeg

View original post 863 more words

Usage of “?.” in Angular

Knoldus Blogs

In some code snippets, you may have seen that Angular expressions have a question mark after them. Angular refers to it as ‘Safe Navigation Operator’.It makes sure that we are not seeing null and undefined values in our application when we want to access properties of an object.

“Safe Navigation Operator” can be used to prevent Angular from throwing errors, when trying to access object properties of an object that don’t exist.

E.g.       {{ employee?.name }}

Here it is, protecting against a view render failure if the employee is null. This will evaluate name only if an employee is not null or undefined. This comes in handy when an employee is something that is loaded asynchronously.

Consider a scenario where we have an object created in our component which is populated by data coming from HTTP Service.


Sample Code :

EmployeeRatingComponent.ts file

exportclass EmployeeRating

View original post 262 more words

BASIC UNDERSTANDING OF WEBSOCKETS IN ANGULAR

Knoldus Blogs

angular-logoHere we are going to get a basic understanding of how to establish a connection between client browsers and the application server over a TCP connection using WebSockets in angular.

Websockets

WebSocket is a technology that allows two-way communication over single TCP socket. It is designed to be used between client’s browser and application server.

Role of RxJS in Websockets

  • RxJS(Reactive extension for Javascript) plays a vital role in angular, RxJS is a library which is mainly used for asynchronous code. Also, it provides an environment for implementing observables, It also helps in adapting language and browser specific environment.

  • RxJS allow us to listen to the new notifications using the WebSocket connection and perform an action if an event occurs.

Now lets see how we can implement Websockets in angular:

1: First and foremost thing we do here is import * from RxJs library which allows us to create a 

View original post 438 more words

Manipulating Server returned dates at client Side in Angular

Knoldus Blogs

Hey Visitor,

Have you ever faced an issue with the date values on your website, when the date and time returned from server doesn’t change according to the client timezone , for eg. values returned from server at US will be different for a client at India so we need to manipulate the received time according to Indian Time Zone.

So recently, I encountered this issue in my project in displaying time in the correct format on the required time zone.

Handling date/time when dealing with time-zones is a tedious task.

As a developer, getting date from server then converting, comparing and displaying date and time might get tricky on front-end side.

So the best solution for this is use of a library called –

Moment.js

Moment.js is a Javascript library for parsing, validating, manipulating, formatting and displaying dates and times.

It’s easy documentation also makes it the first choice for…

View original post 259 more words

Jenkins – Integrating Email Service

Jenkins is one open source tool to perform continuous integration and build automation. Using it, all development work can be integrated as early as possible. The resulting artifacts are automatically created and tested and as a result the process of identification of errors becomes faster. But there must be a way so that build status and testing results can be reported to the team. And then email service come into picture.

To dig further into the details of jenkins and building jobs in it, you can refer to Introduction to Jenkins and Building Jobs with Jenkins.

Now the next question comes i.e “ How email service can be integrated with Jenkins? ”

So I will be discussing two approaches to configure email service:

1) Default Email Notifier: This is what comes default with Jenkins. It contains a default message containing the Build number and its status.

2) Email Extension PluginThis plugin allows you to configure every aspect of email notifications. You can customize when an email is sent, who should receive it, and what the email says.

Next step will be, what all configurations we require to use the above two.

Configuring Default Email Notifier

Step 1: Configure Default Email Notifier

To provide the configuration, Go to Jenkins System configuration page:

Jenkins -> Manage Jenkins -> Configure System

You will be able to see the below screen to add email configurations.

1

Below is the detail for some of the fields :

  • SMTP Server: It denotes name of the mail server. Leave it empty to use the default server (which is normally the one running on localhost). Jenkins uses JavaMail for sending out e-mails, and JavaMail allows additional settings to be given as system properties to the container.

  • SMTP Port: It define the port number for the mail server. Leave it empty to use the default port for the protocol.

Once the mail configurations are set, you can test whether it is working fine or not by checking “Test configuration by sending test email”.

Step 2: To add email notification to projects

To allow your projects to send email you need to add “Post Build Action” and select “Email Notification” from the drop down list.

This will provide you the below interface where you can add list of email addresses to whom email is required to be sent.

2

Configuring Email Extension Plugin

This plugin extends Jenkins built in email notification functionality by giving you more control. It provides customization of 3 areas.

  • Triggers – Select the conditions that should cause an email notification to be sent.

  • Content – Specify the content of each triggered email’s subject and body.

  • Recipients- Specify who should receive an email when it is triggered.

Step 1: Set Global Configuration

Before using email-ext on a project, you must configure some global settings. Go to Jenkins System configuration page i.e, Manage Jenkins -> Configure System

The section titled Extended E-mail Notification is where you can configure global email-ext properties. The properties here should match the settings for your smtp mail server. This section is set up to mirror Jenkins own email Publisher Reporter (It’s different extension point), however there are a few additions. The inputs labeled Default Subject and Default Content, allow you to configure the email content on a global level. The input labeled Default Recipients can be used to set a default list of email addresses for all projects using plugin (it can be overridden at the project level). This can be used to greatly simplify the configuration you need to do for all projects.

3

Step 2: Project configuration

For a project to use the email-ext plugin, you need to enable it in the project configuration page. Select the checkbox labelled “Editable Email Notification” in the “Post-build Actions” section and you will be able to see something like below screenshot.

4

Basic Configuration

There are three fields that you can edit when the plugin is enabled.

  • Global Recipient List – This is a comma (or whitespace) separated list of email recipients. Allows you to specify a single recipient list for each email that is sent.

  • Default Subject – This allows you to configure a token (more about tokens later), that can be used to easily configure all email subjects for the project.

  • Default Content – Same as Default Subject, but for the email body instead of the subject.

Advanced Configuration

To see the advanced configuration for the plugin, first click on Override Global setting checkbox, then click the “Advanced” button. This section allows you to specify recipients for each type of email trigger as well as a pre-send script that can be used to modify the email just prior to sending.

Triggers

By default, the only trigger configured is the “Failure” trigger. To add more triggers, select one from the dropdown, and it will be added to the list. Once you have added a trigger, you have several options.

References:

Jenkins Build Jobs

In continuation to my previous blogs Introduction to Jenkins and Jenkins – Manage Security , I will now be talking about creating build jobs with Jenkins.

It is easy and simple to create a new build job in Jenkins. Follow the given steps to get started:

  • From the Jenkins Dashboard, Click on “New Item”
  • Name your project and select project type.

1

  • Click on “Ok” to continue.

Understanding Build Job Types

Jenkins supports different types of build jobs. Some of them are :

  • Freestyle software project

Freestyle build jobs are general-purpose build jobs, which provides a maximum of flexibility.This is the central feature of Jenkins. Jenkins will build your project, combining any SCM with any build system, and this can be even used for something other than software build.

  • Maven project

Build a maven project. Jenkins takes advantage of your POM files and drastically reduces the configuration.

  • External Job

This type of job allows you to record the execution of a process run outside Jenkins, even on a remote machine. This is designed so that you can use Jenkins as a dashboard of your existing automation system.

  • Multiconfiguration job

The “multiconfiguration project” (also referred to as a “matrix project”) lets you run the same build job in many different configurations. This powerful feature can be useful for testing an application in many different environments, with different databases, or even on different build machines.

Configure project

Once you are done creating the project, your next step is to setup configurations which include :

1) Configuring Global Tool Configuration

Here you can set JDK path, git configurations and Gradle/ Ant or Maven installations as per your project requirement.

2) Creating or updating(if already created during project creation) project configuration settings which may include General settings, Source code management, build triggers, Pre-steps, Build, Post Steps, Emailing notification regarding build and post build actions as per your project need.

Reporting on Test results

Once all the test cases are executed during build, one of the essential part is to generate the test results. This can be specified in the “post build action” part.

For maven jobs, you need not to provide any special configuration. All you need to do is to make sure you invoke a goal that will run your test cases. Ex: mvn test(for unit test) or mvn verify(for unit and integration test).

2

For “freestyle build jobs”, you need to do a bit of more configuration work. In addition to ensuring that your build actually runs the tests, you need to tell jenkins additionally to publish the Junit test report. It can be configured in the “Post build actions” section.

  • From Post Build Actions –> Select “Publish Junit test result report”.
  • Apply and save the configurations.

3

NOTE: Jenkins looks for the “Test report XMLs” path from the workspace root for generating the report. Ensure that the given path is correct or use wildcards to look in multiple locations. To know workspace you can run “echo ${WORKSPACE}”.

Manage Nodes

Jenkins handles parallel and distributed builds well. Jenkins runs simultaneously, and, if you are using distributed builds, set up build nodes. A build node is another machine that Jenkins can use to execute its builds.

1. To execute builds on another machine, you need to create a node in jenkins and then you need to tell your project about the node where it has to run.

Follow the given steps to create a node:

Jenkins Dashboard –> Manage Jenkins –> Manage Nodes –> New Node

2. Name your node and Select the appropriate option and Click “OK”.

3. After that you will be able to see your node under the nodes

Provide “Remote root directory” where you want your jenkins projects to be stored on your machine along with the credentials (if required) to connect to that machine.

Once a node is created, it can be used to run the projects which are setup on jenkins. This can be done by selecting “Restrict where the project can run field.” which can be found in the respective project configuration in “General Tab”.

Suppose if the created node has name “194.178.2.59_Node”. Then it could be set as per the image:

4

In my next blog, I shall be elaborating on “Setting Email Configurations” with Jenkins. Stay tuned !

References:

  1. Jenkins-The Definitive Guide by John Ferguson Smart

Jenkins – Manage Security

Jenkins is one of a powerful continuous integration tool with a great community. It is an opensource tool and hence can be easily used by anyone. So why not to start knowing a tool like this. To read about the basics and installation steps, you can refer to my previous blog Introduction to jenkins

Creating Users

There can be multiple users that can operate jenkins for the same set of project holding different responsibility. So to create a user we can follow the below steps:

  • Click on Manage Jenkins on the home page.
  • Search for Manage Users in Manage Jenkins Page and click on it.
  • You will see a users page.
  • Search for “Create User” in the left side of the page and click to create user.

Here users can be created and its username can further be used to grant access.

Managing Access

After users are created it is very important to restrict access of users so they can operate on their responsibility in a controlled manner.

To manage access of users :

  • Click on Manage Jenkins from the home page.
  • Click on the link Configure global security.
  • Under “Access Control” you will see an “Authorisation” label.

This is the place where you can decide between different options to control access. Following are the available options:

  • Anyone can do anything
  • Legacy Mode: If you have the “admin” role, you’ll be granted full control over the system, and otherwise (including anonymous users) you’ll only have the read access.
  • Logged in users can do anything: It also provides a sub-option “Allow anonymous read access” . If checked, this will allow users who are not authenticated to access Jenkins in a read-only mode.
  • Matrix based security: In this scheme, you can configure who can do what by using a big table.Each column represents a permission. Each row represents a user or a group (often called ‘role’, depending on the security realm).Permissions are additive. That is, if an user X is in group A, B, and C, then the permissions that this user actually has are the union of all permissions given to X, A, B, C, and anonymous.
  • Project based Matrix Authorisation Strategy: This mode is an extension to “Matrix-based security” that allows additional ACL matrix to be defined for each project separately (which is done on the job configuration screen). This allows you to say things like “John Doe can access project A, B, and C but he can’t see D.

Configuring a Proxy

In most enterprise environments, Jenkins server will be situated behind a firewall, and doesn’t have direct access to the Internet and if the server JVM is not configured appropriately to enable internet connection (Jenkins needs Internet access to download plugins and updates, and also to install tools such as the JDK, Ant and Maven from remote sites). If you need to go through an HTTP proxy server to get to the Internet, you can configure the connection details (the server and port, and if required the username and password).

To configure connection details, follow the given steps:

1) Click on “Manage Jenkins” from Jenkins Dashboard.

2) Click on “Manage Plugins” from Manage Jenkins Page.

3) Move to the “Advanced” tab to set “HTTP Proxy Configuration”.

Please note that Jenkins uses HTTPS to communicate with the update center to download plugins.

If Microsoft’s NTLM authentication scheme is used by your proxy, then you will have to provide a domain name as well as a username in the “User name” field. For example: “DomainName\Sangeeta”.

Finally, if you are setting up Proxy access on your Jenkins build server, remember that all of the other tools running on this server will need to know about the proxy as well. In particular, this may include tools such as Subversion (if you are accessing an external repository) and Maven.

Introduction to Jenkins

What is Jenkins

Jenkins is a powerful application that allows continuous integration and continuous delivery of projects, regardless of the platform you are working on. It is a free source that can handle any kind of build or continuous integration. You can integrate Jenkins with a number of testing and deployment technologies.

How Jenkins work?

Below point explains the work flow of jenkins:

1) Developers commit and push their code

2) Jenkins will pick up the changed source code and trigger a build and run the test cases.

3) The build output will be available in the jenkins dashboard. Automatic notification can be sent back to the developer.

Getting Started with Jenkins:

1) System Requirements

  • Jenkins requires Java7 or above to function. Java8 is recommended.
  • Jenkins requires a fair amount of memory to operate well.

Setup JAVA_HOME in .bashrc :

export JAVA_HOME=path_to_java

2) Download Jenkins

Starting Jenkins:

Open the command prompt. From the command prompt, browse to the directory where the jenkins.war file is present. Run the following command:

java -jar jenkins.war

After the command is run, various tasks will run, one of which is the extraction of the war file which is done by an embedded webserver called winstone. Once the processing is complete without major errors, the following line will come in the output of the command prompt.

INFO: Jenkins is fully up and running

Note: Jenkins by default use port 8080. To run it on different port :

java -jar jenkins.war –httpPort=9090

It will run jenkins. at port 9090
Accessing Jenkins

Once Jenkins is up and running, one can access Jenkins from the link −http://localhost:8080 (default)

This link will bring up the Jenkins dashboard.

Setting up Jenkins with Tomcat

The following prerequisites must be met for Jenkins Tomcat setup.

Step 1: Verify Java Installation

To verify Java installation, open the console and execute the following java command.

$java –version

It should display output as per the version like :

java version “1.8.0_92”

Java(TM) SE Runtime Environment (build 1.8.0_92-b14)

Java HotSpot(TM) 64-Bit Server VM (build 25.92-b14, mixed mode)

Step2: Verify whether JAVA_HOME variable is set

It can be done using command, where output must specify the path of Java Home:

echo $JAVA_HOME

If the path is not set, then set JAVA_HOME environment variable to point to the base directory location where Java is installed on your machine. Eg.

export JAVA_HOME=path_to_java

Step 3: Download Tomcat

1) From official website of tomcat: https://tomcat.apache.org/download-70.cgi, download tar.gz file From “Binary Distributions” under Core bullet.

2) Then extract the contents of the downloaded tar.gz file.

Step 4: Jenkins and Tomcat Setup

Copy the Jenkis.war file which was downloaded from the previous section and copy it to the webapps folder in the tomcat folder.

Now open the command prompt. From the command prompt, browse to the directory where the tomcat folder is located. Browse to the bin directory in this folder and run the below command

./startup.sh

Once the processing is complete without major errors, the following line will come in the output of the command prompt.

INFO: Server startup in 1310 ms

Open the browser and go to the link −http://localhost:8080/jenkins. Jenkins will be up and running on tomcat.

NOTE : If the port is already in use, it can be changed in file server.xml inside conf folder of apache-tomcat.

Following lines define the port number in server.xml:

<Connector port=”8080″ protocol=”HTTP/1.1″

connectionTimeout=”20000″

redirectPort=”8443″ />

Here 8080 can be replaced with any other unused port (say 9090)

Now to open the browser and go to the link −http://localhost:9090/jenkins. Jenkins will be up and running on tomcat.

Jenkins-Git Setup

Following are the steps to be followed:

1) In your Jenkins Dashboard (Home screen), click the Manage Jenkins option on the left hand side.

2) In the next screen, click the ‘Manage Plugins’ option.

3) In the next screen, click the Available tab. This tab will give a list of plugins which are available for downloading. In the ‘Filter’ tab type ‘Git plugin’.

4) The list will then be filtered. Check the Git Plugin option and click on the button ‘Install without restart’.

5) The installation will then begin and the screen will be refreshed to show the status of the download.

6) Once all installations are complete, restart Jenkins by issuing the following command in the browser.

http://localhost:9090/jenkins/restart

7) After Jenkins is restarted, Git will be available as an option whilst configuring jobs. To verify, click on New Item in the menu options for Jenkins. Then enter a name for a job, in the following case, the name entered is ‘Demo’. Select ‘Freestyle project’ as the item type. Click the Ok button.

8) In the next screen, if you browse to the Source code Management section, you will now see ‘Git’ as an option.

NOTE: In case Git is already setup then it will be available in Installed tabs in “Manage Plugins”

Jenkins-Maven Setup

Maven Download

1) From official homepage of maven ://maven.apache.org/download.cgi download the link to the Binary.zip file under “Files Section”

2) Once the file is downloaded, extract the files at your local directory say /opt/local-maven/apache-maven-3.3.9

Jenkins-Maven setup

1) In the Jenkins dashboard (Home screen), click Manage Jenkins from the left-hand side menu.

2) Click on ‘Configure System’ from the right hand side.

3) In the Configure system screen, scroll down till you see the Maven section and then click on the ‘Add Maven’ button.

4) Uncheck the ‘Install automatically’ option.

5) Add any name for the setting and the location of the MAVEN_HOME. For example:

Name: localmaven

MAVEN_HOME: /opt/local-maven/apache-maven-3.3.9

6) Then, click on the ‘Save’ button at the end of the screen.

Note: If you are using 2.x version of jenkins, you can find maven at

Jenkins(Home) -> Manage Jenkins -> Global Tool Configuration

You can now create a job with the ‘Maven project’ option. In the Jenkins dashboard, click the New Item option.

If you still cannot see “Maven Project” in “New” window then perform the given steps:

1) Click Manage Jenkins

2) Click Manage Plugins

3) Search for “Maven Integration plugin” using filter and click on “Install without Restart”

4) Once the download is complete, restart jenkins by hitting URL :

http://IP_ADDRESS: TOMCAT_PORT/jenkins/restart

Example:  http://192.168.2.145:9090/jenkins/restart

Now you are ready to build and test your maven projects.

References:

Hive Database : A Basic Introduction

What is Hive?

Hive is a data warehouse infrastructure tool which process structured data in Hadoop. It resides on top of Hadoop to summarize Big Data, and makes querying and analyzing easy.

Why to use Hive?

1) Most of the data warehousing application work with SQL based quering language, Hive supports easy portability of SQL-based application to Hadoop

2) Faster results even for tremendous datasets.

3) As data volume and variety increases, more machines can be added without any corresponding reduction in the performance

Features of Hive

1) It accelerate queries as it provide indexes, including bitmap indexes.

2) It stores metadata which reduce the time to perform semantic checks during query execution.

3) It provide built-in functions to manipulate dates, strings and other data-mining tools.

4) It supports different file formats like Avro Files, ORC Files, Parquet etc

Architecture of Hive

Major components of hive are as follows :

1) Metastore : This component is responsible for storing all the structure information of the various tables and partitions in the warehouse including column and column type information, the serializers and deserializers necessary to read and write data and the corresponding HDFS files where the data is to be stored.

2) Driver : It acts like a controller which receives the HiveQL statements and starts the execution of statement by creating sessions and monitors the life cycle and progress of the execution. It stores the necessary metadata generated during the execution of an HiveQL statement. The driver also acts as a collection point of data or query result obtained after the Reduce operation.

3) Compiler : The component that parses the query, does semantic analysis on the different query blocks and query expressions and eventually generates an execution plan with the help of the table and partition metadata looked up from the metastore.

In other words, the process can be described by the following flow :

Parser —> Semantic Analyser —> Logical Plan Generator —> Query Plan Generator.

4) Optimizer : Performs various transformations on the execution plan to get an optimized DAG(Directed Acyclic Graph).

5) Executor : After compilation and Optimization, the Executor executes the tasks according to the DAG. It interacts with the job tracker of Hadoop to schedule tasks to be run. It takes care of pipelining the tasks by making sure that a task with dependency gets executed only if all other prerequisites are run

6) CLI, UI and Thrift Server : Interface that can be used by the users to submit queries and get the result.

How hive interact with hadoop ?

1) Execute Query : From hive interface(UI or Command Line) query is sent to driver

for execution.

2) Check Syntax and Get Plan : The driver takes the help of query compiler which parses query to check the syntax and query plan or requirement of query.

3) Get Metadata: The compiler sends metadata request to Metastore and in return metastore sends metadata as response to compiler.

4) Execute Plan: The compiler checks the requirement and resends the plan to the driver. The driver sends the execute plan to the execution engine.

5) Execute Job: An execution engine, such as Tez or MapReduce, executes the compiled query. The resource manager, YARN, allocates resources for applications across the cluster. The execution engine sends the job to JobTracker, which is in Name node and it assigns this job to TaskTracker, which is in Data node. Here, the query executes MapReduce job.

6) Fetch Query Result : The execution engine receive the result from data nodes and send it to the driver, which return the result to the hive interface over a JDBC/ODBC connection.

After knowing all these internal stuff about execution of hive queries, I hope you will find it more interesting to work on it.

References:

  1. https://www.tutorialspoint.com/hive/hive_introduction.htm
  2. https://en.wikipedia.org/wiki/Apache_Hive