Shell Scripting Examples

Generate comma separated dates for last seven days:

# Fetch the date 7 days ago
DATE=$(date -d '7 day ago' '+%Y-%m-%d');

# Generate comma-separated string for last 7 days
for i in {6..1}; do DATE=$DATE","$(date -d "$i day ago" '+%Y-%m-%d'); done

Considering current date is 2016-04-08, DATE variable value will be as follows:

2016-04-01,2016-04-02,2016-04-03,2016-04-04,2016-04-05,2016-04-06,2016-04-07

We can use different date formats on a case to case basis. This thing comes in handy when we have to feed a date range to some reporting tool.

Generate comma separated dates for last month and from today to start of the day:

	# This script generate comma separated dates for last month in yyyy-mm-dd format.
	# Ex:
	# "2018-04-01,2018-04-02,2018-04-03,2018-04-04,2018-04-05,2018-04-06,2018-04-07,
	# 2018-04-08,2018-04-09,2018-04-10,2018-04-11,2018-04-12,2018-04-13,2018-04-14,
	# 2018-04-15,2018-04-16,2018-04-17,2018-04-18,2018-04-19,2018-04-20,2018-04-21,
	# 2018-04-22,2018-04-23,2018-04-24,2018-04-25,2018-04-26,2018-04-27,2018-04-28,
	# 2018-04-29,2018-04-30"
	set `date +%m" "%Y`
	CURMTH=$1
	CURYR=$2
	if [ $CURMTH -eq 1 ]
	then
	PRVMTH=12
	PRVYR=`expr $CURYR - 1`
	else
	PRVMTH=`expr $CURMTH - 1`
	PRVYR=$CURYR
	fi
	if [ $PRVMTH -lt 10 ]
	then PRVMTH="0"$PRVMTH
	fi
	LASTDY=`cal $PRVMTH $PRVYR | egrep "28|29|30|31" |tail -1 |awk '{print $NF}'`
	FROM_DATE="$PRVYR-$PRVMTH-01"
	TO_DATE="$PRVYR-$PRVMTH-$LASTDY"
	# convert in seconds sinch the epoch:
	start=$(date -d$FROM_DATE +%s)
	end=$(date -d$TO_DATE +%s)
	cur=$start
	while [ $cur -le $end ]; do
	# convert seconds to date:
	REPORT_DATE=$REPORT_DATE$(date -d@$cur +%Y-%m-%d)",";
	let cur+=24*60*60
	done
	# Remove last comma (,)
	REPORT_DATE=${REPORT_DATE::-1}
	echo $REPORT_DATE;

view raw generate-dates-for-last-month.sh hosted with ❤ by GitHub

	# This script generates the dates from today till the start of the month in yyyy-mm-dd format.
	# Ex. If current date is 2018-05-03 then it will generate "2018-05-01,2018-05-02,2018-05-03".
	set `date +%m" "%Y`
	CURMTH=$1
	CURYR=$2
	CURRENT_DATE=`date +%d`;
	FROM_DATE="$CURYR-$CURMTH-01"
	TO_DATE="$CURYR-$CURMTH-$CURRENT_DATE"
	# convert in seconds sinch the epoch:
	start=$(date -d$FROM_DATE +%s)
	end=$(date -d$TO_DATE +%s)
	cur=$start
	while [ $cur -le $end ]; do
	# convert seconds to date:
	REPORT_DATE=$REPORT_DATE$(date -d@$cur +%Y-%m-%d)",";
	let cur+=24*60*60
	done
	# Remove last comma (,)
	REPORT_DATE=${REPORT_DATE::-1}
	echo $REPORT_DATE;

view raw generate-dates-from-today-till-start-of-the-month.sh hosted with ❤ by GitHub

Java - SQL Parsing

Recently I had a use case where I have to validate a SQL query. Basically I had to look at the JOINS and WHERE clauses in the query.

I thought that why to parse a SQL on my own and extract table names and where clauses from the query. Why to reinvent the wheel and write your own logic when you can re-use existing open source libraries. Keeping that in mind I started searching for a simple open source library to serve the purpose.

Surprisingly I found only a few and those were not serving the purpose then I found Apache-Calcite(https://calcite.apache.org/). Apache-Calcite is a huge project and SQL parsing is just one part of it.

I didn't find a good example though using its SQL parser and had to dig in a bit to get a working example. Following is a working example:

	public class BaseQueryValidator {
	private static List<String> extractTableAliases(SqlNode node) {
	final List<String> tables = new ArrayList<>();
	// If order by comes in the query.
	if (node.getKind().equals(SqlKind.ORDER_BY)) {
	// Retrieve exact select.
	node = ((SqlSelect) ((SqlOrderBy) node).query).getFrom();
	} else {
	node = ((SqlSelect) node).getFrom();
	}
	if (node == null) {
	return tables;
	}
	// Case when only 1 data set in the query.
	if (node.getKind().equals(SqlKind.AS)) {
	tables.add(((SqlBasicCall) node).operand(1).toString());
	return tables;
	}
	// Case when there are more than 1 data sets in the query.
	if (node.getKind().equals(SqlKind.JOIN)) {
	final SqlJoin from = (SqlJoin) node;
	// Case when only 2 data sets are in the query.
	if (from.getLeft().getKind().equals(SqlKind.AS)) {
	tables.add(((SqlBasicCall) from.getLeft()).operand(1).toString());
	} else {
	// Case when more than 2 data sets are in the query.
	SqlJoin left = (SqlJoin) from.getLeft();
	// Traverse until we get a AS.
	while (!left.getLeft().getKind().equals(SqlKind.AS)) {
	tables.add(((SqlBasicCall) left.getRight()).operand(1).toString());
	left = (SqlJoin) left.getLeft();
	}
	tables.add(((SqlBasicCall) left.getLeft()).operand(1).toString());
	tables.add(((SqlBasicCall) left.getRight()).operand(1).toString());
	}
	tables.add(((SqlBasicCall) from.getRight()).operand(1).toString());
	return tables;
	}
	return tables;
	}
	private static Map<String, String> extractWhereClauses(SqlNode node) {
	final Map<String, String> tableToPlaceHolder = new HashMap<>();
	// If order by comes in the query.
	if (node.getKind().equals(SqlKind.ORDER_BY)) {
	// Retrieve exact select.
	node = ((SqlOrderBy) node).query;
	}
	if (node == null) {
	return tableToPlaceHolder;
	}
	final SqlBasicCall where = (SqlBasicCall) ((SqlSelect) node).getWhere();
	if (where != null) {
	// Case when there is only 1 where clause
	if (where.operand(0).getKind().equals(SqlKind.IDENTIFIER)
	&& where.operand(1).getKind().equals(SqlKind.LITERAL)) {
	tableToPlaceHolder.put(where.operand(0).toString(),
	where.operand(1).toString());
	return tableToPlaceHolder;
	}
	final SqlBasicCall sqlBasicCallRight = where.operand(1);
	SqlBasicCall sqlBasicCallLeft = where.operand(0);
	// Iterate over left until we get a pair of identifier and literal.
	while (!sqlBasicCallLeft.operand(0).getKind().equals(SqlKind.IDENTIFIER)
	&& !sqlBasicCallLeft.operand(1).getKind().equals(SqlKind.LITERAL)) {
	tableToPlaceHolder.put(((SqlBasicCall) sqlBasicCallLeft.operand(1)).operand(0).toString(),
	((SqlBasicCall) sqlBasicCallLeft.operand(1)).operand(1).toString());
	sqlBasicCallLeft = sqlBasicCallLeft.operand(0); // Move to next where condition.
	}
	tableToPlaceHolder.put(sqlBasicCallLeft.operand(0).toString(),
	sqlBasicCallLeft.operand(1).toString());
	tableToPlaceHolder.put(sqlBasicCallRight.operand(0).toString(),
	sqlBasicCallRight.operand(1).toString());
	return tableToPlaceHolder;
	}
	return tableToPlaceHolder;
	}
	public static void main(String[] args) throws SqlParseException {
	final String query = "SELECT e.first_name AS FirstName, s.salary AS Salary from employee AS e join salary AS s on e.emp_id=s.emp_id where e.organization = 'Tesla' and s.organization = 'Tesla'";
	final SqlParser parser = SqlParser.create(baseQuery);
	final SqlNode sqlNode = parser.parseQuery();
	final SqlSelect sqlSelect = (SqlSelect) sqlNode;
	final SqlJoin from = (SqlJoin) sqlSelect.getFrom();
	// Extract table names/data sets, For above SQL query : [e, s]
	final List<String> tables = extractTableNames(from);
	// Extract where clauses, For above SQL query : [e.organization -> 'Tesla', s.organization -> 'Tesla']
	final Map<String, String> whereClauses = extractWhereClauses(sqlSelect);
	}
	}

view raw BaseQueryValidator.java hosted with ❤ by GitHub

So this is just about extracting table names and where clauses. This SQL parser can do definitely more and can be used to extract other parts of the query as well. Of course you can dig in the java docs and API to cater your requirements. Happy Learning !!!

Unit testing your POJO classes in Java.

Ever bothered about testing your pojo classes in java. Probably NO. We tend to ignore these pojo classes saying why should I test lame methods i.e. setter, getters, toString etc.

However these pojo classes adds to your test coverage report and if you don't have tests written for these classes you'll end up with a reduced test coverage stats for your java project.

What about delegating the testing responsibility to a framework or library which will ensure the quality of your pojo classes. Cool Huh !!!

I have got such a situation in my project and found this cool library POJO-TESTER to serve the purpose.

What about giving it a quick look around? Happy Testing !!!!

Push messages to browser/clients with NodeJS WebSocket

Hi Folks, off late I have been trying to do a POC on pushing messages from server to all the connected clients and found that websocket can be used for it in conjunction with NodeJS. So I am putting it all over to this post.

Following are the prerequisites in order to run the setup:

1. NodeJS Installation (We need npm to install packages) Download Link
2. NPM Packages (websocket, http-server, finalhandler, serve-static).

NPM package installation commands:

• npm install websocket
• npm install http-server
• npm install finalhandler
• npm install serve-static

After successful installation of NodeJS and above mentioned npm packages we can start off writing the code. We need following three files hosted on a folder (All the npm package installation commands will be executed on this folder).

1. frontend.html.
2. frontend.js (NodeJS Frontend).
3. backend.js (NodeJS Backend).

Following is the source code of above files:

frontend.html

	<html>
	<head>
	<meta charset="utf-8">
	<title>WebSockets - Simple chat</title>
	<style>
	#content { padding:5px; background:#ddd; border-radius:5px; overflow-y: scroll;
	border:1px solid #CCC; margin-top:10px; height: 160px; }
	#label { margin-top:20px; }
	#primary-panel { margin:10px 10px 10px 10px; }
	#input { border-radius:2px; border:1px solid #ccc;
	margin:10px 10px 10px 10px; padding:5px; width:400px; }
	#status { display:block; float:left; margin:15px -5px 10px 10px; }
	</style>
	<link rel="stylesheet" type="text/css" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css"/>
	</head>
	<body>
	<h2 align="center"><u>NodeJS WebSocket Tutorial</u></h2>
	<div id="primary-panel" class="panel panel-primary">
	<div class="panel-heading">
	<h3 class="panel-title">Chat</h3>
	</div>
	<span id="status">Connecting...</span>
	<input type="text" id="input" disabled="disabled" /><br/>
	</div>
	</div>
	<div id="primary-panel" class="panel panel-primary">
	<div class="panel-heading">
	<h3 class="panel-title">Content from server</h3>
	</div>
	<div id="server-content" class="panel-body">
	</div>
	</div>
	<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.1.1/jquery.min.js"></script>
	<script src="./chat-frontend.js"></script>
	<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js"></script>
	</body>
	</html>

view raw frontend.html hosted with ❤ by GitHub

frontend.js

	$(function() {
	"use strict";
	// for better performance - to avoid searching in DOM
	var content = $('#content');
	var input = $('#input');
	var status = $('#status');
	// my color assigned by the server
	var myColor = false;
	// my name sent to the server
	var myName = false;
	// if user is running mozilla then use it's built-in WebSocket
	window.WebSocket = window.WebSocket || window.MozWebSocket;
	// if browser doesn't support WebSocket, just show some notification and exit
	if (!window.WebSocket) {
	content.html($('<p>', {
	text: 'Sorry, but your browser doesn\'t ' +
	'support WebSockets.'
	}));
	input.hide();
	$('span').hide();
	return;
	}
	// open connection
	var connection = new WebSocket('ws://127.0.0.1:1337');
	connection.onopen = function() {
	// first we want users to enter their names
	input.removeAttr('disabled');
	status.html('<b>Message to send:</b>');
	};
	connection.onerror = function(error) {
	// just in there were some problems with conenction...
	content.html($('<p>', {
	text: 'Sorry, but there\'s some problem with your ' +
	'connection or the server is down.'
	}));
	};
	// most important part - incoming messages
	connection.onmessage = function(message) {
	// try to parse JSON message. Because we know that the server always returns
	// JSON this should work without any problem but we should make sure that
	// the massage is not chunked or otherwise damaged.
	try {
	var json = JSON.parse(message.data);
	prepareMessage(json);
	input.removeAttr('disabled').focus();
	} catch (e) {
	console.log('This doesn\'t look like a valid JSON: ', message.data);
	return;
	}
	};
	/**
	* Send mesage when user presses Enter key
	*/
	input.keydown(function(e) {
	if (e.keyCode === 13) {
	var msg = $(this).val();
	if (!msg) {
	return;
	}
	// send the message as an ordinary text
	connection.send(msg);
	$(this).val('');
	// disable the input field to make the user wait until server
	// sends back response
	input.attr('disabled', 'disabled');
	}
	});
	/**
	* This method is optional. If the server wasn't able to respond to the
	* in 3 seconds then show some error message to notify the user that
	* something is wrong.
	*/
	setInterval(function() {
	if (connection.readyState !== 1) {
	status.text('Error');
	input.attr('disabled', 'disabled').val('Unable to comminucate ' +
	'with the WebSocket server.');
	}
	}, 3000);
	/**
	* This methods appends the message coming from server as part of JSON to 'content' <div> in the HTML file.
	*/
	function prepareMessage(message) {
	content.append('<br/>' + message.data.text);
	}
	});

view raw frontend.js hosted with ❤ by GitHub

backend.js

	// http://ejohn.org/blog/ecmascript-5-strict-mode-json-and-more/
	"use strict";
	// Optional. You will see this name in eg. 'ps' or 'top' command
	process.title = 'node-chat';
	// Port where we'll run the websocket server
	var webSocketsServerPort = 1337;
	// websocket and http servers
	var webSocketServer = require('websocket').server;
	var http = require('http');
	/**
	* Global variables
	*/
	// latest 100 messages
	var history = [];
	// list of currently connected clients (users)
	var clients = [];
	var finalhandler = require('finalhandler');
	var serveStatic = require('serve-static');
	var serve = serveStatic("./");
	/**
	* Helper function for escaping input strings
	*/
	function htmlEntities(str) {
	return String(str).replace(/&/g, '&').replace(/</g, '<')
	.replace(/>/g, '>').replace(/"/g, '"');
	}
	/**
	* HTTP server
	*/
	var server = http.createServer(function (request, response) {
	var done = finalhandler(request, response);
	serve(request, response, done);
	});
	server.listen(webSocketsServerPort, function () {
	console.log('[' + (new Date().toLocaleString()) + '] -- Server is listening on port ' + webSocketsServerPort);
	});
	/**
	* WebSocket server
	*/
	var wsServer = new webSocketServer({
	// WebSocket server is tied to a HTTP server. WebSocket request is just
	// an enhanced HTTP request. For more info http://tools.ietf.org/html/rfc6455#page-6
	httpServer: server
	});
	// This callback function is called every time someone
	// tries to connect to the WebSocket server
	wsServer.on('request', function (request) {
	console.log('[' + (new Date().toLocaleString()) + '] -- Connection from origin ' + request.origin + '.');
	// accept connection - you should check 'request.origin' to make sure that
	// client is connecting from your website
	// (http://en.wikipedia.org/wiki/Same_origin_policy)
	var connection = request.accept(null, request.origin);
	// we need to know client index to remove them on 'close' event
	var index = clients.push(connection) - 1;
	var userName = false;
	var userColor = false;
	console.log('[' + (new Date().toLocaleString()) + '] -- Connection accepted.');
	// user sent some message
	connection.on('message', function (message) {
	if (message.type === 'utf8') { // accept only text
	// log and broadcast the message
	var msg = '[' + (new Date().toLocaleString()) + '] -- Received Message from Client-' + index + ': ' + message.utf8Data;
	console.log(msg);
	// we want to keep history of all sent messages
	var obj = {
	text: htmlEntities(msg)
	};
	// broadcast message to all connected clients
	var json = JSON.stringify({
	type: 'message',
	data: obj
	});
	for (var i = 0; i < clients.length; i++) {
	clients[i].sendUTF(json);
	}
	}
	});
	// user disconnected
	connection.on('close', function (connection) {
	console.log((new Date()) + " Peer " +
	connection.remoteAddress + " disconnected.");
	// remove user from the list of connected clients
	clients.splice(index, 1);
	});
	});
	setInterval(function () {
	var msg = '[' + (new Date().toLocaleString()) + '] -- Active Clients: ' + clients.length;
	var obj = {
	text: htmlEntities(msg)
	};
	var json = JSON.stringify({
	type: 'message',
	data: obj
	});
	for (var i = 0; i < clients.length; i++) {
	clients[i].sendUTF(json);
	}
	}, 3000);

view raw backend.js hosted with ❤ by GitHub

Following is the directory structure of project:

Limit typing numbers in input type 'number' HTML

Recently I have started front-end development using AngularJS where I had the following requirement:

1. Open a modal window.
2. Paint form on the modal.
3. Form will have a text field of type "number" with following criteria:
1. User can put only positive numbers in the text field nothing else.
2. User can put numbers only from 0-999.

At first above requirement seems to be very easy to implement with min and max properties of input type "number". However there is a catch that min and max restricts only the spinner (increase and decrease) but user can still type numbers which does not belong to 0-999 (Even characters).

To address first criteria we can use onkeypress event and pass it a function which will listen to 0-9 number key presses only and discard the rest:

onkeypress="return (event.charCode == 8 || event.charCode == 0) ? null : event.charCode >= 48 && event.charCode <= 57"

To address second criteria we can use oninput event and pass it a function which will clip/remove the characters after 3rd position:

oninput="this.value.length > 3 ? this.value= this.value.slice(0,3) : this.value"

Note: I have seen suggestions of an alternate approach to accomplish the above using onkeydown event however I have found that if we use onkeydown event it screws up the spinner and also allows text to be pasted and dragged having length more than 3 characters whereas oninput handles all these cases.

Following is the complete input tag:

<input type="number"
       name="sampleTextbox"
       min="0"
       max="999"
       oninput="this.value.length > 3 ? this.value= this.value.slice(0,3) : this.value"
       onkeypress="return (event.charCode == 8 || event.charCode == 0) ? null : event.charCode >= 48 && event.charCode <= 57"/>

Frequency of occurrence of Strings/Words in a list.

I had a problem where I have to keep a track of the frequency of occurrence of a given string in a record (row). I was thinking of doing this using a Map (Probably the best candidate here) where the key is the string and value is the frequency of occurrence.

But there was a catch here, what should we do on the first occurrence of the string as it won't be present in the Map. Below is the solution I came up with:

import java.util.HashMap;

public class MyHashMap {
       private HashMap<String, Integer> myHashMap;

       public MyHashMap(final HashMap<String, Integer> myHashMap) {
           this.myHashMap = myHashMap;
       }

       public void put(String key) {
           Integer integer = myHashMap.get(key);

           if (integer == null) {
                myHashMap.put(key, new Integer(1));
           } else {
                myHashMap.put(key, ++integer);
          }
        }

        @Override
        public String toString() {
            return "MyHashMap [myHashMap=" + myHashMap + "]";
         }

        public static void main(String[] args) {
             MyHashMap m = new MyHashMap(new HashMap<>());

             m.put("Steve");
             m.put("Steve");
             m.put("Simon");

             System.out.println(m);
       }
}

Output:

        MyHashMap [myHashMap={Simon=1, Steve=2}]

Above implementation works fine however you can see we had to do first occurrence handling in put method from our side which I wanted to avoid so I was looking at some other solution. No problem Java 8 Stream API is here and does this thing very smoothly in one liner.

Following is the Java 8 code snippet :

import java.util.ArrayList;

import java.util.List;

import java.util.Map;

import java.util.stream.Collectors;

import java.util.stream.Stream;

public class StreamsDemo {

public static void main(String[] args) {

List<String> list = new ArrayList<>();

list.add("Steve");

list.add("Steve");

list.add("Frank");

list.add("Tom");

list.add("Steve");

Stream<String> stream = list.stream();

Map<String, Long> collect = stream.collect(Collectors.groupingBy(e -> e, Collectors.counting()));

System.out.println(collect);

}

}

Output:

{Frank=1, Tom=1, Steve=3}

Above implementation which uses Java 8 Stream API does the trick as if we are executing a aggregate function COUNT(*) with GROUP BY. Cool !!

Create executable jar with Maven

If you are using Maven and wants to create an executable jar file but not getting an exact solution (Google is providing too many results and nothing is straightforward :-)). So let me get this straight and there is a plugin for it which is maven-assembly-plugin.

Below is the pom.xml which makes use of maven-assembly-plugin to generate executable jar:

<?xml version="1.0" encoding="UTF-8"?>
<project
    xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
       http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.example</groupId>
    <artifactId>test-project</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <properties>
        <targetJdk>1.7</targetJdk>
        <project.build.sourceEncoding>UTF-8
                          </project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8
                          </project.reporting.outputEncoding>
    </properties>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.5.1</version>
                <configuration>
                    <source>1.7</source>
                    <target>1.7</target>
                </configuration>
            </plugin>
            <plugin>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>2.4</version>
                <configuration>
                    <archive>
                        <manifest>
                            <addClasspath>true</addClasspath>
                            <mainClass>com.test.App</mainClass>
                        </manifest>
                    </archive>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies
                                                </descriptorRef>
                    </descriptorRefs>
                </configuration>
                <executions>
                    <execution>
                        <id>make-jar-with-dependencies</id>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
    <dependencies>
        <dependency>
            ......
            ......
        </dependency>
    </dependencies>
</project>

Now run "Maven install" and following executable jar file will be created : test-project-0.0.1-SNAPSHOT-jar-with-dependencies.jar