Skip to content

Assignment 3: HTCondor

While continuing towards Grid Computing, our batch was asked to work on HTCondor (50% of the batch) and Globus (the remaining 50%) which are the softwares for Grid Computing.
I got HTCondor. None of my batchmates I know was successful in carrying out the assignment beyond the installation part.

OK, so let’s begin with what HTCondor is.

HTCondor is a specialized workload management system for compute-intensive jobs. Like other full-featured batch systems, HTCondor provides a job queueing mechanism, scheduling policy, priority scheme, resource monitoring, and resource management. Users submit their serial or parallel jobs to HTCondor, HTCondor places them into a queue, chooses when and where to run the jobs based upon a policy, carefully monitors their progress, and ultimately informs the user upon completion.
While providing functionality similar to that of a more traditional batch queueing system, HTCondor’s novel architecture allows it to succeed in areas where traditional scheduling systems fail. HTCondor can be used to manage a cluster of dedicated compute nodes (such as a “Beowulf” cluster). In addition, unique mechanisms enable HTCondor to effectively harness wasted CPU power from otherwise idle desktop workstations. For instance, HTCondor can be configured to only use desktop machines where the keyboard and mouse are idle. Should HTCondor detect that a machine is no longer available (such as a key press detected), in many circumstances HTCondor is able to transparently produce a checkpoint and migrate a job to a different machine which would otherwise be idle. HTCondor does not require a shared file system across machines – if no shared file system is available, HTCondor can transfer the job’s data files on behalf of the user, or HTCondor may be able to transparently redirect all the job’s I/O requests back to the submit machine. As a result, HTCondor can be used to seamlessly combine all of an organization’s computational power into one resource.

HTCondor is the product of years of research by the Center for High Throughput Computing in the Department of Computer Sciences at the University of Wisconsin-Madison (UW-Madison), and it was first installed as a production system in the UW-Madison Department of Computer Sciences over 15 years ago. This HTCondor installation has since served as a major source of computing cycles to UW-Madison faculty and students. Additional HTCondor installations have been established over the years across our campus and the world. Hundreds of organizations in industry, government, and academia have used HTCondor to establish compute installations ranging in size from a handful to many thousands of workstations.

The HTCondor software, source code, and complete documentation are freely available under an open source license. Linux, MacOS, and Windows platforms are supported.

Installation
I downloaded the .deb file from this site: http://research.cs.wisc.edu/htcondor/downloads/.
I followed the installation instructions given on
After installation, I did the following changes:
My machine was the central manager. While the other two partners’ machines were allowed to submit jobs to my pool. For this purpose, the configuration file of HTCondor (/etc/condor/condor_config) was modified as follows:I wrote my IP against the “central manager” row. And listed the IPs of the other two machines against the ALLOW_WRITE row. In the FLOCK_FROM, I did the same thing.
We also did the same thing on the other two machines except that the FLOCK_FROM field in those machines were set to my IP address.

Then we also modified firewall settings to bypass local access to one another’s PC using the command ufw allow from <IP>.

After doing all this, I started the condor service using sudo service condor restart and tried submitting jobs to my pool using condor_submit <job name> command. But the other machines were not able to submit jobs. They saw a Permission denied error.

We tried to solve this problem multiple times but were unsuccessful in doing so. There’s not much content about Condor on the internet other than that on the University of Wisconsin’s website. Ultimately, we had to leave the assignment upto halfway mark only.

Assignment 2: Building a simple Beowulf cluster with Ubuntu

“A Beowulf cluster is a group of what are normally identical, commercially available computers, which are running a Free and Open Source Software (FOSS), Unix-like operating system, such as BSD, GNU/Linux, or Solaris. They are networked into a small TCP/IP LAN, and have libraries and programs installed which allow processing to be shared among them.”

This means a Beowulf cluster can be easily built with “off the shelf” computers running GNU/Linux in a simple home network. So building a Beowulf like cluster is within reach if you already have a small TCP/IP LAN at home with desktop computers running Ubuntu Linux (or any other Linux distribution).

There are many ways to install and configure a cluster. There is OSCAR(1), which allows any user, regardless of experience, to easily install a Beowulf type cluster on supported Linux distributions. It installs and configures all required software according to user input.

There is also the NPACI Rocks toolkit, which incorporates the latest Red Hat distribution and cluster-specific software. Rocks addresses the difficulties of deploying manageable clusters. Rocks makes clusters easy to deploy, manage, upgrade and scale.

Both of the afore mentioned toolkits for deploying clusters were made to be easy to use and require minimal expertise from the user. But the purpose of this tutorial is to explain how to manually build a Beowulf like cluster. Basically, the toolkits mentioned above do most of the installing and configuring for you, rendering the learning experience mute. So it would not make much sense to use any of these toolkits if you want to learn the basics of how a cluster works. This tutorial therefore explains how to manually build a cluster, by manually installing and configuring the required tools.

I have followed the exact steps from the following blog:

http://techtinkering.com/2009/12/02/setting-up-a-beowulf-cluster-using-open-mpi-on-linux/

So, I am not writing the same thing here. Please see the contents of the already written blog. I and one of my friends Kuldeep served as the compute nodes while Amit’s PC was the master node.

Note that since the MPICH on my machine was > 1.2.x so I installed Hydra in place of MPD.
We encountered a hell lot of trouble in creating the Beowulf cluster.
Also, note that you have to enable root password on your Linux machine as well. Plus, allow firewall on every compute node so that the master node can be able to access it.

Assignment 1: Virtualization

The task is to perform virtualization by downloading Virtual Box and creating virtual machine with CentOS/BSD/Backtrack/Windows as the guest OS on linux platform*.
*The guest OS can be any other than host OS(linux)

I tried installing CentOS on Virtual Box but was unable to complete the installation. It got stuck at the end of installation. I tried 2-3 times but didn’t work on my PC. So, I used another virtualization software-VMWare. I am writing how I created virtual machine on VMware.

First of all, we’ve to download CentOS. CentOS is a freely-available operating system that is based on Red Hat Enterprise Linux system. You can download the latest release from its website. To download, please refer to download page of CentOS web site

http://isoredirect.centos.org/centos/5/isos/i386/

Select nearest location and begin downloading *.iso files.

After you finish downloading *.iso files, you have to mount it and let your VMware recognize it as a virtual “cd” or “dvd”. For this purpose, I prefer to download one DVD *.iso installation file. Please open VMware, double click on CD-ROM (IDE 1-0) item, select Use ISO image option and click Browse button to select your recently downloaded *.iso installation file of CentOS.

Now you virtually inserted you installation DVD into DVD-ROM. And you’re ready to install your OS virtually.

You should change your Memory to 1024 MB for your further Oracle installation. Double click on Memory item and set it to 1024.

Then click Power button to start your virtual pc. When your virtual pc starts, you’ll see welcome screen of installation CentOS.
Then you’ll get another welcome screen, you only have to click next button on this screen. Select your language of installation and click Next. Then, select your keyboard configuration to use it for the system you’re installing and click Next. Then, you’ll be warned to create new partitions by erasing ALL DATA on your newly created hard drive. As there’s no information in this drive, you’ll select Yes. Then there’ll be warning “You have chosen to remove all Linux partitions (and ALL DATA on them) on the following drives: /dev/sda Are you sure you want to do this?” Click Yes

Then, you’ll see Network Configurations screen, as it is your test virtual pc, don’t change anything and click next. You can change whatever you want after installation, for now, let it continue in this way and click Next

After the installation is completed, click Reboot button to reboot your OS.

After reboot, you’ll face with another Welcome screen. There’re some steps need to be completed
Click Forward button on the screen.
Our Centos is ready to use!

Balance BST (AVL Tree) Tutorial

AVL tree is height balance binary search tree.

a tree is called balanced if all of its nodes have their balance factor lies between [-1,0,1],otherwise it is unbalanced.

Balance Factor : It is the difference between heights of left subtree and right subtree of a node.

here is solution of avl tree.

step 1: Insert newnode in to Binary search tree.

step 2: Find balance factor of each node (by applying any traversal (inorder , preorder , postorder)).

step 3: Find node which is not balanced.(Note that there may be more than one unbalanced node but you will have to find the node which have maximum defth because by balancing that node whole tree will automatically balance otherwise it may or may not be balanced).

step 4: Find parent node and child node of the node which you obtained in step 3.

step 5: Rotate the tree for balancing balancing the tree.

Rotation Rules :

let say unbalanced node is ptr and its child is child(Note that here child is that side of child of ptr in which node was inserted so you have to take care of it).

1. If balance factor(bf) of both child and ptr is >= 0 then tree is left heavy its imply right rotation.

2. If balance factor(bf) of both child and ptr is < 0 then tree is right heavy its imply left rotation.

3. If balance factor(bf) of child <0 and  ptr is >= 0 then tree is left heavy its imply right rotation then left rotation.(refer cormen page no. 314)

4. If balance factor(bf) of child >=0 and  ptr is < 0 then tree is right heavy its imply left rotation then right rotation.

please post your ideas in comments.

and like us on facebook CLICK HERE

Just Next

DevG likes too much fun to do with numbers. Once his friend Arya came and gave him a challenge, he gave DevG an array of digits which is forming a number currently (will be called as given number). DevG was challanged to find the just next greater number which can be formed using digits of given number. Now DevG needs your help to find that just next greater number and win the challenge.

Input: 

The first line have t number of test cases (1<=t<=100). In next 2*t lines for each test case first there is number n (1<=n<=1000000) which denotes the number of digits in given number and next line contains n digits of given number separated by space.

Output:

Print the just next greater number if possible else print -1 in one line for each test case.

Note : There will be no test case which contains zero in starting digits of any given number.

Sample Test Cases :

Input : 

2

5

1 5 4 8 3

10

1 4 7 4 5 8 4 1 2 6

Output :

15834

1474584162

Inversion Count

Let A[0…n – 1] be an array of n distinct positive integers. If i < j and A[i] > A[j] then the pair (i, j) is called an inversion of A. Given n and an array A your task is to find the number of inversions of A.

Input

The first line contains t, the number of testcases followed by a blank space. Each of the t tests start with a number n (n <= 200000). Then n + 1 lines follow. In the ith line a number A[i – 1] is given (A[i – 1] <= 10^7). The (n + 1)th line is a blank space.

Output

For every test output one line giving the number of inversions of A.

Example

Input:

2

3
3
1
2

5
2
3
8
6
1


Output:

2
5

Minimum From An Array

You are given a large array.you have to find minimum element from the array.Time complexity should not be greater than O(log n).

Meeting Point

There is an infinite integer grid at which N people have their houses on. They decide to unite at a common meeting place, which is someone’s house.
From any given cell, all 8 adjacent cells are reachable in 1 unit of time.
eg: (x,y) can be reached from (x-1,y+1) in a single unit of time.
Find a common meeting place which minimises the sum of the travel times of all the persons.

Input Format:
N
The following N lines will contain two integers saying the x & y coordinate of the i-th person.

Output Format:
M M = min sum of all travel times;

Constraints:
N <= 10^5
The absolute value of each co-ordinate in the input will be atmost 109
HINT: Please use long long 64-bit integers;

Input #00:
4
0 1
2 5
3 1
4 0

Output #00:
8

Explanation: Sums of travel times of the houses are 11, 13, 8 and 10. 8 is the minimum.

Input #01:
6
12 -14
-3 3
-14 7
-14 -3
2 -12
-1 -6
Output #01:
54

Download Sample TestCases

Pairs

Given N numbers , [N<=10^5] we need to count the total pairs of numbers that have a difference of K. [K>0 and K<1e9]

Input Format:
1st line contains N & K (integers).
2nd line contains N numbers of the set. All the N numbers are assured to be distinct.
Output Format:
One integer saying the no of pairs of numbers that have a diff K.

Sample Input #00:
5 2
1 5 3 4 2

Sample Output #00:
3

Sample Input #01:
10 1
363374326 364147530 61825163 1073065718 1281246024 1399469912 428047635 491595254 879792181 1069262793
Sample Output #01:
0

Grid Walking

You are situated in an N dimensional grid at position(x_1,x2,…,x_N). The dimensions of the grid are (D_1,D_2,…D_N). In one step, you can walk one step ahead or behind in any one of the N dimensions. (So there are always 2N possible different moves). In how many ways can you take M steps such that you do not leave the grid at any point ? You leave the grid if you for any x_i, either x_i <= 0 or x_i > D_i.

Input:
The first line contains the number of test cases T. T test cases follow. For each test case, the first line contains N and M, the second line contains x_1,x_2…,x_N and the 3rd line contains D_1,D_2,…,D_N.

Output:
Output T lines, one corresponding to each test case. Since the answer can be really huge, output it modulo 1000000007.

Constraints: 1 <= T <= 10
1 <= N <= 10
1 <= M <= 300
1 <= D_i <= 100
1 <= x_i <= D_i

Sample Input #00:
10
1 287
44
78
1 236
25
87
1 122
41
63
1 260
7
64
1 127
3
73
1 69
6
68
1 231
14
63
1 236
13
30
1 259
38
70
1 257
11
12
Sample output: 
38753340
587915072
644474045
423479916
320130104
792930663
846814121
385120933
60306396
306773532

Download Testcases As Zip