Linux | Open Source | Agility | CI | CD

Google Apps Active Directory sync.

2012-01-11T02:41:00.000-08:00

Are you planning to migrate from MS suit to Google Apps?

Some time back I did one such migration and came across several posts where people said "Google AD sync doesn't work properly", etc etc.

But I found it very useful and usable, with a fact that not much of the documentation is available around it.

This lack of documentation inspired me to write this blog, hopefully it will ease up your work.

To begin with, you need to have:
* Working MS AD and an account with Google App
* Administrative credentials for both (AD and Google App) accounts.
* "Google Apps Directory Sync tool" installed on your system.

Now lets see how it is done:
* Start the "Configuration Manager" from your startup menu.

* Fill the required details as shown in screenshots.

General Settings: Check the options that you want to migrate to Google Apps. In my case I used only Users and Groups.

Google Apps Settings: Fill the email and password of user with Administrative privileges on Google Apps.

LDAP Connection: This part needs your local LDAP/AD details. The provided user nay not be admin, but needs only read access.

User Attributes: Provide the local (LDAP/AD) attribute for mail ID (in my case it was mail).

User Extended Attribute: This configuration maps local given name, sir name and password attributes to Google Apps.
Specifying Suspension Policy enables the user to decide what to do if a user is present in Google Apps and not in local AD.

User Sync: Add a new rule. This rule will define which users to sync (or which user not to sync). The rule field accepts RegEx, allowing us to add multiple users through one line.

Notification: In this section define your SMTP server and other details required to relay email on success or failure of this sync up.

Sync Limit: This section is to specify the suspension limit. In simpler words "The script should fail if more users are being suspended (than the specified limit)".

Log File: Specify a valid log file path here. The sync tool will save its output in this file.

Anything Else: There are many more options available (example syncing contacts etc), but in this blog I documented only the minimalistic configuration (and that was what I required fro my project).

* Test it out.

Simulate Sync: This tab will tell you if your configuration is correct. And if you don't find any error (warnings are still OK), then just hit "Simulate Sync" @ the bottom.

Result: Hitting Simulate Sync will take you to a new view. Check if you see users being synced and you don't have errors.

* Save this configuration to a file. I saved it as google-sync.xml.

* Run it: As of now this sync can only be run through command line, so just run this command and see the magic.

sync-cmd.exe -a -c "google-sync.xml" -r C:\Sync-logs\Google.Sync.logs

* Schedule it with the help of tool like Jenkins and you are done with one aspect of migration which gives head ache to people.

PS: Scheduling it through a tool makes it easy to monitor and/or run it on demand.

I am confused (Google docs Vs Dropbox)

2012-01-05T03:55:00.000-08:00

One of the interesting thing that I worked recently on was a small decision of either using Google docs or Dropbox for storing and sharing documents.

Seems unimportant?

I also thought the same on hearing this for the first time, but believe me this proved to be an interesting study.

Let me take you to the tour of both.

Google docs is an online solution provided by Google to collaboratively work (sharing, editing, etc) on the documents.
Several addons like Cloud Connect, etc helps us to sync our native MS office documents to Google docs.
One of the advantage of Google docs is that multiple people can edit one single document @ the same time.

On the other hand Dropbox is used to share files, irrespective of the file being a document, image or some thing else.
Same account can be either synced to multiple computers OR same files can be shared to multiple accounts, hence giving feasibility of accessing the files across the globe (on multiple systems).
Feature like sharing and selective sync help us selective share our files with different people.
One of the advantage of Dropbox is the ability to share all sorts of files (not only just documents)

And the confusion starts :-)

It looks pretty simple that Google docs should be used for documents and Dropbox for other files, but the problem statement lies in the fact that Google changes format of our existing MS office/Open office documents to Google docs format (on uploading). In most of the cases this is not an issue, but it created trouble when you have formulas, macros and images in your documents, as
"They may disappear from your uploaded document".
Google also provides a solution to this, in the form of uploading the documents (in this case other files can also be uploaded) in original format (not changing the format to Google docs format).

Here comes another "BUT".

But this also has a drawback, that we can't maintain single source of truth for the documents. To edit any document we have to download it and then again upload, which creates multiple copies in Google docs (which have to be manually deleted).

Conclusion after digging deaper:
Using any of these tool is subjective.

IF:
* You are creating the documents from scratch.
* And the priority is to have simulanious editing of documents, rather than maintaing MS office formatting, example: Attendance sheet etc.
Then Google docs is THE tool.

ELSE IF:
* You want to use your existing docs.
* And the priority is to use the MS office format.
Then Dropbox is THE tool.

Advantages:
* Google can save money sent on MS office licenses.
* Both the tools can save the hassle of maintaining local shared file server.
* Both have there recovery mechanism, so no danger of getting fully F**k*d.
* Both provide full fleged files/documents sharing system (sharing sccross the globe).
* Both provide the confidence of having one single "Source Of Truth" for documents.
* Dropbox provides offline editing of files (as they gets synced when connected back to internet). Same is provided by Google docs when used with Cloud Connect.

As my requirement was more of sharing and offline working, so I choose Dropbox for now :-)

Cross Platform scripting with bash.

2012-01-04T04:10:00.000-08:00

Is this title a bit confusing?

I asked this question to many people and got the same answer as "YES"
In our day to day life when we talk about bash, we refer to Linux as it's OS.

But the main punch of this blog is to mak BASH break the boundaries of Operating System.
I agree that till now bash is not capable of doing this alone.

Now any guesses about the tool?

If you guessed "CYGWIN" then consider yourself as winner ;-)

To make it more clear for people who were not able to guess it, I can put my example here:

Background: I am a consultant and work with all sorts of projects, during my last couple of projects I was automating build & release cycle for a product that is built on four platforms (Linux32, Linux64, Windows32, Windows64), hence I was desperately looking for some cross OS/platform scripting solution.

My Thoughts: I am a BASH guy and as any developer, my first preference was to look for a solution around what I already know, rather than straight away jumping into any new technology and learning it (it was my second option though).
If it was not bash then I was looking into ant or others to do the job.
As
"The best way to do a work, is to do it." --Harpreet Singh

For Newbies: Cygwin is Linux-like environment for Windows making it possible to port software running on POSIX systems (such as Linux, BSD, and Unix systems) to Windows.

Solution: Now lets put this together.
Install Cygwin on a windows system with basic features like "openssh", "vim", etc.
That's it you are ready to rock 'n' roll.
All most all your Linux commands can run in this shell, which includes doing "ssh" into a Windows system ;-)

Need More: The story is not over yet. Combining the above setup with a CI tool (Jenkins in my case), gave me what I wanted.
A full fledged automated build system, running on four platforms simultaneously (using Jenkins' multi configuration project feature), with ease of writing the script only once and that too using BASH.

Now that's what I call a win ;-)

The winners prize: As a prize of my effort done for the above setup I got:
* Time saved per build/release cycle.
* Time saved in developing automation scripts.

And more important a "Happy Client"

Nagios: Monitoring Systems & Applications

2011-03-16T09:46:00.000-07:00

"Someone please help me."
These are the first words we usually say when we see our production system stuck/crashed due to some reasons.
But "Someone helps those who help themselves." --Harpreet Singh ;-)

Now jokes apart.
But suppose we get an alert before the systems fails may be during the first stroke or when the load starting going high or may be when total processes were more or any thing related to our applications running on the server.

Wouldn't this be like a boon, a chance to save the system in time?
If you have seen my earlier posts you will find monit/munin doing the same, but as on the way of my learning, I found that nagios is a better (easy and more flexible/plugable) tool.
Before starting to explain on why my opinion changed I will ask you one question here.
What do you expect/need from any system/application monitoring tool?

The general answers would be:
- Stable.
- Good UI.
- Easy Installation.
- Easy configuration.
- Good coverage over different applications and system.
- etc. etc.

Now lets see if nagios answers all of these?

Like other tools nagios also has client-server architecture, which gives us freedom to monitor any number of systems/applications from one nagios server.
It has a easy to understand & configure UI, through which you can do many things like scheduling, controlling alerts etc. And if you are a CLI lover (as most of Linux geeks are) then you can do all those from command line also.

Now here comes the most impressive part.
Nagios is highly flexible. First of all it has huge plugin base already available for you to work with.
But if that is not enough for you, then ask yourself just one questions.
Do I know how to write a script (bash, python etc)?

I usually say one line for nagios, that "If you can do it through CLI, you definitely can do it with nagios." Same is the answer for the question you asked yourself (above). If you can write a script to perform any action (login check, api calls, application query etc.) and get a small readable/understandable output (for both
success and failure cases). Then it's like a kids play to integrate it to nagios and see the same results in UI.

In simple words:
- Write a script to perform certain action.
- Copy that to the nagios script directory (just to ensure that you/anyone doesn't accidentally deletes it).
- Add that to the nagios commands.
- Call that command for the host you want.
- And done.

Another plus part is that you can flaunt in front of your seniors about the work done (with minimal effort involved) ;-)

Why is my application slow?

2011-03-14T10:21:00.000-07:00

Wondering what kind of blog post is this?
So let me take you on a brief tour.

Some time back I got a chance to work on Performance Testing of our application (@ openbravo).

On this opportunity one quick questions popped up in my mind: "what do we seek in any performance testing tool?"
- Easy to adjust with our requirement.
- Easy to operate.
- Easy reports at the end.

In our case Jmeter answered them all.
- With Jmeter recorder we were able to record the flow of the application. And with some adjustments like variables, assertions, halts, etc. we reproduced the scenario where a normal user will be using the application.
- After creating the script we can easily run it either from Jmeter UI console or CLI.
- Now for reports Jmeter itself has good number of ways in which we can view reports (graphs, tree view etc), but for sharing those reports is a bit of overhead, so another simple way is to run the test through Hudson (CI tool) and view reports using Hudson's Jmeter plugin.

So basically before any end user/customer shouts "Why is my application slow?", this tool integrated with hudson does a regular check of the delay, failed requests, etc to track the performance and give you an upper hand over other similar applications.

When I was on my way with this work someone asked me "RM and performance testing?", "shouldn't QA do that for you?"
I have a single answer to such question and truly writing there is nothing like mine or your work. It's only a perception, if we have time and scope of doing something then I think we should extend our hands and do it.

Fun With SFTP

2010-10-29T06:06:00.000-07:00

Until now I never knew that using FTP can be so easy.
Many of you would have started thinking about the FTP servers till now, but to be more clear here I am talking about SFTP (SSH File Transfer Protocol).

But as Shakespeare said "What's in name, the thing we call rose would smell as sweet with some other name".
And so is true for SFTP, as this provided the usability that I was looking for and that too with minimal configuration with some extra benefits which we will talk about in the last.

And not only that I was able to do this in two different ways.

One is like my as colleague Juan Pablo says "It should be a JAIL for the user", so that he cannot move outside the directory.
And the other one is like simple ftp which allows you to move around but not able to read or write unless you have permissions.

To know it better I think a use case will be really helpful.
So I will put down the requirement that pushed me to learn about it.
We needed to grant permissions to one directory to a user, with one directory I literally mean that, as we wanted to block him from peeping into other things.
That too with minimal access to system binaries and it should be secure etc etc.

And SFTP was the best suit for the requirement, you will get the answer of how in the next section where I have shown the configurations of both the cases and believe me it couldn't have been simpler.

Lets get into the jail first ;-)

Edit /etc/ssh/sshd_config to include this.

Subsystem sftp internal-sftp
Match User sftpuser
ChrootDirectory /var/www/sftpdir (this makes the user stay under one directory)
ForceCommand internal-sftp

Create the sftpuser and set it's shell acess to false, so that user is not able to do ssh.

useradd -m -s /bin/false sftpuser

Give correct permissions to sftpdir.

chown root:root /var/www/sftpdir

For increasing security I changed these also in /etc/ssh/sshd_config:

PasswordAuthentication no
PubkeyAuthentication yes

And I also added my pub key to /home/sftpuser/.ssh/authorized_keys file, but this is optional as this doesn't make any effect on the SFTP working.

For this jail method we are done.

Try connecting:
sftp sftpuser@localhost
Connecting to localhost...
Enter passphrase for key '/home/user/.ssh/id_dsa':
sftp>

Now lets get back and see the next way (I know most of us will not read this, as first one will work like a charm):

Create a user with /usr/lib/openssh/sftp-server as shell and /var/www/sftpdir as home dir.

sudo useradd -s /usr/lib/openssh/sftp-server -d /var/www/sftpdir sftpuser

Add this /usr/lib/openssh/sftp-server to /etc/shells file

echo "/usr/lib/openssh/sftp-server" >> /etc/shells

For increasing security I changed /etc/ssh/sshd_config and also added my key to /var/www/sftpdir/.ssh/authorized_keys file.

PasswordAuthentication no
PubkeyAuthentication yes

Set correct permissions of the sftpdir.

chmod go-w /var/www/sftpdir
chmod 700 /var/www/sftpdir/.ssh
chmod 600 /var/www/sftpdir/.ssh/authorized_keys

And done.

Try connecting:
sftp sftpuser@localhost
Connecting to localhost...
Enter passphrase for key '/home/user/.ssh/id_dsa':
sftp>

Now about the extra benefits:

Easy to configure.
Good in Security.
Can work with PubkeyAuthentication.
No extra installation (as uses SSH).
Easy to use SFTP client.

at least these things are enough to pull me towards it.

That's it for taday. Happy SFTPing.

Sizing Openbravo: EC2 cost calculation and experimenting with around 270 concurrent users

2010-10-26T04:44:00.000-07:00

To help Openbravo partners and users we have extended our Sizing Tool Guidelines to include some more facts and findings.

"As more the better" --Harpreet Singh ;-)
Same has been proved by the new Amazon Cluster Compute Instance (cc1.4xlarge).

Amazon recently anounced the availability of it's biggest Instance, ideal for cluster infrastructure, as it promises high connectivity between cluster instances (as high as 10 Gigabit Ethernet).
But we tested this instance for the standalone test with Oracle DB and Openbravo on same instance.
The results were realy exciting as it was able to handle around 270 concurrent users.

Now truely speaking "Thats what I call results."
And the same results have been added to our Sizing Tool Results.

"There is no such thing as a free lunch." --Milton Friedman
This instance also has some drawbacks:
- It costs a lot (almost $1.60 per hour)
- Till now it is only available in US (N Virginia) region.
- And is only available with CentOS.

Now here comes another one.
As cost is the biggest concern when we think of any new infrastructure. For example: Running an instance (which can support 10 concurrent users) for 3 (THREE!!!) years on Amazon EC2 would cost only $1217.60, I think these figures can help one think about on-site and in-cloud (EC2) deployments.
So we extended our Sizing Guidelines to help you choose your Amazon Instance.

In the last section of the Guidelines we have added:
- Steps you can follow to calculate your yearly cost with Amazon cost calculator.
- As Amazon calculator is a bit complex so we created simple calculator to help you out.
- And pre calculated cost for most common scenarios.

Openbravo Ubuntu Maverick (be the first to use it).

2010-10-18T04:48:00.000-07:00

We were planning to blog and call for users to come up for early testing of your own trusted Openbravo with the upcoming Ubuntu's 10.10. But as Ubuntu Maverick (10.10) is already released, I think now all of us have a chance to be the first to use Openbravo with Ubuntu Maverick.

To be the first all you need to do is get Ubuntu Maverick Meerkat up and running any where you like may it be your hardware system, Amazon EC2 or a virtual machine.
This link will help you if you are planning to install on hardware system or virtual machine and these AMIs to boot one in EC2

Once you are set then lets rock and roll. I mean start installation.

So all you have to do to install Openbravo is:
- Enable the Partner’s Repository:
* sudo add-apt-repository "deb http://archive.canonical.com/ubuntu maverick partner"

- Install the openbravo-erp package:
* sudo apt-get update
* sudo apt-get install openbravo-erp

You can also install it using Synaptic or the Ubuntu Software Center:

Can installing of any comprehensive ERP be simpler than this?

You can do it even on a Friday. ;-)
As installing most ERPs on a Friday means forgetting about your Friday and Saturday night fun. With Openbravo on Maverick Meerkat, you can start the process at 7 and be at the party by 9!

So once you are done with the party sorry I mean installation you are set to use it and be a proud user of Openbravo ERP.
As it's you love and support that we have been able to live up to your expectations.

Also the users/developers who want to upgrade from 10.04 to 10.10, can do that without a fear of breaking the installation, the only concern should be that 10.10 is not a LTS version :-(

For more on installing Openbravo in Ubuntu please follow this wiki.

PostgreSQL: Performance Tuning

2010-07-16T06:32:00.000-07:00

"Need is the mother of discovery" -Harpreet Singh
I wrote this line just few minutes before writing this blog, as my need of optimizing PostgreSQL's performance lead me to search/discover for some cool facts and features of postgres and tools related to it.

For any postgres user thinking about 100 or more concurrent users is like a nightmare. I will admit that some time back I was also a bit scared on thinking about 100 concurrent users with postgres, but with the end of my search I am happy that I found a usable way to achieve that.

"Knowledge increases by sharing"

So I thought I will pass it to everyone who is searching for it on the internet.

The need that triggered me to search for this was to recommend Hardware as well as Software configuration to support 100-200 concurrent users on Openbravo ERP and postgres/Oracle as the database.

For me as I am a postgres supporter I believed that postgres will be able to handle it. And yippee I was right.

Coming back to the main point:
Postgres doesn't support too many users (concurrent) by default, it comes with very solid configuration aimed at everyone's best guess as to how an "average" database on "average" hardware should be.
Postgres has some default configuration options to fine tune it, like:
- max_connections
- shared_buffers
- effective_cache_size
- etc etc.

But these are not enough for postgres to support 100+ (concurrent) users.

In a reply of my query to postgres performance mailing list, I came to know about connection pooling.
One and the only con that I saw in this is that it is external, I mean we have to configure an external tool to do connection pooling.

There are tools like pgpool to make the job easy for us (pgpool is a middleware that works between PostgreSQL servers and a PostgreSQL database client).

Connection pooling tools provide us features like:
- Connection Pooling: It reduces connection overhead, and improves system's overall throughput.
- Replication: Using the replication function enables creating a real-time backup on 2 or more physical disks.
- Load Balance: As the name suggests it distributes the queries on two or more replicated servers.
- Limiting Exceeding Connections: With the use of this extra connections are queued instead of returning an error immediately.
- Parallel Query: Using the parallel query function, data can be divided among the multiple (replicated) servers.

Configuring these properly can fine tune postgres' performance to handle 100-200 concurrent users.

Happy *postgresing*

To read more about performance tuning in postgreSQL read this.
For more on pgpool click here.

Module Integration with CI (Hudson)

2010-07-01T22:14:00.000-07:00

Module Integration with CI (Hudson)

Long back we (RM @ Openbravo) introduced CI (Continuous Integration) tool (Hudson) for testing code of our core ERP development branch.
Which allowed our developers to do:
- Daily Builds (Full/Incremental).
- Smoke Tests.
- DB Consistency Tests.
- etc.

But as most of the developers were becoming modular (Openbravo became modular with Openbravo ERP Version 2.50), CI was not able to maintain the pace and provide similar help for the module testing.

To set things in place we enabled our developers to integrate and test their modules using new CI even after committing even a single changeset to their module repository.

According to me the goal or I will say the purpose of this whole effort was to enable a developer to have a nice and sound sleep after he pushes his commit to the module repository.
Sounds confusing?

Let me explain it.
Earlier developers use to develop a module and used to do time consuming small manual testing to make sure that their code is bug free.
From a developers perspective he cannot sleep properly until his module is tested and deployed properly.

To enable CI for modules and save developers time (from manual testing) we created the setup which will help developers to directly configure a new job in Hudson to test their changesets. This new setup empowers them to do:
- Sanity Check
- Source compilation check and Create OBX
- Database consistency test
- Module's JUnit test
- Installation of the generated OBX
- Un-Installation of the installed module
- Selenium test (module smoke)
- Upgrade the module from previous published version in CR (Central Repository) to generated OBX
Even for a single new changeset in the module's repository.

And still it has endless possibilities, where we can integrate new test cases to this.

We have also created a template job (which is pre-configured with all these test cases) to help developers configure and run tests for their modules easily.
Developers will just have to copy the template job to a new job, change the variables to their modules related variables and then run the job. We have also created a simplified wiki for step by step instructions.

* Currenty Openbravo developers working on any Modules can take benefits of this tool (but sky is the limit, maybe someday we can allow partners/community to take advantage of this tool).

RM Updates: Amazon backup stratergy, Mantis Upgrade, Establish automatic process for releasing 2.40, OB@OB

2010-01-07T04:12:00.000-08:00

These are the latest news from the Openbravo's Release Management Team:

Backup Strategy: EBS boot.

Amazon has a new feature ebs boot, this helps us to keep our root partition in ebs volume and also allows us to have data up to 1TB in root partition. This helps us in the following way : better backup strategy and from now we can pause & re-start an instance and thus saving cost. My colleague gnuyoga has a blog about the same.

Mantis Upgrade: Upgrade issues.openbravo.com to mantis-1.2.0

As you know our existing issue tracker is based on mantis 1.1.8. With the release of mantis 1.2.0, it promises lot of interesting productivity boosters. We are migrating our current mantis to latest. This involves quiet bit of challenge. In this sprint we address customization like SSO (Single Sign On, etc), and custom css. If you want to be a beta tester to testing our new mantis please drop us an email for us to give you a test account.

Continuous release of 2.40 branch

So the mantra of 2.40 branch is continuous release as detailed in my colleague juan pablo's blog post. Now this task is complete and for details see here

OB@OB: Documentation and Linux tool

This task was about documenting the process of replicating production environment to testing environment and creating a new tool that automates this process in linux.

HTTP Basic Auth

2009-12-01T01:41:00.000-08:00

As while working with apache many of us feel like having a secure way to access our data.
Simple but effective http basic auth is probably the quickest and the easiest answer.

Setting it up requires only two things:
- htpasswd file (containing valid user name and password)
- And apache configuration file to read it.

Creating a htpasswd file:
- htpasswd -cm </Path/tp/htpasswd-file> <username>
- While adding more users just remove c from the above command.

Configuring apache:
- Add this to default (vhost file) configuration file
     <location>
           Allow from all
           AuthType Basic
           AuthName "Restricted Area"
           AuthUserFile </Path/to/htpasswd-file>
           Require valid-user
     </location>

- Now reload apache and enjoy.

RM updates: Automation and upgrade of Mantis @ Openbravo

2009-11-17T22:56:00.000-08:00

We are almost close to achieve milestone 2 of Continuous Integration. The team is working really hard on finding a solution for existing challenges as well as proposing ways to automate current repetitive tasks.

Last sprint we have completed one of the most challenging tasks "automated code migration from pi - main". Now we have obx generated from main branch if all the tests are successful. Plans to generate an obx on every commit are heavily debated within the team.

Now we have tecnicia14 resurrected. This will help our developers as well as our QA team to see the code changes in the live environment (live and liveqa).

Apart from the CI infrastructure, we have also upgraded the Issue Tracker version to 1.1.8 which is the latest stable mantis version available. We are also in the process of ensuring we have a hard backup of all the important instances running in Amazon ec2.

For a complete list of the on-going stories that we are working on, please check the Sprint 28 page of our Scrum spreadsheet

UML (User Mode Linux)

2009-06-29T05:31:00.000-07:00

In this Blog I will take you to a tour to a new dimension of Virtualization. That is the world of UML.

Many of us have spent time using VMware, VBox, Qemu etc, or debugging on how to start Xen/OpenVZ.
But the truth of above two is, applications like VMware are heavy on system resources and applications like Xen are a bit tricky (kernel should be Xen specific).

One thing I will admit, that I was also amongst the people who have spent a huge amount of time on different Virtualization technologies, until here at Openbravo I was given an opportunity to setup virtual environment on Amazon servers.
* For those who haven't worked with Amazon servers, Amazon servers are like domU of Xen virtual environment.

The aim was to setup an Openbravo instance (which can be made available on demand) on the top of an Amazon instance.
The main problem in working with amazon server is that you don't have access to host (domO) and neither Amazon people allow you to use custom kernel for domU (custom kernel: so that we can build another xen domU over it).
So taking these things in to account we were left with options like VMware but looking upon their load on system we needed something light and usable.
In our hunt for a perfect virtual environment my manager told me about UML. And truly speaking it turned up to be the perfect tool (it matched all our requirements and expectations).

UML works very differently from all other Virtualization techniques, all it needs to work smoothly is uml-utilities package, one kernel (binary script) and a block device(which contains minimal/full OS).
It is like doing chroot to a directory and installing a full OS in that.
It has many other features like

Mounting host filesystem.
Adding a COW (copy on write) file.
etc.

This COW file is used if you want to use same block device (filesystem) for more than one Virtual system. This COW file writes the difference to a separate file (just like a diff patch).

The use of UML gave us one more flexibility, that was using Xen image (pre-installed Openbravo instance) with UML kernel.
And an advantage over Xen (i.e. access over host system is not required in UML).

Reference:
UML Howto
UML Wiki
Download Page

Screenshots and DAlbum

2009-06-15T22:54:00.000-07:00

In one of my recent tasks I explored a couple of tools available to publish screen shots (Web Based).

The aim was to get the screen shots when Hudson (CI tool) is running a build and to publish them using some web based gallery creator, for later reference of the developer or QA team.

Well the task was not tough but was a bit tricky, as the screen shot took by Xvnc plugin of Hudson was not sufficient (it took the screen shot at the end of the build, mostly an empty screen), so we decided to get a command line tool to get screen shots. And in the hunt for a command line tool we got ImageMagic and discovered that Xvnc plugin was also using the same tool.
Now by running a FOR loop in the background (when required) gave us a directory full of relevant screen shots.

Now the job remaining was to publish them, for that lot of tools are available in opensource world, basically most of them are either python or php based, but we wanted some thing which required less/no installation (extra) and could run on Apache, so we choose DAlbum (php based). Selecting this tool was not enough as it required placing the images in to it's root directory and clicking or executing Reindex.php script. Here also we made some changes as doing it with the above mentioned script increased disk usage, it created 3 copies of each screen shot (1 for thumbnail, 2nd for full screen view and the 3rd for downloading).

We overcome this issue by creating our own script which did the same job but now the size of all the three images were defined by us, the script (bash) converted the screen shot in to three different sized imaged (using ImageMagic) and then placed them into the respective directory required by DAlbum.

Here is sample command we used to get the screen shots
import -window root -display $display $screenshotpath/screenshot$i.jpg

DAlbum looks like this

For more information on ImageMagic click here.
For more information on DAlbum click here.

Monitoring system with munin/monit

2009-06-03T23:40:00.000-07:00

Munin with Muninnode or Muninlite

Recently while deploying monitoring system on Amazon servers I came across a new tool Muninlite, I would rather prefer to call it a script.
And the hands on experience of this script forced me to think on to which one is better (muninnode / muninlite) and to write this blog so that it can be of help to others.
The goal was to monitor several Amazon servers from one centralized Master and do the resource planning based on the usage graphs.
So we decided to do the job using munin (as front end) and muninnode as client nodes to collect data from different servers.
While exploring on how to deploy and take the best use of the setup we came across muninlite, it's a bash script which works just similar to muninnode, but as name suggests it's light on system resources and has has less response time.

To go through the installation steps for munin and muninnode click here.
And if looking for a better and lighter way to replace muninnode click here.

The resultant graph produced by munin looks like this

Monit and M/Monit

While accomplishing the job mentioned above we also deployed monit, it has proven itself in the terms of alerts (i.e sending mails etc. as alerts at specified times).
Despite of monitoring system as a whole (CPU load, memory usage etc.) monit is also capable of monitoring services eg. apache, ssh, mysql etc.
It also has an extended hand as M/Monit which gives a cool dashboard to manage and monitor different monit instances, but the worst part is yet to come this (M/Monit) is not free.

To go through the installation steps of monit click here.
And if you need a demo version of M/Monit click here.

In a web browser monit looks like this

And M/Monit something like this

I think one thing I left out to share is as we were working on the domU of Amazon servers the only thing that would have worked for us was the thing which can read the desired data from proc files.
As during the process of finalizing munin and monit as our tools we stopped on cacti also for sometime as it had more features and to collect data it used SNMP (Simple Network Management Tool), which gives it a great flexibility to work on almost every linux environment, but it failed in our case.

Even it was not useful for me at that instance but I can share a veiw of cacti

Compiling Linux Kernel 2.6.27

2008-11-23T00:09:00.000-08:00

How to: Compile Linux kernel 2.6.27

Compiling custom kernel has its own advantages and disadvantages. However, new Linux user / admin find it difficult to compile Linux kernel. Compiling kernel needs to understand few things and then just type couple of commands. This step by step howto covers compiling Linux kernel version 2.6.xx under Debian GNU Linux. However, instructions remains the same for any other distribution except for apt-get command.

Step # 1 Get Latest Linux kernel code

Visit http://kernel.org/ and download the latest source code. File name would be linux-x.y.z.tar.bz, where x.y.z is actual version number. For example file
linux-2.6.27.tar.bz represents 2.6.27 kernel version. Use wget command to download kernel source code:
$ cd /tmp
$ wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-x.y.z.tar.bz

Note: Replace x.y.z with actual version number.

Step # 2 Extract tar (.tar.bz) file

Type the following command:
# tar -xzvf linux-2.6.27.tar.bz -C /usr/src
# cd /usr/src

Step # 3 Configure kernel

Before you configure kernel make sure you have development tools (gcc compilers and related tools) are installed on your system. If gcc compiler and tools are not installed then use apt-get command under Debian Linux to install development tools.
# apt-get install gcc

Now you can start kernel configuration by typing any one of the command:

* $ make menuconfig - Text based color menus, radiolists & dialogs. This option also useful on remote server if you wanna compile kernel remotely.
* $ make xconfig - X windows (Qt) based configuration tool, works best under KDE desktop
* $ make gconfig - X windows (Gtk) based configuration tool, works best under Gnome Dekstop.

For example make menuconfig command launches following screen:
$ make menuconfig

You have to select different options as per your need. Each configuration option has HELP button associated with it so select help button to get help.

Step # 4 Compile kernel

Start compiling to create a compressed kernel image, enter:
$ make
Start compiling to kernel modules:
$ make modules

Step # 5 Install kernel modules

Install kernel modules (become a root user, use su command):
$ su -
# make modules_install
It will install all the module under /lib/modules/ directory.

Step # 6 Install kernel

So far we have compiled kernel and installed kernel modules. It is time to install kernel itself.
# make install

It will install three files into /boot directory as well as modification to your kernel grub configuration file:

* System.map-2.6.27
* config-2.6.27
* vmlinuz-2.6.27

But if don't want to install the kernel on the same machine as on which it i being compiled then.
# make bzImage
Where bzImage is a files created after step 4 in the current working directory.

It will create the three files into current directory.

Step # 7: Create an initrd image

Type the following command at a shell prompt:
# cd /boot
# mkinitramfs -o initrd.img-2.6.27 2.6.27

If using fedora/redhat type the following command at a shell prompt:
# mkinitrd -o initrd.img-2.6.27 2.6.27

initrd images contains device driver which needed to load rest of the operating system later on. Not all computer requires initrd, but it is safe to create one.

Step # 8 Modify Grub configuration file - /boot/grub/menu.lst

Open file using vi:
# vi /boot/grub/menu.lst

title Debian GNU/Linux, kernel 2.6.27 Default
root (hd0,0)
kernel /boot/vmlinuz-2.6.27 root=/dev/hdb1 ro
initrd /boot/initrd.img-2.6.27
boot

Remember to setup correct root=/dev/hdXX device. Save and close the file. If you think editing and writing all lines by hand is too much for you, try out update-grub command to update the lines for each kernel in /boot/grub/menu.lst file. Just type the command:
# update-grub

Neat. Huh?

Step # 9 : Reboot computer and boot into your new kernel

Just issue reboot command:
# reboot

Networking Basics

2008-11-20T03:44:00.000-08:00

Networking Basics

Basic Networking Funda
The traffic/data leaves one machine, goes towards the Gateway, which is then redirected towards the destination by the Gateway.
* In this process first routing table in the source machine should point towards the correct Gateway.
* The Gateway should have correct route, IPtables and natting rules pointing towards the destination.
* The destination machine should have correct IPtables to accept the data comming from that source.

In general Networking depends on three things.

* Routing
* Nating
* IPtables/Firewalls

Routing

It tells tell the system about where to send a packet for any destination.
Routing is the process of selecting paths in a network along which to send network traffic. Routing is performed for many kinds of networks, including the telephone network, electronic data networks (such as the Internet), and transportation (transport) networks. This article is concerned primarily with routing in electronic data networks using packet switching technology.

In packet switching networks, routing directs forwarding, the transit of logically addressed packets from their source toward their ultimate destination through intermediate nodes; typically hardware devices called routers, bridges, gateways, firewalls, or switches. Ordinary computers with multiple network cards can also forward packets and perform routing, though they are not specialized hardware and may suffer from limited performance. The routing process usually directs forwarding on the basis of routing tables which maintain a record of the routes to various network destinations. Thus constructing routing tables, which are held in the routers' memory, becomes very important for efficient routing. Most routing algorithms use only one network path at a time, but multipath routing techniques enable the use of multiple alternative paths.

Routing, in a more narrow sense of the term, is often contrasted with bridging in its assumption that network addresses are structured and that similar addresses imply proximity within the network. Because structured addresses allow a single routing table entry to represent the route to a group of devices, structured addressing (routing, in the narrow sense) outperforms unstructured addressing (bridging) in large networks, and has become the dominant form of addressing on the Internet, though bridging is still widely used within localized environments.

To explore it more you should stop reading and take a look at the routing table in you system
* route -n
This command will give you a table ex.

Kernel IP routing table.

Destination	Gateway	Genmask	Flags	Metric	Ref	Use	Iface
192.168.20.0	192.168.20.1	255.255.255.0	UG	0	0	0	tap0
192.168.1.0	0.0.0.0	255.255.255.0	U	0	0	0	eth0
0.0.0.0	192.168.1.5	0.0.0.0	UG	0	0	0	eth0

To add a route to this table type
* route add -net 192.168.1.0/24 gw 192.168.1.1

You can replace 192.168.1.0/24 with network ip to which you want to send data and 192.168.1.1 with the gateway through which that network is connected.

To delete a route to this table type
* route del -net 192.168.1.0/24 gw 192.168.1.1

Natting

In computer networking, network address translation (NAT) is the process of modifying network address information in datagram packet headers while in transit across a traffic routing device for the purpose of remapping a given address space into another.

Most often today, NAT is used in conjunction with network masquerading (or IP masquerading) which is a technique that hides an entire address space, usually consisting of private network addresses , behind a single IP address in another, often public address space. This mechanism is implemented in a routing device that uses stateful translation tables to map the "hidden" addresses into a single address and then rewrites the outgoing Internet Protocol (IP) packets on exit so that they appear to originate from the router. In the reverse communications path, responses are mapped back to the originating IP address using the rules ("state") stored in the translation tables. The translation table rules established in this fashion are flushed after a short period without new traffic refreshing their state.

As described, the method only allows transit traffic through the router when it is originating in the masqueraded network, since this establishes the translation tables. However, most NAT devices today allow the network administrator to configure translation tables entries for permanent use. This feature is often referred to as "static NAT" or port forwarding and allows traffic originating in the 'outside' network to reach designated hosts in the masqueraded network.

Because of the popularity of this technique, see below, the term NAT has become virtually synonymous with the method of IP masquerading.

Network address translation has serious consequences (see below, Drawbacks & Benefits) on the quality of Internet connectivity and requires careful attention to the details of its implementation. As a result, many methods have been devised to alleviate the issues encountered. See article on NAT traversal.

Nating can be taken care off in 2 ways one is through the router and the other way is if you are dealing with Linux machines it can be done using IPtables.

To check nating tables on a Linux machine type
* iptables -nvL -t nat
This command will give you a table ex.

Chain PREROUTING (policy ACCEPT 3791 packets, 359K bytes)

pkts

bytes

target

prot

opt

out

source

destination

Chain POSTROUTING (policy ACCEPT 6573 packets, 406K bytes)

pkts

bytes

target

prot

opt

out

source

destination

Chain OUTPUT (policy ACCEPT 6405 packets, 394K bytes)

pkts

bytes

target

prot

opt

out

source

destination

NAT or DNAT

Address translation occurs before routing. Facilitates the transformation of the destination IP address to be compatible with the firewall's routing table. Used with NAT of the destination IP address, also known as destination NAT or DNAT

SNAT

Address translation occurs after routing. This implies that there was no need to modify the destination IP address of the packet as in pre-routing. Used with NAT of the source IP address using either one-to-one or many-to-one NAT. This is known as source NAT, or SNAT

PACKET FLOW UNDER NAT

* The packet is first examined by rules in the PREROUTING chain, if any. It is then inspected by the rules in the nat table's PREROUTING chain to see whether the packet requires DNAT. It is then routed.

*If the packet is destined for a protected network, then it is filtered by the rules in the FORWARD chain of the filter table and, if necessary, the packet undergoes SNAT in the POSTROUTING chain before arriving to the network.

Masquerading (Many to One NAT)

Masquerading is another name for what many call many to one NAT. Traffic from all devices on one or more protected networks will appear as if it originated from a single IP address on the Internet side of the firewall.

Note: The masquerade IP address always defaults to the IP address of the firewall's main interface. The advantage of this is that you never have to specify the NAT IP address.

You can configure many to one NAT to an IP alias, using the POSTROUTING and not the MASQUERADE statement. An example of this can be seen in the static NAT section that follows.

Port Forwarding Type NAT

In many cases home users may get a single DHCP public IP address from their ISPs. If a Linux firewall is also your interface to the Internet and you want to host a Web site on one of the NAT protected home servers, then you will have to use port forwarding

Port forwarding is handled by the PREROUTING chain of the nat table

examples:

* iptables -t nat -A PREROUTING -p tcp -i eth0 -d $external_ip --dport 80 -j DNAT --to 192.168.1.200:8080

This rule allows the port forwarding for traffic destined to port 80 of the firewall's IP address to be forwarded to port 8080 on server 192.168.1.200

SNAT
----

SNAT is used to NAT all other outbound connections initiated from the protected network to appear to come from single IP address.

POSTROUTING 1 to 1
------------------

* iptables -t nat -A POSTROUTING -s 192.168.1.100 -o eth0 -j SNAT --to-source 97.158.253.26

Many to 1
---------

* iptables -t nat -A POSTROUTING -s 192.168.1.0/24 -j SNAT -o eth0 --to-source 97.158.253.29

* DNAT ----- Used to do destination network address translation. ie. rewriting the destination IP address of the packet

* SNAT ----- Used to do source network address translation rewriting the source IP address of the packet The source IP address is user defined

* MASQUERADE Used to do Source Network Address Translation.By default the source IP address is the same as that used by the firewall's interface

NOTE:

ipforwarding should be enabled..then only your rules will work

cat /proc/sys/net/ipv4/ip_forward

This should give 1.

IPtables/Firewalls

iptables is a user space application program that allows a system administrator to configure the tables provided by Xtables (which in turn uses Netfilter) and the chains and rules it stores. Because iptables requires elevated privileges to operate, it must be executed by user root, otherwise it fails to function. On most Linux systems, iptables is installed as /usr/sbin/iptables and documented in its man page [1], which can be opened using "man iptables" when installed. iptables is also commonly used to inclusively refer to the kernel-level component Xtables that does the actual table traversal and provides an API for kernel-level extensions.

iptables works with Linux kernels 2.4 and 2.6. Older Linux kernels use ipchains (Linux 2.2) and ipfwadm (Linux 2.0).

The source of the packet determines which chain it traverses initially. There are three predefined chains (INPUT, OUTPUT, and FORWARD) in the "filter" table. Predefined chains have a policy, for example DROP, which is applied to the packet if it reaches the end of the chain. The system administrator can create as many other chains as desired. These chains have no policy; if a packet reaches the end of the chain it is returned to the chain which called it. A chain may be empty.

Each rule in a chain contains the specification of which packets it matches. It may also contain a target. As a packet traverses a chain, each rule in turn examines it. If a rule does not match the packet, the packet is passed to the next rule. If a rule does match the packet, the rule takes the action indicated by the target, which may result in the packet being allowed to continue along the chain or it may not.

The packet continues to traverse the chain until either (1) a rule matches the packet and decides the ultimate fate of the packet (for example by calling one of the ACCEPT or DROP targets); or (2) a rule calls the RETURN target, in which case processing returns to the calling chain; or (3) the end of the chain is reached.

This example shows an already-configured workstation firewall. The command "iptables -L" is executed by user root to display the firewall configuration.

* iptables -nvL

Chain INPUT (policy DROP)
target prot opt source destination
ACCEPT all -- localhost.localdomain localhost.localdomain
ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED
REJECT all -- anywhere anywhere

Chain FORWARD (policy DROP)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Before proceeding further one should be aware of the three predefined chains of IPtables.
* INPUT
* OUTPUT
* FORWARD

INPUT
As the name explains this chain comes into picture when any packet is destined towards the system.

OUTPUT
This chain has to be configured when thinking of sending packet to other machines in the network.Usually we don't configure this chain.As it's default policy is ACCEPT.

FORWARD
This chain comes into picture when any packet in the network travels from 1 system 2 the other via the gateway, then FORWARD chain on the gateway has to be configured in such a way that it should accept the packets for forwarding.

Adding rules to IPTABLES
* iptables -I INPUT -s 192.168.20.0/24 -j ACCEPT
* iptables -I FORWARD -s 192.168.20.0/24 -j ACCEPT
* iptables -A FORWARD -s 192.168.20.0/24 --dport 80 -j ACCEPT
* iptables -I FORWARD -d 192.168.20.0/24 -j DROP

(Where I/A are to specify the priority of the ip rule, I is the inserting to the highest priority and A is appending it to the last)

Deleting rules from IPTABLES.
* iptables -D INPUT -s 192.168.20.0/24 -j ACCEPT
* iptables -D FORWARD -s 192.168.20.0/24 -j ACCEPT
* iptables -D FORWARD -s 192.168.20.0/24 --dport 80 -j ACCEPT
* iptables -D FORWARD -d 192.168.20.0/24 -j DROP

(Remember the rule should be in the same format as it was added only change is the -D)

To check if you are connected to your network properly commands like ping. telnet and traceroute will help.
See man pages of these commands for more help.
Use tcpdump to view the flow of packets FOR BETTER RESULTS.

Hope this helps.

Network Bridging

2008-11-04T05:28:00.000-08:00

TAP (Network Bridging)

In computer networking, TUN and TAP are virtual network kernel drivers. They implement network devices that are supported entirely in software, which is different from ordinary network devices that are backed up by hardware network adapters.

TAP (as in network tap) simulates an Ethernet device and it operates with layer 2 packets such as Ethernet frames. TUN (as in network TUNnel) simulates a network layer device and it operates with layer 3 packets such as IP packets. TAP is used to create a network bridge, while TUN is used with routing.

Packets sent by an operating syst/sbin/modprobe tunem via a TUN/TAP device are delivered to a user-space program that attaches itself to the device. A user-space program may also pass packets into a TUN/TAP device. In this case TUN/TAP device delivers (or "injects") these packets to the operating system network stack thus emulating their reception from an external source.

TUN is mostly used for OpenVPN, VTun.
Where as TAP is used for bridging Virtual Machines to the host machine.

We have one more concept that is BRIDGE an this is used if you want to put a number of Virtual Machines into a 1 single subnet.
Else with tap devices you can not do that.

Configuring Network Bridges.

TAP

To configure tap you need to install uml-utilities, to do this ->

* apt-get install uml-utilities

This will give you tunctl command, which is further used to build tap interface.
To build tap interface run ->

* tunctl

This will give you (Set 'tap0' persistent and owned by uid 0).

You can see this interface with <ifconfig -a> command.
Now when the interface is ready give it an ip, make it up and link it to your Virtual Machine.

* ifconfig tap0 192.168.1.1

The ip which you assign to tap0 should be of different subnet as your host network.

Just remember doing this will not solve your full problem of networking in Virtual Machines, to make it fully up and running it requires routing, iptables and nating(NAT) to be configured properly.
You can expect these things to be covered in some of my later posts.

Bridge

Bridge comes into picture when you have more than 1 Virtual Machine.
It helps to get all the Virtual Machines in 1 single network which is not possible with TAP alone.
With this one can also get the Virtual Machines into the same network as the host.

To configure Bridge you need to install bridge-utils, to do this ->

* apt-get install bridge-utils

This will give you brctl command, which is further used to build tap interface.
Running brctl alone will give a list of option which can be used with this command.
But for now run ->

* brctl addbr br0

This will add a bridging interface named br0.Now add an ip to it.

* ifconfig br0 192.168.1.1

Here comes a time to select the scenario that either you need a different network or the same net work for VM(Virtual Machines).

If you need the network to be same then.

* brctl addif br0 eth0
* brctl addif br0 tap0
* ifconfig eth0 0.0.0.0
* ifconfig tap0 0.0.0.0

This will add eth0 and tap0 interfaces to bridge br0.

And if their is no need of the same network then.

* brctl addif br0 tap0

This will add tap0 interface to bridge br0.
In this case to make network live and working configure routing, iptables and nating(NAT) on the host machine.

That it.
Hope it helps.

Vitualization

2008-11-04T03:27:00.000-08:00

Virtualization

To start with Virtualization I can say it is the one of the most helpful technology known to man. The basics which make it a great technology is the feature that user can have more than one machine at the cost of one single machine.
Or for a software guy he can get a number of machines to test his development.

In Virtualization we have 2 different types.
They are ->
* Full Virtualization.
* Para Virtualization.

Full Virtualization, in computer science, is a Virtualization technique used to implement a certain kind of virtual machine environment: one that provides a complete simulation of the underlying hardware. The result is a system in which all software capable of execution on the raw hardware can be run in the virtual machine. In particular, this includes all operating systems. (This is different from other forms of Virtualization – which allow only certain or modified software to run within a virtual machine.)
Some examples of such are VMware/QEMU/KVM.

Para Virtualization, In computing, paravirtualization is a Virtualization technique that presents a software interface to virtual machines that is similar but not identical to that of the underlying hardware.

Paravirtualization may allow the virtual machine monitor (VMM) to be simpler or virtual machines that run on it to achieve performance closer to non-virtualized hardware. However, operating systems must be explicitly ported to run on top of a paravirtualized VMM.

And according to me I personally prefer this due to its feature of not using too much of RAM and other resources, as it uses host's resources and that too when required.
Some examples of such are XEN/OpenVZ.

How to?
In terms of how to do virtualization it can be done through GUI or Command line. I prefer Command line for doing the job.

Full Virtualization.

I will start explaining this hoping that you have QEMU installed.

QEMU

qemu-img create -f raw image.img 5G

This will create a blank image named image.img you can define the size of the image by changing the 5G option to any size.
Now it's time to boot it with a cdrom to give it an Operating System. To do this you need either a CDROM on host machine o an ISO image.
If you have an ISO image then enter this command ->

qemu -cdrom linux/windows.iso -hda image.img -m 600 -boot d

Where linux/windows.iso is your ISO image, image.img is the disk image you just created,-m 600 is the RAM you want to allocate to the Virtual Machine and -boot d is to tell the Virtual Machine to boot from the CDROM.

After the installation is over then run the Virtual Machine with the following command ->

qemu -hda image.img -m 600 -net tap -net nic

Where -net tap tell it to connect to a virtual interface(tap device -- tap0/tap1 etc) on the host and -net nic defines 1 network card present on the Virtual Machine.
And if you don't know how to configure tap device you can refer my next post which will be TAP/TUN.

Para Virtualization.

XEN

Under this I will go on with XEN, to have virtualization with xen you need to have the kernel which supports XEN virtualization.
I now suppose that you have upgraded your kernel to support XEN.

To start with XEN you must be familiar with two terms which we will be using a number of times.
* dom0 -> this refers to the host on which the Virtual Machine or the domU is situated.
* domU -> this refers to the guest machine which we are going to create now.

Now if you have upgraded your kernel of the host machine to support XEN, this means yours dom0 is ready.
And it's time to create domU but before that just reboot your machine to boot into the upgraded kernel.

Creating - domU

File Based Setup of Virtual Disk

* mkdir -p /home/haps/
* dd if=/dev/zero of=/home/haps/diskimage.img bs=1024k count=5000
* dd if=/dev/zero of=/home/haps/swapimage.img bs=1024k count=512

* mkfs.ext3 /home/haps/diskimage.img
* mkswap /home/haps/swapimage.img

* mount -o loop /home/haps/diskimage.img /mnt

Debootstrap New OS onto Virtual Disk

* debootstrap --arch i386 lenny /mnt http://ftp.de.debian.org/debian/

* mv /mnt/lib/tls /mnt/lib/tls.disabled

* cp /etc/apt/sources.list /mnt/etc/apt/
* vi /mnt/etc/apt/sources.list

* cp -a /lib/modules/2.6.16-1-xen-k7/ /mnt/lib/modules/

* cp /etc/resolve.conf /mnt/etc/

* cp /etc/network/interfaces /mnt/etc/network/
* vi /mnt/etc/network/interfaces (and make it look like this)

#To use Specific IP address - edit the /mnt/etc/network/interfaces manually.
#To use DHCP, edit and include the following:
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto eth0
iface eth0 inet dhcp

* vi /mnt/etc/hostname

* vi /mnt/etc/fstab (and make it look like this)

proc /proc proc defaults 0 0
/dev/sda1 / ext3 defaults 0 0
/dev/sda2 none swap sw 0 0

Setup domU Xen Config

* vi /etc/xen/haps (and make it look like this)
kernel = "/boot/vmlinuz-2.6.16-1-xen-k7"
ramdisk = "/boot/initrd.img-2.6.16-1-xen-k7"
memory = 192
name = "haps"
vif = ['bridge=xenbr0']
ip = "ip 192.168.1.5"
gateway = "ip add"
netmask = "255.255.0.0"
root = "/dev/sda1 rw"

* ln -s /etc/xen/haps /etc/xen/auto/haps
//link in the config file so that the Virtal Machine starts on Bootup of Dom0

Run and Update DomU

* xm create haps -c
// CTRL + ] Gets out of the Console

All going well you should see domU booting up. Login as "root" with no password. Execute the following commands to update:

* passwd

* apt-get update
* apt-get upgrade

* apt-get install module-init-tools //for iptables
* apt-get install iptables

* iptables -L

For other commands on xen do .

Thats it.
Hope this helps a lot of people.

BackTrack3 How To

2008-10-15T10:42:00.000-07:00

1. Boot the Live CD

2. Open Terminal

3. Partition the disks
a. fdisk /dev/hda # The name of the hard disk depends on your configuration
b. Press n for new partition p for Primary partition
Partition number (1-4): 1 (for first partition)
c. Starting cylinder number and Size specification (+4000M for 4Gb)
d. Create one more partition for SWAP area with Partition ID as 82

4. mkfs.ext3 /dev/sda1

5. mkswap /dev/sda2

6. swapon /dev/sda2

7. mkdir /mnt/backtrack

8. mount /dev/sda1 /mnt/backtrack/

9. mkdir /mnt/backtrack/boot

10. cp --preserve -R /{bin,dev,home,pentest,root,usr,etc,lib,opt,sbin,var} /mnt/backtrack/

11. mkdir /mnt/backtrack/{mnt,proc,sys,tmp}
12. mount --bind /dev/ /mnt/backtrack/dev/
13. mount -t proc proc /mnt/backtrack/proc/
14. cp /boot/vmlinuz /mnt/backtrack/boot/
15. Edit /mnt/backtrack/etc/lilo.conf file. The
lba32
boot = /dev/hda
prompt
timeout = 60
change-rules
reset
vga = 773
image = /boot/vmlinuz
root = /dev/hda1
label = Backtrack_Final
read-only

16. chroot /mnt/backtrack/ /bin/bash

17. lilo -v
And if you have another OS running with grub then you have to install either grub to BackTrack or add lilo to that OS.

18. exit

19. reboot

10. Remove the CD

20. Enjoy H@CK!NG

LTSP(Linux Terminal Server Project)

2008-09-17T02:25:00.002-07:00

LTSP

To define LTSP I can only say, it's an efficient and most usable technology which one could have seen till now.
As the name denotes it is a Linux terminal or to be more precise i can say that it is a client terminal and a light weight too, it means that it neither uses much of your workstations memory nor the LAN's bandwidth.

Working

When a client boots from a local boot device (like a hard disk, CD-ROM or USB disk), it loads a small Linux kernel from that device which initializes the system and all of the peripherals that it recognizes. When configured for network booting the client first requests its own IP address and the IP address for the LTSP server via DHCP and loads the Linux kernel from a preconfigured Linux image on the LTSP server via the Trivial File Transfer Protocol (TFTP) service running on the LTSP server.

During this process the client makes a (new) DHCP request for the IP address of the LTSP server and the path to its chroot environment. When this information is retrieved, the client mounts the path on its root file system via the Network File System (NFS) service running on the LTSP server.

The client loads Linux from the NFS mounted root file system and finally starts the X windowing system. The client connects to the XDMCP login manager on the LTSP server. From this point forward, all programs are started on the LTSP server, but displayed and operated from the client.

Prerequisites

Server: A system with enough RAM and hard disk to support as many systems as you want to run.
Workstations: A system with no hard disk, only RAM and other basic parts are required.
Connection: Cable and switch

The basic requirements to run LTSP on your system you need :-

DHCP - This is required to allocate IP's to the client machines.
TFTP - This is required to copy kernel image from host to the client, which helps the client to boot up.
NFS - This is required to mount a file system on the client which has no memory of its own.
GDM - This is required to give Thin Clients Display Manager.

Isn't this sounds great running workstations with no hard-disk, no OS.
I think it is an ideal setup for schools, colleges and other places where you don't have much to store and with this you can cut down the cost also.
And if some one is worried about the crashing of the server then he/she can go for RAID disks, which can save you from loosing your data at least.

Installation

As far as installation is considered I am using Debian machine,

If you have a DHCP server already up and running on your network:
apt-get install ltsp-server openssh-server
Otherwise, if you want your LTSP server to function as the DHCP server:
apt-get install ltsp-server-standalone openssh-server
Build the LTSP client environment:
ltsp-build-client

If you change the IP data after you have done the initial setup and run ltsp-update-sshkeys on the server.

Configure /etc/dhcp3/dhcpd.conf:
See examples in /usr/share/doc/ltsp-server/examples/dhcpd.conf or /etc/ltsp/dhcpd.conf and adapt to your network.

Add next server to dhcpd.conf and Restart dhcpd:
/etc/init.d/dhcpd3-server restart.

And if you are running your DNS server on the same machine then you may have to configure /etc/dnsmasq.conf:
See example in
/usr/share/doc/ltsp-server/examples/dhcpd-dnsmasq
and adapt to your network.
If you configured your DNS then Restart dnsmasq:
/etc/init.d/dnsmasq restart
Configure /etc/exports:
/opt/ltsp/i386 *(ro,no_root_squash,async,no_subtree_check)

This tells the NFS server to export this particular directory.

Restart nfs:
/etc/init.d/nfs-kernel-server restart

And if required you can export the file system through exportfs (man exportfs for further details).

Start tftpd:
/etc/init.d/tftpd-hpa start
But a change has to be made in edit /etc/default/tftpd-hpa:
RUN_DAEMON="yes"
Restart tftpd:
/etc/init.d/tftpd-hpa restart
You can also check if the tftp server is running:
type tftp and press enter you will get tftp prompt(tftp>)

And most important thing i.e. to start a GDM session .

Troubleshooting

DHCP:

No DHCP or poxyDHCP offers were received.

This error can be due to many reasons, few of which I was able to cover were:
* Network not connected.
* DHCP server is down.
* Faulty dhcpd.conf file.
To solve this kind of problem start step-by-step ensuring that it won't happen again:
* Check the network cable (if it is connected properly or not).
* Check if the DHCP server is running:
* Check if the dhcpd.conf file is configured properly
i.e. every thing from subnet to ip range is set
properly(refer the sample file).

TFTP:

Connection timed out.

This error can occur due to firewalls/iptables blocking TFTP server's path to connect to the system.
And can be resolved by removing the corresponding reject rules from the firewalls/iptables.

ARP timed out

It's a problem which can make one think even thought the solution to his is very simple.
This occurs due to Network is not able to resolve the servers MAC address,
and can be solved by connecting a switch(layer 2 device) in the network.
Switch maintains a ARP table which helps the systems to Know the MAC address of the system which it wants to
connect.

NFS:

Mounting file system timed out.

I found only one error of such kind and that also didn't bothered me for a long time.
This was due to wrong path in the /etc/exports file i.e. the path of the exported Dir is wrong.
And as you may guess the solution is as simple as it seems go to the /etc/exports and change the path.

Initialization

Restart DHCP, TFTP and NFS server and check if they are running properly.
Change the bios setting on the client machine to boot from the network.
Create users to access the client as you can not login as root.

Fedora-DS

2008-09-17T02:25:00.001-07:00

To start with one can follow some simple steps---------------->

1> get installation file of fedora-ds.
2> install fedora-ds.
3> configure fedora-ds.
4> buid schema file as required.
5> configure replication(as required).
6> add required tree structure to the fedora-ds database(i.e. nodes like addressbooks and groups).

All these points are in details in next lines----------------->

1> Get fedora-ds either by direct downloading or wget http://directory.fedoraproject.org/download/fedora-ds-1.0.4-1.FC5.i386.opt.rpm
It is recomended to download the compatible version from http://directory.fedoraproject.org/wiki/Download

2> Installing fedora-ds is like any other RPM package.
rpm -ivh fedora-ds-1.0.4-1.FC5.i386.opt.rpm
If you are planning to run console on the same machine on which the server is then you also have to install the java.
rpm -ivh jre-6u6-linux-i586.rpm

3> To configure fedora-ds run--->
/opt/fedora-ds/setup/setup
before running this make a user for fedora-ds(this is to avoid giving root user as fedora-ds user).
On running this we have to give certain required specifications like domainname, fedora-ds user, admin login, manager login, suffix etc
if the setup succeds only then start-admin will apear in dir(/opt/fedora-ds/)
And if fails the possible reason may be wrong specification sgiven at the time of setup.
At the end of the configuration it will give you command to run console.
to run it first--->
cd /opt/fedora-ds
./start-admin
that command------>(./startconsole -u root -a http://yourdomain.com:(port no.)/)

4> To buid schema file one should have sufficient knoledge of fedora-ds or the simple way is to do this via console.
Go to the console click directory server ----> config ----> schemas ----> first buid atributes then object classes.
the additions you make can be seen in the form of ldif file /opt/fedora-ds/slapd-(yourdomain)/config/schema/99user.ldif

5> As per replication is considered this can be done only via console.
To do this go to the directory server config ----> replication ----> first activate replication here ----> now select the suffix you wan tto replicate --->
now configure its replication requirements like binddn(i.e. the dn which have sufficient previlleges to access the database) , replication type(master-consumer/multi-master) etc
After that build the replication agreements of that suffix by doing right click on the suffix under replication.
On agreement has to buid for each machine with which you want it to replicate it's data.

6> This step is as per requirement of your organization andit can b edone by both commandline using ldap-utils(ldapadd, ldapmodify etc) or through console.

#####################################################

In this installation some OS specific dependencies may occur that can be solved by installing some helping-utils which you can reffer from link given billow

http://directory.fedoraproject.org/wiki/Download

#####################################################

Openldap

2008-09-17T02:23:00.000-07:00

sudo apt-get install slapd ldap-utils phpldapadmin libnss-ldap libpam-ldap

dpkg-reconfigure slapd

vim /etc/ldap/slapd.conf
include -------------------> all required scema files(build them if u can)

And now for client configuration.

vim /etc/ldap/ldap.conf

HOST 127.0.0.1
BASE o=example.net

To do rest in GUI mode.

http://localhost/phpldapadmin/

Create the database(account book), you can do it in GUI also.

vim ****.ldif

to create authenticated user under admin group

dn: uid=***,cn=admin,dc=example,dc=net
uid: ****
objectClass: account
objectClass: simpleSecurityObject
objectClass: top
userPassword: ******

to create authenticated user under other groups

dn: uid=***,ou=****,dc=example,dc=net
uid: ****
objectClass: account
objectClass: simpleSecurityObject
objectClass: top
userPassword: ******

to create objectclass like organizationalUnit

{{

dn: ou=admin,dc=example,dc=net objectclass: organizationalunit ou: admin

}}}
to create normal addressbook entries under any group

dn: cn=***,ou=***,dc=example,dc=net
objectClass: organizationalRole
cn: ***
roleOccupant: cn=***,dc=example,dc=net

Now to add entries either add them from GUI or by comand line

ldapadd -x -f *.ldif -vD "cn=***,dc=example,dc=net" -w ***

And to test the directory try searching

ldapsearch -x -b 'dc=example,dc=net' '(objectclass=*)'

To give permmisions to authenticated users

vim /etc/ldap/slapd.conf\

access to *
     by dn="cn=admin,dc=example,dc=net" write
     by dn="uid=***,cn=admin,dc=example,dc=net" write
     by * read

* line no. 2 of permissions was to give the user full permissions.
      read --------> to see

      write--------> to edit add delete

pam-nss-ldap

PAM is an intermediator which provides the application the required information about the user.
and helps him clearing authentication without changing it in applications configuration files.
What it does is , it checks for the authentication(user name and password) into the system files then into the programs data.
And lets the user login if it gets the required information from any of the source.

server--------------------->

client-------------------->

install

sudo apt-get install ldap-auth-client
sudo apt-get install ldap-utils libpam-ldap libnss-ldap nscd

edit /etc/ldap/ldap.conf
to look like this-->

host 10.10.5.3
base dc=example,dc=net
pam_filter objectclass=prosixaccount
pam_login_attribute uid
pam_member_attribute memberuid
pam_password crypt
ssl on
sslpath /etc/ssl/certs

Now edit pam.conf
to look like this-->

login   auth sufficient /usr/lib/security/pam_ldap.so.1
login   auth required   /usr/lib/security/pam_unix.so.1 try_first_pass
login   auth required   /usr/lib/security/pam_dial_auth.so.1

telnet  auth sufficient /usr/lib/security/pam_ldap.so.1
telnet  auth required   /usr/lib/security/pam_unix.so.1 try_first_pass

rlogin  auth sufficient /usr/lib/security/pam_rhosts_auth.so.1
rlogin  auth sufficient /usr/lib/security/pam_ldap.so.1
rlogin  auth required   /usr/lib/security/pam_unix.so.1 try_first_pass

dtlogin auth sufficient /usr/lib/security/pam_ldap.so.1
dtlogin auth required   /usr/lib/security/pam_unix.so.1 try_first_pass

rsh     auth required   /usr/lib/security/pam_rhosts_auth.so.1

other   auth sufficient /usr/lib/security/pam_ldap.so.1
other   auth required   /usr/lib/security/pam_unix.so.1 try_first_pass

login   account required /usr/lib/security/pam_ldap.so.1
login   account required /usr/lib/security/pam_unix.so.1

dtlogin account required /usr/lib/security/pam_ldap.so.1
dtlogin account required /usr/lib/security/pam_unix.so.1

other   account required /usr/lib/security/pam_ldap.so.1
other   account required /usr/lib/security/pam_unix.so.1

other   session required /usr/lib/security/pam_unix.so.1

other   password required /usr/lib/security/pam_ldap.so

Now edit /etc/nsswitch.cong
to look like this-->

passwd:         files   ldap
group:          files   ldap
shadow:         files   ldap

Now edit /etc/nscd.conf
to look like this-->

 enable-cache            passwd          yes
      positive-time-to-live   passwd          600
      negative-time-to-live   passwd          20
      suggested-size          passwd          211
      check-files             passwd          yes
      persistent              passwd          yes
      shared                  passwd          yes
      max-db-size             passwd          33554432
      auto-propagate          passwd          yes

      enable-cache            group           yes
      positive-time-to-live   group           3600
      negative-time-to-live   group           60
      suggested-size          group           211
      check-files             group           yes
      persistent              group           yes
      shared                  group           yes
      max-db-size             group           33554432
      auto-propagate          group           yes

enable-cache            hosts           no
      positive-time-to-live   hosts           3600
      negative-time-to-live   hosts           20
      suggested-size          hosts           211
      check-files             hosts           yes
      persistent              hosts           yes
      shared                  hosts           yes
      max-db-size             hosts           33554432

      enable-cache            services        yes
      positive-time-to-live   services        28800
      negative-time-to-live   services        20
      suggested-size          services        211
      check-files             services        yes
      persistent              services        yes
      shared                  services        yes

Restart the server

service slapd restart

restart nscd

servive nscd restart

the configuration is done just check if u can modify the data from the same client and other clients.

Snort+Barnyard+MySQL+Base+RRD-Snort

2008-09-05T04:31:00.001-07:00

Index:-

1> Snort

a> install and configure snort.
b> configure snort to give binary output.

2> Barnyard

a> install and configure barnyard.
b> run two instences of barnyard to get output in local as well as remort machine.

3> Mysql

a> install and configure mysql.
b> install mysql on both local as well as remote machine.
c> configure remote mysql to get data from more than 1 machine.

4> Base

a> install and configure base on the remort machine.
b> configure it to generate report of data got from several machines.

5> RRD-Snort

a> install and configure rrd-snort.
b> configure rrd to generate graph of mysql-snort data.

Snort

To start witth install snort depending on the machine you are running.
1> tar -xvzf snort*.tar.gz
2> rpm -ivh snort*.rpm
3> *.deb

mkdir /var/log/snort  ----------> for snort to log snort.log files.

Configure snort.conf.
Edit these lines of your snort.conf file.

var HOME_NET (your ip)

var RULE_PATH (TO THE DIR. WHERE YOU HAVE YOUR RULES FILES)


# unified: Snort unified binary format alerting and logging

output alert_unified: filename snort.alert, limit 128
output log_unified: filename snort.log, limit 128

Starting snort.

snort -c /etc/snort/snort.conf -i eth0 -l /var/log/snort

-c -----> for snort to look into this configuration file.
-i -----> for snort to look into the packages comming from this intrphase.
-l ----> for snort to log the alert file into this dir.

Barnyard

To start with barnyard install barnyard .
Comile it either with mysql or get a pre compiled rpm or deb package(compiled with mysql).

1> tar -xvzf barnyard*.tar.gz

cp barnyard.conf to 2 places  ---------> to run 2 instences of barnyard.

Configure barnyard.conf.
Edit these lines in barnyard.conf file to give data to mysql on local machine

output alert_acid_db: mysql, sensor_id 1, database snort, server localhost, user root, password root
output log_acid_db: mysql, database snort, server localhost, user root, password root

Edit second barnyard.conf file to give data to mysql on remote machine.

output alert_acid_db: mysql, sensor_id 1, database snort, server remorthost, user root, password root
output log_acid_db: mysql, database snort, server remotehost, user root, password root

Starting barnyard.

barnyard -c /etc/snort/barnyard.conf -g /etc/snort/gen-msg.map -s /etc/snort/sid-msg.map -d /var/log/snort -f snort.log -w /var/log/snort/barnyard.waldo

-c -----> for barnyard to look into this configuration file.
-g -----> for barnyard to look into this gen.map file.
-s -----> for barnyard to look into this sid.map file.
-d -----> for barnyard to look into dir. for the alert files logged by snort.
-f -----> for barnyard to look for files with name starting with.
-w -----> for barnyard to log the temp. dta into the file.

Mysql

Install mysql using apropriate package.

Configure mysql. (on both machines)

1> mysql -u root -p
2> create database snort;
3> grant all on snort.* to ***@***** identified by 'password'
4> quit;

zcat (make_mysql file) | mysql -u root -p snort

1> mysql -u root -p
2> use snort;
3> show tables;
4> quit;

(FOR REMOTE MACHINE)
you can make more than 1 database as required

1> either to get data in a single database from diff. machines.
2> or to get dat from diff. machines into diff. databases.

Base

To make base working first install php, adodb, apache, libapache-mod-php, php-pear. php-mysql, php-gd, libphp-adodb.(look for compatible versions for your machine)

Install base using appropriate version.

Configure base_conf.php.

cp base_conf.php.dist base_conf.php

Edit these lines in your base_conf.php file.

$BASE_urlpath = '/base';  (path where apache server look for files  ex. ->http://localhost/base)

$DBlib_path = '/var/www/adodb5'; (path to adodb libraries)

$alert_dbname   = 'snort';
$alert_host     = 'localhost';
$alert_port     = '';
$alert_user     = 'mysql-snort-user';
$alert_password = 'mysql-snort-password';

Running base.

go to a browser----->

and type the ip of your system.

There after you can see diff. files to choose from---> click on the base.

RRD-Snort

To get rrd-snort working first install rrdtool.

Then get rd-snort.pl file.

always run rrd-snort in the dir where you want it to place the resulting graph.

run rrd-snort.

perl rrd-snort.pl -H hostip -u (mysql-snort-user) -p(snort-user-password)

Snort Basics

2008-09-04T03:52:00.000-07:00

To start with you can get snort from www.snort.org.

In the first part of our discussion we will configure snort on an ubuntu machine.
And in the later blogs we will procede towards others like Redhat or Deb.

The main problem which one faces in installing snort is meeting dependences.

So to solve this i think we should start first.
start with ----->
tar -xvzf snort-2.8.1.tar.gz
cd snort-2.8.1
./configure -------------------> (for IDS)
./configure --enable-inline -------------------> (for IPS)

You can add --enable-mysql to any one of them to push your snort alerts to mysql database.

Generally you wont find any error while configuring IDS.

So hoping your IDS does not gave you a trouble we end it here and even if it gives they wont be different from IPS, that means you can refer to them if you find one(trouble).

To start snort with IDS first get rules
(that too you can get from www.snort.org).
Untar then at /etc/snort/ DIR.
now
cp snort-2.8.1/* /etc/snort/
this makes it convenient as you(and your system) get all files at one place reducing the chances of error.

mkdir /var/log/snort ----------> for snort to log its alert files.

Now make some changes to your snort.conf file lying in /etc/snort DIR.

var HOME_NET localhost ---------->this should be valid ip of your system.

var RULE_PATH /etc/snort/rules

Now initiate snort:>

snort -c /etc/snort/snort.conf -i eth0 -l /var/log/snort

You can also put -A console to get alerts on screen or put -D to run snort in deamon mode.

Now its time we should be concentrating on IPS with or without mysql.

Start with trying to compile snort by--------------->
tar snort-2.8.1.tar.gz
cd snort-2.8.1
./configure --enable-inline --enable-mysql

at this you are likely to get an error i.e.
libipq.h not found
This can be eliminated by
copy libipq.h to /usr/include/
And if incase u don't have libipq.h file
then do-------------->
apt-cache search libipq.h
And install the package which provides you the file
apt-get install *

Other errors include errors like----------->
libpcap not found
same is to deal with these kids of errors.

But to do it with --enable-mysql you have to get some prerequisites.

Libpcap0.8-dev
libmysqlclient15-dev
mysql-client-5.0
mysql-server-5.0

After this i don't think it will give you any kind of trouble.

To get snort-inline working you have to get iptables up
And add rules to iptables in such a way that packets stand in a queue.
Such as--------->
iptables -A INPUT -p tcp -j QUEUE
iptables -A INPUT -p icmp -j QUEUE
iptables -A INPUT -p udp -j QUEUE

You can do this for any chain of iptables, depending upon what kind of work you want from your snort to perform.

To initiate snort-inline just use-------------------->
snort -QC /etc/snort/snort.conf -i eth0 -l /var/log/snort

You can also put -A console to get alerts on screen or put -D to run snort in deamon mode.