fail2ban: Protecting Servers

Generally recommending security measures is considered as the work of Application and Network Security team. But with introduction of DevOps and change in culture, everyone can contribute to it.

I usually recommend teams to embed these security tools into the development practices to ensure better end results. Coz if used properly tools like these help in maintaining better code quality and keeping infrastructure protected.

Few Open Source tools:
YASCA: http://www.scovetta.com/yasca.html
PMD: https://pmd.github.io/
SNORT: https://www.snort.org/
Fail2Ban: http://www.fail2ban.org/wiki/index.php/Main_Page

Let me explain more with an example:

With expanding infrastructure, we need tools to keep any on malicious attempts and take appropriate actions against them.
I use Fail2ban for this, which is one of the best Open Source tool available for this purpose.
How it works? Fail2ban can monitor logs/files based on defined patterns and take action when match exceeds the threshold.
Example: Pattern defined: <HOST> - - .*/create-account.html .*
Threshold Definition:

• Take action if finds 20 connections within 20 seconds from one IP.
• Blocks it for 1800 seconds. - Can be set to any other number or forever.

Actions: Actions like blocking IP via iptables, denying host via hosts.deny file, sending email notification, etc can be triggered once IPs/Users are caught for malicious activities.

Conclusion
Reach out to people who know about security tools and can help you setup these. Leave the rest to the tools to do their duty, small effort from every team member will help in making internet world more secure.

Important links:
Fail2ban Installation instructions: http://www.fail2ban.org/wiki/index.php/MANUAL_0_8
More Security Tools: https://www.owasp.org/index.php/Tools

Sample Configurations:
jail.conf:

[apache-custom-rule]

enabled  = true

action   = iptables-multiport[name=qa, port="http,https"]

filter   = apache-custom-rule

logpath  = /var/log/apache/access.log

maxretry = 20

findtime = 20

iptables-multiport.conf:

actionban = iptables -t nat -A PREROUTING -p tcp -s <ip> --dport 80 -j DNAT --to <Your Private IP>:80

                iptables -t nat -A PREROUTING -p tcp -s <ip> --dport 443 -j DNAT --to <Your Private IP>:443

actionunban = iptables -t nat -D PREROUTING -p tcp -s <ip> --dport 80 -j DNAT --to <Your Private IP>:80

                iptables -t nat -D PREROUTING -p tcp -s <ip> --dport 443 -j DNAT --to <Your Private IP>:443

apache-custom-rule:

failregex = <HOST> - - .*/accounts/u .*

                  <HOST> - - .* 403 .*

What the "F" is DevOps?

Recently Attended "DevOps Days India" here in Bangalore. Those two days with people who either are in DevOps role OR want to adopt it, forced me to rethink: "What the F is DevOps?"

A little background first
This two day conference had lot of stuff, but to me it missed the basic concept of DevOps days. And was the introduction to DevOps.
To fill that gap I proposed the same topic for "Open House/Session" and to my surprise lot of people voted to discuss that :-)

During the Discussion
Lot of people gave their opinion about this, some of them were:

• DevOps means Developer + Operations. It doesn't contain QA else it should be QAOps.
• It is a Developer who knows Ops (specifically system administration).
• It is an Ops guy who knows about Development.
• The guy who builds and deploys.
• The guy who sets up tools for daily use.

My Idea about DevOps
DevOps originated from two words: Development (not Developer) and Operations (not sys admin or similar)
Here Development includes any activity done to build the product/service (including QA) and Operations consists of all the activities included to keep it up & making money out of it.

So in my opinion DevOps is all about solving issues, resolving blame games and bridging gaps between teams.

So In short it is not a guy or a team and definitely not a tool.
DevOps is a philosophy that anyone in the team can implement. It is about solving any issue using: Processes, Tools or People. It is also about solving the issue for good (not just for now).

Few example would be:
Issue: Bugs are caught later in the SDLC
Solution: Introduce CI and SCM tools to get results faster. Introduce Agile process make people react faster incase of issue.
Type: Solved issue with Tools and process.

Issue: Teams have no idea about the other teams work.
Solution: Get the team working on Collaborative development. Get them to work together (may be in a single room) on a periodic basis. Conduct brown bag sessions etc.
Type: Solved issue with people and bridging gaps between them.

Issue: Who owns the build failure?
Solution: CI tool send the email based on the failure. Example: If the build fails while compiling or test cases fail then the culprit's email goes to developer, but incase the test case have issue in running then QA gets the culprit's email.
Type: Solving blame game.

Then who is a DevOps guy?
I don't believe in such a role, but still it is a guy who has experience with solving issues. Who can look at a problem like a consultant and give a solution without being getting stuck in the problem itself.
So in generalised term:
DevOps guy is the person responsible for implementing the DevOps Philosophy across the Organisation.

Continuous Delivery WhitePaper

Continuous Delivery in a Nut Shell

Whatever may be the type or size of the industry, transformation is the key to improvement. Continuous Delivery (aka CD) is emerging as the front runner in the race of transforming IT business from “Slow Delivery” to “Quick and Reliable Delivery”. In combination with right Agile methodology, CD is helping companies to reduce time from planning to production and hence allowing them to earn cash for their product/services faster and with better margins.

What’s in it for Me?
CD has something for everyone in different teams, product, companies or domains. For engineers in any domain (from textile to Software) it means to automate production plan and shipping finished product/services to customer faster and on regular intervals.

CD for Software Development.
From Software Development perspective, CD is divided in 5 components.

#1: Configuration Management. “Single Source Of Truth”
#2: CI and Build Management. “Early and Often”
#3: Testing. “Early, Repeatable and Ever Improving”
#4: IT and Infrastructure Management. “On-Demand and Scalable”
#5: Release Management. Ship Right Release at Right Time”

To set things right CD evolved with 8 basic principles.

Principle #1: The process for releasing/deploying software MUST be repeatable and reliable.
Principle #2: Automate everything possible!
Principle #3: If something is difficult or painful, do it more often.
Principle #4: Keep everything in source control.
Principle #5: Done means “released”.
Principle #6: Build quality in!
Principle #7: Everybody has responsibility for the release process.
Principle #8: Improve continuously.

Summary in a Nut Shell. 

* Special thanks to  David Farley & Jez Humble (as some text of this post is sourced from their book "Continuous Delivery")

Good to be Agile

It's been a long time that I wrote any blog, so now here I am again.
This time I think the best topic to share is working with Agile methodologies.

It's been more than four year now that I have been working with the same. Some initial questions may occur in the mind of the reader of this blog.

Questions like:
- What is actually AGILE?
- How can it make my work better?
- What are it's pros and cons?
- etc. etc.

By definition agile means quickly producing something useful and improving it over a period of time to the best of it's type.
But quick doesn't means to forget the real iterations required for the process.

I will put an example here:
Suppose you are developing a software, under waterfall model we start coding finish the project over a period of time and pass it to the customers.
But in agile we start coding get some thing usable (something like a prototype) pass it to the customers, then start improving it (taking help from customer's feedback). This ends up in a better end product as it includes customers feedback also. And a software which has passed phases like creating prototype, testing, retrofitting etc. (real time iterations), is of greater value to the customer.

There are various Agile methods, our need defines as which one will suit us the best. Example:
KANBAN:  A method for developing products and processes with an emphasis on just-in-time delivery (using KAN="Visual" BAN="Board" technique) while not overloading the developers. With proper use of Kanban, projects are able to eliminate bottlenecks from any part of development cycle.

SCRUM: It is an iterative and incremental development process, where effort is tracked by burndown charts (each sprint) and Velocity tracking (over many sprints). These metrics help in better, predictable sprint planning and hence delivering happy faces.

In the Soup

As trying to consolidate all my accounts under one roof, I tried migrating my blogs from Yahoo ID to Gmail ID.
But what a mess, I could only migrate the posts and not the profile data. I lost data like as "Since when I have been blogging" etc.

Hoping to get such feature in blogger.com which would allow to change Username OR the User ID without loosing blog stats.

Google Apps Active Directory sync.

Are you planning to migrate from MS suit to Google Apps?

Some time back I did one such migration and came across several posts where people said "Google AD sync doesn't work properly", etc etc.

But I found it very useful and usable, with a fact that not much of the documentation is available around it.

This lack of documentation inspired me to write this blog, hopefully it will ease up your work.

To begin with, you need to have:
* Working MS AD and an account with Google App
* Administrative credentials for both (AD and Google App) accounts.
* "Google Apps Directory Sync tool" installed on your system.

Now lets see how it is done:
* Start the "Configuration Manager" from your startup menu.

* Fill the required details as shown in screenshots.

General Settings: Check the options that you want to migrate to Google Apps. In my case I used only Users and Groups.

Google Apps Settings: Fill the email and password of user with Administrative privileges on Google Apps.

LDAP Connection: This part needs your local LDAP/AD details. The provided user nay not be admin, but needs only read access.

User Attributes: Provide the local (LDAP/AD) attribute for mail ID (in my case it was mail).

User Extended Attribute:  This configuration maps local given name, sir name and password attributes to Google Apps.
Specifying Suspension Policy enables the user to decide what to do if a user is present in Google Apps and not in local AD.

User Sync: Add a new rule. This rule will define which users to sync (or which user not to sync). The rule field accepts RegEx, allowing us to add multiple users through one line.

Notification: In this section define your SMTP server and other details required to relay email on success or failure of this sync up.

Sync Limit: This section is to specify the suspension limit. In simpler words "The script should fail if more users are being suspended (than the specified limit)".

Log File: Specify a valid log file path here. The sync tool will save its output in this file.

Anything Else: There are many more options available (example syncing contacts etc), but in this blog I documented only the minimalistic configuration (and that was what I required fro my project).

* Test it out.

Simulate Sync:  This tab will tell you if your configuration is correct. And if you don't find any error (warnings are still OK), then just hit "Simulate Sync" @ the bottom.

Result: Hitting Simulate Sync will take you to a new view. Check if you see users being synced and you don't have errors.


* Save this configuration to a file. I saved it as google-sync.xml.

* Run it: As of now this sync can only be run through command line, so just run this command and see the magic.

sync-cmd.exe -a -c "google-sync.xml" -r C:\Sync-logs\Google.Sync.logs

* Schedule it with the help of tool like Jenkins and you are done with one aspect of migration which gives head ache to people.

PS: Scheduling it through a tool makes it easy to monitor and/or run it on demand.

I am confused (Google docs Vs Dropbox)

One of the interesting thing that I worked recently on was a small decision of either using Google docs or Dropbox for storing and sharing documents.

Seems unimportant?

I also thought the same on hearing this for the first time, but believe me this proved to be an interesting study.

Let me take you to the tour of both.

Google docs is an online solution provided by Google to collaboratively work (sharing, editing, etc) on the documents.
  Several addons like Cloud Connect, etc helps us to sync our native MS office documents to Google docs.
One of the advantage of Google docs is that multiple people can edit one single document @ the same time.

On the other hand Dropbox is used to share files, irrespective of the file being a document, image or some thing else.
Same account can be either synced to multiple computers OR same files can be shared to multiple accounts, hence giving feasibility of accessing the files across the globe (on multiple systems).
Feature like sharing and selective sync help us selective share our files with different people.
One of the advantage of Dropbox is the ability to share all sorts of files (not only just documents)

And the confusion starts :-)

It looks pretty simple that Google docs should be used for documents and Dropbox for other files, but the problem statement lies in the fact that Google changes format of our existing MS office/Open office  documents to Google docs format (on uploading). In most of the cases this is not an issue, but it created trouble when you have formulas, macros and images in your documents, as
"They may disappear from your uploaded document".
Google also provides a solution to this, in the form of uploading the documents (in this case other files can also be uploaded) in original format (not changing the format to Google docs format).

Here comes another "BUT".

But this also has a drawback, that we can't maintain single source of truth for the documents. To edit any document we have to download it and then again upload, which creates multiple copies in Google docs (which have to be manually deleted).

Conclusion after digging deaper:
Using any of these tool is subjective.

IF:
* You are creating the documents from scratch.
* And the priority is to have simulanious editing of documents, rather than maintaing MS office formatting, example: Attendance sheet etc.
Then Google docs is THE tool.

ELSE IF:
* You want to use your existing docs.
* And the priority is to use the MS office format.
Then Dropbox is THE tool.

Advantages:
* Google can save money sent on MS office licenses.
* Both the tools can save the hassle of maintaining local shared file server.
* Both have there recovery mechanism, so no danger of getting fully F**k*d.
* Both provide full fleged files/documents sharing system (sharing sccross the globe).
* Both provide the confidence of having one single "Source Of Truth" for documents.
* Dropbox provides offline editing of files (as they gets synced when connected back to internet). Same is provided by Google docs when used with Cloud Connect.

As my requirement was more of sharing and offline working, so I choose Dropbox for now :-)

Cross Platform scripting with bash.

Is this title a bit confusing?

I asked this question to many people and got the same answer as "YES"
In our day to day life when we talk about bash, we refer to Linux as it's OS.

But the main punch of this blog is to mak BASH break the boundaries of Operating System.
I agree that till now bash is not capable of doing this alone.

Now any guesses about the tool?

If you guessed "CYGWIN" then consider yourself as winner ;-)

To make it more clear for people who were not able to guess it, I can put my example here:

Background: I am a consultant and work with all sorts of projects, during my last couple of projects I was automating build & release cycle for a product that is built on four platforms (Linux32, Linux64, Windows32, Windows64), hence I was desperately looking for some cross OS/platform scripting solution.

My Thoughts: I am a BASH guy and as any developer, my first preference was to look for a solution around what I already know, rather than straight away jumping into any new technology and learning it (it was my second option though).
If it was not bash then I was looking into ant or others to do the job.
As
"The best way to do a work, is to do it." --Harpreet Singh

For Newbies: Cygwin is Linux-like environment for Windows making it possible to port software running on POSIX systems (such as Linux, BSD, and Unix systems) to Windows.

Solution: Now lets put this together.
Install Cygwin on a windows system with basic features like "openssh", "vim", etc.
That's it you are ready to rock 'n' roll.
All most all your Linux commands can run in this shell, which includes doing "ssh" into a Windows system ;-)

Need More: The story is not over yet. Combining the above setup with a CI tool (Jenkins in my case), gave me what I wanted.
A full fledged automated build system, running on four platforms simultaneously (using Jenkins' multi configuration project feature), with ease of writing the script only once and that too using BASH.

Now that's what I call a win ;-)

The winners prize: As a prize of my effort done for the above setup I got:
* Time saved per build/release cycle.
* Time saved in developing automation scripts.

And more important a "Happy Client"

Nagios: Monitoring Systems & Applications

"Someone please help me."
These are the first words we usually say when we see our production system stuck/crashed due to some reasons.
But "Someone helps those who help themselves."  --Harpreet Singh ;-)

Now jokes apart.
But suppose we get an alert before the systems fails may be during the first stroke or when the load starting going high or may be when total processes were more or any thing related to our applications running on the server.

Wouldn't this be like a boon, a chance to save the system in time?
If you have seen my earlier posts you will find monit/munin doing the same, but as on the way of my learning, I found that nagios is a better (easy and more flexible/plugable) tool.
Before starting to explain on why my opinion changed I will ask you one question here.
What do you expect/need from any system/application monitoring tool?

The general answers would be:
 - Stable.
 - Good UI.
 - Easy Installation.
 - Easy configuration.
 - Good coverage over different applications and system.
 - etc. etc.

Now lets see if nagios answers all of these?

Like other tools nagios also has client-server architecture, which gives us freedom to monitor any number of  systems/applications from one nagios server.
It has a easy to understand & configure UI, through which you can do many things like scheduling, controlling alerts etc. And if you are a CLI lover (as most of Linux geeks are) then you can do all those from command line also.

Now here comes the most impressive part.
Nagios is highly flexible. First of all it has huge plugin base already available for you to work with.
But if that is not enough for you, then ask yourself just one questions.
Do I know how to write a script (bash, python etc)?

I usually say one line for nagios, that "If you can do it through CLI, you definitely can do it with nagios." Same is the answer for the question you asked yourself (above). If you can write a script to perform any action (login check, api calls, application query etc.) and get a small readable/understandable output (for both
success and failure cases). Then it's like a kids play to integrate it to nagios and see the same results in UI.

In simple words:
 - Write a script to perform certain action.
 - Copy that to the nagios script directory (just to ensure that you/anyone doesn't accidentally deletes it).
 - Add that to the nagios commands.
 - Call that command for the host you want.
 - And done.

Another plus part is that you can flaunt in front of your seniors about the work done (with minimal effort involved) ;-)

Why is my application slow?

Wondering what kind of blog post is this?
So let me take you on a brief tour.

Some time back I got a chance to work on Performance Testing of our application (@ openbravo).

On this opportunity one quick questions popped up in my mind: "what do we seek in any performance testing tool?"
 - Easy to adjust with our requirement.
 - Easy to operate.
 - Easy reports at the end.

In our case Jmeter answered them all.
 - With Jmeter recorder we were able to record the flow of the application. And with some adjustments like variables, assertions, halts, etc. we reproduced the scenario where a normal user will be using the application.
 - After creating the script we can easily run it either from Jmeter UI console or CLI.
 - Now for reports Jmeter itself has good number of ways in which we can view reports (graphs, tree view etc), but for sharing those reports is a bit of overhead, so another simple way is to run the test through Hudson (CI tool) and view reports using Hudson's Jmeter plugin.

So basically before any end user/customer shouts "Why is my application slow?", this tool integrated with hudson does a regular check of the delay, failed requests, etc  to track the performance and give you an upper hand over other similar applications.

When I was on my way with this work someone asked me "RM and performance testing?", "shouldn't QA do that for you?"
I have a single answer to such question and truly writing there is nothing like mine or your work. It's only a perception, if we have time and scope of doing something then I think we should extend our hands and do it.

Fun With SFTP

Until now I never knew that using FTP can be so easy.
Many of you would have started thinking about the FTP servers till now, but to be more clear here I am talking about SFTP (SSH File Transfer Protocol).

But as Shakespeare said "What's in name, the thing we call rose would smell as sweet with some other name".
And so is true for SFTP, as this provided the usability that I was looking for and that too with minimal configuration with some extra benefits which we will talk about in the last.

And not only that I was able to do this in two different ways.

• One is like my as colleague Juan Pablo says "It should be a JAIL for the user", so that he cannot move outside the directory.
• And the other one is like simple ftp which allows you to move around but not able to read or write unless you have permissions.

To know it better I think a use case will be really helpful.
So I will put down the requirement that pushed me to learn about it.
We needed to grant permissions to one directory to a user, with one directory I literally mean that, as we wanted to block him from peeping into other things.
That too with minimal access to system binaries and it should be secure etc etc.

And SFTP was the best suit for the requirement, you will get the answer of how in the next section where I have shown the configurations of both the cases and believe me it couldn't have been simpler.

Lets get into the jail first ;-)

• Edit /etc/ssh/sshd_config to include this.

Subsystem sftp internal-sftp
Match User sftpuser
ChrootDirectory /var/www/sftpdir (this makes the user stay under one directory)
ForceCommand internal-sftp

• Create the sftpuser and set it's shell acess to false, so that user is not able to do ssh.

useradd -m -s /bin/false sftpuser

• Give correct permissions to sftpdir.

chown root:root /var/www/sftpdir

• For increasing security I changed these also in /etc/ssh/sshd_config:

PasswordAuthentication no
PubkeyAuthentication yes

And I also added my pub key to /home/sftpuser/.ssh/authorized_keys file, but this is optional as this doesn't make any effect on the SFTP working.

For this jail method we are done.

Try connecting:
sftp sftpuser@localhost
Connecting to localhost...
Enter passphrase for key '/home/user/.ssh/id_dsa':
sftp>

Now lets get back and see the next way (I know most of us will not read this, as first one will work like a charm):

• Create a user with /usr/lib/openssh/sftp-server as shell and /var/www/sftpdir as home dir.

sudo useradd -s /usr/lib/openssh/sftp-server -d /var/www/sftpdir sftpuser

• Add this /usr/lib/openssh/sftp-server to /etc/shells file

echo "/usr/lib/openssh/sftp-server" >> /etc/shells

• For increasing security I changed /etc/ssh/sshd_config and also added my key to /var/www/sftpdir/.ssh/authorized_keys file.

PasswordAuthentication no
PubkeyAuthentication yes

• Set correct permissions of the sftpdir.
  

chmod go-w /var/www/sftpdir
chmod 700 /var/www/sftpdir/.ssh
chmod 600 /var/www/sftpdir/.ssh/authorized_keys

And done.

Try connecting:
sftp sftpuser@localhost
Connecting to localhost...
Enter passphrase for key '/home/user/.ssh/id_dsa':
sftp>


Now about the extra benefits:

• Easy to configure.
• Good in Security.
• Can work with PubkeyAuthentication.
• No extra installation (as uses SSH).
• Easy to use SFTP client.

at least these things are enough to pull me towards it.

That's it for taday. Happy SFTPing.

Sizing Openbravo: EC2 cost calculation and experimenting with around 270 concurrent users

To help Openbravo partners and users we have extended our Sizing Tool Guidelines to include some more facts and findings.

"As more the better" --Harpreet Singh ;-)
Same has been proved by the new Amazon Cluster Compute Instance (cc1.4xlarge).

Amazon recently anounced the availability of it's biggest Instance, ideal for cluster infrastructure, as it promises high connectivity between cluster instances (as high as 10 Gigabit Ethernet).
But we tested this instance for the standalone test with Oracle DB and Openbravo on same instance.
The results were realy exciting as it was able to handle around 270 concurrent users.

Now truely speaking "Thats what I call results."
And the same results have been added to our Sizing Tool Results.

"There is no such thing as a free lunch." --Milton Friedman
This instance also has some drawbacks:
- It costs a lot (almost $1.60 per hour)
- Till now it is only available in US (N Virginia) region.
- And is only available with CentOS.

Now here comes another one.
As cost is the biggest concern when we think of any new infrastructure. For example: Running an instance (which can support 10 concurrent users) for 3 (THREE!!!) years on Amazon EC2 would cost only $1217.60, I think these figures can help one think about on-site and in-cloud (EC2) deployments.
So we extended our Sizing Guidelines to help you choose your Amazon Instance.

In the last section of the Guidelines we have added:
- Steps you can follow to calculate your yearly cost with Amazon cost calculator.
- As Amazon calculator is a bit complex so we created simple calculator to help you out.
- And pre calculated cost for most common scenarios.

Openbravo Ubuntu Maverick (be the first to use it).

We were planning to blog and call for users to come up for early testing of your own trusted Openbravo with the upcoming Ubuntu's 10.10. But as Ubuntu Maverick (10.10) is already released, I think now all of us have a chance to be the first to use Openbravo with Ubuntu Maverick.

To be the first all you need to do is get Ubuntu Maverick Meerkat up and running any where you like may it be your hardware system, Amazon EC2 or a virtual machine.
This link will help you if you are planning to install on hardware system or virtual machine and these AMIs to boot one in EC2

Once you are set then lets rock and roll. I mean start installation.

So all you have to do to install Openbravo is:
- Enable the Partner’s Repository:
* sudo add-apt-repository "deb http://archive.canonical.com/ubuntu maverick partner"

- Install the openbravo-erp package:
* sudo apt-get update
* sudo apt-get install openbravo-erp

You can also install it using Synaptic or the Ubuntu Software Center:


Can installing of any comprehensive ERP be simpler than this?

You can do it even on a Friday. ;-)
As installing most ERPs on a Friday means forgetting about your Friday and Saturday night fun. With Openbravo on Maverick Meerkat, you can start the process at 7 and be at the party by 9!

So once you are done with the party sorry I mean installation you are set to use it and be a proud user of Openbravo ERP.
As it's you love and support that we have been able to live up to your expectations.

Also the users/developers who want to upgrade from 10.04 to 10.10, can do that without a fear of breaking the installation, the only concern should be that 10.10 is not a LTS version :-(

For more on installing Openbravo in Ubuntu please follow this wiki.

PostgreSQL: Performance Tuning

"Need is the mother of discovery" -Harpreet Singh
I wrote this line just few minutes before writing this blog, as my need of optimizing PostgreSQL's performance lead me to search/discover for some cool facts and features of postgres and tools related to it.

For any postgres user thinking about 100 or more concurrent users is like a nightmare. I will admit that some time back I was also a bit scared on thinking about 100 concurrent users with postgres, but with the end of my search I am happy that I found a usable way to achieve that.

"Knowledge increases by sharing"

So I thought I will pass it to everyone who is searching for it on the internet.

The need that triggered me to search for this was to recommend Hardware as well as Software configuration to support 100-200 concurrent users on Openbravo ERP and postgres/Oracle as the database.

For me as I am a postgres supporter I believed that postgres will be able to handle it. And yippee I was right.

Coming back to the main point:
Postgres doesn't support too many users (concurrent) by default, it comes with very solid configuration aimed at everyone's best guess as to how an "average" database on "average" hardware should be.
Postgres has some default configuration options to fine tune it, like:
- max_connections
- shared_buffers
- effective_cache_size
- etc etc.

But these are not enough for postgres to support 100+ (concurrent) users.

In a reply of my query to postgres performance mailing list, I came to know about connection pooling.
One and the only con that I saw in this is that it is external, I mean we have to configure an external tool to do connection pooling.

There are tools like pgpool to make the job easy for us (pgpool is a middleware that works between PostgreSQL servers and a PostgreSQL database client).

Connection pooling tools provide us features like:
- Connection Pooling: It reduces connection overhead, and improves system's overall throughput.
- Replication: Using the replication function enables creating a real-time backup on 2 or more physical disks.
- Load Balance: As the name suggests it distributes the queries on two or more replicated servers.
- Limiting Exceeding Connections: With the use of this extra connections are queued instead of returning an error immediately.
- Parallel Query: Using the parallel query function, data can be divided among the multiple (replicated) servers.

Configuring these properly can fine tune postgres' performance to handle 100-200 concurrent users.

Happy *postgresing*

To read more about performance tuning in postgreSQL read this.
For more on pgpool click here.

Module Integration with CI (Hudson)

Module Integration with CI (Hudson)

Long back we (RM @ Openbravo) introduced CI (Continuous Integration) tool (Hudson) for testing code of our core ERP development branch.
Which allowed our developers to do:
- Daily Builds (Full/Incremental).
- Smoke Tests.
- DB Consistency Tests.
- etc.

But as most of the developers were becoming modular (Openbravo became modular with Openbravo ERP Version 2.50), CI was not able to maintain the pace and provide similar help for the module testing.

To set things in place we enabled our developers to integrate and test their modules using new CI even after committing even a single changeset to their module repository.

According to me the goal or I will say the purpose of this whole effort was to enable a developer to have a nice and sound sleep after he pushes his commit to the module repository.
Sounds confusing?

Let me explain it.
Earlier developers use to develop a module and used to do time consuming small manual testing to make sure that their code is bug free.
From a developers perspective he cannot sleep properly until his module is tested and deployed properly.

To enable CI for modules and save developers time (from manual testing) we created the setup which will help developers to directly configure a new job in Hudson to test their changesets. This new setup empowers them to do:
- Sanity Check
- Source compilation check and Create OBX
- Database consistency test
- Module's JUnit test
- Installation of the generated OBX
- Un-Installation of the installed module
- Selenium test (module smoke)
- Upgrade the module from previous published version in CR (Central Repository) to generated OBX
Even for a single new changeset in the module's repository.

And still it has endless possibilities, where we can integrate new test cases to this.

We have also created a template job (which is pre-configured with all these test cases) to help developers configure and run tests for their modules easily.
Developers will just have to copy the template job to a new job, change the variables to their modules related variables and then run the job. We have also created a simplified wiki for step by step instructions.

* Currenty Openbravo developers working on any Modules can take benefits of this tool (but sky is the limit, maybe someday we can allow partners/community to take advantage of this tool).

RM Updates: Amazon backup stratergy, Mantis Upgrade, Establish automatic process for releasing 2.40, OB@OB

These are the latest news from the Openbravo's Release Management Team:


Backup Strategy: EBS boot.

Amazon has a new feature ebs boot, this helps us to keep our root partition in ebs volume and also allows us to have data up to 1TB in root partition. This helps us in the following way : better backup strategy and from now we can pause & re-start an instance and thus saving cost. My colleague gnuyoga has a blog about the same.


Mantis Upgrade: Upgrade issues.openbravo.com to mantis-1.2.0

As you know our existing issue tracker is based on mantis 1.1.8. With the release of mantis 1.2.0, it promises lot of interesting productivity boosters. We are migrating our current mantis to latest. This involves quiet bit of challenge. In this sprint we address customization like SSO (Single Sign On, etc), and custom css. If you want to be a beta tester to testing our new mantis please drop us an email for us to give you a test account.


Continuous release of 2.40 branch

So the mantra of 2.40 branch is continuous release as detailed in my colleague juan pablo's blog post. Now this task is complete and for details see here



OB@OB: Documentation and Linux tool

This task was about documenting the process of replicating production environment to testing environment and creating a new tool that automates this process in linux.

HTTP Basic Auth

As while working with apache many of us feel like having a secure way to access our data.
Simple but effective http basic auth is probably the quickest and the easiest answer.

Setting it up requires only two things:
- htpasswd file (containing valid user name and password)
- And apache configuration file to read it.


Creating a htpasswd file:
- htpasswd -cm </Path/tp/htpasswd-file> <username>
- While adding more users just remove c from the above command.

Configuring apache:
- Add this to default (vhost file) configuration file
     <location>
           Allow from all
           AuthType Basic
           AuthName "Restricted Area"
           AuthUserFile </Path/to/htpasswd-file>
           Require valid-user
     </location>

- Now reload apache and enjoy.

RM updates: Automation and upgrade of Mantis @ Openbravo

We are almost close to achieve milestone 2 of Continuous Integration. The team is working really hard on finding a solution for existing challenges as well as proposing ways to automate current repetitive tasks.

Last sprint we have completed one of the most challenging tasks "automated code migration from pi - main". Now we have obx generated from main branch if all the tests are successful. Plans to generate an obx on every commit are heavily debated within the team.

Now we have tecnicia14 resurrected. This will help our developers as well as our QA team to see the code changes in the live environment (live and liveqa).

Apart from the CI infrastructure, we have also upgraded the Issue Tracker version to 1.1.8 which is the latest stable mantis version available. We are also in the process of ensuring we have a hard backup of all the important instances running in Amazon ec2.

For a complete list of the on-going stories that we are working on, please check the Sprint 28 page of our Scrum spreadsheet

UML (User Mode Linux)

In this Blog I will take you to a tour to a new dimension of Virtualization. That is the world of UML.

Many of us have spent time using VMware, VBox, Qemu etc, or debugging on how to start Xen/OpenVZ.
But the truth of above two is, applications like VMware are heavy on system resources and applications like Xen are a bit tricky (kernel should be Xen specific).

One thing I will admit, that I was also amongst the people who have spent a huge amount of time on different Virtualization technologies, until here at Openbravo I was given an opportunity to setup virtual environment on Amazon servers.
* For those who haven't worked with Amazon servers, Amazon servers are like domU of Xen virtual environment.

The aim was to setup an Openbravo instance (which can be made available on demand) on the top of an Amazon instance.
The main problem in working with amazon server is that you don't have access to host (domO) and neither Amazon people allow you to use custom kernel for domU (custom kernel: so that we can build another xen domU over it).
So taking these things in to account we were left with options like VMware but looking upon their load on system we needed something light and usable.
In our hunt for a perfect virtual environment my manager told me about UML. And truly speaking it turned up to be the perfect tool (it matched all our requirements and expectations).

UML works very differently from all other Virtualization techniques, all it needs to work smoothly is uml-utilities package, one kernel (binary script) and a block device(which contains minimal/full OS).
It is like doing chroot to a directory and installing a full OS in that.
It has many other features like

• Mounting host filesystem.
• Adding a COW (copy on write) file.
  
• etc.

This COW file is used if you want to use same block device (filesystem) for more than one Virtual system. This COW file writes the difference to a separate file (just like a diff patch).

The use of UML gave us one more flexibility, that was using Xen image (pre-installed Openbravo instance) with UML kernel.
And an advantage over Xen (i.e. access over host system is not required in UML).

Reference:
UML Howto
UML Wiki
Download Page

Screenshots and DAlbum

In one of my recent tasks I explored a couple of tools available to publish screen shots (Web Based).

The aim was to get the screen shots when Hudson (CI tool) is running a build and to publish them using some web based gallery creator, for later reference of the developer or QA team.

Well the task was not tough but was a bit tricky, as the screen shot took by Xvnc plugin of Hudson was not sufficient (it took the screen shot at the end of the build, mostly an empty screen), so we decided to get a command line tool to get screen shots. And in the hunt for a command line tool we got ImageMagic and discovered that Xvnc plugin was also using the same tool.
Now by running a FOR loop in the background (when required) gave us a directory full of relevant screen shots.

Now the job remaining was to publish them, for that lot of tools are available in opensource world, basically most of them are either python or php based, but we wanted some thing which required less/no installation (extra) and could run on Apache, so we choose DAlbum (php based). Selecting this tool was not enough as it required placing the images in to it's root directory and clicking or executing Reindex.php script. Here also we made some changes as doing it with the above mentioned script increased disk usage, it created 3 copies of each screen shot (1 for thumbnail, 2nd for full screen view and the 3rd for downloading).

We overcome this issue by creating our own script which did the same job but now the size of all the three images were defined by us, the script (bash) converted the screen shot in to three different sized imaged (using ImageMagic) and then placed them into the respective directory required by DAlbum.


Here is sample command we used to get the screen shots
import -window root -display $display $screenshotpath/screenshot$i.jpg

DAlbum looks like this


For more information on ImageMagic click here.
For more information on DAlbum click here.