puppet (header too long (OpenSSL::X509::CRLError)

             In an automated environment where new instances are added automatically and manged by puppet it is a great problem when the puppet master has some issues. It can act as a SPOF.
                   I happened as a accidental problem that puppet master had a 100% disk usage. As a result the requests from puppet clients of new instances were failing with 503 error. On checking the puppet master I could see the following error in puppet master error log.


=============================

Exception PhusionPassenger::UnknownError in PhusionPassenger::Rack::ApplicationSpawner (header too long (OpenSSL::X509::CRLError)) (process 598, thread #): ============================= 

            We have replaced passenger instead of the built in webrick for performance. Now checking the master there were no error. Accidentally when I tried to list out the certificates that are there in the host I got the following error. 
 ============================= 
puppetca --list --all err: Could not call list: header too long ============================= 

            Searching the forums I could see that this can happen if there were 0 byte certificate requests in /var/puppet/ssl/ca/requests or ( /var/lib/puppet/ssl/ca/requests ). In our case it was the /etc/puppet/ssl/ca/ca_crl.pem which was 0 byte. Removed the file and everything was back to normal. 

           It is quite a bad day when the master of automation gets involved in some kind of trouble.

Disable Notifications during Maintainence From Command Line.

We have lot of scripts which do automatic maintenance work during weekends. Eventhough the scripts are written to take care of errors it doesn't have a option to notify nagios that the maintenance work is taking place.

The person who is Oncall also gets frustrated seeing the alerts disturbing his weekend peace. He might even screw up the entire maintenance taking place.

Hence we needed the script to notify nagios that a maintenance is taking place and not to send out notifications.

We were using nagios3 as the monitoring service. The great command line utility curl came in handy here.

We use curl to send a POST request to the nagios admin interface emulating a user experience.


Disable notification to service.

curl -d "cmd_mod=2&cmd_typ=23&host=rayber01&service=MYSQL&btnSubmit=Commit" "https://rayber.blogspot.com/nagios/cgi-bin/cmd.cgi" --insecure -uray:testray

Host= rayber01 Hostname for which you need service notification disabled
-uray:testray = User authentication in nagios interface.




Enable Notification of service:

Once the maintenance work is complete we need to enable notification for the service.

curl -d "cmd_mod=2&cmd_typ=22&host=rayber01&service=MYSQL&btnSubmit=Commit" "https://rayber.blogspot.com/nagios/cgi-bin/cmd.cgi" --insecure -uray:testray


You can do a lot of other custom commands using curl. I use Firebug to translate the POST requests being send to the nagios admin interface.

You can completely disable notifications on a host as well..

Multicast Routing issues in Ganglia

Sometimes the Gmond process does not start and spews the following error.
==========================
gmond -d 10
udp_recv_channel mcast_join=10.16.101.81 mcast_if=NULL port=8664 bind=10.16.101.81
Error creating multicast server mcast_join=10.16.101.81 port=8664 mcast_if=NULL family='inet4'. Exiting.
==========================

This happens due to some multicast routing issues. I am not sure exactly what is causing this problem. The fix is to explicitly add a route.

============================

route add -host 239.2.11.71 dev eth0

=============================


Need to learn what is causing this problem though..

Using CCRB to manage multiple environments with same code base

Usually the code should be tested in a development environment before pushing the code to production. Automating test process is an important process in deployment. CCRB is a great tool to do this.


The code base is written in such a way that it can be deployed to development or production based on environment variables passed using the capistrano deployment script.

The development deployment initiates a CCRB build and testing process in the development cruisecontrol project which has the same code base. During this process the CCRB should be capable of invoking a development environment variable.

In comman setups we have the environmental variable 'development' and 'production' to differentiate the between production and development.

We add the following entries to cruise_config.rb to pass the 'development' environmental variables to the ccrb build.
============
ENV['env'] = 'development'
============

You can create a file named build_requested in project rootdir to initiate a build process.

Capistrano Using User Input

Capistrano does not accept ruby methods. Suppose I need to get user input I can't use gets.strip and it would spew Method not found error.

You can use the following method to get the user input in capistrano deploy scripts.
==============
puts "This is a critical code do you want to proceed (y/n)"
value = STDIN.gets[0..0] rescue nil
exit unless value == 'y' or value == 'Y'
===============

Capistrano is full and fast automation solution. Don't include too much of user interaction in that unless necessary.

Bash Execute a Remote Command as Sudo

Just difficult for me to remember this thing. This is a way to execute remote command which can executed only using sudo privileges.
===============
ssh -t testuser@testserver "/usr/bin/sudo sh -c w"
===============

Migrate Thunderbird email from Mac to Ubuntu

This is a simple step to migrate thunderbird mails from Mac OSX to Ubuntu.


Create the following directories in Ubuntu Desktop

cd /home/rayber
mkdir .thunderbird

Mount the mac HD

mount -t hfsplus /dev/sda2 /media/mac
Copy the data from mac homedir

cp -rpf /media/mac/Users/home/rayber/Library/Thunderbird/* .thunderbird/


You are now good to go. Start thunderbird and you should be able to start from the place where you stopped in Mac.