puppet (header too long (OpenSSL::X509::CRLError)

             In an automated environment where new instances are added automatically and manged by puppet it is a great problem when the puppet master has some issues. It can act as a SPOF.
                   I happened as a accidental problem that puppet master had a 100% disk usage. As a result the requests from puppet clients of new instances were failing with 503 error. On checking the puppet master I could see the following error in puppet master error log.


=============================

Exception PhusionPassenger::UnknownError in PhusionPassenger::Rack::ApplicationSpawner (header too long (OpenSSL::X509::CRLError)) (process 598, thread #): ============================= 

            We have replaced passenger instead of the built in webrick for performance. Now checking the master there were no error. Accidentally when I tried to list out the certificates that are there in the host I got the following error. 
 ============================= 
puppetca --list --all err: Could not call list: header too long ============================= 

            Searching the forums I could see that this can happen if there were 0 byte certificate requests in /var/puppet/ssl/ca/requests or ( /var/lib/puppet/ssl/ca/requests ). In our case it was the /etc/puppet/ssl/ca/ca_crl.pem which was 0 byte. Removed the file and everything was back to normal. 

           It is quite a bad day when the master of automation gets involved in some kind of trouble.

Disable Notifications during Maintainence From Command Line.

We have lot of scripts which do automatic maintenance work during weekends. Eventhough the scripts are written to take care of errors it doesn't have a option to notify nagios that the maintenance work is taking place.

The person who is Oncall also gets frustrated seeing the alerts disturbing his weekend peace. He might even screw up the entire maintenance taking place.

Hence we needed the script to notify nagios that a maintenance is taking place and not to send out notifications.

We were using nagios3 as the monitoring service. The great command line utility curl came in handy here.

We use curl to send a POST request to the nagios admin interface emulating a user experience.


Disable notification to service.

curl -d "cmd_mod=2&cmd_typ=23&host=rayber01&service=MYSQL&btnSubmit=Commit" "https://rayber.blogspot.com/nagios/cgi-bin/cmd.cgi" --insecure -uray:testray

Host= rayber01 Hostname for which you need service notification disabled
-uray:testray = User authentication in nagios interface.




Enable Notification of service:

Once the maintenance work is complete we need to enable notification for the service.

curl -d "cmd_mod=2&cmd_typ=22&host=rayber01&service=MYSQL&btnSubmit=Commit" "https://rayber.blogspot.com/nagios/cgi-bin/cmd.cgi" --insecure -uray:testray


You can do a lot of other custom commands using curl. I use Firebug to translate the POST requests being send to the nagios admin interface.

You can completely disable notifications on a host as well..