Monday, September 26, 2011

puppet (header too long (OpenSSL::X509::CRLError)

             In an automated environment where new instances are added automatically and manged by puppet it is a great problem when the puppet master has some issues. It can act as a SPOF.
                   I happened as a accidental problem that puppet master had a 100% disk usage. As a result the requests from puppet clients of new instances were failing with 503 error. On checking the puppet master I could see the following error in puppet master error log.


Exception PhusionPassenger::UnknownError in PhusionPassenger::Rack::ApplicationSpawner (header too long (OpenSSL::X509::CRLError)) (process 598, thread #): ============================= 

            We have replaced passenger instead of the built in webrick for performance. Now checking the master there were no error. Accidentally when I tried to list out the certificates that are there in the host I got the following error. 
puppetca --list --all err: Could not call list: header too long ============================= 

            Searching the forums I could see that this can happen if there were 0 byte certificate requests in /var/puppet/ssl/ca/requests or ( /var/lib/puppet/ssl/ca/requests ). In our case it was the /etc/puppet/ssl/ca/ca_crl.pem which was 0 byte. Removed the file and everything was back to normal. 

           It is quite a bad day when the master of automation gets involved in some kind of trouble.