Monday, September 26, 2011

puppet (header too long (OpenSSL::X509::CRLError)




             In an automated environment where new instances are added automatically and manged by puppet it is a great problem when the puppet master has some issues. It can act as a SPOF.
                   I happened as a accidental problem that puppet master had a 100% disk usage. As a result the requests from puppet clients of new instances were failing with 503 error. On checking the puppet master I could see the following error in puppet master error log.


=============================

Exception PhusionPassenger::UnknownError in PhusionPassenger::Rack::ApplicationSpawner (header too long (OpenSSL::X509::CRLError)) (process 598, thread #): ============================= 

            We have replaced passenger instead of the built in webrick for performance. Now checking the master there were no error. Accidentally when I tried to list out the certificates that are there in the host I got the following error. 
 ============================= 
puppetca --list --all err: Could not call list: header too long ============================= 

            Searching the forums I could see that this can happen if there were 0 byte certificate requests in /var/puppet/ssl/ca/requests or ( /var/lib/puppet/ssl/ca/requests ). In our case it was the /etc/puppet/ssl/ca/ca_crl.pem which was 0 byte. Removed the file and everything was back to normal. 

           It is quite a bad day when the master of automation gets involved in some kind of trouble.

3 comments:

  1. Just want to say thanks for sharing the info here. Exactly what the case was for me.

    Thanks again!

    ReplyDelete
  2. Same thing here. Cleaning up the disk didn't fix it for me. Had to clean out a few 0 byte files in /var/lib/puppet/ssl/ca/requests/ . Restart wasn't required. Thanks for the help!

    ReplyDelete
  3. I have read your blog its very attractive and impressive. I like it your blog.
    appvn app

    ReplyDelete