In an automated environment where new instances are added automatically and manged by puppet it is a great problem when the puppet master has some issues. It can act as a SPOF.
I happened as a accidental problem that puppet master had a 100% disk usage. As a result the requests from puppet clients of new instances were failing with 503 error.
On checking the puppet master I could see the following error in puppet master error log.
=============================
Exception PhusionPassenger::UnknownError in PhusionPassenger::Rack::ApplicationSpawner (header too long (OpenSSL::X509::CRLError)) (process 598, thread #
):
=============================
We have replaced passenger instead of the built in webrick for performance. Now checking the master there were no error. Accidentally when I tried to list out the certificates that are there in the host I got the following error.
=============================
puppetca --list --all
err: Could not call list: header too long
=============================
Searching the forums I could see that this can happen if there were 0 byte certificate requests in /var/puppet/ssl/ca/requests or ( /var/lib/puppet/ssl/ca/requests ). In our case it was the /etc/puppet/ssl/ca/ca_crl.pem which was 0 byte. Removed the file and everything was back to normal.
It is quite a bad day when the master of automation gets involved in some kind of trouble.
Just want to say thanks for sharing the info here. Exactly what the case was for me.
ReplyDeleteThanks again!
Same thing here. Cleaning up the disk didn't fix it for me. Had to clean out a few 0 byte files in /var/lib/puppet/ssl/ca/requests/ . Restart wasn't required. Thanks for the help!
ReplyDeleteI have read your blog its very attractive and impressive. I like it your blog.
ReplyDeleteappvn app
Great Article. Thank you for sharing! Really an awesome post for every one.
ReplyDeleteBig Data Based Improved Data Acquisition and Storage System for Designing Industrial Data Platform Project For CSE
Compressed Sensing and Its Applications in Risk Assessment for Internet Supply Chain Finance Under Big Data Project For CSE
Exploring behavioral heterogeneities of elementary school students’ commute mode choices through the urban travel big data of Beijing, China Project For CSE
FTLADS Object Logging based Fault Tolerant Big Data Transfer System using Layout Aware Data Scheduling Project For CSE
Intelligent Big Data Summarization for Rare Anomaly Detection Project For CSE
Algorithm A New Probabilistic ProcessLearning Approach for Big Data in Healthcare Project For CSE