Extract Data from Master oplog and Restore it in another MongoDB server

Extract data from oplog in MongoDB and restore in another MongoDB server.





                            Recently I came across a problem where we have to do a lot of modifications in the mongodb server which will be having issues with the production database. We removed the replication between the master and slave and then did the operation in slave and then updated the data using the tool wordnik-oss tools.


                             Unfortunately we did not have a replica set and we had normal Master-Slave setup. While all the updates are happening in slave I need to keep track of the master data so that I can add it to the slave. For this I used a tool named mongodb-admin-utils in wordnik-oss https://github.com/wordnik/wordnik-oss.


Required software:


  1. Java and Git:
                  yum install java-1.6.0-sun-devel java-1.6.0-sun git


  2. Maven:
        recent version of wordnik-oss require maven 3


                  cd /usr/src
                   wgethttp://apache.techartifact.com/mirror/maven/binaries/apache-maven-3.0.4-bin.tar.gz
                    tar zxf apache-maven-3.0.4-bin.tar.gz


Building wordnik:


  1. Download and compile


                            git clone http://github.com/wordnik/wordnik-oss.git wordnik


  2. Compile and build
 
             In my case I only needed mongodb-admin-utils and hence I packaged only   that.


                         cd wordnik/modules/mongo-admin-utils
                        /usr/src/apache-maven-3.0.4/bin/mvn package


                       Once this is complete you can use mongo-admin-utils in the host.


Get Incremental oplog Backup from mongo master server




                cd wordnik/modules/mongo-admin-utils
               ./bin/run.sh com.wordnik.system.mongodb.IncrementalBackupUtil  -o /root/mongo -h mastermongodb


                                   /root/mongo => output directory where the oplog is stored.
                                   mastermongodb => mongodb master host.


** We can't use this tool in slave as there is no oplog in slave.


Replay the Data from the oplog to the database


          I had some problems in restoring data from backup and I had to add the following settings for the restore to work without any issues.

    ulimit -n 20000


           Added the following Java options in run.sh so that it does not fail with Out Of Memory (OOM ) erros.

JAVA_CONFIG_OPTIONS="-Xms5g -Xmx10g -XX:NewSize=2g -XX:MaxNewSize=2g -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:PermSize=2g -XX:MaxPermSize=2g"




Replay Command:


            ./bin/run.sh com.wordnik.system.mongodb.ReplayUtil -i /root/mongo -h localhost
                localhost => the mongodb server in which you want the data to be added.


         If you face any issues you can go ahead and file a issue in https://github.com/wordnik/wordnik-os . The developer is a awesome person and will help you sort out the issue.

Custom puppet master hostname error hostname was not match with the server certificate

 When you want to use custom hostname for puppet it shows the following error.
=============
err: Could not retrieve catalog from remote server: hostname was not match with the server certificate
warning: Not using cache on failed catalog
err: Could not retrieve catalog; skipping run
err: Could not send report: hostname was not match with the server certificate
==============


 In my case I wanted to use the default hostname "puppet" . Add the following entries to puppet master configuration file /etc/puppet/puppet.conf

 certname = puppet

stop puppet master; mv /var/lib/puppet /var/lib/puppet-bak

start puppet

 Ensure that the cert is loaded with the name you want.

==================
puppet cert print $(puppet master --configprint certname)|grep Subject

        Subject: CN=puppet
==================


If the CN field is showing your hostname then this would not work.


Now you can use puppet agents to connect using


puppet agent --test --server puppet








Ensure that you have entries in /etc/hosts for puppet master

Centos create puppet rpm package

 Centos create puppet RPM package.


    I usually upgrade puppet using rpm package and always love to stay on latest stable. The EPEL repo usually does not update the repositories as I want. Recently  I wanted to update to latest stable 2.7.6


 Ensure that you have rpm-build package installed.

yum -y install rpm-build

wget http://www.puppetlabs.com/downloads/puppet/puppet-2.7.6.tar.gz

tar zxf puppet-2.7.6.tar.gz

cp puppet-2.7.6.tar.gz /usr/src/rpm/SOURCES/puppet-2.7.6.tar.gz

rpmbuild -bb puppet-2.7.6/conf/redhat/puppet.spec

  The above command should provide you the new rpms

Wrote: /usr/src/rpm/RPMS/x86_64/puppet-2.7.6-1.x86_64.rpm
Wrote: /usr/src/rpm/RPMS/x86_64/puppet-server-2.7.6-1.x86_64.rpm


 While I was installing 2.7.6 I had issues that the rpmbuild failed with the following error.
==================
sed: can't read lib/puppet/network/http_server/mongrel.rb: No such file or directory
=================


 Replacing http_server with http in line number 79 in the puppet.spec file fixed the issue.

Puppet Dashboard On and external node using external database

 Install and configure puppet dashboard in a host other than puppet master and using an external DB server



 Installation.


package requirements:

ruby-augeas
ruby
ruby-shadow
ruby-rdoc
ruby-mysql
ruby-libs
rubygems
ruby-devel
rubygem
ruby-irb


Gem packages

gem update --system 1.3.6

gem install rack --no-ri --no-rdoc

gem install rake -v 0.9.2 --no-ri --no-rdoc




Download puppet dashboard

wget http://www.puppetlabs.com/downloads/dashboard/puppet-dashboard-1.2.2.tar.gz


tar zxf puppet-dashboard-1.2.2.tar.gz

mv puppet-dashboard puppet_dash
cd puppet_dash
cp config/database.yml.example config/database.yml
cp config/settings.yml.example config/settings.yml


edit the config/database.yml and update database settings.

Add the following entry as it will not there by default

under production settings:

production:
  database: puppetdash_prod
  username: dashboard
  password: yourpasswd
  encoding: utf8
  adapter: mysql
  host: puppethost

Update mysql privileges

create databases

rake RAILS_ENV=production db:create

create table structure

rake RAILS_ENV=production db:migrate


start the server

./script/server -e production


Enable reporting in slave and master


In each client puppet.conf add the following entry

[agent]
report = true



In the puppet master puppet.conf add the following entry


[master]
reports = http, store
reporturl = http://puppetdashboard:3000/reports/upload



Restart the puppet master and all the clients with the new configuration. Now the puppet dashboard should be populated with the new data.

Reliance and Huawei e173 3G datacard on Ubuntu

 Recently I migrated to Reliance 3G services. The installation disk did not come with any applet for linux systems. Hence I had a little trouble connecting it to the internet using default network manager. You can use rcomnet APN.

 I thought of getting a applet like the one they have for windows and mac. After some searching I came across a blog post which covered it.


    Download the following installation file linux.zip ( DOWNLOAD LINK )



unzip linux.zip


cd linux

chmod +x install

./install


Once it is done then it will install the drivers required for the device to be operational and also installs a applet for movistar mobile network which is widely used in europe.


 The applet will automatically pop-up when you insert the datacard. Click connect. Change the language to english from the menu and you should be able to use a english interface.

puppet (header too long (OpenSSL::X509::CRLError)

             In an automated environment where new instances are added automatically and manged by puppet it is a great problem when the puppet master has some issues. It can act as a SPOF.
                   I happened as a accidental problem that puppet master had a 100% disk usage. As a result the requests from puppet clients of new instances were failing with 503 error. On checking the puppet master I could see the following error in puppet master error log.


=============================

Exception PhusionPassenger::UnknownError in PhusionPassenger::Rack::ApplicationSpawner (header too long (OpenSSL::X509::CRLError)) (process 598, thread #): ============================= 

            We have replaced passenger instead of the built in webrick for performance. Now checking the master there were no error. Accidentally when I tried to list out the certificates that are there in the host I got the following error. 
 ============================= 
puppetca --list --all err: Could not call list: header too long ============================= 

            Searching the forums I could see that this can happen if there were 0 byte certificate requests in /var/puppet/ssl/ca/requests or ( /var/lib/puppet/ssl/ca/requests ). In our case it was the /etc/puppet/ssl/ca/ca_crl.pem which was 0 byte. Removed the file and everything was back to normal. 

           It is quite a bad day when the master of automation gets involved in some kind of trouble.

Disable Notifications during Maintainence From Command Line.

We have lot of scripts which do automatic maintenance work during weekends. Eventhough the scripts are written to take care of errors it doesn't have a option to notify nagios that the maintenance work is taking place.

The person who is Oncall also gets frustrated seeing the alerts disturbing his weekend peace. He might even screw up the entire maintenance taking place.

Hence we needed the script to notify nagios that a maintenance is taking place and not to send out notifications.

We were using nagios3 as the monitoring service. The great command line utility curl came in handy here.

We use curl to send a POST request to the nagios admin interface emulating a user experience.


Disable notification to service.

curl -d "cmd_mod=2&cmd_typ=23&host=rayber01&service=MYSQL&btnSubmit=Commit" "https://rayber.blogspot.com/nagios/cgi-bin/cmd.cgi" --insecure -uray:testray

Host= rayber01 Hostname for which you need service notification disabled
-uray:testray = User authentication in nagios interface.




Enable Notification of service:

Once the maintenance work is complete we need to enable notification for the service.

curl -d "cmd_mod=2&cmd_typ=22&host=rayber01&service=MYSQL&btnSubmit=Commit" "https://rayber.blogspot.com/nagios/cgi-bin/cmd.cgi" --insecure -uray:testray


You can do a lot of other custom commands using curl. I use Firebug to translate the POST requests being send to the nagios admin interface.

You can completely disable notifications on a host as well..