Push Thrift Metrics to Ganglia CDH4

 

 Add the following entries in /etc/hbase/conf/hadoop-metrics.properties or $HBASE_HOME/conf/hadoop-metrics.properties file

 

===========

thriftserver.class=org.apache.hadoop.metrics.ganglia.GangliaContext31

thriftserver.period=10

thriftserver.servers=<ganglia_multicast_address>:<ganglia_port>

===========

 

Restart hbase-thrift-server and you should see thrift metrics in ganglia interface.

No such device or address while trying to determine filesystem size on SSD in Centos 5

[root@02 ~]# mkfs.ext4 /dev/sdb1
mke4fs 1.41.12 (17-May-2010)
mkfs.ext4: No such device or address while trying to determine filesystem size

All the usual stuff by playing around with df -h, mount , cat /proc/swaps, /etc/mtab , lsof did not yeild any results.

Checking out some forums suggested that it is related to low level raid stuff and can be found by help of dmsetup and dmraid.

List device name

[root@02 ~]# dmsetup ls
ddf1_4c5349202020202010000411100010044711471181ddbcc4   (253, 0)

Get some more information about the device

[root@02 ~]# dmsetup info ddf1_4c5349202020202010000411100010044711471181ddbcc4
Name:              ddf1_4c5349202020202010000411100010044711471181ddbcc4
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        0
Event number:      0
Major, minor:      253, 0
Number of targets: 1
UUID: DMRAID-ddf1_4c5349202020202010000411100010044711471181ddbcc4

Discover all software raid devices in the system

[root@02 ~]# dmraid -r
/dev/sdb: ddf1, ".ddf1_disks", GROUP, ok, 123045888 sectors, data@ 0

Get the property of the raid disks

[root@02 ~]# dmraid -s
*** Group superset .ddf1_disks
--> Subset
name   : ddf1_4c5349202020202010000411100010044711471181ddbcc4
size   : 123045888
stride : 128
type   : stripe
status : ok
subsets: 0
devs   : 1
spares : 0

Tried to deactivate the raid device

[root@02 ~]# dmraid -an
ERROR: dos: partition address past end of RAID device
The dynamic shared library "libdmraid-events-ddf1.so" could not be loaded:
libdmraid-events-ddf1.so: cannot open shared object file: No such file or directory

Remove all raid devices metadata

[root@02 ~]# dmraid -r -E /dev/sdb
Do you really want to erase "ddf1" ondisk metadata on /dev/sdb ? [y/n] :y
ERROR: ddf1: seeking device "/dev/sdb" to 32779907366912
ERROR: writing metadata to /dev/sdb, offset 64023256576 sectors, size 0 bytes returned 0
ERROR: erasing ondisk metadata on /dev/sdb

Format the entire disk with zeros so that it does not get detected as raid device and cause further problem.

dd if=/dev/zero of=/dev/sdb

Centos 5 with ganglia and rrdcached

 

 As the number of metrics in your environment grows you start to see huge impact in system IO performance. At one stage the disk utilization stays at 100% and cpu spends a lot of time waiting for IO. At this time the IO exceeds the theoratical IO supported by the disk. We might go ahead and add fast 15K disks or raid10 array. The disk contention stays as long as we keep on expanding the cluster and adding new metrics.

 

 A simple solution would be to add the rrdcached layer in the middle. There are certain things to consider such as updating the rrdtool package and recompiling ganglia with the new rrdtool support.

 

 Just a step by step guide of doing it.

 

A. Steps for building and install rrdtool and ganglia.

 

1. uninstall existing rrdtool

yum -y install rrdtool rrdtool-perl

 

2. Download latest rrdtool

 

wget http://apt.sw.be/redhat/el5/en/x86_64/dag/RPMS/rrdtool-1.4.7-1.el5.rf.x86_64.rpm

wget http://apt.sw.be/redhat/el5/en/x86_64/dag/RPMS/perl-rrdtool-1.4.7-1.el5.rf.x86_64.rpm

wget http://apt.sw.be/redhat/el5/en/x86_64/dag/RPMS/rrdtool-devel-1.4.7-1.el5.rf.x86_64.rpm

 

3. Install all three in a single go. Otherwise it will show some weird perl dependency errors.

 

rpm -ivh rrdtool-1.4.7-1.el5.rf.x86_64.rpm rrdtool-devel-1.4.7-1.el5.rf.x86_64.rpm perl-rrdtool-1.4.7-1.el5.rf.x86_64.rpm

 

4. Get the ganglia source rpm 

 wget http://downloads.sourceforge.net/project/ganglia/ganglia%20monitoring%20core/3.4.0/ganglia-3.4.0-1.src.rpm?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fganglia%2Ffiles%2Fganglia%2520monitoring%2520core%2F3.4.0%2F&ts=1360839947&use_mirror=citylan

 

5. Install the dependencies required for building the rpm.

 

yum -y install libpng-devel libart_lgpl-devel python-devel libconfuse-devel pcre-devel freetype-devel

 

6. Build the rpm

 rpm -ivh ganglia-3.4.0-1.src.rpm

rpmbuild -ba /usr/src/redhat/SPECS/ganglia.spec

 

7. Install the new ganglia versions

 

rpm -ivh /usr/src/redhat/RPMS/x86_64/ganglia-* /usr/src/redhat/RPMS/x86_64/libganglia-3.4.0-1.x86_64.rpm

 

B . Configuring rrdcached to work with ganglia.

 

1. Since gmetad runs with ganglia user and rrdcached require access to write to rrd dir and apache needs access of same directory. Add ganglia to apache group.

usermod -a -G apache ganglia

 

2. correct the permissions

chown -R ganglia:apache /var/lib/ganglia/rrds/

 

3. Update rrdcached sysconfig startup options.

cat /etc/sysconfig/rrdcached

OPTIONS="rrdcached -p /tmp/rrdcached.pid -s apache -m 664 -l unix:/tmp/rrdcached.sock -s apache -m 777 -P FLUSH,STATS,HELP -l unix:/tmp/rrdcached.limited.sock -b /var/lib/ganglia/rrds -B"

RRDC_USER=ganglia

5. Update gmetad sysconfig file so that it knows the rrdcached socket information.

cat /etc/sysconfig/gmetad

RRDCACHED_ADDRESS="unix:/tmp/rrdcached.sock"

6. Update ganglia-web config information so that apache communicates with rrdcached daemon to fetch rrd information.

grep rrdcached_socket /var/www/html/gweb/conf_default.php

$conf['rrdcached_socket'] = "/tmp/rrdcached.sock";

7.  stop gmetad

service gmetad stop

8. start rrdcached

service rrdcached start

9 . start gmetad

service gmetad start

 If everything is fine then you should see graphs populating in ganglia frontend.

At the same time you'll see that the IO disk utilization is reduced awesomely :)

Nginx to hadoop hdfs with fluentd

Nginx => hadoop HDFS using Fluentd


        Fluentd is a json everywhere log collector. It transmits logs as json streams so that log processing can be easily managed and processed.

      Hadoop HDFS is a distributed filesystem which can be used to store any amount of logs and run mapreduce jobs for faster log processing.

 We will be using fluent-webhdfs-plugin to send logs over to httpfs interface


1. Install hadoop-httpfs package

        yum install hadoop-httpfs

2. Enable access to HDFS for httpfs user

vi /etc/hadoop/conf/core-site.xml

  <property>  
   <name>hadoop.proxyuser.httpfs.hosts</name>  
   <value>localhost,httpfshost></value>  
  </property>  
  <property>  
   <name>hadoop.proxyuser.httpfs.groups</name>  
   <value>*</value>  
  </property>  

 

3. Restart the hadoop cluster.

4. Start the hadoop-httpfs service

/etc/init.d/hadoop-httpfs start

5. Check whether it is working

 curl -i "http://<namenode>:14000?user.name=httpfs&op=homedir"  
 HTTP/1.1 200 OK  
 Server: Apache-Coyote/1.1  

6. Install treasure date td-agent in nginx servers and log-aggregator server

cat > /etc/yum.repos.d/treasuredate.repo
[treasuredata]
name=TreasureData
baseurl=http://packages.treasure-data.com/redhat/$basearch
gpgcheck=0

 yum install td-agent

7. Install fluentd and fluentd-plugin-webhdfs in log-aggregator host

gem install fluent-logger --no-ri --no-rdoc
/usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-webhdfs

8. Edit td-agent configuration in nginx server

vi /etc/td-agent/td-agent.conf

 # Tail the nginx logs associated with stats.slideshare.net  
 <source>  
  type tail  
  path /var/log/nginx/stats_access.log  
  format apache  
  tag stats.access  
  pos_file /var/log/td-agent/stats_access.pos  
 </source>  
 <match stats.access>  
  type forward  
  <server>  
   host <LOG AGGREGATOR NODE>  
   port 24224  
  </server>  
  retry_limit 5  
  <secondary>  
   type file  
   path /var/log/td-agent/stats_access.log  
  </secondary>  
 </match>  

Edit Nginx configuration to use apache log format.

   log_format main '$remote_addr - $remote_user [$time_local] "$request" '  
            '$status $body_bytes_sent "$http_referer" '  
            '"$http_user_agent"';  

9. Edit td-agent configuration in log aggregator server

 <source>  
  type forward  
  port 24224  
 </source>  
 <match stats.access>  
  type webhdfs  
  host <NAMENODE OR HTTPFS HOST>  
  port 14000  
  path /user/hdfs/stats_logs/stats_access.%Y%m%d_%H.log  
  httpfs true  
  username httpfsuser  
 </match>  

10. Start td-agent in log aggregator host

/etc/init.d/td-agent start

 * ensure that there are no errors in /var/log/td-agent/td-agent.log

11. Start td-agent in nginx servers

/etc/init.d/td-agent start
/etc/init.d/nginx restart

12. Check whether you can see the logs in HDFS

 sudo -u hdfsuser hdfs dfs -ls /user/hdfs/stats_logs/  
 Found 1 items  
 -rw-r--r--  3 httpfsuser group   17441 2012-10-12 01:10 /user/hdfsuser/stats_logs/stats_access.20121012_07.log  

That is all.. Now you have a log aggregation happening

Class org.apache.hadoop.thriftfs.NamenodePlugin not found

 While starting namenode you might come across this error.


Class org.apache.hadoop.thriftfs.NamenodePlugin not found



 In cdh4 it does not require a plug-in on the NameNode or DataNodes. Hence all the configuration related to that should be removed from namenode and datanode hdfs-site.xml

Extract Data from Master oplog and Restore it in another MongoDB server

Extract data from oplog in MongoDB and restore in another MongoDB server.





                            Recently I came across a problem where we have to do a lot of modifications in the mongodb server which will be having issues with the production database. We removed the replication between the master and slave and then did the operation in slave and then updated the data using the tool wordnik-oss tools.


                             Unfortunately we did not have a replica set and we had normal Master-Slave setup. While all the updates are happening in slave I need to keep track of the master data so that I can add it to the slave. For this I used a tool named mongodb-admin-utils in wordnik-oss https://github.com/wordnik/wordnik-oss.


Required software:


  1. Java and Git:
                  yum install java-1.6.0-sun-devel java-1.6.0-sun git


  2. Maven:
        recent version of wordnik-oss require maven 3


                  cd /usr/src
                   wgethttp://apache.techartifact.com/mirror/maven/binaries/apache-maven-3.0.4-bin.tar.gz
                    tar zxf apache-maven-3.0.4-bin.tar.gz


Building wordnik:


  1. Download and compile


                            git clone http://github.com/wordnik/wordnik-oss.git wordnik


  2. Compile and build
 
             In my case I only needed mongodb-admin-utils and hence I packaged only   that.


                         cd wordnik/modules/mongo-admin-utils
                        /usr/src/apache-maven-3.0.4/bin/mvn package


                       Once this is complete you can use mongo-admin-utils in the host.


Get Incremental oplog Backup from mongo master server




                cd wordnik/modules/mongo-admin-utils
               ./bin/run.sh com.wordnik.system.mongodb.IncrementalBackupUtil  -o /root/mongo -h mastermongodb


                                   /root/mongo => output directory where the oplog is stored.
                                   mastermongodb => mongodb master host.


** We can't use this tool in slave as there is no oplog in slave.


Replay the Data from the oplog to the database


          I had some problems in restoring data from backup and I had to add the following settings for the restore to work without any issues.

    ulimit -n 20000


           Added the following Java options in run.sh so that it does not fail with Out Of Memory (OOM ) erros.

JAVA_CONFIG_OPTIONS="-Xms5g -Xmx10g -XX:NewSize=2g -XX:MaxNewSize=2g -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:PermSize=2g -XX:MaxPermSize=2g"




Replay Command:


            ./bin/run.sh com.wordnik.system.mongodb.ReplayUtil -i /root/mongo -h localhost
                localhost => the mongodb server in which you want the data to be added.


         If you face any issues you can go ahead and file a issue in https://github.com/wordnik/wordnik-os . The developer is a awesome person and will help you sort out the issue.

Custom puppet master hostname error hostname was not match with the server certificate

 When you want to use custom hostname for puppet it shows the following error.
=============
err: Could not retrieve catalog from remote server: hostname was not match with the server certificate
warning: Not using cache on failed catalog
err: Could not retrieve catalog; skipping run
err: Could not send report: hostname was not match with the server certificate
==============


 In my case I wanted to use the default hostname "puppet" . Add the following entries to puppet master configuration file /etc/puppet/puppet.conf

 certname = puppet

stop puppet master; mv /var/lib/puppet /var/lib/puppet-bak

start puppet

 Ensure that the cert is loaded with the name you want.

==================
puppet cert print $(puppet master --configprint certname)|grep Subject

        Subject: CN=puppet
==================


If the CN field is showing your hostname then this would not work.


Now you can use puppet agents to connect using


puppet agent --test --server puppet








Ensure that you have entries in /etc/hosts for puppet master

Centos create puppet rpm package

 Centos create puppet RPM package.


    I usually upgrade puppet using rpm package and always love to stay on latest stable. The EPEL repo usually does not update the repositories as I want. Recently  I wanted to update to latest stable 2.7.6


 Ensure that you have rpm-build package installed.

yum -y install rpm-build

wget http://www.puppetlabs.com/downloads/puppet/puppet-2.7.6.tar.gz

tar zxf puppet-2.7.6.tar.gz

cp puppet-2.7.6.tar.gz /usr/src/rpm/SOURCES/puppet-2.7.6.tar.gz

rpmbuild -bb puppet-2.7.6/conf/redhat/puppet.spec

  The above command should provide you the new rpms

Wrote: /usr/src/rpm/RPMS/x86_64/puppet-2.7.6-1.x86_64.rpm
Wrote: /usr/src/rpm/RPMS/x86_64/puppet-server-2.7.6-1.x86_64.rpm


 While I was installing 2.7.6 I had issues that the rpmbuild failed with the following error.
==================
sed: can't read lib/puppet/network/http_server/mongrel.rb: No such file or directory
=================


 Replacing http_server with http in line number 79 in the puppet.spec file fixed the issue.

Puppet Dashboard On and external node using external database

 Install and configure puppet dashboard in a host other than puppet master and using an external DB server



 Installation.


package requirements:

ruby-augeas
ruby
ruby-shadow
ruby-rdoc
ruby-mysql
ruby-libs
rubygems
ruby-devel
rubygem
ruby-irb


Gem packages

gem update --system 1.3.6

gem install rack --no-ri --no-rdoc

gem install rake -v 0.9.2 --no-ri --no-rdoc




Download puppet dashboard

wget http://www.puppetlabs.com/downloads/dashboard/puppet-dashboard-1.2.2.tar.gz


tar zxf puppet-dashboard-1.2.2.tar.gz

mv puppet-dashboard puppet_dash
cd puppet_dash
cp config/database.yml.example config/database.yml
cp config/settings.yml.example config/settings.yml


edit the config/database.yml and update database settings.

Add the following entry as it will not there by default

under production settings:

production:
  database: puppetdash_prod
  username: dashboard
  password: yourpasswd
  encoding: utf8
  adapter: mysql
  host: puppethost

Update mysql privileges

create databases

rake RAILS_ENV=production db:create

create table structure

rake RAILS_ENV=production db:migrate


start the server

./script/server -e production


Enable reporting in slave and master


In each client puppet.conf add the following entry

[agent]
report = true



In the puppet master puppet.conf add the following entry


[master]
reports = http, store
reporturl = http://puppetdashboard:3000/reports/upload



Restart the puppet master and all the clients with the new configuration. Now the puppet dashboard should be populated with the new data.

Reliance and Huawei e173 3G datacard on Ubuntu

 Recently I migrated to Reliance 3G services. The installation disk did not come with any applet for linux systems. Hence I had a little trouble connecting it to the internet using default network manager. You can use rcomnet APN.

 I thought of getting a applet like the one they have for windows and mac. After some searching I came across a blog post which covered it.


    Download the following installation file linux.zip ( DOWNLOAD LINK )



unzip linux.zip


cd linux

chmod +x install

./install


Once it is done then it will install the drivers required for the device to be operational and also installs a applet for movistar mobile network which is widely used in europe.


 The applet will automatically pop-up when you insert the datacard. Click connect. Change the language to english from the menu and you should be able to use a english interface.

puppet (header too long (OpenSSL::X509::CRLError)

             In an automated environment where new instances are added automatically and manged by puppet it is a great problem when the puppet master has some issues. It can act as a SPOF.
                   I happened as a accidental problem that puppet master had a 100% disk usage. As a result the requests from puppet clients of new instances were failing with 503 error. On checking the puppet master I could see the following error in puppet master error log.


=============================

Exception PhusionPassenger::UnknownError in PhusionPassenger::Rack::ApplicationSpawner (header too long (OpenSSL::X509::CRLError)) (process 598, thread #): ============================= 

            We have replaced passenger instead of the built in webrick for performance. Now checking the master there were no error. Accidentally when I tried to list out the certificates that are there in the host I got the following error. 
 ============================= 
puppetca --list --all err: Could not call list: header too long ============================= 

            Searching the forums I could see that this can happen if there were 0 byte certificate requests in /var/puppet/ssl/ca/requests or ( /var/lib/puppet/ssl/ca/requests ). In our case it was the /etc/puppet/ssl/ca/ca_crl.pem which was 0 byte. Removed the file and everything was back to normal. 

           It is quite a bad day when the master of automation gets involved in some kind of trouble.

Disable Notifications during Maintainence From Command Line.

We have lot of scripts which do automatic maintenance work during weekends. Eventhough the scripts are written to take care of errors it doesn't have a option to notify nagios that the maintenance work is taking place.

The person who is Oncall also gets frustrated seeing the alerts disturbing his weekend peace. He might even screw up the entire maintenance taking place.

Hence we needed the script to notify nagios that a maintenance is taking place and not to send out notifications.

We were using nagios3 as the monitoring service. The great command line utility curl came in handy here.

We use curl to send a POST request to the nagios admin interface emulating a user experience.


Disable notification to service.

curl -d "cmd_mod=2&cmd_typ=23&host=rayber01&service=MYSQL&btnSubmit=Commit" "https://rayber.blogspot.com/nagios/cgi-bin/cmd.cgi" --insecure -uray:testray

Host= rayber01 Hostname for which you need service notification disabled
-uray:testray = User authentication in nagios interface.




Enable Notification of service:

Once the maintenance work is complete we need to enable notification for the service.

curl -d "cmd_mod=2&cmd_typ=22&host=rayber01&service=MYSQL&btnSubmit=Commit" "https://rayber.blogspot.com/nagios/cgi-bin/cmd.cgi" --insecure -uray:testray


You can do a lot of other custom commands using curl. I use Firebug to translate the POST requests being send to the nagios admin interface.

You can completely disable notifications on a host as well..

Multicast Routing issues in Ganglia

Sometimes the Gmond process does not start and spews the following error.
==========================
gmond -d 10
udp_recv_channel mcast_join=10.16.101.81 mcast_if=NULL port=8664 bind=10.16.101.81
Error creating multicast server mcast_join=10.16.101.81 port=8664 mcast_if=NULL family='inet4'. Exiting.
==========================

This happens due to some multicast routing issues. I am not sure exactly what is causing this problem. The fix is to explicitly add a route.

============================

route add -host 239.2.11.71 dev eth0

=============================


Need to learn what is causing this problem though..

Using CCRB to manage multiple environments with same code base

Usually the code should be tested in a development environment before pushing the code to production. Automating test process is an important process in deployment. CCRB is a great tool to do this.


The code base is written in such a way that it can be deployed to development or production based on environment variables passed using the capistrano deployment script.

The development deployment initiates a CCRB build and testing process in the development cruisecontrol project which has the same code base. During this process the CCRB should be capable of invoking a development environment variable.

In comman setups we have the environmental variable 'development' and 'production' to differentiate the between production and development.

We add the following entries to cruise_config.rb to pass the 'development' environmental variables to the ccrb build.
============
ENV['env'] = 'development'
============

You can create a file named build_requested in project rootdir to initiate a build process.

Capistrano Using User Input

Capistrano does not accept ruby methods. Suppose I need to get user input I can't use gets.strip and it would spew Method not found error.

You can use the following method to get the user input in capistrano deploy scripts.
==============
puts "This is a critical code do you want to proceed (y/n)"
value = STDIN.gets[0..0] rescue nil
exit unless value == 'y' or value == 'Y'
===============

Capistrano is full and fast automation solution. Don't include too much of user interaction in that unless necessary.

Bash Execute a Remote Command as Sudo

Just difficult for me to remember this thing. This is a way to execute remote command which can executed only using sudo privileges.
===============
ssh -t testuser@testserver "/usr/bin/sudo sh -c w"
===============

Migrate Thunderbird email from Mac to Ubuntu

This is a simple step to migrate thunderbird mails from Mac OSX to Ubuntu.


Create the following directories in Ubuntu Desktop

cd /home/rayber
mkdir .thunderbird

Mount the mac HD

mount -t hfsplus /dev/sda2 /media/mac
Copy the data from mac homedir

cp -rpf /media/mac/Users/home/rayber/Library/Thunderbird/* .thunderbird/


You are now good to go. Start thunderbird and you should be able to start from the place where you stopped in Mac.

SSH with Empty Password and no ssh keys

This is a guide to zero security in linux. It is applicable only in places where security is not a threat and there is no threat from external networks.

The following options would allow a system running ssh to have a user with empty password so that you can use this user to login without any password or ssh keys.


I am adding a test user for this purpose.

useradd noneknowme
passwd -d noneknowme

Configure ssh server to allow empty passwords.

Edit the following line in /etc/ssh/sshd_config
================
PermitEmptyPasswords yes
================

Restart sshd using /etc/init.d/sshd restart

Now you should be able to access the host with ssh with username noneknowme without any issues.
===============
ssh -l noneknowme test
[noneknowme@test ~]$
================

Don't try this . I am using this as a reference. Thanks to linuxquestions.org

script to check swap usage by process ( approximate )

The following command returns the total swap usage.

swapon -s

The following script would obtain the total MB of swap used obtained from proc filesystem and sum it up.

for pid in `ps -ef|grep -i java| grep -v grep|awk '{print $2}'`; do echo -n "Pid: $pid "; cat /proc/$pid/smaps |grep -i swap| awk '{SUM += $2} END {print "SUM: " SUM " kB (" SUM/1024 " MB)"}'; done

Thanks to linuxquestions.org

Zero Padding in Bash

I was wondering what is the best way to do "Zero Padding" in Bash. Suppose you need to iterate through a for loop and you need 01, 02, 03 sequence instead of 1, 2,3 then that is Zero padding.

Some of the methods I could get from the internet.

1. Using printf

for ((i=2;i<16;i++)); do printf "%02d" $i; done

2. Using seq

There were couple of solutions when you use seq

* seq -f '%02g' 1 15

* seq -w 1 15


Enjoy being fast using bash..