Wednesday, December 8, 2010

Backup mails in Thunderbird from Outlook

It has been a hard time working with Outlook and I was worried that it could crash at any time. I thought of backing up all the mails to thunderbird in another linux box. Reading the documentation I couldn't figure out any method using which we can import the .pst files to the thunderbird client.

Here is what I did to do the migration. Initially I took the backup from the outlook

Outlook -> File -> Import or Export -> Export to a file -> Personal Folder File (.pst )

Now it might take some time to take a complete backup. At the end we will get a big backup.pst file.


Transfer the file over to the Linux Desktop.

Since there is no way to import the .pst file directly to thunderbird. We might have to convert to mbox format to get this done. There are many tools to do that. In my case I used readpst.

1. Install Readpst

Readpst is with the package libpst.

yum -y install libpst

2. Convert the .pst file to mbox based format.

mkdir mail

readpst -o mail backup.pst


Now that we have mbox format of all the folders.

3. Install MboxImport utility in thunderbird.

Download the xpi from http://nic-nac-project.de/~kaosmos/ImportExportTools-2.4.1.xpi

Thunderbird-> Tools -> Addons-> extensions -> Install
Select the xpi file that you downloaded earlier and install it. Restart tomcat for the new extensions to take effect.

4. Import the Mbox mails to thunderbird.

Now that we have the mbox based mails in mails directory we can import the mails and account to thunderbird.

Thunderbird -> Tools -> ImportExportTools -> Import Mbox File -> Import entire directory (including sub directory)

Now you can select the "mails" directory in which we saved the mbox format files.


It might take some time for the import to happen. Now that you have a backup..Don't worry about old mails..

Monday, September 6, 2010

Add a Recovery Disk to KVM and Recover the VM

There might be situations when you see a corrupted File system in KVM VM. In such situation you can add a ISO rescue cd and fix the file system errors.

For example I have a VM named "centos". Take the VM configuration file /etc/libvirt/qemu/centos.xml

Add the following lines in the configuration file above the disk image of the VM.









Then execute the command virsh and enter the following command.

virsh # define /etc/libvirt/qemu/centos.xml
Domain centos defined from /etc/libvirt/qemu/centos.xml

virsh # start centos
Domain centos started


During booting process press f12 so that you get the booting option and select CDROM. Now that the VM boots with the ISO image and you can do the filesystem recovery process.

fsck -y /dev/mapper/VolGroup00-LogVol00

Reboot the VM and ensure that you are able to access the VM without any file system errors.

Friday, August 6, 2010

Enable GPRS with Vodafone INDIA in Motorolo Milestone Android device.

Don't ever try to get help from the Vodafone Customer care. I searched google to get the following working settings. I use vodafone internet as they provide 2GB limited internet access for 95 RS which is quite affordable.

1. Ensure that VMC is activated.

make sure  that vmc is activated send a sms ACT VMC to 144

2. Configure the following settings..


This is where u find the settings in milestone:
Settings->Wireless  & networks->Mobile networks->Access point names

 Here are the settings:



Add new APN.

Homepage: Any ****.

Connection Name: Any ****
Access Point Name: www
User Name:Leave it.

Password: Leave it.
Authentication: none.
Proxy: None

Port Number: 0

Leave rest of the settings as they are - in default state. n enjoy.

I guess no more calls to vodafone cc. -

Thursday, May 6, 2010

KVM Virtualization in Fedora

WHY KVM

KVM is a full virtualization technology which uses the linux kernel for efficient hardware virtualization.

It consists of a loadable kernel module, kvm.ko, that provides the core virtualization infrastructure and a processor specific module, kvm-intel.ko or kvm-amd.ko. KVM also requires a modified QEMU although work is underway to get the required changes upstream.

Using KVM, one can run multiple virtual machines running unmodified Linux or Windows images. Each virtual machine has private virtualized hardware: a network card, disk, graphics adapter, etc.

* Ensure that your processor support virtualization

[root@fedora ~]# egrep '(vmx|svm)' --color=always /proc/cpuinfo
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc arch_perfmon pebs bts aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 xsave lahf_lm tpr_shadow vnmi flexpriority
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc arch_perfmon pebs bts aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 xsave lahf_lm tpr_shadow vnmi flexpriority

* Install the required kvm packages

yum install kvm qemu libvirt python-virtinst

qemu: we use qemu for processor emulation. We use full system emulation. QEMU emulates a full system (for example a PC), including one or several processors and various peripherals. It can be used to launch different Operating Systems without rebooting the PC or to debug system code.

libvirt: It is a tool to interact with virtualization capabilities of linux kernel.


In order to use Qemu support in libvirt we have to use the libvirt daemon.

/etc/init.d/libvirtd start

Qemu communicates with libvirt daemon using URIS.


So to connect to the daemon, one of two different URIs is used:

* qemu:///system connects to a system mode daemon.
* qemu:///session connects to a session mode daemon.

Inorder to check whether they communicate without any issues. Check the following command and output.

[root@fedora ~]# virsh -c qemu:///system list
Id Name State
----------------------------------


Usually all the documents mention to use a bridged setup so that internet is available to the hosts. In my setup I configured iptables with SNAT so that the VMS can communicate to the external network.

Install Virt Manager:

Virt Manager provides you a GUI to manage the VMS.

yum install virt-manager


Creating a centos VM.

virt-install --connect qemu:///system -n centos5 -r 512 --vcpus=2 -f /opt/kvm/centos5.qcow2 -s 12 -c CentOS-5.4-i386-netinstall.iso --vnc --noautoconsole --os-type linux --os-variant rhel5.4 --accelerate --hvm


-r 512 = RAM
--vcpus = virtual cpus
-f /opt/kvm/centos5.qcow2 = Storage file used to store VM data
-c ISO file to install the VM from
--os-variant = You can find the OS - Variant list by using man virt-install

If you have additional partitions then you can use it as a LVM disk and configure individual LVM disks for individual hosts.

You can use virt-manager to proceed with the rest of the installation process. You can also use vncviewer to proceed with the installation

yum install vnc
vncviewer localhost:0 # If you are having a single host.

Suppose that you are having multiple hosts then you find the vnc ip and port and then connect using vncviewer

[root@fedora ~]# virsh vncdisplay centos5
:1

root@fedora ~]$ vncviewer localhost:1

TigerVNC Viewer for X version 1.0.0 - built Oct 26 2009 10:57:15
Copyright (C) 2002-2005 RealVNC Ltd.
Copyright (C) 2000-2006 TightVNC Group
Copyright (C) 2004-2009 Peter Astrand for Cendio AB
See http://www.tigervnc.org for information on TigerVNC.

Wed May 5 22:20:18 2010
CConn: connected to host localhost port 5901
CConnection: Server supports RFB protocol version 3.8
CConnection: Using RFB protocol version 3.8
TXImage: Using default colormap and visual, TrueColor, depth 24.
CConn: Using pixel format depth 24 (32bpp) little-endian rgb888
CConn: Using Tight encoding







Configuring Network for the VM.

In my case virt-manager uses the ip range 192.168.122.0/24 which is default. You can change the default by running the following commands and modifying the entries.
virsh net-edit default
virsh net-destroy default
virsh net-start default


I used SNAT based forwarding. You can also use MASQUERADE.

My eth0 interface ip was 192.168.1.221 and hence my iptables command was

iptables -t nat -A POSTROUTING -s 192.168.122.0/24 -j SNAT --to 192.168.1.221

echo 1 >/proc/sys/net/ipv4/ip_forward



Now you should be able to connect to external network from VM.


This seems to be the future to virtualization as we are using the kernel to manage the hardware virtualization which would improve effeciency of the system.

Monday, March 29, 2010

Adding HTTP Authentication to Tomcat Container

I used the following method to secure the entire application. We use .htaccess based authentication setup in apache. I wanted a same sort of setup in tomcat instance.

Initially I added a user in the $catalina_home/conf/tomcat-users.xml







Then I add the security constraint in the web.xml of the application. In my case it was the default application ROOT. Hence I added the following entries in $catalina_home/webapps/ROOT/WEB-INF/web.xml





Entire Application
/*



agroup





BASIC
Authenticate yourself





The role that is required to log in to the Manager Application

agroup




Hence all the users under the security role "agroup" will be able to access the applicationn.

Hope this helps some one somewhere..

Thursday, March 18, 2010

Bash Script to send email using smtp authentiation and bash file descriptors

Sometimes we require bash script to perform some operation and use a remote mail server to send mails. In this instance we require smtp authentication for the remote server to allow this operation.

The script was copied from the link http://linuxshellaccount.blogspot.com/2008/04/shell-script-to-send-mail-using-bash.html which describes networking with the help of file descriptors.

I just added a smtp authentication part.

Please ensure that others do not have access to your bash history for security reasons.

Usage

sh mail.sh report@yourserver.com youremail@gmail.com yourserver.com yourmailserver.com filecontainingmessage email@yourmailserver.com youremailpassword





SCRIPT CONTENTS:


mail.sh




#!/bin/bash

#
# mail.sh
#
# 2008 - Mike Golvach - eggi@comcast.net
# 2010 - Rayber
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

if [ $# -ne 7 ]
then
echo "Usage: $0 FromAdress ToAdress Domain MailServer MailTextFile AuthEmail AuthPass"
exit 1
fi

from=$1
to=$2
domain=$3
mailserver=$4
mailtext=$5
authemail=`echo $6|openssl enc -base64|awk 'sub("..$", "")'`
authpass=`echo $7|openssl enc -base64|awk 'sub("..$", "")'`

if [ ! -f $mailtext ]
then
echo "Cannot find your mail text file. Exiting..."
exit 1
fi

exec 9<>/dev/tcp/$mailserver/25
echo "HELO $domain" >&9
read -r temp <&9
echo "$temp"
echo "auth login" >&9
read -r temp <&9
echo "$authemail" >&9
read -r temp <&9
echo "$authpass" >&9
read -r temp <&9
echo "Mail From: $from" >&9
read -r temp <&9
echo "$temp"
echo "Rcpt To: $to" >&9
read -r temp <&9
echo "$temp"
echo "Data" >&9
read -r temp <&9
echo "$temp"
cat $mailtext >&9
echo "." >&9
read -r temp <&9
echo "$temp"
echo "quit" >&9
read -r temp <&9
echo "$temp"
9>&-
9<&-
echo "All Done Sending Email. See above for errors"
exit 0



When you run the script you should get a output similar to the following output.

235 2.7.0 Authentication successful
250 2.1.0 Ok
250 2.1.5 Ok
354 End data with .
250 2.0.0 Ok

Tuesday, March 9, 2010

Remove or Disable svn authentication data

When you issue a svn co statement the authentication gets cached in the home directory of the user.

svn co http://192.168.1.13/repo
Authentication realm: Subversion Repository
Username: test
Password for 'test':

The authentication information gets stored in a cache /root/.subversion/auth/svn.simple directory.


o remove cached data go to “.subversion/auth/svn.simple” folder and delete the particular file.
There will be key (K)-value (V) pairs. “username” and “svn:realmstring” together can identify the user.
So use for example:

grep servername ./*

to find the right file.

Disable caching by opening “config” file in “.subversion” folder and setting the values of “store-passwords” and “store-auth-creds” to “no” or use --no-auth-cache as command line argument.


/root/.subversion/config

Wednesday, February 3, 2010

Mysql Database size calculation in MB

These are some small queries to find the total disk space used by the mysql database.

1. Get the Size of all the tables as well are databases.

SELECT TABLE_SCHEMA AS 'Database', TABLE_NAME AS 'Table', CONCAT(ROUND(((DATA_LENGTH + INDEX_LENGTH - DATA_FREE) / 1024 / 1024),2)," Mb") AS Size FROM INFORMATION_SCHEMA.TABLES;


2. Get the size of specific tables in the database.

SELECT TABLE_SCHEMA AS 'Database', TABLE_NAME AS 'Table', CONCAT(ROUND(((DATA_LENGTH + INDEX_LENGTH - DATA_FREE) / 1024 / 1024),2)," MB") AS Size FROM INFORMATION_SCHEMA.TABLES where TABLE_SCHEMA like 'testdb';

Where testdb is the database for which you want to know the details.

3. Get the size of the entire DB

select CONCAT(sum(ROUND(((DATA_LENGTH + INDEX_LENGTH - DATA_FREE) / 1024 / 1024),2))," MB") AS Size FROM INFORMATION_SCHEMA.TABLES where table_schema like 'testdb';



Another query to display the above result would be

4. display the disk size of all the databases

SELECT table_schema "Data Base Name", sum( data_length + index_length ) / 1024 / 1024 "Data Base Size in MB"
FROM information_schema.TABLES GROUP BY table_schema ;








The 3'rd query would not give you a correct result for large database in AMAZON RDS at the time of writing. I had to use the 4'th query to obtain a approximate value.

Monday, January 11, 2010

Monitor Memcached status

It is quite a important task to manage and monitor the memcache caching system. There are quite a lot of options to carry out this operation.

1. Using Telnet Command Line interface

List of supported commands are available at

http://code.sixapart.com/svn/memcached/trunk/server/doc/protocol.txt



telnet localhost 11211



List of keys used

stats items


STAT items:5:number 2
STAT items:5:age 47
STAT items:5:evicted 0
STAT items:5:evicted_nonzero 0
STAT items:5:evicted_time 0
STAT items:5:outofmemory 0
STAT items:5:tailrepairs 0
END



2. Using memcached-tool

The memcached tool is available in the source directory .

/usr/src/memcached-1.4.4/scripts/memcached-tool

Stats

/usr/src/memcached-1.4.4/scripts/memcached-tool 127.0.0.1:11211 stats

Show slab information

/usr/src/memcached-1.4.4/scripts/memcached-tool 127.0.0.1:11211 display

Show keys and values

/usr/src/memcached-1.4.4/scripts/memcached-tool 127.0.0.1:11211 dump


3. Using the damemtop

A flexible 'top' like utility for viewing memcached clusters. You can use it to manage a interactive cluster status.

The utility is available in the source directory. /usr/src/memcached-1.4.4/scripts/damemtop


It require you to install the following perl modules

cpan AnyEvent Term::ReadKey YAML


Now you need the /etc/damemtop.yaml file with the following contents.


delay: 3
mode: t
top_mode:
sort_column: "hostname"
sort_order: "asc"
columns:
- hostname
- all_version
- all_fill_rate
- hit_rate
- evictions
- bytes_written
- "2:get_hits"
servers:
- 127.0.01:11211


In a distributed environment you can add all the servers in the cluster to the file and issue the following command.


/usr/src/memcached-1.4.4/scripts/damemtop


That would give you a ideal monitoring setup..

Sunday, January 10, 2010

Install Memcached caching sytem in centos from source

memcached is a high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load.

It is mainly used in MYSQL + PHP or MYSQL + JAVA applications which are quite database intensive.

Install LIBEVENT


yum -y install libevent libevent-devel


Download Source

wget http://memcached.googlecode.com/files/memcached-1.4.4.tar.gz
tar zxf memcached-1.4.4.tar.gz
cd memcached-1.4.4
./configure && make && make install



Test the installation


memcached -m 256 -u nobody -vv
slab class 1: chunk size 80 perslab 13107
slab class 2: chunk size 104 perslab 10082
<29 send buffer was 110592, now 268435456
<29 server listening (udp)
<29 server listening (udp)



Start the Daemon


memcached -d -m 256 -u nobody -p 11211 -l 192.168.1.33


The above command will start the memcached daemon in port 11211

Ensure that it is working.

netstat -plan|grep 11211
tcp 0 0 192.168.1.33:11211 0.0.0.0:* LISTEN 7049/memcached




Thats it.. Enjoy the power of caching system..

Friday, January 8, 2010

Check Ubuntu 8.04 server security using OpenVAS

Install openvas server from source in ubunutu 8.04


apt-get install build-essential libgtk2.0-dev libglib2.0-dev libssl-dev htmldoc libgnutls-dev libpcap0.8-dev bison libgpgme11-dev libsmbclient-dev snmp pnscan


add the following line to /etc/apt/sources.list


deb http://ftp.de.debian.org/debian sid main


Install the server


apt-get install openvas-server openvas-server-dev openvas-plugins-dfsg openvas-plugins-base libopenvas2 libopenvas2-dev libopenvasnasl2 libopenvasnasl2-dev


Add your user


openvas-adduser


Update the feeds


openvas-nvt-sync



start the server


/etc/init.d/openvas-server start


Install the client

It would ask you to upgrade php etc and hence I compiled the agent from source. The trunk is incompatible with the openvas-server provided by debian repository.


svn co https://svn.wald.intevation.org/svn/openvas/branches/openvas-client-2-0/ openvas-client



cd openvas-client

./configure && make && make install


Running the agent in batch mode:

You can run the client to produce beautiful html with graphs using the following command line.

create a file named target with the host you want to check

cat >target
localhost
ctrl+c


OpenVAS-Client -q localhost 9390 ovasuser mypasswd target localhost-result -T html_graph -x


The run would take around 15 minutes and you will get the results in the localhost-result directory. View the HTML files using browser.