Stuff I Do

I have been playing around with a Linux server in my home lab and wanted to try expanding a Linux software RAID 1 array to a 3 disk RAID 5 array.

So I already have a two disk RAID 1 array with a pair of 1TB SATA drives at /dev/sdb and /dev/sdc. I am going to add a third drive at /dev/sdd to the array.

I start by removing the array description from /etc/mdadm.conf
I also comment out the line from the /etc/fstab

Next I stop the array with
# mdadm --stop /dev/md0

I also unmount that drive
# umount /mnt/vg001/lvol1

Next I change the array configuration to RAID 5
# mdadm --create /dev/md0 --level=5 -n 2 /dev/sdb /dev/sdc
At this point it says that the drives are already part of an array and asks if you really want to do this. Answer Y.

This part takes a little while so go find something else to do while it reconfigures the array.
Once it is done, I add the third drive to the array.
# mdadm --add /dev/md0 /dev/sdd

Now it is time to grow the array
# mdadm --grow /dev/md0 --raid-disks=3 --backup-file=/tmp/raid.backup
I started this 24 hours ago and it is only at 43% complete at this point and it is estimating that it will be another 100+ hours before it completes. Wow. So it works, just painfully slowly.

What I take way from this is if you need to add more space to your array and you are using Linux software RAID, put all your data on another device and build a brand new array, move your data back. It would take less time to move all the data than it would be to wait for the array to grow.

**UPDATE**
I checked in the morning and the grow had completed. So it took between 24 and 33 hours to complete the grow.

I have never really done much with PHP. I do very little coding and when I do it has always been a Microsoft centric language. For whatever reason, it can’t be boredom I have too much to be doing, this weekend I started fiddling around with PHP. I am actually working through a PHP tutorial so I actually learn the correct ways to use the language instead of just mashing things together like I have done in the past.

Not sure if it is the procedure or the documentation that I am using but it seems to be a very easy language with incredible power available to use. At this point I have only worked through operators, arrays and loops but I have been impressed with the little bit that I have done with it. I am curious once I get to the point of pulling data from a MySQL database and that should be fun.

I have no idea what I am going to do with this knowledge, we use ASP and ASP.net at work.

Ok so about a year ago I was fretting over the idea that MySQL was now owned buy Sun and I actually said, “…Worse of all is if Oracle had purchased them, we would never see another MySQL product again same with IBM.” Uhm yeah, well the news comes out today that Oracle is buying Sun and there by owns MySQL now too.

Interesting.

These instruction are for setting up rsyslog and phpLogCon on CentOS 5.3. It is assumed you have a working Apache and MySQL setup. Configuration of these two packages are outside the realm of this document.

On the Linux server I wanted to host the rsyslog database I ran yum install -y rsyslog-mysql on my CentOS 5.3 machine. Once installed went to the path of cd /usr/share/doc/rsyslog-mysql-2.0.6/. In this location you will see a file called createDB.sql which is the scheme needed to make using phpLogCon easy to use. To create this database type mysql -u MySQLusername -p < createDB.sql You will be prompted for a password for the account you entered and then the database will be created.

Edit the /etc/rsyslog.conf file to add in the mysql connection information at the top of the config file. I am using FROMHOST instead of HOSTNAME as I wasn't getting any hostnames with the former.
$ModLoad ommysql.so
$template dbFormat,”insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values (‘%msg%’, %syslogfacility%, ‘%FROMHOST%’, %syslogpriority%, ‘%timereported:::date-mysql%’, ‘%timegenerated:::date-mysql%’, %iut%, ‘%syslogtag%’)”,sql

*.* : ommysql:192.168.1.100,Syslog,MySQLusername,MySQLpassword;dbFormat

Stop the normal syslog daemon
service syslog stop

And set it to never come on
chkconfig syslog off

Set rsyslog to start at boot
chkconfig rsyslog on

And then start the service
service rsyslog start

You should now have messages showing in your MySQL database. If you do not check the /var/log/messages to see if there is an error in your rsyslog.conf file.

Once you are sure you are getting messages in your database you can use the phpLogCon application to have a nice web interface over the top of the database. Download from http://www.phplogcon.org/, I am using version 2.6.2 for this tutorial. Once downloaded unpack it and move the src directory to the web server path mv src /var/www/html/phplogcon Out of the contrib directory copy both the configure.sh and secure.sh to /var/www/html/phplogcon. Change to the web directory cd /var/www/html/phplogcon and then run the configure script by typing ./configure.sh Open a web browser and enter the path to this site http://servername/phplogcon to start the configuration of the site. Click NEXT until you get to Step 7 and make sure the source type is MySQL, the Database Name is Syslog and the table name is SystemEvents and then enter your MySQL username and password. You now have a function phpLogCon page.

On the Linux clients make sure you install the rsyslog-mysql module the same way you did on the server by typing yum install -y rsyslog-mysql

Edit the /etc/rsyslog.conf file to add in the mysql connection information at the top of the config file.
$ModLoad ommysql.so
$template dbFormat,"insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%FROMHOST%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",sql

*.* : ommysql:192.168.1.100,Syslog,MySQLusername,MySQLpassword;dbFormat

Stop the normal syslog daemon
service syslog stop

And set it to never come on
chkconfig syslog off

Set rsyslog to start at boot
chkconfig rsyslog on

And then start the service
service rsyslog start

Refresh your phpLogCon web page and you will now see two entries from the Linux client, one is a kernel message the other is a syslog message.

Anything that has a syslog capability can now point to the rsyslog server, such as HP JetDirect cards and Cisco network gear. All the messages are available in one location and you can filter and search for issues on your network.

In our Active Directory domain we do not allow the clients to resolve outside domain names. This is because we have a very restrictive web browsing policy and people are only allowed access out if a manager says they can, and even at that point it goes through a proxy server that blocks 99% of the internet. The added benefit is that this really curbs any virus or spyware issues. Even though we don’t let user workstations out of the network, there are other vlans that definitely need out. At the time we were very limited on the number of Windows servers we could purchase and place so we went with Linux to provide this name server for these vlans. Below is the config and how resolve both internal and external names and I will describe what each section involves.

Line 6-14 list the subnets that are allowed access to this name server. If you are not in this subnet and you request a name resolution from this server, you will be denied.
Line 18-21 lists the name servers for outside (internet) name resolution. These are the OpenDNS ip addresses.
Line 25-33 tells named that for the domain ‘domain.com’ resolve those names to the internal Active Directory DNS servers.
Line 35-55 lists the zones for reverse name resolution. These zone names must match the reverse name in your AD DNS servers.

options {
        directory "/var/named";
        dump-file "/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named_stats.txt";
        version "not currently available";
        allow-query {
                127.0.0.1;
                10.1.4.0/24;
                10.1.6.0/24;
                10.1.7.0/24;
                10.1.15.0/24;
                10.2.4.0/24;
                10.2.7.0/24;
        };
        allow-transfer {
                none;
        };
        forwarders {
                208.67.220.220;
                208.67.222.222;
        };
        forward only;
};

zone "domain.com" IN {
        type forward;
        forwarders {
                10.1.100.100;
                10.1.101.101;
                10.2.100.100;
                10.2.101.101;
        };
};

zone "1.10.in-addr.arpa" {
    type slave;
    file "slaves/1.10.in-addr.arpa";
    masters {
                10.1.100.100;
                10.1.101.101;
                10.2.100.100;
                10.2.101.101;
    };
};

zone "2.10.in-addr.arpa" {
    type slave;
    file "slaves/2.10.in-addr.arpa";
    masters {
                10.1.100.100;
                10.1.101.101;
                10.2.100.100;
                10.2.101.101;
    };
};

include "/etc/rndc.key";

Our maintenance department uses a CMMS package called MP2 for all maintenance work orders. The software is based on a MSSQL back end which has allowed us to create many additional reports and web interfaces to help the maintenance staff handle the work orders. Some of the work orders are scheduled PMs and they have a due date. Part of their review is making sure they have the work orders closed on time so to help them out I created a SQL Server Reporting Services report that will email them a list of work orders that are due the next business day.

First I have to get a listing of all the work orders that are due the next business day and then if it is Friday, get all the work orders that are due up through Monday. Below is the TSQL to get that info:

select
	wo.wonum,
	woeqlist.location,
	equip.description,
	wo.taskdesc,
	wo.wotype,
	wo.atfirstname + ' ' + wo.atlastname as fullname,
	wo.schedfinishdate
from
	wo left join woeqlist on wo.wonum = woeqlist.wonum
		join equip on woeqlist.eqnum = equip.eqnum
where
	schedfinishdate
		between
			cast(convert(char, getdate(), 101) as datetime) + 1 and
			case
				when (select datename(dw, getdate())) = 'Friday' then cast(convert(char, getdate(), 101)as datetime) + 3
				else cast(convert(char, getdate(), 101) as datetime) + 1
			end
	and status != 'C'
order by wonum

This will return the work order number, which plant location the equipment’s name, what the title of the work order is, type of work order, who is assigned to the work order and the date it is due as a extreme reminder which day they need to complete this job.

Throw that in a Reporting Services report and we are almost ready. I don’t want to send an email unless there is actually a work order that is due for the following day. For this I create a Data Driven Subscription in the Reporting Services web page for this report.

To do this go to the report you want to send out in the Reporting Services web page and click on the Subscriptions tab. Select the New Data Driven Subscription. Give this subscription a description and select the Shared Data Sources option and click Next. Pick your shared data source and click Next. Now we will make a slightly modified query to get back an email address if any records are available.

select
	email = 'MaintenanceDept@domain.com'
from
	wo
where
	schedfinishdate
		between
			cast(convert(char, getdate(), 101) as datetime) + 1 and
			case
				when (select datename(dw, getdate())) = 'Friday' then cast(convert(char, getdate(), 101)as datetime) + 3
				else cast(convert(char, getdate(), 101) as datetime) + 1
			end
	and status != 'C'
group by email

This will return us just one record of MaintenanceDept@domain.com if there are any number of work orders due for the next business day.

On the next page in the TO: field select the database field email to include the needed email address. Set any other properties on this page and click Next. Set the days and hours that you want the report to run and you are done.

Now an email will go out only when there are work orders that are due tomorrow and if no work orders are do, no one gets an email.

In a later post, I will show a super cool user defined function that you can use to take into account holidays and other office closed events.

There are plenty of other productive things I could have done tonight around the house but instead I setup a new MythTV box. In the past I have install Fedora and then piece by piece installed and configured the MythTV suite so tonight I decided to try out MythDora 5 which is a distribution that gives you everything. The install went really good and then when I went to watch live TV and got nothing but a blank screen. Crap.

Watching the /var/log/mythtv/mythbackend.log I get the following log:

2008-11-19 23:16:32.817 TVRec(1): Changing from None to WatchingLiveTV
2008-11-19 23:16:32.821 TVRec(1): HW Tuner: 1->1
2008-11-19 23:16:33.972 NVR(/dev/video0): Unknown video codec. Please go into the TV Settings, Recording Profiles and setup the four ‘Software Encoders’ profiles. Assuming RTjpeg for now.
2008-11-19 23:16:33.976 NVR(/dev/video0) Error: Unknown audio codec
2008-11-19 23:16:33.993 AutoExpire: CalcParams(): Max required Free Space: 2.0 GB w/freq: 15 min
2008-11-19 23:16:33.991 NVR(/dev/video0): Won’t work with the streaming interface, falling back
VIDIOCGMBUF:: Invalid argument
2008-11-19 23:17:14.017 TVRec(1): Changing from WatchingLiveTV to None
2008-11-19 23:17:14.023 Finished recording Seinfeld “The Stakeout”: channel 1002
2008-11-19 23:18:54.847 Expiring 0 MBytes for 1002 @ Wed Nov 19 23:00:00 2008 => Seinfeld “The Stakeout”

Seems kind of odd since it is talking about a Software Encoder and I have a PVR-350 which is a hardware MPEG card. It appears the problem comes from being too trusting that the software would properly set itself up correctly. On the tuner page it said it auto probed the card and I went with it. Since the auto probe was wrong it set the card as a V4L card when I should have selected a hardware encoder card. As soon as I did that Live TV worked the way you would expect it to.

If you can’t trust a computer who can you trust?!

USB flash drives are great little devices. Gigs of information stored quickly and easily on a light weight unit you can stuff in any pocket. The downside is that these things get lost often too, and users of these devices apparently are too relaxed about them and put data that should never be on a removable drive on it. Recently I got a letter from a college I have attended, informing me that some moron in their organization of higher learning had lost a flash drive that contained a spread sheet with hundreds of students names, addresses and social security numbers, and that I should be on the look out for my hits against my credit report or any other identity theft issues. {Climbing onto soapbox} Personally I think I should not have to do a thing, that the organization that was so flippant with such sensitive data should shed large sums of money to protect those they have harmed and prosecute those who use the data. Maybe these places would then take seriously securing our information when they have huge dollars of losses resulting in their sloppiness. {Climbing back down off soapbox}.

Anyway, since I have never carried anything sensitive on my flash drives, I figured I would see how difficult, inconvenient it was to encrypt data on a flash drive to protect it. I selected the OpenSource product TrueCrypt. It is free, open and works on Windows and Linux and I can only assume that you could build it from source on a Mac. While free it has no institutional control from a central IT perspective which could be hard to deploy for an IT department, although it would not cost an organization anything to use it, so you can rule out “its too expensive to do” excuse. I installed the small application onto my Windows laptop and plugged in a flash drive. Started the application and you have two options, you can either encrypt the entire drive or just a file. Pluses and minus to each and you would have to decide which worked better for your organization, personally if I were handing out flash drives to uses the entire thing would be encrypted so that users don’t take the lazy way of copying data do the unencrypted portion. You can select the type of encryption you want and can even test the device and TrueCrypt will tell you what one is quickest and explains each method of encryption. Once the device has been encypted you launch the TrueCrypt application and mount the device. It can no longer be mounted as a regular device and has to be mounted through the application. Once mounted it simply shows as another drive in the My Computer window, start coping files to it and they are encrypted and protected. Pretty simple.

The downside I see is that it does take a few extra clicks to mount the encrypted device, but if organizations were financially responsible for the data they lost, a couple extra clicks would be no big deal. The other would be people in the organization who will always try to circumvent IT policies be cause it is too inconvenient for them. Again if you were to make the person who lost the drive, use their personal finances to offset the costs to the victims, you would not see these people try to workaround the security. The other major downside is that if you travel with the device you either need the application install on the remote computer or use the Traveler Mode of the application. The bad thing with Traveler Mode requires you to have Admin rights on the machine.

Bottom line is, it is not too difficult to protect data on removable drives and people and companies need to be held financially responsible when they do loose data.

We have a RAIC (Redundant Array of Inexpensive Computers) that control our entire conveyor system through out the manufacturing plant. This conveyor system is responsible for staging product and placing it behind machine when a particular order has been scheduled to sending final product all the way down to the truck docks. If this system were to go offline it makes getting product through a much more labor intensive process. So we have this system that keeps track of all order, where they are on the conveyors, when they are scheduled etc. The base of this system runs off of MySQL on Linux servers. If we loose this database everything is lost and we would have to hand enter every product and where it is staged on the line. Not something we want to do.

So I did a little reading and setup MySQL replication with a Master-Slave relationship. Really quite an easy process and extremely quick in getting the data out to the slave. Below are the steps I used to complete this task. This was done on RedHat Enterprise 4.4 and MySQL 4.1.20.

Setup The Master
On the master server edit the /etc/my.cnf file. In the [mysqld] section add the following lines:

# database replication
log-bin
binlog-do-db=database-name
server-id=1

Restart MySQL.
/etc/init.d/mysql restart

Log into the MySQL server /etc/init.d/mysql restart and enter the following query:
GRANT REPLICATION SLAVE ON *.* TO ‘replication-user-account‘@’%’ IDENTIFIED BY ‘password‘;
FLUSH PRIVILEGES;

USE database-name;
FLUSH TABLES WITH READ LOCK;
SHOW MASTER STATUS;

Make a note of the file and position from the query results you will need this later.

QUIT;

Make a dump of the database to copy to the slave machine.
mysqldump -u root -p –opt database-name > /tmp/database-name.sql

Once you have the dump log back into the MySQL server and issue the following commands;
UNLOCK TABLES;
QUIT;

Copy the database dump to the slave machine:
scp /tmp/database-name.sql root@slave.domain.com:/tmp

Setup The Slave
Log into the slave server.

Log into the MySQL server:
mysql -u root -p

Create a new database:
CREATE DATABASE database-name;
QUIT;

Copy sql dump into the database:
mysql -u root -p database-name < /tmp/database-name.sql

On the slave server edit the /etc/my.cnf file. In the [mysqld] section add the following lines:

# database replication
server-id=2
master-host=master.domain.com
master-user=replication-user-account
master-password=password
master-connect-retry=60
replicate-do-db=database-name

Restart the MySQL service.
/etc/init.d/mysqld restart

Log into the MySQL server and issue the following commands:
SLAVE STOP;
CHANGE MASTER TO MASTER_HOST=‘master.domain.com’, MASTER_USER=‘replication-user-account’, MASTER_PASSWORD=‘password’, MASTER_LOG_FILE=’server-bin.000001′, MASTER_LOG_POS=101091;
Be sure to replace MASTER_LOG_FILE and MASTER_LOG_POS with the values you noted when setting up the master server
SLAVE START;
QUIT;

Ran into an issue with the Infrant NAS devices. Apparently there is a problem with the MTU size that the Infrant device likes and when I am trying to do an Rsync between the two devices over the WAN they will just time out and die. If possible you can set the MTU of the port the NAS device is connected on to 1492. Unfortunately for me there is no way to do this on the HP ProCurve switch I am currently using, can’t wait until I get the new Cisco gear in! When P10 firmware comes out it is suppose to deal with this problem where you can set the MTU on the NAS itself, but until then I have to NFS mount the NAS device from one of my Linux servers and use Rsync on the Linux server to the remote NAS to keep the devices in sync.