blog.pWnSpeak

I developed the below script to log the status of my server throughout a given day.  This can be used to investigate the moments leading up to a crash, or even investigate why load may have spiked at certain points during the day.  Code below:

#!/bin/sh

day=$(date +%Y%m%d)
min=$(date +%H%M)

/bin/mkdir -p /var/log/system/$day

/usr/bin/top -b -n 1 >  /var/log/system/$day/$min
/bin/netstat -an >>     /var/log/system/$day/$min
/bin/ps aux >>          /var/log/system/$day/$min
/usr/bin/free >>        /var/log/system/$day/$min
exit 0

I have a script I run weekly which automatically utilizes sa-learn to learn spam and ham message in my Inbox and multiple Spam folders.  This script also removes spam after a month and uses a last chance mail dir in case I missed something:

#!/usr/local/bin/bash
# Written by: Tim Henderson
#
# Simple shell script for learning junk using bayesian filtering
# Also clears my junk file and copies to a bigger junk file
#
# becoming not so simple
#

LEARN=/usr/local/bin/sa-learn
DOMBOX=/usr/boxes/$USER/pwnspeak.com
CAT=/bin/cat
CP=/bin/cp
DAY=$(date +%d)
WEEK=08
TWOWEEK=15
SPAM=$DOMBOX/bizdev^/.imap/Spam
FIVE=$DOMBOX/bizdev^/.imap/5plus.spam
TEN=$DOMBOX/bizdev^/.imap/10plus.spam
#PERCENT=$DOMBOX/bizdev^/.imap/percent.spam
RM=/bin/rm

# Learn Spam
$LEARN --spam --mbox $SPAM
$LEARN --spam --mbox $FIVE
$LEARN --spam --mbox $TEN
#$LEARN --spam --mbox $PERCENT

# Learn Ham
$LEARN  --ham --mbox /var/mail/acmeinc

######$$$$$$$##############$$$$$$$$$$###########
#
#    Must clean up stuff in first and second week of month
#    before moving stuff to the Bayesian Done folder
#

# if first week of month, then move bayes to last chance mail file
#
if [ $DAY -lt $WEEK ];
then
$CAT $DOMBOX/bizdev^/.imap/Bayes >> $DOMBOX/bizdev^/.imap/lastchance
$CP /dev/null $DOMBOX/bizdev^/.imap/Bayes
fi

# if second week of month, destroy last chance
if [ $DAY -lt $TWOWEEK -a $DAY -gt $WEEK  -o  $DAY -eq $WEEK ];
then
$CP /dev/null $DOMBOX/bizdev^/.imap/lastchance
fi


# Finally: Move to learned spam to bayes and empty create and secure
$CAT $SPAM     >> $DOMBOX/bizdev^/.imap/Bayes
$CAT $FIVE     >> $DOMBOX/bizdev^/.imap/Bayes
$CAT $TEN      >> $DOMBOX/bizdev^/.imap/Bayes
#$CAT $PERCENT      >> $DOMBOX/bizdev^/.imap/Bayes

$RM $SPAM $FIVE $TEN
touch $SPAM $FIVE $TEN
chmod 600 $SPAM $FIVE $TEN $DOMBOX/bizdev^/.imap/Bayes $DOMBOX/bizdev^/.imap/lastchance

I use the below script to turn over my procmail log.  It deletes logs more than 7 days old and keep my logs organized for easy checking:

#!/usr/local/bin/bash
#runs just before midnight to turn over logs

PROC="/usr/home/$USER/.procmail/"
DATE=$(date +%Y%m%d)

#  change the name to incorporate date
for var in `find $PROC -mtime 1 | grep log`;
do mv $var $var.$DATE;
done

# gzip the output
for var in `find $PROC -mtime 1 | grep log`;
do gzip $var;
done

# remove logs more then 7 days old
for var in `find $PROC -mtime 8 | grep log`;
do rm $var;
done

exit

I love procmail as my Mail Delivery Agent.  I utilize it to the max.  Qmail first accepts the messages for my entire domain (all of them) then delivers it to my procmail script utilizing my .procmailrc. 

I am currently developing an automatic .procmailrc generator for ease of creating useful rc files.  This program will allow advanced users to create their rc in a simplistic fashion and allow novices to use procmail with little to no knowledge of how it works in the backend.

I currently utilize clamscan and spamassasin for almost all messages.  I divide the scanned items into sub sections related to how high they score making my spam folders very clean.  Also I belong to mailing lists and this script automatically places them in IMAP folders for me, keeping clutter out of my Inbox. 

An example of my .procmailrc is below:

LOGDIR=/usr/home/$USER/.procmail
LOGFILE=$LOGDIR/procmail.log
VERBOSE=on
USER=myemailaddy@mydomain.com
DELIVER=/usr/local/libexec/dovecot/deliver
SPAMC=/usr/local/bin/spamc
FORMAIL=/usr/local/bin/formail

################################
# virus filter with clamscan
#
:0
VIRUS=|/usr/local/bin/clamdscan --no-summary --stdout -

:0fw
* VIRUS ?? ^.*: \/.* FOUND
| $FORMAIL -b -f -t -I "X-Clamav-Status: Yes, $MATCH"

:0Efw
| $FORMAIL -b -f -t -I "X-Clamav-Status: No"

#put virus messages in a folder called VIRUS
:0
* ^X-Clamav-Status: Yes
| $DELIVER -d $USER -m VIRUS


#######################################################
#          Pre filter out messages
#

# Parse xubuntu mailing list
:0:
* ^Subject:.*\[xubuntu-users\].*
| $DELIVER -d $USER -m xubuntu

# Parse ubuntu mailing list
:0:
* ^Return-Path:.*ubuntu-users.*
| $DELIVER -d $USER -m ubuntu



#########################################################
#        Filter with Spamassassin
#

# SA all messages
:0fw
| $SPAMC

# place spam that score >10 in special folder, eventually /dev/null
:0
* ^X-Spam-Status: Yes, score=[1-9][0-9].*
| $DELIVER -d $USER -m 10plus.spam

# place >5 && <10 spam in special place
:0
* ^X-Spam-Status: Yes, score=[5-9].*
| $DELIVER -d $USER -m 5plus.spam

# put <5 spam in a folder called SPAM
:0
* ^X-Spam-Status: Yes
| $DELIVER -d $USER -m Spam

:0
* ^X-Spam-Flag: YES
| $DELIVER -d $USER -m Spam

:0:
* ^Subject:.*JUNK.*
| $DELIVER -d $USER -m Spam

#
#       These were previous pre filtered, but I'd like to see if they
#       get marked as spam now
#
#            Update::::: they are indeed getting marked as spam! commenting
#            out code now
#
# messages with % in the subject line
#:0:
#* ^Subject:.*%.*
#| $DELIVER -d $USER -m percent.spam
#
# rx-store in from
#:0:
#* ^From:.*RX-Store.*
#| $DELIVER -d $USER -m percent.spam


######## Remove obvious spam before anything
# This is not needed now
#:0:
#* ^Subject:.*80% OFF.*
#| $DELIVER -d $USER -m pre.Spam
#
#:0:
#* ^From:(.*viagra.*|.*cialis.*)
#| $DELIVER -d $USER -m pre.Spam
#
#:0HB:
#* (.*russian lady.*|.*beautiful russian.*|.*russian woman.*|.*russian dat.*)
#| $DELIVER -d $USER -m pre.Spam
#
#:0:
#* ^Subject:.*Your wife photos attached.*
#| $DELIVER -d $USER -m Spam

#############
############## Content Filtering ###############
#############


# Deliver the rest, hopefully only good messages :)
:0:
| $DELIVER -d $USER

I have decided to set up monitoring for my disk drives.  It's very nice to know when you disk may fail so you can get it swapped out before disaster.

Also, enabling disk size monitoring is a must to prevent file loss and server instability.

The code for my drive failure monitoring is below:

#!/bin/bash

CWD="/home/$USER/bin/monitor/drive"
LOGFILE="/var/log/messages"
READNUM=$CWD/readnum
CACHELOG=$CWD/buffer
MESSAGE=$CWD/mess
READ=$(cat $READNUM)
TOTAL=$(wc -l $LOGFILE|awk '{print$1}')
TAILNUM=`expr $TOTAL - $READ`
HEADER=$CWD/header
SENDMAIL=/usr/sbin/sendmail

#exit if no change has been made to logs
if [ $TAILNUM = '0' ] ; then
exit
fi

# cache the log file
cat -n $LOGFILE | tail -$TAILNUM > $CACHELOG

# look for stuff in log
egrep -i "I/O error" $CACHELOG > $MESSAGE

# send message if there is
if [[ -s $MESSAGE ]] ; then
cat $HEADER $MESSAGE > $MESSAGE.msg
$SENDMAIL -t -oi <  $MESSAGE.msg
fi ;

# define new readnum
tail -1 $CACHELOG|awk '{print$1}' > $READNUM

exit

And for disk size monitoring:

#!/bin/bash
# check for large drive

CWD="/home/$USER/bin/monitor/size"

READNUM=$CWD/readnum
HEADER=$CWD/header
SENDMAIL=/usr/sbin/sendmail
MESSAGE=$CWD/mess
CONT=`cat $READNUM`

# has this already been determined today?
if  [ `echo $CONT` -eq 1 ]
then
exit
fi

# determine percentage use of drive
PERCENT=`df -h |grep sdb1|awk '{print$5}'|cut -d % -f 1`

if [ $PERCENT -gt 94 ]
then
echo `df -h |grep sdb1` > $MESSAGE
cat $HEADER $MESSAGE > $MESSAGE.msg
$SENDMAIL -t -oi <  $MESSAGE.msg
echo 1 > $READNUM
fi

exit

Both utilize sendmail to e-mail me in the case of imminent failure.  Also they use caching for optimum log searching.

Following a security breach I decided to enable a few monitors setup to e-mail me any time certain log entries contain a set of defined values.

For instance each FTP login from a specific username will result in an e-mail.  The following set of code defines this:

#!/bin/bash

# intialize all my vars
CWD="/home/$USER/bin/monitor/ftp"
LOGFILE="/var/log/vsftpd.log"
READNUM=$CWD/readnum
CACHELOG=$CWD/buffer
MESSAGE=$CWD/mess
READ=$(cat $READNUM)
TOTAL=$(wc -l $LOGFILE|awk '{print$1}')
TAILNUM=`expr $TOTAL - $READ`
HEADER=$CWD/header
SENDMAIL=/usr/sbin/sendmail

#exit if no change has been made to logs
if [ $TAILNUM = '0' ] ; then
exit
fi


# cache the log file
cat -n $LOGFILE | tail -$TAILNUM > $CACHELOG

# look for stuff in log
egrep "ftpuser|tricorp" $CACHELOG > $MESSAGE

# send message if there is
if [[ -s $MESSAGE ]] ; then
cat $HEADER $MESSAGE > $MESSAGE.new
$SENDMAIL -t -oi <  $MESSAGE.new
fi ;

# define new readnum
tail -1 $CACHELOG|awk '{print$1}' > $READNUM

exit

As you can see this set of code only looks in the log for the last entries since the previous check.  This makes for optimum resource usage.

This script runs every minute on the server.  Each night at midnight the 'readnum' file is zero'd out to reset the log since it was cleared by syslogd.

LG, we have always loved and respected the quality of your products.  YOU BASTARDS.  As we all know LG is selling their optical media with the Bluebird software preinstalled.  Yea, thats right, even in Linux when you close the drive without a disk your DVD drive will mount itself to Bluebird. 

Luckily there is a fix.  Unfortunately it must be installed via Windows :)  Fail me not, I use kvm for Vista and XP use when I need it.  I downloaded the fix to the Windows machine and ran it.  The annoying pop-ups have seized.

Link:

  Bluebird Fix

This was taken from MSFN forums:

http://www.msfn.org/board/index.php?act=ST&f=5&t=135300

And we all go crazy for Phil Ivey making the final table at the World Series of Poker.  God damn this is awesome.  It's like your home team making the Super Bowl or Stanley Cup (Pens and Steelers).  We are waiting in anticipation.  Go Phil, go!.  All of us in Pittsburgh are rooting for you Ivey. 

IVEY IVEY IVEY!!!!!

~pWnSpeak

If you experience odd 10-45 second delays when attempting to SSH into your server/computer, I have found the solution.  Append or change the below line in the ssh configuration file:

/etc/ssh/sshd_config

 to the below value:

UseDNS no

Finally, I haven't posted in awhile!  But this deems a post.  I have started my first HTPC, and it has been fun.  These things rock!  Lets go over the specs:

Pentium 4 2.8 gHz
2 GB RAM
ThermalTake Media Lab
80GB HD

Would you believe it, thats it!

Actually this HTPC is a front end machine, very very lightweight.  I have a local 'hog' which holds all of my files and does the brute force for this front end machine.  FE machine has only the MythBuntu OS installed and is remotely connected to the 'hog'.  The 'hog' has a 1.5TB drive and is used for download all of my media.  This machine is also the NAT'r for the front end machine, so the FE machine only has a ethernet connection to the 'hog'. 

In the future I will be setting up the HTPC with other mixes of setups.  The next one will be a FE/BE machine (front end back end) which does everything.  I will also be experimenting with other front end app's such as XBMC.   I'm pretty set on the Media Lab, but there is still alot of work, and further posting to be had.

Pictures and details, soon to come.