Scanning the log files for last 30 minutes of data

▲
0
▼

I have to write a shell/perl script to scan a log file for last 30 mins worth Data. The requirement is to schedule this script in Cron to run every 30 minutes and look for a error string.

OS: Solaris
Shell:Bash

I have tried below script, but it has become too long and clumsy, do we have other way to make it a bit shorter?

blogs=/opt/docs/datapower/prod/business.log
slogs=/opt/docs/datapower/prod/system.log


starttime=$(date +'%H')
currmin=$(date +'%M')
curdate=`date|cut -d' ' -f5`
echo $(date)

if [ $currmin -le 29 ] && [ $starttime -ne 00 ] ; then
starttime1=`echo "$(date +'%H') - 1" | bc`
logtime="$starttime1"
logtime="$logtime:[3-5][0-9]"
echo $logtime



elif [ $currmin -le 29 ] && [ $starttime -eq 00 ] ; then
logtime="23:[3-5][0-9]"
echo $logtime



else
logtime="$starttime"
logtime="$logtime:[0-2][0-9]"
echo $logtime

fi

if ( grep "$logtime" $slogs | egrep "AAA Authentication Failure|AAA Authorization Failure") > dptest 2>&1;then

       Do something

fi

UPDATE: Adding example log statement. Below is the example of log statement: Nov 20 06:06:58 business-log-sta [DP-Domain-STAGING][0x80000001][business-log][info] mpgw(GenServiceMPG): trans(31513092)[request]: AAA Authentication failure/>

bash
perl
shell
solaris
perlscript
asked on Stack Overflow Nov 20, 2014 by user2607367 • edited Nov 20, 2014 by user2607367

2 Answers

▲
1
▼

I think you're doing it a little backwards - building an RE to grep a date out of a log file.

Approaching this in perl I'd be looking to read the whole log file, tokenise it - to extract the time stamp - and then alert based on message content.

Perl has a nice module for the first part - Time::Piece. It goes a bit like this:

use strict;
use warnings;

use Time::Piece;

my $HALF_HOUR = 30 * 60;

while (<DATA>) {
    #extract timestamp via regular expression
    my ( $timestamp, $message ) = (m/\A(\w+\s+\d+\s+\d+:\d+:\d+) (.*)/);

    #convert text timestamp to 'unix time'. 
    #need the year in here because your log doesn't include it. 
    my $t = localtime();
    $t = $t->strptime( $timestamp . " " . $t->year, "%b %d %H:%M:%S %Y" );


    #skip if parsed time is more than half an hour ago. 
    next if ( $t < time() - $HALF_HOUR );
    if (   $message =~ m/AAA Authentication failure/i
        or $message =~ m/AAA Authorization failure/i )
    {
        print "Alert: ( $t )  $message\n";
    }
}

__DATA__
Nov 20 13:46:58 business-log-sta [DP-Domain-STAGING][0x80000001][business-log][info] mpgw(GenServiceMPG): trans(31513092)[request]: AAA Authentication failure/>
Nov 20 13:00:58 business-log-sta [DP-Domain-STAGING][0x80000001][business-log][info] mpgw(GenServiceMPG): trans(31513092)[request]: AAA Authentication failure/>
Nov 20 10:06:58 business-log-sta [DP-Domain-STAGING][0x80000001][business-log][info] mpgw(GenServiceMPG): trans(31513092)[request]: AAA Authentication failure/>

Followup Q:

"Could you please explain what this statement does, my ( $timestamp, $message ) = (m/\A(\w+\s+\d+\s+\d+:\d+:\d+) (.*)/);"

This does two things:

  • One of the tricks in Perl, is that you can capture parts of a regular expression, by putting it in brackets. So \A(\w+\s+\d+\s+\d+:\d+:\d+) - will match from the start of line:
    • One or more 'word' characters.
    • One or more 'digits'
    • \d+:\d+:\d+ will capture a time. (Any 3 colon separated numbers).

The other part, of course, captures 'the rest'.

  • Then, we assign the array returned by the pattern match, into an array of named variables ( $timestamp and $message).

Net result is - given the line:

 Nov 20  13:46:58       business-log-sta [DP-Domain-STAGING][0x80000001][business-log][info] mpgw(GenServiceMPG): trans(31513092)[request]: AAA Authentication failure/>
(\w+ \d+ \d+:\d+:\d+)   (.*)

Our regular expression returns the two 'chunks' separately, and then we put them into the two variables.

answered on Stack Overflow Nov 20, 2014 by Sobrique • edited Nov 20, 2014 by Sobrique
▲
0
▼

How is your opinion to use sqlite3 to do the filtering stuff - the benefit it parses the time for you could be very handy. The only backside is you have to normalize the data.

function sqlite-filter-time() {
    if [ '0' = "$#" ]; then
        echo "Usage: $FUNCNAME <file> <timespan> <where>"
        return
    fi
    local year="$(date '+%Y')"
    local ofs='___FS___'
    sed  "s,^\([^ ]* [^ ]*\) \([^ ]*\),\1 \2$ofs," "$1" | sed "s,Jan ,$year-01-,;s,Feb ,$year-02-,;s,Mar ,$year-03-,;s,Apr ,$year-04-,;s,May ,$year-05-,;s,June ,$year-06-,;s,July ,$year-07-,;s,Aug ,$year-08-,;s,Sep ,$year-09-,;s,Oct ,$year-10-,;s,Nov ,$year-11-,;s,Dec ,$year-12-," > "$1.tmp" # normalize data for sqlite - command to extract the date and the rest of the text
    {
        echo '.mode csv'
        echo 'DROP TABLE IF EXISTS sft;'
        echo 'CREATE TEMPORARY TABLE sft ('
        echo '  sft_date TEXT,'
        echo '  sft_text TEXT'
        echo ');'
        echo ".headers off"
        echo ".nullvalue ''"
        echo ".separator '$ofs'"
        echo ".import $1.tmp sft"
        echo ".separator ' '"
        echo "SELECT *"
        echo "FROM sft"
        echo "WHERE sft_date > datetime('now', '$2')"
        echo " AND (sft_text like '%AAA Authentication Failure%'"
        echo "   OR sft_text like '%AAA Authorization Failure%'"
        echo " )"
        echo ";"
    } | sqlite3
    rm "$1.tmp"
}

$ sqlite-filter-time "$slogs" '-30 minutes'
"2014-11-20 16:01:58" " business-log-sta [DP-Domain-STAGING][0x80000001][business-log][info] mpgw(GenServiceMPG): trans(31513092)[request]: AAA Authentication failure/>"
$
answered on Stack Overflow Nov 20, 2014 by lx42.de • edited Nov 20, 2014 by lx42.de

User contributions licensed under CC BY-SA 3.0