Monitoring those NTLM authentication Proxies

So, now that we have discussed how to overcome the challenge of testing those NTLM proxies, we move on to a better use. Load testing is fine and good, but how often do you really need to load test. Let's say though, that you have a couple dozen of these proxies spread out all over the globe, and for some reason MOM just doesn't cut it with monitoring the actual request performance on these proxies.

Using the base design of the previous script, I created one that is set to test each proxy in the environment once, through the same URL, and measure the delay in response. This is not 100% accurate as internal networking issues can cause some unaccounted fluctuation, but it is good enough for general purposes. So I created a mysql database with two tables. One is a status table, which contains the proxy, a counter, and the current known status. This is especially useful as the script pulls the proxies to test from this table, so adding or removing proxies is just a matter of doing it in the database, instead of altering code. the other table is a simple log file.

The script times the delay in the final response from the initiation of the request and then assigns a status based on this result. It compares it to the current status listed for that proxy, if it is different, it updates the table and emails out an alert. If it continues in a persistent bad state, it will send out a new alert again on the 12 straight return of that bad status. This ensures we are notified that the status is persisting, but doesn't flood us every 10 minutes, which is how frequently the script runs. Anyways, without further ado, here is my simplistic little proxy monitoring script

----------------code--------------------

#!/usr/bin/perl -w
use threads;
use DBI;
use LWP;
use LWP::UserAgent;
use HTTP::Request;
use Authen::NTLM(nt_hash, lm_hash);
use Authen::NTLM::HTTP;
use Time::HiRes qw( gettimeofday );
use Math::Round;
use Net::SMTP;

#Opens the connection to the datbase and prepares the statement handles we will need to call on.
our $dbh = DBI->connect("DBI:mysql:Proxy_Health", , ) or die "Unable to connect to database $DBI::errstr";
our $statuschk = $dbh->prepare("SELECT * from status WHERE proxy=?");
our $statusupd = $dbh->prepare("UPDATE status SET status=? , count=? where proxy=?");
our $logsth=$dbh->prepare("INSERT INTO chklog(proxy,delay) VALUES(?,?)");

#pulls the lsit of proxies from the datbase and maps them to a hash
%proxies= map {$_ => 0 } @{ $dbh->selectcol_arrayref("Select proxy from status" )};

#generates a worker thread for each proxy to test
 my $threadcount = 0;
 foreach (keys %proxies){
$threadcount+=1;
$thrs[$threadcount]= threads->create(\&Test, $_);

}
#performs blcoking for the threads, and returns the result of each test and inserts them into the chklog table
 foreach (keys %proxies){
$proxies{$_}= $thrs[$threadcount]->join;
$proxy_human = $_ ;
$proxy_human=~s/http:\/\///;
$proxy_human=~s/:80//;
$logsth->execute($proxy_human, $proxies{$_});
$threadcount-=1;
}

#Takes the results, and comapres the current status of the proxy to the last recorded status of the proxy. If the status has changed, it updates the status table and sends an alert. If the status has remained the same but is in a negative state, it increments a counter. Every 12 checks that return that negative result will generate a new Alert.
foreach (keys %proxies){
my $scount = 0;
if ($proxies{$_}>= 120){ $status = 'DOWN';}
elsif ($proxies{$_}>= 90){ $status = 'CRITICAL';}
elsif ($proxies{$_}>= 60){ $status = 'MAJOR';}
elsif ($proxies{$_}>= 40){ $status = 'MINOR';}
elsif ($proxies{$_}>= 20){ $status = 'SLOW';}
else{$status = 'GOOD';}
$statuschk->execute($_);
my @statusline = $statuschk->fetchrow_array;

if ($status eq $statusline[1]){
if ($status eq'GOOD'){last;}
elsif ($statusline[2]==11){
$scount =1;
&Alert($_, $status);
print "ALERT $_ !\n";
}
else{
$scount= $statusline[2] +1;
}
if ($scount==1){
&Alert($_, $status);
print "ALERT $_ !\n";
}
$statusupd->execute($status,$scount,$_);
}
else{
if ($status eq'GOOD'){$scount=0;}
else{$scount=1;}
$statusupd->execute($status,$scount,$_);
&Alert($_, $status);
print "ALERT $_ !\n";
}
}

 #

 #This function is what the worker threads run to test their given proxy.
sub Test{
#pulls the proxy from the passed parameters, sets the target as maps.google.com because that site is set to 'private' meaning the proxy will not cache it. It then retrieves the hostname of the local machine and the login credentials, so that it can properly negotiate NTLM authentication with the proxy server
my $proxy=$_[0];
my $url="http://maps.google.com";
our $workstation = `hostname` ;
my $user=;
my $my_pass = ;

#instanatiates the LWP user agent , sets the proxy, and sets the timeout to 120 seconds, because this is the timeout used on our ISA installs
my $ua =  new LWP::UserAgent(keep_alive=>1);
$ua->proxy('http', $proxy);
$ua->timeout(120);

#Creates the first request for the target website, starts the counter running and then fires off the request
my $req = HTTP::Request->new(GET => $url);
my $start = gettimeofday();
my $res = $ua->request($req);


#Sets up the data about the client to send the NTLM Authentication Negotiation Message
$client = new_client Authen::NTLM::HTTP(lm_hash($my_pass), nt_hash($my_pass),Authen::NTLM::HTTP::NTLMSSP_HTTP_PROXY, $user, , , $workstation, );

$flags = Authen::NTLM::NTLMSSP_NEGOTIATE_ALWAYS_SIGN | Authen::NTLM::NTLMSSP_NEGOTIATE_OEM_DOMAIN_SUPPLIED | Authen::NTLM::NTLMSSP_NEGOTIATE_OEM_WORKSTATION_SUPPLIED | Authen::NTLM::NTLMSSP_NEGOTIATE_NTLM | Authen::NTLM::NTLMSSP_NEGOTIATE_OEM ;
$negotiate_msg = $client->http_negotiate($flags);

#Takes the negotiation message and sets it as a header in the request and resends the request
$negotiate_msg = "Proxy-" . $negotiate_msg ;
@pa = split(/:/,$negotiate_msg);
$req->header($pa[0] => $pa[1]);
$res = $ua->request($req);

#Strips the NTLM challenge message from the response header and parses it
my $challenge_msg = "Proxy-Authenticate: " . $res->header("Proxy-Authenticate");

($domain, $flags, $nonce, $ctx_upper, $ctx_lower) = $client->http_parse_challenge($challenge_msg);

if ($domain or $ctx_upper or $ctx_lower){$placeholder=1;}

#Takes the nonce and flags from the challenge message , calculates the final authentication message, sets it as a header and sends it in the final request, recieving the originally requested page in response
$flags = Authen::NTLM::NTLMSSP_NEGOTIATE_ALWAYS_SIGN | Authen::NTLM::NTLMSSP_NEGOTIATE_NTLM | Authen::NTLM::NTLMSSP_REQUEST_TARGET;
$auth_msg = $client->http_auth($nonce, $flags);

@pa = split(/:/,$auth_msg);
$req->header($pa[0] => $pa[1]);
$res = $ua->request($req);

#Stops the timer, calculates the elapsed time rounding to the nearest hudnredth of a second and returns that value to the main thread
my $end = gettimeofday();
my $delta = ($end - $start);
$delta= nearest(.01,$delta);
print "Finished getting $url through $proxy in $delta seconds! \n";
return $delta;

}

#This function actually handles the generation of the email alert for a status change. Depending on the status it picks from different wordings in the email subject and message.
sub Alert{
my $proxy = $_[0];
my $status=$_[1];


if ($status eq 'GOOD'){
$subject="Subject: $proxy has returned to Normal Operation";
$message = "The ProxyHealth Monitor has detected that proxy $proxy has returned to a 'GOOD' status and is retrieving pages within an acceptable timeframe.";
}
elsif ($status eq 'SLOW'){
$subject="Subject: $proxy is experiecing delay";
$message="The ProxyHealth Monitor has detected that the proxy $proxy is experiencing slowness in processing web requests. The system will continue to monitor and will send an update when the status changes.";
}
elsif($status eq 'MINOR'){
$subject="Subject: $proxy is experiencing a Performance Problem";
$message="The ProxyHealth Monitor has detected that the proxy $proxy is suffering noticeable slowness in processing web requests. It's current status is rated as 'MINOR'. The system will continue to monitor and will send an update when the status changes.";
}
elsif($status eq 'MAJOR'){
$subject="Subject: $proxy is experiencing a Major Performance Problem";
$message="The ProxyHealth Monitor has detected that the proxy $proxy is suffering serious slowness in processing web requests. It's current status is rated as 'MAJOR'. The system will continue to monitor and will send an update when the status changes.";
}
elsif($status eq 'CRITICAL'){
$subject="Subject: $proxy is experiencing a Critical Performance Problem";
$message="The ProxyHealth Monitor has detected that the proxy $proxy is facing a 'CRITICAL' performance decrease. Web traffic throguh this proxy will be extremely slow. The system will continue to monitor and will send an update when the status changes.";
}
elsif($status eq 'DOWN'){
$subject="Subject: $proxy is DOWN!";
$message="The ProxyHealth Monitor has detected that traffic through $proxy is exceeding the timeout limit of 2 minutes. This has led to the system declaring the proxy as being 'DOWN'. Web requests through this proxy will FAIL due to timeout. The system will continue to monitor and will send an update when the status changes.";
}



my $mailer= Net::SMTP->new(, Hello=> );
$mailer->mail();
$mailer->to();
$mailer->data();
#Sets the UK and US Security Team Distribution lists as the Recipients
$mailer->datasend('To: , ');
$mailer->datasend("\n");
$mailer->datasend('Return-Path:');
$mailer->datasend("\n");
#Sets a header that will tell the mail client that replies are to go to the Security Distribution lists and not back to the fake address used to send the alert.
$mailer->datasend('Reply-To:, ');
$mailer->datasend("\n");
$mailer->datasend('FROM:');
$mailer->datasend("\n");
#Sets the message importance to high
$mailer->datasend('Importance: High');
$mailer->datasend("\n");
$mailer->datasend($subject);
$mailer->datasend("\n\n");
$mailer->datasend($message);
$mailer->dataend();
$mailer->quit;



}