LWP and NTLM Proxy Authentication

During the course of my duties, I had a need to load test some proxy servers. To do this, we decided to use ISA logs as sources for test traffic. so the objective was seemingly simple, write a quick LWP script that parses an ISA log for urls, then goes and tries to retrieve them through the target proxy. Oh and , of course, make it multi-threaded so we can send tons and tons of traffic at a time. Where it gets a little more complicated is this: the proxies in question all use NTLM Authentication. I wasn't discouraged, at first, but soon discovered that I could not find anyone who had managed to make LWP work with an NTLM proxy. Sure, I could have kludged it together with something like CNTLM, but that didn't feel right, and didn't provide for solid re usability.

Fortunately, I did find Yee Man Chan's Authen::NTLM Module which I was able to appropriately adapt to my purposes.It  is important to not that this Yee Man Chan's module not the one with the same namespace . You can tell which is which by the version numbers. Anyways, the script I wrote takes a proxy address, an isa log file and a number of threads as arguments, and proceeds to slam said proxy into oblivion. Here it is.Please feel free to leave comments and/or feedback.


-------------------------------------code-------------------------------------------------------------

#!/usr/bin/perl


use threads;
use Thread::queue;
use LWP;
use LWP::UserAgent;
use HTTP::Request;
use Authen::NTLM(nt_hash, lm_hash);
use Authen::NTLM::HTTP;


#Checks to ensure the user has invoked the script correctly
unless(scalar(@ARGV) ==3){
print "Proper usage is proxytest.pl <# of threads> \n";
print "Proxy must be entered as http://: or http://:\n ";
exit;
}

#Begin instantiating our queues
our $users = new Thread::Queue;
our $urls = new Thread::Queue;

#Takes the apssed parameters and sets them. This is the ISA log file being parsed for test URLS, the number of threads to use in testing, and the proxy being tested
my $logfile = $ARGV[0];
my $numthreads = $ARGV[1];
my $proxy = $ARGV[2];

#Collect the hostname for the local machine, this is important for the NTLM Negotiation that will be happening later
our $workstation = `hostname` ;
our $placeholder = 0;

#Verifies that the proxy was entered in the correct format
unless ($proxy=~/^http:\/\/[A-Za-z0-9\.]+:\d+$/){
print "Proxy must be entered as http://: or http://:\n ";
exit;
}

#Enqueues the test accounts to use
$users->enqueue();

#Reads through the supplied log file, and collects all of the URLs and enqueues them for the worker threads to use
open ISALOG, "<$logfile";

while (){
chomp;
if($_=~/\banonymous\b/i){next;}
if($_=~/\bhttp:\/\/\S+\b/i){$urls->enqueue($&);}
}

close ISALOG;
print "\n Done Reading Log! \n\n";

#Instantiates a number of worker threads based on the parameter passed when invoking the script
for($tcount=1; $tcount<=$numthreads;$tcount++){
$thrs[$tcount]= threads->create(\&printoff, $tcount );
}
#Sets blockings joins for each one of these asynchronous worker threads
for($tcount=1; $tcount<=$numthreads;$tcount++){
$thrs[$tcount]->join;
}

#foreach(@thrs){$_->join;}


#The meat and potatoes
sub printoff{
#Dequeues a URL and username to use. It then re-enqueues the username, sticking back at the end of the Queue to be used over again
my $tid = $_[0];
my $url = $urls->dequeue_nb;
my $user = $users->dequeue;
$users->enqueue($user);

#While it had a valid URL, it will perform the below tests
while ($url){


#Password is set here. This password is static for all of the used test accounts
my $my_pass = ;

#Creates the LWP User Agent, tells it to use the supplied proxy, and sends the initial HTTP GET request for the supplied URL and takes in a response
my $ua =  new LWP::UserAgent(keep_alive=>1);
$ua->proxy('http', $proxy);
$ua->timeout(30);
my $req = HTTP::Request->new(GET => $url);
my $res = $ua->request($req);

#Once the initial request has been sent out, the proxy will send back an NTLM negotiate message
#We set up the NTLM authentication client response by passing ntlm hashes of the username, password, domain, and workstation hostname
$client = new_client Authen::NTLM::HTTP(lm_hash($my_pass), nt_hash($my_pass),Authen::NTLM::HTTP::NTLMSSP_HTTP_PROXY, $user, , , $workstation, );
#Here we set the NTLM protocol flags that we wish to be accepted
$flags = Authen::NTLM::NTLMSSP_NEGOTIATE_ALWAYS_SIGN | Authen::NTLM::NTLMSSP_NEGOTIATE_OEM_DOMAIN_SUPPLIED | Authen::NTLM::NTLMSSP_NEGOTIATE_OEM_WORKSTATION_SUPPLIED | Authen::NTLM::NTLMSSP_NEGOTIATE_NTLM | Authen::NTLM::NTLMSSP_NEGOTIATE_OEM ;

#We then take the client data, and the flags and jam them into a header, and add it back to the original request, and resend it.
$negotiate_msg = $client->http_negotiate($flags);

$negotiate_msg = "Proxy-" . $negotiate_msg ;
@pa = split(/:/,$negotiate_msg);

$req->header($pa[0] => $pa[1]);
#The proxy then sends back an NTLM challenge response, which we strip from the message and parse using the NTLM methods provided by the module
$res = $ua->request($req);

my $challenge_msg = "Proxy-Authenticate: " . $res->header("Proxy-Authenticate");

($domain, $flags, $nonce, $ctx_upper, $ctx_lower) = $client->http_parse_challenge($challenge_msg);
#Kludged together fix. for some reason it generates errors if you do not do this. Possibly an oddity about the way we are using the NTLM module
if ($domain or $ctx_upper or $ctx_lower){$placeholder=1;}

#We set the next round of flags, take the Nonce which we gained from parsing the challenge message, and send back a final authentication message. Once the proxy recieves this, it processes the original GET request
$flags = Authen::NTLM::NTLMSSP_NEGOTIATE_ALWAYS_SIGN | Authen::NTLM::NTLMSSP_NEGOTIATE_NTLM | Authen::NTLM::NTLMSSP_REQUEST_TARGET;
$auth_msg = $client->http_auth($nonce, $flags);

@pa = split(/:/,$auth_msg);
$req->header($pa[0] => $pa[1]);
$res = $ua->request($req);
print "Finished getting $url \n";
#my $bytes = length $res->content;
#print " $url was $bytes bytes \n";
#print $res->code;
#print "\n\n" . $res->content;
#We then dequeue the next URL and continue on until there are no more URLs. The worker thread will then attempt to join. when all worker threads have joined, the code exits.
$url = $urls->dequeue_nb;


}

}