Recording and Replaying Web Interaction Using Perl Modules

CPAN, the Central Perl Archiving Network, is the Holy Grail of pre-packaged Perl scripts in module form. Before you tackle any coding project, check CPAN first, as it may save you tons of time and a huge headache. For instance, if you want to code a quick and dirty Perl script to automate a web user's clicks for site validation, you can do it with a couple of CPAN modules. The browser-independent HTTP::Recorder records all of the clicks a user makes when interacting with websites and creates a script for automated playback using WWW::Mechanize, a Perl module for stateful programmatic web browsing, used for automating interaction with websites.

After downloading and installing both HTTP::Recorder and WWW::Mechanize from CPAN, create a simple Perl script as specified by the HTTP::Recorder documentation as a proxy script. You may want to change the location to which the script saves the log file. Kick off your script, then go into your browser's proxy settings and change the proxy to point to localhost::8080. Voilà, the script is running!

Once the script has started it records all user browser actions in the specified file. You can see what is being written to the file using the module's control panel, which you can access by visiting http://http-recorder.

[caption id="attachment_88670" align="alignnone" width="396" caption="The HTTP::Recorder Control Panel"][/caption]

The HTTP::Recorder process only records user actions from a standard HTML page. If you are logging into page via SSL you must go to the http://http-recorder/ control panel and paste the https:// URL into the goto page section, then click the go button. You can then open a second browser tab or window, visit the site, and HTTP::Recorder will record your actions as usual, showing the results in the control panel window.

When you have recorded all the actions you want, stop the recording process by closing the browser tab or window that shows the website you are working in, and kill the HTTP::Recorder process.

You can now use an automated script to replay the actions you recorded. The script calls the WWW::Mechanize module to rerun the now automated steps against the specified URL.

Here is a sample script I completed in about 15 minutes that goes to a form on a company's login page (https://somesite/login) and logs in using username "XXXXXX" and password "passXXXXX". The script returns a successful statement to the terminal if the username and password combination are correct and an exception otherwise.

#!/usr/bin/Perl -w

use strict;
use WWW::Mechanize;
use Test::More qw(no_plan);

my $agent = WWW::Mechanize->new();
my $ccm_http_value;
my $monitoring_ccm;

$agent->get('https://somesite/login');
$agent->form_name('loginForm');
$agent->field('loginForm:loginBody:Password', 'passXXXXX');
$agent->field('Username', 'XXXXXX');
$agent->click('login');
$ccm_http_value = $agent->get('https://somesite');

   if($ccm_http_value->is_success){
      print "Successfully logged into ccm: $ccm_http_value 
"; 
      #print $ccm_http_value->content;
      print "Getting monitoring link at this time. 
";
    $monitoring_ccm = $agent->get('https://somesite');
   if($monitoring_ccm->is_success){
      print "Validation completed successfully logged into monitoring:  $monitoring_ccm ";
     # print $monitoring_ccm->content;
  
    } else {
            print STDERR $monitoring_ccm->status_line, "
"; 
      
    }  
 }
    else {
        print STDERR $ccm_http_value->status_line, "
";       
      }

You can kick off your script from the terminal or command line like any other Perl script. [and it stops by itself when it reaches the bottom of the recorded log?]

A couple of caveats: HTTP::Recorder doesn't record JavaScript very well. Also, you will not be able to record sites that use any type of heavy encryption, such as banking sites. And, as I noted earlier, if you want to record an SSL session, you must enter the site's URL into the control panel before bringing it up and recording your session.

For help with HTTP::Recorder, in addition to the doc page I linked to above, you can browse the HTTP::Recorder source tree, subscribe to a mailing list for users and developers, or browse the mailing list archives.

Follow @openlogic
Follow @CloudSwing

This work is licensed under a Creative Commons Attribution 3.0 Unported License
.Follow @openlogic
Follow @OSCloudServices

This work is licensed under a Creative Commons Attribution 3.0 Unported License
.