Perl Tutorials - Herong's Tutorial Examples

∟LWP::UserAgent and Web Site Testing

∟LWP-UserAgent-Request.pl - GET, POST and Cookies

This section provides a tutorial example on how to send a GET or POST request with a single generic LWP::UserAgent object and manages cookies automatically like a real Web browser.

Now I think we are ready to write a more generic Perl program to send a GET or POST request to a Web server, follow any redirects, automatically manage cookies like a Web browser.

Here is my first version of LWP-UserAgent-Request.pl: 

#- LWP-UserAgent-Request.pl
#- Copyright (c) HerongYang.com. All Rights Reserved.

   use LWP::UserAgent;
   use HTTP::Cookies;

   my ($method, $uri, $data) = @ARGV;
   $method = 'GET' unless $method;
   $uri = 'http://herongyang.com/' unless $uri;
   $data = 'author=Herong&lang=Perl' unless $data;

   my $browser = new LWP::UserAgent();
   $browser->requests_redirectable(['GET', 'HEAD', 'POST']);
   $browser->max_redirect(5);
   $browser->agent("Mozilla/5.0 (Windows NT 6.1)");

   my $request = HTTP::Request->new();
   $request->method($method);
   $request->uri($uri);
   if (uc($method) eq 'POST') {
      $request->header('Content-Type'
         => 'application/x-www-form-urlencoded');
      $request->content($data);
   }

   my $jar = HTTP::Cookies->new();
   $jar->load('HTTP-Cookies.jar');
   $jar->add_cookie_header($request);

   my $response = $browser->request($request);
   $request = $response->request();
   $jar->extract_cookies($response);
   $jar->save('HTTP-Cookies.jar');

   my @redirects = $response->redirects();
   foreach my $res (@redirects) {
      my $req = $res->request();
      print($req->as_string());
      print($res->as_string());
   }
   print($request->as_string());
   print($response->as_string());

   exit;

sub LWP::UserAgent::redirect_ok {
   my ($self, $req, $res) = @_;
   $jar->extract_cookies($res);
   $jar->add_cookie_header($req);
   return true;
}

What's new in LWP-UserAgent-Request.pl is that we are using an HTTP::Request object to send a GET or POST request message depending on the first command-line option. And if the method is POST, we are using the third command-line option as the form data to be posted.

When testing the program on google.com, I got the following: 

herong> LWP-UserAgent-Request.pl GET http://www.google.com

GET http://www.google.com
User-Agent: Mozilla/5.0 (Windows NT 6.1)

HTTP/1.1 200 OK
Set-Cookie: PREF=ID=39f3c35845eeda94:FF=0:TM=1393709033:LM=1393709...
Set-Cookie: NID=67=v4hLLg1a_TYSA0jgDZmyiEFmp2r3qWujpGhCRAx9by8hSo0...
...

herong> LWP-UserAgent-Request.pl POST http://www.google.com
   "author=Herong&lang=Perl"

POST http://www.google.com
User-Agent: Mozilla/5.0 (Windows NT 6.1)
Content-Type: application/x-www-form-urlencoded
Cookie: PREF=ID=39f3c35845eeda94:FF=0:TM=1393709033:LM=1393709033:...
   NID=67=v4hLLg1a_TYSA0jgDZmyiEFmp2r3qWujpGhCRAx9by8hSo0s6I5t9QrM...
Cookie2: $Version="1"

user=me&lang=en

HTTP/1.1 405 Method Not Allowed
...

The testing result shows that:

Table of Contents

 About This Book

 Perl on Linux Systems

 ActivePerl on Windows Systems

 Data Types: Values and Variables

 Expressions, Operations and Simple Statements

 User Defined Subroutines

 Perl Built-in Debugger

 Name Spaces and Perl Module Files

 Symbolic (or Soft) References

 Hard References - Addresses of Memory Objects

 Objects (or References) and Classes (or Packages)

 Typeglob and Importing Identifiers from Other Packages

 String Built-in Functions and Performance

 File Handles and Data Input/Output

 Open Files in Binary Mode

 Open Directories and Read File Names

 File System Functions and Operations

 Image and Picture Processing

 Using DBM Database Files

 Using MySQL Database Server

 Socket Communication Over the Internet

 XML::Simple Module - XML Parser and Generator

 XML Communication Model

 SOAP::Lite - SOAP Server-Client Communication Module

 Perl Programs as IIS Server CGI Scripts

 CGI (Common Gateway Interface)

 XML-RPC - Remote Procedure Call with XML and HTTP

 RPC::XML - Perl Implementation of XML-RPC

 Integrating Perl with Apache Web Server

 CGI.pm Module for Building Web Pages

►LWP::UserAgent and Web Site Testing

 What Is LWP::UserAgent?

 What Is HTTP::Response?

 What Is HTTP::Request?

 LWP-UserAgent-GET.pl - Sending a GET Request

 LWP-UserAgent-GET-Redirect.pl - Following HTTP Redirects

 http-equiv="Refresh" Meta Tag not Followed

 LWP-UserAgent-POST.pl - Posting Form Data

 post() Method not Following Redirect Location

 LWP-UserAgent-POST-Redirect.pl - Posting with Redirects

 What Is HTTP::Cookies?

►LWP-UserAgent-Request.pl - GET, POST and Cookies

 LWP-UserAgent-Request.pl - Login to facebook.com

 HTTP::Cookies save() not Saving Temporary Cookies

 LWP-UserAgent-Request-Cookies.pl - Sending Request with Cookies

 Converting Perl Script to Executable Binary

 Managing Perl Engine and Modules on macOS

 Archived Tutorials

 References

 Full Version in PDF/EPUB