LWP-UserAgent-GET-Redirect.pl - Following HTTP Redirects

Perl Tutorials - Herong's Tutorial Examples

∟LWP-UserAgent-GET-Redirect.pl - Following HTTP Redirects

This section provides a tutorial example on how to send a GET request to a Web server, follow up with redirects, and dump request and response messages of all redirects.

Here is the improved LWP::UserAgent test program:

#- LWP-UserAgent-GET-Redirect.pl
#- Copyright (c) HerongYang.com. All Rights Reserved.

   use LWP::UserAgent;

   my ($uri) = @ARGV;
   $uri = 'http://herongyang.com/' unless $uri;

   my $browser = new LWP::UserAgent();
   $browser->max_redirect(5);
   $browser->agent("Mozilla/5.0 (Windows NT 6.1)");

   my $response = $browser->get($uri);
   my $request = $response->request();
   my @redirects = $response->redirects();

   foreach my $res (@redirects) {
      my $req = $res->request();
      print($req->as_string());
      print($res->as_string());
   }
   print($request->as_string());
   print($response->as_string());

   exit;

3 changes are made on the program:

"$browser->max_redirect(5);" is added to limit automatic redirects to maximum 5 times.
"$browser->agent("Mozilla/5.0 (Windows NT 6.1)");" is added to tell the Web server that we are a "Mozilla/5.0" browser on a Windows system.
"my @redirects = $response->redirects();" is added to retrieve all redirected response objects.
"foreach my $res (@redirects) {...}" loop is added to print out all redirected requests and responses Note that the last request and response is not in the @redirects list.

Now try this new testing program on facebook.com, I got the following:

herong> LWP-UserAgent-GET-Redirect.pl http://www.facebook.com

GET http://www.facebook.com
User-Agent: Mozilla/5.0 (Windows NT 6.1)

HTTP/1.1 301 Moved Permanently
Cache-Control: private, no-cache, no-store, must-revalidate
Connection: close
Pragma: no-cache
Location: https://www.facebook.com/
Content-Length: 0
Content-Type: text/html; charset=utf-8
...


GET https://www.facebook.com/
User-Agent: Mozilla/5.0 (Windows NT 6.1)

HTTP/1.1 200 OK
Cache-Control: private, no-cache, no-store, must-revalidate
Connection: close
Pragma: no-cache
Content-Type: text/html; charset=utf-8
Client-Response-Num: 1
Client-SSL-Cert-Issuer: /C=US/O=VeriSign, Inc./OU=VeriSign Trust N...
Client-SSL-Cert-Subject: /C=US/ST=California/L=Palo Alto/O=Faceboo...
Client-SSL-Cipher: ECDHE-RSA-AES128-GCM-SHA256
Client-SSL-Socket-Class: IO::Socket::SSL
Client-Transfer-Encoding: chunked
Link: <https://www.facebook.com/>; media="handheld"; rel="alternate"
Link: <https://www.facebook.com/>; hreflang="en"; rel="alternate"
P3P: CP="Facebook does not have a P3P policy. Learn why here: http...
Refresh: 0; URL=/?_fb_noscript=1
Set-Cookie: reg_fb_ref=https%3A%2F%2Fwww.facebook.com%2F; path=/; ...
Title: Welcome to Facebook - Log In, Sign Up or Learn More
X-Content-Type-Options: nosniff
X-FB-Debug: HshLC4adnJI8cXDgh8tGUSyMrbso7p3Wfw1EDU61Swo=
X-Frame-Options: DENY
X-Meta-Charset: utf-8
X-Meta-Description: Facebook is a social utility that connects peo...
X-Meta-Referrer: default
X-Meta-Robots: noodp, noydir
X-XSS-Protection: 0

<!DOCTYPE html>
<html lang="en" id="facebook" class="no_js">
<head><meta charset="utf-8" /><script>function envFlush(a){functio...
    <link type="text/css" rel="stylesheet" href="https://fbstatic-...
...
<form id="login_form"
action="https://www.facebook.com/login.php?login_attempt=1"
method="post" onsubmit="return window.Event && Event.__inline...
<input type="text" class="inputtext" name="email" id="email" value...
<input type="password" class="inputtext" name="pass" id="pass" tab...
...

As you can see, facebook.com server is happier now.

The first response from http://www.facebook.com was a redirect instruction: "301 Moved Permanently". The new URI is https://www.facebook.com/.
The LWP::UserAgent object made a second request to the new URI. The second response is the normal facebook home page with a login form.

If we want to continue to play with the facebook web server, we need use LWP::UserAgent post() method to submit the login form. Continue with the next tutorial.