A Curl One-Liner to Retrieve Headers or Response Data

In this post I'll cover a simple curl one-liner that will let you read response content from multiple sites. I had a list of URLs and for each URL I wanted to check if the site redirected the user to another page via the location header. As usual a magic one liner came to the rescue!

Retrieving our data

First up I needed a way to send a request and receive a response from the remote server. Wget and Curl are both great options. I went with curl. Because of the company proxy I needed to include the --proxy flag.

curl -i -s --proxy http://myproxy.com:8080 www.google.com|grep Location:

• By default Curl doesn't include the response headers, to actually see them you need to use the -i flag. 
• Curl often displayed download messages I didn't want to see so I included the silent (-s) flag. 
• As I was just looking for redirections I added a bit of grep on the end to show just the Location header.

For Loopin'

As I wanted to grab the header from multiple sites I needed to perform the Curl within a for loop.

for i in $(cat hosts.txt);do curl -i -s --proxy http://myproxy.com:8080 http://www.$i|grep Location:;done;

Here I read in the hostnames from hosts.txt and for each host download the content and search for the Location header. My hosts were in the format "test.com", so I included the http://www. in front of the $i.

Sorting Output

Using the above one-liner you get the grep output and nothing else. I wanted to know which site had which header. To do this I assigned the curl output to a variable and then performed a check with double brackets to see if the variable was empty (whether the site had the header or not). And finally outputted the results.

for i in $(cat hosts.txt);do data=$(curl -i -s --proxy http://myproxy.com:8080 http://www.$i|grep Location:);if [[$data]];then echo $i $data;fi;done;

My final output looked like this:

You can see Google will redirect me to the local version (.com.ph) and Paypal/Facebook redirect to the https version of the site.

Final Thoughts

I love a good one-liner and this simple script can easily be altered for different headers or data depending on what you're looking for.

If you're just interested in the headers it's also worth checking out the curl special variables as described here:
http://beerpla.net/2010/06/10/how-to-display-just-the-http-response-code-in-cli-curl/

Hope you guys have found this useful. Improvements and comments always welcome!

Cheers, 

PwnDizzle out.