PHP Screenscraping Using Curl

// October 22nd, 2005 // Technology Bits

Php-1Tim was looking to try something new, so I decided to introduce him to Client URL (CURL) functions. As the example at hand, we looked at hitting the USPS site to lookup city and states based on ZIP code.

For the uninitiated, CURL basically lets you programmatically simulate a user browsing a web site. You can POST, GET, PUT, maintain cookies and session information. In the following example we are using a technique called "screen scraping" which is rarely recommended, but a good skill to have because sometimes its the only solution.

The reason its bad is because it is extremely fragile. If a webmaster of the site you are accessing makes even a slight change, it could break your page parsing. The other reason to shy away from this is some web sites really don't like when you do this. As a rule, if the webmaster of the site you are scraping contacts you and wants you to stop, you should, immediately. Though you should also recommend they provide the info you are scraping as a service through something like REST or SOAP. It would be very Web 2.0 of them to comply, it's worth a shot.

Anyway, check out this example code, it's kinda fun.

PLAIN TEXT
PHP:
  1. <?php
  3.  
  4. $ch = curl_init();
  5. curl_setopt($ch, CURLOPT_URL, "http://zip4.usps.com/zip4/zcl_3_results.jsp");
  6. curl_setopt ($ch, CURLOPT_POST, 1);
  7. curl_setopt ($ch, CURLOPT_POSTFIELDS, "zip5=".$_GET['zip']);
  8.  
  9. $data = curl_exec($ch);
  10. $string = ob_get_contents();
  11.  
  13.  
  14. list(,$second) = explode('Actual City name', $string);
  15. list($first) = explode('images/spacer.gif', $second);
  16. $junk = explode(“\n”,$first);
  17.  
  18. list($city,$state) = explode(', ',trim(strip_tags($junk[6])));
  19.  
  20. $city = ucwords(strtolower($city));
  21.  
  22. print $city.','.$state;
  23. ?>

web20, web2.0, code, programming, screen scraping, php, soap, rest, curl, usps, address information, zip code, zip, zip codes

Related items

21 Responses to “PHP Screenscraping Using Curl”

  1. brian says:
    October 4, 2006 at 11:17 am

    good job with the most incomplete piece of code ive ever seen.

  2. zbtirrell says:
    October 6, 2006 at 8:12 am

    This is purely an example of how one might use cURL and output bufferring. There is certainly no error checking or particularly useful output… but this is not a chunk of code anyone would drop in and use somewhere. It’s an example… From this example you should be able to extrapolate your own useful application, api, or what have you.

    If there is some other incompleteness beyond it being short with no error checking, please enlighten people with your insight. I’d love constructive discourse on this. I’d love someone to show me other ways to accomplish this, or better uses of output buffering and/or cURL.

  3. georges says:
    October 16, 2006 at 9:50 pm

    incomplete or not, it was handy to see a decent exmaple. thanks.

  4. zbtirrell says:
    October 17, 2006 at 8:42 am

    Thanks georges, that was my intent…

  5. Ashish Srivastava says:
    November 8, 2006 at 7:54 am

    hi,

    How to maintain the session in curl in php.

  6. Kumi Rauf says:
    November 8, 2006 at 11:43 pm

    Nice example… pretty black and white. Time to scrape some pages!

  9. madsh says:
    February 12, 2007 at 4:02 pm

    And if you want to do the same in 5 min without writing a single line of code… have a look at http://www.openkapow.com

  10. Lee says:
    May 10, 2007 at 8:19 pm

    Thanks a lot for the excellent example! Great job!

  11. Tine MГјller says:
    September 6, 2007 at 4:12 am

    I can’t get it to functioning - is the code not functioning any more?

    /Tine

  13. Neo says:
    April 10, 2009 at 2:06 am

    nicholas jonas
    [URL=http://xidimaco.100megsfree8.com/nicholas-jonas/]nicholas jonas[/URL]
    fraction worksheets
    [URL=http://xidimaco.100megsfree8.com/fraction-worksheets/]fraction worksheets[/URL]
    fairfield county court records
    [URL=http://xidimaco.100megsfree8.com/fairfield-county-court-records/]fairfield county court records[/URL]
    spotting in early pregnancy
    [URL=http://xidimaco.100megsfree8.com/spotting-in-early-pregnancy/]spotting in early pregnancy[/URL]
    lowes cinema
    [URL=http://xidimaco.100megsfree8.com/lowes-cinema/]lowes cinema[/URL]
    racquel welch
    [URL=http://xidimaco.100megsfree8.com/racquel-welch/]racquel welch[/URL]
    free webkinz unused code
    [URL=http://xidimaco.100megsfree8.com/free-webkinz-unused-code/]free webkinz unused code[/URL]
    olympic medal count
    [URL=http://xidimaco.100megsfree8.com/olympic-medal-count/]olympic medal count[/URL]
    spinach salad recipe
    [URL=http://xidimaco.100megsfree8.com/spinach-salad-recipe/]spinach salad recipe[/URL]
    kci and jojo
    [URL=http://xidimaco.100megsfree8.com/kci-and-jojo/]kci and jojo[/URL]

Leave a Reply

Sponsored Links