PHP Crawler Help

miamiandy · **Posted:** Sat May 01, 2010 6:22 am

Hey,
I was wondering if someone could help me with a problem I am having. I have a crawler, that is indexing a particular domain (*.rpi.edu) It works fine for the most part.

However, when it tries to get_headers($row['url'], 1) from links such as:

http://prod3.web.server.rpi.edu/peopledirectory/search.do?query=employee1_dept_org:3040&datasetName=directory&qct=200
http://webct.rpi.edu

It times out since it isn't getting any response at all within the maximum execution time. I am at a loss on how to check.

Below is the full error. I can provide other details if needed. Thank you for your help.

Quote:

Warning: get_headers(http://prod3.web.server.rpi.edu/peopledirectory/search.do?query=employee1_dept_org:3040&datasetName=directory&qct=200) [function.get-headers]: failed to open stream: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. in C:\xampp\htdocs\Crawler\pagetitle.php on line 12

Liger · **Posted:** Sat May 01, 2010 8:12 pm

You just want to check to see if it failed?

Line number On/Off | Expand/Contract | Select all

if(!$results = @get_headers(...))
{
    echo 'Timeout';
}
else
{
    ...
}

The function returns an array of the headers on success, and false on failure.

miamiandy · **Posted:** Sun May 02, 2010 12:25 am

Thanks for the suggestion, I tried that, the problem is the get header i believe times out longer than 60 seconds which is the time out time of php.

Liger · **Posted:** Sun May 02, 2010 5:01 am

If this is on a local or dedicated server, then you can change the max_execution value in php.ini to zero to disable it. If not, you can set it in a .htaccess file with:

Line number On/Off | Expand/Contract | Select all

php_value max_execution_time 0

or at the beginning of your script:

Line number On/Off | Expand/Contract | Select all

set_time_limit(0);

That only works if safe mode isn't enabled.

miamiandy · **Posted:** Sun May 02, 2010 7:16 am

Ok, i'll try setting the limit to 5 minutes.

Welcome to Mango12!

PHP Crawler Help

Who is online