Backend Development
PHP Tutorial
Detailed explanation of how to crawl and analyze web pages with PHP, detailed explanation of php crawling and analysis_PHP tutorial
Detailed explanation of how to crawl and analyze web pages with PHP, detailed explanation of php crawling and analysis_PHP tutorial
Jul 12, 2016 am 08:53 AM
Detailed explanation of the method of crawling and analyzing web pages with PHP, detailed explanation of php crawling and analysis
This article describes the method of crawling and analyzing web pages with PHP. Share it with everyone for your reference, the details are as follows:
Crawling and analyzing a file is very simple. This tutorial will take you step by step through an example to implement it. Let's get started!
First, we must decide which URL addresses we will crawl. This can be set in a script or passed via $QUERY_STRING. For simplicity, let's set the variable directly in the script.
<?php $url = 'http://www.php.net'; ?>
In the second step, we grab the specified file and store it in an array through the file() function.
<?php $url = 'http://www.php.net'; $lines_array = file($url); ?>
Okay, now we have the files in the array. However, the text we want to analyze may not all be in one line. To resolve this file, we can simply convert the array $lines_array into a string. We can use the implode(x,y) function to achieve this. If you want to use explode later (array of string variables), it may be better to set x to "|" or "!" or other similar delimiter. But for our purposes, it's better to set x to a space. y is another necessary parameter because it is the array you want to process with implode().
<?php
$url = 'http://www.php.net';
$lines_array = file($url);
$lines_string = implode('', $lines_array);
?>
Now that the crawling work is done, it’s time to analyze. For the purposes of this example, we want to get everything between
and . In order to parse out the string, we also need something called a regular expression.
<?php
$url = 'http://www.php.net';
$lines_array = file($url);
$lines_string = implode('', $lines_array);
eregi("<head>(.*)</head>", $lines_string, $head);
?>
Let’s take a look at the code. As you can see, the eregi() function is executed in the following format:
eregi("<head>(.*)</head>", $lines_string, $head);
"(.*)" means everything, which can be interpreted as, "analyze everything between
and ". $lines_string is the string we are analyzing, and $head is the array where the analyzed results are stored.Finally, we can enter the data. Since there is only one instance between
and , we can safely assume that there is only one element in the array, and it is the one we want. Let's print it out.
<?php
$url = 'http://www.php.net';
$lines_array = file($url);
$lines_string = implode('', $lines_array); eregi("<head>(.*)</head>", $lines_string, $head);
echo $head[0];
?>
That’s all the code.
<?php
//獲取所有內(nèi)容url保存到文件
function get_index ( $save_file , $prefix = "index_" ){
$count = 68 ;
$i = 1 ;
if ( file_exists ( $save_file )) @ unlink ( $save_file );
$fp = fopen ( $save_file , "a+" ) or die( "Open " . $save_file . " failed" );
while( $i < $count ){
$url = $prefix . $i . ".htm" ;
echo "Get " . $url . "..." ;
$url_str = get_content_url ( get_url ( $url ));
echo " OK/n" ;
fwrite ( $fp , $url_str );
++ $i ;
}
fclose ( $fp );
}
//獲取目標(biāo)多媒體對象
function get_object ( $url_file , $save_file , $split = "|--:**:--|" ){
if (! file_exists ( $url_file )) die( $url_file . " not exist" );
$file_arr = file ( $url_file );
if (! is_array ( $file_arr ) || empty( $file_arr )) die( $url_file . " not content" );
$url_arr = array_unique ( $file_arr );
if ( file_exists ( $save_file )) @ unlink ( $save_file );
$fp = fopen ( $save_file , "a+" ) or die( "Open save file " . $save_file . " failed" );
foreach( $url_arr as $url ){
if (empty( $url )) continue;
echo "Get " . $url . "..." ;
$html_str = get_url ( $url );
echo $html_str ;
echo $url ;
exit;
$obj_str = get_content_object ( $html_str );
echo " OK/n" ;
fwrite ( $fp , $obj_str );
}
fclose ( $fp );
}
//遍歷目錄獲取文件內(nèi)容
function get_dir ( $save_file , $dir ){
$dp = opendir ( $dir );
if ( file_exists ( $save_file )) @ unlink ( $save_file );
$fp = fopen ( $save_file , "a+" ) or die( "Open save file " . $save_file . " failed" );
while(( $file = readdir ( $dp )) != false ){
if ( $file != "." && $file != ".." ){
echo "Read file " . $file . "..." ;
$file_content = file_get_contents ( $dir . $file );
$obj_str = get_content_object ( $file_content );
echo " OK/n" ;
fwrite ( $fp , $obj_str );
}
}
fclose ( $fp );
}
//獲取指定url內(nèi)容
function get_url ( $url ){
$reg = '/^http:////[^//].+$/' ;
if (! preg_match ( $reg , $url )) die( $url . " invalid" );
$fp = fopen ( $url , "r" ) or die( "Open url: " . $url . " failed." );
while( $fc = fread ( $fp , 8192 )){
$content .= $fc ;
}
fclose ( $fp );
if (empty( $content )){
die( "Get url: " . $url . " content failed." );
}
return $content ;
}
//使用socket獲取指定網(wǎng)頁
function get_content_by_socket ( $url , $host ){
$fp = fsockopen ( $host , 80 ) or die( "Open " . $url . " failed" );
$header = "GET /" . $url . " HTTP/1.1/r/n" ;
$header .= "Accept: */*/r/n" ;
$header .= "Accept-Language: zh-cn/r/n" ;
$header .= "Accept-Encoding: gzip, deflate/r/n" ;
$header .= "User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Maxthon; InfoPath.1; .NET CLR 2.0.50727)/r/n" ;
$header .= "Host: " . $host . "/r/n" ;
$header .= "Connection: Keep-Alive/r/n" ;
//$header .= "Cookie: cnzz02=2; rtime=1; ltime=1148456424859; cnzz_eid=56601755-/r/n/r/n";
$header .= "Connection: Close/r/n/r/n" ;
fwrite ( $fp , $header );
while (! feof ( $fp )) {
$contents .= fgets ( $fp , 8192 );
}
fclose ( $fp );
return $contents ;
}
//獲取指定內(nèi)容里的url
function get_content_url ( $host_url , $file_contents ){
//$reg = '/^(#|<a href="http://lib.csdn.net/base/18" class='replace_word' title="JavaScript知識庫" target='_blank' style='color:#df3434; font-weight:bold;'>JavaScript</a>.*?|ftp:////.+|http:////.+|.*?href.*?|play.*?|index.*?|.*?asp)+$/i';
//$reg = '/^(down.*?/.html|/d+_/d+/.htm.*?)$/i';
$rex = "/([hH][rR][eE][Ff])/s*=/s*['/"]*([^>'/"/s]+)[/"'>]*/s*/i" ;
$reg = '/^(down.*?/.html)$/i' ;
preg_match_all ( $rex , $file_contents , $r );
$result = "" ; //array();
foreach( $r as $c ){
if ( is_array ( $c )){
foreach( $c as $d ){
if ( preg_match ( $reg , $d )){ $result .= $host_url . $d . "/n" ; }
}
}
}
return $result ;
}
//獲取指定內(nèi)容中的多媒體文件
function get_content_object ( $str , $split = "|--:**:--|" ){
$regx = "/href/s*=/s*['/"]*([^>'/"/s]+)[/"'>]*/s*(.*?<//b>)/i" ;
preg_match_all ( $regx , $str , $result );
if ( count ( $result ) == 3 ){
$result [ 2 ] = str_replace ( "多媒體： " , "" , $result [ 2 ]);
$result [ 2 ] = str_replace ( " " , "" , $result [ 2 ]);
$result = $result [ 1 ][ 0 ] . $split . $result [ 2 ][ 0 ] . "/n" ;
}
return $result ;
}
?>
Readers who are interested in more PHP-related content can check out the special topics of this site: "Summary of PHP Regular Expression Usage", "Summary of PHP Ajax Skills and Applications", "Summary of PHP Operations and Operator Usage", "PHP Network Summary of Programming Skills", "Introduction Tutorial on PHP Basic Syntax", "Summary of PHP Office Document Operation Skills (Including Word, Excel, Access, PPT)", "Summary of PHP Date and Time Usage", "Introduction Tutorial on PHP Object-Oriented Programming" , "php string (string) usage summary", "php mysql database operation introductory tutorial" and "php common database operation skills summary"
I hope this article will be helpful to everyone in PHP programming.
Hot AI Tools
Undress AI Tool
Undress images for free
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Clothoff.io
AI clothes remover
Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
How to get the current session ID in PHP?
Jul 13, 2025 am 03:02 AM
The method to get the current session ID in PHP is to use the session_id() function, but you must call session_start() to successfully obtain it. 1. Call session_start() to start the session; 2. Use session_id() to read the session ID and output a string similar to abc123def456ghi789; 3. If the return is empty, check whether session_start() is missing, whether the user accesses for the first time, or whether the session is destroyed; 4. The session ID can be used for logging, security verification and cross-request communication, but security needs to be paid attention to. Make sure that the session is correctly enabled and the ID can be obtained successfully.
PHP get substring from a string
Jul 13, 2025 am 02:59 AM
To extract substrings from PHP strings, you can use the substr() function, which is syntax substr(string$string,int$start,?int$length=null), and if the length is not specified, it will be intercepted to the end; when processing multi-byte characters such as Chinese, you should use the mb_substr() function to avoid garbled code; if you need to intercept the string according to a specific separator, you can use exploit() or combine strpos() and substr() to implement it, such as extracting file name extensions or domain names.
How do you perform unit testing for php code?
Jul 13, 2025 am 02:54 AM
UnittestinginPHPinvolvesverifyingindividualcodeunitslikefunctionsormethodstocatchbugsearlyandensurereliablerefactoring.1)SetupPHPUnitviaComposer,createatestdirectory,andconfigureautoloadandphpunit.xml.2)Writetestcasesfollowingthearrange-act-assertpat
How to split a string into an array in PHP
Jul 13, 2025 am 02:59 AM
In PHP, the most common method is to split the string into an array using the exploit() function. This function divides the string into multiple parts through the specified delimiter and returns an array. The syntax is exploit(separator, string, limit), where separator is the separator, string is the original string, and limit is an optional parameter to control the maximum number of segments. For example $str="apple,banana,orange";$arr=explode(",",$str); The result is ["apple","bana
JavaScript Data Types: Primitive vs Reference
Jul 13, 2025 am 02:43 AM
JavaScript data types are divided into primitive types and reference types. Primitive types include string, number, boolean, null, undefined, and symbol. The values are immutable and copies are copied when assigning values, so they do not affect each other; reference types such as objects, arrays and functions store memory addresses, and variables pointing to the same object will affect each other. Typeof and instanceof can be used to determine types, but pay attention to the historical issues of typeofnull. Understanding these two types of differences can help write more stable and reliable code.
Using std::chrono in C
Jul 15, 2025 am 01:30 AM
std::chrono is used in C to process time, including obtaining the current time, measuring execution time, operation time point and duration, and formatting analysis time. 1. Use std::chrono::system_clock::now() to obtain the current time, which can be converted into a readable string, but the system clock may not be monotonous; 2. Use std::chrono::steady_clock to measure the execution time to ensure monotony, and convert it into milliseconds, seconds and other units through duration_cast; 3. Time point (time_point) and duration (duration) can be interoperable, but attention should be paid to unit compatibility and clock epoch (epoch)
How to pass a session variable to another page in PHP?
Jul 13, 2025 am 02:39 AM
In PHP, to pass a session variable to another page, the key is to start the session correctly and use the same $_SESSION key name. 1. Before using session variables for each page, it must be called session_start() and placed in the front of the script; 2. Set session variables such as $_SESSION['username']='JohnDoe' on the first page; 3. After calling session_start() on another page, access the variables through the same key name; 4. Make sure that session_start() is called on each page, avoid outputting content in advance, and check that the session storage path on the server is writable; 5. Use ses
How does PHP handle Environment Variables?
Jul 14, 2025 am 03:01 AM
ToaccessenvironmentvariablesinPHP,usegetenv()orthe$_ENVsuperglobal.1.getenv('VAR_NAME')retrievesaspecificvariable.2.$_ENV['VAR_NAME']accessesvariablesifvariables_orderinphp.iniincludes"E".SetvariablesviaCLIwithVAR=valuephpscript.php,inApach
