我抓取受密碼保護(hù)的網(wǎng)站的步驟：

1. 擷取 HTML 表單元素：使用者名稱 ID、密碼 ID 和登入按鈕類別
2. - 使用 requests 或 Selenium 等工具自動(dòng)登入：填入使用者名，等待，填入密碼，等待，點(diǎn)選登入
3. - 儲(chǔ)存會(huì)話 cookie 以進(jìn)行身份驗(yàn)證
4. - 繼續(xù)抓取經(jīng)過(guò)驗(yàn)證的頁(yè)面

免責(zé)聲明：我已在 https://www.scrapewebapp.com/ 上為此特定用例建立了一個(gè) API。因此，如果您想快速完成它，請(qǐng)使用它，否則請(qǐng)繼續(xù)閱讀。

讓我們使用這個(gè)範(fàn)例：假設(shè)我想從我的帳戶 https://www.scrapewebapp.com/ 中抓取我自己的 API 金鑰。在此頁(yè)面：https://app.scrapewebapp.com/account/api_key

1. 登入頁(yè)面

首先，您需要找到登入頁(yè)面。如果您嘗試造訪登入後的頁(yè)面，大多數(shù)網(wǎng)站都會(huì)為您重新導(dǎo)向303，因此如果您嘗試直接抓取https://app.scrapewebapp.com/account/api_key，您將自動(dòng)取得登入頁(yè)面https:// app.scrapewebapp.com/login。因此，如果尚未提供，這是自動(dòng)查找登入頁(yè)面的好方法。

好的，現(xiàn)在我們有了登入頁(yè)面，我們需要找到新增使用者名稱或電子郵件以及密碼和實(shí)際登入按鈕的位置。最好的方法是建立一個(gè)簡(jiǎn)單的腳本，使用類型「電子郵件」、「使用者名稱」、「密碼」來(lái)尋找輸入的 ID，並尋找類型為「提交」的按鈕。我在下面為您編寫(xiě)了程式碼：

from bs4 import BeautifulSoup


def extract_login_form(html_content: str):
    """
    Extracts the login form elements from the given HTML content and returns their CSS selectors.
    """
    soup = BeautifulSoup(html_content, "html.parser")

    # Finding the username/email field
    username_email = (
        soup.find("input", {"type": "email"})
        or soup.find("input", {"name": "username"})
        or soup.find("input", {"type": "text"})
    )  # Fallback to input type text if no email type is found

    # Finding the password field
    password = soup.find("input", {"type": "password"})

    # Finding the login button
    # Searching for buttons/input of type submit closest to the password or username field
    login_button = None

    # First try to find a submit button within the same form
    if password:
        form = password.find_parent("form")
        if form:
            login_button = form.find("button", {"type": "submit"}) or form.find(
                "input", {"type": "submit"}
            )
    # If no button is found in the form, fall back to finding any submit button
    if not login_button:
        login_button = soup.find("button", {"type": "submit"}) or soup.find(
            "input", {"type": "submit"}
        )

    # Extracting CSS selectors
    def generate_css_selector(element, element_type):
        if "id" in element.attrs:
            return f"#{element['id']}"
        elif "type" in element.attrs:
            return f"{element_type}[type='{element['type']}']"
        else:
            return element_type

    # Generate CSS selectors with the updated logic
    username_email_css_selector = None
    if username_email:
        username_email_css_selector = generate_css_selector(username_email, "input")

    password_css_selector = None
    if password:
        password_css_selector = generate_css_selector(password, "input")

    login_button_css_selector = None
    if login_button:
        login_button_css_selector = generate_css_selector(
            login_button, "button" if login_button.name == "button" else "input"
        )

    return username_email_css_selector, password_css_selector, login_button_css_selector


def main(html_content: str):
    # Call the extract_login_form function and return its result
    return extract_login_form(html_content)

2。使用 Selenium 實(shí)際登入

現(xiàn)在您需要建立一個(gè) selenium webdriver。我們將使用 chrome headless 來(lái)透過(guò) Python 運(yùn)行它。安裝方法如下：

# Install selenium and chromium

!pip install selenium
!apt-get update 
!apt install chromium-chromedriver

!cp /usr/lib/chromium-browser/chromedriver /usr/bin
import sys
sys.path.insert(0,'/usr/lib/chromium-browser/chromedriver')

然後實(shí)際登入我們的網(wǎng)站並儲(chǔ)存 cookie。我們將保存所有 cookie，但您只能根據(jù)需要儲(chǔ)存身分驗(yàn)證 cookie。

# Imports
from selenium import webdriver
from selenium.webdriver.common.by import By
import requests
import time

# Set up Chrome options
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')

# Initialize the WebDriver
driver = webdriver.Chrome(options=chrome_options)

try:
    # Open the login page
    driver.get("https://app.scrapewebapp.com/login")

    # Find the email input field by ID and input your email
    email_input = driver.find_element(By.ID, "email")
    email_input.send_keys("******@gmail.com")

    # Find the password input field by ID and input your password
    password_input = driver.find_element(By.ID, "password")
    password_input.send_keys("*******")

    # Find the login button and submit the form
    login_button = driver.find_element(By.CSS_SELECTOR, "button[type='submit']")
    login_button.click()

    # Wait for the login process to complete
    time.sleep(5)  # Adjust this depending on your site's response time


finally:
    # Close the browser
    driver.quit()

3. 儲(chǔ)存 Cookie

就像透過(guò) driver.getcookies() 函數(shù)將它們保存到字典中一樣簡(jiǎn)單。

def save_cookies(driver):
    """Save cookies from the Selenium WebDriver into a dictionary."""
    cookies = driver.get_cookies()
    cookie_dict = {}
    for cookie in cookies:
        cookie_dict[cookie['name']] = cookie['value']
    return cookie_dict

從 WebDriver 儲(chǔ)存 cookie

cookie = save_cookies(驅(qū)動(dòng)程式)

4. 從我們登入的會(huì)話中取得數(shù)據(jù)

在這部分中，我們將使用簡(jiǎn)單的庫(kù)請(qǐng)求，但您也可以繼續(xù)使用 selenium。

現(xiàn)在我們想從此頁(yè)面取得實(shí)際的 API：https://app.scrapewebapp.com/account/api_key。

因此，我們從請(qǐng)求庫(kù)建立一個(gè)會(huì)話並將每個(gè) cookie 新增到其中。然後請(qǐng)求 URL 並列印回應(yīng)文字。

def scrape_api_key(cookies):
    """Use cookies to scrape the /account/api_key page."""
    url = 'https://app.scrapewebapp.com/account/api_key'

    # Set up the session to persist cookies
    session = requests.Session()

    # Add cookies from Selenium to the requests session
    for name, value in cookies.items():
        session.cookies.set(name, value)

    # Make the request to the /account/api_key page
    response = session.get(url)

    # Check if the request is successful
    if response.status_code == 200:
        print("API Key page content:")
        print(response.text)  # Print the page content (could contain the API key)
    else:
        print(f"Failed to retrieve API key page, status code: {response.status_code}")

5. 取得您想要的實(shí)際數(shù)據(jù)（獎(jiǎng)勵(lì)）

我們得到了我們想要的頁(yè)面文本，但是有很多我們不關(guān)心的數(shù)據(jù)。我們只想要 api_key。

最好、最簡(jiǎn)單的方法是使用像 ChatGPT（GPT4o 模型）這樣的人工智慧。

這樣提示模型：「您是專家抓取工具，您只會(huì)提取從上下文中詢問(wèn)的資訊。我需要來(lái)自 {context} 的 api-key 值」

from bs4 import BeautifulSoup


def extract_login_form(html_content: str):
    """
    Extracts the login form elements from the given HTML content and returns their CSS selectors.
    """
    soup = BeautifulSoup(html_content, "html.parser")

    # Finding the username/email field
    username_email = (
        soup.find("input", {"type": "email"})
        or soup.find("input", {"name": "username"})
        or soup.find("input", {"type": "text"})
    )  # Fallback to input type text if no email type is found

    # Finding the password field
    password = soup.find("input", {"type": "password"})

    # Finding the login button
    # Searching for buttons/input of type submit closest to the password or username field
    login_button = None

    # First try to find a submit button within the same form
    if password:
        form = password.find_parent("form")
        if form:
            login_button = form.find("button", {"type": "submit"}) or form.find(
                "input", {"type": "submit"}
            )
    # If no button is found in the form, fall back to finding any submit button
    if not login_button:
        login_button = soup.find("button", {"type": "submit"}) or soup.find(
            "input", {"type": "submit"}
        )

    # Extracting CSS selectors
    def generate_css_selector(element, element_type):
        if "id" in element.attrs:
            return f"#{element['id']}"
        elif "type" in element.attrs:
            return f"{element_type}[type='{element['type']}']"
        else:
            return element_type

    # Generate CSS selectors with the updated logic
    username_email_css_selector = None
    if username_email:
        username_email_css_selector = generate_css_selector(username_email, "input")

    password_css_selector = None
    if password:
        password_css_selector = generate_css_selector(password, "input")

    login_button_css_selector = None
    if login_button:
        login_button_css_selector = generate_css_selector(
            login_button, "button" if login_button.name == "button" else "input"
        )

    return username_email_css_selector, password_css_selector, login_button_css_selector


def main(html_content: str):
    # Call the extract_login_form function and return its result
    return extract_login_form(html_content)

如果您想要一個(gè)簡(jiǎn)單可靠的 API 來(lái)實(shí)現(xiàn)這一切，請(qǐng)嘗試我的新產(chǎn)品 https://www.scrapewebapp.com/

如果你喜歡這篇文章，請(qǐng)給我鼓掌並關(guān)注我。確實(shí)有很大幫助！

以上是如何使用 Selenium 抓取受登入保護(hù)的網(wǎng)站（逐步指南）的詳細(xì)內(nèi)容。更多資訊請(qǐng)關(guān)注PHP中文網(wǎng)其他相關(guān)文章！