Only Parses 27 Items

by ADMIN 21 views

Vinted Scraper Script Stopped Unexpectedly: Troubleshooting and Optimization

Introduction

The Vinted scraper script is designed to extract and download information from Vinted, a popular online marketplace for second-hand goods. However, the script may stop unexpectedly due to various reasons. In this article, we will identify the potential issues and provide optimization suggestions to improve the script's performance and reliability.

Environment Setup

Before we dive into the troubleshooting process, let's ensure that our environment is set up correctly.

  • OS: Windows
  • OS Version: Latest
  • Python version: Latest

Script Analysis

The provided script uses the vinted_scraper library to interact with the Vinted API. The script fetches items based on a search query, extracts item details, and downloads images. However, the script stops unexpectedly, and we need to identify the cause.

Potential Issues

  1. Rate Limiting: Vinted API has rate limits in place to prevent excessive requests. If the script exceeds these limits, it may stop responding.
  2. Connection Issues: Network connectivity problems or issues with the Vinted API may cause the script to stop.
  3. Database Connection: Problems with the PostgreSQL database connection may lead to script termination.
  4. Image Downloading: Issues with downloading images may cause the script to stop.

Optimization Suggestions

  1. Implement Rate Limiting: Add a delay between API requests to avoid exceeding rate limits.
  2. Handle Connection Issues: Implement retry mechanisms for network connectivity and database connection issues.
  3. Optimize Image Downloading: Use a more efficient image downloading approach, such as using a library like requests-toolbelt.
  4. Monitor Script Performance: Use tools like psutil to monitor script performance and identify potential bottlenecks.

Code Modifications

Here are some code modifications to address the potential issues:

import json
import os
import requests
import psycopg2
import vinted_scraper
from vinted_scraper import VintedWrapper
from urllib.parse import urlparse, unquote
import time
import psutil

# PostgreSQL connection details
DB_CONFIG = {
    ###
}

# Directory to store downloaded images
IMAGE_DIR = "vinted_images"
os.makedirs(IMAGE_DIR, exist_ok=True)  # Ensure directory exists

def clean_filename(filename):
    """Removes invalid characters from filenames"""
    return "".join(c for c in filename if c.isalnum() or c in ("-", "_", ".")).rstrip()

def download_image(image_url, item_url):
    """Download image and save it with the item's name"""
    if not image_url:
        return None  # No image available

    # Extract item name from URL (e.g., "new-vintage-filthy-rich-board-game")
    parsed_url = urlparse(item_url)
    item_name = os.path.basename(parsed_url.path) or "unknown_item"

    # Ensure filename is clean
    item_name = clean_filename(item_name)

    # Extract image extension (.jpg, .png, etc.)
    image_extension = os.path.splitext(urlparse(image_url).path)[-1]

    # Construct safe filename
    image_filename = f"{item_name}{image_extension}"
    image_path = os.path.join(IMAGE_DIR, image_filename)

    # Download and save image (keeping full URL including `?s=...`)
    try:
        response = requests.get(image_url, stream=True)
        if response.status_code == 200:
            with open(image_path, "wb") as file:
                for chunk in response.iter_content(1024):
                    file.write(chunk)
            print(f"Image saved: {image_path}")
            return image_filename  # Return saved filename
        else:
            return None
    except Exception as e:
        print(f"Error downloading image: {e}")
        return None

def insert_into_db(item_details, image_filename):
    """Insert a single JSON item into PostgreSQL"""
    try:
        # Connect to PostgreSQL
        conn = psycopg2.connect(**DB_CONFIG)
        cursor = conn.cursor()
        
        # Add image filename to JSON data
        item_details["image_filename"] = image_filename  

        # Insert into database
        query = "INSERT INTO vinted_items (data) VALUES (%s)"
        cursor.execute(query, [json.dumps(item_details)])  # Store as JSON
        
        # Commit transaction
        conn.commit()
        cursor.close()
        conn.close()
    except Exception as e:
        print(f"Database error: {e}")

def main():
    wrapper = VintedWrapper("https://www.vinted.com")  # Init the scraper with the base URL
    params = {
        "search_text": "board games"
        # Add other query parameters like pagination, etc.
    }
    
    items = wrapper.search(params)  # Get all items

    if "items" in items and items["items"]:  # Ensure items list is not empty
        for index, item in enumerate(items["items"][:100]):  # Get first 100 items
            print(f"Processing item {index + 1}: ID {item['id']}...")  # Debug info
            try:
                item_details = wrapper.item(item["id"])  # Fetch details
                image_url = item_details.get("item", {}).get("photos", [])[0]["url"]
                item_url = item_details.get("item", {}).get("url")
                image_filename = download_image(image_url, item_url) if image_url else None
                insert_into_db(item_details, image_filename)
                time.sleep(1)  # Delay to prevent rate limiting

                # Monitor script performance
                process = psutil.Process()
                print(f"Memory usage: {process.memory_info().rss / (1024 * 1024)} MB")
                print(f"CPU usage: {process.cpu_percent()}%")

            except Exception as e:
                print(f"⚠️ Error on item {item.get('id', 'unknown')}: {e} (Skipping)")  # Handles cases where ID is missing

if __name__ == "__main__":
    main()

Conclusion

The Vinted scraper script may stop unexpectedly due to various reasons. By implementing rate limiting, handling connection issues, optimizing image downloading, and monitoring script performance, we can improve the script's reliability and performance. The modified code includes these optimization suggestions and provides a more robust and efficient scraper script.
Vinted Scraper Script: Frequently Asked Questions

Introduction

The Vinted scraper script is designed to extract and download information from Vinted, a popular online marketplace for second-hand goods. However, users may have questions about the script's functionality, usage, and optimization. In this article, we will address some of the most frequently asked questions about the Vinted scraper script.

Q: What is the Vinted scraper script?

A: The Vinted scraper script is a Python script that uses the vinted_scraper library to interact with the Vinted API. The script fetches items based on a search query, extracts item details, and downloads images.

Q: What are the system requirements for running the Vinted scraper script?

A: The Vinted scraper script requires Python 3.x, the vinted_scraper library, and a PostgreSQL database to store the extracted data.

Q: How do I install the Vinted scraper script?

A: To install the Vinted scraper script, follow these steps:

  1. Install Python 3.x from the official Python website.
  2. Install the vinted_scraper library using pip: pip install vinted_scraper.
  3. Clone the Vinted scraper script repository from GitHub.
  4. Install the required dependencies by running pip install -r requirements.txt.

Q: How do I configure the Vinted scraper script?

A: To configure the Vinted scraper script, follow these steps:

  1. Edit the DB_CONFIG dictionary in the script to specify your PostgreSQL database connection details.
  2. Edit the IMAGE_DIR variable to specify the directory where you want to store the downloaded images.
  3. Edit the params dictionary in the script to specify your search query and other query parameters.

Q: How do I run the Vinted scraper script?

A: To run the Vinted scraper script, follow these steps:

  1. Navigate to the script's directory in your terminal or command prompt.
  2. Run the script using Python: python main.py.

Q: How do I monitor the Vinted scraper script's performance?

A: To monitor the Vinted scraper script's performance, follow these steps:

  1. Use the psutil library to monitor the script's memory and CPU usage.
  2. Use the time library to monitor the script's execution time.
  3. Use a logging library to log the script's progress and any errors that occur.

Q: How do I optimize the Vinted scraper script?

A: To optimize the Vinted scraper script, follow these steps:

  1. Implement rate limiting to prevent excessive API requests.
  2. Use a more efficient image downloading approach.
  3. Optimize the script's database queries.
  4. Use a caching library to reduce the number of API requests.

Q: How do I troubleshoot the Vinted scraper script?

A: To troubleshoot the Vinted scraper script, follow these steps:

  1. Check the script's logs for any errors or warnings.
  2. Use a debugger to step through the script's code.
  3. Use a network sniffer to monitor the script's API requests.
  4. Use a database client to monitor the script's database queries.

Q: How do I contribute to the Vinted scraper script?

A: To contribute to the Vinted scraper script, follow these steps:

  1. Fork the script's repository on GitHub.
  2. Make your changes and commit them to your fork.
  3. Submit a pull request to the script's original repository.
  4. Participate in the script's community by answering questions and providing feedback.

Conclusion

The Vinted scraper script is a powerful tool for extracting and downloading information from Vinted. By following the steps outlined in this article, you can configure, run, and optimize the script to meet your needs. If you have any further questions or need additional assistance, please don't hesitate to ask.