Scrapy Jobs

I’ve created a Google Sheet that will act as a central “lead collector,” and I need it filled with fresh, accurate contact data pulled from publicly available sites. The focus is simple yet crucial: for every company you find, capture the homepage URL and a working email address. (ask for details in the sheet ) A completely ethical approach is non-negotiable—no gated content, no third-party lists, and no automated harvesting that violates site terms. I’m happy for you to use tools you’re comfortable with (Python, Scrapy, BeautifulSoup, Selenium, Google Apps Script, etc.) as long as you respect and rate limits. Email addresses must appear in plain text within the sheet; please avoid hyperlinks or HTML encoding. Deliverables • A Google Sheet ...

I need a robust scraping solution that continuously pulls price, full product descriptions, and customer reviews for electronics across roughly ten different e-commerce sites. Each time the script runs, the results should append to a master Excel workbook so that every entry is stored chronologically—allowing me to compare today’s prices with yesterday’s and build long-term trend charts without any manual work. Key expectations • The scraper must visit all target URLs, handle pagination or lazy-loaded content where it exists, and respect each site’s structure. • Collected fields: date/time stamp, site name, product name, current price, full description text, average rating, review count, and a link to the product page. • Excel output: one sheet...

I need an end-to-end AI agent that automatically scouts freelancing websites, general job boards, and specialised training platforms for roles or courses that involve artificial-intelligence work. The agent must: • Crawl and scrape the relevant pages in real time or on a frequent schedule. • Apply NLP or other classification techniques to decide whether a posting is truly AI-related, then tag it by sub-domain (e.g. vision, NLP, MLOps, prompt-engineering). • Deliver concise, deduplicated listings to me through an in-app notification feed—no email or SMS required. For the deployment side I’m open to Python (Scrapy, BeautifulSoup, Selenium), Node, or any stack you are comfortable with so long as it is containerised and can run unattended on a small cloud insta...

Hey! I’m looking to hire an experienced developer to build a universal product-detail scraping pipeline that takes a product URL (any website) and returns a complete structured product record. This is not a “simple HTML parse.” Many target sites are React/Next/Vue, load content via XHR/GraphQL, hide details behind tabs/accordions/modals, and lazy-load images/PDFs. The solution needs to reliably extract everything a human can see on the page, plus the underlying data used to render it. What the scraper must do (high level) Given a product URL, the pipeline should: Load the page like a real user (handle cookies/overlays). Capture all content from multiple sources (DOM + network + interactions). Use GPT API strategically to increase accuracy (field mapping, variant ext...

I need every product on copied into my existing WooCommerce shop so the catalog mirrors theirs one-to-one. That means grabbing each item’s title, plain-text description and all associated images, then pushing them into WordPress with the correct categories, colour swatches and size variations exactly as they appear on Furnx. Only product details are required right now—reviews and live stock counts can wait for a later phase—so the job is focused on clean data capture and a flawless import workflow. Descriptions must remain in plain text; no extra HTML markup. Images should arrive attached to the right variation, including separate gallery shots where available, and the colour options need to show as clickable swatches in WooCommerce, not just text labels. I’m...

I have two source spreadsheets that I need merged and enriched through automated scraping: • “File 1” – 170 k Spanish local businesses with emails • “File 2” – 65 k additional businesses with websites only Phase 1 – Email extraction Using a Python script and well-known libraries (requests, BeautifulSoup, Scrapy or similar), scan every site listed in File 2, capture all working email addresses you can locate, then append them to the corresponding rows so I can produce a unified “File 3”. Phase 2 – Offer harvesting Next, visit each live site in File 3. Where an offer, deal or promotion is publicly displayed, record the details in a fresh Excel sheet with these exact columns: Business ID | Business Name | Offer...

Project Description: Find school districts and charter schools who use a specific vendor for a large list of domains. I am seeking an experienced web scraping specialist to improve our Python script to analyze a large list of school district websites (approximately 4000+ URLs) and identify the ones who show a specific link on any page found in their sitemap. The primary method of identification must be to scan the website's for specific, known vendor links. Deliverables Required 1. A Production-Ready Python Script (.py file): The script must be commented, easily configurable, and capable of reading the provided CSV list, performing the scan, and generating the output CSV. It should handle timeouts and basic error handling gracefully. 2. The Final Results (CSV/Excel File): A c...

I'm looking for an experienced Python developer to create an algorithmic trading bot based on my own unique strategy for options trading. The bot should incorporate the following key functionalities: - Backtesting: Ability to test the strategy against historical data. - Real-time data feed: Integrate with reliable data sources for live market updates. - Automatic order execution: Execute trades automatically based on the strategy signals. Ideal Skills and Experience: - Proficiency in Python and experience with trading libraries. - Strong understanding of options trading and market dynamics. - Experience in developing and backtesting trading algorithms. - Familiarity with integrating real-time data feeds and order execution systems. Please provide examples of previous work and relev...

I need every public phone number that appears on gathered into a single, well-structured Excel workbook. Please crawl the entire site, not just a few sections, and return each number alongside the key profile details that make the data usable at a glance—name, profile URL, and any other easily captured identifiers shown next to the number. A clean .xlsx with one row per profile, no duplicates, and clearly labelled columns is the only deliverable I’m expecting. If you prefer Python, Scrapy, Selenium, Beautiful Soup or a comparable stack, go ahead; I’m interested in results, not the specific toolset, as long as the script can be rerun later should the site content change. Before delivery, double-check that: • every row contains a valid phone number and url • n...

I need a senior-level specialist to harvest product data from several e-commerce sites and deliver it in a single, well-structured CSV file. The task demands production-ready techniques—think Scrapy spiders hardened with rotating proxies, Selenium or Playwright for dynamic content, and solid anti-bot countermeasures. The information I’m after is very specific: product names, prices, pictures, and SKU. Nothing less, nothing more. Your solution must run reliably at scale, cope with frequent layout changes, and leave no trace that could trigger blocks. Python is the preferred stack, but if you have a proven alternative that meets the same bar, I’m open to hearing it. To be considered, include in your proposal: • At least one example of a comparable e-commerce scrapi...

I’m expanding our Florida outreach list and need a reliable web-scraped data set of school, college, and university administrators who oversee Nursing or other Healthcare programs. You’ll pull the information directly from two source types only—official institution websites and reputable educational directories—so every entry must be traceable back to one of those pages. Here’s exactly what must land in the spreadsheet: • Institution name • Contact’s first and last name • Job title (Administrator, Director of Nursing, CTE Healthcare lead, etc.) • Verified email address • State (always Florida) Format & delivery – Send the file in Excel (.xlsx). – First progress drop: within 5 days so I can spot-c...

Je veux constituer un annuaire exhaustif de toutes les bases nautiques en France métropolitaine et DOM-TOM offrant jetski, bouée tractée, flyboard, ski nautique, wakeboard, parachute ascensionnel ou encore location de bateaux. Pour y parvenir, j’ai besoin d’un workflow automatisé reposant exclusivement sur Google Maps – c’est la source retenue – capable de collecter, dédupliquer puis nettoyer les données avant de les mettre en forme dans un CSV directement importable dans le CMS Wix. Les coordonnées devront être fournies en degrés décimaux. Livrables attendus • Un script réutilisable (Python + Selenium, Scrapy ou équivalent) qui interroge Google Maps, gère le r...

Proyecto: API / Sistema de búsqueda de productos en múltiples tiendas (Texto + Foto tipo Lens) Estoy buscando un desarrollador freelance con experiencia en web scraping, desarrollo de APIs y reconocimiento de productos por imagen, para crear un sistema que permita buscar un producto en múltiples tiendas online al mismo tiempo. Objetivo Desarrollar una API (o sistema backend) donde yo pueda buscar productos de dos maneras: 1) Búsqueda por texto Ej: “iPhone 15 Pro 256GB” / “Zapatillas Nike Air Force blancas” 2) Búsqueda por imagen (tipo Google Lens) Subir una foto del producto y que el sistema intente identificarlo (marca/modelo) y luego busque coincidencias en las tiendas. El sistema debe consultar automáticamente una lista...