
In Progress
Posted
Paid on delivery
I need a fully automated pipeline that fetches fresh data from specific Facebook assets and a public website twice a week and drops the results into a structured destination I can immediately work with. Scope • Facebook: once the job starts I will supply a short list of page, group and/or profile URLs. The solution should pull all visible posts, their comments, basic engagement metrics and any media links for each run. Please stay within Facebook’s Terms of Service and use the official Graph API wherever possible (access tokens will be provided). • Website: a single-domain crawl that captures several predefined sections. I will point you to the exact pages; think product blocks, review snippets and article text. Automation cadence Run every Monday and Thursday (UTC). A simple cron, GitHub Action, Cloud Function or comparable scheduler is fine as long as it is hands-off for me. Output CSV is preferred, with each run stored in its own timestamped folder in Google Drive. If a small SQLite or Postgres database is easier for deduping, feel free to propose it. Deliverables - Clean, well-commented code (Python is ideal, open to Node/Go) - Setup instructions and a one-click deployment script or Dockerfile - A short README outlining field mappings and runtime steps - Basic logging so I can verify each execution Acceptance criteria 1. Pipeline runs unattended on the agreed schedule for one full week during hand-off. 2. Each target record appears exactly once per run, with no missing fields. 3. Any rate-limit or auth error is reported in the log and does not crash the job. Please let me know which libraries or scraping tools you plan to use and any questions you have about access.
Project ID: 40753875
101 proposals
Remote project
Active 1 min ago
Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs

We can deliver this better then anyone,while meeting all of your requirements. Maintenance, upgrades,fixes everything will be taken care of proper best for optimizing site for future growth, while maintaining low cost.
$100 USD in 30 days
3.1
3.1
101 freelancers are bidding on average $33 USD for this job

Hello, Leveraging my 15+ years of experience, I am confident in delivering an automated pipeline that fetches and structures data from Facebook and websites in accordance with your specific needs. My proficiency in Python and PHP, along with expertise in web scraping tools such as Beautiful Soup and Selenium, will enable me to create a bi-weekly automated script that adheres to Facebook's Terms of Service while ensuring all data fields are extracted accurately. Given your preference for storing each run's output in its own timestamped folder in Google Drive, my proficiency in content management systems like WordPress and Shopify equips me well. Furthermore, since you're open to using a small SQLite or Postgres database for deduping, I can leverage my solid understanding of database management to provide an optimal solution. In summary, from the hand-off phase to ensuring smooth unattended pipeline runs on the agreed schedule, I am equipped with the necessary skills and experience to provide effective yet efficient solutions. Let's collaborate to build a system that will fetch you fresh data without any hiccups! Hire me now and let's build something great together! Thanks!
$30 USD in 1 day
8.6
8.6

Hello I checked your project and the main point is an unattended Python pipeline that collects Facebook and website data twice weekly, then stores clean, timestamped CSV files in Google Drive. In the past I built data extraction systems using Scrapy, Selenium, Puppeteer, and Python, including large-scale Google Maps scraping with deduplication and structured output. I would use the official Facebook Graph API wherever the supplied permissions support it, with Scrapy or Playwright for the public website sections. I’ll add field validation, run-level deduplication, Google Drive upload, structured logging, and Docker or a deployment script for easy setup. The scheduler can run through GitHub Actions, Cloud Functions, or cron, while rate-limit and authentication errors will be recorded clearly without stopping unrelated processing. Could you confirm which Facebook asset types and Graph API permissions will be available, and whether the website content is server-rendered or JavaScript-loaded? Best regards
$20 USD in 3 days
7.7
7.7

Hi there, I hope this proposal finds you well. I am excited about the opportunity to work on your bi-weekly Facebook and website scraper project. With extensive experience in PHP, Python, and data processing, I am well-equipped to deliver a robust and efficient solution that meets your needs. I understand the importance of adhering to Facebook's Terms of Service, and I plan to leverage the Facebook Graph API to ensure compliance while extracting posts, comments, engagement metrics, and media links. For the website crawl, I will use Python's BeautifulSoup and Scrapy to navigate the specified sections and gather the necessary data efficiently. To automate the process, I propose using a cron job or a similar scheduler such as GitHub Actions to ensure the pipeline runs seamlessly every Monday and Thursday. The output will be structured in CSV format, and each run will be stored in a timestamped folder on Google Drive. I am also open to implementing an SQLite or Postgres database for deduplication if needed. The deliverables will include clean, well-commented code, a Dockerfile for easy deployment, and a README to guide you through the setup and execution process. Basic logging will be incorporated to ensure any potential issues, such as rate limits or authentication errors, are reported without disrupting the job. Please let me know if you have any questions or specific preferences regarding libraries or tools. I am looking forward to the possibility of collaborating with you on this project. Best Regards,
$10 USD in 2 days
6.9
6.9

Hi! My name is Marjan and I'm here to offer you my services as a skilled applicant with over a decade of experience working on Freelancer.com. I believe I am the best fit candidate for this project due to my extensive experience; I would like to have a discussion to get to know that we both are on the same page. Once the scope will be locked, I will start working on it right away.
$140 USD in 7 days
6.8
6.8

Hello! i can build this automated pipeline in Python Facebook data through the official Graph API, website data with Requests/BeautifulSoup or Playwright if needed CSV files will be saved to timestamped Google Drive folders and the job will run automatically twice a week will also add logging, deduplication and proper handling of API/rate-limit errors
$390 USD in 4 days
6.7
6.7

Hello, I'm Denis, a full-stack developer. I've built data pipelines that run on schedule, pull from multiple sources, and deliver clean results where you need them. Your goal is to stop manually checking Facebook pages and your website twice weekly. Instead, you want a hands-off system that fetches posts, comments, engagement metrics, and media from Facebook, plus specific website content, and delivers everything to Google Drive as timestamped CSV files. The data must be deduplicated and reliable for immediate use. I'd start by confirming which Facebook assets and website sections to target, then build the scraper in Python using the official Facebook Graph API and a focused crawler for your specified pages. I'll set up a Cloud Function or cron job to run Monday and Thursday at UTC, write results to Google Drive with timestamps, and add logging so you can track each run. Testing will cover a full week during handoff to ensure reliability. The main challenge is handling Facebook's rate limits and auth errors without crashing. I'll add retry logic with exponential backoff and graceful error handling so the pipeline logs issues but keeps running. The other risk is unclear scope on website sections, so I'll need you to specify exact pages upfront to avoid rework. I can start right away. Let's connect and discuss the details. Thanks, Denis.
$30 USD in 1 day
6.4
6.4

Your pipeline will fail silently if Facebook revokes your Graph API token mid-run or if the website changes its DOM structure between scrapes. Both scenarios will corrupt your dataset without triggering an alert unless you build token refresh logic and schema validation into the orchestration layer. Quick questions - are you planning to store historical snapshots for trend analysis, or do you only need the latest bi-weekly state? And do you have existing OAuth credentials for the Facebook Graph API, or will I need to set up the app registration and permission scopes? Here is the architectural approach: - PYTHON + GRAPH API: Build a token-refresh wrapper around Facebook's official SDK to handle expired credentials gracefully, then map posts/comments/engagement to a normalized schema that survives API version changes. - WEBSITE SCRAPING: Use Playwright with headless Chrome to handle JavaScript-rendered content, paired with CSS selectors that fall back to XPath if the site redesigns its markup. - AUTOMATION: Deploy as a containerized Cloud Run job triggered by Cloud Scheduler (cron syntax), with structured logs piped to a monitoring dashboard so you catch failures before the next run. I've built similar ETL pipelines for 2 SaaS clients who needed daily LinkedIn and Shopify scrapes without manual intervention. Let's schedule a 15-minute call to confirm your OAuth setup and finalize the deduplication strategy before I start the build.
$20 USD in 3 days
6.5
6.5

As a seasoned software engineer with an impressive portfolio, I am well-versed in all the skills required for this project. My specialized knowledge of web scraping, particularly using Python and Node.js, has proven invaluable in automating data retrieval processes similar to what you're requesting. Additionally, my experience with Facebook's Graph API will ensure we respect all its terms of service while extracting the necessary data from specific Facebook assets. Moreover, I'm distinctly comfortable working within diverse hosting platforms such as AWS and Google Cloud, which will come in handy when deploying the scraper and files for you to access comfortably via Google Drive. My commitment to clean code and thorough documentation will further ensure that you can maneuver the pipeline effortlessly even after delivery. Trust me to implement a solution that runs undisturbed on Mondays and Thursdays(UTC) while gracefully handling any rate-limit or authentication errors along the way. Remember: "There's extraordinary chemistry in long-term relationships," - let's build one through this successful project venture!
$20 USD in 1 day
6.3
6.3

Hi, can the Facebook assets be accessed through the official Graph API with the provided tokens, including comments and engagement fields? I’d build this in Python with Graph API requests, Playwright/BeautifulSoup for the specified public website pages, CSV exports to timestamped Google Drive folders, and SQLite for deduplication/logging. I’ll add Monday/Thursday UTC scheduling, retry/rate-limit handling, authentication checks, Docker deployment, and clear field mappings in the README. Can you provide the Facebook asset URLs and the exact website sections so I can confirm API coverage and the crawl scope?
$20 USD in 7 days
6.1
6.1

noticed your requirement and here is my understanding - Automated bilingual data ingestion pipeline delivering structured Facebook and website intelligence directly to Google Drive. I've gone through your requirement for a fully automated data ingestion pipeline. I can build this with your preferred stack - Python, Official Graph API, Beautiful Soup, Google Drive API, Docker - with clean, scalable codebase within 7 days. Milestone 1: Build core extraction modules for Facebook Graph API pagination and targeted single-domain web scraping with robust error-handling, retry logic, and deduplication. Milestone 2: Implement the bi-weekly UTC scheduler, Google Drive automated folder structuring, and end-to-end logging verified through a 7-day unattended test run. Let's review target timelines and map core milestones. I may have few questions before suggesting full architecture. Lets hit CHAT and discuss now ! Gaurav P. 15 yrs experience in debugging systems :)
$49 USD in 2 days
6.4
6.4

Hi, I can build a reliable Python-based automated pipeline that collects the required Facebook and website data every Monday and Thursday and exports clean, structured CSV files to timestamped Google Drive folders. Proposed stack: Python Facebook Graph API for permitted Facebook data Requests + BeautifulSoup, with Playwright where JavaScript rendering is required Google Drive API for automatic uploads SQLite for deduplication/state tracking Docker for simple deployment Cron, GitHub Actions, or Cloud Scheduler for twice-weekly execution The pipeline will collect available posts, comments, engagement metrics and media links from the Facebook assets supported by the provided API permissions. For the website, I’ll build a targeted crawler for the specified product, review and article sections. I’ll include retry/rate-limit handling, authentication error reporting, structured logging and safeguards so one failed source does not crash the entire job. Deliverables: clean documented code, Docker/deployment setup, README with field mappings, automated scheduling, Google Drive CSV exports and hand-off testing for one full week. I have experience with Python/Node.js, APIs, automation, databases, cloud deployment and data-processing workflows. I’ll also ensure the Facebook integration stays within the capabilities of the official Graph API and provided access permissions.
$20 USD in 4 days
6.0
6.0

Hi, I have read the job description and now ready to start. I will provide 100% quality. Please send me a message for discussion. Let's discuss the project. Thanks
$10 USD in 1 day
5.5
5.5

Hello, I’d build this pipeline using Python with `facebook-scraper` and `requests-html` to handle both Facebook’s Graph API and the website, while avoiding IP bans through proper rate limiting. For Facebook, I’ve automated similar workflows for clients by structuring the Graph API calls to fetch posts, comments, and media links in batches—this keeps the data clean and avoids hitting rate limits. The system would run on a Cloud Function with a cron trigger, storing each run’s CSV in Google Drive with timestamps. I’d include error handling to log issues without stopping the job, and a simple SQLite database for deduplication if needed. This setup will save you time by delivering structured data automatically, while the logging ensures you can spot issues early. After deployment, I’ll test the pipeline for a week to confirm it runs smoothly and meets all requirements. If this approach works for you, I can share more details about the implementation. Thanks, Lazar.
$10 USD in 1 day
5.4
5.4

Could you clarify the specific Facebook pages or groups you want to scrape? This will help ensure the pipeline is tailored to your needs. For the website scraping, I recommend using BeautifulSoup or Scrapy in Python to effectively capture the required sections. Is that acceptable? I have extensive experience in Python and data processing, which will allow me to create a robust and efficient scraping solution that adheres to Facebook's API guidelines. I will set up a cron job to automate the data fetching every Monday and Thursday. The output will be structured in CSV format and stored in Google Drive, ensuring easy access and organization. Please feel free to reach out to me. I look forward to working with you. Best regards, Billy Bryan
$10 USD in 3 days
5.6
5.6

Hi, I am a Python developer with 10 years of experience. I can build an automated pipeline using the Facebook Graph API and Python scraping, with scheduled Monday/Thursday runs, CSV exports to Google Drive, deduplication, logging, and Docker/cron deployment. Could you share the Facebook asset URLs and the website sections you want to crawl? Thanks Jaffer I.
$15 USD in 2 days
5.0
5.0

Facebook assets plus a public site every two weeks means two different scrapers that need to survive layout changes without breaking the pipeline. I would build this in Python with requests or Playwright for the dynamic parts, store results in MySQL, and run it on a cron schedule with logging so you see exactly what ran. Can start today. These numbers are based on the post as written. We will refine them once we go through the exact fields and sources you need. Want me to send a quick scope doc so we can get moving?
$30 USD in 5 days
4.7
4.7

Hi, would you like the pipeline to prioritize historical completeness on the first run and then use deduplication to capture only newly visible Facebook/website records on subsequent runs? I can build a Python-based, Dockerized pipeline using the official Facebook Graph API where supported, a targeted website crawler, Google Drive CSV exports, structured logging, and Monday/Thursday UTC scheduling. I’ll include resilient rate-limit/auth handling, timestamped run folders, field mapping, deployment instructions, and a clean README so the system can run unattended. Which Facebook assets and exact website sections should be included in the initial configuration?
$15 USD in 2 days
5.1
5.1

Hi, I can build this as a lightweight Python pipeline using the Facebook Graph API for the Facebook data and requests/BeautifulSoup for the website, with Playwright only if any target content requires JavaScript rendering. I’d keep each run timestamped, normalize and deduplicate the records, export them to CSV, and upload the output automatically to the requested Google Drive folder. The Monday/Thursday schedule can run through cron or GitHub Actions depending on where you want to host it. I’ll also add logging for API errors, rate limits and failed pages so one failure doesn’t stop the entire run. Before starting, I’d like to check the Facebook URLs and the permissions available with your access token, since Graph API access differs between pages, groups and profiles. I can deliver the initial working pipeline in around 3 days, then verify the scheduled runs during the hand-off period. Thanks, Albert
$30 USD in 3 days
4.7
4.7

With years of experience in web and app development, I am confident that I am the perfect fit for your bi-weekly Facebook and website scraping project. My expertise in Python, Node.js or Go (as per your preference) combined with a deep understanding of APIs and databases such as MySQL will ensure a reliable, efficient and hands-off hands-off pipeline for you. JSON parsing through Facebook's Graph API is something I've done extensively in previous projects, so handling rate limits or other authentication issues won't be a problem. For website scraping, It will be a breeze to navigate through the structure of each page and accurately collect all the specified information using Python libraries such as Beautiful Soup or Selenium. The choice of using Google Drive for storage with timestamps for every run is well-suited for my skills. Furthermore, I'm committed to delivering clean, well-commented code along with detailed setup instructions and maintenance scripts. My proficiency in problem-solving will also ensure smooth job execution devoid of any missing fields or crashes. Lastly, I believe my commitment to quality, deadlines and client satisfaction aligns perfectly with what you're searching for - a dependable developer to automate this part of your work while you focus on the more important tasks at hand. Let's work together and turn your vision into reality! Let’s turn your idea into a powerful digital experience!
$10 USD in 1 day
4.7
4.7

Hello, I checked your project "Bi-Weekly Facebook & Website Scraper" and already have a clear understanding of your requirements and how to deliver them efficiently. I have solid experience in PHP, Python, MySQL, and I've successfully completed similar projects by delivering high-quality, scalable, and reliable solutions. Why choose me? * Strong expertise in PHP, Python, MySQL * Clean, optimized, and scalable implementation * Fast communication and regular progress updates * Focused on delivering results, not just completing the task I'm ready to start immediately and would be happy to discuss the details. Best regards, Umer
$25 USD in 1 day
4.3
4.3

Burnley, United Kingdom
Payment method verified
Member since Feb 21, 2018
$10-50 USD
$30-250 USD
$199-200 USD
$30-250 USD
$10-35 USD
₹1500-12500 INR
₹12500-37500 INR
₹12500-37500 INR
$30-250 USD
₹1500-12500 INR
₹10000-15000 INR
$30-250 USD
$30-250 CAD
₹750-1250 INR / hour
$250-750 USD
$1500-3000 USD
$100000-300000 USD
₹750-1250 INR / hour
$10-30 USD
₹1500-12500 INR
₹600-1500 INR
₹12500-37500 INR
$250-750 USD
$25-50 USD / hour
$250-750 USD