Scrape data from website to MS Power BI using PBI Python

Closed Posted 1 year ago Paid on delivery
Closed Paid on delivery

To execute this project, the freelancer will need the following skill and experience set:

Proficiency in Python programming language

Experience with web scraping using Python libraries such as BeautifulSoup or Selenium

Knowledge of Power BI and its integration with Python

Experience with incremental data refresh in Power BI

Understanding of website structure and HTML elements

The project will involve the following steps:

Setting up the Python environment and importing the necessary libraries:

Installing necessary Python libraries such as BeautifulSoup, Selenium, and Pandas

Importing the libraries into the script

Setting up the web driver for Selenium

Creating a script to navigate to the website and select the start and end dates:

Using Selenium to open the website "[login to view URL]"

Locating the start and end date pickers on the website using Selenium's find_element_by_* methods

Entering the start and end dates into the date pickers

Ticking the 3 boxes located below the date pickers to select all options of their related drop down lists:

Locate the 3 boxes using Selenium's find_element_by_* methods

Use Selenium's click() method to tick the boxes

Clicking the retrieve button to retrieve the data and storing it in a variable:

Locate the retrieve button using Selenium's find_element_by_* method

Use Selenium's click() method to click the button

Use BeautifulSoup to parse the HTML of the page and extract the table data

Store the table data in a variable for further processing

Looping through the pagination links and retrieving the data from each page:

Locate the pagination links using Selenium's find_elements_by_* method

Iterate through the pagination links and click on each one

Retrieve the table data on each page using BeautifulSoup and store it in the variable

Continue looping until the end of the pages is reached

Cleaning and formatting the data for better understanding:

Use Pandas to convert the data into a dataframe

Perform any necessary cleaning and formatting on the data, such as removing null values or renaming columns

Integrating the data with Power BI for visualization and analysis:

Use Power BI's Python integration to import the data into a report

Create visualizations and perform analysis on the data

Setting up incremental refresh in Power BI to update the data daily:

Use Power BI's incremental refresh feature to schedule the data to be updated daily

Configure the refresh settings and test the refresh process to ensure it is working properly.

The estimated hours required for this project is around 40-50 hours, assuming 2 revision rounds with the project owner. This includes time for debugging and testing the script, as well as integration with Power BI and setting up incremental refresh.

Python Web Scraping BeautifulSoup Selenium Power BI

Project ID: #35823355

About the project

71 proposals Remote project Active 1 year ago

71 freelancers are bidding on average $497 for this job

chirgeo

Hello, my name is George and I'm an experienced web crawler expert with proven success in scraping websites and optimizing dataflows with web crawling software. With my extensive experience and knowledge in the Python More

$750 USD in 3 days
(151 Reviews)
8.0
livegoodlife

Hello There, Experienced in web scraping / bot / software using Python, Java, PHP. I am experienced with scrap data from the given website, Also I am an expert to build a web scraper, desktop software, or script to pul More

$250 USD in 7 days
(238 Reviews)
7.8
PoojaRautela417

Hi There, I am Python script developer with 5 years of experience. I can develop web scraper to scrape your required website by Python script/Bot with your instructions . Can we discuss please? Thanks & Regards Pooj More

$250 USD in 2 days
(104 Reviews)
6.6
AwaisChaudhry

Hello Good evening , I hope you are doing great. Just finished reading the brief details of your job and currently going through attached files . I see you have been looking for a freelancer who has experience with P More

$750 USD in 17 days
(11 Reviews)
6.8
sapnathakur14

Hello, Warm Greetings! I am pretty sure I can produce high-quality and perfect results for your project. By using Python, I used to make AI engines, BOT, Web Scraping Tools, and so on. PHP and Python are my majors, so More

$750 USD in 7 days
(11 Reviews)
6.1
prakash2813

⭐⭐⭐⭐⭐ Hi there, I have strong expertise in web crawler, web scraper, web monitor, web automation and b-o-t-s using selenium webdriver framework. I have already made lots of scrapers and b-o-t-s for sites such as Googl More

$650 USD in 3 days
(27 Reviews)
5.8
Akmanyu

Hi There, Good Morning! "First use then believe". I don't want to mention much here but I have 6+ years of experience in Power BI, Python along with Power Platform & Power Automate in total 8+ years of IT experie More

$250 USD in 1 day
(69 Reviews)
5.8
Lagripe

Hi. I am a freelancer working in 24/7 service. I have much experience of +6 years in Scraping field, I can do your job with Python. I will produce the output in any format you desire. I think this project is for me and More

$250 USD in 2 days
(51 Reviews)
5.7
ganeshrasekar

I can do it

$500 USD in 7 days
(25 Reviews)
5.3
diachukoleksand1

Happy new year, Nice to meet you. I am very interested in your project. I am a scraping expert that have developed many scraping bot, web scraping, data mining, Data processing. I am very familiar with python, c#, and More

$500 USD in 7 days
(4 Reviews)
5.3
Parveenlamba77

Hi There, I am web scraping expert. I have 10 years of working exposure on web scraping using python+selenium, BeautifySoap, UiPath etc. I have delivered many scraping projects to different clients. I can start work More

$250 USD in 3 days
(16 Reviews)
4.8
suyashdhoot

Scrape data from website to MS Power BI using PBI Python Good morning Fahad A. , Hi I am a very experienced statistician, data scientist and academic writer. I have completed several PhD level thesis projects involvin More

$750 USD in 3 days
(30 Reviews)
6.0
karlmarkus609

Hello. I read your description carefully. I am python expert you want and am familiar with Power BI so I can make perfect result in time. Let's discuss about your project.

$500 USD in 7 days
(27 Reviews)
5.3
anenkovakateryna

Hi! I have 8 years of experience in ⭐ python with power BI ⭐. I've read the project description carefully and can start work immediately. Let`s discuss about more detail in chat box. Best regards!

$350 USD in 3 days
(20 Reviews)
5.2
rexzetsolutions

hii i went with the attachment kindly see my profile already done with the scraping part with help of python i know this how to do kindly tell me the details and text me once if interested and i'll show you my previous More

$500 USD in 4 days
(23 Reviews)
5.4
MrAndreyZhdanov

Hi, there. I read your project description carefully and I am interested in your proposal. I'm a Python and Web Scraping developer and I have a lot of experience with web scraping. I usually use Beautiful Soup, Seleni More

$250 USD in 2 days
(17 Reviews)
4.6
anaya8777

Hey! I am python and web scrapping expert. I did many scrapping tasks using python libraries. Please initiate a message to discuss further.

$350 USD in 8 days
(7 Reviews)
4.8
ARTICLEYOUWANT

I will create scraping tool with PHP & CURL that extracts all requested data, And provides data in the required format (JSON, CSV, EXCEL, SQL...). Can we discuss more details about this project? Then I can provide sam More

$250 USD in 7 days
(25 Reviews)
5.0
vladimirrudic

Hi, I just read your job posting and can see what you want exactly. As a senior developer, I have done many scarping projects using both python and extensions, fully mastered python as well as selenium, puppeteer, am More

$700 USD in 10 days
(2 Reviews)
4.3
vamshi6

Hi, We have experience in Developing Web Scraper using Python and its various modules like BeautySoup and also works with Power BI and Selenium can fulfill your requirements mentioned. We have done this Before for our More

$500 USD in 7 days
(13 Reviews)
4.5