Completed

Python web crawler

I need to crawl the real estate section of a local marketplace website which has an awful html structure, for market research purposes.

Furthermore, the crawler should be uploaded to [login to view URL] for scheduling weekly jobs.

This project must be deployed with:

- Python 3.6

- Scrapy

- Shub (for deployment)

Other things to take into account:

- The website is in spanish

- CSS selectors won't work

- The comments section of the listings is variable

- The website has no [login to view URL] nor API

The starting url is:

[login to view URL]

Just to be clear:

Starting URL (results page)
http://www.compraensanjuan.com/b_l_in.php?cat=&orden=QueDes%2Cfchact+desc&tipo=0&operacion=0&zona=0&vendedor=Todos&precio_desde=&precio_hasta=&resultado=listado

Example of individual listing layout
http://www.compraensanjuan.com/anuncio/929788/loteo-propio-b-pbco-sobre-mendoza-y-centenario-proximo-al-natania-8-400-m2-

Data that needs to be extracted:
- Listing number ("codigo aviso")
- URL
- Title
- Price
- Seller ("anunciante")
- Number of visits ("visitas")
- Publishing date ("publicado")
- Last update ("actualizado")
- Description ("Características")
- Lot/land size ("Superficie total")
- Construction size ("Superficie cubierta")
- Age ("antiguedad")
- Rooms ("ambientes")
- Bedrooms ("dormitorios")
- Car parking lots ("garages")
- Street name ("Calle")
- Street number ("numero")
- Neighborhood ("barrio")
- District ("Departamento")
- Latitude and longitude
- Questions to seller ("Consultas al vendedor") [here I only need the url of each person's profile and names]
- Pictures URL's

Skills: HTML, Python, Regular Expressions, Scrapy, Web Crawling

See more: url http www webdesign org miscellaneous web design inspiration 100 free patterns 22198 html 100 free patterns to boost your cre, python web crawler mining, python web crawler scrapy, python web crawler documentation, python web crawler source code, python web crawler code, python web crawler github, scrapy python, python web crawler beautifulsoup, python scrapy example, scrapy python 3, python, web scraping, scrapy, advanced python web crawler, python web crawler mysql, python web crawler mp3, python basic web crawler, python web crawler scraper, programming web crawler python

About the Employer:
( 1 review ) San Juan, Argentina

Project ID: #16577045

Awarded to:

gangabass

I can create Scrapy spider and deploy it to your Scrapinghub account in less than 36 hours. It will output all fields you want: - Listing number ("codigo aviso") - URL - Title - Price - Seller ("anunciante") More

$2200 ARS in 2 days
(164 Reviews)
6.5

23 freelancers are bidding on average $2588 for this job

$2472 ARS in 3 days
(10 Reviews)
6.5
hunmin888

hi, sir. i have a good experience in web scrapping. i made a lot of scrapers. if you want, i can show my previous projects to you. thanks.

$2472 ARS in 3 days
(36 Reviews)
6.2
wayne4989

These are some scraped urls by myself. - [login to view URL] - [login to view URL] (login) - [login to view URL] - [login to view URL] - [login to view URL] - https:/ More

$2472 ARS in 3 days
(14 Reviews)
5.5
furqaanwar

Hello Sir I am an expert web developer with an experience of over 5 years and a university bachelors degree in computer science major. with web development i am also doing web scraping work very efficently . i have d More

$5555 ARS in 3 days
(9 Reviews)
5.2
shingjin

Hello. After reviewing your post, I am very interested in that due to my experience. I’d like to be considered for your project position. Whether you need a friendly, attractive or a more playful project, I can mak More

$2222 ARS in 3 days
(24 Reviews)
4.8
zekovicm

Hi there,I am Miljan,IT expert from Bosnia & Herzegovina,Europe. I have carefully gone through with your requirements and I would like to help you with this job ! I can start immediately and finish it within the agree More

$3333 ARS in 3 days
(6 Reviews)
4.0
vamzee

I have 5 years of experience in python and data scrapping. I have done many projects in webscrapping. You can check my profile. We can negotiate on the price.

$2222 ARS in 3 days
(5 Reviews)
3.4
damilareisaac

Hi there, This is an interesting project. I am willing to work with you on this project to deliver an excellent result. The can provide both the data and analysis if you want. I will be looking forward to hear from y More

$500 ARS in 3 days
(8 Reviews)
3.7
ts199756

hey! I have seen the attachment crawler build in selenium will work perfect with the type of extraction you want, do try me I can show you some sample results before getting the project, Secondly I have an experience i More

$525 ARS in 7 days
(1 Review)
2.9
$2500 ARS in 5 days
(2 Reviews)
3.0
samescolas

Hello, First, I agree, this HTML is a mess! I'm not positive I can do this yet. I verified that I can isolate each property, but what data points are you looking for? Also, is Scrapy necessary or would you mind i More

$2472 ARS in 3 days
(3 Reviews)
2.7
htmn

IT engineer, 6 years of experience in web crawling/ web scraping, I have such project ready (running in python, django, cron, psql), Please contact me for details: website to scrape, deadlines, .. here is a sa More

$3600 ARS in 3 days
(1 Review)
2.4
Tikdragon

Hi! Very full confidence and experience on Web Scarping with python. If you hire me, I will prove my potentials with good service. Regards!

$2472 ARS in 3 days
(4 Reviews)
2.2
yogeshpawar159

Hi, I have successfully coded and tested the webpage parser for all required parameters. Now just need to setup the scrapy. I am an experienced python developer. I have done web scrapping using scrapy framework an More

$4000 ARS in 5 days
(2 Reviews)
1.5
shawnzhaowu

A proposal has not yet been provided

$2222 ARS in 3 days
(0 Reviews)
0.0
xcrapper142

I know how to use scrappy well. I should have this ready before my exams start on Tuesday.

$525 ARS in 10 days
(0 Reviews)
0.0
projectdevelope8

Your ad called out to me because the position, as described, is such a perfect match with my skills, as you will see when you review my attached resume. I am a highly skilled Devops with eight years of experience progr More

$3333 ARS in 7 days
(0 Reviews)
0.0
codemonkey1110

Hi, I took a look at the site's HTML and it is indeed awful. It should still be possible to extract data with some effort. As I'm new here and don't have a good profile to show you, I did a sample job with some of t More

$1411 ARS in 3 days
(0 Reviews)
0.0
khannanav

We are a team of experts with more than 8 years of rich cloud experience in AWS [login to view URL], we have worked extensively on Machine Learning. I have reviewed the attachment and is confident of delivering your solution More

$3333 ARS in 3 days
(1 Review)
0.0
hoseyn

A proposal has not yet been provided

$2222 ARS in 7 days
(0 Reviews)
0.0