Closed

Scrape Wayback for AU domain name and first archive entry, output to CSV

Unless one of the Freelancer community knows different, there doesn't seem to be a way of querying the AU TLD whois records for the creation date of Australian domain names, even with DIG or nay other commandline tools. My imperfect solution is to harvest the earliest Wayback archive entry .

Image 1 shows the two data elements I need to extract from the zonefile csv I have : (1) the domain name (2) the date the domain name was first archived in Wayback.

Using a script I found in the Wayback APIs I have built a [clunky] batch script [[url removed, login to view], attached] that captures data into a series of files that I batch rename to a csv [[url removed, login to view] example attached].

Each capture from [url removed, login to view] produces a file with the oldest archive date on line 1. The domain name is obvious from each line, the date is in the YYYY:MM:DD:HH:MM:SS format.

It will need a chunk of regex written into the script to filter off the first line of each successful data capture, clean up time data element to readable format (eg 19981202014938 becomes 02/12/1998) and appends the domain name and date to an external csv file.

Some of the domains have no entry in Wayback but the script will still need to write the URL to the csv with a 'nul' value as the date element. so I can see which have no Wayback records.

The script will preferably draw the urls from an external text file.

The successful bid can use the URLs listed in the '[url removed, login to view]' file attached as a test list for the script. I dont need the scrape, just the script (there are 2.1 million domain names to query).

Autobids will be deleted if the proposal is not read, and acknowledged within 12 hours. My script took approx 2 hours to write with minimal experience so bids over 4 hours will be automatically deleted.

Any other questions, DM me.

Some resources:

[url removed, login to view]

[url removed, login to view]

[url removed, login to view]

[url removed, login to view]

Skills: Data Processing, JSON, Shell Script, Web Scraping

See more: read csv file output csv vbscript, python script output csv file, data website output csv, javascript output csv, scrubyt output csv, output csv javascript, web scraping output csv, data entry excel output csv, vba paste excel output csv, read text file save output csv, output csv file javascript, perl output csv cron, web output csv, writing output csv file format python

About the Employer:
( 2 reviews ) Melbourne, Australia

Project ID: #14358495

15 freelancers are bidding on average $35/hour for this job

mantislin

Hi sir, I am scraping expert, I have did more than 350+ scraping project, please check my feedback then you will know. Can we discuss more details about this project? then I will provide example data/script More

$250 AUD / hour
(238 Reviews)
7.3
schoudhary1553

Greetings sir, i am an expert freelancer for this job and your 100% satisfaction is assured if you allow me to serve. Here is the reason. Why you should pick me? a) I am a very expert and have the same kind of ex More

$22 AUD / hour
(136 Reviews)
7.2
dghq123

Hello. I can do it the BEST for you. I have read the details completely. Please have a look onto my employer's reviews on my profile so you can get an idea of my quality work :) Inbox me so we can discuss and conti More

$21 AUD / hour
(6 Reviews)
4.9
cheungkc

Hi I have consulted the api document. There is a limit switch to return just the first entry of the result, which is usually sorted by date. So we can always get only 1 row of data per search. Using the result. More

$22 AUD / hour
(6 Reviews)
3.1
labibhasan01718

I am highly interested to do data research and extraction work. I can guarantee you that I will extract such data that are missed by others.I can start your work right away very soon. Please note that I will collect al More

$15 AUD / hour
(4 Reviews)
2.5
shubhahp

A proposal has not yet been provided

$22 AUD / hour
(4 Reviews)
1.9
BlueViSion7

Good day, My name is Nikola Nuspahic and I am business manager at BlueViSion IT Solutions. It is a team of web developing and market research experts from East Europe. Our expertise are Web Development, Database De More

$15 AUD / hour
(0 Reviews)
0.0
zdesign77

Hey, how is it going today? My name is Michael, I'm a web designer/developer from Boston. I've had a look at your project description and feel that my skills match your requirements perfectly. In fact, I did a very sim More

$22 AUD / hour
(0 Reviews)
0.0
RaniAkter786

Hello This is a simple task for [login to view URL] I have much experience about that kind of task Give me the opportunity and show how my talent is Waiting For your Kind Response Regard`s Rani Akter

$22 AUD / hour
(0 Reviews)
0.0
Zubairashraf123

Hi I am Zubair, expert in web scrapping,Data Processing, JSON, Shell Script, Data mining, and web research. I have scraped over 1000 sites from last 6 years during which I have undertaken many challenging aspects. I h More

$15 AUD / hour
(0 Reviews)
0.0
Shopify

Hello, It's a pleasure to let you know that I've Completed and Delivered similar project before. All I need to work on customization part if we can proceed towards more discussion. I have gone through your project de More

$20 AUD / hour
(0 Reviews)
3.4
ishtiyaqahmad

Hello, My name is Ishtiyaq Lone, I am expert in web scrapping, data crawling and data mining, I have 6 years+ experience in data mining, website scraping, screen scraping and I am certified web scrapper expert. I have More

$15 AUD / hour
(0 Reviews)
0.0
pixelagency1

Hello! I have got acquainted with your requirements and have a proposal as well as some questions that we should discuss. The price and timeline are negotiable. We will agree on final estimation after discussio More

$25 AUD / hour
(0 Reviews)
0.0
anirbanbanerjee

Hi, I was going through your project specs and I have some questions to ask you. Are you available to have a quick chat now? Thanks and Best Regards Krish

$20 AUD / hour
(1 Review)
0.0
mobileappdevin

Hi, I have a good amount of experience in web scraping using Python,Django, Phantomjs and nodejs. This is my latest project on web scraping using python: Scraping using Python: Electronics Parts Intelligence Pr More

$15 AUD / hour
(0 Reviews)
0.0