Multi-layered text and image scraping project for industry in Australia

Stagecoach Software Pty Ltd requires a MySQL database containing details of companies and particular individuals operating within a particular industry operating in the states of Victoria and New South Wales, Australia. There are approximately 30,000 individuals employed in the industry in these states and the details of about half of those are required. The companies are all named on a single website. Most of the companies belong to a franchise. The MySQL database must contain a table listing the names of every franchise and its website. Each franchise has a logo which must also be scraped and stored in an individual image file. A third table must contain the names and other given details of each company and, where applicable, the name of the franchise it belongs to. The main website contains a link to a profile of each company. Where those profiles contain details of individuals working for the company (almost always less than 20), the name and contact details and a copy of the profile photograph must be scraped. The individual photograph filenames must include the name of the person and their employer. Where these details are not available through the site profile, they must be sought from the company’s own website. A further website containing transaction details must also be scraped to ensure all individuals named on it are in the database. All image files must be stored in a hierarchical folder system arranged by franchise, then company. All data must be normalized. This project follows freelancer Project ID’s 5129765, 5066285 and 4975478. In turn it will be followed by the creation of further projects culminating in a Ruby on Rails website that will present graphs showing aggregate and individual data obtained from the database and allowing user searches. Preference will be given to freelancers interested in and qualified to carry on work in those following projects. Bidders without demonstrable experience in MySQL, and Web Scraping will be rejected. Preference will be given to bidders experienced with Ruby on Rails. A fuller confidential project description will be provided to certain bidders. Bids may be revised after reading the project requirements.