Find Jobs
Hire Freelancers

Hadoop Spark/Scala Project

$30-250 USD

Completed
Posted over 6 years ago

$30-250 USD

Paid on delivery
Homework A) Using Hadoop hdfs & Spark-scala programming Source dataset: [login to view URL] download data for 1999,2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008 1) Download and combine all data for the years specified about 2) Data Cleanup: Find and remove /filter out outliers & bad data 3) Perform statistics analyis on the data: counts /averages /sums / min /max 3) Using spark/scala programming on the entire dataset, what percent (%) is a) on-time flight b) cancelled flight c) Delays flights d)TOP 5 Causes of delays e) Most causes of flight delays f) Airlines with the most delays to a destination g) Airline with the most cancellations h) Airline with the most on-time i) Flight on-time / delays and cancellation national averages J) Perform some visualization in Tableau (Send me output data file,I will do visualisation myself) K) All of the above Code in a separate PDF file B) Create 10-15 pages (in word) to include the following topics: 1) Data source 2) Description the data and its schema 3) Data pre-processing required (parsing, filtering, etc.) 4) Any bad data issues encountered 5) Describe Your Spark algorithm 6) Describe any other ecosystem or additional tools used 7) Describe the output 8) How did you verify that your output is correct? 9) discuss the Performance/scale characteristics 10) what would you have done differently if you did this again? 11) Draw a conclusions from this excercise Please NOTE: This must be your original work. Someone else code cannot be copied from online and used in this project. Doing so will cause you an F grade in this course Deliverable Timeline: 1) Code in separate document -- Deliver by NOV 25 2) Documentation (10-15 pages in word) -- Deliver by NOV 27 3) Output dataset file --- Deliver by NOV 30 Deadline: NOV 30 for all of the above NB: Your personal hadoop cluster or I can provide access to cloud based hadoop cluster with data files already download onto HDFS folder
Project ID: 15652432

About the project

6 proposals
Remote project
Active 6 yrs ago

Looking to make some money?

Benefits of bidding on Freelancer

Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs
Awarded to:
User Avatar
I am a data scientist and have experience with Big data Technologies like Spark and Hadoop. I have previously worked with this dataset and can complete all your given task before date specified. Relevant Skills and Experience Spark, Hadoop, NoSQL, Scala, Python, R, Tableau Proposed Milestones $180 USD - Project Milestone
$135 USD in 3 days
4.9 (11 reviews)
4.8
4.8
6 freelancers are bidding on average $168 USD for this job
User Avatar
Hi, I am java expert and have experience on Big data, so this kind of data processing will be done perfectly. Hope to discuss with you. regards, Relevant Skills and Experience Java Proposed Milestones $133 USD - Init
$133 USD in 5 days
5.0 (2 reviews)
2.8
2.8
User Avatar
I have worked with spark and made recommendation system earlier so I will be able to fulfill your task
$55 USD in 10 days
0.0 (0 reviews)
0.0
0.0

About the client

Flag of UNITED STATES
North saint Paul, United States
5.0
29
Payment method verified
Member since Feb 16, 2011

Client Verification

Thanks! We’ve emailed you a link to claim your free credit.
Something went wrong while sending your email. Please try again.
Registered Users Total Jobs Posted
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759) & Freelancer Online India Private Limited (CIN U93000HR2011FTC043854)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Loading preview
Permission granted for Geolocation.
Your login session has expired and you have been logged out. Please log in again.