spark DataFrame Operations - Spark Session - 27/04/2018 01:01 EDT

Data Frame1:

EmpNo EmpName Salary

E123 Tom 2000

E124 RAM 2000

E125 TAM 2000

E126 SAM 2000

E124 RAM 4000

E126 SAM 6000

E125 TAM 9000

E123 Tom1 4000

Transform this DataFrame to


EmpNo EmpName Salary rownum

E123 Tom 2000 1

E124 RAM 2000 1

E125 TAM 2000 1

E126 SAM 2000 1

E126 SAM1 5000 2

E124 RAM 4000 2

E126 SAM 6000 3

E125 TAM 9000 2

E123 Tom1 4000 2

Here is the summary:

-- Duplicate EMpnos should be indexed(as shown in the rownum column)

-- The order of index should be based on salary.

-- Need All of the below approaches

-- Should be optimized and should be runnable on a cluster.

1. Using RDD(a. using SparkContext, [login to view URL])

2. Using DataFrame(a. using SparkContext, [login to view URL])

3. Using DataSet(a. using SparkContext, [login to view URL])

DataFrame Operations should contain both . notation and sql notation.

Action Items:

1) Development

2) Testing

3) Demo

4) Any corrections/small enhancements(if required)

Skills: Data Processing, Data Science, Hadoop, Scala, Spark

See more: With reference to your application date26.04.2017 and further interview date 27.04.2017, we are pleased to offer you the positio, spark plugin messenger spark, spark graphics philippines, openfire spark api, spark remote desktop, spark client remote desktop control, remote control spark, bright spark multimedia, spark messenger view log, spark messanger, vnc spark, spark messenger, spark graphix, spark plugin website, spark plugin source code

About the Employer:
( 1 review ) orlando, United States

Project ID: #16796241

12 freelancers are bidding on average $25 for this job


hello, senior data engineer, I have worked from more than 6 years on software development, I know well spark, scala etc, I believe I can resolve your problem.

$25 USD in 1 day
(2 Reviews)
$25 USD in 1 day
(8 Reviews)

Hi I am interested in this job. I believe that the skills and experiences I have gained at this position make me an ideal candidate for this position. I have extensive experience in implementing Spark-core, Spark-s More

$25 USD in 1 day
(1 Review)
$25 USD in 1 day
(2 Reviews)

Hi Dear, I'm available now for urgent working for you 40/hr for data entry work with 100% correction I will do Fast and accurate Data entry and Data processing form according to the requirement. The First prio More

$10 USD in 1 day
(0 Reviews)
$25 USD in 1 day
(0 Reviews)

Scala and spark expert. Have written numerous algorithms in spark/scala. Most recently wrote cure algorithm in scala/spark. The problem presented is interesting. Can complete this project with specs2 mocking. More

$35 USD in 2 days
(0 Reviews)

A proposal has not yet been provided

$30 USD in 2 days
(0 Reviews)
$30 USD in 1 day
(0 Reviews)

excel expert Relevant Skills and Experience BS in maths

$25 USD in 1 day
(0 Reviews)
$25 USD in 1 day
(0 Reviews)
$20 USD in 2 days
(0 Reviews)