Spark Scala development

Closed Posted 6 years ago Paid on delivery
Closed Paid on delivery

how to dynamically loop thorugh cols in df in spark and get their values

I have a list of tables let say x,y,z and each table is having some cols for example test,test1,test2,test3 for table x. just like we have cols like rem,rem1,rem2 for table y. Similarly is the case for table z. Now the requirement is that we have to loop through each col in a table and have to get row count based on below scenario's.

If test is not NULL and all other are NULL(test1,test2,test3) then it will be one count.

Now we have to loop through each table and then find cols like which start with test like test1,test2,test3 then match the above condition then marked that row as one 1 count if it satisfy above condition. different tables can have different no of cols.

how to load cols dynamically in a df and then validate the condition then mark it as count.

Scala Spark

Project ID: #16587205

About the project

2 proposals Remote project Active 5 years ago

2 freelancers are bidding on average ₹1025 for this job

nmogilip

Hi, I have very good experience in developing the big data applications using spark dataframes with java, scala and python as well. I very much clear with your requirement which I came across many times. I will deliver More

₹1300 INR in 1 day
(4 Reviews)
3.2
rbajaniy

I have extensively worked on -Spark Transformations/Actions. -Spark DataFrames API, -Different file formats like Avro,Json,Parquet etc Questions: 1)What is the input file format?

₹750 INR in 1 day
(0 Reviews)
0.0