Spark Scala development
₹600-1500 INR
Paid on delivery
how to dynamically loop thorugh cols in df in spark and get their values
I have a list of tables let say x,y,z and each table is having some cols for example test,test1,test2,test3 for table x. just like we have cols like rem,rem1,rem2 for table y. Similarly is the case for table z. Now the requirement is that we have to loop through each col in a table and have to get row count based on below scenario's.
If test is not NULL and all other are NULL(test1,test2,test3) then it will be one count.
Now we have to loop through each table and then find cols like which start with test like test1,test2,test3 then match the above condition then marked that row as one 1 count if it satisfy above condition. different tables can have different no of cols.
how to load cols dynamically in a df and then validate the condition then mark it as count.
Project ID: #16587205
About the project
2 freelancers are bidding on average ₹1025 for this job
I have extensively worked on -Spark Transformations/Actions. -Spark DataFrames API, -Different file formats like Avro,Json,Parquet etc Questions: 1)What is the input file format?