PySpark Jobs

Responsible for designing and implementing large-scale data migration and ingestion pipelines to move high-volume data from diverse sources into cloud platforms. Sources include HDFS, relational databases such as MySQL and PostgreSQL, and real-time streaming systems like Kafka. Develop and maintain robust data pipelines using PySpark, ensuring efficient processing of batch and streaming data. Implement automated scheduling mechanisms to orchestrate data workflows on daily and monthly intervals, ensuring reliability and timely data availability. Optimize data ingestion and storage through advanced performance tuning, partitioning, and compaction strategies to handle large-scale datasets efficiently. Ensure data quality, consistency, and fault tolerance across all pipelines. Deploy and ma...

We are hiring an experienced Databricks Developer for support our data platform initiatives. This is a remote role aligned with US working hours. Location: Remote (Offshore – US Timings) Experience Required: 5+ Years (Data Engineering), 3+ Years (Databricks) Note: Budget is fixed. Please do not apply if you are looking to negotiate. Key Responsibilities Develop and optimize data pipelines using Databricks, PySpark, and Spark SQL Design and implement Delta Lake architecture (Bronze / Silver / Gold layers) Work on Lakehouse architecture and manage Unity Catalog Apply DataOps practices for scalable and reliable data workflows Optimize Spark jobs for performance and cost efficiency Required Skills Strong hands-on experience with Databricks Proficiency in PySpark and Spark SQL Experie...