Hadoop Jobs

I have a Hadoop cluster holding several large data sets, and I need a seasoned PySpark developer who also writes rock-solid SQL. The immediate aim is to connect to the cluster (YARN/HDFS with Hive metastore), develop or refine PySpark jobs, optimise the accompanying SQL, and make sure everything runs smoothly end-to-end. You’ll receive access to a staging namespace plus a sample of the data. Once the logic checks out we’ll promote the code to the full environment. Deliverables • A clean, well-commented PySpark notebook or .py job that executes successfully on the cluster • The corresponding SQL script or view definitions ready for Hive or spark-sql • A concise README detailing execution steps, parameters, and expected outputs Acceptance criteria &bul...

I need an experienced Python engineer who works confidently with AWS Glue to build and manage a small suite of data-integration jobs for a Hyderabad-based project. The core of the work is to design and automate Glue ETL pipelines that pull data from our production databases, catalog it accurately, and transform it into analytics-ready tables. Here is what I expect from the engagement: • Develop, test, and deploy Glue ETL jobs in Python. • Populate and maintain the Glue Data Catalog so new tables are discoverable and properly version-tracked. • Implement efficient transformation logic that cleans, enriches, and partitions data for downstream reporting. • Optimise job performance and cost by selecting the right worker types, job parameters, and database connectio...

Hiring: DevOps Developers DevOps Engineer Requirements: -Expertise in CentOS, Ubuntu, Debian -Experience with HBase, Hadoop, Storm, ArangoDB, Prometheus -Strong benchmarking and system scalability testing skills Responsibilities: -Configure and optimize high-availability servers and databases -Implement monitoring and performance evaluation tools -Review and enhance DevOps workflows Long-term opportunities available!

- **Core Architecture:** Spring Cloud + Kafka + Hadoop + Python Automation，This project requires a certain level of technical expertise.

I have an existing analytics initiative that now needs a dedicated Redshift-based warehouse. The core objective is to design and implement a robust schema in Amazon Redshift, then ingest data coming from three different sources—our operational SQL databases, a set of RESTful APIs, and periodic flat-file drops in CSV or JSON. Here is what I’m aiming for: • A well-structured Redshift warehouse (star or snowflake schema, whichever is most appropriate) built to scale and documented clearly. • Reliable, automated ingestion pipelines for each source type. For SQL we currently use PostgreSQL and MySQL; for APIs the payloads are mostly JSON; the flat files live in S3. • Transformations that standardise data types, handle slowly changing dimensions, and enforce dat...

I need a propensity-modelling software package that plugs directly into our CRM system, website analytics, and sales database, unifying those streams into a clean, continuously updated dataset. On top of that data layer, the build must train, evaluate, and deploy the best-performing predictive models—whether regression, decision-tree, neural-network, or any other technique that proves superior—then surface the results through a lightweight web interface and an API our teams can call in real time. Key deliverables • Automated ETL jobs and data-quality checks for the three sources mentioned above • Modular training pipeline with experiment tracking, lift/ROC reporting, and feature-importance visuals • Scoring service exposed via REST (or GraphQL) endpoints plu...

Key Responsibilities: • Design, develop, and deploy AI/ML solutions end-to-end • Lead AI architecture and solution design for enterprise applications • Build and optimize machine learning and deep learning models • Deploy and monitor models in production environments • Collaborate with cross-functional teams including product and engineering • Mentor junior AI engineers and contribute to technical leadership • Conduct research and implement state-of-the-art AI techniques • Ensure data quality, security, and model performance optimization Required Skills & Qualifications: • 10+ years of experience in AI/ML or Software Engineering roles • Strong proficiency in Python and data processing libraries (NumPy, Pandas) • Hands-on experienc...

Project Description My current production workload runs entirely on AWS EC2, and while usage grows, monthly costs are increasing and application response times are degrading. I’m looking for an experienced AWS / DevOps engineer to perform a data-driven infrastructure audit and recommend optimizations. The engagement will begin with a deep-dive assessment of the existing environment, followed by a cost- and performance-balanced architecture proposal. All findings must be supported with measurable evidence, not assumptions. Phase 1 — Infrastructure Audit The audit should cover: EC2 CPU, memory, and disk I/O utilization Database behavior and performance (MySQL) Network flow and latency analysis CloudWatch metrics and logs review Identification of bottlenecks, ...

**Operating System:** CentOS 7.x (7.9 recommended) **Core Architecture:** Spring Cloud + Kafka + Hadoop + Python Automation **Core Package Name:** ``