Senior Data Engineer - Python & PySpark

Senior Data Engineer - Python & PySpark

Job Summary

We are seeking an experienced Senior Data Engineer with strong expertise in Python, PySpark, SQL, and Big Data technologies .

The ideal candidate will be responsible for designing, developing, and optimizing scalable data pipelines and ETL/ELT workflows for processing large volumes of structured and unstructured data. The role requires hands-on experience with distributed data processing, cloud platforms, orchestration tools, and performance optimization of big data applications.

️ Key Responsibilities

Data Pipeline Development

• Design, develop, and maintain scalable data pipelines using:
  • Python
  • Apache Spark / PySpark
• Build reusable and efficient data processing frameworks.

ETL / ELT Development

• Develop and optimize ETL/ELT workflows for:
  • Data ingestion
  • Data transformation
  • Data processing
• Process large volumes of structured and unstructured data.

Big Data Processing

• Work with big data technologies such as:
  • Hadoop ecosystem
  • Hive
  • Spark
• Implement distributed computing solutions for high-performance processing.

Data Modeling & Warehousing

• Support:
  • Data modeling
  • Data architecture
  • Data warehousing solutions
• Ensure scalability and maintainability of data systems.

SQL & Database Management

• Write and optimize:
  • Complex SQL queries
  • Data transformation logic
• Work with:
  • Relational databases
  • Non-relational databases

Cloud & Orchestration

• Deploy and manage data solutions on cloud platforms such as:
  • AWS
  • Azure
  • GCP
• Work with orchestration tools like:
  • Apache Airflow

Data Quality & Governance

• Perform:
  • Data validation
  • Data cleansing
  • Data transformation
• Ensure compliance with:
  • Data governance
  • Security standards

Performance Optimization

• Optimize:
  • Spark jobs
  • SQL queries
  • Data pipelines
• Improve:
  • Scalability
  • Reliability
  • Processing performance

Collaboration & Agile Delivery

• Collaborate with:
  • Data Analysts
  • Data Scientists
  • DevOps teams
  • Business stakeholders
• Participate in:
  • Agile ceremonies
  • Sprint planning
  • Continuous improvement initiatives

✅ Required Skills

Programming & Data Engineering

• Python
• PySpark
• Apache Spark
• SQL

Big Data Technologies

• Hadoop ecosystem
• Hive
• Distributed computing platforms

ETL / ELT & Orchestration

• ETL / ELT pipelines
• Apache Airflow or similar orchestration tools

Cloud Platforms

• AWS / Azure / GCP
• Cloud-based data services

Databases & Data Warehousing

• Relational databases
• NoSQL databases
• Data warehousing concepts
• Data modeling

File Formats

• Parquet
• Avro
• JSON
• CSV

Soft Skills

• Strong analytical and troubleshooting skills
• Excellent communication and collaboration abilities
• Ability to work with cross-functional teams

Experience Required

• 6-10+ years of experience in:
  • Data Engineering
  • Big Data technologies
  • Distributed data processing

Preferred Skills

• Performance tuning and optimization expertise
• Experience with scalable cloud-native data architectures
• Exposure to DevOps and CI/CD for data platforms