Data Engineer Databricks / Python / Database
Overview:
We are seeking an experienced Data Engineer with strong expertise in Databricks, Python, and Database engineering. The ideal candidate will design, build, and optimize scalable data pipelines, ensure data quality, and support advanced analytics and business intelligence initiatives across the organization.
- Design, build, and maintain ETL/ELT workflows on Databricks.
- Develop scalable data pipelines using Python and Spark.
- Optimize data transformation logic for performance and cost efficiency.
- Work with structured and unstructured data from various sources.
- Develop and manage data models, schemas, and tables across databases (SQL/NoSQL).
- Implement data quality, validation, and monitoring frameworks.
- Collaborate with Data Scientists, Analysts, and Cloud Engineers to support analytics and ML workloads.
- Ensure data security, governance, and compliance in all solutions.
- Troubleshoot pipeline failures and continuously improve pipeline reliability.
- Integrate data from APIs, streaming sources, and cloud storage.
Required Skills:
Core Technical Skills:-
- Databricks (mandatory) – Delta Lake, Notebooks, Spark SQL, Jobs, Workflows
- Python – Data processing, Spark APIs, automation, optimization
- Database Engineering – Strong SQL, query tuning, schema design, stored procedures
- Apache Spark – RDDs/DataFrames, optimisation, partitioning
- Cloud Platforms (one or more): AWS / Azure / GCP
- ETL/ELT tools: ADF, Glue, Dataflow, or equivalent
- Experience working with large-scale datasets
- CI/CD for data pipelines (Git, DevOps, Jenkins, etc.)
Recommended Jobs
SAP/APO consultant (Newark, NJ)
This role provides support from IT perspective in Supply Chain and Master data area utilizing ECC and SAP APO applications. Job Duties - Provide support for the client from IT perspective in …
Project Management Intern
Job Description Job Title: Project Management Intern Posting Start Date: 11/4/25 Job Description: Primary Role About the Internship: Join American Water as an Project Management Intern …
Photonics Support Engineer I
Purpose of the Position The Photonics Support Engineer I is responsible for providing entry level technical support by solving problems, troubleshooting, and working with customers to guide them t…
Wildlife Technician
Cowley’s mission is to build a winning team that delivers exceptional service and consistently exceeds client expectations. Compensation: $55,000 - $85,000 per year (based on licensing, experien…
3rd Shift Machine Operator
Job Title: Machine Operator Job Description As a Machine Operator, you will be responsible for operating blow-mold, grinders, trimmer machines, and conveyor systems. Your role will involve trou…
Lead Guest Services Team Member
Pilot Company, formerly known as Pilot Flying J, is the leader in providing premium services to travelers and truck drivers across North America. As we continue to expand our hospitality offerings at …
Outreach Sales Specialist - NJ/NY
Position: Outreach Sales Specialist Location: On site, New Jersey/New York At Whizz, we are on a mission to revolutionize the transportation industry for delivery drivers. We believe that eve…
Veterinary Assistant - Route 516 Animal Hospital
Overview: Care for animals. Support your team. Make a difference every single shift. At Route 516 Animal Hospital , we know it takes a village to care for pets and Veterinary Assistants are t…
Activity Assistant (Trio Sun-Thu)
ACTIVITY ASSISTANT -Trio (Arts & Crafts Focused) (Sunday-Thursday Schedule) Job Details: Plan, develop, organize, implement, evaluate, and direct activity programs of the community, assuming r…