Azure Databricks Engineer with Python & Spark 2 Positions (Senior & Lead) Only W2

Role: Azure Databricks Engineer with Python & Spark (Senior / Lead)

Location: Weehawken NJ (Onsite from Day 1)

Job Type: W2 Contract

Note: 2 positions are available - Senior & Lead

Experience Level: Senior OR Lead

Must have skills (Mandatory):

• Apache Spark

• Python

• Azure Data Factory

• Azure SQL Microsoft

• Azure Synapse Analytics

• ETL/ELT Solutions

Nice to have skills:

• Apache Kafka

• CI/CD

Experience & Qualifications:

• Expert level understanding of Azure Data Factory Azure Synapse Azure SQL Azure Data Lake and Azure App Service

• Working knowledge of Python

• Expert level skills writing and optimizing complex SQL

• Experience with complex data modeling ETL design and using large databases in a business environment

• Experience with building data pipelines and applications to stream and process datasets at low latencies

• Fluent with Big Data technologies like Spark Kafka and Hive

• Designing and building data pipelines using API ingestion and Streaming ingestion methods

• Knowledge of Dev-Ops processes (including CI/CD) and Infrastructure as code

• Experience in developing NoSQL solutions using Azure Cosmos DB

• Thorough understanding of Azure and AWS Cloud Infrastructure offerings

Key Responsibilities:

• Design and implement scalable and secure data processing pipelines using Azure Data Factory Azure Databricks and other Azure services

• Manage and optimize data storage using Azure Data Lake Storage Azure SQL Data Warehouse and Azure Cosmos DB

• Monitor and troubleshoot data-related issues within the Azure environment to maintain high availability and performance

• Automate data pipelines and workflows to streamline data ingestion processing and distribution tasks

• Utilize Azures analytics services such as Azure Synapse Analytics to provide insights and support data-driven decision-making

• Provide guidance and support for data governance including metadata management data lineage and data cataloging

Note: Visa Independent candidates are highly preferred