Data Engineer

**Role Summary:**

We are seeking a Data Engineer with strong experience in **Microsoft Purview and Metadata Management** to lead the integration of non-Microsoft data sources into our enterprise data governance platform. This role will focus on designing, developing, and optimizing metadata ingestion pipelines, enabling end-to-end visibility, classification, and lineage of enterprise data assets across diverse platforms (cloud, on-premises, and hybrid).

**Responsibilities**:

* Design and implement connectors, pipelines, and workflows to integrate non-Microsoft data sources (databases, data lakes, SaaS applications) with Microsoft Purview.
* Configure and manage Self-hosted Integration Runtime (SHIR) for on-premises and third-party cloud source scanning.
* Develop custom metadata ingestion solutions using Purview REST API, Python SDK, and Apache Atlas/OpenMetadata-compatible frameworks.
* Implement and optimize classification rules, sensitivity labels, and lineage tracking for integrated data assets.
* Collaborate with stakeholders to map metadata schemas from source systems to Purview’s Data Map.
* Ensure compliance with data governance, security, and privacy regulations (GDPR, HIPAA, CCPA, etc.).
* Monitor and troubleshoot integration pipelines, resolving connectivity, authentication, and performance issues.
* Automate metadata synchronization using scripting (Python, PowerShell) and workflow orchestration (Azure Data Factory, Synapse Pipelines).
* Maintain documentation for integration processes, API usage, and source system configurations.
* Provide technical support for ongoing Purview platform enhancements and integration scalability.

**Qualifications:**

* 5+ years of experience as a Data Engineer, Integration Engineer, or similar role in cloud and hybrid environments.
* Hands-on experience with Microsoft Purview (Data Map, Data Catalog, Classification, Lineage, and APIs).
* Proficiency in Python and/or PowerShell for automation and custom connector development.
* Strong experience with ETL/ELT tools (Azure Data Factory, Synapse Pipelines, Databricks, or similar).
* Knowledge of data source authentication methods (OAuth2, Kerberos, API Keys, Service Principals).
* Experience integrating with non-Microsoft platforms (e.g., Oracle, Snowflake, AWS S3, SAP).
* Understanding of metadata standards and APIs (Apache Atlas, OpenMetadata, JSON-based schemas).
* Familiarity with network security configurations (private endpoints, firewalls, VPN, ExpressRoute).