Senior Databricks Lakehouse Architect

Description:

URGENT HIRING – Senior Databricks Lakehouse Architect / Lead
Job Title: Senior Databricks Lakehouse Architect / Lead
Location: New York, NY
Work Arrangement: 100% Onsite
Experience: 10+ Years

Job Overview
We are seeking an experienced Senior Databricks Lakehouse Architect / Lead to design, develop, and lead enterprise-scale Lakehouse solutions. The ideal candidate must have strong hands-on expertise in Databricks, PySpark, Delta Lake, Unity Catalog, and AWS data services, along with enterprise data architecture experience.

Required Skills and Experience
Strong expertise in Databricks Lakehouse architecture and workspace administration.
Hands-on experience with PySpark and Spark SQL.
Strong knowledge of Delta Lake and Lakehouse implementation.
Experience implementing Unity Catalog for data governance, security, and metadata management.
Experience with MLflow.
Strong AWS integration experience, especially AWS S3.
Experience with Auto Loader and Structured Streaming.
Knowledge of CI/CD pipelines and workflow orchestration.
Expertise in Spark performance tuning and optimization.
Strong understanding of enterprise data architecture, modeling, and governance.
Experience working with enterprise and insurance data domains.

Key Responsibilities
Lead Databricks Lakehouse architecture, administration, and implementation.
Design and develop scalable data pipelines using PySpark, Spark SQL, Delta Lake, Auto Loader, and Structured Streaming.
Implement Unity Catalog for enterprise data governance and security.
Integrate Databricks with AWS services, including S3.
Establish CI/CD, workflow orchestration, and production deployment practices.
Optimize Spark workloads and Lakehouse performance.
Manage enterprise data modeling, governance, and security.
Coordinate technical delivery across enterprise data teams.

Critical Requirements
10+ years of overall experience.
Strong hands-on Databricks Lakehouse architecture experience.
New York, NY local candidates only.
100% onsite work is mandatory.
Face-to-face interview is required.