Data Engineer

Iris's direct client, a leader Payroll, is looking to hire a strong Databricks Platform Engineer for a Long Term Contractual role.

Job title: Data & AI Platform Engineer — (Databricks + AWS)

Location: Hybrid (3 days onsite, Roseland, NJ)

Duration: 18 Months

Skills: Databricks, Pyspark, Python, Genie Space Or Genie Agent

Job Description

About the Role

Own the Databricks lakehouse layer and AI/LLM platform for Finance Insights. You'll design the data architecture on Databricks Unity Catalog, build Genie Spaces for natural language analytics, and integrate LLMs to power intelligent financial workflows — all running on AWS.

Responsibilities

• Design and build Delta Lake tables on Databricks (bronze/silver/gold medallion architecture) for financial datasets (AR, GL, invoices, collections)
• Build and manage Databricks Genie Spaces — configure semantic models, write GENIE-compatible SQL, and tune NL-to-SQL behavior for finance use cases
• Develop PySpark and Python-based data pipelines (Databricks Jobs + Workflows) for ERP data ingestion, transformation, and aggregation
• Integrate Databricks Model Serving with LLMs (Claude, GPT-4, or custom fine-tuned models) for AI-driven insights, anomaly detection, and financial forecasting
• Build RAG pipelines using Databricks Vector Search over financial documents, contracts, and GL notes
• Expose Databricks SQL Warehouse endpoints to Python API services and the React frontend
• Manage Unity Catalog governance — row-level security, column masking, data lineage for finance compliance
• Deploy infrastructure on AWS (S3 as Databricks external storage, IAM roles, VPC networking, Secrets Manager)

Required Skills

• Databricks (PySpark, Delta Lake, Jobs/Workflows, SQL Warehouses) — 3+ years
• Databricks Unity Catalog and governance
• Databricks Genie Spaces (semantic layer, NL analytics)
• Python 3.11+ for data engineering and API integration
• AWS (S3, IAM, VPC, Secrets Manager, optionally Lambda/Glue)
• LLM integration: Databricks Model Serving, MLflow, or direct API calls to Claude/OpenAI
• SQL (advanced — window functions, CTEs, query optimization)