Lead Data Engineer / Architect

Fulltime / Direct Hire Position

Location - Chicago, IL / Remote (should be local to Chicago and be able to go into office if needed)

Our client is looking to add a Lead Data Engineer / Architect to their team that will have full authority over data architecture decisions and the engineering roadmap. You will stand up a Databricks (or Snowflake, or Hybrid)-based Data Lakehouse from greenfield, define the architectural patterns the team will build on for years, and grow a small high-caliber team.

Key Responsibilities

Data Architecture & Lakehouse Design

• Design and own the end-to-end data architecture. Example: Databricks(or Snowflake, or Hybrid)-— Unity Catalog, Delta Lake, medallion layer structure (Bronze / Silver / Gold)
• Define data modeling standards: Example: star schema, dimensional modeling, and semantic layer design using dbt
• Establish data governance frameworks including lineage, cataloging, and access controls
• Own the technology selection process in partnership with the COO, consultants, and other stakeholders.
• Ensure the architecture can support investment analytics, portfolio reporting, and operational workflows at scale

Engineering & Pipeline Development

• Build and own the ingestion layer: Example: Fivetran/Airbyte connectors for custodians, Bloomberg, and internal systems
• Develop and maintain dbt transformation models, tests, and documentation
• Configure and manage pipeline orchestration using Airflow or Prefect
• Write clean, production-grade Python for data pipelines, custom connectors, and analytical tooling
• Build internal dashboards and data applications using for example Plotly Dash or Streamlit
• Integrate Claude Enterprise (Anthropic) into data workflows for AI-assisted analysis and automation

Team Leadership & Consultant Oversight

• Recruit, interview, onboard, and mentor direct reports — you define who joins this team
• Define scope, manage delivery, and sign off on all work from external consultants (pipeline specialists, viz specialists)
• Ensure all consultant engagements include knowledge transfer and documentation before close
• Build a team culture grounded in engineering excellence, documentation discipline, and continuous improvement

Stakeholder Communication & COO Reporting

• Report directly to the COO with regular updates on build progress, risks, and resource decisions
• Translate complex technical decisions into clear business terms for investment and operations leadership
• Collaborate with portfolio managers, operations, and compliance teams to understand and prioritize data needs
• Own the data roadmap and communicate trade-offs transparently

Required Qualifications

Experience

• 7+ years in data engineering, data architecture, or a combined role
• Demonstrable experience designing and building a modern Data Lakehouse or cloud data warehouse from the ground up — not just maintaining one
• 3+ years working with cloud data platforms (Databricks or snowflake strongly preferred)
• Strong hands-on dbt experience: model design, testing, documentation, and semantic layer development
• Experience managing or mentoring engineers — you have grown people, not just pipelines
• Track record of working directly with senior business stakeholders and translating requirements into technical solutions

Technical Skills

• Databricks, Snowflake: Delta Lake, Unity Catalog, Databricks SQL, cluster configuration
• dbt Cloud: transformations, testing, documentation, multi-environment deployment
• Data ingestion: Fivetran, Airbyte, custom connector development
• Orchestration: Apache Airflow or Prefect (managed or self-hosted)
• Python: production-grade scripting, pipeline development, Plotly/Dash or Streamlit
• Data modeling: star schema, dimensional modeling, slowly changing dimensions
• SQL: expert-level; complex transformations, window functions, query optimization
• Git, CI/CD, and engineering best practices

Domain Knowledge

• Financial services or asset management experience strongly preferred
• Familiarity with custodian data feeds, Bloomberg API, or investment data sources is a significant advantage
• Working knowledge of data governance, lineage, and quality frameworks

Preferred Qualifications

• Direct experience with Databricks Unity Catalog rollout
• Hands-on experience with data catalog tooling (Atlan, Monte Carlo, or similar)
• Familiarity with Claude Enterprise or Anthropic AI tools in a production context
• Experience with SOC 2 data environments and security-conscious engineering practices
• Exposure to portfolio management systems, order management systems (OMS), or similar investment platforms
• Experience standing up a data team from zero — first hire, first pipeline, first architecture decision

For all non-bonus, non-commission direct hire positions: The anticipated salary range for this position is ($180,000 - $200,000). Actual salary will be based on a variety of factors including relevant experience, knowledge, skills and other factors permitted by law. A range of medical, dental, vision, retirement, paid time off, and/or other benefits are available.