Lead Data/AI Engineer

Our client is the leading provider of insights and analysis for the world's top insurers, distributors, service providers, and investors. At the moment they need a senior or Lead Data/AI Engineer.

If you are not afraid to:

✔︎ Design, build, and maintain data pipelines and orchestration workflows using Databricks, Python, and/or TypeScript/Node.js – covering ingestion, transformation, scheduling, and monitoring.

✔︎ Write clean, well-structured scripts and services to ingest, process, and transform data from APIs, databases, and third-party sources – using Python for data work and Node.js/TypeScript where appropriate for integrations or lightweight services

✔︎ Line-manage, mentor, and grow a small team of data engineers – running 1:1s, supporting career development, giving clear feedback, and helping the team do the best work of their careers.

✔︎ Set technical direction and standards – making sound architectural decisions, defining how the team works, and balancing speed of delivery against long-term maintainability

✔︎ Define requirements for billing, subscriptions, invoicing, and revenue workflows in partnership with Product and Finance

✔︎ Lead the end-to-end build-out of a new billing system – from discovery and vendor/build evaluation through to delivery and adoption

✔︎ Integrate a usage-based billing system such as Orb or Metronome – including designing the usage logging layer that captures metering events (API calls, data queries, MCP tool invocations, records returned) from pipelines and services, and making that data reliably available to the billing platform for customer charging and entitlement enforcement

✔︎ Configure and maintain dashboards, reports, and analytics tools to surface insights from content and data – connecting pipeline outputs to BI tools and keeping metrics accurate and up to date.

✔︎ Work closely with editorial, product, and commercial teams to understand their data needs and translate them into practical metrics and visualisations.

✔︎ Connect content assets with structured data using scripted pipelines – enabling smarter search, tagging, categorisation, and intelligence features across our platform.

✔︎ Identify new data sources and APIs, scope their integration, and build the connectors needed to operationalise them – fast.

✔︎ Build and maintain MCP server integrations that expose Insurance Insider’s proprietary data to AI agents and client workflows.

✔︎ Design and implement data-sharing infrastructure – enabling structured, permissioned delivery of datasets to customers via APIs, data-sharing platforms, or direct integrations with client systems.

✔︎ Leverage AI coding tools (e.g., GitHub Copilot, Claude, Cursor) to write, debug, and iterate on data scripts, pipelines, and orchestration logic quickly and independently.

✔︎ Develop working prototypes for AI-based products and/or AI internal tools, using AI coding tools to create quickly and iterate

✔︎ Work across teams – product, editorial, commercial, and leadership – to understand data needs and surface the right insights at the right time.

✔︎ Communicate data findings clearly to non-technical audiences, making complex outputs accessible and actionable.

✔︎ Monitor data quality across pipelines and outputs, flagging issues proactively and implementing fixes to maintain reliable, trustworthy data.

✔︎ Continuously improve data processes, tooling, and documentation so that the team’s data assets are scalable, maintainable, and easy to build on.

And your profile looks like that:

✔︎ 5–8 years of hands-on software or data engineering experience, with strong proficiency in Python and working knowledge of TypeScript or Node.js – comfortable writing production-quality code, not just scripts.

✔︎ Proven experience designing and running data pipelines and orchestration workflows in production – with solid understanding of scheduling, dependency management, retries, alerting, and data quality patterns.

✔︎ Solid experience with Databricks, Spark, or a comparable cloud data platform; confident with SQL, Delta tables, and both notebook-based and job-based pipeline architectures.

✔︎ Experience connecting data pipelines to analytics or BI tools (e.g., Power BI, Looker, Metabase) and maintaining the data models that underpin them.

✔︎ A strong engineering foundation combined with a generalist mindset – able to move between data engineering, analytics, and integration work and make sound technical decisions independently.

✔︎ Experience working at a business that sells data as a product – whether through direct licensing, API access, or data sharing platforms – with a solid understanding of the data product lifecycle, access control, and the commercial sensitivities of proprietary datasets.

✔︎ Some experience managing or leading other data roles – whether line managing analysts or engineers, setting technical direction for a small team, conducting code reviews, or mentoring junior colleagues. You don’t need to have run a large team, but you should be comfortable owning quality and direction across the data function.

✔︎ Demonstrated use of AI coding assistants (e.g., Copilot, Claude, and Cursor) as a genuine multiplier – not a crutch. You write good code, use AI to go faster, and know the difference.

✔︎ A degree in computer science, engineering, or a related technical field – or demonstrably equivalent experience. We care more about the quality of your engineering than where you studied.

✔︎ Experience with orchestration tools such as Airflow, Dagster or Databricks Workflows; be confident building REST API integrations, webhook consumers or event-driven data patterns in Node.js or Python.

✔︎ Hands-on experience with MCP (Model Context Protocol) – building or consuming MCP servers, understanding tool definitions, and thinking through the access, security, and versioning challenges of exposing commercial data via this protocol.

✔︎ Familiarity with usage-based billing platforms such as Orb or Metronome – including how to design and instrument a usage logging layer that reliably captures metering events (such as API calls, data queries, MCP tool invocations, and records returned) from pipelines and services; how those events are ingested and processed by a billing platform; how usage is tracked against customer entitlements and plans; and how billing data surfaces in finance, product, and customer-facing reporting.

... then you’re the right person in the right place!

They offer, not just a nice team and environment, but:

✔︎ Flexibility with true hybrid working - expected to be in the office 1-2 days a week.

✔︎ 25 holiday days per year, plus your birthday off.

✔︎ Opportunities for professional growth and development.

✔︎ Competitive compensation and benefits package.

Are you still here? Great! Let’s discuss further at [email protected]

(Recruitment License № 2709/ 17.01.2019)