Big Data Engineer

Role OverviewWe are looking for a powerhouse Big Data Engineer to architect and manage our global data infrastructure. You will be responsible for handling petabyte-scale datasets, ensuring they are structured for both lightning-fast real-time ingestion and deep historical analytics. Your primary mission is to unify our distributed private cloud data lakes—stretching across our nodes in the US, Europe, and Budapest—into one orderly, high-performance ecosystem. This role is central to our ability to process massive market data feeds and provide our quantitative teams with the raw power they need to win.Key Responsibilities

• Data Lake Orchestration: Build and interconnect distributed private cloud data lakes, transforming disparate data silos into a unified, orderly structure.
• Real-Time Ingestion: Design and maintain robust pipelines for real-time data "injection" and streaming analytics, ensuring zero-loss capture of high-velocity market feeds.
• High-Performance Interfaces: Develop and optimize low-latency interfaces with external data providers, ensuring reliable data flow under extreme load.
• Database Architecture: Manage and tune petabyte-scale databases, selecting the appropriate structures for storage and retrieval efficiency.
• Pipeline Automation: Implement automated ETL/ELT processes that prioritize data integrity and performance, utilizing modern columnar storage formats.

Minimum Qualifications

• Database Mastery: Expert-level knowledge of SQL and high-performance columnar DBMS. You must have hands-on experience with tools like ClickHouse, BigQuery, or DuckDB.
• Storage Formats: Deep understanding of optimized file formats, specifically Parquet, Avro, or ORC, and how to leverage them for analytical performance.
• Programming Excellence: Strong software engineering fundamentals are a must. Proficiency in Python, Go, Java, or Scala for building data tools and pipelines.
• Distributed Systems: Experience managing data across distributed environments and understanding the complexities of data consistency and network latency.
• Data Engineering Ecosystem: Proficiency with tools like Spark, Flink, or Kafka for stream and batch processing.

Preferred Qualifications

• Capital Markets Expertise: Previous experience in High-Frequency Trading (HFT) or a deep understanding of market data structures (Level 2/3 feeds).
• kdb+/Q Experience: Experience with kdb+ and the q programming language for time-series data is a massive advantage.
• Cloud Infrastructure: Experience with Hybrid Cloud environments, specifically optimizing data egress/ingress costs and performance.
• HPC Knowledge: Familiarity with high-performance computing clusters and hardware-accelerated data processing.

Benefits

• Competitive Salary: A top-tier compensation package tailored to your experience and the global market.
• Ownership Stake: Employee Share Options (ESOP), allowing you to directly benefit from the company’s long-term growth and success.
• Professional Growth: Access to International Conferences (US, Europe, and beyond) to stay at the forefront of your field.
• Career Development: Structured Career Building Plans with clear milestones and the support needed to transition into leadership or specialized roles.
• Premium Mobility: A Company Car provided as part of your compensation package (subject to role/seniority level).
• Daily Perks: Complimentary lunch, snacks, and premium coffee to keep you fueled throughout the day.
• International Environment: The opportunity to work in a truly global setting with sister companies in the US and EU.