Job description

The Senior Databricks Data Engineer is responsible for supporting the development, migration, and optimization of an enterprise data platform within the investment domain, leveraging Databricks, PySpark, Python, SQL, and cloud-based data platforms. The role focuses on building and maintaining scalable data pipelines and lakehouse data models using Delta Lake and lakehouse architecture patterns to support investment analytics, portfolio management, risk analysis, and trading data workflows. The position requires hands-on expertise in Databricks architecture, including job orchestration, cluster management, and performance tuning, along with advanced SQL and data modelling techniques such as dimensional modeling and query optimization. Responsibilities include migrating datasets, pipelines, and transformation logic from Palantir Foundry to Databricks Delta Lake, including pipeline conversion, data model redesign, and production cutover. The engineer works extensively with financial datasets including market data, portfolio holdings, transactions, pricing data, risk metrics, and financial instruments. The role also involves building ETL and ELT pipelines to enable data availability for analytics, reporting, and portfolio management systems, implementing data validation, reconciliation, and data quality frameworks to ensure accuracy and consistency, and maintaining data lineage, documentation, and governance aligned with financial industry standards. Additional responsibilities include optimizing Spark jobs, cluster configurations, and storage formats for performance and cost efficiency, and supporting monitoring, troubleshooting, and performance tuning of production pipelines using tools such as Dynatrace and Datadog. The position operates in a cloud environment (Azure preferred) with responsibilities spanning storage, security, networking, and enterprise data architecture, requiring collaboration with data engineers, quantitative analysts, and investment stakeholders to deliver reliable and performant data solutions.