Senior Data Engineer
by Deriv in FinTech & Digital Payments
The Senior Data Engineer is responsible for designing, building, optimizing, and maintaining highly reliable, scalable, and governed data infrastructure that powers trading decisions, product analytics, compliance reporting, fraud detection, artificial intelligence, machine learning systems, and business intelligence across Deriv's global fintech operations. The role owns end-to-end data accuracy, reliability, observability, governance, and performance for enterprise data pipelines supporting millions of traders operating continuously across multiple regulatory jurisdictions. The position develops and manages ETL and ELT pipelines for both batch and real-time workloads using AI-assisted development while ensuring reliability, scalability, and operational excellence. Responsibilities include embedding observability into every pipeline through data freshness monitoring, completeness validation, schema drift detection, lineage tracking, automated alerting, anomaly detection, and automated data quality controls to proactively identify issues before they impact analysts, stakeholders, compliance reporting, or trading operations. The Senior Data Engineer defines and manages data contracts, including SLAs, SLOs, schema agreements, producer-consumer alignment, access controls, PII handling, auditability, and governance frameworks suitable for regulated financial environments. The role optimizes cloud data warehouse performance, query efficiency, partitioning strategies, clustering, orchestration reliability, warehouse cost optimization, dimensional modeling, semantic layers, reusable abstractions, and scalable data platform architecture while continuously improving engineering standards and platform capabilities. The position partners with analysts, product teams, finance, compliance, engineering, and business stakeholders to transform business requirements into reliable, governed, and production-ready data products. Additional responsibilities include peer reviewing pipeline code, mentoring engineers, improving software engineering practices, onboarding new team members, documenting reusable resources, resolving root causes of data issues, implementing systematic corrective actions, supporting AI/ML feature pipelines, maintaining version-controlled pipelines, and ensuring production-grade reliability. The role requires extensive expertise in Data Engineering, ETL, ELT, Batch Data Processing, Real-time Data Processing, Data Pipelines, Data Infrastructure, Data Governance, Data Quality, GCP, BigQuery, Airflow, Python, SQL, dbt, Dataform, Kafka, Pub/Sub, CI/CD, Data Modeling, Kimball Star Schema, Data Vault, Medallion Architecture, Schema Registries, Data Contracts, Data Lineage, Observability, Anomaly Detection, Streaming Architectures, Cloud Data Warehousing, Compliance Reporting, Fraud Detection, AI-assisted Development, Workflow Orchestration, Query Optimization, Warehouse Cost Optimization, Semantic Layers, Partitioning, Clustering, Version Control, Peer Review, PII Protection, Auditability, Regulatory Compliance, and Containerized Data Platforms while contributing to mission-critical fintech infrastructure supporting global trading operations.