Middle+/Senior Data Engineer

Responsibilities Building both batch and streaming pipelines in production environments

Requirements (Must-have) 4-5 years of experience in Data Engineering Strong experience with Apache Spark (including Structured Streaming) Experience building both batch and streaming pipelines in production environments Proven experience deg AWS-based data lake architectures (S3, EMR, Glue, Athena) Experience with event streaming platforms such as Apache Kafka or Amazon Kinesis Experience implementing lakehouse formats such as Delta Lake Strong understanding of partitioning strategies and schema evolution Experience using SparkUI and AWS CloudWatch for profiling and optimization Strong understanding of Spark performance tuning (shuffle, skew, memory, partitioning) Proven track record of cost optimization in AWS environments Experience with Docker and CI/CD pipelines Experience with Infrastructure as Code (Terraform, AWS CDK, or similar) Familiarity with monitoring and observability practices

Nice to Have Experience in the financial domain Experience running Spark workloads on Kubernetes Exposure to or interest in Large Language Models (LLMs) and AI integration Experience implementing data quality frameworks or metadata/lineage systems

Hiring Process Intro call Technical discussion (focused on real experience) Offer Start: ASAP