Senior Data Engineer

1. Data Pipeline Development & Management
• Design, build, and maintain robust data pipelines using AWS Glue • Implement ETL/ELT processes for data ingestion from multiple sources • Optimize data workflows for performance and scalability • Monitor and troubleshoot data pipeline failures and performance issues
2. Data Infrastructure & Engineering (with IT Department) • Manage and optimize AWS Redshift data warehouse operations • Configure and maintain data storage solutions (AWS S3, data lakes) • Implement data partitioning, indexing, and compression strategies • Support Infrastructure as Code (IaC) for data infrastructure deployment
3. CI/CD & DevOps for Data (with AWS partners and IT Department) • Develop and maintain CI/CD pipelines for data workflows using GitLab • Implement automated testing for data pipelines and data quality • Support version control and deployment strategies for data assets • Configure AWS Lambda functions for data processing automation
4. Monitoring...