Data Engineer
JOB SUMMARY
We are looking for an experienced Data Engineer to take ownership of our data infrastructure's efficiency, performance, and reliability. The ideal candidate will be responsible for auditing our current systems, identifying areas for improvement, and implementing robust solutions to ensure our data is both performant and trustworthy. This role is critical in establishing a "ground truth" data source that will empower our data analytics and data science teams to derive actionable insights with confidence. You will be instrumental in building and maintaining a scalable and resilient data ecosystem that supports the company's strategic objectives.
KEY RESPONSIBILITIES
Infrastructure Auditing & Optimization
- Conduct comprehensive audits of the existing data infrastructure to assess its efficiency, scalability, and performance.
- Identify and analyze performance bottlenecks, and propose and implement optimization strategies.
- Re-design and modernize data pipelines for greater scalability and reduced latency.
- Implement monitoring and alerting systems to proactively identify and address infrastructure issues.
Data Pipeline & ETL Development
- Design, build, and maintain robust and scalable ETL/ELT processes to ingest data from a wide variety of sources.
- Assemble large, complex datasets that meet both functional and non-functional business requirements.
- Automate manual data processes to improve efficiency and reduce the potential for human error.
Ground Truth Data Source Management
- Establish and maintain a centralized, reliable source of truth for all key business data.
- Implement rigorous data quality checks and validation processes to ensure data accuracy and consistency.
- Develop and enforce data governance best practices, including data lineage and metadata management.
Collaboration & Support
- Work closely with data scientists, data analysts, and other stakeholders to understand their data requirements and provide them with...