Sr. Data Engineer – Big Data
SUMMARY OF JOB
As a part of the Data Tribe the Sr. Data Engineer is responsible to (a) help support the existing big data infrastructure on Google BigQuery ensuring that data is continuously and promptly available for analysis, and (b) devise new methodologies and infrastructure for reconstructing and enriching data at scale. This position is also the key subject matter expert in supporting internal users who leverage the big data environment to upload additional data sets. Your efforts will contribute to the great team mandate to deliver derivative big data products and solutions to customers and partners.
Key Areas of Responsibility:
Build and maintain optimized reliable data pipelines that facilitate deeper analysis, additionally develop queries for ad hoc business projects, that translate raw data into powerful features and signals as well as ongoing reporting.
Builds data processing frameworks that handle the business’s growing database and enrich big data with tertiary data at scale. Working with stakeholders in leveraging data with reporting and scientific tools while striving to continuously develop new and improved data engineering capabilities.
Provide senior-level contribution to a team that is responsible for the design, deployment, and maintenance of the business’s data platforms. Understand data processing needs and develop infrastructure solutions to support these initiatives.
Assist in the support and maintenance of existing big data infrastructure on Google BigQuery and develop processes and alert systems to monitor the health of the big data infrastructure (Java).
Create and maintain documentation for architecture, requirements, and process flows - supporting internal teams to assist with data integration with newly developed big data platform
Use SQL and Python to interact with big data infrastructure as needed for data engineering.
- Experience working closely with data science teams to arrive at data infrastructure solutions that are optimized for scale.
- Requires strong customer focus, organizational skills, emotional intelligence, and ability to manage through change.
- Excellent communications skills – comfort level with all levels of management and business and technical stakeholders.
- Bachelor's degree in Computer Science, Software Engineering, Computer Engineering, or other related degree
- 8 – 10 + years of concrete programming experience in core Java/C# with Software Development Life Cycle
- 2+ years of SQL experience with large data sets (experience with Google BigQuery an asset)
- Hands-on experience in writing complex, highly-optimized SQL queries across large data sets
- Must have experience in data modeling, ETL development, and Data warehousing.
- Deep understanding of object-oriented design principles
- Familiar with various design patterns, good component and modeling ability
- Solid understanding and experience of any relational databases (MySQL, Postgres, SQLServer, etc)