Job Information
C-Vision IT Data and Cloud Solutions Engineer in Troy, Michigan
Collaborate closely with cross-functional teams including Data Stewards, Architects, Developers and operations to identify needs and data-related technical issues in order to support the data infrastructure. Work with data stewards to refine or develop business-oriented metadata definitions. Work with architects to assemble complex data pipelines that meet functional/non-functional business requirements involving terabytes and petabytes of data using SQL, Python, Pyspark, BigQuery, Shell Scripting, DataProc, and Airflow. Develop complex and cost-efficient queries in BigQuery for data analysis, extraction and validation purposes. Identify and design internal process improvements including automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability and translating requirements into technical solutions by implementing POCs. Drive system design and architecture, development and unit testing of applications following established standards. Work with the engineering team to identify the right open-source tools to deliver product features by researching industry trends and best practices while adhering to enterprise direction and governance. Work hands-on to fine-tune compute parameters, configurations and resource allocation to reduce cost and performance bottlenecks on Cloud Platform. Conduct regular reviews with the product and engineering teams to gauge the performance of the pipelines vs. expectations and then rework and deliver if deemed necessary. Oversee and contribute to the creation and maintenance of relevant data artifacts (data lineages, source-to-target mappings, high-level designs, interface agreements). Use SQL, Python, Shell Scripting, Google Cloud Platform (DataProc, Cloud SQL, Compute Engine, GCS), Spark, Pyspark, Airflow, Jupyter Notebooks, Hive, BigQuery, GitHub, Jenkins, Jira, ServiceNow, Agile.
Work Location: Various unanticipated work locations throughout the United States; relocation may be required. Must be willing to relocate.
Minimum Requirements:
Education: Bachelor -- Computer Science or Electrical and Electronic Engineering (will accept foreign education equivalent)
Experience: Five (5) years