•Main Objective:
We are looking for a Data Engineer to join our growing Data Infrastructure team.
•Main Responsibilities:
-The hired will be responsible for expanding and optimizing our data ingestion pipeline architecture
-The Data Engineer will support our data scientists and business intelligence on data initiatives and will ensure our optimal data infrastructure is consistent, stable, and robust throughout ongoing projects
-The right candidate will be excited by the prospect of optimizing or even re-designing TAP30’s data infrastructure architecture to support our next generation of products and data initiatives
-She/He must be proactive and self-motivated supporting the data needs of multiple teams, systems and products
Requirements
-Bachelor’s in Computer Science or another quantitative field
-Over 3 years of experience in a Data Engineer role
-She/He should also have experience using the following software/tools:
•Big data tools: Hadoop (YARN, HDFS), Spark, Kafka
•Relational SQL and NoSQL databases, including Postgres and Cassandra
•Data pipeline and workflow management tools: Airflow, Luigi
•Stream-processing systems: Storm, Spark-Streaming
•Object-oriented/object function scripting languages: Python
-Experience building and optimizing data ingestion pipelines
-Being comfortable with working on Linux operated machines
-A successful history of manipulating, processing and extracting value from large disconnected datasets
-Strong project management and organizational skills
-Experience supporting and working with cross-functional teams in a dynamic environment