Data Engineer

Arzdigital Karaj

Posted 2 years ago

Job

Company

Salary

Applicant Insights

Job Description

As the data industry matures, more priorities are being placed on Data Pipelines and Infrastructure. So, design and building production data pipelines as well as automating data engineering (DE) workflows are the main tasks we are looking for assistance with. In a Python developer role, you’re responsible for extracting, transforming, and loading data (ETL) from a wide variety of data sources using SQL and big data technologies. Moreover, as a Python developer, you work closely with data scientists to make high-quality data available to them. Therefore, you are responsible for building and maintaining pipelines that feed data to the data scientists. Responsibilities: ● Develop and automate large-scale, high-performance, scalable data pipelines (batch and streaming) to drive faster analytics. ● Design new data architecture with excellent run-time characteristics such as low latency, fault tolerance, and availability. ● Maintain and monitor real-time analytics and big data systems to make sure about their reliability and resolve issues. ● Collaborate with a business intelligence team, data Scientists team, Ventures, and other teams to build data insights and help them to achieve their business goals. ● Design, implement and maintain scalable real-time and batch data pipelines handling billions of records. ● Maintain real-time analytics systems and big data systems and make sure their reliability and maintenances. ● Setup real-time analytics solutions depending on services. ● Propose new data architecture for new requirements and fine-tune the existing ones. ● Monitor data services and resolve issues in case of any incident.

Requirements

● Bachelor’s degree in Computer Science, Engineering, Mathematics, or a related technical discipline, or equivalent experience. ● More than two years of industry experience in software development, data engineering, business intelligence, data science, or related field with a track record of manipulating, processing, and extracting value from large datasets. ● More than a year of experience with traditional RDBMS technologies, SQL, and SQL tuning. ● More than a year of experience in data modeling, ETL development, and data warehousing. ● Experience in one or more scripting or programming language, especially Bash and Python. ● Having a deep understanding of Python (Asyncio and Lazy Evaluation) and data science libraries (NumPy, Pandas, TensorFlow). ● Having prior experience of Docker, Apache Kafka (or Google Pub-Sub), and Big Query. ● Know the ins and outs of SQL and NoSQL database systems, especially PostgreSQL. ● Be comfortable using ETL (extract, transform, load) systems. ● Knowledge of Data modeling and data warehousing. ● Knowledge of software engineering best practices. ● Knowledge of graph databases including Neo4j is a plus. ● Familiar with Blockchain core and/or cryptocurrency market is a plus. ● Able to work in teams and in a fast-paced agile development environment.

Employment Type