Data Engineer

Responsibilities:
Maintain and optimize the data infrastructure required for accurate extraction, transformation, and loading of data from a wide variety of data sources.
Design, build, and maintain batch or real-time data pipelines in production.
Develop ETL/ELT Data pipeline (extract, transform, load) processes to help extract and manipulate data from multiple sources.
Automate data workflows such as data ingestion, aggregation, and ETL processing and should have good experience with different types of data ingestion techniques: File-based, API-based, and streaming data sources (OLTP, OLAP, ODS etc) and heterogeneous databases.
Prepare raw data in Data Warehouses into a consumable dataset for both technical and non-technical stakeholders.
Requirements:
Should have a minimum 4+ years in Data Engineering platform(Pyspark, AWS is mandatory).
Should have a strong hands-on design and engineering background in AWS, across a wide range of AWS services, with the ability to demonstrate working on large engagements.
Should be involved in Requirements Gathering and transforming them into functional and technical design.
Strong experience and implementation of Data lakes, Data warehousing, and Data Lakehousing architectures.
Ensure data accuracy, integrity, privacy, security, and compliance through quality control procedures.
Monitor data systems performance and implement optimization strategies.
Leverage data controls to maintain data privacy, security, compliance, and quality for allocated areas of ownership.
Experience with AWS tools (AWS S3 EC2 Athena, Redshift, Glue, EMR, Lambda, RDS, Kinesis, DynamoDB, QuickSight, etc. ).
Strong experience with Python, SQL, pySpark, Scala, Shell Scripting etc.
Strong experience with workflow management & Orchestration tools (Airflow).
Should hold decent experience and understanding of data manipulation/wrangling techniques.
Demonstrable knowledge of applying Data Engineering best practices (coding practices to DS, unit testing, version control, and code review).
Big Data ecosystems, Cloudera/Hortonworks, AWS EMR etc.
Snowflake Data Warehouse/Platform.
Streaming technologies and processing engines: Kinesis, Kafka, Pub/Sub and Spark Streaming.
Experience of working with CI/CD technologies, Git, Jenkins, Spinnaker, Ansible etc.
Experience building and deploying solutions on AWS Cloud.
Good experience on NoSQL databases like DynamoDB, Redis, Cassandra, MongoDB, Neo4j, etc.
Experience with working on large data sets and distributed computing (e. g., Hive/Hadoop/Spark/Presto/MapReduce).
Good to have working knowledge on Data Visualization tools like Tableau, Amazon QuickSight, Power BI, QlikView etc.

Experience
4-7 yrs