Tools & Technologies
Redshift, DynamoDB, Data Pipeline, RDS, EMR, EC2, Python, SQL, Bash/Shell, ETL
Worked as a Data Engineer for a famous books publisher based in the US.
Role & Responsibilities:
- Data analysis and design of source tables as well as warehouse tables.
- Write and use Sqoop scripts to source data into staging environment in Redshift.
- Create Data Pipeline templates as per new requirements of data ingestion.
- Transformation of existing DML logic written in AS400 iSeries into AWS Redshift.
- Design and write ETL scripts to load facts and dimensions.
- Create mapping and analysis documents outlining the transformation logic and mapping between source table's columns and target table's columns.
- Write scripts and build frameworks in Python for different data manipulations and configurations.
- Work directly with the client team in Agile model. Requirements gathering and sprint planning are the major agendas of daily/weekly meetings.