Big-Data Hadoop developer
Primologic System Pvt. Ltd.
- Hands on experience in Implementing Hadoop Cluster and different eco-system tools for analysis of Hadoop framework.
- Moved all crawl data flat files generated from various retailers to HDFS for further processing.
Developed Map-Reduce programs in HIVE and PIG to validate and cleanse the data in HDFS, obtained from heterogeneous data sources, to make it suitable for analysis.
The Hive tables created as per requirement were Internal or External tables defined with appropriate Static and Dynamic partitions, intended for efficiency.
- Worked extensively with Sqoop for importing and exporting data from MySQL into HDFS and Hive.
- Load and transform large sets of structured, semi structured using Hive.
- Connected Hive to Tableau reporting tool and generated graphical reports.
- Written the Apache PIG complex scripts to process the HDFS data.
- Deployed and configured Flume agents to stream log events into HDFS for analysis.