Data Platform Engineer
- Zagreb, Croatia
- [email protected]
Innovative IT novice offering his full knowledge and experience to deliver highly effective and creative solutions to technology challenges. Big data enthusiast whose hunger for dealing with constant challenges is infinite.
Master thesis about building/implementing Lambda architecture and how it can help in some business cases, save products and make engineers life easier.
In thesis you can find whole architecture built from scratch and business case where this kind of architecture saved the same from collapsing.
Final work about distributed databases in Apache Cassandra.
Managing alarms and writing plugins
Data modeling, cluster 6 node modeling
Setting up Hadoop, YARN, MR2, Hbase, Hive, Spark2, Zookeeper cluster
Experimenting with AKS engine and configuring hybrid Linux & Windows K8 cluster.
Administrating Kubernetes with Confluent Platform environment on it.
Setting up platform which handles fault-tolerance, availability, Real-Time and Near Real-Time data processing.
Intention was on implementing Lambda Architecture where for batch layer was used HDP and stream/serving layer outsourced Confluent Platform, KStreams and Cassandra. Kafka-Connect was heavily there.
I was tuning and administrating whole stack and my main job was to take care on infrastructure as on my own kids where Nagios helped big time.
The most experience is in Kafka and Cassandra as most projects were based just on streaming layer and most issues came from that side.
Cassandra, HortonworksPlatform (Hadoop, YARN, Spark2, Zookeeper, Hive), Confluent Kafka, Docker, Nagios, Scala/Java, Bash.
Tested throughput and latency of Apache Cassandra on Multi-node Cluster using Apache Spark as engine for big data processing. Also trying to compare the same results with Microsoft SQL Server performance.
Also was testing performance of Hadoop multi-node cluster configuration with Apache Drill and Zookeeper along with.
COMPLETED AS THE TOP 10% OF ALL STUDENTS
Created kafka-connect-hbase-sink for Hortonworks Data Platform v2.6 which has HBase v1.1.2 (by then, was available sinker for HBase v1.2)