Data engineering as a field is still relatively new. And with that many tools and people with different backgrounds it started to resemble the wild west somehow. Data engineers yearn for a concrete set of principles and design patterns just like what...
Here is a little fun side project to help you get your feet wet with Apache Kafka and Apache Streaming and at the same time give you a little overview of how different tools can work together to provide real world value. We are going to build a simpl...
Many projects rely on Apache Zookeeper for cluster coordination and Apache NiFi is no different. But what makes Apache NiFi different is that it comes with an embedded Zookeeper, so you don’t need to set up a dedicated Zookeeper cluster if you are ju...
While designing any system, one must prepare an error handling mechanism. This is even more critical when you are designing a data pipeline. With data being one of the most abandon assets out there, it can come with lots of issues. As well as the dat...
