My Attempt - And Failure - In Creating A Poor Man's Data Warehouse Using DuckDBI know what some of you are probably thinking: "Oh great, another article about DuckDB." And honestly, I get it. A few months ago, I was in the same boat. It felt like DuckDB was following me around—blog posts, tweets, random mentions. It was everywh...May 9, 2025·5 min read
My Data Engineering PrinciplesData engineering as a field is still relatively new. And with that many tools and people with different backgrounds it started to resemble the wild west somehow. Data engineers yearn for a concrete set of principles and design patterns just like what...May 31, 2022·6 min read
Building a data pipeline using BeautifulSoup, Apache Kafka, Apache Spark Streaming and MySQLHere is a little fun side project to help you get your feet wet with Apache Kafka and Apache Streaming and at the same time give you a little overview of how different tools can work together to provide real world value. We are going to build a simpl...Mar 25, 2022·8 min read
Apache Nifi: Move from embedded Zookeeper to external oneMany projects rely on Apache Zookeeper for cluster coordination and Apache NiFi is no different. But what makes Apache NiFi different is that it comes with an embedded Zookeeper, so you don’t need to set up a dedicated Zookeeper cluster if you are ju...Feb 23, 2022·5 min read
Error Handling: Expecting the UnexpectedWhile designing any system, one must prepare an error handling mechanism. This is even more critical when you are designing a data pipeline. With data being one of the most abandon assets out there, it can come with lots of issues. As well as the dat...Dec 12, 2021·5 min read
