Miswag Engineering Blog

Not every system is ready to listen the moment something worth capturing happens. Event buffering is the practice of temporarily storing events until the consuming system is prepared to receive them — turning a race condition into a guarantee. This article introduces the pattern, explains when and why it's needed, and walks through how it works.

Understanding the core functionality of your tools — and the features they already provide — can save you valuable time and lead to cleaner, more reliable solutions. Often, the best fix isn’t rewriting logic, but fully leveraging the capabilities that are already built in.

Building effective batch data pipelines is fundamental to modern data engineering. This article focuses on implementing batch data pipeline patterns specifically in a SQL, Python, Mage AI, and ClickHouse ecosystem.

In Amazon RDS, binary logs (binlogs) serve as critical components for recording all changes to your database, including INSERT, UPDATE, and DELETE operations. These logs enable essential functionality such as replication, point-in-time recovery, and change data capture (CDC).

OpenMetadata and DataHub are both open-source platforms. Both tools offer similar functionalities for data cataloging, search, discovery, governance, and quality.