Data pipelines transport raw data from software-as-a-service (SaaS) platforms and database sources to data warehouses for use by analytics and business intelligence (BI) tools. Developers can build pipelines themselves by writing code and manually interfacing with source databases — or they can avoid reinventing the wheel and use a SaaS data pipeline instead. To understand [...]

This is a presentation on how to migrate from data lake to data mesh architecture. It provides pros and cons of data mesh over data lake. Evolution of mesh and need for data mesh in today’s data world.

Data engineers are data professionals that are tasked with creating and maintaining complex systems that are capable of collecting, storing, and analyzing big data for companies. The way Kafka treats the concept of data is entirely different from what we have thought of data to be. Kafka provides a more robust, fault tolerant and distributed [...]

Data engineers must be adept in several areas. With years of SME experience on conventional tools, moving data to Hadoop will be a steep learning curve. Hadoop platform also offer a wide variety of services that are custom built to serve a specific purpose. A data engineer is expected to integrate with assortment of data [...]

The future of machine learning is declarative and data first. This talk will make the case for declarative ML and provide a peek at what this potential future could look like.

GitOps is an operational framework that takes DevOps best practices used for application development such as version control, collaboration, compliance, and CI/CD, and applies them to infrastructure automation. In cloud environments Terraform IAC is commonly used to deploy infrastructure resources. The automation of this is now encapsulated into well established patterns. In this presentation, using [...]