Large Volume Data Replication from MySQL to Snowflake

In this talk, we will be covering the journey that we took from originally introducing StreamSets into our architecture to where we are today. We will go through the challenges around syncing and replicating large volumes of data from MySQL to Snowflake to support near real-time use cases. Additionally, we will explore how StreamSets, MySQL, and Snowflake can work in cohesion to minimize bottlenecks within the system. Finally, we will be touching on how Kubernetes can be utilized for deploying StreamSets resources.

Sean Vanderbach

Data Engineer

Skillz

Biography:

Sean has been working at Skillz the past 2 years primarily data infrastructure projects. A large portion of this time has been spent implementing and optimizing large volume data syncs between MySQL and Snowflake. Before that he was a student at the University of Colorado, studying Computer Science.