GitOps – StreamSets Platform Automation

GitOps is an operational framework that takes DevOps best practices used for application development such as version control, collaboration, compliance, and CI/CD, and applies them to infrastructure automation. In cloud environments Terraform IAC is commonly used to deploy infrastructure resources. The automation of this is now encapsulated into well established patterns. In this presentation, using StreamSets DataOps platform, Chris will look at how the existing best practices for infrastructure deployment automation can be extended into the control plane for ongoing management of the full data platform in respect to management of all metadata objects - e.g. 'Deployments','Engines','User Groups','Parameters' and security 'ACLs'. Chris will demonstrate how Terraform IAC can be extended to act as the complete orchestrator of every environment layer, showcasing the use of a custom Terraform provider. In addition, it will be shown that using a declarative approach for defining the full environment stack carries significant efficiency benefits, risk reduction and increases visibility and collaboration across the organisation, fully enabling a self service approach of the data platform for its developers.

Chris Finlayson

Co - Founder / Principal Engineer

bigspark

Biography:

Chris is a co-founder at bigspark and a multi-disciplined Principal Engineer, spanning Data Engineering, Software Engineering and DevOps. He has spent the last 10 years mainly building data platforms within financial services and has a particular focus in data engineering enablement and workload optimisation.