Devoxx Poland 2019
from Monday 24 June to Wednesday 26 June 2019.
Serverless Jenkins DevOps pipeline with K8s and Spark
Methodology & Culture
|Room 5||Tuesday from 13:20 til 14:10|
Building a Spark/BigData application is hard because involves a universe of different technologies and components that, in a large enterprise, are typically centralized.
The need for centralized services often leads Development Teams to surrender and abandon a DevOps lifecycle.
However, this talk is about a different story, where a strong and motivated development Team decided to try harder and leverage the latest advances in Jenkins and its support for K8s to implement a true DevOps lifecycle where nothing is long-running and all resources needed to build, test and integrate the software are created on-demand.
This talk is a live session of a step-by-step construction of the DevOps pipeline: - creation of the ephemeral Jenkins on demand using the configuration-as-code plugin - allocation of the build agents on the clusters with the K8s plugin - tailoring of the minified version of the Hadoop/HDFS cluster for the tests - execution of the tests and collection of the results
The entire DevOps lifecycle is built using the declarative style Jenkinsfile pipeline, with some extra additions to integrate seamlessly to different runtime environments.DevOps serverless computing Kubernetes
Software engineer of the young generation. Used to engage in open source contribution for XWiki project as part of Google GSOC program. Mostly working with JVM environment, firstly developing web apps in Java, currently fighting Big Data in Scala for third largest retailer in the world.
Strong follower of the rule: think twice, code once. Still looking for his path through IT lands.