By now you have a good understanding of what is DataOps and the various practices that can be adopted to increase DataOps maturity. Let’s explore how DataOps can be implemented and the benefits that can be realized.

Implementing DataOps

There are multiple ways in which DataOps practices can be implemented to drive value for the business. Typically, greenfield applications as well as applications that are getting re-architected can derive the most benefit from DataOps programs. Here we discuss a few approaches.

Data platforms

There are on-premise as well as SaaS Data platforms that provide the key features to enable fast and secure build and deployment. Selection of the data platform will depend on the business use-case as well as the technology direction of the organization. These data platforms have many of the important capabilities required for DataOps (eg: DBT, DataKitchen, StreamSets, Composable)

Hyperscalers

Hyperscalers like Azure, AWS and Google offer multiple data and analytics products and packages which can be leveraged to quickly build and scale out applications (eg: Azure Machine Learning, Amazon SageMaker, Google Cloud AI). They also have tools to facilitate data transformation and data pipeline migration (eg: Azure DevOps, AWS CodePipeline)

Custom built data platforms

Teams can also stitch together a platform using a combination of open-source and licensed products. The technologies and tools should support integration and the strategies mentioned for DataOps. This approach offers more customization but may take longer to build. (eg: Apache Kafka, Airflow, Java/Python, Hadoop using tools like Git, Jenkins, Sonarqube, Artifactory)

It is recommended to follow a platform-based approach for greenfield and rearchitected applications. The DataOps platform can cater to all the key technologies used in the organization for DataOps and provide self-serve capabilities for multiple application teams. This approach allows for standardization of tools, consistent adoption, cost saving in licenses and helps achieve no-touch automation. The platform will address CI-CD along with automation testing and data security needs.

Many organizations already have Data and Analytics systems built with legacy COTS based systems and database packages. Implementing DataOps in COTS and database packages is a function of the automation support available in these platforms. End to end DataOps might either not be feasible or the ROI against effort and cost may not be justified. In such cases a practice-based approach for DataOps is recommended. A detailed time motion study helps uncover bottlenecks in the application and data pipeline and which are impacting release cycle, cost and effort the most. Addressing these bottlenecks first will give maximum cost and speed benefits.

Benefits of DataOps

Adopting DataOps provides multiple benefits to the organization.

• It accelerates feature releases for data programs from biannual/quarterly to weekly/daily giving agility to business users
• It improves end-to-end efficiency by providing capabilities to manage, govern, curate and provision data at speed
• DataOps allows data discovery and democratization with effective data lineage and catalog generation
• It reduces data related incidents by delivering high quality data on time while complying with security requirements
• DataOps provides observability and data provenance. It gives enhanced visibility into the data pipeline, data versions and allows scenario recreation
• It frees up the teams from manual activities and gives an opportunity to focus on innovative use cases like AI and ML

In summary, DataOps can assist in unleashing the power of information for driving business agility.