Big Data Analytics Drives Organizational Agility to Bring Change

7 min readApr 15, 2021

Introduction

Business Analytics is not something new. Some companies have been doing this for years (e.g. forecasting sales using traditional systems of records). However, currently with the unprecedented growth of data and content from sources such as social media, web logs, mobile phones, wearables, sensors, etc. Mid and large companies around the globe have realized the importance of leveraging Big Data Analytics as part of their business strategy to boost sales. To go further in materializing the adoption of Big Data Analytics, companies should asses the acquisition of platforms (Hadoop, Storm, Spark, MongoDB , HBase, Hive) and related analytics software (i.e. Tableau, MS Power BI). These could be at their premises in their private cloud, or they could evaluate a partial hybrid or a complete solution provided by different vendors in the public cloud. At the same time, they should effectively use their current assets and optimize their coming expenditures. In this document, I will review the concepts of Grid, Cluster, and Cloud Computing and its flavors that united with Virtualization technologies have allowed Big Data to be processed over the “Cloud”, review the vendor offerings and tendencies, that will provide agility to any business.

Cluster, Grid, Cloud Computing

As described by “Heisterberg, Verma; 2014” Creating Business Agility, Cluster Computing tries to unify compute efforts of low cost computers working together to be viewed “as a single system”, while Grid Computing uses networked distributed computers to perform “multiple applications”, instead of having a dedicated computer and storage for each application. On the other hand, Cloud computing with the use of virtualization techniques, seem to unify both of these concepts to offer on-demand self service, broad network access, resource pooling, rapid elasticity and a measured service.

Cloud Computing definition provided by the National Institute of Standards and Technologies:

“…a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources…that can be rapidly provisioned and released with minimal management effort or service provider interaction.”

Virtualization Types and Usage

As companies have experimented in the recent years with the adoption of hardware virtualization, having the possibility to use a single hardware to create virtual machines (VM), each virtual machine managing its own guest kernel operating system (OS). Either using host OS or Hypervisor to manage VMs directly upon bare metal, virtualization have allowed scalability and flexibility in the deployment of grid computing or cluster computer, adding boxes utilization efficiency. Even more, virtualization is a term that englobes other different aspects as well, like desktop virtualization, networking virtualization, software virtualization, memory virtualization, file system virtualization, application virtualization and services virtualization. In addition, when talking about virtualization comes the term container (a virtual environment that isolate applications own resources and depends on host OS), which have spread in popularity with the use of microservices and kubernetes to orchestrate containers also in the cloud. See figure 1 by by “Brew P., 2018” in blog.netapp.com

Figure 1. Virtual Machine architecture vs Container architecture

Cloud Native Applications

Another aspect we need to add to this business agility are the strategies used in the development of software applications that provide services facing either internal or external systems/users, i.e. front end web sites and API (application programing interfaces) services. Developers in each specific case evaluate the use of SOA (Service Oriented Architecture) / SOAP (Simple Object Access Protocol) / REST(Representational State Transfer) to elaborate software that provide business capabilities. Nowadays, the cloud computing vendors talk about Cloud Native Applications and serverless. As shown in the following figure by “Hall S. 2018” in the her blog at thenewstack.io

Cloud offering: Models and deployments

Cloud models include Software as a Service (SaaS), Platform as a Service (Paas) and Infrastructure as a Service (IaaS). As stated in ““Heisterberg, Verma; 2014” Creating Business Agility ”, Software as a service is a model where the user only makes use of the software but does not need to operate or support the hardware or software below. E.g. Email, Cloud Microsoft 360 or CRM Sales force. Platform as a Service, gives companies the capability to deploy onto the cloud infrastructure their own applications using programming languages and tools supported by the provider. E.g. AWS Lambda. Infrastructure as a Service, give companies the capability to provision processing, storage, networks, and the other fundamental computing resources where the consumer is able to deploy and run arbitrary software, which can include operating systems and applications. In addition, Cloud Computing deployments include “Private Cloud” where companies build their own cloud targeting to serve only their internal users, “Public Cloud” (e.g. Amazon) where the vendor manages the cloud and companies make is of it; Hybrid Cloud where companies deploy make use of their private cloud and for example use the public cloud for peak events.

Public Cloud Vendors

In this perfect storm where technologies are converging to the cloud arena, cloud offering of IaaS, PaaS and SaaS from vendors like Amazon AWS, Google Platform, Microsoft Azure, Oracle Cloud, IBM Cloud, between others have reached to the point to charge their customers up to the second, millisecond, or per transaction depending on the products used, benefiting at the end companies adoption of Public Cloud. All of these vendors have specific solutions for developing for example: serverless Cloud Native Applications that at the end will deposit the data either in the company Data Warehouse or Data Lake (that can also be located in the public cloud or in the company private cloud). More importantly for our discussion, these vendors are offering the whole Big Data architecture and Analytics combining their different product alternatives (Hadoop, Storm, etc). With all of that, companies in different industries have the possibility to develop in the vendor’s cloud, the company’s Lambda or Kappa architecture (see Figure 4 and 5 by Marz N., Warren J., 2015” in Principles and Best Practices of Scalable Realtime Data Systems) and add Artificial Intelligence and Machine learning (ML) tool capabilities offered by these vendors, even with already conceived ML models developed for an easy to use Use Case development.

The Trend on Analytics and BI platforms

According to the latest Gartner magic quadrant for Analytics and Business Intelligence Platforms resumed by “King T. 2018” in his blog at solutionsreview.com, Gartner highlights augmented analytics as a strategic planning topic, a paradigm that includes natural language query and narration, augmented data preparation, automated advanced analytics, and visual-based data discovery capabilities. Gartner characterizes a modern analytics and BI platform as “easy-to-use tools that support a full range of analytic workflow capabilities.” These tools “do not require significant involvement from IT to predefine data models upfront as a prerequisite to analysis, and in some cases, will automatically generate a reusable data model.”

Figure 5. Gartner Magic Quadrant for Analytics and Business Intelligence Platforms

Until now we have reviewed the technological architecture, and the application architecture that drive the adoption of Cloud computing offering either Private, Public or Hybrid cloud allow companies a quick adoption requested to bring agility into the business and provide the final user a better user experience. This agility is actually referred to the business agility architecture that needs to be implemented.

Converging to Business Agility

As described by “Heisterberg, Verma, 2014” in the book Creating Business Agility : Business agility, defined as innovation via collaboration to be able to anticipate challenges and opportunities before they occur, produces a sustainable competitive advantage requiring people, process and tools to be align in order to provide a customer-centric business virtual enterprise in this competitive arena where adopting Big Data strategies is a must. One example aligned with the business is described in the CRISP-DM (Cross-Industry Standard Process for Data Mining) presented in the following figure:

Figure 6. CRISP methodology for Data Mining projects

As seen in the picture the development of data mining models start by understanding the business and the data both from sources of records and sources of engagements to develop data mining models oriented to bring insights that enable a competitive advantage and promote business agility actionable steps

These actionable steps in the business to promote agility can be measured in a Balanced Score Card, as shown in the column initiatives in a real-life example of the Sonoma County Wine Country presented by “Heisterberg and Verma, 2014”,

Figure 7. Balance Score Card for the Sneakaway “as is” SOR

Conclusions

In sum, in this competitive landscape adopting Big Data Analytics has become crucial, companies need to evaluate accordingly either to adopt a private, public or hybrid cloud to develop their analytical and BI business strength and develop value added services around the data driven insights. In the public cloud, vendors offerings are quite extensive as SaaS, PaaS or IaaS. Cloud computing combining grid, cluster, and virtualization with their elasticity offerings of pay as you consume, have demonstrated to make the business cases to offer all of the technological ground for companies to start their own Big Data Analytics deployments in either private, public, hybrid clouds. Something is for sure, adding business agility is being used by companies in order to maximize their productivity.