To maximize the ROI of Advanced Analytics (AA) the enterprise must design and deploy an organizational structure that supports its mission.
As companies recognize the predictive power of advanced analytics, many are hoping to use AA to drive business decisions and strategies. Many of my clients have made AA a key component of their digital transformations.
While most companies understand the importance of analytics and have adopted best practices, fewer than 20 percent, according to a recent Forrester survey, maximize the potential and achieved AA at scale.
Because of this, most companies are frustrated as they see their efforts not meeting their goals and objective and watch as more mature, analytically driven companies leverage their enterprise data. The democratization of data has suddenly blurred sector boundaries and because of this, businesses will find themselves disrupted not by the competitor that they have been monitoring, but by an unknown upstart from another industry. Industry leadership is no longer enough; companies must aim to be at par (or better) across industries to compete effectively and ensure survival. Functional expertise, beyond specific business sector expertise, will become more and more vital. I believe that AA performance is achievable only by developing functional expertise, strategic partnerships, and defining clear objectives for organizing human capital.
Companies attempting an AA transformation can incorporate these elements into one of several organizational models assuming that clear governance is in place and the company fosters an analytics driven culture across all business units.
The three organizational models are:
- Centralized — the company creates a single AA organization that stands alone in a Center of Excellence (COE) that supports all business units.
- Decentralized — analytics expertise is embedded within individual business units.
- Hybrid — combine centralized AA with embedded analytics expertise in some business units.
All organizations change over time, particularly as the AA transformation evolves. Many companies move back and forth between centralized and decentralized models ultimately settling on a hybrid strategy.
Data governance, however, should be centralized, even if data ownership/stewardship is not. Most large organizations have data centralized within business units like marketing, sales, operations, and finance.
Another important human capital decision is whether AA-COE talent should be partially outsourced and if so, what guidelines should be established. Outsourcing may be limited to low-level data analytics activities, but successful companies establish tactical and strategic partnerships to help with both tactical and strategic roles. These strategic relationships, managed by a single COE unit, provide the COE with guidance, mentorship and skill transfer. Many of the companies I work with protect certain analytical subject areas that produce a competitive advantage — such as pricing analytics — and staff solely within the organization.
Most organizations centralize partnership management in order to remove the likelihood of creating redundant or competing partnerships thereby risking efficiency and security.
So where does the AA unit live in the larger organizational hierarchy? AA is most valuable when it focuses cross-functionally and is accessible from all business units. To be most effective the COA should be visible from, and have access to, the C-suite. A COA with a bi-directional enterprise view will have the most transformational potential.
Over time I’ve seen most COEs collocated as sub-units of Business Intelligence. Many of our customers locate their AA units in IT, but this arrangement can be challenging. IT staff accustomed to managing longer-term projects that are often disconnected from the business, may not be prepared to manage short-term, agile AA projects.
Centralized Organizational Structure
Figure 1 illustrates the ideal organization structure for constructing a centralized Advanced Analytics Center of Excellence. Several job roles that traditionally were thought to be decentralized and distributed across other functional groups are added to this hierarchy in order to achieve maximum efficiency.
Let’s take a deeper dive into the key roles and responsibilities within an Advance Analytics Development trunk of the Centralized COE depicted within Figure 1.
Centralized COE Roles
Role: AA Program and Project Management
For the sake of brevity, I won’t define program/project management but it’s important to point out that those roles need to be cultivated within the COA because, quite frankly, AA projects need to be approached a bit differently than your typical IT project.
Because of the exploratory nature of AA projects, they’re often more similar to R&D efforts than traditional development efforts. R&D is challenging to plan, track and manage.
Moreover, AA projects can reap the benefits of an agile methodology. Agile Advanced Analytics (AAA) business processes should be adopted in order to reduce the time it takes for AA to return value to the organization while quickly adapting to change.
Role: Data Scientist
The job of data scientist is threefold:
- Enhancing business intelligence by taking data that the enterprise collects and getting it in front of the right stakeholders in the form of dashboards, reports, other visualizations
- Decision science, which takes enterprise data and uses it to help a company make a decision
- Machine learning, which builds predictive data science models and places them into production
There is a bit of overlap here with the BI Analyst role however, digging a bit deeper, we can classify data science activities into two categories:
- Type 1 — statistical analytics, or using a background in statistics or actuarial science to construct decision support assets
- Type 2 — machine learning, using AI frameworks to build autonomous/semi-autonomous decision support assets.
The Data Scientist role also overlaps somewhat with that of the Data Engineer’s. Data Scientists frequently transform data within statistical analysis or machine learning experiments in order to perform feature engineering tasks like clustering or normalization.
There are many tools and technologies that facilitate data science activities some of which are:
- Computer languages: R, Python, Java/Scala, Octave, and MATLAB
- DBMS: SQL/NOSQL, Hadoop, Spark, and many OEM versions from companies like Databricks and Snowflake
- Machine learning pipelines: tensor flow, AZURE ML Studio, Keras, etc.,
- Platforms: SAS, Watson Analytics
Role: Solution Architect
IT Architecture is a well-established discipline that links business strategy, objectives and constraints into a viable, robust and cost-effective implementation plan. It is essential for ensuring solutions meet current requirements and can evolve to support future requirements without costly rework and disruptions.
An AA Solution Architect leverages this discipline to deal with the complex aspects of information management and advanced analytics while focusing on activities from user experience and performance to security and governance to platform and infrastructure.
The solutions architect has a deep understanding of the enterprise information model, the enterprise application portfolio, the enterprise technology infrastructure environment and has a deep understanding of the current advanced analytics technology landscape.
Key responsibilities of the analytics architect include operationalizing analytics, mapping business requirements to implementation approaches, selecting technology, and evaluating non-functional attributes such as usability, security, governance, and stability.
Role: AA Business Analyst
A Business Analyst combine analytics knowledge with strong domain expertise. In many organizations, within the context of advanced analytics, the business analyst might also be known as an operations research analyst or a business data analyst.
Within the AA COE Business Analyst duties typically include:
- Evaluating business processes and the data they produce to meet the reporting and business intelligence needs of the users they support.
- Develop business intelligence and advanced analytics use cases.
- Communicating discoveries/insights with business teams, key stakeholders and the AA development team.
- Preparing strategic recommendations for tracking metrics, KPIs and the deployment of business intelligence and advanced analytic assets.
- Define how the business will make decisions based on the BI and advanced analytics assets that will be delivered.
The AA Business Analyst must have a working knowledge of the technology involved in analytics platforms used within the enterprise, though the need for hard technical skills is generally lower than for the BI Developer.
Role: BI Developer
Business Intelligence (BI) Developers transform data into insights that drive business value. Through use of data analytics, data visualization and data modeling techniques and technologies, BI analysts identify trends that help other departments, managers and executives make business decisions to modernize and improve processes in the organization.
BI developers typically are chartered to uncover improvements that can be made to save the enterprise money or increase profits. Working with business stakeholders they help define KPIs (Key Performance Indicators), Scorecards and Dashboards. This is done by mining complex data using BI software and tools, comparing data to competitors and industry trends and by creating visualizations that communicate findings to others in the organization. BI Analysts are proficient in computer programming languages (Python, R, Java, Scala), BI tools (Power BI, SAS, Tableau, Excel), and underlying technologies (RDBMS, NOSQL, Data Lakes).
Job responsibilities vary by organization, but the following day-to-day tasks are performed by BI Developers:
- Review and validate enterprise data as it’s collected and stored.
- Model data
- Oversee the deployment of data to the data warehouse
- Review enterprise data to ensure integrity/quality of data warehoused
- Develop policies and procedures for the collection, analysis, and distribution of data
- Design, create and monitor analytics assets
- Monitor analytics consumption within the enterprise
- Implement new data analysis methodologies, techniques, and tools
Role: Data Engineer
Data engineers prepare and transform data using pipelines. This involves extracting data from various data source systems, transforming it into a staging area or staged state, and loading it into a data warehouse system. This process is known as ETL (Extract, Transform, Load). Data engineers are also experienced in ELT (Extract Load Transform) technologies that delegate the transformation of the data to the underlying data management technology.
Data engineers are typically responsible for finding and analyzing patterns in datasets. This requires transforming large amounts of data into formats that can be processed and analyzed. The role of a data engineer requires significant technical skills, including multiple programming languages, ETL/ELT tools and knowledge of SQL, NOSQL, Big Data and Data Lake technologies.
Data engineers are expected to:
- Create and maintain optimal data pipeline architecture
- Assemble large, complex data sets that meet business/analytic requirements
- Optimize data delivery and re-design infrastructure for greater scalability
- Be conversant in data manipulation technologies like SQL, Big Query, Hadoop/Spark or other modern, cloud-scale data management technologies (Databricks, Snowflake)
- Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using EDW/Data Lake technologies
- Build the analytics infrastructure that creates the data pipelines that provide raw data in formats that can be used by BI Analysts and Data Scientists
- Work with internal and external stakeholders to assist with data-related technical issues (data quality) and support data infrastructure needs
- Understand and implement DevOps processes that supports the high availability of the EDW and Data Lake.
Role: Data Acquisition Manager
In order to understand the role of the Data Acquisition Manager we should first define what we mean by data acquisition.
Data acquisition is the processes for bringing data that has been created by a source outside the organization, into the organization, for production use.
Prior to the Big Data revolution, companies were predominantly inward-looking in terms of data consumption. Traditional, data-centric environments like data warehouses dealt only with data created within the enterprise.
With the advent of data science and predictive analytics, many organizations have come to the realization that enterprise data must be fused with external data to enable and scale a digital business transformation. This means that processes for identifying, sourcing, understanding, assessing and ingesting such data must be developed.
This highlights two points of terminological confusion. First, “data acquisition” is sometimes used to refer to data that the organization produces, rather than (or as well as) data that comes from outside the organization. This is a fallacy, because the data the organization produces is already acquired and likely warehoused in one or more data stores.
Second, the term “ingestion” is often used in place of “data acquisition.” Ingestion is merely the process of copying data from outside an environment to inside an environment and is very much narrower in scope than data acquisition. It seems to be a term that is more commonplace, because there are mature ingestion tools in the marketplace. (These are extremely useful, but ingestion is not data acquisition.) In fact, ingestion is the primary objective of the data engineer.
The following set of tasks constitute a data acquisition process and are overseen by the Data Acquisition Manager:
- A need for externally sourced data is identified, perhaps with use cases
- Prospecting for the required data is carried out
- Data sources are disqualified, leaving a set of qualified sources
- Vendors providing the sources are contacted and legal agreements entered into for evaluation and sample data sets are acquired for evaluation
- Semantic analysis of the data sets is undertaken, so they are adequately understood
- The data sets are evaluated against originally established use cases
- Legal, privacy and compliance issues are understood, particularly with respect to permitted use of data
- Vendor negotiations occur to purchase the data
- Implementation specifications are drawn up, usually involving Data Operations who will be responsible for production processes
- Source onboarding occurs, such that ingestion is technically accomplished
- Production ingest is undertaken
Several things that stand out about this list. The first is that it consists of a relatively large number of multidisciplinary tasks. The second is that many different groups will be involved with the process.
Business Analysts, BI Analysts and Data Scientists will likely identify the need and document the use cases, whereas Data Governance, and perhaps the Office of General Counsel, will render opinions on legal, privacy and compliance requirements.
Conclusion
Creating a Center of Excellence (COE) will ensure that your organization reaps all the advantages of Advanced Analytics (AA) technologies. In this article I’ve identified a number of best practices that will help you kickstart efforts to build an AA driven culture, an organizational structure supporting that culture and the role human capital plays within the COE.
