guardrails

3. Guardrails Maturity – Data Analytics

What are the different levels of platform maturity for providing innovation, and what does it entail in terms of business results and Data Analytics guardrails maturity?

Maturity Levels of Guardrails and Principles

The following table explains on how the maturity of an organisation to implement the guardrails is linked to the business and architectural outcomes.

Depending on the need and scope of the work, there can be different types of guardrails. The one below helps us understand the Maturity Levels of Principles and Guardrails within IT Architecture:

Level	State	Description	Business Outcome
1	Not written	The guardrails are not written, as there is a lack of understanding of what “good” looks like.	Missing North Star.
2	Written but not referred	The guardrails are written but only in documents and are not put into practice.	This could potentially impact the business outcome and quality of the product.
3	Written, referred, but not assured	The guardrails are documented and referred but work on a 100% trust framework. This model can break when put under stress or if there are changes to the team, e.g. outsourcing, attrition, etc.	The business may invest thinking there is minimal risk, which can be a false image. This is similar to the blind spot and can lead to accidental spending, retrofitting costs and risk to business outcomes.
4	Written, referred, manually assured	The team understands the boundaries, and it is ensured that the boundaries are not crossed. However, this is monitored manually and will require a lot of hand-holding.	The quality of the product is guaranteed, but this state is not scalable. Because of manual interventions, the time to market will increase significantly, cause frustration to the product developers and owners, and delay the launch of new products.
5	Written, referred, automated assurance	The significance of the guardrails is well understood; they are implemented as part of the way of life and partially or fully automated depending on the risk, e.g. traffic lights govern traffic and are automated but are not at all the crossroads.	Enormous potential to not just bottom-line efficiency savings and unlock top-line use cases but also reduce regulatory risk exposure by amalgamating it with regulatory and compliance frameworks.

Maturity Levels of Guardrails within IT Architecture

CMM Levels

The CMM framework defines five levels of process maturity:

▶️ Initial (Level 1)

(chaotic, ad hoc, reactive, individual heroics): At this level, processes are typically chaotic and ad hoc, with no formal methods or processes in place. Work is often reactive, with a focus on fixing problems as they arise rather than preventing them.

🔁 Repeatable (Level 2)

(managed, measured, controlled): At this level, processes are more formalized and controlled, with some degree of measurement and management in place. Work is still reactive, but processes are more consistent and predictable.

📜Defined (Level 3)

(documented, standardized, integrated): At this level, processes are documented, standardized, and integrated into the organization’s overall project management processes. Work is proactive, with a focus on continuous process improvement.

🚓Managed (Level 4)

(quantitatively managed, optimized): At this level, processes are quantitatively managed and optimized for efficiency and effectiveness. Work is proactive, with a focus on continuous process improvement and the use of data to drive decision making.

🚀Optimizing (Level 5)

(innovative, responsive): At this level, processes are innovative and responsive, with a focus on continuous improvement and the ability to adapt to changing business needs.

It is important to note that these levels of process maturity are not meant to be rigid or prescriptive but are intended to provide a general guide for assessing an organization’s current level of process maturity and identifying areas for improvement.

CMM Levels and Architecture Goals

Generally, most organisations are at the level of maturity of Level 1 or Level 2, i.e. the guardrails are either not written or are written but are not referred to or assured. It is essential to obtain support from the leadership team and stakeholders to establish the data infrastructure and provide the groundwork for security before commencing or executing data analytics use cases/projects/programs. The foundation must be laid properly to avoid major delays or unforeseen costs in the future.

Mapping of Guardrails Maturity to Architectural Outcomes

Data architecture within analytics is evolving at a great pace, e.g. Hadoop, which was thought to be the answer to big data just 5 to 7 years ago, is being seriously questioned and challenged. This, along with concepts of evolutionary architecture, domain-driven design, advanced analytics use cases covering AI/ML, low-code/no-code requirements, micro-services architecture or new architectural constructs, e.g. Data Mesh, cannot be embraced if the right foundation is not put in place.

The 27 points covered in the figure above give a flavour but do not cover all the details and should not be looked at in isolation. It is expected that other CMM levels, e.g. data integration, information architecture, data governance, etc., should also work alongside the data analytics maturity.

Examples of Data Analytics Guardrails

Coming Next.

The author is heading one of the largest data analytics transformations and modernizations while working as British Telecom’s Principal Enterprise Architect for Data and Analytics. The above write-up is author’s personal opinion. Contact the author at shuklabhavin@yahoo.com, on LinkedIn, or through Twitter if you have any questions.

2. Guardrails for Data Analytics

My previous article discussed an introduction to guardrails and their significance. This is the second blog I plan to write about this topic in a series. This article mainly focuses on the challenges of writing the guardrails in data analytics that uses Data Mesh as the architectural construct and how they can be approached.

📈 Data Analytics Guardrails

Data analytics guardrails are rules, procedures, or controls put in place to help ensure that data analytics processes and systems are being used safely and ethically. These guardrails are designed to prevent data misuse or the unintentional disclosure of sensitive information, and to help ensure that data analytics results are accurate and reliable. Data analytics guardrails can include data governance policies, data privacy regulations, and quality assurance processes. They can also include technical controls, such as data encryption or access controls, to help protect against unauthorized access to data or data breaches. Overall, data analytics guardrails are an important part of any organization’s data analytics strategy, as they help to ensure the data’s safety, security, and integrity and the results of the data analytics process.

🚧 Challenges in Writing Guardrails

The challenge is to “write” guardrails that are easy to interpret and socialise, are feasible and implementable, are machine-readable and can be assured by machines/systems.

But the first hurdle is to agree on what “good” looks like with all the stakeholders, including the team that is expected to live within the boundaries.

⚒️ Breaking the Problem into Manageable Chunks

The first question that comes to mind is where to begin writing the guardrails. The most effective method is to start from the top and then componentise the entire architecture canvas across the Enterprise Architecture into domains, further dividing domains into sub-domain areas. One sub-domain listed below within the Data Domain can be Data Analytics.

A sub-domain, e.g. Data Analytics, can be further modularised by creating independent solution building blocks. We can then understand the scope of these blocks, e.g. operational vs analytical plane, and define finer guardrails for each solution building block within these planes, inside that finite scope and context.

The figure above on Data Analytics Reference Architecture shows how Operational Plane and Analytical Planes are coming close together in a Domain-Driven Design and Data Mesh Architecture. Data Product Management, Data Security and Privacy, Data Quality and others overlap significantly between the operational and analytical planes. On the other hand, Data Integration has overlaps but is still developing as a self-serve, evolutionary architecture and may be maintained by IT rather than the business.

⚛️ Identifying Fundamental Building Block – Data Products

The Data Analytics Reference Architecture and the Solution Building Blocks shown above then use the architectural construct of Data Mesh. Data Mesh helps create the data infrastructure, within the data analytics sub-domain to connect the Operational Plane and the Analytical Plane.

The reference architecture above provides the blueprint to create manageable and reusable technical infrastructure per higher principles from IT guardrails separation of concern. Whereas the Data Mesh construct then provides the governance layer and data architecture to help create Data Products using principles of Domain-Driven Design, making it manageable within the bounded context but interoperable across domains, sub-domains and bounded context.

Data Products accessed via the Data Analytics tools/demand via Data Marketplace will be the actual tangible output to the personas consuming information. As a result, Data Product can be used as a fundamental building block within the Data analytics subdomain.

The advanced analytical plane in the illustration above does not detail the AI / ML aspect, which will be discussed in future blogs.

🌏 Fundamental Building Block within the Data Ecosystem

The responsibility of the architecture is to ensure the quality of the output. If the quality of the fundamental building blocks, i.e. Data Products, and the solution building blocks, i.e. technical components, is well maintained, a high-quality output delivering use cases will be assured.

Each Data Product should ensure that it fits well into the data ecosystem and complies with different dimensions of data governance, data architecture and data management frameworks.

Area	Description	Data Product Context
Data Architecture	Defines the blueprint for organising and managing data assets	Lifecycle Management of Data Products
Data Modelling & Design	The process of discovering, analysing, representing, and communicating data requirements and designs	It deals with how Data Products are internally structured and connected to the use cases. This area brings the discipline ensuring that the data products are reused, data is shared, and redundant copies of data are not created outside of the domain where the data belongs.
Data Storage & Operations	The design, implementation, and support of stored data. Operations provide support throughout the data lifecycle from planning for to disposal of data	This is the Information Lifecycle management of the data stored within the Data Products.
Data Security and Privacy	The protection of data from unauthorized access or use.	The measures and practices are implemented to prevent unauthorized access to or disclosure of Data Products. It also deals with the data within the Data Products around the rights of individuals to control how their personal information is collected, used, and shared.
Data Integration & Interoperability	The movement, consolidation, and translation of data between data stores, applications, and organisations. Integration: messaging, formats etc. Interoperability: semantics, understanding, and interworking	Helps in the consolidation, convergence and connection of the Data Products.
Document & Content Management	Storage, management, and access to digital content, such as documents, images, videos, and audio files.	Unstructured data managed within the Data Products. The architecture managing Data Lake vs. Data Warehouse use cases for documents, images, videos and audit files.
Reference & Master Data	Ongoing reconciliation, maintenance, and sharing of core enterprise data to enable consistent access to a single version of truth	Storing hierarchies to map levels, codes to harmonise data from various systems, lookups to map descriptions, enumerations and any domain/sub-domain specific patterns within or across Data Products.
BI & Advanced Analytics	Planning, implementation, and control processes and technology to manage decision support data, analytics, and reporting	The data structure design within Data Products, e.g. dimensional approach, denormalised, etc. And how the data/information is consumed within the reports, dashboards and advanced analytics.
Metadata Management	High quality integrated metadata on: definitions, data models, data flows, and other information critical to understanding data throughout its lifecycle	This discipline aims to link Data Products to the Enterprise Metadata Management Service. It should be noted that Domain Boundaries are subject to change as a result of modifications to business processes or upcoming Domain Distillations. The domain, sub-domain, bounded context and data product boundaries should not be materialised/physicalised but mapped logically within the metadata repository.
Data Quality Management	The planning and implementation of quality management techniques and systems to measure, assess, and improve the fitness of data	Checking Data Integrity within the Data Products in the Data Analytics sub-domain. Data Integrity is a subset of overall Data Quality Management. The Data Integrity check within Data Analytics should be integrated into the organisation’s Enterprise Data Quality Service.

When the Data Product complies with the guidelines established by each of the dimensions above, it can be shared and reused and is expected to provide the use cases’ specified benefits.

📏Maturity Levels of Guardrails and Principles

The third write-up in the series will focus on the maturity levels for data analytics guardrails.

Coming Next.

1. Guardrails, An Introduction

Guardrails (Plural Noun): A rail at the edge of something, such as a cliff or the deck of a boat, that prevents people from falling off.

What, in your opinion, should be the first priority before embarking on any new venture or innovation? First and foremost, safety?

At first glance in the photograph, the white car appears to be leaving the arrival gate. The fact is that this is a departure gate. If you look closer, there is a small “No Entry” board in blue, suggesting that this is a departure gate and not an arrival gate. The white double line on the road also indicates that this is the pause for the outgoing vehicle, i.e. the departure area before a vehicle takes the major road.

How did we reach this confusing stage, and how is this linked to the topic of guardrails?

This is called “hard coding” within the IT Engineering area. The “Arrival” and “In” signs were hardcoded into a decorative gate when the train station was built with a thought process to make it look artistic, grand and elegant. However, the most important point missed was that there can be a state change at some point in the future. An entry gate can also become an exit gate in future due to an increase in traffic or a change to other external parameters, e.g. road design, taxi ranks, car size, etc. Avoiding hardcoding and making it configurable would be a guardrail similar to something in IT that could have avoided this situation.

🗽 Boundaries Liberate

Guardrails / Boundaries sound claustrophobic and negative in nature. They are being seen as restrictive, an unnecessary rule that takes away freedom.

Another way to look at it is that boundaries provide safety, it helps us liberate rather than constrain us. For example, children from a school near a busy road could only play in areas where adults could keep an eye on them. This was true until a fence was built, after which the children could play in any area of the playground of their choosing.

Boundaries limit options but also provide freedom of choice from the available options. As Greg Mckeown puts nicely in his book Essentialism – The Disciplined Pursuit of Less, it gives freedom to select the vital few from the trivial many and is central to Essentialism.

It is not about policing and restricting freedom but taking out unnecessary adventures that can risk the overall goal/vision, may lead one to danger, or waste crucial time exploring things that do not really matter. Too many options may not lead us anywhere; guardrails allow us to experiment within the chosen option.

Exploring Examples of Different Types of Guardrails:

📿Guardrails: Religious Beliefs

This is Kutubiya mosque, Marrakech, Morrocco, and no building can be taller than this Meenaret in the old town. This is by design, as it reminds people that principles and values are higher than anything else.

The principles of Islam are Surrender, Submission, Sincerity, Obedience and Peace.

🏳️ Guardrails: Political Beliefs

India’s Republic Day falls on 26th January. This was when the Constitution of India came into effect on 26th January 1950, and India became a Republic.

The word republic comes from the Latin term res publica, which literally means “public affair” or “public matter”, i.e. the power resides with the people.

The constitution is a book of rules, principles and guidance on how the country should operate. It is above any law, politics or religion of the country, which brings “real” freedom to the people by setting up boundaries.

The constitution sets up three pillars,

🏛️ The Legislative Power: Task of passing laws and supervising their implementation

👮‍♂️ The Executive Power: This is the federal government tasked to implement laws created by the legislation.

⚖️The Judicial Power: Interprets law and administers justice; their task is to ensure the laws are complied with.

The constitution is above all these pillars because the power resides with the people, and the three pillars work for them.

It is important to note that the Constitution defines the duty, roles and responsibilities, and power between the Union and the States, as India is a Union of States, i.e. establishing the fine balance between Centralisation and Decentralisation. Without this balance, it is impossible to scale up the governance of such a big country, having a rich diversity of cultures, 1.3B people, different religions and roots in ancient civilisations.

🏨 Guardrails: Organisational Beliefs

British Telecom has a 178-year history, employs over 100,000 people, and operates in 187 countries.

A code of conduct for all BT employees and principles is established within the Digital area. It is impossible to scale and establish the right culture unless these are defined and agreed upon by the citizens of the organisation.

👴 Guardrails: Personal Beliefs

The three guiding principles in my life of which are constructive.

❤️ Speak Truth (First Person)

Speaking the truth means being true to myself. This ensures that I am always consistent with my responses and do not have to think about them.

🤗 Love (Second Person, Singular)

Loving everyone individually and treating them the way I want to be treated.

♾️ Compassion (Third Person, Plural)

Compassion is for the third person, and it is plural. It is about being kind to everyone as a group.

The other three principles are contrarian and instruct me on what to avoid doing. This is specific to me and my way of life; there is no right or wrong.

🍾 Do Not Intoxicate

Trying not to drink alcohol.

🍗 Avoid Meat

Do not eat anything that is not plant-based, except milk and honey. It’s more like a way of life I’ve grown up with.

🎲 Do Not Gamble

Don’t gamble, and don’t do anything that has to do with gambling. So far in my career, I haven’t worked for any company that deals with gambling.

What are your personal and family guardrails? I would love to hear from you!

Guardrails: IT Architecture, Data and Analytics

As an Enterprise Architect, this is a great topic that is close to my heart. I promise to write in detail about the guidelines for data and analytics in IT architecture. This will include figuring out where to begin, how to define, and giving an assessment of the maturity model for data architecture guardrails.

Coming Soon!

Bhavin Shukla