Digital Transformation Reference Architecture
Introduction
I am not an apps expert on the initial day of my technologies journey. Electronics and science were fascinating for me and a kid’s curiosity mind made me to learn, observe and dream.
The industry I work in revolves around apps, I have learned it and started to put it into practice to meet business demand. This reference architecture in my mind for some time, want to share it with my fellow Architects.
Business’s Business
Organizations undertake digital transformation initiatives to utilize technology as much as possible to support business operations. Therefore, many applications have to be developed for various business operations carried out by an organization such as inventory management, procurement operations, supplier management, payroll handling, advertising/sales campaigns, building space management, and vehicle fleet management.
Standard Enterprise
Even before a digital transformation project, enterprises use many IT systems. These can be commercial systems, which may be deployed on-premise data centers, on cloud VMs (e.g. AWS, Azure, etc), or consumed as SaaS systems (e.g. Salesforce, M365, etc). Further, an organization may have some custom-developed systems for certain specific business operations (e.g. developed using Spring Boot, .NET, Go, etc). In addition, an organization may have multiple databases, containing information about different entities and applications.
Although most of these systems can connect with a central user store (e.g. NDS/LDAP/AD) to get user information, each application has to authenticate users before offering its services. Each of these systems will be providing some services (e.g. as APIs) and those service interfaces have to be protected from unauthorized and uncontrolled access. Furthermore, as most of these systems do not exist in isolation and need to interact with other systems, many integrations need to be performed within the deployment. Finally, when a deployment gets larger and more complex, monitoring and troubleshooting tasks become more complicated.
What is in digital transformation
As the outcome of digital transformation initiatives, it is necessary to integrate these different existing
systems, data stores, business processes, and physical entities (where possible) and offer a seamless user experience tailored for the enterprise’s business operations. If we take an example of a retail chain, below are few possible use cases expected from such an initiative:
When inventory levels run low, purchasing orders have to be automatically sent to registered suppliers (steps such as selecting suppliers and approving large purchase orders may need human intervention).
Once delivery is ready, logistics providers should be assigned automatically (again some human approval steps might be needed) to transport goods from suppliers to warehouses or to deliver online orders made by customers.
When a new employee is recruited, his/her details should be propaga
ted to POS systems, payroll systems, procurement systems, etc based on assigned roles.
Web portals and mobile apps should be available for customers,
store staff, management staff, and suppliers to perform relevant operations.
When a customer is signed into one business application, he should be able to use the services of all business applications without signing in again.
All business service usages by customers, staff and partners should be monitored and controlled.
Most importantly, it should be possible to introduce new services to the enterprise’s IT platform and integrate those with existing services with a minor effort.
Generic Architecture – Digital
Now let’s consider a generic architecture for building an IT platform for such digital transformation initiatives (Figure 1).
Let’s consider the main components of this platform:
Integration layer
This layer performs all integration activities among on-premise backend systems, file-based systems, data stores, and SaaS systems. It can perform protocol bridging, message transformations, message enrichment, content validation, service orchestration, etc to implement integration flows among backend systems.
This layer is deployed within the internal network and secure connections (e.g. VPN, protected
endpoints, etc) should be available for it to integrate systems that are not deployed within the data center. Usually, components of the integration layer can be scaled as necessary to match the load (e.g. auto-scale by utilizing the capabilities of hyper scalars provided by infrastructures such as Kubernetes pod auto-scaling and AWS auto-scaling groups). Furthermore, it is also possible to deploy more than one integration cluster depending on the requirements for segregating business services and load.
API gateway clusters
API Gateways intercept incoming traffic to the deployment to enforce security and other policies as well as to capture API usage statistics. It is possible to deploy one or more API gateway clusters as necessary to isolate API traffic based on different client parties (e.g. general users and partners) and enforce relevant network security policies (e.g. limit partner API gateway to IP ranges of partners).
API Gateways are usually deployed within the internal network and incoming traffic is routed to API gateways via a load balancer placed within the DMZ. However, it is also possible to deploy external-facing API gateways in DMZ depending on the organization’s policies. Similar to the integration layer, API gateway clusters can also be auto-scaled by utilizing auto-scaling capabilities provided by the infrastructure.
API management layer
API management plane facilitates API publishing, policy definition, application registration, API subscription, and API life cycle management activities. This layer interacts with API gateway clusters to publish API definitions and to exchange API authorization information. As the API management layer usually provides web portals for API creators, API users, and administrators, it may be necessary to facilitate access via a load balancer placed within DMZ (e.g. if external users need to subscribe for APIs).
Identity and Access Management (IAM)
IAM layer provides user management, authentication, authorization (policy evaluation), and auditing functions to the whole deployment. Below are some of the specific features an IAM layer can provide related to the above areas:
Support for OpenID Connect and SAML2 for authentication and exchanging user information
Support for authorizations based on OAuth2 / XACML.
Single Sign-On (SSO) to enable multiple services to be accessed without having to authenticate with each service (typically implemented using OpenID Connect or SAML2)
Multi-Factor Authentication (MFA) to enhance the authentication process (e.g. password and SMS OTP-based authentication).
PAM and Conditional or context-based authentication (e.g. users in the store-admin role are allowed to authenticate only during office hours and if connecting using a certain IP address range).
Support for connecting with multiple user stores such as LDAP/Active Directory and RDBMS.
Connecting / federation with external identity providers to authenticate users who are not registered in the organization’s IAM systems (e.g. sign-in using Google, Facebook, etc, or external IAM systems).
Web portals, mobile apps, or other interfaces for users to perform self-sign-up, profile management, password recovery, etc.
Usually, the IAM layer is also deployed within the internal network and clustered as necessary to meet scalability and high availability requirements. As the IAM layer provides portals/interfaces for users, it may be required to provide external access to it via a load balancer placed within the DMZ.
Business Process Management (BPM)
Some business operations require many interactions with users at multiple steps. Examples would be selecting suppliers, approving large orders, handling customer complaints, etc. These operations may have one or more approval steps or steps that require user input. Such business operations (or parts of those operations) may have to be implemented as workflows modeled in standard workflow languages such as BPMN or BPEL.
BPM systems support deployment, execution, administration, and analysis of such human-oriented workflows. Once a BPM system is deployed, it can be integrated with other business systems and user portals via the integration layer. Furthermore, similar to any other business system, BPM systems can also utilize SSO and other IAM features provided by the IAM layer (e.g. for assigning users to workflow tasks).
Observability layer
All systems in the deployment, including the API layer, integration layer, IAM layer, BPM systems as well as backend systems, should provide mechanisms to monitor their behaviors. This includes collecting and analyzing logs, collecting and summarizing operational metrics, and tracing message flows across multiple systems. The observability layer performs these tasks by collecting logs, metrics, and traces from all systems and providing a unified view of the deployment. Therefore, if something goes wrong in the deployment, administrators don’t have to inspect each and every system individually, as they can obtain all information by just logging into the observability layer.
Business Distributed in Multiple Locations
An enterprise may have user bases distributed across multiple geographical locations (e.g. in multiple states or countries). In such cases, it may be necessary to deploy business services in multiple data centers or cloud regions closer to client locations to satisfy performance and regularity requirements. The architecture discussed here can be extended in such scenarios to support multiple data centers/cloud regions as shown in figure 2. In this case, usually only the API gateway clusters and integration layer clusters have to be deployed in additional data centers. API management plane, IAM layer, BPM systems, and the observability layer deployed in the main data center can be used by all additional data centers.
As API gateways and integration components handle the main runtime load of the deployment, having a single deployment of other components does not affect the performance in most cases. Furthermore, having a single management and monitoring layer naturally supports such tasks to be carried out centrally.
Architecture Enterprise Apps Development
So far, we have discussed an architecture that can be used for digital transformation projects. Now we can look at how this architecture can be used to build enterprise applications.
When developing an enterprise application, we can first think about the APIs that need to be exposed. This can be modeled in the Open API specification and deployed in the API layer. It is possible to deploy APIs as prototypes so that client application developers can start testing them and provide feedback.
Then we can focus on the back-end implementation. Usually, many new applications can be composed using existing business services. In such cases, we can implement most parts of the new application in the integration layer. If the new application requires complex logic, in addition to integrations, we may have to implement some parts in a programming platform (e.g. Spring Boot, Go. NET., etc) and use an integration layer to integrate it with the rest of the systems. Another option is to use an integration-focused programming language such as Ballerina, so that complex logic and integrations can be implemented in the same platform.
All client applications and backend systems can use the IAM layer to perform authentications (e.g. SSO for client apps) and authorizations, without implementing those features within each application. Similarly, all applications can publish logs, metrics, and traces to the observability layer to enable centralized monitoring of the deployment. Finally, it is necessary to maintain a CI/CD pipeline to build, test and deploy all artifacts related to new applications, so that API definitions, application logic, and integration artifacts can be maintained in a source control system and deployed via a controlled deployment flow.
References :
