What is a data architecture
Data Management Body of Knowledge (DMBOK) describes data architecture as “data strategy specifications that outline the current state, describe data requirements, direct data integration, and manage data assets.”
Every organization’s strategy should be based on effective use of data. Your data architecture contains policies for a solid foundation under the business model. In addition, the Data Architecture describes guidelines for many processes. These processes include data collection, storage, application, processing, and interaction with multiple systems.
The right data architecture is indispensable to prevent chaos by adding tools, dashboards and technical adjustments ad-hoc. A clear architecture ensures that there is no loss of productivity and that operational costs remain manageable and low. Data architecture is an integral part of the IT architecture and the overall enterprise architecture. Together they determine the structure and functioning of the entire organisation.
Designing your data architecture
If we look at our use case and the route the data follows, which we call the dataflow, we see that it starts with the need and associated request of users.
It is important to remember that a dataflow has as its starting point the business user and through the data architecture also ends again with the answer and presentation to the user. The data architecture facilitates all routes which are used to collect and distribute data the required data.
When designing the data architecture, it is good practice for data architects to work from consumer needs to data sources. So data architects have to take many things into account. In this 4-minutes video we present the most relevant criteria.
Frameworks to guide you
When setting up a data strategy and data architecture, it is good to use existing standards and frameworks as much as possible. That is why we list three widely used standard bodies and frameworks, that can support you when designing your own architecture.
DAMA DMBoK, known as the Data Management Body of Knowledge, a framework that includes data architecture, and related topics. the model is specifically designed for high quality data management.
It contains standard definitions of terminology, functions, deliverables, data management roles and also provides guidance on data management principles.
You can read more about the DAMA community via clicking on this link.
Zachmann, more particular John Zachman created a complete Enterprise ontology while working at IBM in the 1980s.
The ‘data’ column of this framework includes multiple layers, such as key architectural standards for the business, a semantic model for business data (conceptual layer), an application data model and the coherence (logical model) and a data model for the actual databases and storage (physical model).
This framework connects the Enterprise, IT and Data architectures.
Click on this link for more detailed information about Zachmann.
TOGAF is the most widely used Enterprise architecture methodology that provides a framework for designing, planning, implementing and managing data architecture best practices.
It helps define business goals and align them with architecture goals.
Click here for the Wikidescription of TOGAF.
Best practice recommendations for data architecture design
From best practice studies we are convinced that following the below mentioned steps will guide you towards an effective data architecture. Before you continue reading we like you to evaluate the existing architecture in your organisation by answering the question in this online form.
Data architecture must support the data strategy
In the words of Donna Burbank, Global Data Strategy’s MD: “Your organization’s business model and strategy inform the direction you take as you create your data strategy. The data strategy then gives you a clear picture of your client. You should be able to tailor your product line to fit the needs of the customer. You get to improve customer service in the long run.”
The business impact of an organisation can only be increased if extensive and relevant information can be obtained from the data architecture in place. The data strategy highlights all areas that can impact business performance. After all, data is a crucial part of the business strategy.
Consider governance when designing your data architecture
Having a proper data architecture without data management and governance is a recipe for failure. Employees who can undisturbed adjust data or its structure pose an uncontrollable risk to business operations. A difference of opinion with the existing organisation is often a reason for such unstructured adjustments. Although these variations seem harmless at first glance, they can cause great damage to the organisation.
With Data Governance you ensure that everyone uses data in the right way. Data governance also ensures that the effects of your data architecture go beyond the technical infrastructure. Data usage practices and processes are being centralised.
Governance ensures that any upfront errors do not have a disruptive effect on data processing. Good data governance also reduces the chance of errors and associated risks in any process.
Harmonize data architecture and data modeling
It is becoming increasingly clear that a data architecture cannot be designed to work independently. The data strategy prescribes what must be included in the architecture, while data governance can guarantee an optimal return on the architecture.
Data modeling is ideally suited for capturing data relationships. It provides a clear picture of how data structures in different databases work together. Data models allow architects to use different data components to support business goals.
To create the optimal user experience, it is important that we use data models that meet the needs and that we integrate them into the data architecture. In the next section, we take a closer look at designing a data model.
Make your data strategy as agile as your business
Data architectures must enable fast access to information with minimal effort, so employees can spend maximum time on work rather than waiting for slow systems.
From research we know that an employee who carries out data processing processes in a business environment spends on average more than 10% of the time looking for and making available the right information.
For data specialists, that is even an alarming percentage of 80% of the workable time that is not spent effectively. In addition to inefficiency, this also causes boredom from repetitive work with the associated risk of overrun for specialists.
By designing a scalable system from the ground up, you ensure that there are no issues with scale change and that performance is optimal and costs are minimal.
Embed security by design
Protecting your data architecture requires knowledge and understanding of each component’s role in securing information. Security must be incorporated from the initial design concepts to the implementation of the data architecture and usage during day-to-day operations by following best practices. This ensures that everyone involved is aware of their responsibilities when it comes to protecting critical assets.
When choosing a data architecture in the cloud and on-premises, it is very important that employees and customers feel that their personal and sensitive information is well protected. This means that security measures must be incorporated into every aspect of the system, including who can access what data when they need it.
A data breach can cost an organization dearly, so good quality software with robust encryption will help protect against outside threats while mitigating internal risks.
Data architectures should also be regularly reviewed for compliance requirements, such as meeting industry standards around privacy governance. By demonstrating good management of customer and employee data, companies can increase trust internally and externally.
Take innovation into account during construction
Not long ago, IT organisations built static data warehouses. These barely reacted to the constant changes in the business environment. New data solutions were devised based on the limitations, which adapt to changes in the market. Organisations have also used data lakes to store raw data. Although the data lakes require large storage capacities, companies can analyse the data for any purpose. However, the lack of efficient data management and data quality strategies has not made this resource particularly popular.
A data architecture must take into account the limitations or possibilities of technical solutions. Elsewhere in this training, we already mentioned innovative technical possibilities that can be incorporated into a data architecture, such as Linked Data and FAIR data.
Benefits of a well-thought-out architecture
We would like to inspire you with the following summary that shows how a solid, modern architecture will optimally support your organization.
- Enhanced integration ensures that distributed information is effectively combined to gain accurate business insights. A good data architecture enables users to easily extract relevant input from different data sources. With a converged architecture, your organization is undoubtedly on the path to more innovation and better creativity.
- Increased efficiency through the use of dynamic platforms. Cloud and Edge Computing have enabled organizations to share data between and within departments and business units. Such technologies should certainly be included in a good data architecture.
- Support for various data types. A system that effectively and reliably handles different types of data also makes it possible to leverage robust, innovative and disruptive technologies. A good data architecture is flexible and makes the organization agile.
- Dynamic evolution of your products. The technology landscape with manufacturers and their products like Redshift, Snowflake, BigQuery to Azure SQL Database, is always evolving. The Data architecture must be flexible enough to allow the organization to move in real-time with technological developments. It is up to organizations themselves to ensure that their systems are robust enough to adapt to such technologies.
- Better storage and compute management. Cloud suppliers now offer robust and secure storage solutions that your organization can use. The trade-off between proprietary investments or cloud computing and storage has become much easier.