Unity Catalog and open source: compliance and security for small and medium-sized businesses

‘Data for everyone!’ is the dream of data scientists – and often a warning sign for every German CISO. Especially in regulated SMEs, security is often seen as a brake on innovation, which is exacerbated by new requirements from the EU AI Act and NIS-2.

But what if governance didn't block progress, but accelerated it? In this deep dive, we'll show you how Unity Catalog and open standards can help you overcome technical compliance and organisational hurdles without losing your agility.

Governance workshops often reveal a classic conflict of objectives: data scientists need broad and fast access to data in order to train models efficiently. Security and data protection officers, on the other hand, are forced to restrict access in order to minimise personal liability risks (GDPR, NIS-2). This tension can paralyse digitalisation initiatives in small and medium-sized enterprises. However, the solution lies not in more extensive regulations, but in technology that combines control and access.

What exactly is the Unity Catalog?

Before we can resolve compliance issues, we need to understand the tool. The Unity Catalog (UC) is much more than just a simple inventory of tables. It acts as a centralised, unified governance layer for data and AI on the Databricks platform.

Think of UC as a kind of intelligent control layer that sits above all your workspaces and clouds. It centralises metadata, access rights, audit logs and data lineage in a single location. The principle is ‘define once, secure everywhere’: a security policy is defined centrally once and then automatically applies to all users, regardless of whether they access it via SQL, Python or a dashboard.

With this understanding of the technology, we can now tackle the four key challenges of the German market.

When we at adesso lead governance workshops in German companies, we often feel like mediators between two worlds.

True data sovereignty: Why open source is your best insurance

A key point of discussion in cloud transformations is the preservation of data sovereignty. While closed SaaS solutions (‘walled gardens’) often present high migration barriers due to proprietary storage formats, Databricks pursues a different architectural approach that is also relevant in terms of the EU Data Act.

The platform is consistently based on open standards. The data is not persisted in a closed storage system of the provider, but remains physically in the company's own cloud account (e.g. AWS S3 or Azure Data Lake). Delta Lake, which is based on the open industry standard Apache Parquet, serves as the storage format.

This has significant strategic implications for the exit strategy: In the event of termination of use, no physical data migration is required. Since the data is stored in open formats in the customer's own storage, technical control remains with the company. Access would also be possible directly through other open source engines (such as Apache Spark or Trino). This ensures digital sovereignty and avoids effective vendor lock-in at the data level.

NIS-2 and C5: The technical foundation for audits

While regulatory requirements such as BAIT/VAIT are already established in the financial sector, the NIS-2 Directive now extends cyber resilience requirements to critical sectors such as mechanical engineering, logistics and the energy industry. This is accompanied by stricter due diligence obligations for management.

The Unity Catalog supports the compliance strategy with certified technical building blocks. Since the Databricks platform is certified according to the BSI's C5 criteria catalogue, a tested infrastructure basis is guaranteed. A core aspect of NIS-2 is complete traceability: in case of suspicion, companies must be able to precisely identify which identities have accessed critical data.

Instead of having to manually consolidate audit information from isolated data silos, Unity Catalog provides a central governance layer for logging. This enables efficient technical integration into higher-level ISMS and incident management processes. While the technology does not replace organisational responsibility, it significantly reduces the operational effort required to fulfil documentation obligations.

Scalable GDPR compliance: attribute-based rights and the ‘right to be forgotten’

In complex organisational structures, classic role-based access control (RBAC) often reaches its administrative limits. The need to define permissions individually for each permutation of project, location and data classification leads to role inflation that is difficult to maintain.

Unity Catalog enables the transition to attribute-based access control (ABAC). Instead of static assignment lists, dynamic sets of rules are defined. In practice, data columns (such as tax IDs or health data) are identified as PII (personally identifiable information) by automated classification mechanisms and tagged accordingly in the catalogue.

Based on this, a central policy applies, for example: ‘Data with the tag “PII” is masked by default, except for the user group “HR Admin”.’ This establishes a ‘compliance by default’ approach: as soon as a new table with sensitive data is created, the protection mechanisms take effect immediately without the need for manual intervention.

Another critical aspect is the technical implementation of the right to be forgotten (Art. 17 GDPR). In classic data lakes (e.g. based purely on Parquet files), the selective deletion of individual data records is often technically complex and inefficient. This is where the underlying Delta Lake format offers a decisive advantage: thanks to its support for ACID transactions, targeted DELETE or UPDATE operations can be performed at row level. Deletion requests can thus be executed in a transaction-secure manner and verified in the transaction log in an audit-proof manner without having to rewrite entire data partitions.

Fit for the EU AI Act: Transparency in the ‘transparent factory’

The EU AI Act defines high transparency requirements for so-called ‘high-risk AI systems’ used, for example, in human resources management or credit scoring. A key compliance requirement is the complete documentation of data origin. It must be possible to verify which data sets have been used in training and evaluation processes and whether appropriate quality assurance measures, such as bias prevention, have been taken.

In combination with the integrated Model Registry, Unity Catalog provides the technological basis for fulfilling these documentation requirements. Automated data lineage creates end-to-end transparency. Data flows can be traced across tables, views and pipelines to the specific AI model.

In conjunction with established MLOps processes, this enables precise governance:

• Traceability: For each model version, it is technically possible to reconstruct which specific data sources it is based on and which transformation steps were performed.
• Versioning & approval: Models are versioned, enriched with metadata and can be linked to binding approval workflows.

This allows you to trace which data sources each relevant model version is based on, which processing steps were involved, and which teams were involved. While the technology alone does not ensure compliance ‘out of the box,’ it does provide the necessary tools to efficiently integrate the required technical documentation, risk assessment, and audit compliance into the development processes.

The ‘works council factor’: transparency without monitoring

A critical success factor for data projects in the DACH region is the protection of employee rights, particularly with regard to the prohibition of unlawful monitoring of behaviour and performance. For technical reasons, the Unity Catalog logs detailed metadata on user activities, executed queries and runtimes. This data is indispensable for security operations and forensics, but requires differentiated handling in the context of co-determination.

To avoid compliance risks and project delays, it is advisable to coordinate a governance concept with the data protection officer and the works council at an early stage. Proven technical measures include:

• Restrictive authorisation management: Access to granular audit logs (such as system tables and audit events) is strictly limited to a defined group of security administrators.
• Pseudonymisation in monitoring: For general platform reports (e.g. for performance optimisation or cost control), user IDs are aggregated or pseudonymised by default. This technically prevents conclusions about individual work performance.
• Logged de-anonymisation: Pseudonyms are only resolved to real names in defined cases of suspicion (such as security incidents) and under the application of a documented dual control principle.

By transparently defining these parameters, Unity Catalog enables platform operation that reconciles high security standards with the requirements of operational co-determination.

Conclusion & outlook: Governance as the foundation for scalable innovation

Information security is not a static state, but a continuous process. The use of the Unity Catalog transforms governance from manual control mechanisms to automated, code-based policies (‘Policy as Code’). This creates a development environment in which regulatory requirements from the GDPR, the NIS 2 Directive and the EU AI Act – such as access control, logging and lineage – are already architecturally embedded. Governance is thus transformed from a limiting factor to a necessary enabler for productive AI operations.

Technological independence is crucial here: through the consistent use of open-source standards (Delta Lake, Apache Parquet) at the core of the architecture, security is not compromised by proprietary dependencies. Companies thus benefit from enterprise-grade governance without sacrificing their data sovereignty or the long-term interoperability of their data assets.

Outlook

With an open standards-based, compliance-compliant data architecture, the foundation has been laid. In the next part of our series, we will focus on the value-adding application of this database.