Releases

Release 2024.12.00

Featured

Access control

You can now create access control policies to manage access to entire golden records or specific vocabulary keys. Together with source control, access control helps you configure reliable and secure access to data in CluedIn.

Export target renaming

You can now give a user-friendly name for the export target, making it easier to quickly locate in the list of all export targets.

Ownership

Almost every element in CluedIn now has the Owners tab with a list of users and/or roles who are considered owners. The owner can approve or reject change requests submitted by non-owner users as well as make direct changes to the element.

If you created an element in CluedIn and trust other users to process change requests and make direct changes to the element, you can add those users to the list of owners. You can also add roles to the list of owners; this way, all users who have that role will get the ownership permissions.

Normalize Date rule action

We’re turning off the default Date Time normalization option (Administration > Settings > Processing Property Data Type Normalization). Now, you can add the Normalize Date rule action to convert the values of the vocabulary key to ISO 8601 format (YYYY-MM-DDT00:00:00+00:00).

For dates that have already been converted when the Date Time option was enabled, the Normalize Date rule action won’t have any effect at all because the dates are already normalized.

Workflow module (Power Automate)

We’re introducing a new Workflow module. It is designed to help you automate and track approvals and notifications for specific activities in CluedIn. With workflows, you can easily streamline such processes as modifying vocabularies, inviting users, creating rules, and more. Please note that the approvals and notifications that are sent outside of CluedIn are intended only for SSO users.

Duplicate actions

Now you can easily duplicate clean projects, rules, deduplication projects, glossary terms, streams, and more. This action allows you to create new elements in CluedIn using the configurations of the existing elements. For example, when you duplicate a clean project, you get a new clean project with the same configuration, but without the cleaning activities and rules from the original project.

Accountable Task approval

When a change request is sent off for approval. Only the owners of the rule, stream, deduplication project etc used to be able to approve the changes. We now send the request to anyone that has the relevant claim at the Accountable level as well so that anyone that is set as an owner or has the required claim at the Accountable level will now be able to approve the changes.

RabbitMQ consumer loss fix

This fix was originally released in the 2024.07.01 update, but we wanted to highlight it again in this release. In the past, you may have encountered issues where data didn’t process or jobs didn’t run properly under heavy system load, requiring assistance from CluedIn. These problems were often caused by dropped consumers, which prevented messages from being processed.

If you’ve experienced these issues and haven’t upgraded to the previous patches yet, rest assured they’ve been resolved and are included in this release.

Elasticsearch Dynamic Templates

In version 2024.01.00, we introduced strong typing to CluedIn, allowing you to store and query typed data—for example, filtering dates within a specific range. Since then, we’ve expanded the system’s functionality, which sometimes required changes to how data is stored in Elasticsearch. This occasionally caused issues when writing data before those changes were fully applied.

To address this, we’ve implemented “dynamic templates.” These templates automatically handle data storage in Elasticsearch, so you don’t need to take any action—it all happens behind the scenes. If you’ve experienced issues with strongly typed data or filtering in the past, this update should resolve them.

Release Notes

CluedIn

Features

• The ability to define Access Control Rules
• Speed up processing of entities containing Lookup dictionary keys
• Improve deduplication generation throughput, by inserting groups in bulk
• Add endpoints to delete entities
• Add endpoints to delete dataparts in an entity
• Allow Deduplication Result to be filtered by name and group status
• Ability to create Access Control Rules to control visibility of entities
• Clean now only shows data thaat the clean project creator can see
• Add stages and transitions for the initial work on deduplication automation
• Allow for additional connector configuration within a stream
• Improved the reliability of writing/reading data to Elasticsearch
• Can set the display name of a provider definition when it is created
• Owners and users in roles that have the relevant claim with level Accountable will now receive the approval tasks
• Allow permissions to be added using roles to Connector, Enricher, Export Targets, and Integrations
• Rule action for date normalization has been added
• Improved the performance of clean generation/commiting when there are multiple projects and a lot of data has been generated
• Rework streams to have single Backlog queue instead of a chain of 3 queues to improve performance and reliability
• Add centralized Ownership of system objects and services
• Replicate static Vocabs and Keys in database on startup to avoid fatal errors if a vocabulary gets removed when a provider is removed and the vocabulary is referenced within the system
• Support external workflows via Power Automate

• Additional liveness checks to detect rabbitmq disconnects on processing pods Normally rabbitmq connections should auto recover within 5 min. If this doesn’t happen. Then the liveness probe will report “Red” status. From this point Kubernetes liveness thresholds will kick in

  Effectively this means:
  CluedIn server will not report failing liveness probes within the first 5 minuets (to allow the connection to auto recover by it self) Kubernetes liveness probe 10min-25min (depending if there is liveness probe timeouts)
  In total 15min-30min before Kubernetes will restart the pod

  The following settings have been added:

  Configuration key	Default Value
  Health.SystemLivenessChecks.Enabled	true
  Health.SystemLivenessChecks.InitialDelaySeconds	600
  Health.SystemLivenessChecks.CheckIntervalSeconds	30
  Health.SystemLivenessChecks.ConsecutiveErrorCountThreshold	10

• Add raci checks when activating/deactivating rules
• Improve the stability of streams when working with large datasets
• Do not allow glossary terms to be deleted when they are used
• Creating a glossary term now requires a filter to be set
• Deduplication project rules are audited

• Add outgoing edge property values to IndexedText so they are searchable via normal entity search The following configuration key have been added to enable / disable this feature

  Key	Default Value
  ClueProcessing.IndexedText.EdgeProperties.Enabled	true

• Users added RACI requirements to execute relevant copilot skills
• User that are not owners/accountable now will require approval to change the state of a stream between start/pause/stop
• Access Control Policies rules are now applied to the relations tab and related entities
• Introduce platform upgrade mode
• Show user notification if platform fails to configure stream in the background (e.g. properly start it or update stream container)
• Improved resilience when querying Elasticsearch
• Made property value normalization in the processing pipeline configurable through the organization settings
• Disabled the property value normalization of date time values by default in the processing pipeline
• Add edge properties to the entity history api
• Health monitoring for StreamLog, StreamCache & CleanCache
• Make RabbitMQ messaging more robust by using confirmation of the delivery
• Enhance streams robust nous when making changes to stream configuration
• Enhance explain log to display formatted “added & removed” edge tables for edge merging

• Change prefetch count defaults

  Setting	Old value	New Value	Description
  ClueProcessing.Subscribe.DefaultPrefetchCount	50	x10	Default prefetch count for processing queues
  ClueProcessing.Subscribe.ClueProcessing.PrefetchCount	25	x14	Prefetch count for Clue queue
  ClueProcessing.Subscribe.*.PrefetchCount	25	x10	Prefetch count for most of the processing queues
  ClueProcessing.Subscribe.DeleteEntity.PrefetchCount	5	x1	Prefetch count for delete entity commands
  Messaging.Consumer.PrefetchCount	implicit 50	x10	Prefetch count for Robust Messaging subsystem

• Change default setting value

  Setting	Old value	New Value	Description
  ProcessingCommandTaskScheduler.MaxThreads	8	x16	Concurrency Level for the Processing Task Scheduler

  • Link the hierarchy records back to the hierarchy project in the history tab

Fixes

• Stream actions are not applied when previewing data within streams
• Elasticsearch has a small chance under certain circumstances to store data in incorrect fields
• Actions are not applied if reprocessing is triggered from within the rule builder page for certain filters whilst using special characters
• Changes made using the Excel add-in shows the default source icon when viewing changes in history tab
• Synchronous call paths over asynchronous operations lead to reduced throughput in the task scheduler
• Hierarchy builder sources are displayed in filters
• Money data type does not work with some rule operators
• Filter rules that use dates return the incorrect values under certain circumstances
• Various date formats do not work with the rule builder for strongly typed date times
• Users can remove golden records using GQL even if they don’t have the correct permissions We now require admin.datamanagement at a Consulted level to delete records via the GQL endpoint
• Incorrect vocabulary usage is returned if the vocabulryprefix is a subset of another vocabulary. For example if you had a vocabulary with a prefix my.prefix and another vocabulary with a prefix of my it would should usage for both vocabularies
• Rules are not applied depending on the filter combination and order
• Possible error in the rule builder when the vocabulary does not exist anymore
• Incorrect RACI requirement when trying to view the list of glossaries
• Source is absent for some vocabularies
• Can’t update filters after change data type for a vocabulary key
• Delete ‘Stream/Rule/Glossary’ change request lacks information about the deletion
• Glossary approval does not log tag changes correctly
• Manual data entry project not properly updated with renamed vocabulary key
• Adding an entity type to a page template is not applied for a minute
• Inactive glossary terms are showing in the filter glossary selection list
• Blank organization settings that are secret are not saved
• Part id compaction during entity merging could cause save errors

• EntityEdge with same reference point would merge regardless of properties being different in EntityEdgeCollection

  Old behaviour can be re-enabled with configuration key Feature.EntityEdgeCollection.LegacyEdgeMergingWithoutConsideringPropertiesEnabled

• Reprocessing rules does apply if the rule filter is empty
• Rule audit log entries appear out of order
• Export target details are not audited when adding or changing the configuration on a stream
• Vocabulary Key approval tasks are not displayed in the list of pending changes on the vocabulary key page
• Entities are not enriched by an enricher if a previous enricher fails
• Related tag changes are not displayed in the glossary term audit log
• Under certain circumstances not all approved deduplication groups would be merged, when committing groups
• DeduplicateRelationships are not always handling edge properties correctly during entity merging
• EntityEdge with same name only reference would merge regardless of properties being different in EntityEdgeCollection
• Edge processing skips edges with the same entity code but different properties during shadow entity creation
• Provider definition ids are not being returned in GraphQL
• Glossary terms can be deleted whilst they are in use in the system
• Providers with different configurations can be flagged as duplicates
• Regex replace rule action does not allow you to replace with empty values

CluedIn.MicroServices

Features

• Introduced an API to display the status of switching to edit mode
• Enabled the ‘Clear Content’ functionality for endpoint data set types
• Provided an API to verify access via the global security filter
• Introduced an API to display the Data tab at the data source level
• Implemented paging for existing annotations and updated annotation naming convention to ‘{dataSetName} - mapping’
• Allowed access to the Data tab based on user claims
• Displayed additional information about received data in relevant sections
• Expanded available time ranges on the Data Set Monitoring tab for ingestion-data-set and commit-data-set queues
• Leveraged the quarantine DTO in applicable workflows
• Added a sanitize option for creating data sets
• Applied integrity transactions across the ingestion and mapping processes
• Updated RabbitMQ connection strategy with improved channel reconnection logic, allowing three retries before crashing
• Introduced a setting to create deletion policies for logs and submissions retention
• Transitioned ingestion endpoint responsibilities to processing for better performance
• Enhanced stability in operations involving Elasticsearch
• Resolved an issue where an odd submission was created unexpectedly
• Displayed validation errors for data type mismatches in the data type selector
• Provided API support to validate access through global security filters
• Implemented query parameter validation to prevent SQL injection vulnerabilities
• Optimized SQL calls during the consumption of ingest-data-set messages
• Introduced the ‘Integrity Transactions’ feature to retry critical operations in case of failure
• Logged ‘Integrity Transactions’ details in Markdown format, configurable through the INTEGRITY_TRANSACTIONS_LOGS_FILES_PATH environment variable
• Monitored upload processes with a job running every three minutes, marking datasources with broken uploads as error states
• Deactivated SQL scheduled jobs when datasets are removed
• Added notifications for failed uploads to improve error visibility

Fixes

• Resolved an issue with loading more datasets when the first column was a UUID
• Addressed content clearing issues with Elasticsearch
• Corrected the behaviour of the ‘To Boolean’ operation
• Prevented dataset preview crashes
• Improved record removal functionality after purging submissions
• Adjusted retention jobs to avoid removing all logs inadvertently
• Resolved errors after switching to the default endpoint mode
• Updated exclusive queues to durable types for stability
• Enhanced claim validation logic
• Disabled submissions and log retention by default for better control
• Displayed dataset processing notifications only after the last submission is complete
• Improved UI benchmarks and addressed logs retention inconsistencies
• Updated database schema and UI validations for display names in annotations and vocabularies
• Fixed validation for legacy endpoints
• Addressed indexing issues during dataset creation
• Provided clearer messaging when data is successfully received
• Corrected processing issues when hash check is enabled
• Hidden _id property rules for better data handling
• Adjusted annotation name length restrictions
• Enabled clearing of original fields and added ingestion logs for new fields
• Limited dataset listing for strict edge to mapped items only
• Resolved issues where data was returned despite failing global filter checks
• Improved handling of quarantine clues and operations for datasets
• Adjusted JWT parsing for consistency
• Enhanced dataset operation fixes, including filter rollback and sanitation for Unicode
• Allowed newline characters (\n) and numbers in mappings
• Corrected URLs and field duplication in CSV exports
• Followed up on Unicode sanitization with improved regex handling
• Validated new column names and addressed mapping crashes due to incorrect data
• Added the ability to refresh datasets in the UI
• Fixed UI updates after receiving data or quarantine changes
• Resolved issues with executing integrity transactions and dataset preview pagination
• Improved recovery file handling and submission statistics refreshing
• Extended image mapping timeouts to prevent crashes
• Addressed data type validation errors during mappings
• Improved prompts for AI-based mappings
• Implemented measures to prevent crashes during checkQueue on missing queues
• Ensured message persistence during publishing
• Enhanced AI mapping functionality
• Resolved stuck operations during dataset processes
• Added bulk upsert support for retry loading scenarios
• Fixed XLSX export of golden records and processing message parsing
• Deactivated SQL jobs when archiving or removing datasets
• Prevented logging errors when uploading Parquet files with duplicated columns
• Purged manual project logs more effectively
• Addressed an issue where the progress for switching to edit mode would incorrectly reset to 0
• Resolved a problem where public tokens were unable to parse claims properly
• Improved functionality for switching to edit mode, ensuring smoother transitions

CluedIn.UI.Gql

Features

• Feature flag for ‘Knowledge Hub’ to access the knowledge hub
• Feature flag ‘CluedIn News’ to access articles and videos
• Feature flag ‘Nwe Home Dashboard View’ for use with the new dashboard home view
• ‘Advanced Mapping’ feature is no longer under a feature flag and is in general release
• ‘Allow basic XLS’ feature is no longer under a feature flag and is in general relea
• ‘File upload status’ feature is no longer under a feature flag and is in general relea
• ‘Import databases in data source module’ feature is no longer under a feature flag and is in general relea
• ‘Manual data entry’ feature is no longer under a feature flag and is in general relea
• ‘Manual hierarchies’ feature is no longer under a feature flag and is in general relea
• ‘Mapping property rules’ feature is no longer under a feature flag and is in general relea
• ‘New entity relations graph with D3’ feature is no longer under a feature flag and is in general relea
• ‘Profiling dashboards’ feature flag is now in Beta and enabled by default
• ‘Remove golden records’ feature flag is now in Production and enabled by default
• ‘Data set filters and operations’ feature flag is now in Beta and disabled by default feature flag
• ‘Duplicate actions’ feature flag is now disabled default
• ‘Annotation UI’ (deprecated) feature flag has been disabed and removed
• Removed feature flags for New clean project, New Mapping Screen, Data Set Logs , Endpoint bridge mode, Data Set Monitoring, Data Set Pre-process Rules, Data Set Quarantine, Entity Relations with new API, New Entity History
• Removed Potential Duplicates, Clean Legacy feature flags
• Changed feature flag status to ‘BETA’ for Parquet File Support
• Changed feature flag status to ‘Production’ for New Entity Relations with D3

CluedIn.UI

Features

• Added searchable Vocabulary Key selector for:
  1. Add edge relations panel
  2. Map entity - Map Entity Name
  3. Map entity - Description
  4. Map entity - Date Created
  5. Map entity - Date Modified
• Update the organization settings page to have a save all button and display validation messages
• You can filter by Edge Type on the relations tab
• New settings for retention and submission logs settings in datasource
• ‘Delete’ button has been added to the record page, enabling the deletion of a record
• You can delete records from the search page, enabling the deletion of one or multiple records
• Progress bar is displayed when switching to edit mode in a data set
• Added ‘Clear content’ ability for endpoint data set type
• Changed generate rules button to be always available
• Added a delete button to node record panels in the Topology graph, enabling the deletion of a data part
• Added a delete button to history item’s record panels and under the history item’s status badge in the History view, enabling the deletion of a data part
• Fixed SplitIcons not rendering correctly in the Topology graph
• Improved validation display for data type selection when mismatched with provided data
• Ensured global security filter access checks on dataset content
• Ensured global security filter access checks on dataset content in the Data tab
• Enabled definition of Access Control Rules
• Displayed warnings on the Entities Properties page when entities are filtered
• Added a Data tab to the data source level to display records by provider definition ID
• Implemented paging for existing annotations
• Enabled search functionality within the annotations selector
• Updated annotation naming to {dataSetName} - mapping
• Introduced a disabled state and tooltips for the Delete button in History/Topology views
• Added a three-dot menu in the History view for deletion options with tooltips
• Fixed alignment of history item indicators in the History view
• Introduced a Back button on the Import From Database panel (step 2)
• Displayed information about received data
• Expanded dataset monitoring with additional tiles for ingestion and commit queues
• Resolved missing name and entityType values in the delete data part modal
• Added a Disk Usage tile and related charts to the Engine Room - Statistics section
• Enabled setting of provider display names
• Updated saving toast notifications
• Supported custom data providers for dropdown creation
• Allowed permissions to be assigned via roles for Connector, Enricher, Export Targets, and Integrations
• Added a Documentation module to the Knowledge Hub for workflow support
• Introduced feature flag checks to toggle menu visibility in the Knowledge Hub
• Added an Articles page to the Knowledge Hub rendering content from cluedin.com
• Added a Videos page to the Knowledge Hub rendering content from cluedin.com
• Enabled a CluedIn News feature flag to access Articles and Videos pages
• Added a sanitize option when creating datasets
• Enabled previewing Rules, Conditions, and Filters when approving requests
• Displayed mapped key information in the vocab keys list
• Supported notifications for replicating static vocabs and keys
• Improved caching for Knowledge Hub articles/videos and added View More links
• Allowed glossary terms to be duplicated
• Adjusted global pillar module tiles to display descriptive text
• Enhanced submodules with descriptions and improved dashboard styling
• Removed Actions headers from module and submodule dashboards
• Updated the Help Center with external documentation links, support ticket forms, and suggestion forms
• Enabled stream duplication
• Refined Copilot and My Tasks modules in the user panel
• Removed unused links from the user panel
• Allowed role-based ownership support
• Integrated external workflows module with Power Automate widgets
• Ordered global menu and claim sorting in Role settings
• Restricted unauthorized data source editing in Role settings
• Added KPI tiles with data for the new home dashboard
• Enhanced business domain statistics and tooltips for the new home dashboard
• Enabled feature flags for the new home dashboard view
• Updated headers and KPI button text in the new home dashboard
• Supported approval status handling for rule bulk actions
• Added icons to the Help Center panel
• Enhanced handling of RACI claims in business domain charts
• Added data tabs for processed data specific to enrichers
• Added data tabs for processed data specific to integrations
• Introduced a Data Set Admin page to analyze ES mappings and create missing ones
• Added a Copilot button to the top navbar
• Enhanced tooltips for the Business Domain chart
• Added prompts for Copilot settings when configurations are empty or erroneous
• Enabled API support for glossary term usage
• Disabled selection of unhealthy export targets during stream configuration
• Disabled stream start for unhealthy export targets
• Restricted data visibility in certain UI elements when the global security filter is active
• Deprecated features: Potential Duplicates and Clean Legacy removed; Parquet File Support moved to BETA; Entity Relations with D3 transitioned to Production. Feature flags for New Clean Project, New Mapping Screen, Data Set Logs, Endpoint Bridge Mode, and others removed
• Introduced save confirmation modals on the export target page
• Displayed previously merged results on the Merging tab while processing matches
• Enhanced custom icon upload functionality for data sources
• Centralized ownership introduced for system objects and services
• Improved tooltip for export target names when editing streams; added useful details to export list columns; introduced filtering by Active and Inactive terms in Glossary
• Updated image display in the Data Set Preview
• Enabled duplication of rules (Data Parts, Survivorship, Golden) and Clean Projects directly from their respective lists
• Adjusted button appearances throughout the application
• Updated explanation messages for “no entities” on the Data Tab
• Simplified creation by enabling duplication of deduplication projects
• Added support for approval workflows in stream and deduplication projects
• Disabled the “Add Filter” button in preview mode
• Refined dropdown menus and buttons across the application
• Adjusted “Open” and “More Vertical” button styles in Hubspot CRM entity details
• Updated confirmation pop-up logic in Access Control
• Removed the status column from Entity Type -> Vocabularies
• Added the ability to view role permissions directly on the role page
• Improved validation logic for Glossary entries
• Corrected the “search not found” message in the Deduplication Owners tab
• Enable XLSX export
• Adding max length validation for ‘Add property’ modal when adding a property to an added edge in mapping
• Adding max length validation for ‘Map Field’ panel when mapping a field during manually mapping

Fixes

• Amending the vocabulary key value length when uploading a file
• Added name validation for exporting golden records
• Fixing record deletion in the search for both table and tile views
• Added confirmation modal to confirm changes will be lost when navigating away
• Cleared all data after clicking the Back button in the import flow
• Avoided page numbers conflict for Data and Data sets tabs
• Adding data management edit claims at CONSULTANT requirement for selecting entities for merge
• Fix data set preview page crashes
• Fix displaying errors in data set preview
• Added warning message in case the same vocab key as deleting is selected
• Modified logic of global checkbox with filtered and unfiltered list
• Adding No Name fallback if name value is missing in the Delete modal
• Fix remove records when submissions were purged
• Fixed an issue with selecting of logs statutes to purge
• Hided advanced mapping code preview when endpoint in ‘bridge’ mode
• Added usages of Access Control Policies
• Added Data Catalog claim in the Management section at Consulted Level to add or edit property
• Fixed issue when switching to edit mode caused progress to reset
• Reduced overall query calls for checking if data parts can be deleted
• Fixed ‘Delete’ button for data parts in Topology and History panels
• Allow data tab visibility if claims allow
• Disabling deletion of temporal entities on a temporal entity’s page toolbar
• Adjusted DeleteGoldenRecord to support disabled state and tooltip messages
• Fixed search menu ‘select’ option for ‘Select for manual merge’ and ‘Select for deletion’
• Adding locale for when a user tries to delete themselves
• Disabling the ‘Next’ button in the delete record modal if a record cannot be deleted
• Fixed the omission of the isTemporalEntity key in the delete record modal
• Enabled deletion of records of differing entity types in the search list
• Displaying No Name instead of ‘[No Title]’ in the delete golden record modal
• Showing the entityId in the delete golden record modal
• Fixed an issue with removing the last character in inputs for max/min range in Add property rule modal
• Adding ‘No topology to show’ message/illustration if a record has no topology records/nodes
• Made bigger tooltips for advanced filters in Exported Files
• Removed double scrollbar from delete golden record modal
• Hided odd tiles in bridge mode
• Changed ingest URL when running the UI locally
• Added message about restricted author information when the user cannot view author info
• Fixed issue when no data sent was shown
• Centering deletable icon in the golden record deletable modal
• Fixed entity type ellipsis in the golden record deletable modal
• Fixed issue when empty state component was shown while loading data
• Pending changes tab is not shown if Mesh center is disabled
• Fixed an issue with incorrect badge if submission logs were purged
• Fixing error when undoing manual or deduplication merge
• Omitting the entityType if missing when deleting a data part
• Fixed an issue with no matches displayed for Deduplication
• Removed the displayName field from the AnnotationProperty database
• Extended the displayName field in the VocabularyKey database to 500 characters
• Updated all UI validation to reflect changes for the display name
• Adding a warning message to the delete data part modal if enrichers may reintroduce deleted data parts
• Adding a disabled red color for the delete data part menus
• Adding an ellipsis to the delete data part modal entityId
• Adding a disabled state to the delete button on a record page if a record cannot be deleted
• Adding error text and disabled buttons if there are errors when fetching data part info or failing to delete data parts
• Avoided cache for matching results requests
• Fix page update after clearing logs
• Removed Explain Log tab if the user doesn’t have access
• Fixing relations page filters for ‘Hide Shadow Records’ and ‘Hide Temporal Records’
• Fixed issue with incorrect button shown after purging submission logs
• Fixed an issue with create export target form validation
• Fixed an issue with a missed uploaded datasource logo
• Fix edit name panel title
• Fixed issue when reset mapping UI was broken
• Fixed an issue with a broken sidebar layout for vocab key remapping
• Fixed an issue with flickering “Load more” button during merge matches
• Added ability to clear originalFields during clearing records
• Added logs if ingestion introduces new fields
• Auto disable suggested origin if its field is ignored
• In list of datasets for strict edge, show only mapped ones
• Fixed issue when some data was returned despite the global filter failing
• Fixed issue with odd warning tooltip in case of auto-fixed fields
• Data set filters & operations fixes
• Fixing mapping preview header cells to show an icon and link for obsolete vocabulary keys
• Fixed global filter check to respect permissions
• Fixed reverting data set operation
• Fixed preview refresh after performing operations
• Fixed process tab’s actions tooltip
• Fixed locales for data set operations
• Removed purge quarantine button
• Disabled generate clues option if no global access filter
• Fixed remove golden records modal layout
• Adjusting mapping preview header cells for obsolete vocabulary keys
• Disable adding column if user has unsaved changes
• Fixing missing id and name values in Topology for non-admin users
• Buttons in the Term section on the Glossary page conform to style guidelines
• Fixed issue when user could create edge with empty origin
• Fixed rules when the key was reprocessed
• Added explanation about reusing mapping
• Amending the user activation sign-up screen text typos and text
• Changing appearance of Glossary page buttons
• Dataset operations fixes
• Fixed border visibility issue in “Entity Page Layouts” page
• Globally changing appearance of buttons
• Change glossary page button appearance
• Fixed alignment of elements bug in the Data Source > Logs page
• Fixed validation during add extra column
• Changing appearance of buttons and text in App
• Changing appearance of pop-up window
• Changing appearance of long vocabulary keys preview in filters
• Changing appearance of image in Data Set Preview
• Changing appearance of toggle button
• Changing appearance of export target name configuration
• Changing space between buttons on the Data Part Rules page
• Disable editing computed fields
• Added support for normalize date rule action
• Fixed operations layout and computed fields column
• Changing appearance of rounded buttons within App
• Changing appearance of Data Set browser title
• Added missing claims for “Select All” button on deduplication matching
• Changing browser title name in Data Set Preview
• Added sorting by created date in Stream columns
• Fixed tooltip position
• Fixed tabs highlighting on Rules/Glossary
• Fixed glossary layout and added error handling for activating glossary
• Fixed supported files list according to feature flags
• Fixed streams/export targets claims
• Changing appearance of “More Menu” icon and buttons in global search
• Changing hover style for import dropdown menu
• Reverted edit mode on Stream
• Fixed removing extra fields and reverting dataset to original
• Show message when deleting term awaits approval
• Show waiting for approval info on glossary
• Changing hover style for column options panel
• Fixed lineage zoom area issues
• Fixed an issue with Accountable level for Data Catalog
• Changing hover style for CTA buttons without color containers
• Fixed an issue with glossary term short description validation
• Fixed rules duplication
• Removed zoom ability in Processing Pipeline
• Added new column names validation and disabled actions menu during operations
• Updated data set after adding a column operation
• Fixed edit mapping
• Show data tab on datasource level
• Fixed dataset filters layout
• Removed “Actions” and “Report” headers from the “Integrations” page
• Fixed an issue with an incorrect date format example
• Fixed duplicating a rule
• Fixed a 500 error when trying to activate a newly created Glossary term
• Fixed issue when UI wasn’t updating after receiving data or quarantine updates
• Fixed module names, URL paths, and breadcrumb titles for Profiling pages in Governance and Engine Room
• Fixed issue after editing rules when changes were not applied
• Fixed broken message for Manual Data Entry project with no owners
• Changing hover styles for buttons
• Fixed “Pending changes” and “Audit log” tabs being hidden for Accountable level users
• Prevented users without data management claims from seeing detail pages, showing a claims-required page instead
• Allowed access to Deduplication Page without management.duplicates claim
• Fixed missing info about wrong field names and duplicate actions
• Restricted access to Duplicate modal for users lacking necessary permissions, with tooltip guidance
• Changed text alignment of buttons in empty available integrations list
• Fixed Part ID section theme on Links in History Relations Page
• Displayed access control policy description
• Fixed issue with editing vocabulary key with special characters in name
• Added badge display for aborting Clean Project generation results in all items, not just List
• Hidden Action Bar showing rows selected for Preview Mapping Screen
• Fixed Preparation and Hierarchy side panels pagination issues
• Fixed refresh submission stats
• Added filter by Entity Type on datasource data tab
• Fixed odd scrollbar issues at datasource level
• Fixed search page filters not persisting after refresh
• Fixed flickering “Load more” button during merge matches
• Fixed refresh data set issue after data posting
• Fixed missing tooltip on Remove Entity button
• Fixed text validation for Help Center ‘Report Issue’ and ‘Suggest Idea’ forms
• Edited preprocess rule label
• Added confirmation modals to Help Center for unsaved forms, updated error illustration colors, and made support links clickable
• Displayed task title when related entity link has empty ID
• Changed default items per page to 20 on data source groups list
• Fixed data types validation and SQL datasource name validation
• Adjusted preprocess rules vocab key select
• Fixed property rules details modal
• Added Help Center “Ticket ID” locale
• Fixed validation during dataset creation
• Corrected source control disabled message
• Adjusted business domain bar chart label positioning
• Added “Home” item and reordered navigation menu items
• Fixed issue preventing access to records on the second page after adding all roles from the first page
• Added cancel/remove functionality for golden records
• Showing ‘Use option’ in select while options are loading
• Added new badge to Streams status
• Changed text in “Clean” modal
• Updated deduplication merge modal text and corrected option order
• Removed scrollbar from deduplication modal
• Amended clean and deduplication modals to select a default option
• Fixed hidden “Pending Changes” and “Audit Log” tabs for users with Accountable claim level in Vocabularies
• Prevented non-owners from starting/stopping/pausing Streams without approval
• Fixed SQL dataset name validation and vocab key select in preprocess rules
• Displayed friendly message when clicking a deleted entity in the hierarchy project
• Adjusted “No Duplicates” message alignment in the reprocess panel
• Fixed navigation menu order in Governance to match module tile order
• Improved responsiveness of the business domain pie chart legend in the home dashboard
• Resolved page crash when non-owners edited data part rules with multiple paragraphs
• Updated hierarchy project version on save/publish actions
• Corrected datasource and manual project claims on home tiles
• Added external link icon to open used business domains in a new tab
• Fixed issue with manual data entry tile being enabled for users without proper claims
• Adjusted Streams status badges alignment
• Fixed integration type filter not showing selected type
• Rendered vocab key select on preprocess rules
• Added labels for mapped keys in mapping
• Reset changes after saving organization settings
• Improved behaviour of the sidebar panel in mapping vocabulary key
• Updated appearance of the page after inviting a user
• Enabled permission requests for Explain Log tab access
• Changed tooltip visibility in the Column options panel
• Updated behaviour of the invite user panel in “All Users” page
• Updated appearance of long-name tabs in overview
• Prevented horizontal overflow in the Help Center panel description
• Removed black cell border from the business domain pie chart in the home dashboard
• Adjusted tooltip positioning in the Relations tab
• Improved logic for Add Actions panel closing
• Fixed handler logic for adding roles or users in Access Control Members cell
• Adjusted tooltips for long business domain labels in the home dashboard
• Disabled “Show More” button for users lacking permissions to view charts
• Updated tooltip text for “Clean” and “Deduplication” KPI tiles
• Removed RACI claims requirement for viewing “Billable Record” tiles
• Added notification for non-owners attempting to remove Glossary categories
• Improved vocabulary key creation logic in Edge Properties
• Added remove button and notification for failed datasource uploads
• Fixed SQL scheduled job badge and disabled job creation for archived datasets
• Improved error message display on settings page actions
• Retrieved tags info on Streams Data tab and Preview Condition tab
• Hidden “Retention settings” in manual data entry
• Set “Push” as the default option in Clean ‘Process’ modal and reset state on close
• Updated claims levels for role access
• Fixed grouped relations issues in Hierarchy projects
• Adjusted claim levels for Export Targets access
• Added warning dialog for unsaved Copilot settings
• Consistent font-weight for Copilot button across themes
• Changed labels and added links for ownership actions in audit logs
• Improved add rules panel appearance for users
• Updated availability of Export Target tab on Streams page
• Fixed vocab key panel issues in custom layout editing
• Enhanced Survivorship Rules with fined data options: Best Quality, Latest Manual, Latest Modified Date, Most Frequent Value, Most Properties, Oldest Modified Date, and Provider Definition
• Fixed stream claims and adjusted Deduplication > Matches page button availability
• Updated topology after splitting records
• Adjusted button availability in Integrations based on claims level
• Updated Rules page and Grant/Revoke access configurations for claim-dependant display
• Fixed entity page header appearance
• Fixed export panel functionality
• Enabled RACI permissions and adjusted styles for Copilot chat
• Fixed glossary term removal issues
• Displayed correct filter type in review task changes
• Fixed edit entity code issues on Data Sources and Manual Projects
• Adjusted entity codes panel width
• Updated roles display on the Users page when claims are missing
• Improved Configuration Permissions tab appearance when no users are present
• Updated topology handling after splitting records
• Fixed Home Dashboard Golden Record count refresh and RACI-based record hiding
• Resolved ellipsis issues in various locations, including entity detail vocabs
• Added ability to display Streams actions in the Audit Logs tab
• Fixed infinite loader in Manual Data Entry Permission tab
• Displayed invalid filter status badge only when a stream stops with an incorrect status
• Allowed users with consulted-level claims or higher to view Audit Logs even without project ownership
• Fixed crash in the Vocabulary Owners tab
• Adjusted Glossary create button availability based on claims
• Updated Streams tab in Export Targets for missing claims scenarios
• Restricted editing in Export Targets without necessary claims
• Updated Permissions tab appearance in Export Targets for users without claims
• Amended ‘Add Record’ button for Sources KPI to redirect to the Integrations page
• Updated Help Center panel close button behaviour to skip confirmation when a ticketId is present
• Streamlined API calls for History tab functionality to reduce redundancy and improve efficiency
• Changed Save Settings button availability based on conditions
• Updated Access Status tab appearance
• Adjusted preview action button availability in Streams
• Updated Matches tab appearance in Deduplication for missing claims scenarios
• Limited business domains shown in Home Dashboard ‘Business Domain Count by Type’ bar chart to 9 and removed ‘Show More’ button
• Improved Author column appearance across different contexts
• Fixed vocabulary key creation in Form Field creation forms
• Improved refresh rate for Business Domain charts in the new Home Dashboard
• Prevented additional executions when the total items length is reached in Grouped Relation Panel List
• Fixed “No data found for the stream” placeholder display in the “Preview Condition” tab when expected
• Resolved issue where the delete golden records button was not active
• Introduced new claim requirement for adding users/roles in the Permissions tab for Data Sources, Manual Data Entry Projects, and Enrichers
• Ensured page refreshes correctly after removing a data part
• Fixed record removal issues
• Disabled hierarchy publish button during publishing
• Resolved export target configuration crash
• Resolved an issue where the ‘No data sent’ badge persisted after data was sent
• Addressed deduplication group size slider causing excessive requests
• Improved various UI components related to datasets
• Prevented users from adding themselves as owners and making unauthorized changes
• Corrected validation issues in Access Control
• Enhanced filtering functionality for Ownership items
• Adjusted the display of the Add Role button on the Owners tab’s Add role sidebar
• Ensured the paging selector is no longer hidden behind the scrollbar on Owners -> Add Users sidebar
• Fixed display of long names in the Revoke Ownership confirmation modal
• Updated the Stream icon and resolved empty page issues after deleting a stream
• Improved filter updates when changing the data type for a vocabulary key
• Addressed validation of the Email field during user invitations
• Enhanced validation for data types during mapping
• Resolved issues switching back to the original mode
• Updated the short description display for glossary terms
• Made the “Save” button in the Retention Settings panel inactive until changes are made
• Improved the sidebar layout for vocabulary key remapping
• Ensured proper validation for glossary term names, matching rules, and criteria names
• Boosted performance for the dataset filters modal
• Added a notification for waiting approval status
• Corrected spelling errors in the Access Status tab
• Addressed truncation issues for Entity Types and Vocabulary Keys

Runtime-Environment

Features

• Introduce custom table type needed for bulk group insertion on deduplication projects
• Added submission retention and log retention columns to the data sets table
• Added DeduplicationRejection, DeduplicationRejectionEntry and EntityIdAndOriginEntityCode tables to support the automatic rejection of duplicates
• Added a ConnectorProperties column to the streams table to allow us to store additional configuration settings for export targets
• Unique constraint added to the StreamMapping table to avoid adding the same vocabulary/property twice to a stream
• Added a RecurringJobData table
• Added columns to store sanitization configuration values on the DataSets and DataSourceFiles tables
• Dropped OrganizationProviderOwner, TaskOwners and DeduplicationProjectOwner tables and dropped all owner related columns as we have added and migrated to a centralized AccessControlOwnership table
• Increased the Vocabulary Key Prefix from 50 to 250 chars
• Added columns to the Vocabulary and VocabularyKey Definition to support storing all vocabularies and vocabulary keys in the database
• Added table to support PowerAutomate
• Added constraints to ensure VocabularyKeyDefinition cannot have a Text DataType without having Keyword Storage
• Added column on DataSet to track when content was cleared
• Added lastSwitchedToDefaultMode column to filter receipts on the DataSets table
• Change annotation name length on the Annotations table
• Added field to track data receiving on the DataSets table

Packages

For this release, kindly utilize the precise versions listed below for the following packages

Connectors

Name	Version
CluedIn.Connector.AzureDataLake	4.4.0
CluedIn.Connector.AzureDedicatedSqlPool	4.0.2
CluedIn.Connector.AzureEventHub	4.0.1
CluedIn.Connector.AzureServiceBus	4.0.2
CluedIn.Connector.Http	4.0.0
CluedIn.Connector.SqlServer	4.1.1
CluedIn.PowerApps	4.4.0
CluedIn.Connector.Dataverse	4.4.0
CluedIn.Connector.OneLake	4.4.0

Enrichers

Name	Version
CluedIn.ExternalSearch.Providers.DuckDuckGo.Provider	4.4.0
CluedIn.ExternalSearch.Providers.PermId.Provider	4.4.0
CluedIn.ExternalSearch.Providers.Web	4.4.0
CluedIn.Provider.ExternalSearch.Bregg	4.4.0
CluedIn.Provider.ExternalSearch.ClearBit	4.4.0
CluedIn.Provider.ExternalSearch.CompanyHouse	4.4.0
CluedIn.Provider.ExternalSearch.CVR	4.4.0
CluedIn.Provider.ExternalSearch.Gleif	4.4.0
CluedIn.Provider.ExternalSearch.GoogleMaps	4.4.0
CluedIn.Provider.ExternalSearch.KnowledgeGraph	4.4.0
CluedIn.Provider.ExternalSearch.Libpostal	4.4.0
CluedIn.Provider.ExternalSearch.OpenCorporates	4.4.0
CluedIn.Provider.ExternalSearch.Providers.VatLayer	4.4.0
CluedIn.ExternalSearch.Providers.PermId.Provider	4.4.0

Crawlers

Name	Version
CluedIn.Crawling.MasterDataServices	4.3.0
CluedIn.Purview	4.4.0

Other

Name	Version
CluedIn.Vocabularies.CommonDataModel	4.4.0
CluedIn.EventHub	4.4.0
CluedIn.EnterpriseFlows.PowerAutomate	4.4.0

Tags

Controller

Docker Image	Tags
cluedin/controller	2024.12.00, 2024.12, 4.4, 4.4.0, 4.4.0_103483

Gql

Docker Image	Tags
cluedin/cluedin-ui-gql	2024.12.00, 2024.12, 4.4, 4.4.0, 4.4.0_103482

Microservices

Docker Image	Tags
cluedin/data-source	2024.12.00, 2024.12, 4.4, 4.4.0, 4.4.0_103480
cluedin/data-source-processing	2024.12.00, 2024.12, 4.4, 4.4.0, 4.4.0_103480
cluedin/data-source	2024.12.00, 2024.12, 4.4, 4.4.0, 4.4.0_103480
cluedin/data-source-processing	2024.12.00, 2024.12, 4.4, 4.4.0, 4.4.0_103480
cluedin/data-source-submitter	2024.12.00, 2024.12, 4.4, 4.4.0, 4.4.0_103480
cluedin/data-source	2024.12.00, 2024.12, 4.4, 4.4.0, 4.4.0_103480

Runtime

Docker Image	Tags
cluedin/neo4j	2024.12.00, 2024.12, 4.4, 4.4.0, 4.4.0_103484
cluedin/openrefine	2024.12.00, 2024.12, 4.4, 4.4.0, 4.4.0_103484

Server

Docker Image	Tags
cluedin/cluedin-server	2024.12.00, 2024.12, 4.4, 4.4.0, 4.4.0_103490, 4.4.0_103490-alpine, 4.4.0-alpine, 4.4-alpine
cluedin/cluedin-server	2024.12.00, 2024.12, 4.4.0_103490-ubuntu, 4.4.0-ubuntu, 4.4-ubuntu
cluedin/nuget-installer	2024.12.00, 2024.12, 4.4, 4.4.0, 4.4.0_103490, 4.4.0_103490-alpine, 4.4.0-alpine, 4.4-alpine
cluedin/nuget-installer	2024.12.00, 2024.12, 4.4.0_103490-ubuntu, 4.4.0-ubuntu, 4.4-ubuntu

Ui

Docker Image	Tags
cluedin/ui	2024.12.00, 2024.12, 4.4, 4.4.0, 4.4.0_103481