Releases

Release 2024.07.02-beta

Notable Changes

Removal of legacy systems

We have removed the deprecated Prepare, Documentation and Old Search since there functionality is now incorporated elsewhere in the system

Deduplication Filtering

We now allow you to order the Matches table on name, group size or status and we have added the ability to filter by the groups size.

Release Notes

CluedIn

Features

• Improved the final stage of the job system such that it completes significantly faster
• Expose billable record count, golden record count and CPU usage
• Persist deduplication group states for accepted and rejected groups when discarding and re-generating groups
• New connection string StreamCache to support AzureDataLake connector buffering

Fixes

• Concurrency handling issue during processing command execution
• Creation of indexes failing when Feature.Neo4.CreateIndexes is set to true
• Deleting a stream does not remove queues in RabbitMQ
• Entity merge & split edge processing does not work correctly in some cases when edges exist from/to a shadow entity
• Deduplication project can get stuck, if cancelling under certain conditions
• Circular reference can be created when deleting or adding a vocabulary key that has been mapped
• Certain date formats do not work in the ContainsOperator
• Vocabulary key usage can report duplicate values
• Stream ingestion queues could grow immensely due to retrying of failed messages
• History changeset does not always show the author when it is available
• Global metrics do not work under certain conditions
• Generating rules from clean with long names causes rule generation to fail

CluedIn.MicroServices

Features

• Processing logs can be purged
• Improved recovery from missing RabbitMQ connections

Fixes

• New fields that are added to a mapping are not displayed on the subsequent forms
• Fields containing column names that have characters that are substituted can cause data to merge into the same vocabulary key
• On the preview screen adding a column can restore previously deleted columns
• When uploading a CSV without a header, one record would be skipped
• Processing fails when there are 2 or more strict edges
• Cannot remove records after the data set was processed under certain conditions
• Data ingestion stops working after adding a table from a Postgres database
• Load more does not work if the first column on the preview tab is a unique identifier
• Changing an ingestion endpoint to bridge mode can block data being received by the endpoint
• Errors when searching with insufficient permissions

CluedIn.UI.Gql

Features

• Remove the validate dataset feature flag
• Moved Export golden records to a development feature flag

CluedIn.UI

Features

• Export your search results to a file
• Can add additional columns to a data set:
  1. Stored column - manually populated
  2. Computed column - dynamically populated
• New Number of billable records and Number of golden records tiles on the dashboard
• Improved messages and logs when sending invalid JSON to ingestion endpoints
• Added a new domain to the data set log filters
• Purge processing logs
• Deduplication projects use the rule builder for better filtering
• Rule automatic generation checkbox is now on by default on clean projects
• Processing pipeline in the engine room now displays all currently processing data sets
• Additional information warning which claims have special privileges when editing a role
• Improved the create hierarchy from entity process
• Removed the validation tab on a data set
• Included a hint to suggest the format of a duration type
• Improve formatting of the total in the history tab
• Allow all column types to be sorted in search
• Display a message when editing a role to inform the user of any special data access rules
• Removed deprecated Prepare projects
• Removed deprecated Old Search
• New badge to show if a deduplication project generated the matches successfully or if it was cancelled or failed part way through the generation process
• Filter deduplication project groups by number of matches
• Can order the deduplication groups table
• Display the count of groups found whilst the deduplication results are being generated
• Information message added into the streams Preview Condition/Data tabs to inform the user the data they see might different to the complete data that is streamed due to the security configuration withing CluedIn and the data they have access to view
• Disable the ability to switch to bridge/default mode and change it to auto submit if the data set is currently processing

Fixes

• Paging does not work on the export files list
• Incorrect maximum length validation for vocabulary keys
• The grouped relations panel shows more edges than it should under certain conditions
• We are not trimming spaces when searching for target records in the add edge panel on the relations tab
• Reset filters button in the relations tab is active when it should not be
• Add property form on the relations tab keeps the previously entered values if you close the form
• On the relations tab the add property and cancel buttons are not visible under certain conditions
• Lookup values are not being displayed when adding a property on the entity properties tab
• Links to vocabulary keys that cannot be found are displayed on the entity history tab
• Archived data sets incorrectly displaying monitoring stats
• Users with incorrect claims can see buttons they cannot utilise whilst working with data sets
• Long names are not displayed correctly on the relations tab
• Incorrect casing validation when adding properties to an edge on the relations tab
• Long data set names are not being properly truncated
• A misleading message is displayed when you have removed all data sets from a data source
• Incorrect origin validation when selecting an existing origin
• The action is not displayed on the reason panel in quarantine
• A warning is missing when a field is ignored whilst mapping but is set to be used as an entity code
• Broken icon image on the relations tab under certain conditions
• Creating a vocabulary that already exists whilst using the select vocabulary list during mapping results in an unhelpful error message
• Spaces are not allowed in the remove line breaks and replace actions
• Codes are not displayed during mapping but are submitted on the clue
• When going to deduplication from the management tile or left side panel, we get the incorrect breadcrumb in the header
• Unable to select one table from the list for Postgres database
• Error after adding a second property rule in a dataset
• Value field disappears if you change the action to Set Vocabulary key or Set Record property value when editing a pre process rule
• Tags show the incorrect icon in search
• Entity types created through clues lack a display name and icon
• Bridge mode description displays incorrectly
• Incorrect tooltip when you reach the maximum number of columns that can be added to a view
• Incorrect text when adding an edge
• Incorrect text in the validation of a vocabulary key name when adding a vocabulary key
• Missing claim access requirement information when accessing an entity
• Error when hierarchy has a node that has too many children
• User with only informed claims can remove entity code or unmerge entities on the topology tab
• Deleting a term does not update the list of terms when returned to the list view
• ”-“ value is sent when selecting all in the resolve conflicts stage of a deduplication project if there are missing vocabulary keys
• Incorrect RACI permission tooltip for profiling in governance
• Columns added in search are not included when exporting as csv or json
• The entity layouts page can show an error if the user has the incorrect claims
• Rules with invalidated or unsaved filters can be activated
• Arrows to change the month/year are not visible on date picker
• The dataset logs panel is much wider when there are no clues to display

Runtime-Environment

Features

• FileExports table for generating files based on the search
• Extra fields in DataSet table
• Added tables and custom table types, needed for persisting and retrieving previous deduplication group evaluations
• Added columns to store the date and user that purged the logs
• New StreamsCache data base to allow for bulk stream updates
• New table for DataProtectionKeys
• Ensure Vocab Keys cannot map to themselves in VocabularyKeyDefinition table
• Dropped the DeduplicationProjectEntityType table

Packages

For this release, kindly utilize the precise versions listed below for the following packages

Connectors

Name	Version
CluedIn.Connector.AzureDataLake	4.3.0
CluedIn.Connector.AzureDedicatedSqlPool	4.0.0
CluedIn.Connector.AzureEventHub	4.0.0
CluedIn.Connector.AzureServiceBus	4.0.0
CluedIn.Connector.Http	4.0.0
CluedIn.Connector.SqlServer	4.1.0
CluedIn.PowerApps	4.3.0
CluedIn.Connector.Dataverse	4.3.0
CluedIn.Connector.OneLake	4.3.0

Enrichers

Name	Version
CluedIn.ExternalSearch.Providers.DuckDuckGo.Provider	4.0.0
CluedIn.ExternalSearch.Providers.PermId.Provider	4.0.0
CluedIn.ExternalSearch.Providers.Web	4.1.0
CluedIn.Provider.ExternalSearch.Bregg	4.0.0
CluedIn.Provider.ExternalSearch.ClearBit	4.1.0
CluedIn.Provider.ExternalSearch.CompanyHouse	4.0.0
CluedIn.Provider.ExternalSearch.CVR	4.1.0
CluedIn.Provider.ExternalSearch.Gleif	4.0.0
CluedIn.Provider.ExternalSearch.GoogleMaps	4.1.0
CluedIn.Provider.ExternalSearch.KnowledgeGraph	4.0.0
CluedIn.Provider.ExternalSearch.Libpostal	4.1.0
CluedIn.Provider.ExternalSearch.OpenCorporates	4.0.0
CluedIn.Provider.ExternalSearch.Providers.VatLayer	4.0.0

Crawlers

Name	Version
CluedIn.Crawling.MasterDataServices	4.0.0
CluedIn.Purview	4.3.0

Other

Name	Version
CluedIn.Vocabularies.CommonDataModel	4.3.0
CluedIn.EventHub	4.3.0

Tags

Controller

Docker Image	Tags
cluedin/controller	2024.07.02-beta, 2024.07-beta, 4.3, 4.3.0, 4.3.0_91677

Gql

Docker Image	Tags
cluedin/cluedin-ui-gql	2024.07.02-beta, 2024.07-beta, 4.3, 4.3.0, 4.3.0_91676

Microservices

Docker Image	Tags
cluedin/data-source	2024.07.02-beta, 2024.07-beta, 4.3, 4.3.0, 4.3.0_91673
cluedin/data-source-processing	2024.07.02-beta, 2024.07-beta, 4.3, 4.3.0, 4.3.0_91673
cluedin/data-source	2024.07.02-beta, 2024.07-beta, 4.3, 4.3.0, 4.3.0_91673
cluedin/data-source-processing	2024.07.02-beta, 2024.07-beta, 4.3, 4.3.0, 4.3.0_91673
cluedin/data-source-submitter	2024.07.02-beta, 2024.07-beta, 4.3, 4.3.0, 4.3.0_91673
cluedin/data-source	2024.07.02-beta, 2024.07-beta, 4.3, 4.3.0, 4.3.0_91673

Runtime

Docker Image	Tags
cluedin/neo4j	2024.07.02-beta, 2024.07-beta, 4.3, 4.3.0, 4.3.0_91835
cluedin/openrefine	2024.07.02-beta, 2024.07-beta, 4.3, 4.3.0, 4.3.0_91835

Server

Docker Image	Tags
cluedin/cluedin-server	2024.07.02-beta, 2024.07-beta, 4.3, 4.3.0, 4.3.0_91680, 4.3.0_91680-alpine, 4.3.0-alpine, 4.3-alpine
cluedin/cluedin-server	2024.07.02-beta, 2024.07-beta, 4.3.0_91680-ubuntu, 4.3.0-ubuntu, 4.3-ubuntu
cluedin/nuget-installer	2024.07.02-beta, 2024.07-beta, 4.3, 4.3.0, 4.3.0_91680, 4.3.0_91680-alpine, 4.3.0-alpine, 4.3-alpine
cluedin/nuget-installer	2024.07.02-beta, 2024.07-beta, 4.3.0_91680-ubuntu, 4.3.0-ubuntu, 4.3-ubuntu

Ui

Docker Image	Tags
cluedin/ui	2024.07.02-beta, 2024.07-beta, 4.3, 4.3.0, 4.3.0_91675