Releases

Release 2023.07.00

Notable Changes

WebApi removal

We have removed the WebApi container freeing up some resources to be allocated to other containers. All WebApi functionality is now supported within the Server itself.

RACI and Role Changes

For greater flexibility we have improved the RACI authorization flow within CluedIn. This allows us to remove the requirement of specific roles such as OrganizationAdmin as part of the authorization flow.

This is a fundamental change and may require you to check your current claims configuration.

In addition to the above changes ALL references to the IsAdmin flag have been removed and the column has been dropped from the AspNetUser table.

To facilitate the move away from roles in the authorization flow, we added additional claims to replace the OrganizationAdmin responsibilities. They are as follows

Claim Name	Description
admin.datamanagement	Managing data related endpoints
admin.providermanagement	Managing provider related endpoints
admin.messagingmanagement	Managing messaging related endpoints
admin.tenantmanagement	Managing traditional administration endpoints like configuration or creating organizations etc
admin.tokenmanagement	Managing token related endpoints

Please note this could potentially cause some unexpected side effects. Please make sure that you have the correct claims set for your roles so you do not encounter any unexpected behavior.

There are new prompts and warnings throughout the entire site that will inform users when they do not have sufficient claims to access specific features or perform certain actions.

Performance and Resiliency Improvements

Overhauled the entire processing pipeline to significantly increase throughput of data ingestion.

Improved the availability and scalability of internal services resulting in less downtime.

Updated all internal service resource specifications to improve pod efficiency.

Provided an option to switch to premium disks to further improve data processing throughput and increase IOPs but this would occur additional costs

Added priority classes for better pod eviction and recovery policies.

Entity History Improvements

Ability to filter entity history by source and sort entity history by modified/created/discovery/sort date.

Rule Action UI

Added rich controls to the rule action panel, providing a consistent user experience with rules.

Added validation to prevent incompatible action configurations.

Properties on Edges Whilst Mapping

Added the ability to add properties to edges whilst mapping data.

Annotation

Features

• Support for creating and submitting properties on edges

Fixes

• Allow manual edits to not pass a ProviderDefinitionId

Data Source

Features

• Support for creating properties on edges
• Support for monitoring the submission of clues
• Improve performance when uploading files
• Improve performance when using ingestion endpoints

Fixes

• Manual data entry project fields are now displayed in the order they were created

Gql

Features

• Show estimates of total nodes to load in Hierarchy Builder’s Load Entities panel
• Support for refreshing hierarchy data
• Support for rule action input controls
• Support filtering entity history by source and sorting by date
• Support for creating properties on edges
• Support for monitoring the submission of clues
• Allow the creation of a vocabulary key during manual data entry form setup
• Support the removal of WebApi

Fixes

• Create entity type whilst adding an edge relationships

Runtime

Features

• Support for improved file upload stability
• Support the new edge properties panel
• New admin claims have been added
• Support the archiving of unwanted manual data entry form fields

Fixes

• Add indexes to Clean tables
• Deletion of an organization should remove page templates

Server

Features

• Added a kind attribute to the rule actions, so that the UI can display the correct UI input control
• Allow the use of EntityCodes in Search and Rule Builder
• Open AI Action Rules have specified input control types with friendly names
• Allow filtering and sorting of entity history on source name and date respectively
• Ability to only undo deduplication merges, if the merged entity has been updated after merge took place
• You can select additional meta-data properties in Rule/Query Builder
• Move responsibility of CluedInApp endpoints to Server
• Purge dead-letter queue when we reprocess or alter a Stream
• Enhance entity split execution, making it more reliable, safe and robust
• Organization settings toggle for config setting ClueProcessing.ContentDataExtraction.Enabled
• Validate the entity type when created whilst configuring a rule action
• Improve Clean Project Generation and commit performance
• We have removed the dependency on Roles and any authorization is now carried out using our claims based system. As such we have added a few new claims to cover the additional administrative functions.
• Properties of actions on rules now only show valid combinations

• Options to override sql connection timeouts for health checks to avoid blocking health checks for a long time.

  Configuration Key	Default Value
  DataStores.HealthCheck.SqlConnectTimeoutSeconds	10
  DataStores.HealthCheck.SqlCommandTimeoutSeconds	10

• Deduplication projects allow deduplication across multiple entity types
• Improve performance when adding Vocabulary Keys
• Legacy and LegacyFuzzyEntityMatcherV2 matchers are now disabled by default

Fixes

• Deduplication project could get stuck whilst changing state
• Potential match lookups in deduplication projects could sometimes cause search queries to fail because of too many normalization value variations
• Show estimates and improve loading time on load entities panel
• Some exceptions were not being treated as transient errors
• Incorrect property name was being sent to the connector
• Possible concurrency issue in AuthenticationRepository
• Entity Framework error thrown when calling Token endpoint with incorrect password
• Rule properties lookup throws and exception when null
• Job System progress counters would not update correctly
• Streams Data Preview shows all records when using entity code property in condition
• The action regex expression that had the property isCaseSensitive set to true ignored case sensitivity
• Rules builder using origin entity code throwing an expression tree error
• Prevent unwanted registration of users as admins
• Edge properties are not being created correctly in some cases during edge processing
• Street address normalization function not normalizing values correctly in some cases
• In some cases /Person entity type is not being set correctly when merging data into the golden record
• Properties that are stored in a list (e.g. tags) do not deserialize correctly in streams
• Entity type does not revert when un-merging a deduplication project
• Can add self referencing edges during processing
• In some cases entity does not get inserted into Elastic Search or the Blob store when processing fails
• Delete value rule action in come certain circumstances would not remove the value
• Data parts could be not compacted away due to temporal edges
• Using PropertyCount in glossary rule throws error
• An error was thrown when creating a stream with a condition that used an empty glossary
• Streams can send data that has not changed when in event mode
• SSO role mapping can map to the incorrect role
• Cloning a hierarchy project does not set the modified date

UI

Features

• Added new functionality to the preview data set list. This includes adding pagination, column options, full screen mode and column resizing
• Allow changing column options on the quarantine page
• Show estimate for loading entities in hierarchy
• Add a project properties panel in the hierarchy builder
• Allow the user to un-merge deduplication groups based on if the rules no longer apply
• Search pagination changes to Load More if there are more than 10k results
• Enable the addition of properties on edges during mapping
• Change security model to only support the RACI model and stop relying on roles
• Improve latency of status checks
• Monitoring tab on the data set to view the submission of clues

Fixes

• Added confirmation dialog on delete rule action item, added validation for text inputs inside rule action sidebar
• Select all in lists when there is only one item
• Added status message to inform the user that there is data already in the queue on the data set preview page
• System health check displaying incorrect status when GQL was unavailable
• In certain circumstances editing of entity types could fail

Tags

Annotation

Docker Image	Tags
cluedin/cluedin-micro-annotation	2023.07.00, 2023.07, 3.7, 3.7.0, 3.7.0_62588

Clean

Docker Image	Tags
cluedin/cluedin-micro-clean	2023.07.00, 2023.07, 3.7, 3.7.0, 3.7.0_62527

Controller

Docker Image	Tags
cluedin/controller	2023.07.00, 2023.07, 3.7, 3.7.0, 3.7.0_62530

Datasource

Docker Image	Tags
cluedin/cluedin-micro-datasource	2023.07.00, 2023.07, 3.7, 3.7.0, 3.7.0_62554

Docs

Docker Image	Tags
cluedin/cluedin-micro-documentation	2023.07.00, 2023.07, 3.7, 3.7.0, 3.7.0_62529

Gql

Docker Image	Tags
cluedin/cluedin-ui-gql	2023.07.00, 2023.07, 3.7, 3.7.0, 3.7.0_62556

Runtime

Docker Image	Tags
cluedin/neo4j	2023.07.00, 2023.07, 3.7, 3.7.0, 3.7.0_62531
cluedin/openrefine	2023.07.00, 2023.07, 3.7, 3.7.0, 3.7.0_62531
cluedin/sqlserver	2023.07.00, 2023.07, 3.7, 3.7.0, 3.7.0_62531

Server

Docker Image	Tags
cluedin/cluedin-server	2023.07.00, 2023.07, 3.7, 3.7.1, 3.7.1_62683, 3.7.1_62683-alpine, 3.7.1-alpine, 3.7-alpine
cluedin/cluedin-server	2023.07.00, 2023.07, 3.7.1_62683-ubuntu, 3.7.1-ubuntu, 3.7-ubuntu
cluedin/nuget-installer	2023.07.00, 2023.07, 3.7, 3.7.1, 3.7.1_62683, 3.7.1_62683-alpine, 3.7.1-alpine, 3.7-alpine
cluedin/nuget-installer	2023.07.00, 2023.07, 3.7.1_62683-ubuntu, 3.7.1-ubuntu, 3.7-ubuntu

Submitter

Docker Image	Tags
cluedin/cluedin-micro-submitter-node	2023.07.00, 2023.07, 3.7, 3.7.0, 3.7.0_62555

Ui

Docker Image	Tags
cluedin/ui	2023.07.00, 2023.07, 3.7, 3.7.0, 3.7.0_62528