Releases

Release 3.3.0

Please follow the Upgrade Guide to update existing CluedIn installations.

Annotation

Docker Image	Tags
cluedin/cluedin-micro-annotation	3.3, 3.3.0, 3.3.0_35534

Fixes

• Supporting release for platform
• Move database table creation scripts into the runtime (AB#3533)

App

Docker Image	Tags
cluedin/app	3.3, 3.3.0, 3.3.0_35538

Fixes

• The environment variable EMAILSETTINGS__ENABLESSL can now be set to false to disable ssl connections to smtp clients
• Adding roles to user now uses the legacy endpoints to allow for multi tenant roles
• Implement multi tenant users with matching emails (AB#5540)

Clean

Docker Image	Tags
cluedin/cluedin-micro-clean	3.3, 3.3.0, 3.3.0_35535

Features

• Supporting release for platform
• Move database table creation scripts into the runtime (AB#3534)

Controller

Docker Image	Tags
cluedin/controller	3.3, 3.3.0, 3.3.0_35543

Features

Added support for sendMail CRD type to send simple on-off emails based on a configmap.

apiVersion: api.cluedin.com/v1
kind: SendMail
metadata:
  name: 'sendmail-example'
spec:
  emailSecret: "cluedin-email"
  emailTemplateResource: "email-template"

apiVersion: v1
kind: ConfigMap
metadata:
  name: email-template
data:
  subject: "WELCOME to CLUEDIN!"
  body: "Some talk about things.."
  to: "stp@cluedin.com"
  from: "no-reply@cluedin.com"

Datasource

Docker Image	Tags
cluedin/cluedin-micro-datasource	3.3, 3.3.0, 3.3.0_35536

Features

• Supporting release for platform
• Move database table creation scripts into the runtime (AB#3532)

Gql

Docker Image	Tags
cluedin/cluedin-ui-gql	3.3, 3.3.0, 3.3.0_35537

Features

• Udpated to Node.js 16
• Add config for engine room charts (#11359)
• Move database table creation scripts into the runtime (AB#3532)

Fixes

• Engine room chart name (AB#7305)

Runtime

Docker Image	Tags
cluedin/neo4j	3.3, 3.3.0, 3.3.0_35539
cluedin/neo4j-init	3.3, 3.3.0, 3.3.0_35539
cluedin/openrefine	3.3, 3.3.0, 3.3.0_35539
cluedin/sqlserver	3.3, 3.3.0, 3.3.0_35539
cluedin/sqlserver-init	3.3, 3.3.0, 3.3.0_35539

Features

• Added sqlserver-init container which will push the scripts to an already running instance of SQL Server.
  • Parameters:
    • MSSQL_HOST - SQL Server Host ( Default : localhost )
    • MSSQL_PORT - SQL Server Host Port ( Default : 1433 )
    • MSSQL_USER - SQL Server User ( Default : sa )
    • MSSQL_PASSWORD - SQL Server Password ( Default: yourStrong(!)Password )
    • MSSQL_DACPACS - SQL Scripts directory ( Default: ./dacpacs )
    • SQLCMD - SQLCMD exexcutable location ( Default: sqlcmd )
    • SQLPACKAGE - SQLPACKAGE exexcutable location ( Default: sqlpackage )
    • MSSQL_CLIENTUSER - Non-admin user ( Default: `` - Skips creation if blank)
    • MSSQL_CLIENTUSER_PASSWORD - Non-admin user password ( Default: `` - Skips creation if blank)
• Added schema for Dynamic Vocabularies
• Extend processing rules to support an ingestion rule (AB#3916)
• Add table scripts for all micro services
• Add tables to track dataset endpoint ingressions
• Add upgrade notes

Server

Docker Image	Tags
cluedin/cluedin-server	3.3, 3.3-alpine, 3.3.0, 3.3.0_35562, 3.3.0_35562-alpine, 3.3.0-alpine
cluedin/cluedin-server	3.3-ubuntu, 3.3.0_31367-ubuntu, 3.3.0-ubuntu
cluedin/nuget-installer	3.3, 3.3-alpine, 3.3.0, 3.3.0_35562, 3.3.0_35562-alpine, 3.3.0-alpine
cluedin/nuget-installer	3.3-ubuntu, 3.3.0_35562-ubuntu, 3.3.0-ubuntu

Fixes

• Resolved an issue where changing categories on a glossary term would not save the changes
• Resolved issue where RemoveEvent’s OriginHost was not deseralized correctly

• Resolved issue where crawlers won’t run on servers with IPv6 Addresses

• RACI Permission Levels and Configurable Roles
• RACI Approvals mechanism
• Tasks for upcoming Dashboard
• Role Request Tasks
• Audit Log
• IN Operator

• Dynamic Vocab

• Fixed merge entities user action to correctly persist user preserved property values in separate version branch within the entity
• Fixed merge entities processing event handler in certain conditions not creating mesh commands
• Fixed data part InputSource not being serialized correctly if attribute have been manually set on the data part instead of using the InputSource property
• Added check in entity blob datastore to check if the entity being saved is a shallow loaded entity This is to prevent a shallow loaded entity overwriting entire entity when Update is called on the data store. This behaviour can be turned off using DataStore.EntityBlob.OnUpdate.VerifyFullEntity
• Fixed WorkflowFinishedEvent in some cases would have the wrong NoChangesMade value
• Fixed unhandled exceptions in PhoneNumberEx when quering carrier names
• Fixed incorrectly defined Object Type as VocabularyKey when submitting changes for approval
• Not able to delete the organisation (AB#6768)
• Role names should be in alphabetical order (AB#5260)
• Rules duplicated in Vocabulary Key Usage tab (AB#6896)
• Creation Date Sorting not working for Rule Builder list (AB#6729)
• Stream data preview now serializes data into friendly values
• Ensure Rule ordering takes Organization into account (AB#7255)
• Fixed move relationship row limit not being respected in graph datastore
• Add configurable MoveRelationships limit for how many will be moved within a batch
• Implement multi tenant users with matching emails (AB#5540)
• Fixed Clean Projects not commiting if the description was left blank (AB#7863)

Configuration:

Key	Default	Notes
DataStores.Graph.MoveRelationships.Limit	10	How many relationships is moved within a single batch

• Fixed Neo4jEntityGraphDataStore.MoveRelationshipsByNodeId runtime database error
• Change organization data shard caching to use 10 min sliding expiration instead of 1 min absolute expiration
• Add configuration keys to be able to configure organization data shard caching policy

Configuration:

Key	Default	Notes
Caching.Organization.DataShard.SlidingExpirationSeconds	600	Sliding expiration in seconds
Caching.Organization.DataShard.AbsoluteExpirationSeconds		Absolute expiration in seconds in the future from current date

• Do not allow update of vocabulary name, if name already exists (AB#7435)

Features

• Added the ability to ‘Clean with history’ when creating a cleaning project.
• Added the ability to disabled the IDataContentExtractors in bulk using ClueProcessing.IDataPartProcessor.Enabled set to false. The PII/Sensitivity detection in CluedIn can be disabled through this. If you do not need to use this feature you will find that disabling it will increase performance in CluedIn.
• You can now force the Global Data Quality Metrics to run through a REST API call instead of waiting the 24 hours for the next job to run.
• All Glossary Searches will now search through the Long and Short Description as well.
• Added a new Search Endpoint to allow for paged term results
• Added a new EntityInfo Endpoint to allow for paged results
• Improved crawling throughput when lots of rules exist
• Normalized depedency tree to enable public nuget feed
• Added User Input Version Branch Evaluator
• Added new endpoint for adding a user to a role without sending the email confirmation
• Rule activation ordering starts from 2(AB#6797)
• Authentication certificate set from environment variables (AB#6011)
• Added ability to upgrade Enrichers to the new Provider Model (AB#7729)

Configuration:

Key	Default	Notes
ClueProcessing.VersionHistory.BranchEvaluator.UserInput.Enabled	True	Turn this feature on or off
ClueProcessing.VersionHistory.BranchEvaluator.UserInput.ExcludedInputSources		Specify semicolon separated data part input sources to ignore
ClueProcessing.VersionHistory.BranchEvaluator.UserInput.ExcludedEntityTypes		Specify semicolon separated entity types to ignore

• Added configuration keys to be able to control how merge entities user action should persist the user preserved values

Configuration:

Key	Default	Notes
UserActions.MergeEntities.ApplyTargetEntity.Properties	True	Apply all property no matter if they changed or not
UserActions.MergeEntities.ApplyTargetEntity.Codes	True	Set all entity codes of the target entity instead of only the origin entity code
UserActions.MergeEntities.ApplyLastChangedBy	True	Set last changed by if available
UserActions.MergeEntities.ApplyChangeVerb	False	Apply change verb
UserActions.MergeEntities.ChangeVerb	Merged	The change verb to apply

• Added the ability to configure metrics settings through the API. This gets stored in the DB but if a configuration file has a different value to the default value that value will override what is stored in the DB.
• Added Enricher Providers (AB#3531)
• Added a new EntityInfo Endpoint to allow for paged results
• Extend processing rules to support an ingestion rule (AB#3916)
• When uploading a file which is mapped to a “new” vocabulary key, the value does not work with the “TERM” search (AB#6108)
• Create an endpoint that returns if Vocabulary keys exists or not (AB#5258)

Submitter

Docker Image	Tags
cluedin/cluedin-micro-submitter-node	3.3, 3.3.0, 3.3.0_35542

UI

Docker Image	Tags
cluedin/ui	3.3, 3.3.0, 3.3.0_35541

Features

• Mapping screens in data sources has been redesigned to offer better visibility on the Mapping of the orignal fields of the data set.
• Auto-mapping will now be able to generate the Vocabulary and Vocabulary keys for you.
• You are now able to create Vocabulary keys during the mapping phase of your Data Set.
• New Data Catalog section under Management, you can know view a list of your Vocabularies and Vocabulary key.
• The list of vocabulary supports filtering by Entity type, Integration and search by name.
• The list of vocabulary keys supports filtering by type, by integrations, by classification and search by name or key.
• Core and custom Vocabularies can now be edited.
• Core and custom Vocabulary Keys can now be edited.
• You can create your own Vocabulary.
• You can create your own Vocabulary Key.
• You can edit Voabularies.
• You can edit Vocabulary keys.
• New Usage Tab under a Vocabulary page to see where the vocabulary is being used.
• Owners Tab under a Vocabulary page to define who is the owner of the vocabulary.
• New Usage Tab under a Vocabulary Key page to see where the vocabulary key is being used.
• All values tab under a Voabulary Key page to see what are the differen values for a given Vocabulary key.
• Lineage Tab under a Vocabulary Key page to see the mapping from source to stream and how the data is flowing.
• You can now add (or change) the mapping of a Vocabulary key.
• You can now rename a vocabulary key.
• A new section called Entity Types has been added in the Management pillar.
• You can now see the list of all Entity types in the application and search them by name or code.
• You configure the Entity Type configuration (icon, displayName…) from multiple places.
• You can now create your own Entity Type
• Under the Entity type page, you have a Data tab showing all the entities that has the Entity type as type.
• Under the Entity type page, you have now a Vocabularies tab to see all the vocabularies are attached to an Entity type.
• Rules now support multiple actions.
• Rules now support sub filters for each action.
• New settings to enable/disable metrics (global or individual) has been added.
• A new pillar called ‘Engine Room’ has been added to the main menu.
• In the Engine Room, you can follow the status of the different main processing stage of the data pipeline.
• In the Engine Room, you can now see high level statistics on the Cluster health.
• In the Engine Room, you can now drill down to some specific services to see basic statistic (Search Db, Relational Db, Graph Db…)
• In the Engine Room, you can now see how the CluedIn instance is being setup in a configuraiton point of view.
• In the Entity page, you have now a tab called “Explaing Log” which helps you debug/understand how the Golden record is being produced.
• Enricher can now be added and edited, if the Enricher supports configuration (like API Token), you will be able to enter them from the UI (it was in config file before).
• Enricher supports permissions the same way as Integrations and Data Sources.
• The UI now has the ability to run in multiple languages.

Fixes

• In duplicates section, only call the Queries endpoint to have faster result on what duplicate queries are installed.
• You can now map multiple Edges for the same property.
• The reports of the processing have been improved to be more accurate (total and progression).