The Power of Patterns – Part 2

Check out The Power of Patterns and read on to discover the second type:

Patterns of similarity when people make decisions about duplicate records

The second type of patterns about data (let’s call these type 2 patterns) occurs as humans look at multiple records (an illustration follows) trying to decide, for instance, if they are duplicate records. Without realizing it, they are sensing and evaluating the patterns of data similarities for each of the attributes, and sets of attributes, while of course automatically taking into account any misfielding and differences in the data. What they don’t calculate or really have any knowledge of is the mathematical similarity score that could be calculated for each set of attributes.

Now imagine we get a domain expert to label representative sets of record pairs as being duplicate or not-duplicate, yes or no, true or false, etc. We could build a model of the sets (vectors) of attribute similarity scores that lead the domain expert to their particular conclusions. With enough training, the model would know when it was ready to take over reaching the same conclusions (to converge) as the domain expert would reach. At this point, the model is ready to run and reach highly accurate conclusions based on the prior training by domain experts. Providing more accurate results means that higher levels of automation can be achieved in that particular business process.


Looking at the data and the attribute similarity scores you’ll notice a few keys points: misfieldings are taken into account, differences in spelling are recognized and scored appropriately, and also that the use of nicknames (or any other semantic equivalents for that matter) is supported. 

Learn more here!

Previous articleEnterprise Social Improves Employee Motivation & Engagement – HR Tells All
Next articleGuidelines for Choosing an Enterprise Social Network
Dave Chamberlain has over thirty years of experience in all aspects of software, including marketing, business development, and creating and fostering business partnerships. Dave has worked for TIBCO for the past two years as chief “evangelist” for TIBCO Patterns. Dave brought his considerable ability to develop and articulate the business value of new technologies to TIBCO when Netrics, a software firm specializing in data matching, was acquired. Before Netrics, Dave spent over 10 years at IBM, where he led the IBM WebSphere SOA initiative. He started his software career with Computer Associates in the UK where he was responsible for their mainframe DBMS group.