Multivalue Fields

Each metadata field commonly stores only one value (e.g. one integer, one string, one date). However, you can also set one field to carry more values of the same type.

[Note]Note

Multivalue fields is a new feature available as of Clover v. 3.3.

Example 33.4. Example situations when you could take advantage of multivalue fields

  • A record containing an employee's ID, Name and Address. Since employees move from time to time, you might need to keep track of all their addresses, both current and past. Instead of creating new metadata fields each time an employee moves to a new house, you can store a list of all addresses into one field.

  • You are processing an input stream of CSV files, each containing a different column count. Normally, that would imply creating new metadata for each file (each column count). Instead, you can define a generic map in metadata and append fields to it each time they occur.


As implied above, there are two types of structures:

list - is a set containing elements of a given data type (any you want). In source code, lists are marked by the [] brackets. Example:

integer[] list1 = [1, 367, -1, 20, 5, 0, -79]; // a list of integer elements 
boolean[] list2 = [true, false, randomBoolean()]; // a list of three boolean elements
string[] list3; // a just-declared empty list to be filled by strings

map - is a pair of keys and their values. A key is always a string while a value can be any data type - but you cannot mix them (remember a map holds values of the same type). Example:

map[string,date] dateMap; // declaration

// filling the map with values
dateMap["a"] = 2011-01-01;
dateMap["b"] = 2012-12-31;
dateMap["c"] = randomDate(2011-01-01,2012-12-31);

You will find out more about maps and lists if you go to Data Types in CTL2.

[Important]Important

To change a field from single-value to multi-value:

  1. Go to Metadata Editor.

  2. Click a field or create a new one.

  3. In PropertyBasic, switch Container Type either to list or map. (You will see an icon appears next to the field Type in the left hand record pane.)

Lists and Maps Support in Components

An alphabetically sorted list of components which you can use multivalue fields in:

ComponentListMapNote
CloverDataReader
yes
yes
 
CloverDataWriter
yes
yes
 
ParallelSimpleGather

yes

yes

Round robin

yes

no

Merge by key

yes

yes

Simple gather
ParallelPartition

no

no

Ranges

yes

no

Partition key

yes

yes

Partition class
Concatenate
yes
yes
 
DataGenerator
yes
yes
 
DataIntersection

yes

no

 

yes

yes

Map is not a part of key
DBJoin
yes
yes
Map is not a part of key
Dedup

yes

no

 

yes

yes

Map is not a part of key
Denormalizer

yes

no

 

yes

yes

Map is not a part of key
Filter
yes
yes
 
ExtHashJoin

yes

no

 

yes

yes

Map is not a part of key
ExtMergeJoin
yes
yes
Map is not a part of key
ExtSort

yes

no

 

yes

yes

Map is not a part of key.
JavaBeanReader
yes
no 
JavaBeanWriter
yes
no 
JavaMapWriter
yes
yes
 
JSONExtract
yes
no 
JSONReader
yes
no 
JSONWriter
yes
no 
LookupJoin
yes
yes
 
LookupTableReaderWriter
yes
yes
 
Merge

yes

no

 

yes

yes

Map is not a part of key
Normalizer

yes

no

 

yes

yes

Map is not a part of key
Partition

no

no

Ranges

yes

no

Partition key

yes

yes

Partition class
Reformat
yes
yes
 
RelationalJoin
yes
yes
Map is not a part of key
Rollup

yes

no

Sorted input

yes

yes

Sorted input, map not part of key

yes

yes

Unsorted input
SequenceChecker

yes

no

 

yes

yes

Map is not a part of key
SimpleCopy
yes
yes
 
SimpleGather
yes
yes
 
Sleep
yes
yes
 
SortWithinGroups
yes
no 
XMLExtract
yes
no 
XMLReader
yes
no 
XMLWriter
yes
yes
 

At the moment, neither map nor list structures can be extracted as metadata from flat files.

Joining on Lists and Maps (Comparison Rules)

You can specify fields that are lists or maps as Join keys (see Join Types) just like any other fields. The only question is when two maps (lists) equal.

First of all, let us clarify this. A list/map can:

Two maps (lists) are equal if both of them are not null, they have the same data type, element count and all element values (keys-values in maps) are equal.

Two maps (lists) are not equal if either of them is null.

[Important]Important

When comparing two lists, the order of their elements has to match, too. In maps, there is no 'order' of elements and therefore you cannot use them in Sort key.

Example 33.5.  Integer lists which are (not) equal - symbolic notation

[1,2] == [1,2]
[null] != [1,2]
[1] != [1,2]
null != null // two unspecified lists
[null] == [null] // an extra case: lists which are not empty but whose elements are null


Note: Maps are implemented as LinkedHashMap and thus their properties derive from it.