CloverDataReader

Not available in Community Designer

Short Description

CloverDataReader reads data stored in our internal binary Clover data format files. It can also read data from compressed files, or a dictionary.

ComponentData sourceInput portsOutput portsEach to all outputsDifferent to different outputsTransformationTransf. req.JavaCTLAuto-propagated metadata
CloverDataReaderclover binary file11-n
yes
no
no
no
no
no
no

Ports

Port typeNumberRequiredDescriptionMetadata
Input0
no
For Input Port Reading correct data recordsInclude specific byte/cbyte field
Output0
yes
For correct data recordsAny
1-n
no
For correct data recordsOutput 0

Metadata

CloverDataReader does not propagate metadata.

CloverDataReader has no metadata template, but it can extract metadata from Clover file and propagate it forward as it would have a template. (Available since 4.1.0-M1.)

Metadata on the input port has to include a byte, cbyte or string field.

Metadata on the output port has to be the same as metadata of data from the file.

Metadata can use Autofilling Functions.

CloverDataReader Attributes

AttributeReqDescriptionPossible values
Basic
File URLyes

An attribute specifying what data source(s) will be read (Clover data file, input port, dictionary). See Supported File URL Formats for Readers.

 
Advanced
Number of skipped records 

Number of records to be skipped. See Selecting Input Records.

0-N
Max number of records 

Maximum number of records to be read. See Selecting Input Records.

0-N
Number of skipped records per source Skip the first n records of each file.0-N
Max number of records per source Reads maximally n records from each file.0-N
Deprecated
Index file URL 

The name of an index file, including the path. If not specified, all records are read.

 
Start record 

Has exclusive meaning: Last record before the first that is already read. Has lower priority than Number of skipped records.

0 (default) | 1-n
Final record  Has inclusive meaning: Last record to be read. Has lower priority than Max number of records. all (default) | 1-n

Examples

Reading a Clover Data File
Omitting Leading Records
Omitting Leading Records of Each File
Reading at most n Records in Total
Reading at most n Records per File

Reading a Clover Data File

Read all records from the Clover data file.

Solution

Set up the File URL attribute.

AttributeValue
File URL${DATAIN_DIR}/my-clover-file.cdf

CloverDataReader will read all the records from the file(s).

Omitting Leading Records

You have two Clover data files. First 3 records contain unimportant data and should not be read. The unimportant records are in the first file. (The records have been sorted and partitioned for example.)

greengrocers1.cdf

bread
honey
raisins
pears
plums

greengrocers2.cdf

carrot
peas
radish

Solution

Set up the File URL and Number of skipped records attributes.

AttributeValue
File URL${DATAIN_DIR}/greengrocers1.cdf;${DATAIN_DIR}/greengrocers2.cdf
Number of skipped records3

CloverDataReader reads the following items:

pears
plums
carrot
peas
radish

Omitting Leading Records of Each File

There are two Clover data files: list1.cdf and list2.cdf. Each file starts with one record to be omitted.

list1.cdf

Goods
cardigan
shirt
trousers

list2.cdf

Goods
shoes
sox

Solution

Set up the File URL and Number of skipped records per source attributes.

AttributeValue
File URL${DATAIN_DIR}/list1.cdf;${DATAIN_DIR}/list2.cdf
Number of skipped records per source1

CloverDataReader sends the following records to the output:

cardigan
shirt
trousers
shoes
sox

Reading at most n Records in Total

You have three files stationery1.cdf, stationery2.cdf and stationery3.cdf and you need to read six records in total from all files.

stationery1.cdf

pen
pencil
marker
paintbrush

stationery2.cdf

ink
water colors
oil colors

stationery3.cdf

notebook
coloring book

Solution

Set up the File URL and Max number of records attributes.

AttributeValue
File URL${DATAIN_DIR}/stationery1.cdf;${DATAIN_DIR}/stationery2.cdf;${DATAIN_DIR}/stationery3.cdf
Max number of records6

CloverDataReader sends all 4 records from stationery1.cdf and 2 of 3 records from stationery2.cdf to the output port. No record from the file stationery3.cdf is sent the output port as the limit has been reached already.

pen
pencil
marker
paintbrush
ink
water colors

Reading at most n Records per File

You have three Clover data files (stationery[1-3].cdf) from the previous example. Read at most 3 records from each file.

Solution

Set up the File URL and Max number of records per source attributes.

AttributeValue
File URL${DATAIN_DIR}/stationery1.cdf;${DATAIN_DIR}/stationery2.cdf;${DATAIN_DIR}/stationery3.cdf
Max number of records per source3

CloverDataReader reads 3 records from stationery1.cdf, 3 records from stationery2.cdf and 2 of 2 records from stationery3.cdf.

pen
pencil
marker
ink
water colors
oil colors
notebook
coloring book

Compatibility

2.9

Since CloverETL 2.9, CloverDataWriter also writes a header to output files with the version number. For this reason, CloverDataReader expects that files in Clover binary format contain such a header with the version number. CloverDataReader 2.9 cannot read files written by older versions of CloverETL nor these older versions can read data written by CloverDataWriter 2.9.

4.0

The internal structure of the zip archive has changed. Graphs that rely on the structure will stop working. Graphs that use plain zip file URL without internal entry specification are not affected: CloverDataReader with File URL zip:(${DATAOUT_DIR}/employees.zip)#DATA/employees will need to be fixed to use zip:(${DATAOUT_DIR}/employees.zip) instead.

4.1.0-M1

Since 4.1.0-M1, CloverDataReader can extract metadata template from Clover file. It can be seen as a metadata template corresponding to the file.

4.4.0-M2

Since 4.4.0-M2, CloverDataReader can read from input port just from byte or cbyte field.

See also

CloverDataWriter
Common Properties of Components
Specific Attribute Types
Common Properties of Readers
Readers Comparison