CloverETL Designer
CloverETL Overview
Family of CloverETL Products
CloverETL Designer
CloverETL Engine
CloverETL Server
Getting Additional Information
Integrating CloverETL Designer with CloverETL Server
Creating CloverETL Server project (Basic Principles)
Opening CloverETL Server Projects
Connecting via HTTP
Connecting via HTTPS
Designer has its Own Certificate
Designer does not have its Own Certificate
Connecting via proxy server
Installation Instructions
System Requirements for CloverETL Designer
Related Links
Downloading CloverETL
CloverETL Desktop Edition
CloverETL Desktop Trial Edition
CloverETL Community Edition
Starting CloverETL Designer
Installing Designer as Eclipse plugin
Getting Started
Creating CloverETL Projects
CloverETL Project
CloverETL Server Project
CloverETL Examples Project
Structure of CloverETL Projects
Standard Structure of All CloverETL Projects
Workspace.prm File
Opening the CloverETL Perspective
Appearance of CloverETL Perspective
CloverETL Designer Panes
Graph Editor with Palette of Components
Navigator Pane
Outline Pane
Tabs Pane
Creating CloverETL Graphs
Creating Empty Graphs
Creating a Simple Graph in a Few Simple Steps
Running CloverETL Graphs
Successful Graph Execution
Using the Run Configurations Dialog
Working with CloverETL Designer
Using Cheat Sheets
Common Dialogs
URL File Dialog
Edit Value Dialog
Open Type Dialog
Import
Import CloverETL Projects
Import from CloverETL Server Sandbox
Import Graphs
Import Metadata
Metadata from XSD
Metadata from DDL
Export
Export Graphs
Export Graphs to HTML
Export Metadata to XSD
Export to CloverETL Server Sandbox
Export Image
Advanced Topics
Program and VM Arguments
Example of Setting Up Memory Size
Changing Default CloverETL Settings
Enlarging the Font of Displayed Numbers
Setting and Configuring Java
Setting Java Runtime Environment
Installing Java Development Kit
Graph Elements, Structures and Tools
Components
Edges
What Are the Edges?
Connecting Components by the Edges
Types of Edges
Assigning Metadata to the Edges
Propagating Metadata through the Edges
Colors of the Edges
Debugging the Edges
Enabling Debug
Selecting Debug Data
Viewing Debug Data
Turning Off Debug
Edge Memory Allocation
Metadata
Data Types and Record Types
Data Types in Metadata
Record Types
Data Formats
Data and Time Format
Numeric Format
Boolean Format
String Format
Locale and Locale Sensitivity
Locale
Locale Sensitivity
Autofilling Functions
Internal Metadata
Creating Internal Metadata
Externalizing Internal Metadata
Exporting Internal Metadata
External (Shared) Metadata
Creating External (Shared) Metadata
Linking External (Shared) Metadata
Internalizing External (Shared) Metadata
Creating Metadata
Extracting Metadata from a Flat File
Extracting Metadata from Delimited Files
Extracting Metadata from Fixed Length Files
Extracting Metadata from an XLS(X) File
Extracting Metadata from a Database
Extracting Metadata from a DBase File
Creating Metadata by User
Extracting Metadata from Lotus Notes
Merging existing metadata
Dynamic Metadata
Reading Metadata from Special Sources
Creating Database Table from Metadata and Database Connection
Metadata Editor
Basics of Metadata Editor
Record Pane
Field Name vs. Label vs. Description
Details Pane
Changing and Defining Delimiters
Changing Record Delimiter
Changing Default Delimiter
Defining Non-Default Delimiter for a Field
Editing Metadata in the Source Code
Multivalue Fields
Lists and Maps Support in Components
Joininig on Lists and Maps (Comparison Rules)
Database Connections
Internal Database Connections
Creating Internal Database Connections
Externalizing Internal Database Connections
Exporting Internal Database Connections
External (Shared) Database Connections
Creating External (Shared) Database Connections
Linking External (Shared) Database Connections
Internalizing External (Shared) Database Connections
Database Connection Wizard
Encrypting the Access Password
Browsing Database and Extracting Metadata from Database Tables
Windows Authentication on Microsoft SQL Server
Getting the Native Library
Installation
JMS Connections
Internal JMS Connections
Creating Internal JMS Connections
Externalizing Internal JMS Connections
Exporting Internal JMS Connections
External (Shared) JMS Connections
Creating External (Shared) JMS Connections
Linking External (Shared) JMS Connection
Internalizing External (Shared) JMS Connections
Edit JMS Connection Wizard
Encrypting the Authentication Password
QuickBase Connections
Lotus Connections
Lookup Tables
Internal Lookup Tables
Creating Internal Lookup Tables
Externalizing Internal Lookup Tables
Exporting Internal Lookup Tables
External (Shared) Lookup Tables
Creating External (Shared) Lookup Tables
Linking External (Shared) Lookup Tables
Internalizing External (Shared) Lookup Tables
Types of Lookup Tables
Simple Lookup Table
Database Lookup Table
Range Lookup Table
Persistent Lookup Table
Aspell Lookup Table
Sequences
Internal Sequences
Creating Internal Sequences
Externalizing Internal Sequences
Exporting Internal Sequences
External (Shared) Sequences
Creating External (Shared) Sequences
Linking External (Shared) Sequences
Internalizing External (Shared) Sequences
Editing a Sequence
Parameters
Internal Parameters
Creating Internal Parameters
Externalizing Internal Parameters
Exporting Internal Parameters
External (Shared) Parameters
Creating External (Shared) Parameters
Linking External (Shared) Parameters
Internalizing External (Shared) Parameters
Parameters Wizard
Parameters with CTL Expressions
Environment Variables
Canonizing File Paths
Using Parameters
Internal/External Graph Elements
Internal Graph Elements
External (Shared) Graph Elements
Working with Graph Elements
Advantages of External (Shared) Graph Elements
Advantages of Internal Graph Elements
Changes of the Form of Graph Elements
Dictionary
Creating a Dictionary
Using the Dictionary in a Graph
Notes in the Graphs
Search Functionality
Transformations
Fact table loader
Launching Fact Table Loader Wizard
Wizard with project parameters file enabled
Wizard with the project parameter file disabled
Working with Fact Table Loader Wizard
Created graph
Components Overview
Introduction to Components
Palette of Components
Find / Add Components
Finding Components
Adding Components
Common Properties of All Components
Edit Component Dialog
Component Name
Phases
Enable/Disable Component
PassThrough Mode
Common Properties of Most Components
Group Key
Sort Key
Defining Transformations
Return Values of Transformations
Error Actions and Error Log (deprecated since 3.0)
Transform Editor
Regex Tester
Common Java Interfaces
Common Properties of Readers
Supported File URL Formats for Readers
Viewing Data on Readers
Input Port Reading
Incremental Reading
Selecting Input Records
Data Policy
XML Features
CTL Templates for Readers
Java Interfaces for Readers
Common Properties of Writers
Supported File URL Formats for Writers
Viewing Data on Writers
Output Port Writing
How and Where Data Should Be Written
Selecting Output Records
Partitioning Output into Different Output Files
Java Interfaces for Writers
Common Properties of Transformers
CTL Templates for Transformers
Java Interfaces for Transformers
Common Properties of Joiners
Join Types
Slave Duplicates
CTL Templates for Joiners
Java Interfaces for Joiners
Common Properties of Cluster Components
CTL Templates for Cluster Components
Java Interfaces for Cluster Components
Common Properties of Others
Custom Components
Component Reference
Readers
CloverDataReader
ComplexDataReader
DataGenerator
DBFDataReader
DBInputTable
EmailReader
JavaBeanReader
JMSReader
JSONReader
LDAPReader
LotusReader
MultiLevelReader
ParallelReader
QuickBaseRecordReader
QuickBaseQueryReader
SpreadsheetDataReader
UniversalDataReader
XLSDataReader
XMLExtract
XMLReader
XMLXPathReader
Writers
CloverDataWriter
DB2DataWriter
DBOutputTable
EmailSender
InfobrightDataWriter
InformixDataWriter
JavaBeanWriter
JavaMapWriter
JMSWriter
JSONWriter
LDAPWriter
LotusWriter
MSSQLDataWriter
MySQLDataWriter
OracleDataWriter
PostgreSQLDataWriter
QuickBaseImportCSV
QuickBaseRecordWriter
SpreadsheetDataWriter
StructuredDataWriter
Trash
UniversalDataWriter
XLSDataWriter
XMLWriter
Transformers
Address Doctor 5
Aggregate
Concatenate
DataIntersection
DataSampler
Dedup
Denormalizer
EmailFilter
ExtFilter
ExtSort
FastSort
Merge
MetaPivot
Normalizer
Partition
Pivot
Reformat
Rollup
SimpleCopy
SimpleGather
SortWithinGroups
XSLTransformer
Joiners
ApproximativeJoin
DBJoin
ExtHashJoin
ExtMergeJoin
LookupJoin
RelationalJoin
Cluster Components
ClusterGather
ClusterPartitioner
Others
CheckForeignKey
DBExecute
HTTPConnector
JavaExecute
LookupTableReaderWriter
RunGraph
SequenceChecker
SpeedLimiter
SystemExecute
WebServiceClient
CTL - CloverETL Transformation Language
Overview
CTL1 vs. CTL2 Comparison
Typed Language
Arbitrary Order of Code Parts
Compiled Mode
Access to Graph Elements (Lookups, Sequences, ...)
Metadata
Migrating CTL1 to CTL2
CTL1
Language Reference
Program Structure
Comments
Import
Data Types in CTL
Literals
Variables
Operators
Simple Statement and Block of Statements
Control Statements
Error Handling
Functions
Eval
Conditional Fail Expression
Accessing Data Records and Fields
Mapping
Parameters
Functions Reference
Conversion Functions
Date Functions
Mathematical Functions
String Functions
Container Functions
Miscellaneous Functions
Dictionary Functions
Lookup Table Functions
Sequence Functions
Custom CTL Functions
CTL2
Language Reference
Program Structure
Comments
Import
Data Types in CTL2
Literals
Variables
Dictionary in CTL2
Operators
Simple Statement and Block of Statements
Control Statements
Error Handling
Functions
Conditional Fail Expression
Accessing Data Records and Fields
Mapping
Parameters
Functions Reference
Conversion Functions
Date Functions
Mathematical Functions
String Functions
Container Functions
Record functions (dynamic field access)
Miscellaneous Functions
Lookup Table Functions
Sequence Functions
Custom CTL Functions
CTL2 Appendix - List of National-specific Characters
Regular Expressions
Tutorial
Creating a Project and a Graph
Creating a New Data File
Pasting Components in the Graph Editor Pane
Connecting Components by the Edges
Extracting Metadata from the Input File
Assigning Metadata to the Edges
Propagating Metadata
Setting Up Readers (UniversalDataReader)
Setting Up the ExtFilter Component
Setting Up the ExtSort Component
Setting Up Writers (UniversalDataWriter)
Running the Graph
Refreshing the Data-Out Folder
Opening the Output File
Custom Component
Creating a Custom Component
Component Java Definition
Plugin.xml
Integrating a Custom Component in CloverETL Designer
Plug-in Project
Component XML Definition
Plug-in Dependencies
Plug-in Extensions
Import Requisities
Plug-in Building
Tips for Advanced
CloverETL Overview
Family of CloverETL Products
CloverETL Designer
CloverETL Engine
CloverETL Server
Getting Additional Information
Integrating CloverETL Designer with CloverETL Server
Creating CloverETL Server project (Basic Principles)
Opening CloverETL Server Projects
Connecting via HTTP
Connecting via HTTPS
Designer has its Own Certificate
Designer does not have its Own Certificate
Connecting via proxy server
Installation Instructions
System Requirements for CloverETL Designer
Related Links
Downloading CloverETL
CloverETL Desktop Edition
CloverETL Desktop Trial Edition
CloverETL Community Edition
Starting CloverETL Designer
Installing Designer as Eclipse plugin
Getting Started
Creating CloverETL Projects
CloverETL Project
CloverETL Server Project
CloverETL Examples Project
Structure of CloverETL Projects
Standard Structure of All CloverETL Projects
Workspace.prm File
Opening the CloverETL Perspective
Appearance of CloverETL Perspective
CloverETL Designer Panes
Graph Editor with Palette of Components
Navigator Pane
Outline Pane
Tabs Pane
Creating CloverETL Graphs
Creating Empty Graphs
Creating a Simple Graph in a Few Simple Steps
Running CloverETL Graphs
Successful Graph Execution
Using the Run Configurations Dialog
Working with CloverETL Designer
Using Cheat Sheets
Common Dialogs
URL File Dialog
Edit Value Dialog
Open Type Dialog
Import
Import CloverETL Projects
Import from CloverETL Server Sandbox
Import Graphs
Import Metadata
Metadata from XSD
Metadata from DDL
Export
Export Graphs
Export Graphs to HTML
Export Metadata to XSD
Export to CloverETL Server Sandbox
Export Image
Advanced Topics
Program and VM Arguments
Example of Setting Up Memory Size
Changing Default CloverETL Settings
Enlarging the Font of Displayed Numbers
Setting and Configuring Java
Setting Java Runtime Environment
Installing Java Development Kit
Graph Elements, Structures and Tools
Components
Edges
What Are the Edges?
Connecting Components by the Edges
Types of Edges
Assigning Metadata to the Edges
Propagating Metadata through the Edges
Colors of the Edges
Debugging the Edges
Enabling Debug
Selecting Debug Data
Viewing Debug Data
Turning Off Debug
Edge Memory Allocation
Metadata
Data Types and Record Types
Data Types in Metadata
Record Types
Data Formats
Data and Time Format
Numeric Format
Boolean Format
String Format
Locale and Locale Sensitivity
Locale
Locale Sensitivity
Autofilling Functions
Internal Metadata
Creating Internal Metadata
Externalizing Internal Metadata
Exporting Internal Metadata
External (Shared) Metadata
Creating External (Shared) Metadata
Linking External (Shared) Metadata
Internalizing External (Shared) Metadata
Creating Metadata
Extracting Metadata from a Flat File
Extracting Metadata from Delimited Files
Extracting Metadata from Fixed Length Files
Extracting Metadata from an XLS(X) File
Extracting Metadata from a Database
Extracting Metadata from a DBase File
Creating Metadata by User
Extracting Metadata from Lotus Notes
Merging existing metadata
Dynamic Metadata
Reading Metadata from Special Sources
Creating Database Table from Metadata and Database Connection
Metadata Editor
Basics of Metadata Editor
Record Pane
Field Name vs. Label vs. Description
Details Pane
Changing and Defining Delimiters
Changing Record Delimiter
Changing Default Delimiter
Defining Non-Default Delimiter for a Field
Editing Metadata in the Source Code
Multivalue Fields
Lists and Maps Support in Components
Joininig on Lists and Maps (Comparison Rules)
Database Connections
Internal Database Connections
Creating Internal Database Connections
Externalizing Internal Database Connections
Exporting Internal Database Connections
External (Shared) Database Connections
Creating External (Shared) Database Connections
Linking External (Shared) Database Connections
Internalizing External (Shared) Database Connections
Database Connection Wizard
Encrypting the Access Password
Browsing Database and Extracting Metadata from Database Tables
Windows Authentication on Microsoft SQL Server
Getting the Native Library
Installation
JMS Connections
Internal JMS Connections
Creating Internal JMS Connections
Externalizing Internal JMS Connections
Exporting Internal JMS Connections
External (Shared) JMS Connections
Creating External (Shared) JMS Connections
Linking External (Shared) JMS Connection
Internalizing External (Shared) JMS Connections
Edit JMS Connection Wizard
Encrypting the Authentication Password
QuickBase Connections
Lotus Connections
Lookup Tables
Internal Lookup Tables
Creating Internal Lookup Tables
Externalizing Internal Lookup Tables
Exporting Internal Lookup Tables
External (Shared) Lookup Tables
Creating External (Shared) Lookup Tables
Linking External (Shared) Lookup Tables
Internalizing External (Shared) Lookup Tables
Types of Lookup Tables
Simple Lookup Table
Database Lookup Table
Range Lookup Table
Persistent Lookup Table
Aspell Lookup Table
Sequences
Internal Sequences
Creating Internal Sequences
Externalizing Internal Sequences
Exporting Internal Sequences
External (Shared) Sequences
Creating External (Shared) Sequences
Linking External (Shared) Sequences
Internalizing External (Shared) Sequences
Editing a Sequence
Parameters
Internal Parameters
Creating Internal Parameters
Externalizing Internal Parameters
Exporting Internal Parameters
External (Shared) Parameters
Creating External (Shared) Parameters
Linking External (Shared) Parameters
Internalizing External (Shared) Parameters
Parameters Wizard
Parameters with CTL Expressions
Environment Variables
Canonizing File Paths
Using Parameters
Internal/External Graph Elements
Internal Graph Elements
External (Shared) Graph Elements
Working with Graph Elements
Advantages of External (Shared) Graph Elements
Advantages of Internal Graph Elements
Changes of the Form of Graph Elements
Dictionary
Creating a Dictionary
Using the Dictionary in a Graph
Notes in the Graphs
Search Functionality
Transformations
Fact table loader
Launching Fact Table Loader Wizard
Wizard with project parameters file enabled
Wizard with the project parameter file disabled
Working with Fact Table Loader Wizard
Created graph
Components Overview
Introduction to Components
Palette of Components
Find / Add Components
Finding Components
Adding Components
Common Properties of All Components
Edit Component Dialog
Component Name
Phases
Enable/Disable Component
PassThrough Mode
Common Properties of Most Components
Group Key
Sort Key
Defining Transformations
Return Values of Transformations
Error Actions and Error Log (deprecated since 3.0)
Transform Editor
Regex Tester
Common Java Interfaces
Common Properties of Readers
Supported File URL Formats for Readers
Viewing Data on Readers
Input Port Reading
Incremental Reading
Selecting Input Records
Data Policy
XML Features
CTL Templates for Readers
Java Interfaces for Readers
Common Properties of Writers
Supported File URL Formats for Writers
Viewing Data on Writers
Output Port Writing
How and Where Data Should Be Written
Selecting Output Records
Partitioning Output into Different Output Files
Java Interfaces for Writers
Common Properties of Transformers
CTL Templates for Transformers
Java Interfaces for Transformers
Common Properties of Joiners
Join Types
Slave Duplicates
CTL Templates for Joiners
Java Interfaces for Joiners
Common Properties of Cluster Components
CTL Templates for Cluster Components
Java Interfaces for Cluster Components
Common Properties of Others
Custom Components
Component Reference
Readers
CloverDataReader
ComplexDataReader
DataGenerator
DBFDataReader
DBInputTable
EmailReader
JavaBeanReader
JMSReader
JSONReader
LDAPReader
LotusReader
MultiLevelReader
ParallelReader
QuickBaseRecordReader
QuickBaseQueryReader
SpreadsheetDataReader
UniversalDataReader
XLSDataReader
XMLExtract
XMLReader
XMLXPathReader
Writers
CloverDataWriter
DB2DataWriter
DBOutputTable
EmailSender
InfobrightDataWriter
InformixDataWriter
JavaBeanWriter
JavaMapWriter
JMSWriter
JSONWriter
LDAPWriter
LotusWriter
MSSQLDataWriter
MySQLDataWriter
OracleDataWriter
PostgreSQLDataWriter
QuickBaseImportCSV
QuickBaseRecordWriter
SpreadsheetDataWriter
StructuredDataWriter
Trash
UniversalDataWriter
XLSDataWriter
XMLWriter
Transformers
Address Doctor 5
Aggregate
Concatenate
DataIntersection
DataSampler
Dedup
Denormalizer
EmailFilter
ExtFilter
ExtSort
FastSort
Merge
MetaPivot
Normalizer
Partition
Pivot
Reformat
Rollup
SimpleCopy
SimpleGather
SortWithinGroups
XSLTransformer
Joiners
ApproximativeJoin
DBJoin
ExtHashJoin
ExtMergeJoin
LookupJoin
RelationalJoin
Cluster Components
ClusterGather
ClusterPartitioner
Others
CheckForeignKey
DBExecute
HTTPConnector
JavaExecute
LookupTableReaderWriter
RunGraph
SequenceChecker
SpeedLimiter
SystemExecute
WebServiceClient
CTL - CloverETL Transformation Language
Overview
CTL1 vs. CTL2 Comparison
Typed Language
Arbitrary Order of Code Parts
Compiled Mode
Access to Graph Elements (Lookups, Sequences, ...)
Metadata
Migrating CTL1 to CTL2
CTL1
Language Reference
Program Structure
Comments
Import
Data Types in CTL
Literals
Variables
Operators
Simple Statement and Block of Statements
Control Statements
Error Handling
Functions
Eval
Conditional Fail Expression
Accessing Data Records and Fields
Mapping
Parameters
Functions Reference
Conversion Functions
Date Functions
Mathematical Functions
String Functions
Container Functions
Miscellaneous Functions
Dictionary Functions
Lookup Table Functions
Sequence Functions
Custom CTL Functions
CTL2
Language Reference
Program Structure
Comments
Import
Data Types in CTL2
Literals
Variables
Dictionary in CTL2
Operators
Simple Statement and Block of Statements
Control Statements
Error Handling
Functions
Conditional Fail Expression
Accessing Data Records and Fields
Mapping
Parameters
Functions Reference
Conversion Functions
Date Functions
Mathematical Functions
String Functions
Container Functions
Record functions (dynamic field access)
Miscellaneous Functions
Lookup Table Functions
Sequence Functions
Custom CTL Functions
CTL2 Appendix - List of National-specific Characters
Regular Expressions
Tutorial
Creating a Project and a Graph
Creating a New Data File
Pasting Components in the Graph Editor Pane
Connecting Components by the Edges
Extracting Metadata from the Input File
Assigning Metadata to the Edges
Propagating Metadata
Setting Up Readers (UniversalDataReader)
Setting Up the ExtFilter Component
Setting Up the ExtSort Component
Setting Up Writers (UniversalDataWriter)
Running the Graph
Refreshing the Data-Out Folder
Opening the Output File
Custom Component
Creating a Custom Component
Component Java Definition
Plugin.xml
Integrating a Custom Component in CloverETL Designer
Plug-in Project
Component XML Definition
Plug-in Dependencies
Plug-in Extensions
Import Requisities
Plug-in Building
Tips for Advanced