CloverETL Data Profiler
About CloverETL Data Profiler
Installation
Hardware Requirements
Software Requirements
Licensing
Getting CloverETL Data Profiler
Getting Started
CloverETL Data Profiler essentials
Profiling Session Step-by-step
CloverETL Data Profiler GUI
Working With Projects
Creating Projects
Importing Projects
Exporting Projects
Creating Jobs
File Profiling
Database Table Profiling
Excel Sheet Profiling
Metrics
Data Sampling
Modifying Jobs
Running Jobs
Reporting Console
Screens hierarchy
Reporting Console as Standalone Server
Integration with CloverETL Server
Hardware and Software Requirements
Installation
Tomcat, Jetty
Websphere
Glassfish, Weblogic
JBoss 5, JBoss 6
Default Configuration
Configuration
Base Configuration
Alternative ways of configuration
Database Support
Derby
MSSQL
MySQL
Oracle
PostgreSQL
JNDI
Configuring CloverETL Server
User Accounts Settings
Permissions
Group-level permissions
Sandbox-level permissions
Public API Authorization
Public API
Historic Metric Values
RSS Feed
Advanced Topics
Configuration
Run configurations
Preferences
File URL
Integration with CloverETL
Using Data Profiler Feature within Designer
Profiler job (.cpj) editing
Data Profiler Graph Components
ProfilerProbe
ExecuteProfilerJob
Profiler-Designer Element Sharing
List of Metrics
About CloverETL Data Profiler
Installation
Hardware Requirements
Software Requirements
Licensing
Getting CloverETL Data Profiler
Getting Started
CloverETL Data Profiler essentials
Profiling Session Step-by-step
CloverETL Data Profiler GUI
Working With Projects
Creating Projects
Importing Projects
Exporting Projects
Creating Jobs
File Profiling
Database Table Profiling
Excel Sheet Profiling
Metrics
Data Sampling
Modifying Jobs
Running Jobs
Reporting Console
Screens hierarchy
Reporting Console as Standalone Server
Integration with CloverETL Server
Hardware and Software Requirements
Installation
Tomcat, Jetty
Websphere
Glassfish, Weblogic
JBoss 5, JBoss 6
Default Configuration
Configuration
Base Configuration
Alternative ways of configuration
Database Support
Derby
MSSQL
MySQL
Oracle
PostgreSQL
JNDI
Configuring CloverETL Server
User Accounts Settings
Permissions
Group-level permissions
Sandbox-level permissions
Public API Authorization
Public API
Historic Metric Values
RSS Feed
Advanced Topics
Configuration
Run configurations
Preferences
File URL
Integration with CloverETL
Using Data Profiler Feature within Designer
Profiler job (.cpj) editing
Data Profiler Graph Components
ProfilerProbe
ExecuteProfilerJob
Profiler-Designer Element Sharing
List of Metrics
CloverETL Designer
CloverETL Overview
Family of CloverETL Products
CloverETL Designer
CloverETL Engine
CloverETL Server
Getting Additional Information
Integrating CloverETL Designer with CloverETL Server
Creating CloverETL Server project (Basic Principles)
Opening CloverETL Server Projects
Connecting via HTTP
Connecting via HTTPS
Designer has its Own Certificate
Designer does not have its Own Certificate
Connecting via proxy server
Installation Instructions
System Requirements for CloverETL Designer
Related Links
Downloading CloverETL
CloverETL Desktop Edition
CloverETL Desktop Trial Edition
CloverETL Community Edition
Starting CloverETL Designer
Installing Designer as Eclipse plugin
Getting Started
License Manager
CloverETL License dialog
CloverETL License wizard
Activation using License key
Activation online
Creating CloverETL Projects
CloverETL Project
CloverETL Server Project
CloverETL Examples Project
Structure of CloverETL Projects
Standard Structure of All CloverETL Projects
Workspace.prm File
Opening the CloverETL Perspective
Appearance of CloverETL Perspective
CloverETL Designer Panes
Graph Editor with Palette of Components
Navigator Pane
Outline Pane
Tabs Pane
Creating CloverETL Graphs
Creating Empty Graphs
Creating a Simple Graph in a Few Simple Steps
Running CloverETL Graphs
Successful Graph Execution
Using the Run Configurations Dialog
Working with CloverETL Designer
Using Cheat Sheets
Common Dialogs
URL File Dialog
Edit Value Dialog
Open Type Dialog
Import
Import CloverETL Projects
Import from CloverETL Server Sandbox
Import Graphs
Import Metadata
Metadata from XSD
Metadata from DDL
Export
Export Graphs
Export Graphs to HTML
Export Metadata to XSD
Export to CloverETL Server Sandbox
Export Image
Graph tracking
Advanced Topics
Program and VM Arguments
Example of Setting Up Memory Size
Changing Default CloverETL Settings
Enlarging the Font of Displayed Numbers
Setting and Configuring Java
Setting Java Runtime Environment
Installing Java Development Kit
Graph Elements, Structures and Tools
Components
Edges
What Are the Edges?
Connecting Components by the Edges
Types of Edges
Assigning Metadata to the Edges
Propagating Metadata through the Edges
Colors of the Edges
Debugging the Edges
Enabling Debug
Selecting Debug Data
Viewing Debug Data
Turning Off Debug
Edge Memory Allocation
Metadata
Data Types and Record Types
Data Types in Metadata
Record Types
Data Formats
Date and Time Format
Numeric Format
Boolean Format
String Format
Locale and Locale Sensitivity
Locale
Locale Sensitivity
Autofilling Functions
Internal Metadata
Creating Internal Metadata
Externalizing Internal Metadata
Exporting Internal Metadata
External (Shared) Metadata
Creating External (Shared) Metadata
Linking External (Shared) Metadata
Internalizing External (Shared) Metadata
Creating Metadata
Extracting Metadata from a Flat File
Extracting Metadata from Delimited Files
Extracting Metadata from Fixed Length Files
Extracting Metadata from an XLS(X) File
Extracting Metadata from a Database
Extracting Metadata from a DBase File
Creating Metadata by User
Extracting Metadata from Lotus Notes
Merging existing metadata
Dynamic Metadata
Reading Metadata from Special Sources
Creating Database Table from Metadata and Database Connection
Metadata Editor
Basics of Metadata Editor
Record Pane
Field Name vs. Label vs. Description
Details Pane
Changing and Defining Delimiters
Changing Record Delimiter
Changing Default Delimiter
Defining Non-Default Delimiter for a Field
Editing Metadata in the Source Code
Multivalue Fields
Lists and Maps Support in Components
Joininig on Lists and Maps (Comparison Rules)
Database Connections
Internal Database Connections
Creating Internal Database Connections
Externalizing Internal Database Connections
Exporting Internal Database Connections
External (Shared) Database Connections
Creating External (Shared) Database Connections
Linking External (Shared) Database Connections
Internalizing External (Shared) Database Connections
Database Connection Wizard
Encrypting the Access Password
Browsing Database and Extracting Metadata from Database Tables
Windows Authentication on Microsoft SQL Server
Getting the Native Library
Installation
Hive Connection
JMS Connections
Internal JMS Connections
Creating Internal JMS Connections
Externalizing Internal JMS Connections
Exporting Internal JMS Connections
External (Shared) JMS Connections
Creating External (Shared) JMS Connections
Linking External (Shared) JMS Connection
Internalizing External (Shared) JMS Connections
Edit JMS Connection Wizard
Encrypting the Authentication Password
QuickBase Connections
Lotus Connections
Hadoop Connections
Lookup Tables
LookupTables in CloverETL Cluster environment
Internal Lookup Tables
Creating Internal Lookup Tables
Externalizing Internal Lookup Tables
Exporting Internal Lookup Tables
External (Shared) Lookup Tables
Creating External (Shared) Lookup Tables
Linking External (Shared) Lookup Tables
Internalizing External (Shared) Lookup Tables
Types of Lookup Tables
Simple Lookup Table
Database Lookup Table
Range Lookup Table
Persistent Lookup Table
Aspell Lookup Table
Sequences
Internal Sequences
Creating Internal Sequences
Externalizing Internal Sequences
Exporting Internal Sequences
External (Shared) Sequences
Creating External (Shared) Sequences
Linking External (Shared) Sequences
Internalizing External (Shared) Sequences
Editing a Sequence
Parameters
Internal Parameters
Creating Internal Parameters
Externalizing Internal Parameters
Exporting Internal Parameters
External (Shared) Parameters
Creating External (Shared) Parameters
Linking External (Shared) Parameters
Internalizing External (Shared) Parameters
Parameters Wizard
Parameters with CTL Expressions
Environment Variables
Canonizing File Paths
Using Parameters
Internal/External Graph Elements
Internal Graph Elements
External (Shared) Graph Elements
Working with Graph Elements
Advantages of External (Shared) Graph Elements
Advantages of Internal Graph Elements
Changes of the Form of Graph Elements
Dictionary
Creating a Dictionary
Using the Dictionary in a Graph
Notes in the Graphs
Search Functionality
Transformations
Fact table loader
Launching Fact Table Loader Wizard
Wizard with project parameters file enabled
Wizard with the project parameter file disabled
Working with Fact Table Loader Wizard
Created graph
Jobflow
Jobflow Overview
Introduction
Important concepts
Advanced Concepts
Jobflow Design Patterns
Components Overview
Introduction to Components
Palette of Components
Find / Add Components
Finding Components
Adding Components
Common Properties of All Components
Edit Component Dialog
Component Name
Phases
Enable/Disable Component
PassThrough Mode
Component Allocation
Common Properties of Most Components
Metadata Templates
Time Intervals
Group Key
Sort Key
Defining Transformations
Return Values of Transformations
Error Actions and Error Log (deprecated since 3.0)
Transform Editor
Regex Tester
Common Java Interfaces
Common Properties of Readers
Supported File URL Formats for Readers
Viewing Data on Readers
Input Port Reading
Incremental Reading
Selecting Input Records
Data Policy
XML Features
CTL Templates for Readers
Java Interfaces for Readers
Common Properties of Writers
Supported File URL Formats for Writers
Viewing Data on Writers
Output Port Writing
How and Where Data Should Be Written
Selecting Output Records
Partitioning Output into Different Output Files
Java Interfaces for Writers
Common Properties of Transformers
CTL Templates for Transformers
Java Interfaces for Transformers
Common Properties of Joiners
Join Types
Slave Duplicates
CTL Templates for Joiners
Java Interfaces for Joiners
Common Properties of Cluster Components
Common Properties of Others
Common Properties of Data Quality
Common Properties of Job Control
Common Properties of File Operations
Supported URL Formats for File Operations
Custom Components
Component Reference
Readers
CloverDataReader
ComplexDataReader
DataGenerator
DBFDataReader
DBInputTable
EmailReader
JavaBeanReader
HadoopReader
JMSReader
JSONReader
LDAPReader
LotusReader
MultiLevelReader
ParallelReader
QuickBaseRecordReader
QuickBaseQueryReader
SpreadsheetDataReader
UniversalDataReader
XLSDataReader
XMLExtract
XMLReader
XMLXPathReader
Writers
CloverDataWriter
DB2DataWriter
DBFDataWriter
DBOutputTable
EmailSender
HadoopWriter
InfobrightDataWriter
InformixDataWriter
JavaBeanWriter
JavaMapWriter
JMSWriter
JSONWriter
LDAPWriter
LotusWriter
MSSQLDataWriter
MySQLDataWriter
OracleDataWriter
PostgreSQLDataWriter
QuickBaseImportCSV
QuickBaseRecordWriter
SpreadsheetDataWriter
StructuredDataWriter
Trash
UniversalDataWriter
XLSDataWriter
XMLWriter
Transformers
Aggregate
Concatenate
DataIntersection
DataSampler
Dedup
Denormalizer
ExtFilter
ExtSort
FastSort
Merge
MetaPivot
Normalizer
Partition
LoadBalancingPartition
Pivot
Reformat
Rollup
SimpleCopy
SimpleGather
SortWithinGroups
XSLTransformer
Joiners
ApproximativeJoin
Combine
DBJoin
ExtHashJoin
ExtMergeJoin
LookupJoin
RelationalJoin
Job Control
Barrier
Condition
ExecuteGraph
ExecuteJobflow
ExecuteMapReduce
ExecuteProfilerJob
Input mapping
Output mapping
ExecuteScript
Fail
GetJobInput
KillGraph
KillJobflow
MonitorGraph
MonitorJobflow
SetJobOutput
Success
TokenGather
File Operations
CopyFiles
CreateFiles
DeleteFiles
ListFiles
MoveFiles
Cluster Components
ClusterPartition
ClusterLoadBalancingPartition
ClusterSimpleCopy
ClusterSimpleGather
ClusterMerge
ClusterRepartition
Data Quality
Address Doctor 5
EmailFilter
ProfilerProbe
Others
CheckForeignKey
DBExecute
HTTPConnector
JavaExecute
LookupTableReaderWriter
RunGraph
SequenceChecker
SpeedLimiter
SystemExecute
WebServiceClient
CTL - CloverETL Transformation Language
Overview
CTL1 vs. CTL2 Comparison
Typed Language
Arbitrary Order of Code Parts
Compiled Mode
Access to Graph Elements (Lookups, Sequences, ...)
Metadata
Migrating CTL1 to CTL2
CTL1
Language Reference
Program Structure
Comments
Import
Data Types in CTL
Literals
Variables
Operators
Simple Statement and Block of Statements
Control Statements
Error Handling
Functions
Eval
Conditional Fail Expression
Accessing Data Records and Fields
Mapping
Parameters
Functions Reference
Conversion Functions
Date Functions
Mathematical Functions
String Functions
Container Functions
Miscellaneous Functions
Dictionary Functions
Lookup Table Functions
Sequence Functions
Custom CTL Functions
CTL2
Language Reference
Program Structure
Comments
Import
Data Types in CTL2
Literals
Variables
Dictionary in CTL2
Operators
Simple Statement and Block of Statements
Control Statements
Error Handling
Functions
Conditional Fail Expression
Accessing Data Records and Fields
Mapping
Parameters
Functions Reference
Conversion Functions
Date Functions
Mathematical Functions
String Functions
Container Functions
Record functions (dynamic field access)
Miscellaneous Functions
Lookup Table Functions
Sequence Functions
Custom CTL Functions
CTL2 Appendix - List of National-specific Characters
Regular Expressions
Tutorial
Creating a Project and a Graph
Creating a New Data File
Pasting Components in the Graph Editor Pane
Connecting Components by the Edges
Extracting Metadata from the Input File
Assigning Metadata to the Edges
Propagating Metadata
Setting Up Readers (UniversalDataReader)
Setting Up the ExtFilter Component
Setting Up the ExtSort Component
Setting Up Writers (UniversalDataWriter)
Running the Graph
Refreshing the Data-Out Folder
Opening the Output File
Custom Component
Creating a Custom Component
Component Java Definition
Plugin.xml
Integrating a Custom Component in CloverETL Designer
Plug-in Project
Component XML Definition
Plug-in Dependencies
Plug-in Extensions
Import Requisities
Plug-in Building
Troubleshooting
Tips for Advanced
CloverETL Overview
Family of CloverETL Products
CloverETL Designer
CloverETL Engine
CloverETL Server
Getting Additional Information
Integrating CloverETL Designer with CloverETL Server
Creating CloverETL Server project (Basic Principles)
Opening CloverETL Server Projects
Connecting via HTTP
Connecting via HTTPS
Designer has its Own Certificate
Designer does not have its Own Certificate
Connecting via proxy server
Installation Instructions
System Requirements for CloverETL Designer
Related Links
Downloading CloverETL
CloverETL Desktop Edition
CloverETL Desktop Trial Edition
CloverETL Community Edition
Starting CloverETL Designer
Installing Designer as Eclipse plugin
Getting Started
License Manager
CloverETL License dialog
CloverETL License wizard
Activation using License key
Activation online
Creating CloverETL Projects
CloverETL Project
CloverETL Server Project
CloverETL Examples Project
Structure of CloverETL Projects
Standard Structure of All CloverETL Projects
Workspace.prm File
Opening the CloverETL Perspective
Appearance of CloverETL Perspective
CloverETL Designer Panes
Graph Editor with Palette of Components
Navigator Pane
Outline Pane
Tabs Pane
Creating CloverETL Graphs
Creating Empty Graphs
Creating a Simple Graph in a Few Simple Steps
Running CloverETL Graphs
Successful Graph Execution
Using the Run Configurations Dialog
Working with CloverETL Designer
Using Cheat Sheets
Common Dialogs
URL File Dialog
Edit Value Dialog
Open Type Dialog
Import
Import CloverETL Projects
Import from CloverETL Server Sandbox
Import Graphs
Import Metadata
Metadata from XSD
Metadata from DDL
Export
Export Graphs
Export Graphs to HTML
Export Metadata to XSD
Export to CloverETL Server Sandbox
Export Image
Graph tracking
Advanced Topics
Program and VM Arguments
Example of Setting Up Memory Size
Changing Default CloverETL Settings
Enlarging the Font of Displayed Numbers
Setting and Configuring Java
Setting Java Runtime Environment
Installing Java Development Kit
Graph Elements, Structures and Tools
Components
Edges
What Are the Edges?
Connecting Components by the Edges
Types of Edges
Assigning Metadata to the Edges
Propagating Metadata through the Edges
Colors of the Edges
Debugging the Edges
Enabling Debug
Selecting Debug Data
Viewing Debug Data
Turning Off Debug
Edge Memory Allocation
Metadata
Data Types and Record Types
Data Types in Metadata
Record Types
Data Formats
Date and Time Format
Numeric Format
Boolean Format
String Format
Locale and Locale Sensitivity
Locale
Locale Sensitivity
Autofilling Functions
Internal Metadata
Creating Internal Metadata
Externalizing Internal Metadata
Exporting Internal Metadata
External (Shared) Metadata
Creating External (Shared) Metadata
Linking External (Shared) Metadata
Internalizing External (Shared) Metadata
Creating Metadata
Extracting Metadata from a Flat File
Extracting Metadata from Delimited Files
Extracting Metadata from Fixed Length Files
Extracting Metadata from an XLS(X) File
Extracting Metadata from a Database
Extracting Metadata from a DBase File
Creating Metadata by User
Extracting Metadata from Lotus Notes
Merging existing metadata
Dynamic Metadata
Reading Metadata from Special Sources
Creating Database Table from Metadata and Database Connection
Metadata Editor
Basics of Metadata Editor
Record Pane
Field Name vs. Label vs. Description
Details Pane
Changing and Defining Delimiters
Changing Record Delimiter
Changing Default Delimiter
Defining Non-Default Delimiter for a Field
Editing Metadata in the Source Code
Multivalue Fields
Lists and Maps Support in Components
Joininig on Lists and Maps (Comparison Rules)
Database Connections
Internal Database Connections
Creating Internal Database Connections
Externalizing Internal Database Connections
Exporting Internal Database Connections
External (Shared) Database Connections
Creating External (Shared) Database Connections
Linking External (Shared) Database Connections
Internalizing External (Shared) Database Connections
Database Connection Wizard
Encrypting the Access Password
Browsing Database and Extracting Metadata from Database Tables
Windows Authentication on Microsoft SQL Server
Getting the Native Library
Installation
Hive Connection
JMS Connections
Internal JMS Connections
Creating Internal JMS Connections
Externalizing Internal JMS Connections
Exporting Internal JMS Connections
External (Shared) JMS Connections
Creating External (Shared) JMS Connections
Linking External (Shared) JMS Connection
Internalizing External (Shared) JMS Connections
Edit JMS Connection Wizard
Encrypting the Authentication Password
QuickBase Connections
Lotus Connections
Hadoop Connections
Lookup Tables
LookupTables in CloverETL Cluster environment
Internal Lookup Tables
Creating Internal Lookup Tables
Externalizing Internal Lookup Tables
Exporting Internal Lookup Tables
External (Shared) Lookup Tables
Creating External (Shared) Lookup Tables
Linking External (Shared) Lookup Tables
Internalizing External (Shared) Lookup Tables
Types of Lookup Tables
Simple Lookup Table
Database Lookup Table
Range Lookup Table
Persistent Lookup Table
Aspell Lookup Table
Sequences
Internal Sequences
Creating Internal Sequences
Externalizing Internal Sequences
Exporting Internal Sequences
External (Shared) Sequences
Creating External (Shared) Sequences
Linking External (Shared) Sequences
Internalizing External (Shared) Sequences
Editing a Sequence
Parameters
Internal Parameters
Creating Internal Parameters
Externalizing Internal Parameters
Exporting Internal Parameters
External (Shared) Parameters
Creating External (Shared) Parameters
Linking External (Shared) Parameters
Internalizing External (Shared) Parameters
Parameters Wizard
Parameters with CTL Expressions
Environment Variables
Canonizing File Paths
Using Parameters
Internal/External Graph Elements
Internal Graph Elements
External (Shared) Graph Elements
Working with Graph Elements
Advantages of External (Shared) Graph Elements
Advantages of Internal Graph Elements
Changes of the Form of Graph Elements
Dictionary
Creating a Dictionary
Using the Dictionary in a Graph
Notes in the Graphs
Search Functionality
Transformations
Fact table loader
Launching Fact Table Loader Wizard
Wizard with project parameters file enabled
Wizard with the project parameter file disabled
Working with Fact Table Loader Wizard
Created graph
Jobflow
Jobflow Overview
Introduction
Important concepts
Advanced Concepts
Jobflow Design Patterns
Components Overview
Introduction to Components
Palette of Components
Find / Add Components
Finding Components
Adding Components
Common Properties of All Components
Edit Component Dialog
Component Name
Phases
Enable/Disable Component
PassThrough Mode
Component Allocation
Common Properties of Most Components
Metadata Templates
Time Intervals
Group Key
Sort Key
Defining Transformations
Return Values of Transformations
Error Actions and Error Log (deprecated since 3.0)
Transform Editor
Regex Tester
Common Java Interfaces
Common Properties of Readers
Supported File URL Formats for Readers
Viewing Data on Readers
Input Port Reading
Incremental Reading
Selecting Input Records
Data Policy
XML Features
CTL Templates for Readers
Java Interfaces for Readers
Common Properties of Writers
Supported File URL Formats for Writers
Viewing Data on Writers
Output Port Writing
How and Where Data Should Be Written
Selecting Output Records
Partitioning Output into Different Output Files
Java Interfaces for Writers
Common Properties of Transformers
CTL Templates for Transformers
Java Interfaces for Transformers
Common Properties of Joiners
Join Types
Slave Duplicates
CTL Templates for Joiners
Java Interfaces for Joiners
Common Properties of Cluster Components
Common Properties of Others
Common Properties of Data Quality
Common Properties of Job Control
Common Properties of File Operations
Supported URL Formats for File Operations
Custom Components
Component Reference
Readers
CloverDataReader
ComplexDataReader
DataGenerator
DBFDataReader
DBInputTable
EmailReader
JavaBeanReader
HadoopReader
JMSReader
JSONReader
LDAPReader
LotusReader
MultiLevelReader
ParallelReader
QuickBaseRecordReader
QuickBaseQueryReader
SpreadsheetDataReader
UniversalDataReader
XLSDataReader
XMLExtract
XMLReader
XMLXPathReader
Writers
CloverDataWriter
DB2DataWriter
DBFDataWriter
DBOutputTable
EmailSender
HadoopWriter
InfobrightDataWriter
InformixDataWriter
JavaBeanWriter
JavaMapWriter
JMSWriter
JSONWriter
LDAPWriter
LotusWriter
MSSQLDataWriter
MySQLDataWriter
OracleDataWriter
PostgreSQLDataWriter
QuickBaseImportCSV
QuickBaseRecordWriter
SpreadsheetDataWriter
StructuredDataWriter
Trash
UniversalDataWriter
XLSDataWriter
XMLWriter
Transformers
Aggregate
Concatenate
DataIntersection
DataSampler
Dedup
Denormalizer
ExtFilter
ExtSort
FastSort
Merge
MetaPivot
Normalizer
Partition
LoadBalancingPartition
Pivot
Reformat
Rollup
SimpleCopy
SimpleGather
SortWithinGroups
XSLTransformer
Joiners
ApproximativeJoin
Combine
DBJoin
ExtHashJoin
ExtMergeJoin
LookupJoin
RelationalJoin
Job Control
Barrier
Condition
ExecuteGraph
ExecuteJobflow
ExecuteMapReduce
ExecuteProfilerJob
Input mapping
Output mapping
ExecuteScript
Fail
GetJobInput
KillGraph
KillJobflow
MonitorGraph
MonitorJobflow
SetJobOutput
Success
TokenGather
File Operations
CopyFiles
CreateFiles
DeleteFiles
ListFiles
MoveFiles
Cluster Components
ClusterPartition
ClusterLoadBalancingPartition
ClusterSimpleCopy
ClusterSimpleGather
ClusterMerge
ClusterRepartition
Data Quality
Address Doctor 5
EmailFilter
ProfilerProbe
Others
CheckForeignKey
DBExecute
HTTPConnector
JavaExecute
LookupTableReaderWriter
RunGraph
SequenceChecker
SpeedLimiter
SystemExecute
WebServiceClient
CTL - CloverETL Transformation Language
Overview
CTL1 vs. CTL2 Comparison
Typed Language
Arbitrary Order of Code Parts
Compiled Mode
Access to Graph Elements (Lookups, Sequences, ...)
Metadata
Migrating CTL1 to CTL2
CTL1
Language Reference
Program Structure
Comments
Import
Data Types in CTL
Literals
Variables
Operators
Simple Statement and Block of Statements
Control Statements
Error Handling
Functions
Eval
Conditional Fail Expression
Accessing Data Records and Fields
Mapping
Parameters
Functions Reference
Conversion Functions
Date Functions
Mathematical Functions
String Functions
Container Functions
Miscellaneous Functions
Dictionary Functions
Lookup Table Functions
Sequence Functions
Custom CTL Functions
CTL2
Language Reference
Program Structure
Comments
Import
Data Types in CTL2
Literals
Variables
Dictionary in CTL2
Operators
Simple Statement and Block of Statements
Control Statements
Error Handling
Functions
Conditional Fail Expression
Accessing Data Records and Fields
Mapping
Parameters
Functions Reference
Conversion Functions
Date Functions
Mathematical Functions
String Functions
Container Functions
Record functions (dynamic field access)
Miscellaneous Functions
Lookup Table Functions
Sequence Functions
Custom CTL Functions
CTL2 Appendix - List of National-specific Characters
Regular Expressions
Tutorial
Creating a Project and a Graph
Creating a New Data File
Pasting Components in the Graph Editor Pane
Connecting Components by the Edges
Extracting Metadata from the Input File
Assigning Metadata to the Edges
Propagating Metadata
Setting Up Readers (UniversalDataReader)
Setting Up the ExtFilter Component
Setting Up the ExtSort Component
Setting Up Writers (UniversalDataWriter)
Running the Graph
Refreshing the Data-Out Folder
Opening the Output File
Custom Component
Creating a Custom Component
Component Java Definition
Plugin.xml
Integrating a Custom Component in CloverETL Designer
Plug-in Project
Component XML Definition
Plug-in Dependencies
Plug-in Extensions
Import Requisities
Plug-in Building
Troubleshooting
Tips for Advanced
CloverETL Server
What is CloverETL Server?
Installation
Evaluation Server
Enterprise Server
Apache Tomcat
Jetty
IBM Websphere
Glassfish / Sun Java System Application Server
JBoss
Oracle WebLogic Server
Possible issues during installation
Memory Settings
Upgrading Server to Newer Version
Server Side Job files - Sandboxes
Referencing files from the ETL graph or Jobflow
Sandbox Content Security and Permissions
Sandbox Content
Job config properties
Viewing Job Runs - Executions History
Users and Groups
LDAP authentication
Web GUI section Users
Web GUI section Groups
Scheduling
Timetable Setting
Tasks
Graph Event Listeners
Graph Events
Listener
Tasks
Use cases
Jobflow Event Listeners
Jobflow Events
Listener
Tasks
JMS messages listeners
Optional Groovy code
Message data available for further processing
Universal event listeners
Groovy code
Evaluation Critieria
Manual task execution
File event listeners
Observed file
File Events
Check interval, Task and Use cases
WebDAV
WebDAV clients
WebDAV authentication/authorization
Simple HTTP API
Operation help
Operation graph_run
Operation graph_status
Operation graph_kill
Operation server_jobs
Operation sandbox_list
Operation sandbox_content
Operation executions_history
Operation suspend
Operation resume
Operation sandbox_create
Operation sandbox_add_location
Operation sandbox_remove_location
Cluster status
JMX mBean
JMX configuration
Operations
SOAP WebService API
SOAP WS Client
SOAP WS API authentication/authorization
Launch Services
Launch Services Overview
Deploying Graph in Launch Service
Using Dictionary in ETL Graph/Jobflow for a Launch Services
Configuring the job in CloverETL Server Web GUI
Sending the Data to Launch Service
Results of the Graph Execution
Configuration
Config Sources and Their Priorities
Examples of DB Connection Configuration
Embedded Apache Derby
MySQL
DB2
Oracle
MS SQL
Postgre SQL
JNDI DB DataSource
List of Properties
Graph/Jobflow parameters
Another sets of parameters according the type of execution
executed from Web GUI
executed by Launch Service invocation
executed by HTTP API run graph operation invocation
executed by RunGraph component
executed by WS API method executeGraph invocation
executed by task "graph execution" by scheduler
executed from JMS listener
executed by task "Start a graph" by graph/jobflow event listener
executed by task "graph execution" by file event listener
How to add another graph parameters
Additional "Graph Config Parameters"
Task "execute_graph" parameters
Recommendations for transformations developers
Add external libraries to app-server classpath
Another graphs executed by RunGraph component may be executed only in the same JVM instance
Logging
Main logs
Graph run logs
CloverETL Server API Extensibility
Groovy Code API
Embedded OSGi framework
Extensibility - CloverETL Engine Plugins
Clustering
High Availability
Scalability
Transformation Requests
Parallel Data Processing
Graph allocation examples
Recommendations for Cluster Deployment
Example of Distributed Execution
Details of the Example Transformation Design
Scalability of the Example Transformation
Cluster configuration
Mandatory properties
Optional properties
Example of 2 node cluster configuration
Load balancing properties
Temp Space Management
Overview
Setup
Adding Temp Space
What is CloverETL Server?
Installation
Evaluation Server
Enterprise Server
Apache Tomcat
Jetty
IBM Websphere
Glassfish / Sun Java System Application Server
JBoss
Oracle WebLogic Server
Possible issues during installation
Memory Settings
Upgrading Server to Newer Version
Server Side Job files - Sandboxes
Referencing files from the ETL graph or Jobflow
Sandbox Content Security and Permissions
Sandbox Content
Job config properties
Viewing Job Runs - Executions History
Users and Groups
LDAP authentication
Web GUI section Users
Web GUI section Groups
Scheduling
Timetable Setting
Tasks
Graph Event Listeners
Graph Events
Listener
Tasks
Use cases
Jobflow Event Listeners
Jobflow Events
Listener
Tasks
JMS messages listeners
Optional Groovy code
Message data available for further processing
Universal event listeners
Groovy code
Evaluation Critieria
Manual task execution
File event listeners
Observed file
File Events
Check interval, Task and Use cases
WebDAV
WebDAV clients
WebDAV authentication/authorization
Simple HTTP API
Operation help
Operation graph_run
Operation graph_status
Operation graph_kill
Operation server_jobs
Operation sandbox_list
Operation sandbox_content
Operation executions_history
Operation suspend
Operation resume
Operation sandbox_create
Operation sandbox_add_location
Operation sandbox_remove_location
Cluster status
JMX mBean
JMX configuration
Operations
SOAP WebService API
SOAP WS Client
SOAP WS API authentication/authorization
Launch Services
Launch Services Overview
Deploying Graph in Launch Service
Using Dictionary in ETL Graph/Jobflow for a Launch Services
Configuring the job in CloverETL Server Web GUI
Sending the Data to Launch Service
Results of the Graph Execution
Configuration
Config Sources and Their Priorities
Examples of DB Connection Configuration
Embedded Apache Derby
MySQL
DB2
Oracle
MS SQL
Postgre SQL
JNDI DB DataSource
List of Properties
Graph/Jobflow parameters
Another sets of parameters according the type of execution
executed from Web GUI
executed by Launch Service invocation
executed by HTTP API run graph operation invocation
executed by RunGraph component
executed by WS API method executeGraph invocation
executed by task "graph execution" by scheduler
executed from JMS listener
executed by task "Start a graph" by graph/jobflow event listener
executed by task "graph execution" by file event listener
How to add another graph parameters
Additional "Graph Config Parameters"
Task "execute_graph" parameters
Recommendations for transformations developers
Add external libraries to app-server classpath
Another graphs executed by RunGraph component may be executed only in the same JVM instance
Logging
Main logs
Graph run logs
CloverETL Server API Extensibility
Groovy Code API
Embedded OSGi framework
Extensibility - CloverETL Engine Plugins
Clustering
High Availability
Scalability
Transformation Requests
Parallel Data Processing
Graph allocation examples
Recommendations for Cluster Deployment
Example of Distributed Execution
Details of the Example Transformation Design
Scalability of the Example Transformation
Cluster configuration
Mandatory properties
Optional properties
Example of 2 node cluster configuration
Load balancing properties
Temp Space Management
Overview
Setup
Adding Temp Space