Chapter 2. Integrating a Custom Component in CloverETL Designer

So far we have described how to create your own custom component(s) as a plug-in for CloverETL Engine. Now, we will explain how to integrate the custom component(s) engine plug-in into CloverETL Designer, i.e., create a CloverETL Designer plug-in.

Plug-in Project

First, select FileNewProject...

Selecting Other Project

Figure 2.1. Selecting Other Project


Choose Plug-in Project and click Next.

Creating a Plug-in Project

Figure 2.2. Creating a Plug-in Project


A window displays. Type the project name in the proper field. This will be the plug-in name. Here, we named the project org.company.gui.components. The name was derived from the CloverETL Engine plug-in by interposing "gui".

Giving a Name to the Plug-in Project

Figure 2.3. Giving a Name to the Plug-in Project


Click Next, fill in the plug-in properties (can be later modified). Plugin we create in this tutorial will not need the Activator class, nor will it make contributions to the UI, so you can uncheck these option in the Options groupbox, but it's safe to leave them checked.

Plug-in properties and options

Figure 2.4. Plug-in properties and options


Click Finish button and confirm opening the Plug-in Development perspective.

Opening the Plug-in Development Perspective

Figure 2.5. Opening the Plug-in Development Perspective


Component XML Definition

Custom component definition file is a standard XML file with the structure based on following DTD. Its purpose is to define components input and output ports, parameters which configure component, and other necessary info for CloverETL Designer.

  <!ELEMENT ETLComponentList (ETLComponent*)>
  
  <!ELEMENT ETLComponent (shortDescription, description, inputPorts?, 
                             outputPorts?, properties)>
  <!ATTLIST ETLComponent
	    category (readers | writers | transformers | others | deprecated | 
                             joiners) #REQUIRED
	    className NMTOKEN #REQUIRED
	    iconPath CDATA #IMPLIED
	    name CDATA #REQUIRED
	    smallIconPath CDATA #IMPLIED
	    type NMTOKEN #REQUIRED
	    passThrough (true|false) #IMPLIED
	    defaultVisibility (true|false) #IMPLIED
	    >
	  
  <!ELEMENT shortDescription (#PCDATA)>
	  
  <!ELEMENT description (#PCDATA)>
	  
  <!ELEMENT inputPorts ((singlePort*,multiplePort?)|multiplePort)>
	  
  <!ELEMENT outputPorts (singlePort*|multiplePort)>
	  
  <!ELEMENT properties (property*)>
  
  <!ELEMENT singlePort EMPTY>
  <!ATTLIST singlePort
	    name NMTOKEN #REQUIRED
	    required (true|false) #IMPLIED
	    label CDATA #IMPLIED
	    >
	  
  <!ELEMENT multiplePort EMPTY>
  <!ATTLIST multiplePort
	    required (true|false) #IMPLIED
	    label NMTOKEN #IMPLIED
	    >
	  
  <!ELEMENT property (singleType | enumType | keyType)>
  <!ATTLIST property
	    category (clover|basic|advanced|deprecated) #REQUIRED
	    displayName CDATA #REQUIRED
	    modifiable (true | false) #REQUIRED
	    name NMTOKEN #REQUIRED
	    nullable (true | false) #IMPLIED
	    defaultValue CDATA #IMPLIED
	    required CDATA #IMPLIED
	    >
	  
  <!ELEMENT singleType EMPTY>
  <!ATTLIST singleType
	    name NMTOKEN #REQUIRED
	    inputPortName NMTOKEN #IMPLIED
	    type NMTOKEN #IMPLIED
	    outputPortName NMTOKEN #IMPLIED
	    mappingType NMTOKEN #IMPLIED
	    master NMTOKEN #IMPLIED
	    title CDATA #IMPLIED
	    leftLabel CDATA #IMPLIED
	    rightLabel CDATA #IMPLIED
	    labels CDATA #IMPLIED
	    keyType NMTOKEN #IMPLIED
	    size NMTOKEN #IMPLIED
	    min NMTOKEN #IMPLIED
	    max NMTOKEN #IMPLIED
	    >
  
  <!ELEMENT enumType (item+)>
	  
  <!ELEMENT keyType EMPTY>
  <!ATTLIST keyType
	    inputPortName NMTOKEN #REQUIRED
	    >
	  
  <!ELEMENT item EMPTY>
  <!ATTLIST item
	    value NMTOKEN #REQUIRED
	    displayValue NMTOKENS #REQUIRED
  >

Here is an example of our NewComponent XML definition file customcomponents.xml with the Dedup component properties demonstration in comments. This XML file should be placed in the root directory of the org.company.gui.components plug-in project.

<?xml version="1.0" encoding="UTF-8"?>

<ETLComponentList>

    <ETLComponent
        category="Custom"
        className="org.company.gui.components.NewComponent"
        name="My Component"
        type="NEW_COMPONENT"
        iconPath=
            "platform:/plugin/org.company.gui.components/icons/scopy32.gif"
        smallIconPath=
            "platform:/plugin/org.company.gui.components/icons/simpleCopy16.png"
        passThrough="true">

        <shortDescription>A sample custom component</shortDescription>
        <description>Receives data through connected input port and copies data 
                     records to all connected output ports.</description>

        <inputPorts>
            <singlePort name="0" required="true" />
        </inputPorts>
        <outputPorts>
            <multiplePort required="true" />
        </outputPorts>

        <properties>
        <!--
            <property category="basic" displayName="Dedup key" modifiable="true" name="dedupKey" 
                      nullable="true">
               <singleType name="key" inputPortName="0" keyType="orderVerification" />
            </property>
            <property category="basic" displayName="Keep" modifiable="true" name="keep" 
                      nullable="true" defaultHint="First">
               <singleType name="keep" />
            </property>
            <property category="basic" displayName="Equal NULL" modifiable="true" name="equalNULL" 
                      nullable="true" defaultHint="true">
               <singleType name="bool" />
            </property>
            <property category="basic" displayName="Number of duplicates" modifiable="true" 
                      name="noDupRecord" nullable="true" defaultHint="1">
               <singleType name="int" />
            </property>
           -->
        </properties>

    </ETLComponent>

</ETLComponentList>

The category attribute specifies the Palette of Components category, the component displays under. It can be one of the existing ones (Readers, Writers, Transformers, Joiners or Others) or your own (Custom in our example).

The proper className attribute must be a combination of your CloverETL Designer plugin name (org.company.gui.components) and the component class (NewComponent).

The name attribute specifies the component label displayed in CloverETL Designer.

The type attribute must correspond to COMPONENT_TYPE variable defined in your component Java class file (NEW_COMPONENT in our example).

There are several ways how to define the path(s) to component icons. Each component applies icons of three different sizes (16x16, 32x32, and 64x64 pixels). We recommend to name the files so that they have the same prefix and vary only in the size specificator, e.g. NewComponent16.png, NewComponent32.png, NewComponent64.png. Then, you can specify just the iconpath attribute, e.g., supposing all the icons will be located in the icons folder within your CloverETL Designer plug-in, the path will be:

iconpath="icons/NewComponent"

Another option is to specify the iconpath attribute for the 32x32 icon and the smallIconPath attribute for the 16x16 icon. In this case, the path must include the complete filename. Additionally, as the 64x64 icon can not be specified this way, it will be substituted by the 32x32 icon. There is also a possibility of applying some of the CloverETL icons. In this case, the icon path begins with platform:/plugins/ followed by the name of the plug-in, the icon is located in, and the path to the icon within this plugin.

In the properties tag, component attributes are specified. As our NewComponent has no parameters, the parameters of the Dedup component are given to illustrate the tag structure.

Plug-in Dependencies

The Dependencies page shows the dependencies that your plug-in has on other plug-ins. On this page, you must list all plug-ins that contribute code to your plug-in project and that are required to be on your project's classpath in order to compile. When you modify the list of dependencies and save the file, your classpath will be updated automatically.

Switch to the Dependencies tab and press Add... button to browse the list of all plug-ins.

Adding Dependencies

Figure 2.6. Adding Dependencies


Type the letter c and select com.cloveretl.gui plug-in.

Appearance of the Plug-in Selection Wizard

Figure 2.7. Appearance of the Plug-in Selection Wizard


The added plug-in displays in the Dependencies page:

Dependency Plugin Has Been Added

Figure 2.8. Dependency Plugin Has Been Added


Plug-in Extensions

To register with the CloverETL Designer, your plug-in must define extensions at the following CloverETL Designer extension points:

com.cloveretl.gui.component

com.cloveretl.gui.enginePlugin

To set the relationships, go to Extensions tab and press Add....Select the com.cloveretl.gui.component item and click Finish.

Selecting the Component Extension

Figure 2.9. Selecting the Component Extension


After com.cloveretl.gui.component has been added, set the file property to the customcomponents.xml component definition file, and the classLoaderProvider field to the org.company.gui.CLProvider class name.

Specifying the Extension File

Figure 2.10. Specifying the Extension File


Now, before adding new extension of the extension point com.cloveretl.gui.enginePlugin, new Java class is needed. This class will provide class loader of your plugin to the CloverETL Designer, which is needed for propper initialization of your componet plugin in the CloverETL Engine. You don't need to bother with details, just folow these steps: right-click the org.company.gui.components project, and select NewClass from the context menu. Enter package name of new class, lets use the name org.company.gui, and enter class name CLProvider. Also, fill in the Superclass field with org.cloveretl.gui.plugin.engine.AbstractClassLoaderProvider class name. That's it! Leave body of the class empty, everything is handled by the superclass. The essential thing is that this class is located in your plugin.

OK, let's create the engine plugin extension: press Add... again, select the com.cloveretl.gui.enginePlugin item and click Finish.

Selecting the Engine Plugin Extension

Figure 2.11. Selecting the Engine Plugin Extension


After com.cloveretl.gui.enginePlugin has been added, set the directory property to the plugins folder:

Specifying the Engine Plugin Extension Directory

Figure 2.12. Specifying the Engine Plugin Extension Directory


Save the settings by clicking Ctrl+S.

Import Requisities

Now, you have to import the previously created engine plug-in.

First, create the plugins folder by right-clicking the org.company.gui.components project name and selecting NewFolder in the context menu.

Creating a New Folder

Figure 2.13. Creating a New Folder


Set the Folder name to plugins and click Finish.

Right-click the plugins item and select Import... from the context menu.

Importing Content to the Plugins Folder

Figure 2.14. Importing Content to the Plugins Folder


Expand the General item, select File System and click Next.

Searching for the Content

Figure 2.15. Searching for the Content


Select your workspace folder and expand its item:

Locating the Engine Plugin Folder

Figure 2.16. Locating the Engine Plugin Folder


Expand and select the org.company.components item, check its checkbox and uncheck both .classpath and .project files on the right. Also uncheck all child items of the org.company.components item except the bin folder. Click Finish.

Selecting the Content of the Engine Plugin Extension Directory

Figure 2.17. Selecting the Content of the Engine Plugin Extension Directory


Plug-in Building

Now switch to the Build tab. Check the icons, plugins, customcomponents.xml, plugin.xml and META-INF items.

Creating a Build XML File

Figure 2.18. Creating a Build XML File


Then right-click the plugin.xml item and select PDE ToolsCreate Ant Build File from the context menu. After that, a new build.xml item appears in the Navigator pane. Right-click this item and select Run AsAnt Build... from the context menu. Uncheck build.jars, check zip.plugin, and press the Run button.

Running the Build

Figure 2.19. Running the Build


The console output will inform you whether the build was successful:

Build Has Been Successful

Figure 2.20. Build Has Been Successful


Click the org.company.gui.components item and then F5. It will refresh the file system and a new .jar file will appear in the Navigator pane.

Refreshing the File Structure

Figure 2.21. Refreshing the File Structure


The new .jar file (org.company.gui.components_1.0.0.jar) is the plug-in for CloverETL Designer.

[Important]Important

As Eclipse reads the information about its plug-ins from the cache first (restarting Eclipse with -clean option does not work in this case), it is necessary to unzip your custom component plug-in into a fresh Eclipse copy. The best way is to not restart the Eclipse when prompted after installing CloverETL Designer plug-in, but close it and unzip the custom component plug-in into plugins. Also, we recommend to save the configuration folder now. When you need a clean copy of Eclipse, you will just replace the configuration folder by the one from the clean Eclipse. Now you can start the Eclipse.

Troubleshooting

If your plugins seem not to be loaded by the CloverETL Designer, you may want to check logging messages generated during Designer startup. To get access to these, redirect standard and error output of the Designer into files. On Windows, you achieve this by executing the following command in the installation directory of the CloverETL Designer:

CloverETLDesigner.exe -clean > std.log 2> err.log

After the Designer is started, splash screen appears, all the Designer plugins are being loaded and once the splash screen disappears, you can examine the err.log file which should be empty if everything went well, or should contain (hopefully useful) error message. The std.log contains info about successfully loaded plugins – one of the "Component extension plugin found:" lines should be followed by a line with ID of your Designer plugin. There should be also "Engine plugin extension plugin found:" line followed by your engine plugin ID, plugins directory and classloader provider class name.