Chapter 2. Integrating a Custom Component in CloverETL Designer

So far we have described how to create your own custom component(s) as a plug-in for CloverETL Engine. Now, we will explain how to integrate the custom component(s) engine plug-in into CloverETL Designer, i.e., create a CloverETL Designer plug-in.

Plug-in Project

First, select File → New → Project...

Figure 2.1. Selecting Other Project

Choose Plug-in Project and click Next.

Figure 2.2. Creating a Plug-in Project

A window displays. Uncheck the Create a Java project option and type the project name in the proper field. This will be the plug-in name. Here, we named the project org.company.gui.components. The name was derived from the CloverETL Engine plug-in by interposing "gui".

Figure 2.3. Giving a Name to the Plug-in Project

Click Next, fill in the plug-in properties (can be later modified) and confirm opening the Plug-in Development perspective.

Figure 2.4. Opening the Plug-in Development Perspective

Component XML Definition

Custom component definition file is a standard XML file with the structure based on following DTD. Its purpose is to define components input and output ports, parameters which configure component, and other necessary info for CloverETL Designer.

  <!ELEMENT ETLComponentList (ETLComponent*)>
  
  <!ELEMENT ETLComponent (shortDescription, description, inputPorts?, 
                             outputPorts?, properties)>
  <!ATTLIST ETLComponent
	    category (readers | writers | transformers | others | deprecated | 
                             joiners) #REQUIRED
	    className NMTOKEN #REQUIRED
	    iconPath CDATA #IMPLIED
	    name CDATA #REQUIRED
	    smallIconPath CDATA #IMPLIED
	    type NMTOKEN #REQUIRED
	    passThrough (true|false) #IMPLIED
	    defaultVisibility (true|false) #IMPLIED
	    >
	  
  <!ELEMENT shortDescription (#PCDATA)>
	  
  <!ELEMENT description (#PCDATA)>
	  
  <!ELEMENT inputPorts ((singlePort*,multiplePort?)|multiplePort)>
	  
  <!ELEMENT outputPorts (singlePort*|multiplePort)>
	  
  <!ELEMENT properties (property*)>
  
  <!ELEMENT singlePort EMPTY>
  <!ATTLIST singlePort
	    name NMTOKEN #REQUIRED
	    required (true|false) #IMPLIED
	    label CDATA #IMPLIED
	    >
	  
  <!ELEMENT multiplePort EMPTY>
  <!ATTLIST multiplePort
	    required (true|false) #IMPLIED
	    label NMTOKEN #IMPLIED
	    >
	  
  <!ELEMENT property (singleType | enumType | keyType)>
  <!ATTLIST property
	    category (clover|basic|advanced|deprecated) #REQUIRED
	    displayName CDATA #REQUIRED
	    modifiable (true | false) #REQUIRED
	    name NMTOKEN #REQUIRED
	    nullable (true | false) #IMPLIED
	    defaultValue CDATA #IMPLIED
	    required CDATA #IMPLIED
	    >
	  
  <!ELEMENT singleType EMPTY>
  <!ATTLIST singleType
	    name NMTOKEN #REQUIRED
	    inputPortName NMTOKEN #IMPLIED
	    type NMTOKEN #IMPLIED
	    outputPortName NMTOKEN #IMPLIED
	    mappingType NMTOKEN #IMPLIED
	    master NMTOKEN #IMPLIED
	    title CDATA #IMPLIED
	    leftLabel CDATA #IMPLIED
	    rightLabel CDATA #IMPLIED
	    labels CDATA #IMPLIED
	    keyType NMTOKEN #IMPLIED
	    size NMTOKEN #IMPLIED
	    min NMTOKEN #IMPLIED
	    max NMTOKEN #IMPLIED
	    >
  
  <!ELEMENT enumType (item+)>
	  
  <!ELEMENT keyType EMPTY>
  <!ATTLIST keyType
	    inputPortName NMTOKEN #REQUIRED
	    >
	  
  <!ELEMENT item EMPTY>
  <!ATTLIST item
	    value NMTOKEN #REQUIRED
	    displayValue NMTOKENS #REQUIRED
  >

Here is an example of our NewComponent XML definition file customcomponents.xml with the Dedup component properties demonstration in comments.

<?xml version="1.0" encoding="UTF-8"?>

<ETLComponentList>

    <ETLComponent
        category="Custom"
        className="org.company.gui.components.NewComponent"
        name="My Component"
        type="NEW_COMPONENT"
        iconPath=
            "platform:/plugin/org.company.gui.components/icons/scopy32.gif"
        smallIconPath=
            "platform:/plugin/org.company.gui.components/icons/simpleCopy16.png"
        passThrough="true">

        <shortDescription>A sample custom component</shortDescription>
        <description>Receives data through connected input port and copies data 
                     records to all connected output ports.</description>

        <inputPorts>
            <singlePort name="0" required="true" />
        </inputPorts>
        <outputPorts>
            <multiplePort required="true" />
        </outputPorts>

        <properties>
        <!--
            <property category="basic" displayName="Dedup key" modifiable="true" name="dedupKey" 
                      nullable="true">
               <singleType name="key" inputPortName="0" keyType="orderVerification" />
            </property>
            <property category="basic" displayName="Keep" modifiable="true" name="keep" 
                      nullable="true" defaultHint="First">
               <singleType name="keep" />
            </property>
            <property category="basic" displayName="Equal NULL" modifiable="true" name="equalNULL" 
                      nullable="true" defaultHint="true">
               <singleType name="bool" />
            </property>
            <property category="basic" displayName="Number of duplicates" modifiable="true" 
                      name="noDupRecord" nullable="true" defaultHint="1">
               <singleType name="int" />
            </property>
           -->
        </properties>

    </ETLComponent>

</ETLComponentList>

The category attribute specifies the Palette of Components category, the component displays under. It can be one of the existing ones (Readers, Writers, Transformers, Joiners or Others) or your own (Custom in our example).

The proper className attribute must be a combination of your CloverETL Designer plugin name (org.company.gui.components) and the component class (NewComponent).

The name attribute specifies the component label displayed in CloverETL Designer.

The type attribute must correspond to COMPONENT_TYPE variable defined in your component Java class file (NEW_COMPONENT in our example).

There are several ways how to define the path(s) to component icons. Each component applies icons of three different sizes (16x16, 32x32, and 64x64 pixels). We recommend to name the files so that they have the same prefix and vary only in the size specificator, e.g. NewComponent16.png, NewComponent32.png, NewComponent64.png. Then, you can specify just the iconpath attribute, e.g., supposing all the icons will be located in the icons folder within your custom component plug-in, the path will be:

iconpath="icons/NewComponent"

Another option is to specify the iconpath attribute for the 32x32 icon and the smallIconPath attribute for the 16x16 icon. In this case, the path must include the complete filename. Additionaly, as the 64x64 icon can not be spacified this way, it will be substituted by the 32x32 icon. There is also a possibility of applying some of the CloverETL icons. In this case, the icon path begins with platform:/plugins/ followed by the name of the plug-in, the icon is located in, and the path to the icon within this plugin.

In the properties tag, component attributes are specified. As our NewComponent has no parameters, the parameters of the Dedup component are given to illustrate the tag structure.

Plug-in Dependencies

The Dependencies page shows the dependencies that your plug-in has on other plug-ins. On this page, you must list all plug-ins that contribute code to your plug-in project and that are required to be on your project's classpath in order to compile. When you modify the list of dependencies and save the file, your classpath will be updated automatically.

Switch to the Dependencies tab and press Add... button to browse the list of all plug-ins.

Figure 2.5. Adding Dependencies

Type the letter c and select com.cloveretl.gui plug-in.

Figure 2.6. Appearance of the Plug-in Selection Wizard

The added plug-in displays in the Dependencies page:

Figure 2.7. Dependency Plugin Has Been Added

Plug-in Extensions

Your plug-in modifies behaviour of the following CloverETL plug-ins:

com.cloveretl.gui.component

com.cloveretl.gui.enginePlugin

To set the relationships, go to Extensions tab and press Add....Select the com.cloveretl.gui.component item and click Finish.

Figure 2.8. Selecting the Component Extension

After com.cloveretl.gui.component has been added, set the file property to the customcomponents.xml component definition file.

Figure 2.9. Specifying the Extension File

Press Add... again, select the com.cloveretl.gui.enginePlugin item and click Finish.

Figure 2.10. Selecting the Engine Plugin Extension

After com.cloveretl.gui.enginePlugin has been added, set the directory property to the plugins folder:

Figure 2.11. Specifying the Engine Plugin Extension Directory

Save the settings by clicking Ctrl+S.

Import Requisities

Now, you have to import the previously created engine plug-in, the customcomponent.xml file, and icons (if you use your own).

First, create the plugins folder by right-clicking the org.company.gui.components project name and selecting New → Folder in the context menu.

Figure 2.12. Creating a New Folder

Set the Folder name to plugins and click Finish (use the same procedure to create the icons folder if needed).

Right-click the plugins item and select Import... from the context menu.

Figure 2.13. Importing Content to the Plugins Folder

Expand the General item, select File System and click Next.

Figure 2.14. Searching for the Content

Select your workspace folder and expand its item:

Figure 2.15. Locating the Engine Plugin Folder

Select the org.company.components item, check its checkbox and uncheck both .classpath and .project files and click Finish.

Figure 2.16. Selecting the Content of the Engine Plugin Extension Directory

Use the same Import... mechanism for adding customcomponents.xml into project folder and the icons into icons folder.

Plug-in Building

Now switch to the Build tab. Check the icons, plugins, customcomponents.xml, plugin.xml and META-INF items.

Figure 2.17. Creating a Build XML File

Then right-click the plugin.xml item and select PDE Tools → Create Ant Build File from the context menu. After that, a new build.xml item appears in the Navigator pane. Right-click this item and select Run As → Ant Build... from the context menu. Uncheck build.jars, check zip.plugin, and press the Run button.

Figure 2.18. Running the Build

The console output will inform you whether the build was successful:

Figure 2.19. Build Has Been Successful

Click the org.company.gui.components item and then F5. It will refresh the file system and a new .zip file appears in the Navigator pane.

Figure 2.20. Refreshing the File Structure

The new .zip file (org.company.gui.components_1.0.0.zip) is the plug-in for CloverETL Designer.

Important

As Eclipse reads the information about its plug-ins from the cache first (restarting Eclipse with -clean option does not work in this case), it is necessary to unzip your custom component plug-in into a fresh Eclipse copy. The best way is to not restart the Eclipse when prompted after installing CloverETL Designer plug-in, but close it and unzip the custom component plug-in into plugins. Also, we recommend to save the configuration folder now. When you need a clean copy of Eclipse, you will just replace the configuration folder by the one from the clean Eclipse. Now you can start the Eclipse.