CopyFiles

Jobflow Component

Short Description
Ports
Metadata
CopyFiles Attributes
Details
Examples
See also

Short Description

CopyFiles can be used to copy files and directories.

CopyFiles can copy multiple sources into one destination directory, or a regular source file to a target file. Directories can be copied recursively. Optionally, existing files may be skipped or updated based on the modification date of the files.

[Note]Note

To be able to use this component, you need a separate jobflow license.

Component Inputs Outputs Auto-propagated metadata
CopyFiles0-10-2
yes

Ports

Port typeNumberRequiredDescriptionMetadata
Input0
no
Input data records to be mapped to component attributes.Any
Output0
no
ResultsAny
1
no
ErrorsAny

Metadata

CopyFiles does not propagate metadata from left to right or from right to left.

This component has metadata template available. See Details for details on template fields of CopyFiles or Metadata Templates for general details on metadata templates.

CopyFiles Attributes

AttributeReqDescriptionPossible values
Basic
Source file URLyes [1] Path to the source file or directory (see Supported URL Formats for File Operations).  
Target file URLyes [1] Path to the destination file or directory (see Supported URL Formats for File Operations). When it points to a directory, the source will be copied into the directory.

It must be a path to a single file or directory.

 
RecursivenoCopies directories recursively.false (default) | true
Overwriteno

Specifies whether existing files shall be overwritten.

In always mode, the target will be overwritten.

In update mode, the target will be overwritten only when the source file is newer than the destination file.

In never mode, the target will not be overwritten.

always (default) | update | never
Create parent directoriesnoAttempts to create non-existing parent directories.

When the Create parent directories option is enabled and the Target file URL ends with a slash ('/'), it is treated as the parent directory, i.e. the source directory or file is copied into the target directory, even if it does not exist.

false (default) | true
Input mapping [2] Defines mapping of input records to component attributes.
Output mapping [2] Defines mapping of results to the standard output port.
Error mapping [2] Defines mapping of errors to the error output port.
Redirect error outputnoIf enabled, errors will be sent to the output port instead of the error port. false (default) | true
Verbose outputnoIf enabled, one input record may cause multiple records to be sent to the output (e.g. as a result of wildcard expansion). Otherwise, each input record will yield just one cumulative output record. false (default) | true
Advanced
Stop processing on failno By default, a failure causes the component to skip all subsequent input records and send the information about skipped input records to the error output port. This behavior can be turned off by this attribute. true (default) | false

[1]  The attribute is required, unless specified in the Input mapping.

[2]  Required if the corresponding edge is connected.

Details

Editing any of the Input, Output or Error mapping opens the Transform Editor.

Input mapping

The editor allows you to override selected attributes of the component with the values of the input fields.

Field NameAttributeTypePossible values
sourceURLSource file URLstring 
targetURLTarget file URLstring 
recursiveRecursivebooleantrue | false
overwriteOverwritestring"always" | "update" | "never"
makeParentDirsCreate parent directoriesbooleantrue | false
Output mapping

The editor allows you to map the results and the input data to the output port.

If Output mapping is empty, fields of the input record and result record are mapped to the output by name.

Field NameTypeDescription
sourceURLstringURL of the source file.
targetURLstringURL of the destination.
resultURLstringA new URL of the successfully copied file. Only set in Verbose output mode.
resultbooleanTrue if the operation has succeeded (can be false when Redirect error output is enabled).
errorMessagestringIf the operation has failed, the field contains the error message (used when Redirect error output is enabled).
stackTracestringIf the operation has failed, the field contains the stack trace of the error (used when Redirect error output is enabled).
Error mapping

The editor allows you to map the errors and the input data to the error port.

If Error mapping is empty, fields of the input record and result record are mapped to the output by name.

Field NameTypeDescription
resultbooleanWill always be set to false.
errorMessagestringThe error message.
stackTracestringThe stack trace of the error.
sourceURLstringURL of the source file.
targetURLstringURL of the destination.

Stop processing on fail

If you get file URLs from the input port having Stop processing on fail set to true and error occurs, the record causing error and the following ones are skipped.

Examples

Copying Files from one Directory to Another
Using Stop Processing on Fail
Non-recursive copying directory with mixed content

Copying Files from one Directory to Another

Copy .txt files from ${DATAIN_DIR}/FO/ to ${DATAOUT_DIR}/FO/. The target directory may not exist. If the files in the target directory exist, they should be overwritten.

Solution

Use the attributes Source file URL, Target file URL and Create parent directories.

AttributeValue
Source file URL${DATAIN_DIR}/FO/*.txt
Target file URL${DATAOUT_DIR}/FO/
Create parent directoriestrue

Using Stop Processing on Fail

Copy files f1.txt f2.txt and f3.txt. If you cannot copy a file, continue with the following one.

-rw-rw-r--. 1 clover clover 0 Feb 23 18:11 f1.txt
-rw-------. 1 user23 user23 0 Feb 23 18:11 f2.txt
-rw-rw-r--. 1 clover clover 0 Feb 23 18:11 f3.txt
Solution with Source File URL attribute

The solution without reading file URLs from an input edge is same as in the previous example. Use the attributes Source file URL, Target file URL and Create parent directories.

Note that only files f1.txt and f2.txt will be copied. The file f2.txt cannot be copied as the user 'clover' does not have read access to the file.

Solution with Input Edge

If you read URLs of files from an input edge one by one, use the attributes Target file URL, Create parent directories Input mapping and Stop processing on fail.

AttributeValue
Target file URL${DATAOUT_DIR}/FO/
Create parent directoriestrue
Input mapping 
Stop processing on failfalse

In Input mapping, you map Source URL.

Note that you need to uncheck the attribute Stop processing on fail. Otherwise only the file f1.txt would be copied. (Assuming that the files come in ascending order.)

Non-recursive copying directory with mixed content

This example shows non-recursive copying of files from one directory to another.

Copy files and directories from ${DATAIN_DIR}/abc to ${DATAOUT_DIR}/def. If a directory in ${DATAIN_DIR}/abc contains a file or a directory, these files and directories should not be copied.

Source directory

${DATAIN_DIR}/abc/file.txt
${DATAIN_DIR}/abc/dirA/file2.txt

Expected result

${DATAIN_DIR}/def/file.txt
${DATAIN_DIR}/def/dirA
Solution

Use ListFiles to list the files and directories to be copied. In ListFiles set File URL attribute to the source directory.

AttributeValue
File URL${DATAIN_DIR}/abc/

In CopyFiles, copy the files and directories from the list received from an input edge.

AttributeValue
Target file URL${DATAIN_DIR}/def/
Create parent directoriestrue
Input mapping
//#CTL2

// Transforms input record into output record.
function integer transform() {
	$out.0.sourceURL = $in.0.URL;

	return ALL;
}
[Note]Note

The ListFiles component is necessary to list all files and directories. If you non-recursively copied the files with CopyFiles only, the copying would stop when a first subdirectory is reached.

See also

MoveFiles
Common Properties of Components
Specific Attribute Types
Common Properties of File Operations
File Operations Comparison