Hi,
I am using a CSV file to copy gzipped file to S3 bucket. Even with files as small as 7MB I sometimes get "Pipe not connected" error in the log and the kettle job terminates without returning anything. The kettle job is being called via xaction which is scheduled to run every night. This doesn't happen every night but intermittently - is there a way to capture this error in S3 File Output step so I can attempt to run it again? I am using Kettle-Spoon 4.4.2.1 and Pentaho 4.8.2.1
INFO 13-08 10:41:21,245 - CSV file input - Starting to run...
INFO 13-08 10:41:21,245 - CSV file input - Header row skipped in file '/file_path/file_name.csv'
INFO 13-08 10:41:21,247 - rs_copy_custom_to_s3 - Transformation has allocated 2 threads and 1 rowsets.
INFO 13-08 10:41:21,247 - S3 File Output - Starting to run...
INFO 13-08 10:41:21,612 - S3 File Output - Parent folder exists [s3://access_key:secret_key@s3/bucket_name]!
INFO 13-08 10:41:21,614 - S3 File Output - Opening output stream in gzipped mode
java.io.IOException: Pipe not connected
INFO 13-08 10:41:21,784 - S3 File Output - Opening output stream in encoding: UTF-8
INFO 13-08 10:41:21,784 - S3 File Output - Opened new file with name [s3://access_key:secret_key@s3/bucket_name/file_name.csv.gz]
at java.io.PipedInputStream.read(Unknown Source)
at java.io.PipedInputStream.read(Unknown Source)
at java.io.InputStream.read(Unknown Source)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1025)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:999)
at org.pentaho.s3.vfs.S3FileObject$1.run(S3FileObject.java:126)
at java.lang.Thread.run(Unknown Source)
In addition, I also see these warnings:
WARN [HttpMethodReleaseInputStream] Attempting to release HttpMethod in finalize() as its response data stream has gone out of scope. This attempt will not always succeed and cannot be relied upon!
Please ensure S3 response data streams are always fully consumed or closed to avoid HTTP connection starvation.
WARN [HttpMethodReleaseInputStream] Successfully released HttpMethod in finalize(). You were lucky this time... Please ensure S3 response data streams are always fully consumed or closed.
Are there any updated pentaho vfs libraries which can be installed to prevent these warnings?
Thanks.
I am using a CSV file to copy gzipped file to S3 bucket. Even with files as small as 7MB I sometimes get "Pipe not connected" error in the log and the kettle job terminates without returning anything. The kettle job is being called via xaction which is scheduled to run every night. This doesn't happen every night but intermittently - is there a way to capture this error in S3 File Output step so I can attempt to run it again? I am using Kettle-Spoon 4.4.2.1 and Pentaho 4.8.2.1
INFO 13-08 10:41:21,245 - CSV file input - Starting to run...
INFO 13-08 10:41:21,245 - CSV file input - Header row skipped in file '/file_path/file_name.csv'
INFO 13-08 10:41:21,247 - rs_copy_custom_to_s3 - Transformation has allocated 2 threads and 1 rowsets.
INFO 13-08 10:41:21,247 - S3 File Output - Starting to run...
INFO 13-08 10:41:21,612 - S3 File Output - Parent folder exists [s3://access_key:secret_key@s3/bucket_name]!
INFO 13-08 10:41:21,614 - S3 File Output - Opening output stream in gzipped mode
java.io.IOException: Pipe not connected
INFO 13-08 10:41:21,784 - S3 File Output - Opening output stream in encoding: UTF-8
INFO 13-08 10:41:21,784 - S3 File Output - Opened new file with name [s3://access_key:secret_key@s3/bucket_name/file_name.csv.gz]
at java.io.PipedInputStream.read(Unknown Source)
at java.io.PipedInputStream.read(Unknown Source)
at java.io.InputStream.read(Unknown Source)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1025)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:999)
at org.pentaho.s3.vfs.S3FileObject$1.run(S3FileObject.java:126)
at java.lang.Thread.run(Unknown Source)
In addition, I also see these warnings:
WARN [HttpMethodReleaseInputStream] Attempting to release HttpMethod in finalize() as its response data stream has gone out of scope. This attempt will not always succeed and cannot be relied upon!
Please ensure S3 response data streams are always fully consumed or closed to avoid HTTP connection starvation.
WARN [HttpMethodReleaseInputStream] Successfully released HttpMethod in finalize(). You were lucky this time... Please ensure S3 response data streams are always fully consumed or closed.
Are there any updated pentaho vfs libraries which can be installed to prevent these warnings?
Thanks.