I am using the "S3 File Output" step in transformations in Pentaho Data Integration 5.1.0. It works correctly when I specify the AWS access key and secret key using the "Access Key" and "Secret Key" fields in the settings. It also works with I specify the keys in the Filename field using the "s3://access_key:secret_key@s3/bucket/path/file" URL format.
Now for the problem. I have defined variables for the access and secret keys in my kettle.properties file. When I try to use them in either the two key fields or embedded in the Filename field, it consistently fails with a "Could not determine the type of file" error, which is thrown whenever authentication fails. I have verified that the username and password in the variables are correct--it simply refuses to use the variables correctly. It also fails when I try to use just a variable for the access key. The access key contains just alphanumeric characters, so the problem does not involve incorrect URL encoding of the key. I'd be eager to hear if a workaround is available. Note that this problem is different from bug PDI-10113, in which the key fields did not work at all. The behavior I'm seeing shows that this bug was indeed fixed in this release.
Here is an example of the log output (with redacted S3 URLs):
2014/11/15 04:01:25 - S3 File Output.0 - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : Couldn't open file s3://s3/bucket/path/file
2014/11/15 04:01:25 - S3 File Output.0 - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : org.pentaho.di.core.exception.KettleException:
2014/11/15 04:01:25 - S3 File Output.0 - Error opening new file : org.apache.commons.vfs.FileSystemException: Could not determine the type of file "s3://s3/bucket/path".
2014/11/15 04:01:25 - S3 File Output.0 -
2014/11/15 04:01:25 - S3 File Output.0 - at org.pentaho.di.trans.steps.textfileoutput.TextFileOutput.openNewFile(TextFileOutput.java:655)
2014/11/15 04:01:25 - S3 File Output.0 - at org.pentaho.di.trans.steps.textfileoutput.TextFileOutput.init(TextFileOutput.java:754)
2014/11/15 04:01:25 - S3 File Output.0 - at org.pentaho.di.trans.step.StepInitThread.run(StepInitThread.java:69)
2014/11/15 04:01:25 - S3 File Output.0 - at java.lang.Thread.run(Thread.java:745)
2014/11/15 04:01:25 - CDW Phoenix to S3 - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : Errors detected!
Now for the problem. I have defined variables for the access and secret keys in my kettle.properties file. When I try to use them in either the two key fields or embedded in the Filename field, it consistently fails with a "Could not determine the type of file" error, which is thrown whenever authentication fails. I have verified that the username and password in the variables are correct--it simply refuses to use the variables correctly. It also fails when I try to use just a variable for the access key. The access key contains just alphanumeric characters, so the problem does not involve incorrect URL encoding of the key. I'd be eager to hear if a workaround is available. Note that this problem is different from bug PDI-10113, in which the key fields did not work at all. The behavior I'm seeing shows that this bug was indeed fixed in this release.
Here is an example of the log output (with redacted S3 URLs):
2014/11/15 04:01:25 - S3 File Output.0 - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : Couldn't open file s3://s3/bucket/path/file
2014/11/15 04:01:25 - S3 File Output.0 - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : org.pentaho.di.core.exception.KettleException:
2014/11/15 04:01:25 - S3 File Output.0 - Error opening new file : org.apache.commons.vfs.FileSystemException: Could not determine the type of file "s3://s3/bucket/path".
2014/11/15 04:01:25 - S3 File Output.0 -
2014/11/15 04:01:25 - S3 File Output.0 - at org.pentaho.di.trans.steps.textfileoutput.TextFileOutput.openNewFile(TextFileOutput.java:655)
2014/11/15 04:01:25 - S3 File Output.0 - at org.pentaho.di.trans.steps.textfileoutput.TextFileOutput.init(TextFileOutput.java:754)
2014/11/15 04:01:25 - S3 File Output.0 - at org.pentaho.di.trans.step.StepInitThread.run(StepInitThread.java:69)
2014/11/15 04:01:25 - S3 File Output.0 - at java.lang.Thread.run(Thread.java:745)
2014/11/15 04:01:25 - CDW Phoenix to S3 - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : Errors detected!