Hi all,
I am having trouble with the "S3 File Output" step in Pentaho 5.3 when running on Linux/CentOS.
I have a 2-step PoC transformation that pushes a bunch of fields into Gz-compressed text file in my S3 bucket. This works fine when running in Windows 7. But when running in Linux I get the following:
Code:
2016/12/22 17:03:34 - TestArea S3 PoC - ERROR (version 5.3-SNAPSHOT, build 1 from 2015-04-23 13.43.15 by mgt) : org.pentaho.di.core.exception.KettleException:
2016/12/22 17:03:34 - TestArea S3 PoC - Unexpected error during transformation metadata load
2016/12/22 17:03:34 - TestArea S3 PoC -
2016/12/22 17:03:34 - TestArea S3 PoC - Missing plugins found while loading a transformation
2016/12/22 17:03:34 - TestArea S3 PoC -
2016/12/22 17:03:34 - TestArea S3 PoC - Step : S3FileOutputPlugin
2016/12/22 17:03:34 - TestArea S3 PoC -
2016/12/22 17:03:34 - TestArea S3 PoC - at org.pentaho.di.job.entries.trans.JobEntryTrans.getTransMeta(JobEntryTrans.java:1205)
2016/12/22 17:03:34 - TestArea S3 PoC - at org.pentaho.di.job.entries.trans.JobEntryTrans.execute(JobEntryTrans.java:648)
2016/12/22 17:03:34 - TestArea S3 PoC - at org.pentaho.di.job.Job.execute(Job.java:716)
2016/12/22 17:03:34 - TestArea S3 PoC - at org.pentaho.di.job.Job.execute(Job.java:859)
2016/12/22 17:03:34 - TestArea S3 PoC - at org.pentaho.di.job.Job.execute(Job.java:859)
2016/12/22 17:03:34 - TestArea S3 PoC - at org.pentaho.di.job.Job.execute(Job.java:859)
2016/12/22 17:03:34 - TestArea S3 PoC - at org.pentaho.di.job.Job.execute(Job.java:598)
2016/12/22 17:03:34 - TestArea S3 PoC - at org.pentaho.di.job.entries.job.JobEntryJobRunner.run(JobEntryJobRunner.java:69)
2016/12/22 17:03:34 - TestArea S3 PoC - at java.lang.Thread.run(Thread.java:745)
2016/12/22 17:03:34 - TestArea S3 PoC - Caused by: org.pentaho.di.core.exception.KettleMissingPluginsException:
2016/12/22 17:03:34 - TestArea S3 PoC - Missing plugins found while loading a transformation
2016/12/22 17:03:34 - TestArea S3 PoC -
2016/12/22 17:03:34 - TestArea S3 PoC - Step : S3FileOutputPlugin
2016/12/22 17:03:34 - TestArea S3 PoC - at org.pentaho.di.trans.TransMeta.loadXML(TransMeta.java:2835)
2016/12/22 17:03:34 - TestArea S3 PoC - at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2671)
2016/12/22 17:03:34 - TestArea S3 PoC - at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2623)
2016/12/22 17:03:34 - TestArea S3 PoC - at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2600)
2016/12/22 17:03:34 - TestArea S3 PoC - at org.pentaho.di.job.entries.trans.JobEntryTrans.getTransMeta(JobEntryTrans.java:1154)
2016/12/22 17:03:34 - TestArea S3 PoC - ... 8 more
2016/12/22 17:03:34 - TestArea S3 PoC - ERROR (version 5.3-SNAPSHOT, build 1 from 2015-04-23 13.43.15 by mgt) : org.pentaho.di.core.exception.KettleException:
2016/12/22 17:03:34 - TestArea S3 PoC - Unexpected error occurred while launching entry [CREATE GZ FILE ON S3.0]
2016/12/22 17:03:34 - TestArea S3 PoC -
2016/12/22 17:03:34 - TestArea S3 PoC - Unexpected error during transformation metadata load
2016/12/22 17:03:34 - TestArea S3 PoC -
2016/12/22 17:03:34 - TestArea S3 PoC - Missing plugins found while loading a transformation
2016/12/22 17:03:34 - TestArea S3 PoC -
2016/12/22 17:03:34 - TestArea S3 PoC - Step : S3FileOutputPlugin
2016/12/22 17:03:34 - TestArea S3 PoC -
2016/12/22 17:03:34 - TestArea S3 PoC -
2016/12/22 17:03:34 - TestArea S3 PoC - at org.pentaho.di.job.Job.execute(Job.java:862)
2016/12/22 17:03:34 - TestArea S3 PoC - at org.pentaho.di.job.Job.execute(Job.java:859)
2016/12/22 17:03:34 - TestArea S3 PoC - at org.pentaho.di.job.Job.execute(Job.java:859)
2016/12/22 17:03:34 - TestArea S3 PoC - at org.pentaho.di.job.Job.execute(Job.java:598)
2016/12/22 17:03:34 - TestArea S3 PoC - at org.pentaho.di.job.entries.job.JobEntryJobRunner.run(JobEntryJobRunner.java:69)
2016/12/22 17:03:34 - TestArea S3 PoC - at java.lang.Thread.run(Thread.java:745)
2016/12/22 17:03:34 - TestArea S3 PoC - Caused by: org.pentaho.di.core.exception.KettleException:
2016/12/22 17:03:34 - TestArea S3 PoC - Unexpected error during transformation metadata load
2016/12/22 17:03:34 - TestArea S3 PoC -
2016/12/22 17:03:34 - TestArea S3 PoC - Missing plugins found while loading a transformation
2016/12/22 17:03:34 - TestArea S3 PoC -
2016/12/22 17:03:34 - TestArea S3 PoC - Step : S3FileOutputPlugin
2016/12/22 17:03:34 - TestArea S3 PoC -
2016/12/22 17:03:34 - TestArea S3 PoC - at org.pentaho.di.job.entries.trans.JobEntryTrans.getTransMeta(JobEntryTrans.java:1205)
2016/12/22 17:03:34 - TestArea S3 PoC - at org.pentaho.di.job.entries.trans.JobEntryTrans.execute(JobEntryTrans.java:648)
2016/12/22 17:03:34 - TestArea S3 PoC - at org.pentaho.di.job.Job.execute(Job.java:716)
2016/12/22 17:03:34 - TestArea S3 PoC - at org.pentaho.di.job.Job.execute(Job.java:859)
2016/12/22 17:03:34 - TestArea S3 PoC - ... 5 more
2016/12/22 17:03:34 - TestArea S3 PoC - Caused by: org.pentaho.di.core.exception.KettleMissingPluginsException:
2016/12/22 17:03:34 - TestArea S3 PoC - Missing plugins found while loading a transformation
2016/12/22 17:03:34 - TestArea S3 PoC -
2016/12/22 17:03:34 - TestArea S3 PoC - Step : S3FileOutputPlugin
2016/12/22 17:03:34 - TestArea S3 PoC - at org.pentaho.di.trans.TransMeta.loadXML(TransMeta.java:2835)
2016/12/22 17:03:34 - TestArea S3 PoC - at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2671)
2016/12/22 17:03:34 - TestArea S3 PoC - at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2623)
2016/12/22 17:03:34 - TestArea S3 PoC - at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2600)
2016/12/22 17:03:34 - TestArea S3 PoC - at org.pentaho.di.job.entries.trans.JobEntryTrans.getTransMeta(JobEntryTrans.java:1154)
2016/12/22 17:03:34 - TestArea S3 PoC - ... 8 more
2016/12/22 17:03:34 - TestArea S3 PoC - ERROR (version 5.3-SNAPSHOT, build 1 from 2015-04-23 13.43.15 by mgt) : org.pentaho.di.core.exception.KettleException:
2016/12/22 17:03:34 - TestArea S3 PoC - Unexpected error occurred while launching entry [Get Messages & Property Detail.0]
2016/12/22 17:03:34 - TestArea S3 PoC -
2016/12/22 17:03:34 - TestArea S3 PoC - Unexpected error occurred while launching entry [CREATE GZ FILE ON S3.0]
2016/12/22 17:03:34 - TestArea S3 PoC -
2016/12/22 17:03:34 - TestArea S3 PoC - Unexpected error during transformation metadata load
2016/12/22 17:03:34 - TestArea S3 PoC -
2016/12/22 17:03:34 - TestArea S3 PoC - Missing plugins found while loading a transformation
2016/12/22 17:03:34 - TestArea S3 PoC -
2016/12/22 17:03:34 - TestArea S3 PoC - Step : S3FileOutputPlugin
2016/12/22 17:03:34 - TestArea S3 PoC -
2016/12/22 17:03:34 - TestArea S3 PoC -
2016/12/22 17:03:34 - TestArea S3 PoC -
2016/12/22 17:03:34 - TestArea S3 PoC - at org.pentaho.di.job.Job.execute(Job.java:862)
2016/12/22 17:03:34 - TestArea S3 PoC - at org.pentaho.di.job.Job.execute(Job.java:859)
2016/12/22 17:03:34 - TestArea S3 PoC - at org.pentaho.di.job.Job.execute(Job.java:598)
2016/12/22 17:03:34 - TestArea S3 PoC - at org.pentaho.di.job.entries.job.JobEntryJobRunner.run(JobEntryJobRunner.java:69)
2016/12/22 17:03:34 - TestArea S3 PoC - at java.lang.Thread.run(Thread.java:745)
2016/12/22 17:03:34 - TestArea S3 PoC - Caused by: org.pentaho.di.core.exception.KettleException:
2016/12/22 17:03:34 - TestArea S3 PoC - Unexpected error occurred while launching entry [CREATE GZ FILE ON S3.0]
2016/12/22 17:03:34 - TestArea S3 PoC -
2016/12/22 17:03:34 - TestArea S3 PoC - Unexpected error during transformation metadata load
2016/12/22 17:03:34 - TestArea S3 PoC -
2016/12/22 17:03:34 - TestArea S3 PoC - Missing plugins found while loading a transformation
2016/12/22 17:03:34 - TestArea S3 PoC -
2016/12/22 17:03:34 - TestArea S3 PoC - Step : S3FileOutputPlugin
2016/12/22 17:03:34 - TestArea S3 PoC -
2016/12/22 17:03:34 - TestArea S3 PoC -
2016/12/22 17:03:34 - TestArea S3 PoC - at org.pentaho.di.job.Job.execute(Job.java:862)
2016/12/22 17:03:34 - TestArea S3 PoC - at org.pentaho.di.job.Job.execute(Job.java:859)
2016/12/22 17:03:34 - TestArea S3 PoC - ... 4 more
2016/12/22 17:03:34 - TestArea S3 PoC - Caused by: org.pentaho.di.core.exception.KettleException:
2016/12/22 17:03:34 - TestArea S3 PoC - Unexpected error during transformation metadata load
2016/12/22 17:03:34 - TestArea S3 PoC -
2016/12/22 17:03:34 - TestArea S3 PoC - Missing plugins found while loading a transformation
2016/12/22 17:03:34 - TestArea S3 PoC -
2016/12/22 17:03:34 - TestArea S3 PoC - Step : S3FileOutputPlugin
2016/12/22 17:03:34 - TestArea S3 PoC -
2016/12/22 17:03:34 - TestArea S3 PoC - at org.pentaho.di.job.entries.trans.JobEntryTrans.getTransMeta(JobEntryTrans.java:1205)
2016/12/22 17:03:34 - TestArea S3 PoC - at org.pentaho.di.job.entries.trans.JobEntryTrans.execute(JobEntryTrans.java:648)
2016/12/22 17:03:34 - TestArea S3 PoC - at org.pentaho.di.job.Job.execute(Job.java:716)
2016/12/22 17:03:34 - TestArea S3 PoC - at org.pentaho.di.job.Job.execute(Job.java:859)
2016/12/22 17:03:34 - TestArea S3 PoC - ... 5 more
2016/12/22 17:03:34 - TestArea S3 PoC - Caused by: org.pentaho.di.core.exception.KettleMissingPluginsException:
2016/12/22 17:03:34 - TestArea S3 PoC - Missing plugins found while loading a transformation
2016/12/22 17:03:34 - TestArea S3 PoC -
2016/12/22 17:03:34 - TestArea S3 PoC - Step : S3FileOutputPlugin
2016/12/22 17:03:34 - TestArea S3 PoC - at org.pentaho.di.trans.TransMeta.loadXML(TransMeta.java:2835)
2016/12/22 17:03:34 - TestArea S3 PoC - at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2671)
2016/12/22 17:03:34 - TestArea S3 PoC - at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2623)
2016/12/22 17:03:34 - TestArea S3 PoC - at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2600)
2016/12/22 17:03:34 - TestArea S3 PoC - at org.pentaho.di.job.entries.trans.JobEntryTrans.getTransMeta(JobEntryTrans.java:1154)
2016/12/22 17:03:34 - TestArea S3 PoC - ... 8 more
org.pentaho.di.core.exception.KettleException:
It sounds like this is related to the following issue:
http://jira.pentaho.com/browse/PDI-13575
(Despite being a different plugin)
In the abovementioned JIRA, a workaround is suggested as:
"As a workaround - the one you already found out - is to call the kitchen/pan from inside the folder the scripts are located."
In the documentation subtask on that page, there is also a sample Shell Script for Spoon named "spoon_5.3.sh".
Looking in this file, the only difference I can see between that and my spoon.sh (which I don't have permission to modify) is that in my spoon.sh I have an extra case statement under "setPentahoEnv" in the "Init BASEDIR" section as follows:
Code:
# **************************************************
# ** Init BASEDIR **
# **************************************************
BASEDIR=`dirname $0`
cd $BASEDIR
DIR=`pwd`
cd -
. "$DIR/set-pentaho-env.sh"
setPentahoEnv
# Spoon really needs to be in the "right" directory
case "$1" in
# These are non-spoon calls
-main)
;;
# This is actually spoon
*)
cd $DIR
BASEDIR=.
;;
esac
The kitchen.sh script I am using to access this looks like this:
Code:
#!/bin/sh
BASEDIR="`dirname $0`"
cd "$BASEDIR"
DIR="`pwd`"
cd - > /dev/null
if [ "$1" = "-x" ]; then
set LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$BASEDIR/lib
export LD_LIBRARY_PATH
export OPT="-Xruntracer $OPT"
shift
fi
export IS_KITCHEN="true"
"$DIR/spoon.sh" -main org.pentaho.di.kitchen.Kitchen "$@"
I have interpreted the workaround such that the guts of my shell script (/home/pentaho/testarea/run_s3test.sh) looks like this:
Code:
cd /home/pentaho/testarea
KETTLE_HOME=/home/pentaho/testarea \
PENTAHO_MEM=15G \
/home/pentaho/kettle/kitchen.sh \
/file:///home/pentaho/testarea/s3_poc.kjb \
> logs/s3_poc.`date +%Y%m%d`.log 2>&1
Questions:
1) Is this indeed the same issue as assumed above or is it a separate issue.
2) If this is the same issue, then can someone show me how I can modify my run_s3test.sh shell script to make it work? :)
(If it's the same issue, then my interpretation of the fix is wrong because it ain't workin)
Thanks!!!
Regards,
Stanbridge