Hello all,
We have Amazon Web Service with Amazon EMR up and running.
For building a prototype, I am just trying to use Hadoop Copy Files component to copy one file from local disk to the HDFS.
So I just added a component Hadoop Copy Files and specified the required Input and Output info , but unfortunately the job is failing during execution.
Here below I added the full stack trace.
2014/01/10 10:05:48 - Hadoop Copy Files - ERROR (version 5.0.2, build 1 from 2013-12-04_15-52-25 by buildguy) : Couldn't created parent folder hdfs://machine.compute-1.amazonaws.com:9000/mnt
2014/01/10 10:05:48 - Hadoop Copy Files - ERROR (version 5.0.2, build 1 from 2013-12-04_15-52-25 by buildguy) : org.apache.commons.vfs.FileSystemException: Could not create folder "hdfs://machine.compute-1.amazonaws.com:9000/".
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.commons.vfs.provider.AbstractFileObject.createFolder(Unknown Source)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.commons.vfs.provider.AbstractFileObject.createFolder(Unknown Source)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.pentaho.di.job.entries.copyfiles.JobEntryCopyFiles.CreateDestinationFolder(JobEntryCopyFiles.java:667)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.pentaho.di.job.entries.copyfiles.JobEntryCopyFiles.ProcessFileFolder(JobEntryCopyFiles.java:386)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.pentaho.di.job.entries.copyfiles.JobEntryCopyFiles.execute(JobEntryCopyFiles.java:326)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.pentaho.di.job.Job.execute(Job.java:678)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.pentaho.di.job.Job.execute(Job.java:815)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.pentaho.di.job.Job.execute(Job.java:500)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.pentaho.di.job.Job.run(Job.java:407)
2014/01/10 10:05:48 - Hadoop Copy Files - Caused by: java.io.IOException: Failed on local exception: java.io.IOException: An established connection was aborted by the software in your host machine; Host Details : local host is: "CHEJX05CQ1/2.0.0.2"; destination host is: "machine.compute-1.amazonaws.com":9000;
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.ipc.Client.call(Client.java:1351)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.ipc.Client.call(Client.java:1300)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
2014/01/10 10:05:48 - Hadoop Copy Files - at com.sun.proxy.$Proxy100.mkdirs(Unknown Source)
2014/01/10 10:05:48 - Hadoop Copy Files - at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2014/01/10 10:05:48 - Hadoop Copy Files - at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
2014/01/10 10:05:48 - Hadoop Copy Files - at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
2014/01/10 10:05:48 - Hadoop Copy Files - at java.lang.reflect.Method.invoke(Unknown Source)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
2014/01/10 10:05:48 - Hadoop Copy Files - at com.sun.proxy.$Proxy100.mkdirs(Unknown Source)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:467)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2394)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2365)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.hdfs.DistributedFileSystem$16.doCall(DistributedFileSystem.java:817)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.hdfs.DistributedFileSystem$16.doCall(DistributedFileSystem.java:813)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:813)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:806)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1933)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.pentaho.hdfs.vfs.HDFSFileObject.doCreateFolder(HDFSFileObject.java:81)
2014/01/10 10:05:48 - Hadoop Copy Files - ... 9 more
2014/01/10 10:05:48 - Hadoop Copy Files - Caused by: java.io.IOException: An established connection was aborted by the software in your host machine
2014/01/10 10:05:48 - Hadoop Copy Files - at sun.nio.ch.SocketDispatcher.read0(Native Method)
2014/01/10 10:05:48 - Hadoop Copy Files - at sun.nio.ch.SocketDispatcher.read(Unknown Source)
2014/01/10 10:05:48 - Hadoop Copy Files - at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
2014/01/10 10:05:48 - Hadoop Copy Files - at sun.nio.ch.IOUtil.read(Unknown Source)
2014/01/10 10:05:48 - Hadoop Copy Files - at sun.nio.ch.SocketChannelImpl.read(Unknown Source)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
2014/01/10 10:05:48 - Hadoop Copy Files - at java.io.FilterInputStream.read(Unknown Source)
2014/01/10 10:05:48 - Hadoop Copy Files - at java.io.FilterInputStream.read(Unknown Source)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:457)
2014/01/10 10:05:48 - Hadoop Copy Files - at java.io.BufferedInputStream.fill(Unknown Source)
2014/01/10 10:05:48 - Hadoop Copy Files - at java.io.BufferedInputStream.read(Unknown Source)
2014/01/10 10:05:48 - Hadoop Copy Files - at java.io.DataInputStream.readInt(Unknown Source)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:995)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.ipc.Client$Connection.run(Client.java:891)
2014/01/10 10:05:48 - Hadoop Copy Files - ERROR (version 5.0.2, build 1 from 2013-12-04_15-52-25 by buildguy) : Destination folder does not exist!
2014/01/10 10:05:48 - TestLoadEMR - Finished job entry [Hadoop Copy Files] (result=[false])
Any Suggestion to make this one work guys.
Thanks in advance.
Regards,
VAP.
We have Amazon Web Service with Amazon EMR up and running.
For building a prototype, I am just trying to use Hadoop Copy Files component to copy one file from local disk to the HDFS.
So I just added a component Hadoop Copy Files and specified the required Input and Output info , but unfortunately the job is failing during execution.
Here below I added the full stack trace.
Quote:
2014/01/10 10:05:48 - Hadoop Copy Files - ERROR (version 5.0.2, build 1 from 2013-12-04_15-52-25 by buildguy) : Couldn't created parent folder hdfs://machine.compute-1.amazonaws.com:9000/mnt
2014/01/10 10:05:48 - Hadoop Copy Files - ERROR (version 5.0.2, build 1 from 2013-12-04_15-52-25 by buildguy) : org.apache.commons.vfs.FileSystemException: Could not create folder "hdfs://machine.compute-1.amazonaws.com:9000/".
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.commons.vfs.provider.AbstractFileObject.createFolder(Unknown Source)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.commons.vfs.provider.AbstractFileObject.createFolder(Unknown Source)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.pentaho.di.job.entries.copyfiles.JobEntryCopyFiles.CreateDestinationFolder(JobEntryCopyFiles.java:667)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.pentaho.di.job.entries.copyfiles.JobEntryCopyFiles.ProcessFileFolder(JobEntryCopyFiles.java:386)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.pentaho.di.job.entries.copyfiles.JobEntryCopyFiles.execute(JobEntryCopyFiles.java:326)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.pentaho.di.job.Job.execute(Job.java:678)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.pentaho.di.job.Job.execute(Job.java:815)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.pentaho.di.job.Job.execute(Job.java:500)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.pentaho.di.job.Job.run(Job.java:407)
2014/01/10 10:05:48 - Hadoop Copy Files - Caused by: java.io.IOException: Failed on local exception: java.io.IOException: An established connection was aborted by the software in your host machine; Host Details : local host is: "CHEJX05CQ1/2.0.0.2"; destination host is: "machine.compute-1.amazonaws.com":9000;
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.ipc.Client.call(Client.java:1351)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.ipc.Client.call(Client.java:1300)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
2014/01/10 10:05:48 - Hadoop Copy Files - at com.sun.proxy.$Proxy100.mkdirs(Unknown Source)
2014/01/10 10:05:48 - Hadoop Copy Files - at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2014/01/10 10:05:48 - Hadoop Copy Files - at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
2014/01/10 10:05:48 - Hadoop Copy Files - at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
2014/01/10 10:05:48 - Hadoop Copy Files - at java.lang.reflect.Method.invoke(Unknown Source)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
2014/01/10 10:05:48 - Hadoop Copy Files - at com.sun.proxy.$Proxy100.mkdirs(Unknown Source)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:467)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2394)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2365)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.hdfs.DistributedFileSystem$16.doCall(DistributedFileSystem.java:817)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.hdfs.DistributedFileSystem$16.doCall(DistributedFileSystem.java:813)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:813)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:806)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1933)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.pentaho.hdfs.vfs.HDFSFileObject.doCreateFolder(HDFSFileObject.java:81)
2014/01/10 10:05:48 - Hadoop Copy Files - ... 9 more
2014/01/10 10:05:48 - Hadoop Copy Files - Caused by: java.io.IOException: An established connection was aborted by the software in your host machine
2014/01/10 10:05:48 - Hadoop Copy Files - at sun.nio.ch.SocketDispatcher.read0(Native Method)
2014/01/10 10:05:48 - Hadoop Copy Files - at sun.nio.ch.SocketDispatcher.read(Unknown Source)
2014/01/10 10:05:48 - Hadoop Copy Files - at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
2014/01/10 10:05:48 - Hadoop Copy Files - at sun.nio.ch.IOUtil.read(Unknown Source)
2014/01/10 10:05:48 - Hadoop Copy Files - at sun.nio.ch.SocketChannelImpl.read(Unknown Source)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
2014/01/10 10:05:48 - Hadoop Copy Files - at java.io.FilterInputStream.read(Unknown Source)
2014/01/10 10:05:48 - Hadoop Copy Files - at java.io.FilterInputStream.read(Unknown Source)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:457)
2014/01/10 10:05:48 - Hadoop Copy Files - at java.io.BufferedInputStream.fill(Unknown Source)
2014/01/10 10:05:48 - Hadoop Copy Files - at java.io.BufferedInputStream.read(Unknown Source)
2014/01/10 10:05:48 - Hadoop Copy Files - at java.io.DataInputStream.readInt(Unknown Source)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:995)
2014/01/10 10:05:48 - Hadoop Copy Files - at org.apache.hadoop.ipc.Client$Connection.run(Client.java:891)
2014/01/10 10:05:48 - Hadoop Copy Files - ERROR (version 5.0.2, build 1 from 2013-12-04_15-52-25 by buildguy) : Destination folder does not exist!
2014/01/10 10:05:48 - TestLoadEMR - Finished job entry [Hadoop Copy Files] (result=[false])
Thanks in advance.
Regards,
VAP.