Hello,
I am actually using Pentaho 4.4.0 to do some ETL job. In one of the jobs I created, I use the Bulk Loader step to fill a table (f_localisation). The job runs ok for the first 15-20 minutes, but it stops at some point (always the same moment, after inserting 13200000 rows). I increased the timeout net_write_timeout, thinking it could be it but it didn't solve the problem.
I put the log trace below.
Any idea where this error could come from?
Thanks in advance!
I am actually using Pentaho 4.4.0 to do some ETL job. In one of the jobs I created, I use the Bulk Loader step to fill a table (f_localisation). The job runs ok for the first 15-20 minutes, but it stops at some point (always the same moment, after inserting 13200000 rows). I increased the timeout net_write_timeout, thinking it could be it but it didn't solve the problem.
I put the log trace below.
Any idea where this error could come from?
Thanks in advance!
Code:
INFO 17-07 15:37:08,197 - mysqlbulk_f_localisation - Opening fifo /vol/fifo_f_localisation for writing.
ERROR 17-07 15:41:00,924 - mysqlbulk_f_localisation - Error in step, asking everyone to stop because of:
ERROR 17-07 15:41:00,944 - mysqlbulk_f_localisation - org.pentaho.di.core.exception.KettleException:
Error serializing rows of data to the fifo file
Couldn't execute SQL: LOAD DATA LOCAL INFILE '/vol/fifo_f_localisation' INTO TABLE `f_localisation` FIELDS TERMINATED BY '\t' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '\\' (`d_city`,`d_country`,`d_date`);
Communications link failure
The last packet successfully received from the server was 232,621 milliseconds ago. The last packet sent successfully to the server was 12 milliseconds ago.
at org.pentaho.di.trans.steps.mysqlbulkloader.MySQLBulkLoader.writeRowToBulk(MySQLBulkLoader.java:423)
at org.pentaho.di.trans.steps.mysqlbulkloader.MySQLBulkLoader.processRow(MySQLBulkLoader.java:278)
at org.pentaho.di.trans.step.RunThread.run(RunThread.java:50)
at java.lang.Thread.run(Thread.java:722)
Caused by: org.pentaho.di.core.exception.KettleDatabaseException:
Couldn't execute SQL: LOAD DATA LOCAL INFILE '/vol/fifo_f_localisation' INTO TABLE `f_localisation` FIELDS TERMINATED BY '\t' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '\\' (`d_city`,`d_country`,`d_date`);
Communications link failure
The last packet successfully received from the server was 232,621 milliseconds ago. The last packet sent successfully to the server was 12 milliseconds ago.
at org.pentaho.di.core.database.Database.execStatement(Database.java:1664)
at org.pentaho.di.core.database.Database.execStatement(Database.java:1604)
at org.pentaho.di.trans.steps.mysqlbulkloader.MySQLBulkLoader$SqlRunner.run(MySQLBulkLoader.java:573)
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet successfully received from the server was 232,621 milliseconds ago. The last packet sent successfully to the server was 12 milliseconds ago.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1117)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3871)
at com.mysql.jdbc.MysqlIO.sendFileToServer(MysqlIO.java:3967)
at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:3129)
at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:2299)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2722)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2794)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2155)
at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1379)
at com.mysql.jdbc.StatementImpl.createResultSetUsingServerFetch(StatementImpl.java:658)
at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:830)
at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:737)
at org.pentaho.di.core.database.Database.execStatement(Database.java:1632)
... 2 more
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3852)
... 13 more