Hello!
I'm trying to transform a denormalized table with about 600.000 records into a star schema with several lookup/update steps and a table output step for the fact table.
When I check "Use batch updates for inserts" in the table output step, the process freezes after a random number of rows. There is no error but all load/write operations just stop. It seems like the process gets a little bit further with each time you restart it but that can't be a solution.
If I uncheck "Use batch updates for inserts" in the table output step, the process fails with the following exceptions after about 320.000 rows. Same here, if I restart the process (with truncating the fact table) the process sometimes does some more rows before it fails again, with the same error message.
I'm using mysql 5.6 as data source and target. I have checked the mysql error log, but there are no entries. In the normal log, the last entry is a lookup statement for one of the dimensions. I executed it manually and it returned a value immediatly.
The NULL values shouldn't be the problem since the table accepts NULL values for these columns... Also it's not always the same row/same values at which the process fails.
Anybody has an idea?
Regards!
I'm trying to transform a denormalized table with about 600.000 records into a star schema with several lookup/update steps and a table output step for the fact table.
When I check "Use batch updates for inserts" in the table output step, the process freezes after a random number of rows. There is no error but all load/write operations just stop. It seems like the process gets a little bit further with each time you restart it but that can't be a solution.
If I uncheck "Use batch updates for inserts" in the table output step, the process fails with the following exceptions after about 320.000 rows. Same here, if I restart the process (with truncating the fact table) the process sometimes does some more rows before it fails again, with the same error message.
I'm using mysql 5.6 as data source and target. I have checked the mysql error log, but there are no entries. In the normal log, the last entry is a lookup statement for one of the dimensions. I executed it manually and it returned a value immediatly.
The NULL values shouldn't be the problem since the table accepts NULL values for these columns... Also it's not always the same row/same values at which the process fails.
Code:
2013/04/23 15:06:10 - Table output.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : Because of an error, this step can't continue:
2013/04/23 15:06:10 - Table output.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : org.pentaho.di.core.exception.KettleException:
2013/04/23 15:06:10 - Table output.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : Error inserting row into table [agg_values] with values: [null], [null], [null], [0], [0], [0], [0], [2], [0], [0], [4948], [5835], [286], [3539]
2013/04/23 15:06:10 - Table output.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) :
2013/04/23 15:06:10 - Table output.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : Error inserting/updating row
2013/04/23 15:06:10 - Table output.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : Got error -1 from storage engine
2013/04/23 15:06:10 - Table output.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) :
2013/04/23 15:06:10 - Table output.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) :
2013/04/23 15:06:10 - Table output.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : at org.pentaho.di.trans.steps.tableoutput.TableOutput.writeToTable(TableOutput.java:445)
2013/04/23 15:06:10 - Table output.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : at org.pentaho.di.trans.steps.tableoutput.TableOutput.processRow(TableOutput.java:128)
2013/04/23 15:06:10 - Table output.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : at org.pentaho.di.trans.step.RunThread.run(RunThread.java:50)
2013/04/23 15:06:10 - Table output.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : at java.lang.Thread.run(Unknown Source)
2013/04/23 15:06:10 - Table output.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : Caused by: org.pentaho.di.core.exception.KettleDatabaseException:
2013/04/23 15:06:10 - Table output.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : Error inserting/updating row
2013/04/23 15:06:10 - Table output.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : Got error -1 from storage engine
2013/04/23 15:06:10 - Table output.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) :
2013/04/23 15:06:10 - Table output.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : at org.pentaho.di.core.database.Database.insertRow(Database.java:1411)
2013/04/23 15:06:10 - Table output.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : at org.pentaho.di.trans.steps.tableoutput.TableOutput.writeToTable(TableOutput.java:284)
2013/04/23 15:06:10 - Table output.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : ... 3 more
2013/04/23 15:06:10 - Table output.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : Caused by: java.sql.SQLException: Got error -1 from storage engine
2013/04/23 15:06:10 - Table output.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1074)
2013/04/23 15:06:10 - Table output.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4120)
2013/04/23 15:06:10 - Table output.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4052)
2013/04/23 15:06:10 - Table output.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2503)
2013/04/23 15:06:10 - Table output.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : at com.mysql.jdbc.ServerPreparedStatement.serverExecute(ServerPreparedStatement.java:1399)
2013/04/23 15:06:10 - Table output.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : at com.mysql.jdbc.ServerPreparedStatement.executeInternal(ServerPreparedStatement.java:857)
2013/04/23 15:06:10 - Table output.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2458)
2013/04/23 15:06:10 - Table output.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2375)
2013/04/23 15:06:10 - Table output.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2359)
2013/04/23 15:06:10 - Table output.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : at org.pentaho.di.core.database.Database.insertRow(Database.java:1360)
2013/04/23 15:06:10 - Table output.0 - ERROR (version 4.4.0-GA, build 17542 from 2012-11-01 20.06.29 by buildguy) : ... 4 moreRegards!