Quantcast
Channel: Pentaho Community Forums
Viewing all articles
Browse latest Browse all 16689

PostgreSQL Bulk Loader problem

$
0
0
Hi,

I have 700 million rows in a operational DB on Postgres. I have to export this data into a datawarehouse (also Postgres). After reading some websites, the best way for this task in PDI is the Bulk Loader component.

My flow in PDI :

Input table --> PostgresSQL Bulk Loader

PDI is installed on my computer. To setup the PostgresSQL Bulk Loader component, for the "psql path" I put the path of the psql utility of my localhost Postgre (something like C:\postgres\psql.exe) because I don't know the path of the psql on the operational DB and the datawarehouse.

But it doesn't work. I have errors in PDI with PostgresSQL Bulk Loader component. When I check the log, PDI and the psql utility has succefully connected to the datawarehouse but I think I have a problem wth the statement COPY...FROM STDIN... generated behind PDI.

EDIT : this the error message :

Quote:

PostgreSQL Bulk Loader.0 - Executing command: "C:\Program Files\PostgreSQL\9.3\scripts\runpsql.bat" -U i2bi -h pgsqldbm.server_name.biz -p 5432 Source
2014/02/21 17:14:02 - PostgreSQL Bulk Loader.0 - Launching command: COPY "public".src_gridfeeinvoice ( gridfeeinvoiceid, supplier, gridoperator, invoicenature, gsrn, fromdate, todate, invoicedpointnature, invoicenumber, deliveryperiodid, invoicingdate, messageid, creator, creationdate, latestmodifier, latestmodifdate ) FROM STDIN WITH CSV DELIMITER AS ';' QUOTE AS '"';
2014/02/21 17:14:02 - PostgreSQL Bulk Loader.0 - ERROR {0} psql : attention : option supplémentaire « gridfeeinvoiceid, » ignorée
2014/02/21 17:14:02 - PostgreSQL Bulk Loader.0 - ERROR {0} psql : attention : option supplémentaire « supplier, » ignorée
2014/02/21 17:14:02 - PostgreSQL Bulk Loader.0 - ERROR {0} psql : attention : option supplémentaire « gridoperator, » ignorée
2014/02/21 17:14:02 - PostgreSQL Bulk Loader.0 - ERROR {0} psql : attention : option supplémentaire « invoicenature, » ignorée
2014/02/21 17:14:02 - PostgreSQL Bulk Loader.0 - ERROR {0} psql : attention : option supplémentaire « gsrn, » ignorée
2014/02/21 17:14:02 - PostgreSQL Bulk Loader.0 - ERROR {0} psql : attention : option supplémentaire « fromdate, » ignorée
2014/02/21 17:14:02 - PostgreSQL Bulk Loader.0 - ERROR {0} psql : attention : option supplémentaire « todate, » ignorée
2014/02/21 17:14:02 - PostgreSQL Bulk Loader.0 - ERROR {0} psql : attention : option supplémentaire « invoicedpointnature, » ignorée
2014/02/21 17:14:02 - PostgreSQL Bulk Loader.0 - ERROR {0} psql : attention : option supplémentaire « invoicenumber, » ignorée
2014/02/21 17:14:02 - PostgreSQL Bulk Loader.0 - ERROR {0} psql : attention : option supplémentaire « deliveryperiodid, » ignorée
2014/02/21 17:14:02 - PostgreSQL Bulk Loader.0 - ERROR {0} psql : attention : option supplémentaire « invoicingdate, » ignorée
2014/02/21 17:14:02 - PostgreSQL Bulk Loader.0 - ERROR {0} psql : attention : option supplémentaire « messageid, » ignorée
2014/02/21 17:14:02 - PostgreSQL Bulk Loader.0 - ERROR {0} psql : attention : option supplémentaire « creator, » ignorée
2014/02/21 17:14:02 - PostgreSQL Bulk Loader.0 - ERROR {0} psql : attention : option supplémentaire « creationdate, » ignorée
2014/02/21 17:14:02 - PostgreSQL Bulk Loader.0 - ERROR {0} psql : attention : option supplémentaire « latestmodifier, » ignorée
2014/02/21 17:14:02 - PostgreSQL Bulk Loader.0 - ERROR {0} psql : attention : option supplémentaire « latestmodifdate » ignorée
2014/02/21 17:14:02 - PostgreSQL Bulk Loader.0 - ERROR {0} psql : attention : option supplémentaire « ) » ignorée
2014/02/21 17:14:02 - PostgreSQL Bulk Loader.0 - ERROR {0} psql : attention : option supplémentaire « FROM » ignorée
2014/02/21 17:14:02 - PostgreSQL Bulk Loader.0 - ERROR {0} psql : attention : option supplémentaire « STDIN » ignorée
2014/02/21 17:14:02 - PostgreSQL Bulk Loader.0 - ERROR {0} psql : attention : option supplémentaire « WITH » ignorée
2014/02/21 17:14:02 - PostgreSQL Bulk Loader.0 - ERROR {0} psql : attention : option supplémentaire « CSV » ignorée
2014/02/21 17:14:02 - PostgreSQL Bulk Loader.0 - ERROR {0} psql : attention : option supplémentaire « DELIMITER » ignorée
2014/02/21 17:14:02 - PostgreSQL Bulk Loader.0 - ERROR {0} psql : attention : option supplémentaire « AS » ignorée
2014/02/21 17:14:02 - PostgreSQL Bulk Loader.0 - ERROR {0} psql : attention : option supplémentaire « ';' » ignorée
2014/02/21 17:14:02 - PostgreSQL Bulk Loader.0 - ERROR {0} psql : attention : option supplémentaire « QUOTE » ignorée
2014/02/21 17:14:02 - PostgreSQL Bulk Loader.0 - ERROR {0} psql : attention : option supplémentaire « AS » ignorée
2014/02/21 17:14:02 - PostgreSQL Bulk Loader.0 - ERROR {0} psql : attention : option supplémentaire « ''; -U RLY;0082014062;;2010/06/30 » ignorée
2014/02/21 17:14:02 - PostgreSQL Bulk Loader.0 - ERROR {0} psql : attention : option supplémentaire « 00:00:00;5398873;system;;system; » ignorée
2014/02/21 17:14:02 - PostgreSQL Bulk Loader.0 - ERROR {0} psql : attention : option supplémentaire « -d » ignorée
2014/02/21 17:14:02 - PostgreSQL Bulk Loader.0 - ERROR {0} psql : attention : option supplémentaire « 398852;system;;system; » ignorée
2014/02/21 17:14:02 - PostgreSQL Bulk Loader.0 - ERROR {0} psql : attention : option supplémentaire « -p » ignorée
2014/02/21 17:14:02 - PostgreSQL Bulk Loader.0 - ERROR {0} psql : attention : option supplémentaire « 010/06/30 » ignorée
2014/02/21 17:14:02 - PostgreSQL Bulk Loader.0 - ERROR {0} psql : attention : option supplémentaire « 00:00:00;5398861;system;;system; » ignorée
2014/02/21 17:14:04 - PostgreSQL Bulk Loader.0 - ERROR {0} psql: n'a pas pu traduire le nom d'hôte « COPY » en adresse : Unknown server error
2014/02/21 17:14:04 - PostgreSQL Bulk Loader.0 - OUTPUT {0} Server [localhost]: Database [postgres]: Port [5432]: Username [postgres]: Appuyez sur une touche pour continuer...
2014/02/21 17:14:04 - PostgreSQL Bulk Loader.0 - ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy) : Error in step
2014/02/21 17:14:04 - PostgreSQL Bulk Loader.0 - ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy) : org.pentaho.di.core.exception.KettleException:
2014/02/21 17:14:04 - PostgreSQL Bulk Loader.0 - Error serializing rows of data to the psql command
2014/02/21 17:14:04 - PostgreSQL Bulk Loader.0 - Le canal de communication est sur le point d’être fermé
2014/02/21 17:14:04 - PostgreSQL Bulk Loader.0 -
2014/02/21 17:14:04 - PostgreSQL Bulk Loader.0 - at org.pentaho.di.trans.steps.pgbulkloader.PGBulkLoader.writeRowToPostgres(PGBulkLoader.java:450)
2014/02/21 17:14:04 - PostgreSQL Bulk Loader.0 - at org.pentaho.di.trans.steps.pgbulkloader.PGBulkLoader.processRow(PGBulkLoader.java:316)
2014/02/21 17:14:04 - PostgreSQL Bulk Loader.0 - at org.pentaho.di.trans.step.RunThread.run(RunThread.java:60)
2014/02/21 17:14:04 - PostgreSQL Bulk Loader.0 - at java.lang.Thread.run(Unknown Source)
2014/02/21 17:14:04 - PostgreSQL Bulk Loader.0 - Caused by: java.io.IOException: Le canal de communication est sur le point d’être fermé
2014/02/21 17:14:04 - PostgreSQL Bulk Loader.0 - at java.io.FileOutputStream.writeBytes(Native Method)
2014/02/21 17:14:04 - PostgreSQL Bulk Loader.0 - at java.io.FileOutputStream.write(Unknown Source)
2014/02/21 17:14:04 - PostgreSQL Bulk Loader.0 - at java.io.BufferedOutputStream.flushBuffer(Unknown Source)
2014/02/21 17:14:04 - PostgreSQL Bulk Loader.0 - at java.io.BufferedOutputStream.write(Unknown Source)
2014/02/21 17:14:04 - PostgreSQL Bulk Loader.0 - at java.io.FilterOutputStream.write(Unknown Source)
2014/02/21 17:14:04 - PostgreSQL Bulk Loader.0 - at org.pentaho.di.trans.steps.pgbulkloader.PGBulkLoader.writeRowToPostgres(PGBulkLoader.java:361)
2014/02/21 17:14:04 - PostgreSQL Bulk Loader.0 - ... 3 more
2014/02/21 17:14:04 - PostgreSQL Bulk Loader.0 - Finished processing (I=0, O=162, R=163, W=162, U=0, E=1)
2014/02/21 17:14:04 - gridfeeinvoice - gridfeeinvoice
2014/02/21 17:14:04 - gridfeeinvoice - gridfeeinvoice
2014/02/21 17:14:04 - Table input.0 - ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy) : Unexpected error
2014/02/21 17:14:04 - Table input.0 - ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy) : org.pentaho.di.core.exception.KettleDatabaseException:
2014/02/21 17:14:04 - Table input.0 - Couldn't get row from result set
2014/02/21 17:14:04 - Table input.0 - ERROR: canceling statement due to user request
2014/02/21 17:14:04 - Table input.0 -
2014/02/21 17:14:04 - Table input.0 - at org.pentaho.di.core.database.Database.getRow(Database.java:2302)
2014/02/21 17:14:04 - Table input.0 - at org.pentaho.di.core.database.Database.getRow(Database.java:2270)
2014/02/21 17:14:04 - Table input.0 - at org.pentaho.di.trans.steps.tableinput.TableInput.processRow(TableInput.java:153)
2014/02/21 17:14:04 - Table input.0 - at org.pentaho.di.trans.step.RunThread.run(RunThread.java:60)
2014/02/21 17:14:04 - Table input.0 - at java.lang.Thread.run(Unknown Source)
2014/02/21 17:14:04 - Table input.0 - Caused by: org.postgresql.util.PSQLException: ERROR: canceling statement due to user request
2014/02/21 17:14:04 - Table input.0 - at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2101)
2014/02/21 17:14:04 - Table input.0 - at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1834)
2014/02/21 17:14:04 - Table input.0 - at org.postgresql.core.v3.QueryExecutorImpl.fetch(QueryExecutorImpl.java:2036)
2014/02/21 17:14:04 - Table input.0 - at org.postgresql.jdbc2.AbstractJdbc2ResultSet.next(AbstractJdbc2ResultSet.java:1821)
2014/02/21 17:14:04 - Table input.0 - at org.pentaho.di.core.database.Database.getRow(Database.java:2290)
2014/02/21 17:14:04 - Table input.0 - ... 4 more
2014/02/21 17:14:04 - Table input.0 - Finished reading query, closing connection.
2014/02/21 17:14:04 - Table input.0 - Finished processing (I=20000, O=0, R=0, W=19999, U=0, E=1)
2014/02/21 17:14:04 - gridfeeinvoice - ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy) : Errors detected!
2014/02/21 17:14:04 - Spoon - The transformation has finished!!
2014/02/21 17:14:04 - gridfeeinvoice - ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy) : Errors detected!
2014/02/21 17:14:04 - gridfeeinvoice - ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy) : Errors detected!
2014/02/21 17:14:04 - gridfeeinvoice - gridfeeinvoice
2014/02/21 17:14:04 - gridfeeinvoice - gridfeeinvoice
Thank you.

Viewing all articles
Browse latest Browse all 16689

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>