Quantcast
Channel: Pentaho Community Forums
Viewing all 16689 articles
Browse latest View live
↧

GC Overhead limit Exceeded

May 27, 2015, 10:07 am
$
0
0
Hi all,

I am using Pentaho 5.0.1 CE on a Linux Centos box with 16 GBs of RAM.

I have this transformation,https://www.dropbox.com/s/dz56php7g7...llups.ktr?dl=0, that excutes in a job within Kitchen but I am getting the following error.

This transformation executes a stored procedure that returns back 9000 records with are compared with the existing data of 6000 records and inserts and updates new rolled sku data or updates any existing skus.



2015/05/27 12:19:42 - si_job_calc_sku_roll_ups - Starting entry [Load_insert_yearly_sku_rollups]
java.lang.OutOfMemoryError: GC overhead limit exceeded
at org.pentaho.di.core.row.value.ValueMetaBase.<init>(ValueMetaBase.java:176)
at org.pentaho.di.core.row.value.ValueMetaBase.<init>(ValueMetaBase.java:158)
at org.pentaho.di.core.row.value.ValueMetaDate.<init>(ValueMetaDate.java:15)
at org.pentaho.di.core.row.value.ValueMetaDate.<init>(ValueMetaDate.java:11)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy) : java.lang.OutOfMemoryError: GC overhead limit exceeded
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at java.lang.Throwable.printStackTrace(Throwable.java:658)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at java.lang.Throwable.printStackTrace(Throwable.java:643)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at java.lang.Throwable.printStackTrace(Throwable.java:634)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at org.pentaho.di.core.plugins.PluginRegistry.loadClass(PluginRegistry.java:441)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at org.pentaho.di.core.row.value.ValueMetaFactory.createValueMeta(ValueMetaFactory.java:27)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at org.pentaho.di.core.row.value.ValueMetaFactory.cloneValueMeta(ValueMetaFactory.java:46)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at org.pentaho.di.core.row.value.ValueMetaFactory.cloneValueMeta(ValueMetaFactory.java:42)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at org.pentaho.di.core.row.RowMeta.clone(RowMeta.java:71)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at org.pentaho.di.core.row.RowMeta.clone(RowMeta.java:50)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at org.pentaho.di.core.RowMetaAndData.clone(RowMetaAndData.java:62)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at org.pentaho.di.core.Result.clone(Result.java:214)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at org.pentaho.di.job.Job.execute(Job.java:624)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at org.pentaho.di.job.Job.execute(Job.java:815)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at org.pentaho.di.job.Job.execute(Job.java:815)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at org.pentaho.di.job.Job.execute(Job.java:562)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at org.pentaho.di.job.entries.job.JobEntryJobRunner.run(JobEntryJobRunner.java:73)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at java.lang.Thread.run(Thread.java:745)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy) : org.pentaho.di.core.exception.KettleException:
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - Unexpected error occurred while launching entry [Load_insert_yearly_sku_rollups.0]
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - GC overhead limit exceeded
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups -
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at org.pentaho.di.job.Job.execute(Job.java:818)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at org.pentaho.di.job.Job.execute(Job.java:815)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at org.pentaho.di.job.Job.execute(Job.java:562)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at org.pentaho.di.job.entries.job.JobEntryJobRunner.run(JobEntryJobRunner.java:73)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at java.lang.Thread.run(Thread.java:745)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at java.lang.Throwable.printStackTrace(Throwable.java:658)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at java.lang.Throwable.printStackTrace(Throwable.java:643)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at java.lang.Throwable.printStackTrace(Throwable.java:634)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at org.pentaho.di.core.plugins.PluginRegistry.loadClass(PluginRegistry.java:441)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at org.pentaho.di.core.row.value.ValueMetaFactory.createValueMeta(ValueMetaFactory.java:27)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at org.pentaho.di.core.row.value.ValueMetaFactory.cloneValueMeta(ValueMetaFactory.java:46)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at org.pentaho.di.core.row.value.ValueMetaFactory.cloneValueMeta(ValueMetaFactory.java:42)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at org.pentaho.di.core.row.RowMeta.clone(RowMeta.java:71)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at org.pentaho.di.core.row.RowMeta.clone(RowMeta.java:50)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at org.pentaho.di.core.RowMetaAndData.clone(RowMetaAndData.java:62)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at org.pentaho.di.core.Result.clone(Result.java:214)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at org.pentaho.di.job.Job.execute(Job.java:624)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at org.pentaho.di.job.Job.execute(Job.java:815)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - ... 4 more
org.pentaho.di.core.exception.KettleException:
Unexpected error occurred while launching entry [Tr_Load_the_rolledup_quarters_data.0]


Unexpected error occurred while launching entry [Load_insert_yearly_sku_rollups.0]
GC overhead limit exceeded




at org.pentaho.di.job.Job.execute(Job.java:818)
at org.pentaho.di.job.Job.execute(Job.java:562)
at org.pentaho.di.job.entries.job.JobEntryJobRunner.run(JobEntryJobRunner.java:73)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.pentaho.di.core.exception.KettleException:
Unexpected error occurred while launching entry [Load_insert_yearly_sku_rollups.0]
GC overhead limit exceeded


at org.pentaho.di.job.Job.execute(Job.java:818)
at org.pentaho.di.job.Job.execute(Job.java:815)
... 3 more
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.lang.Throwable.printStackTrace(Throwable.java:658)
at java.lang.Throwable.printStackTrace(Throwable.java:643)
at java.lang.Throwable.printStackTrace(Throwable.java:634)
at org.pentaho.di.core.plugins.PluginRegistry.loadClass(PluginRegistry.java:441)
at org.pentaho.di.core.row.value.ValueMetaFactory.createValueMeta(ValueMetaFactory.java:27)
at org.pentaho.di.core.row.value.ValueMetaFactory.cloneValueMeta(ValueMetaFactory.java:46)
at org.pentaho.di.core.row.value.ValueMetaFactory.cloneValueMeta(ValueMetaFactory.java:42)
at org.pentaho.di.core.row.RowMeta.clone(RowMeta.java:71)
at org.pentaho.di.core.row.RowMeta.clone(RowMeta.java:50)
at org.pentaho.di.core.RowMetaAndData.clone(RowMetaAndData.java:62)
at org.pentaho.di.core.Result.clone(Result.java:214)
at org.pentaho.di.job.Job.execute(Job.java:624)
at org.pentaho.di.job.Job.execute(Job.java:815)
... 4 more
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy) : An error occurred executing this job entry :
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy) : org.pentaho.di.core.exception.KettleException:
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - Unexpected error occurred while launching entry [Tr_Load_the_rolledup_quarters_data.0]
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups -
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - Unexpected error occurred while launching entry [Load_insert_yearly_sku_rollups.0]
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - GC overhead limit exceeded
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups -
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups -
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at org.pentaho.di.job.Job.execute(Job.java:818)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at org.pentaho.di.job.Job.execute(Job.java:562)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at org.pentaho.di.job.entries.job.JobEntryJobRunner.run(JobEntryJobRunner.java:73)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at java.lang.Thread.run(Thread.java:745)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - Caused by: org.pentaho.di.core.exception.KettleException:
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - Unexpected error occurred while launching entry [Load_insert_yearly_sku_rollups.0]
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - GC overhead limit exceeded
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups -
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at org.pentaho.di.job.Job.execute(Job.java:818)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at org.pentaho.di.job.Job.execute(Job.java:815)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - ... 3 more
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at java.lang.Throwable.printStackTrace(Throwable.java:658)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at java.lang.Throwable.printStackTrace(Throwable.java:643)
2015/05/27 12:20:42 - si_job_calc_sku_roll_ups - at java.lang.Throwable.printStackTrace(Throwable.java:634)

I have played around with the Java parameters, increased the heap size memory to 8GB and the stack size to 2048M but still getting the same error.

Is there a solution for this?

Thanks,

Ron
↧
Search

Table output and Update step

May 27, 2015, 11:07 am
$
0
0
Hi,

Could you please suggest me how can we split the data (new records) should insert into Table output step and (incremental records) into update step.

I have Created_date and Updated_date in source step(i.e in Table Input step).

I know we have Insert/Update step but it is very slow in my scenario so i am requesting you to provide some input.

PDI 5.3 CE
Java 1.8
Windows 8 OS
MySql Database

Table output and update.PNG

Thank you
Attached Images
↧
↧

Connect Kettle to a sage100 MAS 90 database

May 27, 2015, 11:41 am
$
0
0
I'm trying to connect to a sage100 (SOTAMAS90) database with Kettle, but I'm not sure how to do this.
I've installed Kettle on a Windows Server 2012. I've created an ODBC data source (System DSN), but have no idea how to get Pentaho to use that connection. I've tried using a general database connection with the ODBC, but get this.
Code:
Driver class 'sun.jdbc.odbc.JdbcOdbcDriver' could not be found
I'm using Kettle 5.3.0.0-213 with java 1.8.0_45.

Thanks
↧

BI for a web hosting company

May 27, 2015, 8:05 pm
$
0
0
Hi guys,

Would anyone have a data model or at least something to start me off? What are the key dimensions?

Think of a domain registrar/website hosting company using CPanel


PS: Sorry for cross-posting
↧

Which Step to use: Table Compare or Merge Rows (diff)?

May 27, 2015, 9:26 pm
$
0
0
I need to set up CDC using a "snapshot" approach (no transaction logging, no usable dates in the tables, I cannot modify the source tables, etc.).

To identify the rows of interest (Changed, New, Deleted), it seems that I could use either Merge Rows (diff) followed by a Filter, or the KFF Table Compare step. Other than the fact that Merge Rows (diff) requires that both streams be sorted, when is one​ approach better than the other?

thank you-
MB
↧
↧

How to translate some messages?

May 27, 2015, 11:44 pm
$
0
0
Hi,

I can't see how to translate some messages like 'This parameter is mandatory' without recompiling the project : I have found a messages.properties, but it is included in a jar.

I have tried to do it once the report is integrated in a dashboard, but without success.

Is there any simpler way to internationalize or should I open a jira?

Thank you

Benoit
↧

Performance: JSON Input

May 28, 2015, 3:07 am
$
0
0
Hi!

Following from this thread: http://forums.pentaho.com/showthread...ted-JSON-array

We noticed that the JSON Input step is not streamlining data as it READS the JSON data file. Instead, correct me if wrong, it commits everything it reads first to memory and when done it will then begin streaming data to the next step and through. This behavior is fine if the number of JSON records is low (e.g. less than 1,000), which includes nested JSON arrays per record GIVEN the default XMX at 512m.

As the number of records increases, we noticed that the amount of time it takes to process increases perhaps disproportionately (or rather exponentially). And when the number of records hits the ceiling, the JSON Input step will fail with a memory issue -- increasing the XMX (above) will recover itself. Having said this, I have set XMX for kitchen.sh to be 2048m and have tried to process a 20,000 rows of JSON record. It finished successfully and the total time taken equals 15h 30m.

That is just taking too long. What other ways are there to improve the reading of the data file? Or, how to streamline the reading of JSON data like text/flat files? (NB: flat files we notice will stream data through as it reads)

Attached here are the transformation file and the sample 20k of JSON data here -- http://www.filedropper.com/1stcredit...ent2015-05-014.

Help please.
Attached Files
↧

Replace value with null

May 28, 2015, 3:28 am
$
0
0
Can anyone tell me how to replace "" with null values.
I have used if field value is null . But im getting '''''''''' instead of "".Then I have used javascript as value='""'. Here also im getting same issue.


Help me.
Ramya
↧
Search

bg-color formula creates empty rows when exporting to Excel / Excel 2007

May 27, 2015, 7:20 pm
$
0
0
Hello,

I have a super-simple report, that I want to export in excel format (vanilla or 2007). In one column however I want to have conditional background color, so the cell's bg-color has the following formula:

Code:
=IF([update_time] < TODAY()-6;"#ff0000"; if([update_time] < today()-3;"#ffff00";"#00ff00"))
If works fine, except that it creates an empty line under each "good" row. Actually it's kinda worse, it makes an empty line except for one cell which is "rowspanned" over the two rows.

This is without the formula, no empty rows:
http://storage1.static.itmages.com/i...5ecc0aa34e.png

This is exactly the same query, report etc, except that I added the bg-color formula in the last column:
http://storage3.static.itmages.com/i...6194a72481.png

I'd really appreciate some help :)

Alban
↧
↧

Creating table structure based on file contents(csv)

May 27, 2015, 11:43 pm
$
0
0
Hi All

I'm currently developing ETL process to load my DataWarehouse. One of the requirement is the ability to load data from files into tables which are not yet created - in other words: file is only put in directory, ETL process read this file to determine table name(from file name) number and names of columns(from headline) and types of columns(reading contents of the file) - based on this process should prepere create statement and run it on a database and in the next step load data from file into it.

I think that best way would be to use JavaScript to read file contents and prepare CREATE statement - save it into a file and in the next steps use standard kettle components to execute this SQL and then load data. So my question to you is - whether I am right or perhaps you have other ideas?

By the way, when I try to run JavaScript to read file
reader = new FileReader();

i got a fallowing error:
Couldn't compile javascript: org.mozilla.javascript.EcmaError: ReferenceError: "FileReader" is not defined. (<cmd>#1)

Could you tell me please what have should be done to use FileReader(); ?
↧

Error authenticating (check service name)

May 27, 2015, 11:50 pm
$
0
0
Hi Everyone,

I am using Pentaho v5.0.4. I am using the google analytics step from a few months. It was working well.
From yesterday, it started saying "check service name" and not allowing me to connect to google server.

Kindly help please.
↧

IDP PINK SAML Integration with Pentaho BI Server 5.2

May 28, 2015, 4:10 am
$
0
0
Hi,
Can anyone help in IDP PINK SAML Integration with Pentaho BI Server 5.2?
Facing trouble while configuring SAML with Pentaho

Regards,
Kumar Goutam
↧

How to convert complex XML and transform to required CSV. Please help on this

May 28, 2015, 5:11 am
$
0
0
hi,

I have multiple XML which is of complex structure. I am to read it and denormalise. but transforming to required CSV I am facing issue
I have tried for two days. Not able to find solution .

Please Please help on this.

I have attached required files and CSV format.

Hema
Attached Images
Attached Files
↧
↧

Question on Analytic query step

May 28, 2015, 6:20 am
$
0
0
I'm trying to figure out what the "group field" setting in the Analytic query step does?

Doc says this:
"Specify the fields you want to group. Click Get Fields to add all fields from the input stream(s). The step will do no additional sorting, so in addition to the grouping identified (for example CUSTOMER_ID) here you must also have the data sorted (for example ORDER_DATE)."

However, if I sort my input by order_date (but not customer_id), and then do a group field on customer_id, nothing happens - customer_ids still come in randomly. If I sort by both order and customer, and then skip the grouping, or turn it on, it doesn't seem to influence the output rows.

Could someone give more details on what this group field is supposed to do?

Thanks!

ps: another random query - why is it called analytic query?
↧

Template preview?

May 28, 2015, 6:56 am
$
0
0
The default template preview window shows a graphical preview of the template (rows, columns, etc); however, when I create a custom corporate template, which shows up under "My Templates", only a big "?" shows in the preview area. How can I get an image to show up there? Thanks.
↧
Search

Cross Join Problems

May 28, 2015, 7:23 am
$
0
0
Julian or anyone who can help,

This problem is driving me crazy

this is the mdx expression:
with set [~FILTER] as '{[FechaEnvio.Dias Fecha Envio Filtro].[2015].[2015-1].[2015-2].[04-2015]}'
set [~ROWS_FechaEnvio.Dias Fecha Envio] as '{[FechaEnvio.Dias Fecha Envio].[Dia Fecha Envio].Members}'
set [~ROWS_Beneficiario.Nombre Beneficiario] as '{[Beneficiario.Nombre Beneficiario].[Nombre Beneficiario].Members}'
set [~ROWS_Cliente.Nombre Cliente] as '{[Cliente.Nombre Cliente].[Nombre Cliente].Members}'
select NON EMPTY {[Measures].[Mto Eviado]} ON COLUMNS,
NON EMPTY (([~ROWS_FechaEnvio.Dias Fecha Envio] * [~ROWS_Beneficiario.Nombre Beneficiario]) * [~ROWS_Cliente.Nombre Cliente]) ON ROWS
from [operacionesdol]
where [~FILTER]

the xml mondrian schema is ok,

the DB star model ([Nombre Beneficiario] and [Nombre Cliente] are in the same table) (the fact table has Nombre Beneficiario foreign key and Nombre Cliente foreign key) and they consist of a list of client and their beneficiaries

the mdx log say:
2015-05-28 09:07:14,594 DEBUG [mondrian.mdx] 883: exec: 500523 ms
2015-05-28 09:11:43,646 DEBUG [mondrian.mdx] 608: exec: 1258083 ms

the SQL log :
2015-05-28 09:10:46,236 DEBUG [mondrian.sql] 90: , exec 13 ms
2015-05-28 09:10:46,237 DEBUG [mondrian.sql] 90: , exec+fetch 15 ms

But it just stay procesing and never give any result

Please! I've partitioned the time table and the client table, the query directly in the db client don't take even a second! I don't know what else I can do
↧

SunburstChart Colors

May 28, 2015, 7:54 am
$
0
0
Hi,
How can i set a pallet of colors for inner ring and outer ring depending on values?
I have already defined colors in chartDefinition like:


Code:
colors: ['#2f5665', '#70a8bd', '#2f5665', '#376576', '#3f7488', '#478399', '#4f92aa', '#5e9db5', '#70a8bd']
This is what i have:
sunburst.jpg

Regards
Attached Images
↧
↧

[CCC2] different valuesLabelStyle 'linked' //'inside' for each Slice?

May 28, 2015, 8:55 am
$
0
0
Hello,

on a CCC pie chart we can show labels inside or outside each slice using valuesLabelStyle on chart properties.

But that setting applies to the whole chart, all slices are affected.

I have a pie chart with too many small slices (so names are not readable) and I am unable to set valuesLabelStyle: outside only for those tiny slices:
2015_05_28_17_11_39_Pie.jpg

Does anyone know if it's even possible to specify a different label behaviour for each slice?

Regards.
Attached Images
↧

Authentication Problem in Google AnalyticsÀQ

May 28, 2015, 8:56 am
$
0
0
Hi All,

till 2 days ago Google Analytics input was working fine. Now there is issue of authentication. Message is "Error Authenticating (Check Service Name)". Is anyone else facing the same problem? How to resolve this problem.

I am using PDI 5.0

Thanks in Advance
Jumin
Attached Images
↧

xml process in transformation step

May 28, 2015, 10:42 am
$
0
0
I have a xml generated from the kettle stream. the next step I am going to process the output xml to create a aggregated xml. for instance, I am going to count how many times the tag</hello> in the xml then output the total numbers of the tag </hello> to the output xml. how to complete this in the kettle step?
↧
Viewing all 16689 articles
Browse latest View live
Search
<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>