Sending data to Apache Kafka Producer

May 29, 2014, 9:27 am

≫ Next: Issue with MySQL upgrade to 5.6 - 'OPTION SQL_SELECT_LIMIT=1'

≪ Previous: How to alter order of fields?

Hi guys,

I'm new with the product and would like to know how to send data from a JSON (or other) file into a Kafka Producer.

Thanks

↧

Issue with MySQL upgrade to 5.6 - 'OPTION SQL_SELECT_LIMIT=1'

May 29, 2014, 9:56 am

≫ Next: como hago un cubo

≪ Previous: Sending data to Apache Kafka Producer

We are upgrading to MySQL 5.6 from 5.1.30 to 5.6.18 .

When using the Pentaho Ketle for exacting daa we are getting the below error, we tried to upgrade the MySQL connector to 5.1.26 and some others as well..

You have an error in your SQL syntax ; check the manual that corresponds to your MySQL version for the right syntax to use near 'OPTION SQL_SELECT_LIMIT=1'.

Pease help with this.

Thanks !

↧

como hago un cubo

May 29, 2014, 10:07 am

≫ Next: exception to connect Hive from Pentaho

≪ Previous: Issue with MySQL upgrade to 5.6 - 'OPTION SQL_SELECT_LIMIT=1'

necesito hacer un cubo y no se por donde empezar que hago??

↧

exception to connect Hive from Pentaho

May 29, 2014, 12:12 pm

≫ Next: Transformation Database Transactional

≪ Previous: como hago un cubo

Hello, I am getting below exception when I try to connect Hive from Pentaho.

Hive JDBC: hive-jdbc-0.10.0-cdh4.3.0.jar

Connection Type: Hadoop Hive
Settings:
Host Name: <hostname>.com
Database Name: default
Port Number: 10000 (also tried 9083), getting same exception

Exception:
Could not establish connecton to jdbc:hive://<hostname>.com:10000/default: java.net.UnknownHostException: <hostname>.com

Connection Type: Hadoop Hive 2
Settings:
Host Name: <hostname>.com
Database Name: default
Port Number: 10000 (also tried 9083), getting same exception

Exception:
Invalid URL: jdbc:hive2://<hostname>.com:10000/default

Pls let me know if any suggestions or pointers to fix this? Thx in advance.

↧

Transformation Database Transactional

May 29, 2014, 12:15 pm

≫ Next: Table Component Expand property

≪ Previous: exception to connect Hive from Pentaho

If we are inserting rows into a table and lkping the row that was just inserted. Is this not possible in PDI? Isnt the data not committed to DB after every step? Or should this be done explicitly?

↧

Table Component Expand property

May 29, 2014, 12:37 pm

≫ Next: Change bar chart minimum

≪ Previous: Transformation Database Transactional

Hi, I made a dashboard with two tables that one expand the other. This was OK. The problem is the advanced properties table do not apply for the second table (the expand one, which have details for each row of the first table). For example the column format do not apply in the second one.
I really appreciate if anyone can help me with it. I would like to know if the extended table do not support the advanced prperties, or if I am doing something wrong.

Thanks.

↧

Change bar chart minimum

May 29, 2014, 1:03 pm

≫ Next: Random Problem - Error determining value metadata from SQL resultset metadata

≪ Previous: Table Component Expand property

I've got a set of data that is typically between 95-99%; we're wanting to display the data with a bar chart, but since the variations that we're interested in are happening in the last 5%, it is hard to see much detail due to the other 95% of the bar that is basically a waste of space. I'd like to display only the last 10%...I've tried setting "orthoAxisFixedMin" to 90. However, while this solves the above problem, it introduces others: 1) when the chart animates, there is a delay as the hidden 90% still has to "load", and 2) if I color the bars with a gradient, most of the gradient is hidden since the gradient starts from 0.

Basically I need to force the chart to accept 90% as the new 0%, and to treat it as such. Is there any way to do this? Thanks in advance!

↧

Random Problem - Error determining value metadata from SQL resultset metadata

May 29, 2014, 1:06 pm

≫ Next: Dimension lookup of SCD type 2 performing slow while loading RDS table.

≪ Previous: Change bar chart minimum

Hi Guys,

I need help with a problem that has been happening since I upgraded to 5.01A

I use mysql to store my repository (Native JDBC).

Every night I run a bunch of jobs that are started automatically using the crontab.

Every few days, I have one job that doesn't start and return the following error, I can't reproduce on demand and it can happen on any of the job randomly...

Could anyone suggest a possible cause for the problem? here's the error I get:

2014/05/29 04:00:02 - Kitchen - Start of run.
2014/05/29 04:00:02 - RepositoriesMeta - Reading repositories XML file: /data/pentaho/data-integration5/repositories.xml
Processing stopped because of an error:
An error occured loading the directory tree from the repository

Error loading a repository directory

ERROR executing query

Error determining value metadata from SQL resultset metadata
at org.pentaho.di.kitchen.Kitchen.main (Kitchen.java:283)
at org.pentaho.di.repository.kdr.KettleDatabaseRepository.loadRepositoryDirectoryTree (KettleDatabaseRepository.java:510)
at org.pentaho.di.repository.kdr.delegates.KettleDatabaseRepositoryDirectoryDelegate.loadRepositoryDirectoryTree (KettleDatabaseRepositoryDirectoryDelegate.java:100)
at org.pentaho.di.repository.kdr.delegates.KettleDatabaseRepositoryDirectoryDelegate.loadRepositoryDirectory (KettleDatabaseRepositoryDirectoryDelegate.java:119)
at org.pentaho.di.repository.kdr.delegates.KettleDatabaseRepositoryDirectoryDelegate.getDirectory (KettleDatabaseRepositoryDirectoryDelegate.java:48)
at org.pentaho.di.repository.kdr.delegates.KettleDatabaseRepositoryConnectionDelegate.getOneRow (KettleDatabaseRepositoryConnectionDelegate.java:1507)
at org.pentaho.di.core.database.Database.openQuery (Database.java:1694)
at org.pentaho.di.core.database.Database.getRowInfo (Database.java:2137)
at org.pentaho.di.core.database.Database.getValueFromSQLType (Database.java:2191)
at org.pentaho.di.core.row.value.ValueMetaBase.getValueFromSQLType (ValueMetaBase.java:4454)
at org.pentaho.di.core.row.value.ValueMetaFactory.createValueMeta (ValueMetaFactory.java:34)
at org.pentaho.di.core.row.value.ValueMetaFactory.createValueMeta (ValueMetaFactory.java:23)
at org.pentaho.di.core.plugins.PluginRegistry.getPlugin (PluginRegistry.java:255)
at org.pentaho.di.core.plugins.PluginRegistry.getPlugins (PluginRegistry.java:226)
at java.util.HashMap$KeyIterator.next (HashMap.java:845)
at java.util.HashMap$Ha****erator.nextEntry (HashMap.java:810)

ERROR: Kitchen can't continue because the job couldn't be loaded.

Here's my repository information:

<?xml version="1.0" encoding="UTF-8"?>
<repositories>
<connection>
<name>*********</name>
<server>*******</server>
<type>MYSQL</type>
<access>Native</access>
<database>***</database>
<port>***</port>
<username>*****</username>
<password>*****</password>
<servername/>
<data_tablespace/>
<index_tablespace/>
<attributes>
<attribute><code>EXTRA_OPTION_MYSQL.defaultFetchSize</code><attribute>500</attribute></attribute>
<attribute><code>EXTRA_OPTION_MYSQL.useCursorFetch</code><attribute>true</attribute></attribute>
<attribute><code>FORCE_IDENTIFIERS_TO_LOWERCASE</code><attribute>N</attribute></attribute>
<attribute><code>FORCE_IDENTIFIERS_TO_UPPERCASE</code><attribute>N</attribute></attribute>
<attribute><code>IS_CLUSTERED</code><attribute>N</attribute></attribute>
<attribute><code>PORT_NUMBER</code><attribute>3306</attribute></attribute>
<attribute><code>PRESERVE_RESERVED_WORD_CASE</code><attribute>N</attribute></attribute>
<attribute><code>QUOTE_ALL_FIELDS</code><attribute>N</attribute></attribute>
<attribute><code>STREAM_RESULTS</code><attribute>N</attribute></attribute>
<attribute><code>SUPPORTS_BOOLEAN_DATA_TYPE</code><attribute>Y</attribute></attribute>
<attribute><code>SUPPORTS_TIMESTAMP_DATA_TYPE</code><attribute>Y</attribute></attribute>
<attribute><code>USE_POOLING</code><attribute>N</attribute></attribute>
</attributes>
</connection>
<repository> <id>KettleDatabaseRepository</id>
<name>****</name>
<description>****</description>
<connection>****</connection>
</repository>
</repositories>

Any help would really be appreciated!

↧

Dimension lookup of SCD type 2 performing slow while loading RDS table.

May 29, 2014, 4:27 pm

≫ Next: Kettle 3.2: Updating specific records in the result set (Execute SQL Script?)

≪ Previous: Random Problem - Error determining value metadata from SQL resultset metadata

Hello all,

I am using Pentaho 5.0.1 community edition and I have table input step to read all the data from a local postgres database at the company headquaters and the next step is a dimension lookup/update step performing SCD type 2 where the order status and shipping status changes, inserts the data into a postgres RDS instance..

To read 300,000 records of data from the local company database, and insert into an amazon RDS instance, it takes almost 2 hours.

I have attached the DDL of the table in amazon RDS instance and the Kettle transformation. Also attached are the logs for test run I did for 2000 records.

How can I make this better to improve the performance?

What suggestions do you give me?

Thanks,

Ron

Attached Files

DDL.txt (3.2 KB)
Tr_Load_AllSubmittedOrders_Frm_Landing_To_PSA.ktr (36.3 KB)
logs.txt (731.4 KB)

↧

Kettle 3.2: Updating specific records in the result set (Execute SQL Script?)

May 29, 2014, 4:53 pm

≫ Next: Pentaho Reporting Video Course

≪ Previous: Dimension lookup of SCD type 2 performing slow while loading RDS table.

Hi Everyone,

Here's what I'm trying to do: I'm querying from a source database and got my result set, below is an example:

Field A	Field B
1	A
2	B
3	C

I want to update the 'A' to 'Andy', that's it, the rest stays the same. But I only have read/write access to the source database as it contains sensitive information. So which step should I use in Kettle 3.2 to update specific records in my result set?

Notes: I'm not updating fields/columns, I want to update specific records, so that eliminates the 'Update' and 'Insert/Update' steps. Also, I DO NOT WANT to actually update the source database, so would the 'Execute SQL Script' step work? Would it update the records that I want from the previous 'Table Input' step, but everything in the source database remains untouched? Any other steps should I consider?

I hope I was clear, any help would be greatly appreciated. Thanks very much.

David

↧

Pentaho Reporting Video Course

May 29, 2014, 8:01 pm

≫ Next: Set default value for a date parameter using a query

≪ Previous: Kettle 3.2: Updating specific records in the result set (Execute SQL Script?)

I would like to recommend this excellent video course created by my friend Francesco Corti and officially reviewed by Paul Hernandez and me. http://www.packtpub.com/pentaho-reporting/video Screenshots: Course Contents: Getting Started with Pentaho Reporting [15:57 minutes] Installing Pentaho Reporting Loading and Saving Reports … Continue reading →

More...

↧

Set default value for a date parameter using a query

May 29, 2014, 10:28 pm

≫ Next: Pentaho BI server error after login..

≪ Previous: Pentaho Reporting Video Course

Dear all

I have a requirement where I need to set the default value for a date parameter.

I need to fetch the date value using a query.

Is it possible to do this in PRD 3.9.1?

Thanks a lot in advance.

Regards
Vadi

↧

Pentaho BI server error after login..

May 29, 2014, 10:43 pm

≫ Next: GetXMLData XPATH's Current node

≪ Previous: Set default value for a date parameter using a query

From Last two week I am facing issue on Pentaho BI server 5.0 Community Edition.

when I am start server using Batch file then It will start and shows Home page after login.

When I am start server from service then after login nothing showing. menu, browse file is hidden....

any one know ????

Code:

INFO: Initiating Jersey application, version 'Jersey: 1.16 11/28/2012 03:18 PM'

May 30, 2014 10:44:27 AM com.sun.jersey.core.spi.component.ProviderFactory __getComponentProvider

SEVERE: The provider class, class com.sun.jersey.json.impl.provider.entity.JacksonProviderProxy, could not be instantiated. Processing will continue but the class will not be utilized

java.lang.RuntimeException: java.lang.reflect.InvocationTargetException

    at com.sun.jersey.core.spi.component.ComponentInjector.setMethodValue(ComponentInjector.java:168)

    at com.sun.jersey.core.spi.component.ComponentInjector.inject(ComponentInjector.java:133)

    at com.sun.jersey.core.spi.component.ComponentConstructor.getInstance(ComponentConstructor.java:180)

    at com.sun.jersey.core.spi.component.ProviderFactory.__getComponentProvider(ProviderFactory.java:166)

    at com.sun.jersey.core.spi.component.ProviderFactory.getComponentProvider(ProviderFactory.java:137)

    at com.sun.jersey.core.spi.component.ProviderServices.getComponent(ProviderServices.java:256)

    at com.sun.jersey.core.spi.component.ProviderServices.getServices(ProviderServices.java:160)

    at com.sun.jersey.core.spi.factory.MessageBodyFactory.initReaders(MessageBodyFactory.java:176)

    at com.sun.jersey.core.spi.factory.MessageBodyFactory.init(MessageBodyFactory.java:162)

    at com.sun.jersey.server.impl.application.WebApplicationImpl._initiate(WebApplicationImpl.java:1310)

    at com.sun.jersey.server.impl.application.WebApplicationImpl.access$700(WebApplicationImpl.java:168)

    at com.sun.jersey.server.impl.application.WebApplicationImpl$13.f(WebApplicationImpl.java:774)

    at com.sun.jersey.server.impl.application.WebApplicationImpl$13.f(WebApplicationImpl.java:770)

    at com.sun.jersey.spi.inject.Errors.processWithErrors(Errors.java:193)

Pentaho_Home_screen.jpg

Attached Images

Pentaho_Home_screen.jpg (41.3 KB)

↧

GetXMLData XPATH's Current node

May 30, 2014, 1:17 am

≫ Next: How can we cache the result set for PRPT reports

≪ Previous: Pentaho BI server error after login..

Hi!

given that the step iterates over a XPATH expression and then fetches a field accoridng to another expression,

I'd like to know how can I refer the current node of the iteration instead of the context node of the evaluated expresison. For instance

Code:

 for xml/entity



 field = xml/another/completly/different/path/field [.=current()/@value]

(. = context node, different than current which is an xml/entity)

↧

How can we cache the result set for PRPT reports

May 30, 2014, 2:52 am

≫ Next: Not able to copy files in Hadoop with regular expression

≪ Previous: GetXMLData XPATH's Current node

Hi ,

I have created some PRPT reports and published this to the BA server (not used pentaho dashboards )

I am using Kettle transformation as a data source as there are lot of data transformations between the actual source database and the presentation layer .

Currently , the report seems to be bit slow when I change the prompts..

Is there a way to cache the result sets in memory ?

Since I am using PDI as input to reports , Will a "Copy rows to Result " step as the data sourcing step work like a cache ?

Any clues ?

TIA

↧

Not able to copy files in Hadoop with regular expression

May 30, 2014, 3:05 am

≫ Next: If ... Else to compare fields

≪ Previous: How can we cache the result set for PRPT reports

Hi All,
Copyfiles.jpg
I have a small problem. I am trying to copy files from my local machine to the HDFS using a regular expression. I have specified the following

File/Folder Source : ${ARCHIVE_LOCATION} (Folder in my local machine.)
File/Folder Destination : ${HDFSSERVER}:${PORT}${TARGETFOLDER}${TARGETFILE} (File)
WildCard : .*abc*\.txt

In the source there is only one file meets the regular expression. But I am getting the error : "Can not copy folder [" to a file (The regular expression is evaluated correctly).
Please find the attached screen shot. Help would be greatly appreciated.

Attached Images

Copyfiles.jpg (29.2 KB)

↧

If ... Else to compare fields

May 30, 2014, 3:12 am

≫ Next: Convert doughnut chart to semi circular chart.

≪ Previous: Not able to copy files in Hadoop with regular expression

Hi guys,

Consider the input data table below - for illustration purposes only:

Customer	In_Car_Make	In_Car_Model
123	Toyota	Toyota Prius
456	Toyota	Toyota

The requirement is to pick the the customer and a car model. If the car model <> car make, I should pick the car model. If the car model is not supplied, I should pick the car make.
Assuming the output field is out_car_model
The logic should be:

Code:

if In_Car_Model == In_Car_Make then 

out_car_model = In_Car_Make

else

out_car_model = In_Car_Model

How do I achieve this in PDI?

↧

Convert doughnut chart to semi circular chart.

May 30, 2014, 5:08 am

≫ Next: Custom functions in mdx

≪ Previous: If ... Else to compare fields

We can convert pie chart into doughnut chart in CDE by writing extension points....

How can make 360* doughnut chart to 180* chart(i.e., semi circular dough nut chart).

No need of start and end points like gauge chart but just need to convert the circular dough nut chart to semi circular dough nut chart..

Any extension points ??

Thank you.

Sadakar

↧

Custom functions in mdx

May 30, 2014, 5:27 am

≫ Next: Error processing component - how to override message?

≪ Previous: Convert doughnut chart to semi circular chart.

Hi.
Can i add custom functions into mdx? For example, i want to add statistics functions from R. But I do not know how.

↧

Error processing component - how to override message?

May 30, 2014, 6:44 am

≫ Next: Upload data into DW

≪ Previous: Custom functions in mdx

We are running some line charts against many potential sets of data. In a few uncaught cases, the data set might be empty/invalid. I'm trying to check for errors with the following code in PostFetch:

var cccOptions = this.chartDefinition;
var eps = Dashboards.propertiesArrayToObject(cccOptions.extensionPoints);

eps.noDataMessage_text = "(No data found - chart cannot be displayed)";
eps.invalidDataMessage_text = "(No data found - chart cannot be displayed)";

These work most of the time; however, I am getting the following message in some cases: Error processing component(chartname). The cda shows that there were no data results. I would think my current text overrides above would work in this instance, but is there another extension point I should be targeting?

Thanks in advance!

↧

Latest Images