I have been attempting to create a transformation in PDI that performs simple K-Means clustering on a dataset. I have the transformation set up and the Weka Scoring plugin is working perfectly.
K-Means is set up to create 2 clusters.
Weka Scoring outputs a field called cluster#_predicted, with values 0 for the first, and values 1 for the second cluster. The problem is that this transformation is to run on multiple datasets, and so the 0 and 1 cluster IDs need to be "standardized".
What I am looking for is a way of performing this inside the Kettle transformation:
So technically I want to make sure that cluster 1 is always the one with a higher average value on Field1, while cluster 0 has the lower average value.
If all else fails, due to the nature of the dataset, this simplification could be made (though I would really prefer the first option)
I have been going to all ends trying to figure out how to achieve this step. I have tried using filter rows, but it doesn't work. I can get the counts by using the Memory Group by step, but it outputs the counts to consecutive rows, and there is no option to compare those.
Can someone help me out?
K-Means is set up to create 2 clusters.
Weka Scoring outputs a field called cluster#_predicted, with values 0 for the first, and values 1 for the second cluster. The problem is that this transformation is to run on multiple datasets, and so the 0 and 1 cluster IDs need to be "standardized".
What I am looking for is a way of performing this inside the Kettle transformation:
Code:
IF(
Average Value for [Field1] where [cluster#_predicted=0] > Average Value for [Field1] where [cluster#_predicted=1]
THEN
Replace Value [cluster#_predicted] 0 --> 1; 1-->0
ELSE
Do Nothing
)If all else fails, due to the nature of the dataset, this simplification could be made (though I would really prefer the first option)
Code:
IF(
Count of [cluster#_predicted=1] > Count of [cluster#_predicted=0]
THEN
Replace Value [cluster#_predicted] 0 --> 1; 1-->0
ELSE
Do Nothing
)Can someone help me out?