Quantcast
Channel: Pentaho Community Forums
Viewing all articles
Browse latest Browse all 16689

Any help in using ETL Metadata Injection step for dynamic number of columns?

$
0
0
Hi ,
I have been struggling to understand the ETL metadata Injection step to use it my scenario with the help of available resources on google. However , Iam definitely missing something. My requirement is :

1.I have a directory where CSV files keep coming in. My kettle job is scheduled to run 2-3 times a day to look for any new files in the folder and then post the CSV file data into MongoDB.

2. The challenge is that the CSV files that come in this directory may have different number of columns. the first 10 columns will be same for any CSV's. However, beyond that some files could have 15 columns, or 12 columns ..i.e the column numbers are dynamic and also the column names.

In most of the examples of ETL Metadata Injection, I notice that the column names are provided. However, in my scenario I do not know in advance the column names or even the number of columns my CSV might have. Since MongoDB will just store the JSON format, dynamic number of columns does not matter.

Is this possible through the ETL Metadata Injection step?

Any help appreciated.

Thanks,
Puneeta

Viewing all articles
Browse latest Browse all 16689

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>