I'm trying to do something that seems like it ought to be simple in PDI, but I haven't been able to find any way to do it in the documentation I've found so far.
I have a situation where I have a number of CVS files being generated with different names and internal columns. Each file contains rows that need to be inserted into a corresponding table in a database. This means that the transformation to do the work has a number of repetitive steps: Text File Input to read a file, Table Output to write it. (It needs to be a single transformation, set as transactional, because all of the table changes need to be rolled back if any of them generates an error.) Text File Input can get the name of its file to read from a field in its input data, and likewise Table Output can take the name of the database table from a field in the input. So this seems to be a natural place to write a single sub-transformation that can then be called with a different input (the source file name) and will handle the steps I've just described in a uniform fashion for every input file.
My problem is that each table has a different set of fields (which is why the update arrives as multiple files), and there seems to be no way to persuade Text File Input to autodetect the fields in its input at runtime. The autodetect logic is present somewhere -- it's what the Get Fields button does in the step editing dialog. But unless I manually predefine the fields that Text File Input is to read, it generates no output. This means, in turn, that there is apparently no way to automate this task to a common sub-transformation; I have to write the steps explicitly, by hand, for each file/table pair. It seems absurd that the autodetection functionality isn't available at runtime, when the code obviously exists to support the action of the "Get Fields" button, but I haven't found anything in the documentation that points to a way to access it.
I have a situation where I have a number of CVS files being generated with different names and internal columns. Each file contains rows that need to be inserted into a corresponding table in a database. This means that the transformation to do the work has a number of repetitive steps: Text File Input to read a file, Table Output to write it. (It needs to be a single transformation, set as transactional, because all of the table changes need to be rolled back if any of them generates an error.) Text File Input can get the name of its file to read from a field in its input data, and likewise Table Output can take the name of the database table from a field in the input. So this seems to be a natural place to write a single sub-transformation that can then be called with a different input (the source file name) and will handle the steps I've just described in a uniform fashion for every input file.
My problem is that each table has a different set of fields (which is why the update arrives as multiple files), and there seems to be no way to persuade Text File Input to autodetect the fields in its input at runtime. The autodetect logic is present somewhere -- it's what the Get Fields button does in the step editing dialog. But unless I manually predefine the fields that Text File Input is to read, it generates no output. This means, in turn, that there is apparently no way to automate this task to a common sub-transformation; I have to write the steps explicitly, by hand, for each file/table pair. It seems absurd that the autodetection functionality isn't available at runtime, when the code obviously exists to support the action of the "Get Fields" button, but I haven't found anything in the documentation that points to a way to access it.