Hey there ---
I'm re-imagining my logging procedures. Trying to use more modular/ standard templates.
I've already done this in SSIS. Pentaho PDI has its uses too of course.
One thing I noticed is useful is a universal logging table -- not logging in separate tables for separate ETL.
One thing SSIS has that Pentaho PDI doesn't is a 'generated' unique job ID (Numeric) that doesn't change with name changes.
Pentaho doesn't have unique Job IDs, right? The only built-in Internal Variable for Job really is "Job Name". Names are changeable and can break referential links.
I'm wondering if anyone here has logging best practices. And before its mentioned, many 'out of the box' logging -- for Pentaho or SSIS -- is not very good. It's good if you're short on time and quite lazy, and then you probably will never look at it or spend way too much time digging throug hit.
I guess the answer is to create a variable "Job ID" for each job. Thing is, this pretty manual (to make it unique). So when the name changes in the job, the ID is persisted like in much technology paradigms. I know there are 'batch IDs' but that's different.
Again, to really make this concrete, you could check a log table and say "when's the last time the Load_Call ran succesfully, and how long did it take?". If this is based on name alone --- names don't change frequently if at all for jobs, but they could change. Right?
I'm re-imagining my logging procedures. Trying to use more modular/ standard templates.
I've already done this in SSIS. Pentaho PDI has its uses too of course.
One thing I noticed is useful is a universal logging table -- not logging in separate tables for separate ETL.
One thing SSIS has that Pentaho PDI doesn't is a 'generated' unique job ID (Numeric) that doesn't change with name changes.
Pentaho doesn't have unique Job IDs, right? The only built-in Internal Variable for Job really is "Job Name". Names are changeable and can break referential links.
I'm wondering if anyone here has logging best practices. And before its mentioned, many 'out of the box' logging -- for Pentaho or SSIS -- is not very good. It's good if you're short on time and quite lazy, and then you probably will never look at it or spend way too much time digging throug hit.
I guess the answer is to create a variable "Job ID" for each job. Thing is, this pretty manual (to make it unique). So when the name changes in the job, the ID is persisted like in much technology paradigms. I know there are 'batch IDs' but that's different.
Again, to really make this concrete, you could check a log table and say "when's the last time the Load_Call ran succesfully, and how long did it take?". If this is based on name alone --- names don't change frequently if at all for jobs, but they could change. Right?