Hi,
I have a PDI 4.4 job that Decrypts a PGP file (to disk) and then starts a transformation to read in the new file (which is CSV once decrypted) and save the contents to a Casandra cluster. That works. Great, I'm happy.
However, I've been given a new requirement that we don't want to dump the unencrypted information to disk for both security and space reasons. I noticed that PDI 5.1 has PGP Decrypt stream step, so I thought I'd install and try out the new version. I used Load file content into memory and then funneled it into PGP Decrypt stream. For my first test with a tiny test file it worked great, I thought I was good. However, I then tried a larger file and realized that I didn't get all my data. The output from PGP Decrypt stream seems to be truncated to 32768 characters (32KiB). And there was no error or warning, it's only because I knew what my test data should have been that I realized there was a problem.
So my questions are:
1] Is this limit a hard stop, or is there a way around it?
2] If it's a limit that I can't get around, is there a way to stream the file from Decrypt files with PGP instead of writing to disk?
Thanks,
Linnea
I have a PDI 4.4 job that Decrypts a PGP file (to disk) and then starts a transformation to read in the new file (which is CSV once decrypted) and save the contents to a Casandra cluster. That works. Great, I'm happy.
However, I've been given a new requirement that we don't want to dump the unencrypted information to disk for both security and space reasons. I noticed that PDI 5.1 has PGP Decrypt stream step, so I thought I'd install and try out the new version. I used Load file content into memory and then funneled it into PGP Decrypt stream. For my first test with a tiny test file it worked great, I thought I was good. However, I then tried a larger file and realized that I didn't get all my data. The output from PGP Decrypt stream seems to be truncated to 32768 characters (32KiB). And there was no error or warning, it's only because I knew what my test data should have been that I realized there was a problem.
So my questions are:
1] Is this limit a hard stop, or is there a way around it?
2] If it's a limit that I can't get around, is there a way to stream the file from Decrypt files with PGP instead of writing to disk?
Thanks,
Linnea