STM_FILE BIFS

[jimwatterson]

Hi

On out Webevent websites we often allow end users to upload csv files to our system to bulk upload data. Once the file is on the IFS (iSeries) we use

Change Field(#OPTIONS) To('''Read Text LineTerminator=ALL''')
Use Builtin(STM_FILE_OPEN) With_Args(#IFSLNM #OPTIONS) To_Get(#FILENO #WRK2A)
Use Builtin(STM_FILE_READ) With_Args(#FILENO) To_Get(#WRK256 #WRK2A)

Occasionally these files contain an invalid character, commonly a user's name has a single quote in it (') but because it has been imported from another system this has been converted to a right handed quote (comma with tail pointing down on the right). When STM_FILE_READ encounters such a character it immediately returns end of file. We may received some lines but the end effect is that we can't read the file to the end.

If I look at the file using wrklnk 5(Display) then I can see all the data fine - the invalid character appears as a blank and a message on the bottom line says something like error on line 12 will be saved as blank.

We can't stop the users from doing this but it should be possible to read the file, identify bad characters, replace them if we can and set them to blank if we can't. But how?

Has anyone got any ideas about how to access and IFS file which may contain dodgy characters in LANSA and allow us to fix them up?

[MarkD]

If you have a sample of the bad data could you maybe try out using the TRANSFORM_FILE built-in function?

https://docs.lansa.com/14/en/lansa015/i ... m_file.htm

Note how it has an option to handle 'bad' characters - but it does not well define what they might be.

If the CSV input data is columunized you could possibly define the target list as a set of fields (i.e. the individual columns in a CSV row).
Then TRANSFORM_FILE should handle stripping the CSV data down to field values for you.

If not, maybe define the list as just one long field and read the raw data, much like the STM BIF does.

[jimwatterson]

Thanks Mark, I'll give transform list a go.

Its a great utility, much, much faster than anything I could write but I do find it sensitive - some files it refuses to read without any explanation of what's wrong with them.

It would be nice if the STM file BIFS had an option to deal with 'bad' characters!

Cheers
Jim

[BrendanB]

Jim,

you may be able to use something like:

Code: Select all

Mthroutine Name(CleanFile)
Define_Map For(*INPUT) Class(#PRIM_DC.UnicodeString) Name(#InputFilePath)
Define_Map For(*RESULT) Class(#PRIM_DC.UnicodeString) Name(#Result)

Define_Com Class(#PRIM_IOC.FileStream) Name(#FileStream)
Define_Com Class(#PRIM_IOC.StreamReader) Name(#Reader) Stream(#FileStream)
Define_Com Class(#PRIM_DC.UnicodeString) Name(#String)

#FileStream.Path := #InputFilePath

Dowhile Cond(#Reader.TryReadLine( #String ))

#Result += #String.ReplaceAll( (96).asUnicodeString) "'" ).ReplaceAll( (180).asUnicodeString) "'" ) 

Endwhile

Endroutine

To acheive what you are after....

#PRIM_IOC.FileStream is not well documented, so can be a bit frustrating to use... but this should read a 'line' at a time (in theory, it reads up until it finds either a CRLF, LF or EOF ).

FileStream is quite fast... so this may be enough... downside is that you must use RDMLX when doing it.

[jimwatterson]

Brendan,

This looks like the kind of thing I'm after. I'll give it a go.

Cheers
Jim