Server routine timeout?

[Paulm]

Hello,

We are working with a file picker (client side) and the built-in function STM_FILE_OPEN + STM_FILE_READ (server side) to upload and parse CSV files but we are having some issues when processing large files in our code.

The upload works fine and the server routine 'CSVToList' works like we want with small sized files. However, when we upload a large file (for example, we have CSV files with 1 million lines), when the server routine takes more than 5 minutes to do its job we have some sort of timeout with a fatal error on the frontend, I have attached an image showing the error message.

Eventhough this error appears, the server routine still works on the IBM i and our database is fully populated after a few more minutes.

We can see the HTTP request staying unanswered for 5 minutes before the error occurs and ends correctly with a status code of 200.

However, the event #sys_web.RequestFailed is fired and the following code gives the following result:

Evtroutine Handling(#sys_web.RequestFailed) Reason(#Reason) Request(#Request)
#sys_web.console.Log( ('request failed: &1 Request: &2').substitute( #Reason #Request.Name ) )
Endroutine

request failed: SERVERERROR Request: CSVTOLIST

Is there a way to solve this? Do we have to change some settings?

Is there maybe a clean way to send the HTTP response to the client before the server routine ends its job?

Thank you for your help.

[BrendanB]

Unfortunately you have run into a BROWSER TIMEOUT...

to put it simply, most browsers will 'timeout' a request after 5 minutes of no response. (i think Safari might be 6 minutes, but the rest are 5 minutes).

a better way might simply to be:

have your server routine:

1. receive the file from the filepicker, SUBMIT a function to process the CSV file and return a 'job tracking number' to the client side.
2. the client can then periodically check on the status of that 'job' (your SUBMITted function should update a table that indicates if it is in-progress, complete, encountered an error etc..

this means that your user will not be waiting more than 5 minutes for a response, and the file will be processed as required.

[Theo de Bruin]

Hi,

This can possibly also be caused by the setting in the LANSA communications,
the the "keep-alive" setting on the listener details should be set to Y - see:
https://docs.lansa.com/15/en/lansa010/c ... r_recs.htm

Further in the LANSA WebAdministrator there are sometime-out settings which can
be set to another value than the default 300 seconds (5 minutes) - see:

https://docs.lansa.com/15/en/lansa085/i ... p_0380.htm
and
https://docs.lansa.com/15/en/lansa085/c ... p_0305.htm

Please try and let us know if that made any difference.

[RomainG]

Unfortunately you have run into a BROWSER TIMEOUT...

to put it simply, most browsers will 'timeout' a request after 5 minutes of no response. (i think Safari might be 6 minutes, but the rest are 5 minutes).

a better way might simply to be:

have your server routine:

1. receive the file from the filepicker, SUBMIT a function to process the CSV file and return a 'job tracking number' to the client side.
2. the client can then periodically check on the status of that 'job' (your SUBMITted function should update a table that indicates if it is in-progress, complete, encountered an error etc..

this means that your user will not be waiting more than 5 minutes for a response, and the file will be processed as required.

Hi Brendan,

I am working with Paul on this project.
Really sorry for the late reply, we were on holiday and we worked on a more urgent project.

I have tried your method of submiting a function to process the CSV but I am having trouble with the Streamreader, it does not seem to read the file. Could this be because the BLOB that I pass through in the exchange does not pass through correctly?

Here is my code:
1. In the srvroutine I call the submit function: Submit Process(POC033PROC) Function(HandleJob) Exchange(#BLOB)
2. In the function:
Function Options(*DIRECT)

Exchange Fields(#BLOB)
#COM_OWNER.ProcessFileTest( #BLOB.FileName )

Mthroutine Name(ProcessFileTest)
Define_Map For(*INPUT) Class(#PRIM_DC.UnicodeString) Name(#inFile)

Define_Com Class(#PRIM_IOC.FileStream) Name(#InFileStream)
Define_Com Class(#PRIM_IOC.StreamReader) Name(#StreamReader) Stream(#InFileStream)

Define_Com Class(#PRIM_DC.UnicodeString) Name(#ReadString)

#InFileStream.Path := #inFile

Dowhile Cond(#StreamReader.TryReadLine( #ReadString ))

Clr_List Named(#ParsedFields)
#COM_OWNER.ParseCSVLine( #ReadString.AsNativeString )
Selectlist Named(#ParsedFields)
#CIT002LINE := #ParsedField.RemoveCharacters( '"' )

Insert Fields(#CIT002LINE #CIT002RTNCOD) To_File(CIT002TEST)
Endselect

#count += 1

Endwhile

#CIT002JOBSTA := 'COMPLETED'
#CIT002JOBID := #CIT002JOBID
Insert Fields(#CIT002JOBID #CIT002JOBSTA) To_File(CIT002JOBSTT)

Endroutine

Any ideas on how I can make this work?

Thank you for your help ,

Romain

[RomainG]

After further investigation, the issue comes from the file being uploaded to a tmp folder in the IFS and when I submit my function I don't have access to the file anymore (maybe because the srvroutine ended before the end of my function?).

Is there a way to prevent this?

Or is there maybe a way to not delete the file until after the function finished its job?

I tried adding my file manually in a folder and it works fine so I just have to find a way for the file to not get deleted straight away.

[BrendanB]

Romain,

I would recommend that the server routine save the file (i usually copy it to somewhere else) and then delete it in the submitted function (so that you can ensure it is only deleted WHEN you are done with it...)

Code: Select all

#xDemoUnicode256 := (*TEMP_DIR + 'test.csv')
#COM_SELF.CopyFile fromFilePath(#STD_BLOB) toFilePath(#xDemoUnicode256.AsNativeString)

Submit Process(SBMPROC) Function(SBMF001) Exchange(#xDemoUnicode256)

Mthroutine Name(CopyFile)
Define_Map For(*INPUT) Class(#prim_alph) Name(#fromFilePath)
Define_Map For(*INPUT) Class(#prim_alph) Name(#toFilePath)
Define_Map For(*RESULT) Class(#prim_boln) Name(#Result)

Use Builtin(OV_FILE_SERVICE) With_Args("COPY_FILE" #fromFilePath #toFilePath) To_Get(#io$sts)

#Result := (#io$sts = "OK")

Endroutine

and your SBMF001 can do simply call a method like:

Code: Select all

#COM_SELF.RemoveFile( #FilePath True )

* =============================================================================
* RemoveFile
*   #Path         -- FilePath to remove
*   #forceRemoval -- Optionally Force Removal for READ_ONLY objects
*   #Result       -- Boolean indicating Success or Failure
*
Mthroutine Name(RemoveFile)
Define_Map For(*INPUT) Class(#prim_alph) Name(#Path)
Define_Map For(*INPUT) Class(#prim_boln) Name(#forceRemoval) Mandatory(True)
Define_Map For(*RESULT) Class(#prim_boln) Name(#Result)

Define_Com Class(#prim_alph) Name(#force) Value('FORCE')

If (#forceRemoval.IsFalse)

#force := ''

Endif

Use Builtin(OV_FILE_SERVICE) With_Args("REMOVE_FILE" #Path #force) To_Get(#io$sts)

#Result := (#io$sts = "OK")

Endroutine

[RomainG]

Brendan,

Thank you very much for your help, it was very helpful! Everything works perfectly now