Page 1 of 1
Reading Json Response that has large base64 strings
Posted: Mon Dec 05, 2022 10:10 pm
by Speedlime
Morning
I hope someone can help regarding this issue.
I am using:
* Reader
* Random Acces Json Reader
DEFINE_COM Class(#XPRIM_RandomAccessJsonReader) Name(#Reader)
DEFINE_COM Class(#XPRIM_ErrorInfo) Name(#ErrorInfo)
I believe this #Reader.ReadStringWithName( 'manifest' ).AsNativeString does not read all the data between " "
Reading like this:
#Reader.SetSourceHttpResponse Httpresponse(#Request.Response) Errorinfo(#ErrorInfo)
#Reader.BeginObjectWithPath Path('paperwork')
The following code reads the base64 block decodes it back to original state(pdf) and writes it to a blob on IFS
* Get manifest
#MyBase64String := #Reader.ReadStringWithName( 'manifest' ).AsNativeString
* Decode Base64 string
#HashBytes.FromBase64String String(#MyBase64String) Errorinfo(#ErrorInfo)
* place in blob
#LMCNCUS := #HashBytes.AsFile
#Reader.EndObject
Thinking my MyBase64String was to small I tried the biggest NVARCHAR variable I could make, got the same result
Then wrote the data to a file that had no size restriction like this
DEFINE Field(#wkOutputFile) Type(*STRING) Length(256)
#wkOutputFile := '/tmp/' + 'BASE64-' + #CPM_DNUM.AsDisplayString( NumString_L ).Trim + '.TXT'
SET Com(#FileStreamWriter) Fileaccess(Write) Filemode(CreateNew) Path(#wkOutputFile)
* Write data from the Json to the File
#FileStreamWriter.WriteChars( #Reader.ReadStringWithName( 'manifest' ).AsNativeString )
However the data was truncated in the file, as compared to the data supplied in the JSON for the label manifest.
How do I read larger blocks of base64 data out of the Json, as the RandomAccessJsonReader looks like it cannot cope with larger blocks of data.
Re: Reading Json Response that has large base64 strings
Posted: Tue Dec 06, 2022 5:48 am
by René Houba
Hi Speedlime,
What LANSA version is this and what platform?
Re: Reading Json Response that has large base64 strings
Posted: Tue Dec 06, 2022 9:12 am
by Dino
Ok, using this page:
https://base64.guru/converter/encode/pdf
I can upload a PDF and generate a json file, content of which looks like:
Code: Select all
{
"file": {
"mime": "application/pdf",
"data": "JVBERi0xLjQKJeLjz9MKMSAwIG9iago8PC9Db2x ...........moreeeeee. ............. PRgo="
}
}
i downloaded this file, and put in my webserver, images/temp, as testpdf.json.
then created this form to test it:
Code: Select all
Function Options(*DIRECT)
Begin_Com Role(*EXTENDS #PRIM_FORM) ClientWidth(904) ClientHeight(340) ComponentVersion(2) Left(524) Top(226)
Define_Com Class(#XPRIM_Binary) Name(#File)
Define_Com Class(#PRIM_PHBN) Name(#Button) DisplayPosition(1) Left(40) Parent(#COM_OWNER) TabPosition(1) Top(16) Caption('Reads PDF in JSON') Width(169)
Define_Com Class(#PRIM_STBR) Name(#StatusBar1) DisplayPosition(2) Height(24) Left(0) MessagePosition(1) Parent(#COM_OWNER) TabPosition(2) TabStop(False) Top(316) Width(904)
Define_Com Class(#XPRIM_RandomAccessJsonReader) Name(#Reader)
Define_Com Class(#XPRIM_ErrorInfo) Name(#ErrorInfo)
Define_Com Class(#XPRIM_HttpRequest) Name(#Request) Scope(*APPLICATION)
Evtroutine Handling(#Button.Click)
* using this page to generate base64 from pdf... https://base64.guru/converter/encode/pdf
#Request.DoGet Url('http://localhost:8080/images/temp/testpdf.json')
* Check if the server returned a response
If (#Request.Response.IsSuccessfulRequest)
If (#Request.Response.IsSuccessHttpStatusCode)
#Reader.SetSourceHttpResponse HttpResponse(#Request.Response) ErrorInfo(#ErrorInfo)
#Reader.BeginObjectWithPath( "file" )
* this returns "mime": "application/pdf"
#std_strng := #Reader.ReadStringWithName( "mime" ).AsNativeString
* this returns "data" : "JVBERi.......
#File.FromBase64String String(#Reader.ReadBase64StringWithPathIntoFile( "data" ))
#STD_STRNG := #File.AsFile
Message Msgtxt('***' + #std_strng + '***')
#Reader.EndObject
Else
Message Msgtxt(#Request.Response.ErrorCode.AsNativeString + ' / ' + #Request.Response.ErrorMessage.AsNativeString)
Endif
Else
Message Msgtxt(#Request.Response.ErrorCode.AsNativeString + ' / ' + #Request.Response.ErrorMessage.AsNativeString)
Endif
Endroutine
End_Com
and that works fine.
the trick was:
Code: Select all
#File.FromBase64String String(#Reader.ReadBase64StringWithPathIntoFile( "data" ))
std_strng contains the blob filename, File is the blob
tested with a 2MB pdf, no problem.
Re: Reading Json Response that has large base64 strings
Posted: Tue Dec 06, 2022 8:09 pm
by Speedlime
Hi Rene we are on V15 EPC 150050 to 56
Dino, thanks for you reply, I am going to go through this and apply to my solution. My pdf in base64 is embedded in a larger Json construct of which I am pulling various bits of the data out.
Re: Reading Json Response that has large base64 strings
Posted: Wed Dec 07, 2022 1:58 am
by Speedlime
Afternoon Dino
Thanks again for you help.
This is my code
#Reader.SetSourceHttpResponse Httpresponse(#Request.Response) Errorinfo(#ErrorInfo)
* Paperwork
#Reader.BeginObjectWithPath Path('paperwork')
* Manifest
#STD_STRNG := #Reader.ReadStringWithName( 'manifest' ).AsNativeString
* Decode Base64
#HashBytes.FromBase64String String(#Reader.ReadBase64StringWithPathIntoFile( "manifest" ))
#STD_STRNG := #HashBytes.AsFile
It works great and decodes the whole of the base64 block from the Json received
#STD_STRNG has the path to the blob. However ?
What I noticed is that 2 blobs are created in /tmp on the ifs, #STD_STRNG reference one that is 1kb and you cannot view anything in it, the second blob is 78kb which holds the PDF in it. I am not sure if the first one points to the second ?? Any ideas
What I am trying to do is write that blob to another directory on the IFS as a actual PDF
Code below.
#FromDocumentPath := #STD_STRNG.Trim
#HashBytes.FromFile Path(#FromDocumentPath.Trim)
* TO (Write Doc to IFS)
#ToDocumentPath :='/SOP/' + 'MANIFEST' + #CPM_DNUM.AsDisplayString( NumString_L ).Trim + '.PDF'
#HashBytes.AsFile Path(#ToDocumentPath.Trim) Errorinfo(#ErrorInfo)
Because #STD_STRNG is pointing to the 1k blob, I am writing a blank pdf
Re: Reading Json Response that has large base64 strings
Posted: Wed Dec 07, 2022 10:59 pm
by Speedlime
Some more testing
This line does produce a blob with the data
#HashBytes.FromBase64String String(#Reader.ReadBase64StringWithPathIntoFile( "manifest" ))
This line creates a second empty blob
#STD_STRNG := #HashBytes.AsFile
I cannot find anything in the debug that references this first blob created. Which is the one I need to work with.
Re: Reading Json Response that has large base64 strings
Posted: Thu Dec 08, 2022 2:15 am
by Dino
I didn't notice that at first, but now I added the Errorinfo parameter anf followed the debug, noted the problem with this line.
Code: Select all
#File.FromBase64String String(#Reader.ReadBase64StringWithPathIntoFile( "data" )) ErrorInfo(#ErrorInfo)
INVALID_INPUT_FORMAT
Not a valid BASE64 encoded string
but the temporarly file was created correctly, so, changed that line to this
(maybe I was converting twice from base64string?):
Code: Select all
#File.FromFile Path(#Reader.ReadBase64StringWithPathIntoFile( "data" )) ErrorInfo(#ErrorInfo)
and that works good. So now the next line:
works fine. Still, I can see two files are temporarily created, not just one. but both have the same content now.
Most likely #XPRIM_Binary needs to have a way to return filename property to avoid the duplication. Seems that #File.AsFile is creating a second blob at that time, effectively doing something like this:
Code: Select all
#File.FromFile Path(#Reader.ReadBase64StringWithPathIntoFile( "data" )) ErrorInfo(#ErrorInfo)
#STD_BLOB := #File.AsFile
#STD_STRNG := #STD_BLOB.FileName
sorry for the mistake
Re: Reading Json Response that has large base64 strings
Posted: Thu Dec 08, 2022 3:32 am
by Speedlime
Afternoon Dino
No apologies needed, I did not see it either, I guess I was so excited to get it to work. You are right in that the two line below work ever so slightly different.
#File.FromBase64String String(#Reader.ReadBase64StringWithPathIntoFile( "data" )) ErrorInfo(#ErrorInfo)
#File.FromFile Path(#Reader.ReadBase64StringWithPathIntoFile( "data" )) ErrorInfo(#ErrorInfo)
Thanks again for you help, and response. All working now and I can write the blob over to the another directory as a pdf.
Regards
Leon