Reading Json Response that has large base64 strings

[Speedlime]

Morning
I hope someone can help regarding this issue.

I am using:
* Reader
* Random Acces Json Reader
DEFINE_COM Class(#XPRIM_RandomAccessJsonReader) Name(#Reader)
DEFINE_COM Class(#XPRIM_ErrorInfo) Name(#ErrorInfo)

I believe this #Reader.ReadStringWithName( 'manifest' ).AsNativeString does not read all the data between " "

Reading like this:

#Reader.SetSourceHttpResponse Httpresponse(#Request.Response) Errorinfo(#ErrorInfo)
#Reader.BeginObjectWithPath Path('paperwork')

The following code reads the base64 block decodes it back to original state(pdf) and writes it to a blob on IFS

* Get manifest
#MyBase64String := #Reader.ReadStringWithName( 'manifest' ).AsNativeString
* Decode Base64 string
#HashBytes.FromBase64String String(#MyBase64String) Errorinfo(#ErrorInfo)
* place in blob
#LMCNCUS := #HashBytes.AsFile
#Reader.EndObject

Thinking my MyBase64String was to small I tried the biggest NVARCHAR variable I could make, got the same result

Then wrote the data to a file that had no size restriction like this

DEFINE Field(#wkOutputFile) Type(*STRING) Length(256)
#wkOutputFile := '/tmp/' + 'BASE64-' + #CPM_DNUM.AsDisplayString( NumString_L ).Trim + '.TXT'
SET Com(#FileStreamWriter) Fileaccess(Write) Filemode(CreateNew) Path(#wkOutputFile)
* Write data from the Json to the File
#FileStreamWriter.WriteChars( #Reader.ReadStringWithName( 'manifest' ).AsNativeString )

However the data was truncated in the file, as compared to the data supplied in the JSON for the label manifest.

How do I read larger blocks of base64 data out of the Json, as the RandomAccessJsonReader looks like it cannot cope with larger blocks of data.

[René Houba]

Hi Speedlime,

What LANSA version is this and what platform?

[Dino]

Ok, using this page:
https://base64.guru/converter/encode/pdf

I can upload a PDF and generate a json file, content of which looks like:

Code: Select all

{
  "file": {
    "mime": "application/pdf",
    "data": "JVBERi0xLjQKJeLjz9MKMSAwIG9iago8PC9Db2x  ...........moreeeeee. ............. PRgo="
  }
}

i downloaded this file, and put in my webserver, images/temp, as testpdf.json.

then created this form to test it:

Code: Select all

Function Options(*DIRECT)
Begin_Com Role(*EXTENDS #PRIM_FORM) ClientWidth(904) ClientHeight(340) ComponentVersion(2) Left(524) Top(226)

Define_Com Class(#XPRIM_Binary) Name(#File)

Define_Com Class(#PRIM_PHBN) Name(#Button) DisplayPosition(1) Left(40) Parent(#COM_OWNER) TabPosition(1) Top(16) Caption('Reads PDF in JSON') Width(169)
Define_Com Class(#PRIM_STBR) Name(#StatusBar1) DisplayPosition(2) Height(24) Left(0) MessagePosition(1) Parent(#COM_OWNER) TabPosition(2) TabStop(False) Top(316) Width(904)

Define_Com Class(#XPRIM_RandomAccessJsonReader) Name(#Reader)
Define_Com Class(#XPRIM_ErrorInfo) Name(#ErrorInfo)
Define_Com Class(#XPRIM_HttpRequest) Name(#Request) Scope(*APPLICATION)

Evtroutine Handling(#Button.Click)
* using this page to generate base64 from pdf... https://base64.guru/converter/encode/pdf
#Request.DoGet Url('http://localhost:8080/images/temp/testpdf.json')

* Check if the server returned a response
If (#Request.Response.IsSuccessfulRequest)
If (#Request.Response.IsSuccessHttpStatusCode)
#Reader.SetSourceHttpResponse HttpResponse(#Request.Response) ErrorInfo(#ErrorInfo)
#Reader.BeginObjectWithPath( "file" )
* this returns     "mime": "application/pdf"
#std_strng := #Reader.ReadStringWithName( "mime" ).AsNativeString
* this returns "data" : "JVBERi.......
#File.FromBase64String String(#Reader.ReadBase64StringWithPathIntoFile( "data" ))
#STD_STRNG := #File.AsFile
Message Msgtxt('***' + #std_strng + '***')
#Reader.EndObject
Else
Message Msgtxt(#Request.Response.ErrorCode.AsNativeString + ' / ' + #Request.Response.ErrorMessage.AsNativeString)
Endif
Else
Message Msgtxt(#Request.Response.ErrorCode.AsNativeString + ' / ' + #Request.Response.ErrorMessage.AsNativeString)
Endif
Endroutine
End_Com

and that works fine.

the trick was:

Code: Select all

#File.FromBase64String String(#Reader.ReadBase64StringWithPathIntoFile( "data" ))

std_strng contains the blob filename, File is the blob
tested with a 2MB pdf, no problem.

[Speedlime]

Hi Rene we are on V15 EPC 150050 to 56
Dino, thanks for you reply, I am going to go through this and apply to my solution. My pdf in base64 is embedded in a larger Json construct of which I am pulling various bits of the data out.

[Speedlime]

Afternoon Dino
Thanks again for you help.

This is my code

#Reader.SetSourceHttpResponse Httpresponse(#Request.Response) Errorinfo(#ErrorInfo)
* Paperwork
#Reader.BeginObjectWithPath Path('paperwork')
* Manifest
#STD_STRNG := #Reader.ReadStringWithName( 'manifest' ).AsNativeString
* Decode Base64
#HashBytes.FromBase64String String(#Reader.ReadBase64StringWithPathIntoFile( "manifest" ))
#STD_STRNG := #HashBytes.AsFile

It works great and decodes the whole of the base64 block from the Json received

#STD_STRNG has the path to the blob. However ?

What I noticed is that 2 blobs are created in /tmp on the ifs, #STD_STRNG reference one that is 1kb and you cannot view anything in it, the second blob is 78kb which holds the PDF in it. I am not sure if the first one points to the second ?? Any ideas

What I am trying to do is write that blob to another directory on the IFS as a actual PDF
Code below.

#FromDocumentPath := #STD_STRNG.Trim
#HashBytes.FromFile Path(#FromDocumentPath.Trim)

* TO (Write Doc to IFS)
#ToDocumentPath :='/SOP/' + 'MANIFEST' + #CPM_DNUM.AsDisplayString( NumString_L ).Trim + '.PDF'
#HashBytes.AsFile Path(#ToDocumentPath.Trim) Errorinfo(#ErrorInfo)

Because #STD_STRNG is pointing to the 1k blob, I am writing a blank pdf

[Speedlime]

Some more testing
This line does produce a blob with the data
#HashBytes.FromBase64String String(#Reader.ReadBase64StringWithPathIntoFile( "manifest" ))
This line creates a second empty blob
#STD_STRNG := #HashBytes.AsFile

I cannot find anything in the debug that references this first blob created. Which is the one I need to work with.

[Dino]

I didn't notice that at first, but now I added the Errorinfo parameter anf followed the debug, noted the problem with this line.

Code: Select all

#File.FromBase64String String(#Reader.ReadBase64StringWithPathIntoFile( "data" )) ErrorInfo(#ErrorInfo)

Ok, FALSE
INVALID_INPUT_FORMAT
Not a valid BASE64 encoded string

but the temporarly file was created correctly, so, changed that line to this
(maybe I was converting twice from base64string?):

Code: Select all

#File.FromFile Path(#Reader.ReadBase64StringWithPathIntoFile( "data" )) ErrorInfo(#ErrorInfo)

and that works good. So now the next line:

Code: Select all

#STD_STRNG := #File.AsFile

works fine. Still, I can see two files are temporarily created, not just one. but both have the same content now.
Most likely #XPRIM_Binary needs to have a way to return filename property to avoid the duplication. Seems that #File.AsFile is creating a second blob at that time, effectively doing something like this:

Code: Select all

#File.FromFile Path(#Reader.ReadBase64StringWithPathIntoFile( "data" )) ErrorInfo(#ErrorInfo)
#STD_BLOB := #File.AsFile
#STD_STRNG := #STD_BLOB.FileName

sorry for the mistake

[Speedlime]

Afternoon Dino

No apologies needed, I did not see it either, I guess I was so excited to get it to work. You are right in that the two line below work ever so slightly different.
#File.FromBase64String String(#Reader.ReadBase64StringWithPathIntoFile( "data" )) ErrorInfo(#ErrorInfo)
#File.FromFile Path(#Reader.ReadBase64StringWithPathIntoFile( "data" )) ErrorInfo(#ErrorInfo)

Thanks again for you help, and response. All working now and I can write the blob over to the another directory as a pdf.

Regards
Leon