Reading huge JSON data in web service invocation....

[tsupartono]

The XPRIM_JsonReader (and XPRIM_RandomAccessJsonReader), which is part of the RDMLX Extended Library, does not perform well when reading large amount of JSON data, as it's an RDML library, as opposed to a native VL feature.
Also, performance on the IBM i is definitely worse than on Windows.

If your web service invocation returns large JSON data, consider switching to the VL native PRIM_JSON.Reader (note it's PRIM, not XPRIM).

As it's completely integrated and built-in to VL, PRIM_JSON.Reader performs much better. On the IBM i, you can expect PRIM_JSON.Reader to run at least 20x faster than XPRIM_JsonReader.

Right now you would have to write the HTTP response first to a temporary file.
See code example below.

Code: Select all

Define_Com Class(#PRIM_IOC.FileStream) Name(#FileStream)
Define_Com Class(#PRIM_IOC.StreamReader) Name(#TextReader) Stream(#FileStream)
Define_Com Class(#PRIM_JSON.Reader) Name(#JsonReader) TextReader(#TextReader)
Define_Com Class(#XPRIM_File) Name(#TempFile)

* Create empty temporary file
#TempFile.CreateTemporaryFile

* Save HTTP response to temporary file
#HttpRequest.Response.AsFile AutoDelete(True) Path(#TempFile)

* Do the reading using JsonReader
#FileStream.Path := #TempFile

Dowhile Cond(#JsonReader.Read)
   * DO your stuff here
Endwhile

[soa]

Tony

As I've mentioned elsewhere in this forum the speed improvement of PRIM_JSON.Reader over XPRIM_JSON.Reader is significant to the point where for one real life use case XPRIM_JSON.Reader was practically useless. I was going to have to poll a service every two minutes and take more than two minutes to read the results!

I have two questions though.

Why is this not documented? If it was not for this forum nobody would know this library existed.

Secondly, in the absence of documentation, I have had to extract the contents of the json by stepping through tokens returned and keeping track of the start and end of every object, array and element. For a single one dimensional array this not hard but for more complex data would soon become unmanageable. Surely its possible to implement in the PRIM_JSON.Reader same capabilities as XPRIM_JSON.Reader - ReadBeginArray, ReadObject and #item.GetString('NAME') etcetera?

I know that there is development effort involved and resources are limited but, as I said, the performance difference is so stark that using XPRIM_JSON.Reader now seems wrong!

Should I raise this with support?

Has anybody else tried this library?

[tsupartono]

Jim,
A couple of comments.

• The RDML Extended Library (XPRIM components) was created so that new & experimental features can be released faster to customers. Some components will, in due time, be made into proper VL PRIMs (such as the JSON reader). The price to pay however is that the XPRIM components (such as XPRIM_HttpRequest) are not interoperable with proper VL PRIMs. For the time being, in order to read your HTTP responses using the native PRIM_JSON.Reader, you would need to write the HTTP response to a temporary file first. For simple HTTP request that does not return a huge amount of data, this could be an unnecessary overhead, so keep this in mind before you change everything over to use the native PRIM_JSON. Until we have a VL native PRIM_HttpRequest component, use native PRIM_JSON only when you see a definite performance improvement.

• The native PRIM_JSON is provided as part of the publishing web services feature, and the team is currently working hard on creating a set of guide and examples for publishing web services, which should include documentation/samples on PRIM_JSON. Please contact LANSA Support for more information on this, include also your question if there are easier ways to extract the JSON without having to go through the tokens.

[soa]

Thanks for the response. For the majority of our use cases XPRIM is the easiest way to go and we will not be making wholesale changes in this area. For those cases with large data sets the PRIM stuff is a lifesaver and has allowed me to look at different ways of doing a couple of things. So no matter what I am happy & grateful for what we are getting.

It makes sense that this is part of the web service stuff - I had forgotten about that. That is another development we're looking forward to seeing.

Thanks again for your good work and your responsiveness in this forum.

[adale]

Does anyone happen to have a working link for documentation on the PRIM_JSON.Reader?

I can find the doc links for the new XPRIM_JSON stuff, but am having trouble finding documentation for the PRIM_JSON ?

[stevelee67]

this is definitely something for me to look in to. i'm only familiar with the XPRIM object (which we used when 14.2 was released).

is this something from later in the 14.2 or perhaps 15.x cycle?

do these kinds of speed improvements translate to a windoz environment? and is there a writer side component to this?

[jimwatterson]

The #PRIM (as opposed to #XPRIM) libraries are part of version 15's API support and are still largely undocumented. The XPRIM stuff on the iSeries is, I believe, built on top of some IBM Java libraries whereas the new stuff is native (C++?). I don't know how this works on the PC but I don't know how bad windows performance is.

If you search this forum for #PRIM_JSON.Reader or #PRIM_JSON.Document I'm sure you will find some examples.

Here is an example of #PRIM_JSON.Document parsing json from a string

Define_Com Class(#PRIM_JSON.Document) Name(#lDocument)
Define_Com Class(#PRIM_JSON.Object) Name(#lRequestObject) Reference(*DYNAMIC)
Define_Com Class(#PRIM_DC.UnicodeString) Name(#UniCodeString)

#UniCodeString := #Request.Response.AsString

#lDocument.LoadFromString( #UniCodeString.AsNativeString )

For Each(#Child) In(#lDocument.RootNode)

#STD_NUM := #Child<'id'>.AsInt32
#STD_STR := #Child<'response'>.AsString

Endfor