Reading huge JSON data in web service invocation....

This forum allows developers to post programming tips and coding techniques that may be useful to other Visual LANSA developers. The information contained in this forum has not been validated by LANSA and, as such, LANSA cannot guarantee the accuracy of the information.
Post Reply
tsupartono
Posts: 74
Joined: Wed Jan 25, 2017 11:12 am

Reading huge JSON data in web service invocation....

Post by tsupartono » Mon Mar 25, 2019 5:53 pm

The XPRIM_JsonReader (and XPRIM_RandomAccessJsonReader), which is part of the RDMLX Extended Library, does not perform well when reading large amount of JSON data, as it's an RDML library, as opposed to a native VL feature.
Also, performance on the IBM i is definitely worse than on Windows.

If your web service invocation returns large JSON data, consider switching to the VL native PRIM_JSON.Reader (note it's PRIM, not XPRIM).

As it's completely integrated and built-in to VL, PRIM_JSON.Reader performs much better. On the IBM i, you can expect PRIM_JSON.Reader to run at least 20x faster than XPRIM_JsonReader.

Right now you would have to write the HTTP response first to a temporary file.
See code example below.

Code: Select all

Define_Com Class(#PRIM_IOC.FileStream) Name(#FileStream)
Define_Com Class(#PRIM_IOC.StreamReader) Name(#TextReader) Stream(#FileStream)
Define_Com Class(#PRIM_JSON.Reader) Name(#JsonReader) TextReader(#TextReader)
Define_Com Class(#XPRIM_File) Name(#TempFile)

* Create empty temporary file
#TempFile.CreateTemporaryFile

* Save HTTP response to temporary file
#HttpRequest.Response.AsFile AutoDelete(True) Path(#TempFile)

* Do the reading using JsonReader
#FileStream.Path := #TempFile

Dowhile Cond(#JsonReader.Read)
   * DO your stuff here
Endwhile

soa
Posts: 269
Joined: Mon Dec 07, 2015 3:15 pm

Re: Reading huge JSON data in web service invocation....

Post by soa » Thu Mar 28, 2019 10:53 am

Tony

As I've mentioned elsewhere in this forum the speed improvement of PRIM_JSON.Reader over XPRIM_JSON.Reader is significant to the point where for one real life use case XPRIM_JSON.Reader was practically useless. I was going to have to poll a service every two minutes and take more than two minutes to read the results!

I have two questions though.

Why is this not documented? If it was not for this forum nobody would know this library existed.

Secondly, in the absence of documentation, I have had to extract the contents of the json by stepping through tokens returned and keeping track of the start and end of every object, array and element. For a single one dimensional array this not hard but for more complex data would soon become unmanageable. Surely its possible to implement in the PRIM_JSON.Reader same capabilities as XPRIM_JSON.Reader - ReadBeginArray, ReadObject and #item.GetString('NAME') etcetera?

I know that there is development effort involved and resources are limited but, as I said, the performance difference is so stark that using XPRIM_JSON.Reader now seems wrong!

Should I raise this with support?

Has anybody else tried this library?

tsupartono
Posts: 74
Joined: Wed Jan 25, 2017 11:12 am

Re: Reading huge JSON data in web service invocation....

Post by tsupartono » Thu Mar 28, 2019 5:00 pm

Jim,
A couple of comments.
  • The RDML Extended Library (XPRIM components) was created so that new & experimental features can be released faster to customers. Some components will, in due time, be made into proper VL PRIMs (such as the JSON reader). The price to pay however is that the XPRIM components (such as XPRIM_HttpRequest) are not interoperable with proper VL PRIMs. For the time being, in order to read your HTTP responses using the native PRIM_JSON.Reader, you would need to write the HTTP response to a temporary file first. For simple HTTP request that does not return a huge amount of data, this could be an unnecessary overhead, so keep this in mind before you change everything over to use the native PRIM_JSON. Until we have a VL native PRIM_HttpRequest component, use native PRIM_JSON only when you see a definite performance improvement.
  • The native PRIM_JSON is provided as part of the publishing web services feature, and the team is currently working hard on creating a set of guide and examples for publishing web services, which should include documentation/samples on PRIM_JSON. Please contact LANSA Support for more information on this, include also your question if there are easier ways to extract the JSON without having to go through the tokens.

soa
Posts: 269
Joined: Mon Dec 07, 2015 3:15 pm

Re: Reading huge JSON data in web service invocation....

Post by soa » Fri Mar 29, 2019 8:33 am

Thanks for the response. For the majority of our use cases XPRIM is the easiest way to go and we will not be making wholesale changes in this area. For those cases with large data sets the PRIM stuff is a lifesaver and has allowed me to look at different ways of doing a couple of things. So no matter what I am happy & grateful for what we are getting.

It makes sense that this is part of the web service stuff - I had forgotten about that. That is another development we're looking forward to seeing.

Thanks again for your good work and your responsiveness in this forum.

Post Reply