Working with XML files with more than 10 million entries

This Q&A forum allows users to post and respond to "How Do I Do ....." questions. Please do not use to report (suspected) errors - you must use your regional help desk for this. The information contained in this forum has not been validated by LANSA and, as such, LANSA cannot guarantee the accuracy of the information.
Post Reply
Joerg Hamacher
Posts: 120
Joined: Thu Feb 11, 2016 12:01 am

Working with XML files with more than 10 million entries

Post by Joerg Hamacher »

HI again,

I need to write a functionality that
  • reads a XML file in an IFS folder
    detects possible erros in the single lines
    corrects these errors and
    writes back the corrected XML file into the original IFS folder.
I plan to realize this with TRANSFORM_FILE, read the list data, correct them, write the correct data into a second list and at last create anew XML using TRANSFORM_LIST.

My question is:
The XML file can contain up to 30 million lines.
Documentation says: The special value ENTRIES(*MAX) is provided to represent 2,147,483,647. Using this value on some operating systems, like 32-bit Microsoft Windows versions, will overflow available memory in a process if an attempt is made to add the maximum number of entries.

Is this also valid when using a Reusable Part (RDMLX) on IBMi?
Or is there a better / other way to handle such big files?

Best regards,
Jörg
User avatar
Dino
Posts: 472
Joined: Fri Jul 19, 2019 7:49 am
Location: Robbinsville, NC
Contact:

Re: Working with XML files with more than 10 million entries

Post by Dino »

Hi Joerg

Not sure about the 30 million records but using the filestream, streamreader, streamwriter is faster.

viewtopic.php?f=3&t=2902&p=8888&hilit=Prim_file#p8888

Kind regards
Joerg Hamacher
Posts: 120
Joined: Thu Feb 11, 2016 12:01 am

Re: Working with XML files with more than 10 million entries

Post by Joerg Hamacher »

Hi Dino,

thank you, I genrally got it running with StreamWriter.

Now I have another problem:

original file contains German umlauts like ä,ö,ü
when I read the lines and write it into my new file these signs are gone

Original xml - read by StreamReader
Define_Com Class(#PRIM_IOC.FileStream) Name(#InFileStream)
Define_Com Class(#PRIM_IOC.StreamReader) Name(#StreamReader) Stream(#InFileStream)
<LBez2>geraute Schlagfläche magn. Nagelhalter</LBez2>
<LBez1_ph>Picard Latthammer</LBez1_ph>
<LBez2_ph>geraut,Vergütungsstahl,magn. Nagelhalter</LBez2_ph>


New xml -written using StreamWriter
Define_Com Class(#PRIM_IOC.FileStream) Name(#FileStream)
Define_Com Class(#PRIM_IOC.StreamWriter) Name(#StreamWriter) Stream(#FileStream)
<LBez2>geraute Schlagflche magn. Nagelhalter</LBez2>
<LBez1_ph>Picard Latthammer</LBez1_ph>
<LBez2_ph>geraut,Vergtungsstahl,magn. Nagelhalter</LBez2_p


Can this be solved?

Best regards,
Joerg
User avatar
Dino
Posts: 472
Joined: Fri Jul 19, 2019 7:49 am
Location: Robbinsville, NC
Contact:

Re: Working with XML files with more than 10 million entries

Post by Dino »

Check the CCSID and/or change it using the chgatr command
Joerg Hamacher
Posts: 120
Joined: Thu Feb 11, 2016 12:01 am

Re: Working with XML files with more than 10 million entries

Post by Joerg Hamacher »

At the moment I am still testing it on my Windows 10 PC. Can anything be done using utf8 or something like this?
Post Reply