September 2017 Entries

I have a huge XML document (over 2GB) exported from an automotive company system which I wanted to be able to export data from it to a CSV file to import into a legacy reporting system. Yes I know this is a bit back to front but a job is a job.

My first thought was to write some C# code to extract the data, but this quickly became difficult as the XML document’s schema was non-trivial due to its use of substitution groups. I may have been able to use XSLT, but this is not a strong point for me, and I think I would have the same complexities with the XSD.

So instead I looked at using the Liquid Data Mapper which is part of Liquid Studio 2017. There is a video explaining how to go from CSV to XML, so I just reversed the Source and Target and this seemed to work OK.

In fact it was quite easy as a lot of the field names matched so the automatic ‘Connect Child Nodes’ connection matching function connected many of these for me.

However, when I clicked the run button I received an ‘Out of Memory’ error. Not so good.

I contacted the Liquid Technologies support team and they responded very quickly suggesting that I try running the generated transform .exe file from the command line rather than from inside Liquid Studio as shown in the video ‘Generate an Executable (.exe) Data Mapping Transform’.

The size of file that can be processed is ultimately dependant on the memory of the PC, but they say they have tested with files > 5GB using a 32GB PC and it works OK.

Anyway, I followed the instructions on the video and it worked straight away. This is very impressive and I would highly recommend using this approach for mapping data.