XML transformations 2

This topic is a continuation of XML Transformations. The previous topic showed a method of creating transformation rules then combining the rules to create a transformation that could be applied to an XML datastructure. This topic takes a different approach of
using a match statement an a recursive method to iterate through the tree.

1. scala> val xml = <library>
2.      | <videos>
3.      | <video type="dvd">Seven</video>
4.      | <video type="blue-ray">The fifth element</video>
5.      | <video type="hardcover">Gardens of the moon</video>
6.      | </videos>
7.      | <books>
8.      | <book type="softcover">Memories of Ice</book>
9.      | </books>
10.      | </library>
11. xml: scala.xml.Elem = 
12. <library>
13.        <videos>
14.        <video type="dvd">Seven</video>
15.        <video type="blue-ray">The fifth element</video>
16.        <video type="hardcover">Gardens of the moon</video>
17.        </videos>
18.        <books>
19.        <book type="softcover">Memories of Ice</book>
20.        </books>
21.        </library>
23. scala> import scala.xml._
24. import scala.xml._
26. scala> def moveElements (node:Node) : Node = node match {
27.      | case n:Elem if (n.label == "videos") => 
28.      |   n.copy( child = n.child diff mislabelledBooks)
29.      | case n:Elem if (n.label == "books") =>
30.      |   val newBooks = mislabelledBooks map { e => e.asInstanceOf[Elem].copy(label="book") }
31.      |   n.copy( child = n.child ++ newBooks)
32.      | case n:Elem => 
33.      |   val children = n.child map {moveElements _}
34.      |   n.copy(child = children)
35.      | case n => n
36.      | }
37. moveElements: (node: scala.xml.Node)scala.xml.Node
39. scala> moveElements(xml)
40. res1: scala.xml.Node = 
41. <library>
42.               <videos>
43.               <video type="dvd">Seven</video>
44.               <video type="blue-ray">The fifth element</video>
45.               
46.               </videos>
47.               <books>
48.               <book type="softcover">Memories of Ice</book>
49.               <book type="hardcover">Gardens of the moon</book></books>
50.               </library>

Xml Transformation 1

Unlike most Java Xml Apis the Scala Object model consists of immutable object. This has two major consequences:

• There is no reference to the parent node because that would cause the XML to be very expensive during transformations
• Transforming the XML requires creating new nodes rather than changing the existing nodes

Both point cause non-functional programmers to feel a little uneasy but in practice only the first restriction causes any real discomfort.

Two methods for XML transformation will be demonstrated in this and the next topic.

1. scala> val xml = <library>
2.      | <videos>
3.      | <video type="dvd">Seven</video>
4.      | <video type="blue-ray">The fifth element</video>
5.      | <video type="hardcover">Gardens of the moon</video>
6.      | </videos>
7.      | <books>
8.      | <book type="softcover">Memories of Ice</book>
9.      | </books>
10.      | </library>
11. xml: scala.xml.Elem = 
12. <library>
13.        <videos>
14.        <video type="dvd">Seven</video>
15.        <video type="blue-ray">The fifth element</video>
16.        <video type="hardcover">Gardens of the moon</video>
17.        </videos>
18.        <books>
19.        <book type="softcover">Memories of Ice</book>
20.        </books>
21.        </library>
23. scala> import scala.xml._
24. import scala.xml._
26. scala> import scala.xml.transform._
27. import scala.xml.transform._
29. // Some of the books are labelled as videos
30. // not books so lets select those elements
31. scala> val mislabelledBooks = xml \\ "video" filter {e => (e \\ "@type").text == "hardcover"}
32. mislabelledBooks: scala.xml.NodeSeq = <video type="hardcover">Gardens of the moon</video>
34. // we can create a rule that will remove all the
35. // selected elements
36. scala> object RemoveMislabelledBooks extends RewriteRule {
37.      | override def transform(n: Node): Seq[Node] ={ 
38.      | if (mislabelledBooks contains n) Array[Node]()
39.      | else n
40.      | }
41.      | }
42. defined module RemoveMislabelledBooks
44. // a quick test to make sure the elements are removed
45. scala> new RuleTransformer(RemoveMislabelledBooks)(xml)
46. res1: scala.xml.Node = 
47. <library>
48.        <videos>
49.        <video type="dvd">Seven</video>
50.        <video type="blue-ray">The fifth element</video>
51.        
52.        </videos>
53.        <books>
54.        <book type="softcover">Memories of Ice</book>
55.        </books>
56.        </library>
59. // Now another rule to add them back
60. scala> object AddToBooks extends RewriteRule {                             
61.      | override def transform(n: Node): Seq[Node] = n match {                                
62.      | case e:Elem if(e.label == "books") =>                                                 
63.      |   val newBooks = mislabelledBooks map { case e:Elem => e.copy(label="book") }
64.      |   e.copy(child = e.child ++ newBooks)                                                 
65.      | case _ => n
66.      | }
67.      | }
68. defined module AddToBooks
70. // voila done
71. scala> new RuleTransformer(RemoveMislabelledBooks, AddToBooks)(xml) 
72. res4: scala.xml.Node = 
73. <library>
74.        <videos>
75.        <video type="dvd">Seven</video>
76.        <video type="blue-ray">The fifth element</video>
78.        </videos>
79.        <books>
80.        <book type="softcover">Memories of Ice</book>
81.        <book type="hardcover">Gardens of the moon</book></books>
82.        </library>

XML matching

The Scala XML support includes the ability to match on XML elements. Here are several examples. One of the most important parts to remember is to not miss the '{' and '}'.

1. scala> <document>
2.      |  <child1/>
3.      |  <child2/>
4.      | </document>
5. res0: scala.xml.Elem =
6. <document>
7.         <child1></child1>
8.         <child2></child2>
9.        </document>
11. scala> res0 match {
12.        // match the document tag
13.        // the {_*} is critical
14.      | case <document>{_*}</document> => println("found document element")
15.      | case _ => println("Found another element")
16.      | }
17. found document element
19. scala> res0 match {
20.        // assign the document element to e
21.      | case e @ <document>{_*}</document> => println(e)
22.      | case _ => println("Found another element")
23.      | }
24. <document>
25.         <child1></child1>
26.         <child2></child2>
27.        </document>
29. scala> res0 match {
30.        // assign the children of document to children
31.        // notice that there are Text elements that are part of children
32.      | case <document>{children @ _*}</document> => println(children)
33.      | case _ => println("Found another element")
34.      | }
35. ArrayBuffer(
36.         , <child1></child1>,
37.         , <child2></child2>,
38.        )
40. // the '\' is xpath like but only returns elements and attributes
41. // in this case the \ "_" returns all element children of res0.  It
42. // will not return the Text elements.
43. scala> res0 \ "_" foreach {
44.      | case <child1>{_*}</child1> => println("child1 found")
45.      | case <child2>{_*}</child2> => println("child2 found")
46.      | case e => println("found another element")
47.      | }
48. child1 found
49. child2 found
51. // another example of how \ does not return any text elements.  This returns
52. // no elements
53. scala> <doc>Hello</doc> \ "_" foreach { case scala.xml.Text(t) => println("a text element found: "+t) }
55. // the .child returns all children of an Elem
56. scala> <doc>Hello</doc>.child foreach { case scala.xml.Text(t) => println("a text element found: "+t) }
57. a text element found: Hello
59. // This example throws a match error because there are whitespace text elements
60. // that cause the match to fail.
61. scala> res0 match {                                                                                    
62.      | case <document><child1/><child2/></document> => println("found the fragment")
63.      | }
64. scala.MatchError: <document>
65.         <child1></child1>
66.         <child2></child2>
67.        </document>
68.        at .< init>(< console>:6)
69.        at .< clinit>(< console>)
70.        at RequestResult$.< init>(< console>:3)
71.        at RequestResult$.< clinit>(< console>)
72.        at RequestResult$result(< console>)
73.        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
74.        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMetho...
76. // The trim method removes the whitespace nodes so now the 
77. // match will work
78. scala> scala.xml.Utility.trim(res0) match {                                                            
79.      | case <document><child1/><child2/></document> => println("found the fragment")
80.      | }
81. found the fragment
83. // you can select part of the tree using matching
84. // child2 is assigned to 3 in this example.
85. scala> scala.xml.Utility.trim(res0) match {                                           
86.      | case <document><child1/>{e @ _*}</document> => println("found the fragment:"+e)
87.      | }
88. found the fragment:RandomAccessSeq(<child2></child2>)

XPath Style XML Selection

The xml API in scala allows xpath like (although not true xpath) queries. In combination with matching this makes it very easy to process XML documents. I am only going to discuss xpath style selection now. The code section is very long but primarily because the results are often quite lengthy.

1. scala> val address = <address>
2.      | <CI_Address>
3.      | <deliveryPoint>
4.      | <CharacterString>Viale delle Terme di Caracalla
5.      | </CharacterString>
6.      | </deliveryPoint>
7.      | <city>
8.      | <CharacterString>Rome</CharacterString>
9.      | </city>
10.      | <administrativeArea>
11.      | <CharacterString />
12.      | </administrativeArea>
13.      | <postalCode>
14.      | <CharacterString>00153</CharacterString>
15.      | </postalCode>
16.      | <country>
17.      | <CharacterString>Italy</CharacterString>
18.      | </country>
19.      | <electronicMailAddress>
20.      | <CharacterString>jippe.hoogeveen@fao.org
21.      | </CharacterString>
22.      | </electronicMailAddress>
23.      | </CI_Address>
24.      | </address>
25. address: scala.xml.Elem =
26. <address>
27.        <CI_Address>
28.       ...
30. // create a pretty printer for writing out the document nicely
31. scala>  val pp = new scala.xml.PrettyPrinter(80, 5);
32. pp: scala.xml.PrettyPrinter = scala.xml.PrettyPrinter@6d87c12a
34. // select the city
35. scala> println( pp.formatNodes( address \ "CI_Address" \ "city" ) )                   
36. <city>
37.      <gco:CharacterString>Rome</gco:CharacterString>
38. </city>
40. // a second way to select city
41. scala> println( pp.formatNodes( address \\ "city" ) )      
42. <city>
43.      <gco:CharacterString>Rome</gco:CharacterString>
44. </city>
46. // select all characterStrings and print then one per line (unless there is a \n in the text)
47. scala> (address \\ "CharacterString").mkString( "\n" )
48. res2: String =
49. <CharacterString>Viale delle Terme di Caracalla
50.        </CharacterString>
51. <CharacterString>Rome</CharacterString>
52. <CharacterString></CharacterString>
53. <CharacterString>00153</CharacterString>
54. <CharacterString>Italy</CharacterString>
55. <CharacterString>jippe.hoogeveen@fao.org
56.        </CharacterString>
58. // iterate over the city node and all of its child nodes.
59. scala> println( pp.formatNodes( address \\ "city" \\ "_"))
60. <city>
61.      <CharacterString>Rome</CharacterString>
62. </city><CharacterString>Rome</CharacterString>
64. // similar as above but iterate over all CI_Address nodes and each of its children
65. scala>println( pp.formatNodes( address \\ "CI_Address" \\ "_")) 
66. <CI_Address>
67.      <deliveryPoint>
68.           <CharacterString>Viale delle Terme di Caracalla </CharacterString>
69.      </deliveryPoint>
70.      <city>
71.           <CharacterString>Rome</CharacterString>
72.      </city>
73.      <administrativeArea>
74.           <CharacterString></CharacterString>
75.      </administrativeArea>
76.      <postalCode>
77.           <CharacterString>00153</CharacterString>
78.      </postalCode>
79.      <country>
80.           <CharacterString>Italy</CharacterString>
81.      </country>
82.      <electronicMailAddress>
83.           <CharacterString>jippe.hoogeveen@fao.org </CharacterString>
84.      </electronicMailAddress>
85. </CI_Address><deliveryPoint>
86.      <CharacterString>Viale delle Terme di Caracalla </CharacterString>
87. </deliveryPoint><CharacterString>Viale delle Terme di Caracalla </CharacterString><city>
88.      <CharacterString>Rome</CharacterString>
89. </city><CharacterString>Rome</CharacterString><administrativeArea>
90.      <CharacterString></CharacterString>
91. </administrativeArea><CharacterString></CharacterString><postalCode>
92.      <CharacterString>00153</CharacterString>
93. </postalCode><CharacterString>00153</CharacterString><country>
94.      <CharacterString>Italy</CharacterString>
95. </country><CharacterString>Italy</CharacterString><electronicMailAddress>
96.      <CharacterString>jippe.hoogeveen@fao.org </CharacterString>
97. </electronicMailAddress><CharacterString>jippe.hoogeveen@fao.org </CharacterString>
100. // print all text
101. scala> address.text                      
102. res4: String =
104.       
105.       
106.        Viale delle Terme di Caracalla
107.       
108.       
109.       
110.        Rome
111.       
112.       
113.       
114.       
115.       
116.        00153
117.       
118.       
119.        Italy
120.       
121.       
122.        jippe.hoogeveen@fao.org
123.       
124. // print all character string text
125. scala> (address \\ "CharacterString").text            
126. res3: String =
127. Viale delle Terme di Caracalla
128.        Rome00153Italyjippe.hoogeveen@fao.org
129.       
130. // print all character string text one per line
131. scala> (address \\ "CharacterString").map( _.text ).mkString("\n")
132. res6: String =
133. Viale delle Terme di Caracalla
134.       
135. Rome
137. 00153
138. Italy
139. jippe.hoogeveen@fao.org
141. // find the longest character string
142. scala> (address \\ "CharacterString").reduceRight(  
143.      | (elem, longest) => {
144.      | if( elem.text.length > longest.text.length ) elem
145.      | else longest
146.      | })
147. res8: scala.xml.Node =
148. <CharacterString>Viale delle Terme di Caracalla
149.        </CharacterString>
152. // find the alphabetically first characterstring
153. scala> (address \\ "CharacterString").reduceRight( (elem, longest) => {
154.      | if( elem.text > longest.text ) elem
155.      | else longest
156.      | })
157. res9: scala.xml.Node =
158. <CharacterString>jippe.hoogeveen@fao.org
159.        </CharacterString>

Creating XML

Scala allows you to embed XML directly into a program and provides several ways to manipulate it. Today we will look at writing XML.

1. scala> val xml = <root>
2.      | <child>text</child>
3.      | </root>
4. xml: scala.xml.Elem =
5. <root>
6.        <child>text</child>
7.        </root>
9. scala> val data = "a string"
10. data: java.lang.String = a string
12. // you can embed logic and variables in the xml by surrounding with {}
13. scala> val xml = <root>{data}</root>
14. xml: scala.xml.Elem = <root>a string</root>
16. scala> val xml = <root>{ for( i <- 1 to 3 ) yield {<xml i={i.toString}/>} } </root>
17. xml: scala.xml.Elem = <root><xml i="1"></xml><xml i="2"></xml><xml i="3"></xml> </root>
19. scala> val xml = <root>{ for( i <- 1 to 3 ) yield <child>{i}</child> } </root>
20. xml: scala.xml.Elem = <root><child>1</child><child>2</child><child>3</child> </root>
22. // save xml to file.  Note Scala 2.8 is changin save API
23. // and will require the encoding and DocType information
24. scala> scala.xml.XML.save( "/tmp/doc.xml", xml)