Groupby - collection processing

Iterator and Iterable have most of the most useful methods when dealing with collections. Fold, Map, Filter are probably the most common. But other very useful methods include grouped/groupBy, sliding, find, forall, foreach, and many more. I want to cover Iterable's groupBy method in this topic.

This is a Scala 2.8 and later method. It is similar to partition in that it allows the collection to be divided (or partitioned). Partition takes a method with returns a boolean and partitions the collection into two depending on a result. GroupBy takes a function that returns an object and returns a Map with the key being the return value. This allows an arbitrary number of partitions to be made from the collection.

Here is the method signature:

1. def groupBy[K](f : (A) => K) : Map[K, This]

A bit of context is require to understand the three Type parameters A, K and This. This method is defined in a super class of collections called TraversableLike (I will briefly discuss this in the next topic.) TraversableLike takes two type parameters: the type of the collection and the type contained in the collection. Therefore in this method definition, 'This' refers to the collection type (List for example) and A refers to contained type (perhaps Int). Finally K refers to the type returned by the function and are the keys of the groups formed by the method.

Examples:

1. scala> val groups = (1 to 20).toList groupBy {
2.      | case i if(i<5) => "g1"
3.      | case i if(i<10) => "g2"
4.      | case i if(i<15) => "g3"
5.      | case _ => "g4"
6.      | }
7. res4: scala.collection.Map[java.lang.String,List[Int]] = Map(g1 -> List(1, 2, 3, 4), g2 -> List(5, 6, 7, 8, 9), g3 -> List(10, 11, 12, 13, 14), g4 -> List(15, 16, 17, 18, 19, 20))
9. scala> groups.keySet
10. res6: scala.collection.Set[java.lang.String] = Set(g1, g2, g3, g4)
12. scala> groups("g1")
13. res7: List[Int] = List(1, 2, 3, 4)
15. scala> val mods = (1 to 20).toList groupBy ( _ % 4 )
16. mods: scala.collection.Map[Int,List[Int]] = Map(1 -> List(1, 5, 9, 13, 17), 2 -> List(2, 6, 10, 14, 18), 3 -> List(3, 7,
17.  11, 15, 19), 0 -> List(4, 8, 12, 16, 20))
19. scala> mods.keySet
20. res9: scala.collection.Set[Int] = Set(1, 2, 3, 0)
22. scala> mods(1)
23. res11: List[Int] = List(1, 5, 9, 13, 17)

Boolean Extractors

As discussed in other topics there are several ways to create custom extractors for objects. See

• Extractors Introduction
• Extract Sequences
• Operator Style Extractors

There is one more custom extractor that can be defined. Simple boolean extractors. They are used as follows:

1. scala> "hello world" match {                   
2.      | case HasVowels() => println("Vowel found")
3.      | case _ => println("No Vowels")            
4.      | }
5. Vowel found

A boolean extractor is an object that returns Boolean from the unapply method rather than Option[_].

Examples:

1. scala> object HasVowels{ def unapply(in:String):Boolean = in.exists( "aeiou" contains _ ) }
2. defined module HasVowels
4. // Note that HasVowels() has ().
5. // This is critical because otherwise the match checks whether
6. // the input is the HasVowels object.
7. // The () forces the unapply method to be used for matching
8. scala> "hello world" match {
9.      | case HasVowels() => println("Vowel found")
10.      | case _ => println("No Vowels")
11.      | }
12. Vowel found
14. // Don't forget the ()
15. scala> "kkkkkk" match {    
16.      | case HasVowels() => println("Vowel found")
17.      | case _ => println("No Vowels")
18.      | }
19. No Vowels
21. scala> class HasChar(c:Char) {           
22.      | def unapply(in:String) = in.contains(c)
23.      | }
24. defined class HasChar
26. scala> val HasC = new HasChar('c')
27. HasC: HasChar = HasChar@3f2f529b
29. // Don't forget the () it is required here as well
30. scala> "It actually works!" match {
31.      | case HasC() => println("a c was found")
32.      | case _ => println("no c found")  
33.      | }
34. a c was found
36. // Don't forget the ()
37. scala> "hello world" match { 
38.      | case HasC() => println("a c was found")
39.      | case _ => println("no c found")       
40.      | }
41. no c found

Extractors 3 (Operator Style Matching)

One of the blessings and curses of Scala the several rules for creating expressive code (curse because it can be used for evil.) One such rule is related to extractors that allows the following style of match pattern:

1. List(1,2,3) match {
2.   case 1 :: _ => println("found a list that starts with a 1")
3.   case _ => println("boo")
4. }

The rule is very simple. An extractor object that returns Option[Tuple2[_,_]] (or equivalently Option[(_,_)]) can be expressed in this form.

In other words: object X {def unapply(in:String):Option[(String,String)] = ...} can be used in a case statement like: case first X head => ... or case "a" X head => ....

Example to extract out the vowels from a string:

1. scala> object X { def unapply(in:String):Option[(RichString,RichString)] = Some(in.partition( "aeiou" contains _ )) }
2. defined module X
4. scala> "hello world" match { case head X tail => println(head, tail) }
5. (eoo,hll wrld)
7. // This is equivalent but a different way of expressing it
8. scala> "hello world" match { case X(head, tail) => println(head, tail) }       
9. (eoo,hll wrld)

Example for Matching the last element in a list. Thanks to 3-things-you-didnt-know-scala-pattern.html:

1. scala> object ::> {def unapply[A] (l: List[A]) = Some( (l.init, l.last) )}
2. defined module $colon$colon$greater
4. scala> List(1, 2, 3) match {
5.      | case _ ::> last => println(last)
6.      | }
7. 3
9. scala> (1 to 9).toList match {
10.      | case List(1, 2, 3, 4, 5, 6, 7, 8) ::> 9 => "woah!"
11.      | }
12. res12: java.lang.String = woah!
14. scala> (1 to 9).toList match {
15.      | case List(1, 2, 3, 4, 5, 6, 7) ::> 8 ::> 9 => "w00t!"
16.      | }
17. res13: java.lang.String = w00t!

Vacation

For the next three weeks (Oct 3,2009 to Oct 25) I am going on vacation which means my Internet access will not be regular so I will post when I can but I cannot guarantee any real schedule. So to keep you interested I recommend looking at this very nice overview of Scala's features:

Another Scala Tour

I really liked the presentation on this site.

Jesse

Strings

This topic simply shows several things that can be done with strings. It is not exhaustive and focusses of things that cannot as easily be done with java strings.

Note: Because I am using Scala 2.7 we often need to use mkString to convert the processed string from a sequence of characters to a string. In scala 2.8 this is not required.

1. /*
2.    Making use of raw strings to create a multi line string
3.    I add a | at the beginning of each line so that we can line up the quote nicely 
4.    in source code then later strip it from the string using stripMargin
5. */
6. scala> val quote = """|I don-t consider myself a pessimist.                                                                                                
7.      |                |I think of a pessimist as someone who is waiting for it to rain.
8.      |                |And I feel soaked to the skin.
9.      | 
10.      |                |Leonard Cohen"""
11. quote: java.lang.String = 
12. |I don-t consider myself a pessimist. 
13.                       |I think of a pessimist as someone who is waiting for it to rain.
14.                       |And I feel soaked to the skin.
15.        
16.                       |Leonard Cohen
19. // capilize the first character of each line
20. scala> val capitalized = quote.lines.
21.      |                         map( _.trim.capitalize).mkString("\n")
22. capitalized: String = 
23. |I don-t consider myself a pessimist.
24. |I think of a pessimist as someone who is waiting for it to rain.
25. |And I feel soaked to the skin.
27. |Leonard Cohen
29. // remove the margin of each line
30. scala> quote.stripMargin        
31. res1: String = 
32. I don-t consider myself a pessimist. 
33. I think of a pessimist as someone who is waiting for it to rain.
34. And I feel soaked to the skin.
35.        
36. Leonard Cohen
38. // this is silly.  I reverse the order of each word but keep the words in order
39. scala> quote.stripMargin.         
40.      |       lines.               
41.      |       map( _.split(" ").   
42.      |              map(_.reverse).
43.      |              mkString (" ")).
44.      |      mkString("\n")
45. res16: String = 
46. I t-nod redisnoc flesym a .tsimissep
47. I kniht fo a tsimissep sa enoemos ohw si gnitiaw rof ti ot .niar
48. dnA I leef dekaos ot eht .niks
50. dranoeL nehoC
52. scala> val myPatch = "-->This is my patch<--"                
53. myPatch: java.lang.String = -->This is my patch<--
55. // I replace the characters from point 10 in the string to myPatch.length 
56. // (the full patch string)
57. scala> quote.patch(10, myPatch, myPatch.length).mkString     
58. res21: String = 
59. |I don-t c-->This is my patch<--mist.
60.                       |I think of a pessimist as someone who is waiting for it to rain.
61.                       |And I feel soaked to the skin.
62.        
63.                       |Leonard Cohen
65. // drop the first 3 lines of the string.  
66. // there is also a take method
67. scala> quote.lines.drop(3).mkString("\n").stripMargin 
68. res25: String = 
69.        
70. Leonard Cohen
72. // a bunch of examples of converting strings
73. scala> "1024".toInt
74. res26: Int = 1024
76. scala> "1024".toFloat
77. res27: Float = 1024.0
79. scala> "1024".toDouble
80. res28: Double = 1024.0
82. scala> "1024".toLong  
83. res29: Long = 1024
85. scala> "102".toByte 
86. res31: Byte = 102
88. scala> "true".toBoolean
89. res32: Boolean = true
91. // format uses the java.util.Formatter class to format a string
92. scala> "Hello %s,\nThe date is %2$tm %2$te,%2$tY".format("Jesse", new java.util.Date()) 
93. res35: String = 
94. Hello Jesse,
95. The date is 09 30,2009
97. /*
98.    More silliness
99.    I am replacing every other character with the character of the reversed string
100.   
101.    this is done by 
102.    1. convert string to a list and zip it together with its reverse
103.       We may still need to cover zipping.  It basically matches up the 
104.       corresponding elements of two lists into one list
105.       so 1,2,3 and one,two,three zipped would be (1,one),(2,two),(3,three)
106.    2. Add an index to each element in the list with zipWithIndex
107.    3. Use map to check if the element is an odd element using the index and return either the original element or the reversed element
108.   
109.    Not useful but interesting use of functional idioms
110. */
111. scala> quote.toList.                                          
112.      |       zip(quote.reverse.toList).                       
113.      |       zipWithIndex.                                    
114.      |       map {                                            
115.      |            case ((original,reversed),index) if(index % 2 == 0) => original
116.      |            case ((original,reversed),index) => reversed                   
117.      |           }.                                                              
118.      |       mkString                                                            
119. res42: String = |e oo -r noes|d r m s l     e s m s .        . i s e t o   e|a shlne  f anp|s i i t a   o e n   h  .siwrioi gifrrfig ioirwis.  h   n e o   a t i i s|pna f  enlhs a|e   o t e s i .        . s m s e     l s m r d|seon r- oo e|
121. // filter out all non-vowels
122. scala> quote.filter( "aeiou" contains _ ).mkString
123. res51: String = ooieeaeiiioaeiiaoeoeoiaiioioaieeoaeoeieoaoe

Multiple type bounds on a generic parameter

Suppose you want to declare a method to take objects that implement two interfaces or parent objects. For example suppose the parameter must implement both Iterable and a function so that you can access the elements of the iterable via object(index). How can you do that in scala?

This topic is derived from How do I setup multiple type bounds in Scala?

Answer:

1. scala> def process[R <: Function1[Int,String] with Iterable[String]] (resource:R) = {
2.      | println("using function:"+resource(0))
3.      | println("using iterable:"+resource.elements.next)
4.      | }
5. process: [R <: (Int) => String with Iterable[String]](R)Unit
7. // Array is a Function1 and iterable so this works
8. scala> process (Array("1","2","3"))                                                  
9. using function:1
10. using iterable:1