Overcoming Type Erasure in Matching 2 (Variance)

A commenter (Brian Howard) on the post Overcoming Type Erasure in Matching 1 made a very good point:

Is there a way to deal with some type arguments being contravariant? Try the following:

class A

class B extends A

val AAFunction = new Def[Function1[A,A]]

((a:A) => a) match {case AAFunction(f) => Some(f(new A)); case _ => None} // this is OK

((a:A) => new B) match {case AAFunction(f) => Some(f(new A)); case _ => None} // this is OK

((b:B) => b) match {case AAFunction(f) => Some(f(new A)); case _ => None} // gives a ClassCastException, since new A is not a B

There is a way to do this, however the information is not captured in the Manifest. A manifest is not designed to do full reflection it is by design very light and has little impact on performance. So to provide the functionality requested by Brian one needs to add that information to the Extractor Definition. Have have a possible solution below.

1. scala> class A
2. defined class A
4. scala> class B extends A
5. defined class B
7. scala> object Variance extends Enumeration {
8.      |     val Co, Contra, No = Value
9.      | }
10. defined module Variance
12. scala> import Variance._
13. import Variance._
15. scala> class Def[C](variance:Variance.Value*)(implicit desired : Manifest[C]) {
16.      |     def unapply[X](c : X)(implicit m : Manifest[X]) : Option[C] = {
17.      |         val typeArgsTriplet = desired.typeArguments.zip(m.typeArguments).
18.      |                                                     zipWithIndex
19.      |         def sameArgs = typeArgsTriplet forall {
20.      |             case ((desired,actual),index) if(getVariance(index) == Contra) => 
21.      |                 desired <:< actual 
22.      |             case ((desired,actual),index) if(getVariance(index) == No) => 
23.      |                 desired == actual 
24.      |             case ((desired,actual),index) => 
25.      |                 desired >:> actual
26.      |         }
27.      |         
28.      |         val isAssignable = desired.erasure.isAssignableFrom(m.erasure) || (desired >:> m)
29.      |         if (isAssignable && sameArgs) Some(c.asInstanceOf[C])
30.      |         else None
31.      |     }
32.      |     def getVariance(i:Int) = if(variance.length > i) variance(i) else No
33.      | }
34. defined class Def
36. scala> val AAFunction = new Def[Function1[A,A]]
37. AAFunction: Def[(A) => A] = Def@64a65760
39. scala> ((a:A) => a) match {case AAFunction(f) => Some(f(new A)); case _ => None}
40. res0: Option[A] = Some(A@1bd4f279)
42. scala> ((a:A) => new B) match {case AAFunction(f) => Some(f(new A)); case _ => None}
43. res1: Option[A] = None
45. scala> ((b:B) => b) match {case AAFunction(f) => Some(f(new A)); case _ => None}
46. res2: Option[A] = None
48. scala> val BAFunction = new Def[Function1[B,A]](Contra,Co)
49. BAFunction: Def[(B) => A] = Def@26114629
51. scala> ((b:A) => b) match {case BAFunction(f) => Some(f(new B)); case _ => None}
52. res3: Option[A] = Some(B@15dcc3ca)

Overcoming Type Erasure in matching 1

Since Scala runs on the JVM (it also runs on .NET but we this tip is aimed at Scala on the JVM) much of the type information that is available at compile-time is lost during runtime. This means certain types of matching are not possible. For example it would be nice to be able to do the following:

1. scala> val x : List[Any] = List(1.0,2.0,3.0)
2. x: List[Any] = List(1, 2, 3)
4. scala> x match { case l : List[Boolean] => l(0) }

If you run this code the list matches the list of ints which is incorrect (I have ran the following with -unchecked so it will print the warning about erasure):

1. scala> val x : List[Any] = List(1.0,2.0,3.0)
2. x: List[Any] = List(1, 2, 3)
4. scala> x match { case l : List[Boolean] => l(0) }         
6. < console>:6: warning: non variable type-argument Boolean in type pattern List[Boolean] is unchecked since it is eliminated by erasure
7.        x match { case l : List[Boolean] => l(0) }
8.                           ^
9. java.lang.ClassCastException: java.lang.Double cannot be cast to java.lang.Boolean
10. at scala.runtime.BoxesRunTime.unboxToBoolean(Unknown Source)
11. at .< init>(< console>:6)

Another example is trying to match on structural types. Wouldn't it be nice to be able to do the following (We will solve this in a future post):

1. scala> "a string" match {
2.      | case l : {def length : Int} => l.length
3.      | }
4. < console>:6: warning: refinement AnyRef{def length: Int} in type pattern AnyRef{def length: Int} is unchecked since it is eliminated by erasure
5.        case l : {def length : Int} => l.length
6.                 ^
7. res5: Int = 8

So lets see what we can do about this. My proposed solution is to create an class that can be used as an extractor when instantiated to do the check.

This is a fairly advanced tip so make sure you have read up on Matchers and Manifests:

• Extractor 1
• Manifests
• Comparing Manifests

The key parts of the next examples are the Def class and the function 'func'. 'func' is defined in the comments in the code block.

The Def class is the definition of what we want to match. Once we have the definition we can use it as an Extractor to match List[Int] instances.

The Def solution is quite generic so it will satisfy may cases where you might want to do matching but erasure is getting in the way.

The secret is to use manifests:

• When the class is created the manifest for the class we want is captured.
• Each time a match is attempted the manifest of the class being matched is captured
• In the unapply method, the two manifests are compared and when they match we can return the matched element but cast to the desired type for the compiler

It is critical to notice the use of the typeArguments of the manifest. This returns a list of the manifests of each typeArgument. You cannot simply compare desired == m because manifest comparisons are not deep. There is a weakness in this code in that it only handles generics that are 1 level deep. For example:

List[Int] can be matched with the following code but List[List[Int]] will match anything that is List[List[_]]. Making the method more generic is an exercise left to the reader.

1. scala> import reflect._ 
2. import reflect._
4. /*
5. This is the key class
6. */
7. scala> class Def[C](implicit desired : Manifest[C]) {
8.      | def unapply[X](c : X)(implicit m : Manifest[X]) : Option[C] = {
9.      |   def sameArgs = desired.typeArguments.zip(m.typeArguments).forall {case (desired,actual) => desired >:> actual}
10.      |   if (desired >:> m && sameArgs) Some(c.asInstanceOf[C])
11.      |   else None
12.      | }
13.      | }
14. defined class Def
16. // first define what we want to match
17. scala> val IntList = new Def[List[Int]]
18. IntList: Def[List[Int]] = Def@6997f7f4
20. /*
21. Now use the object as an extractor.  the variable l will be a typesafe List[Int]
22. */
23. scala> List(1,2,3) match { case IntList(l) => l(1) : Int ; case _ => -1 }
24. res36: Int = 2
26. scala> List(1.0,2,3) match { case IntList(l) => l(1) : Int ; case _ => -1 }
27. res37: Int = -1
29. // 32 is not a list so it will fail to match
30. scala> 32 match { case IntList(l) => l(1) : Int ; case _ => -1 }
31. res2: Int = -1
33. scala> Map(1 -> 3) match { case IntList(l) => l(1) : Int ; case _ => -1 }
34. res3: Int = -1
36. /*
37. The Def class can be used with any Generic type it is not restricted to collections
38. */
39. scala> val IntIntFunction = new Def[Function1[Int,Int]]
40. IntIntFunction: Def[(Int) => Int] = Def@5675e2b4
42. // this will match because it is a function
43. scala> ((i:Int) => 10) match { case IntIntFunction(f) => f(3); case _ => -1}
44. res38: Int = 10
46. // no match because 32 is not a function
47. scala> 32 match { case IntIntFunction(f) => f(3); case _ => -1}
48. res39: Int = -1
50. // Cool Map is a function so it will match
51. scala> Map(3 -> 100) match { case IntIntFunction(f) => f(3); case _ => -1}
52. res6: Int = 100
54. /*
55. Now we see both the power and the limitations of this solution.
57. One might expect that:
58.    def func(a:Any) = {...} 
59. would work.  
61. However, if the function is defined with 'Any' the compiler does not pass in the required information so the manifest that will be created will be a Any manifest object.  Because of this the more convoluted method declaration must be used so the type information is passed in. 
63. I will discuss implicit parameters in some other post
64. */
65. scala> def func[A](a:A)(implicit m:Manifest[A]) = {
66.      |   a match {
67.      |     case IntList(l) => l.head                   
68.      |     case IntIntFunction(f) => f(32)             
69.      |     case i:Int => i                             
70.      |     case _ => -1                                
71.      |   } 
72.      | }
73. func: [A](a: A)(implicit m: Manifest[A])Int
75. scala> func(List(1,2,3))                           
76. res16: Int = 1
78. scala> func('i')                                   
79. res17: Int = -1
81. scala> func(4)
82. res18: Int = 4
84. scala> func((i:Int) => i+2)
85. res19: Int = 34
87. scala> func(Map(32 -> 2))
88. res20: Int = 2

Guard Sugar

This topic is a simple tip for a cleaner syntax for guards. Guards/filters are statements in for-comprehensions and case statements that guard or filter matches. Often you will see them as follows:

1. scala> for (i <- 1 to 10; if (i % 2 == 1)) yield i
2. res0: scala.collection.immutable.IndexedSeq[Int] = IndexedSeq(1, 3, 5, 7, 9)
4. scala> util.Random.nextInt(10) match {
5.      | case i if(i>5) => println("hi")
6.      | case _ => println("low")
7.      | }
8. low

However you have the option to apply a little sugar to the statements cleaning them up a little:

1. scala> for (i <- 1 to 10; if i % 2 == 1) yield i  
2. res2: scala.collection.immutable.IndexedSeq[Int] = IndexedSeq(1, 3, 5, 7, 9)
4. scala> util.Random.nextInt(10) match {          
5.      | case i if i>5 => println("hi")           
6.      | case _ => println("low")                 
7.      | }
8. hi

As you can see the brackets are optional. That is because in both cases the brackets are not required for the parser to determine the start and end of the guard statements. They are added so that the "normal" syntax used in the typical if statements will compile.

1. scala> 10 match { case i if i == 1 || i == 10 => "obviously this is a match"} 
2. res4: java.lang.String = obviously this is a match
4. /*
5. In case statements you can split the if almost any way you want because it is very clearly bound by the if and the => that is required for all case statements with a guard
6. */
7. scala> 10 match { case i if i == 1 ||                                        
8.      | i == 10 => "obviously this is a match"}
9. res5: java.lang.String = obviously this is a match
11. scala> 10 match {        
12.      | case i
13.      | if
14.      | i == 1
15.      | ||
16.      | i == 10 
17.      | => "matched"
18.      | }
19. res6: java.lang.String = matched
21. /*
22. For-comprehensions are not as flexible since it is harder for the compiler to determine where the guard ends. So try to keep it on one line or otherwise make it a function.  That is probably good advice for case statements as well.
23. */
24. scala> for { 
25.      | x <- 1 to 10
26.      | if
27.      | x > 10
28.      | || x == 2 } yield x
29. < console>:5: error: '<-' expected but integer literal found.
30.        || x == 2 } yield x
31.                ^
32. < console>:5: error: illegal start of simple expression
33.        || x == 2 } yield x
34.                  ^
36. scala> for {              
37.      | x <- 1 to 10       
38.      | if x == 1 || x == 10 } yield x
39. res7: scala.collection.immutable.IndexedSeq[Int] = IndexedSeq(1, 10)

Matching with Or

There is a common case where several expressions in a match statement need to have the same result. For example:

1. scala> "word" match {
2.      | case a @ "word" => println(a)
3.      | case a @ "hi" => println(a)
4.      | }
5. word

this could be changed to:

1. scala> "word" match {               
2.      | case a if(a == "word" || a == "hi") => println(a)
3.      | }
4. word

but this is rather verbose. Another option is to use or matching:

1. scala> "word" match { case a @ ("word" | "hi") => println(a)}
2. word

1. /*
2. it is possible to provide alternatives in matching
3. Anything more advanced needs to be handled with a guard
4. See the last examples
5. */
6. scala> 1 match { case _:Int | _:Double => println("Found it")}
7. Found it
9. // v will be the value if v is either an Int or a Double
10. scala> 1.0 match { case v @ ( _:Int | _:Double) => println(v)}
11. 1.0
13. // Having variables as part of the patterns is not permitted
14. scala> 1.0 match { case v:Int | d:Double => println(v)} 
15. < console>:5: error: illegal variable in pattern alternative
16.        1.0 match { case v:Int | d:Double => println(v)}
17.                         ^
18. < console>:5: error: illegal variable in pattern alternative
19.        1.0 match { case v:Int | d:Double => println(v)}
20.                                                     ^
22. /*
23. Variables are not permitted not even when the name is the same.
24. */
25. scala> 1.0 match { case d:Int | d:Double => println(d)}
26. < console>:5: error: illegal variable in pattern alternative
27.        1.0 match { case d:Int | d:Double => println(d)}
28.                         ^
29. < console>:5: error: illegal variable in pattern alternative
30.        1.0 match { case d:Int | d:Double => println(d)}
31.                                 ^

Streams 2: Stream construction

This post just adds to the information about streams in the previous post Introducing Streams. In this topic the methods in the Stream object are looked at a little closer:
(standard warning. Examples were done in Scala 2.8 so some examples may need to be modified for Scala 2.7)

1. scala> import Stream._
2. import Stream._
4. // the empty stream.  good for terminating a stream
5. scala> empty        
6. res0: scala.collection.immutable.Stream[Nothing] = Stream()
8. // one way to declare a stream. 
9. scala> 1 #:: 2 #:: empty foreach (println _)
10. 1
11. 2
12.  // the other main way to create a stream
13. scala> cons(1, cons(2, empty)) foreach println
14. 1
15. 2
17. /*
18. Neither of the previous methods of stream creation are particularily useful because they are explicit and therefore they may as well be Lists or some other collection.  There is very little benefit in those cases.  However when the cons of a stream (tail) is defined as a function then things get a bit more interesting
19. */
20. scala> def i(x:Int,y:Int):Stream[Int] = (x*y) #:: i(x+1,y*2)
21. i: (x: Int,y: Int)Stream[Int]
23. scala> i(2,3) take 3 foreach println
24. 6
25. 18
26. 48
28. // now lets visit a few more methods in the Stream object
31. // create an infinite stream starting at 10
32. scala> from (10) take 3 foreach println     
33. 10
34. 11
35. 12
37. // an infinite stream starting at 10 an increasing by 3
38. scala> from (10,3) take 3 foreach println
39. 10
40. 13
41. 16
43. // converting an interator to a stream
44. scala> (1 until 4).iterator.toStream foreach println
45. 1
46. 2
47. 3
49. // creating an Iterable to a stream
50. scala> (1 until 4).toStream foreach println         
51. 1
52. 2
53. 3
55. // a stream that always returns 7
56. // the following is a pretty convoluted way to compute 7*49 :-P
57. scala> continually(7) take 49 reduceLeft {_ + _}
58. res10: Int = 343
60. // create a stream of 6 streams
61. scala> fill(6)(1 to 2 toStream) foreach println
62. Stream(1, ?)
63. Stream(1, ?)
64. Stream(1, ?)
65. Stream(1, ?)
66. Stream(1, ?)
67. Stream(1, ?)
69. /*
70. create the same stream as the last example but flatten it out so instead of being a stream of 6 streams it is a stream of 12 elements. Each element in each of the streams are visited in order
71. */
72. scala> fill(6)(1 to 2 toStream).flatten take 6 foreach println
73. 1
74. 2
75. 1
76. 2
77. 1
78. 2
80. /*
81. equivalent to:
82. (1 until 20 by 3).toStream foreach println
83. */
84. scala> range(1, 20, 3) foreach println
85. 1
86. 4
87. 7
88. 10
89. 13
90. 16
91. 19
93. /*
94. equivalent to
95. (1 until 3).toStream foreach println
96. */
97. scala> range(1,3) foreach println          
98. 1
99. 2
101. /*
102. iterate is fun! 
103. signature is iterator(start)(elem)
104. basically starting at 3 execute the function on the previous value in the stream.
106. This is an infinite stream
107. */
108. scala> iterate(3){i => i-10} take 5 foreach println _               
109. 3
110. -7
111. -17
112. -27
113. -37
115. /*
116. Another example
117. */
118. scala> iterate(3){i => i*2} take 5 foreach println _ 
119. 3
120. 6
121. 12
122. 24
123. 48
125. /*
126. A final example
127. */
128. scala> iterate(3){i => i} take 5 foreach println _  
129. 3
130. 3
131. 3
132. 3
133. 3
135. /*
136. The real final,final example.  This example uses the optional length parameter.   So the stream is restricted to 5 elements
137. */
138. scala> iterate(3,5){i => i} foreach println       
139. 3
140. 3
141. 3
142. 3
143. 3

Exclude style import semantics

I can't remember if I posted this tip yet so I will do it again (or not :-P ).

If you want to import all classes in a package or object exception for one or two there is a neat trick to exclude the unwanted elements:

1. /*
2. The File class is aliased to _ (oblivion) and then everything is imported (except File since it does not exist in that scope)
3. Note: File can be imported later
4. */
5. scala> import java.io.{File=>_,_}
6. import java.io.{File=>_, _}
8. scala> new File("it is a file")      
9. < console>:8: error: not found: type File
10.        new File("it is a file")
12. scala> object O {
13.      | def one = 1
14.      | def two = 2
15.      | def o = 0
16.      | }
17. defined module O
19. /*
20. Same tricks can apply to importing methods and fields from objects
21. */
22. scala> import O.{o=>_, _}
23. import O.{o=>_, _}
25. scala> one
26. res2: Int = 1
28. scala> two
29. res3: Int = 2
31. scala> o
32. < console>:15: error: not found: value o
33.        o
35. // Once a class is imported into scope it can not be removed
36. scala> import java.io.File
37. import java.io.File
39. scala> import java.io.{File=>_}
40. import java.io.{File=>_}
42. /*
43. this still works because the previous import statement only adds an alias it does not remove the alias
44. */
45. scala> new File("..")          
46. res6: java.io.File = ..
48. // There can be multiple aliases in a scope
49. scala> import java.io.{File=>jfile}
50. import java.io.{File=>jfile}
52. scala> new jfile("..")
53. res0: java.io.File = ..
55. // one more example of importing from objects
56. scala> case class X(a:Int, b:Int, c:Int)
57. defined class X
59. scala> val x = new X(1,2,3)
60. x: X = X(1,2,3)
62. scala> import x.{a=>_,b=>_,_}
63. import x.{a=>_, b=>_, _}
65. scala> c 
66. res1: Int = 3
68. scala> b
69. < console>:14: error: not found: value b
70.        b
71.        ^

Case classes in 2.8

Thanks to Scala 2.8's default parameters case classes have a couple wonderful new features in Scala 2.8. Specifically the copy method and the ability to have case classes with default parameters.

1. // constructor methods have defaults defined
2. scala> case class Node (name:String, left : Option[Node] = None, right : Option[Node] = None)                          
3. defined class Node
5. // the name is the only required parameter because it does not have a default
6. scala> val child = Node("leaf") 
7. child: Node = Node(leaf,None,None)
9. // left is being assigned here because the name of the parameter is not explicit
10. scala> val parent = Node("parent", Some(res0))
11. parent: Node = Node(parent,Some(Node(leaf,None,None)),None)
13. // now the left node is not defined just the right
14. scala> val node = Node("node", right=Some(res0))
15. node: Node = Node(node,None,Some(Node(leaf,None,None)))
17. /* 
18. The real power is the copy constructor that is automatically generated in the case class.  I can make a copy with any or all attributes modifed by using the copy constructor and declaring which field to modify
19. */
20. scala> parent.copy(right = Some(node))
21. res4: Node = Node(parent,Some(Node(leaf,None,None)),Some(Node(node,None,Some(Node(leaf,None,None)))))
23. scala> parent.copy(left=None)         
24. res5: Node = Node(parent,None,None)
26. scala> parent.copy(name="hoho")
27. res6: Node = Node(hoho,Some(Node(leaf,None,None)),None)
29. scala> parent.copy(name="hoho", left=None)
30. res7: Node = Node(hoho,None,None)