Daily scala

zipWithIndex

2010-05-27T23:28:00.000-07:00

A common desire is to have access to the index of an element when using collection methods like foreach, filter, foldLeft/Right, etc... Fortunately there is a simple way.

List('a','b','c','d').zipWithIndex.

But wait!

Does that not trigger an extra iteration through the collection?. Indeed it does and that is where Views help.

List('a','b','c','d').view.zipWithIndex

When using a view the collection is only traversed when required so there is no performance loss.

Here are some examples of zipWithIndex:

scala> val list = List('a','b','c','d')
list: List[Char] = List(a, b, c, d)
/*
I like to use functions constructed with case statements
in order to clearly label the index.  The alternative is 
to use x._2 for the index and x._1 for the value
*/
scala> list.view.zipWithIndex foreach {case (value,index) => println(value,index)}
(a,0)
(b,1)
(c,2)
(d,3)
// alternative syntax without case statement
scala> list.view.zipWithIndex foreach {e => println(e._1,e._2)}
(a,0)
(b,1)
(c,2)
(d,3)
/*
Fold left and right functions have 2 parameters (accumulator, nextValue) 
using a case statement allows you to expand that but watch the brackets!
*/
scala> (list.view.zipWithIndex foldLeft 0) {case (acc,(value,index)) => acc + value.toInt + index} 
res14: Int = 400
// alternative syntax without case statement
scala> (list.view.zipWithIndex foldLeft 0) {(acc,e) => acc + e._1.toInt + e._2} 
res23: Int = 400
/*
alternative foldLeft operator.  The thing I like about this
syntax is that it has the initial accumulator value on the left 
in the same position as the accumulator parameter in the function.
The other thing I like about it is that visually you can see that it starts with
"" and the folds left
*/
scala> ("" /: list.view.zipWithIndex) {                          
     | case (acc, (value, index)) if index % 2 == 0 => acc + value
     | case (acc, _) => acc                                       
     | }
res15: java.lang.String = ac
/*
This example filters based on the index then uses map to remove the index
force simply forces the view to be processed.  (I love these collections!)
*/
scala> list.view.zipWithIndex.filter { _._2 % 2 == 0 }.map { _._1}.force
res29: Seq[Char] = List(a, c)

Return value of a block

2010-05-26T23:41:00.000-07:00

A common misunderstanding is that a code block (without parameters) is a function. That is not the case. A code block is a sequence of statements that are executed and result the last statement is returned. That sounds like a Function0, however, if the block is passed to a method/function only the last statement will be returned to the function/method. If that method/function expects a function as the parameter the last statement maybe returned as a function not a value, this means that the block itself is not a function.

scala> var count = 0                                                                                                                         
count: Int = 0
// the last statement is returned as a function so count
// is incremented only one during the creation of the function
scala> List(1,2,3,4).map{count += 1;_ + 1}
res9: List[Int] = List(2, 3, 4, 5)
scala> count
res10: Int = 1
// now the count increment is within the function
scala> List(1,2,3,4).map{i => count += 1;i + 1}
res11: List[Int] = List(2, 3, 4, 5)
scala> count
res12: Int = 5

The previous example demonstrates a Gotcha if I ever saw one. Map expects a function so the block essentially constructs a function. The last statement being the function. The first line count += 1 executed only once because it is part of creating the function not part of the resulting function. This is equivalent to:

scala> val x = {count += 1 ; i:Int => i +1}
x: (Int) => Int = < function1>
scala> List(1,2,3,4).map(x)
res15: List[Int] = List(2, 3, 4, 5)

Beginning a block with the parameter list signals that the entire block is a function.

Rule of thumb: Functions with placeholder parameters should be a single statement.

Type Inference with Abstract Types

2010-05-20T00:04:00.001-07:00

A second "gotcha" that one might get tripped up when dealing with abstract types is the signature of the concrete class contains type information about the abstract type. So if you are not explicit when assigning a variable or defining a function you can get unexpected compiler errors.

scala> trait S {
     |   type x
     |   def get : x
     | }
defined trait S
scala> var sample = new S{ 
     |   type x = Int
     |   def get = 3
     | }
sample: java.lang.Object with S{type x = Int} = $anon$1@397af435
scala> sample = new S {
     |   type x = Double
     |   def get = 3.0
     | }
< console>:7: error: type mismatch;
 found   : java.lang.Object with S
 required: java.lang.Object with S{type x = Int}
       sample = new S {

In this example sample uses type inference so the actual type is S with underlying type Int. The consequence is that sample can only be assigned with instances of S with type x = Int. The fix is to explicitly declare the variable type:

scala> var sample2 : S = new S{ 
     |   type x = Int
     |   def get = 3
     | }
sample2: S = $anon$1@31602bbc
scala> sample2 = new S {
     |   type x = Double
     |   def get = 3.0
     | }
sample2: S = $anon$1@4de5ed7b

The same thing happens when declaring functions and allows type inference for function definition

scala> class Fac {
     |   def newS = new S {
     |     type x = Int
     |     def get = 3
     |   }
     | }
defined class Fac
scala> class SubFac extends Fac{
     |   override def newS = new S {
     |     type x = Double
     |     def get = 3.0
     |   }
     | }
< console>:8: error: type mismatch;
 found   : java.lang.Object with S
 required: java.lang.Object with S{type x = Int}
         override def newS = new S {

The fix for this example is to be explicit in the definition of the function in the superclass

Instance Type (Abstract type gotcha 1)

2010-05-17T02:24:00.000-07:00

In a previous post about abstract types I showed one of the benefits of using abstract types over parameterized types. Abstract Types vs Parameter. The next several posts will feature potential problems you may encounter when using Abstract Types.

I should point out that abstract types are not inherently difficult to understand but they are rather different from anything you will see when you come from the Java world so if you are new to them I would use them with caution at first.

In the abstract types example you will notice that the abstract type 'I' in Foreach is not within the trait Source rather it is outside in the Foreach trait. At first one might consider putting the type in Source rather than Foreach. The naive change can get you in trouble (but there is a couple easy fixes)

trait Foreach[A] {
  trait Source {
    type I <: java.io.Closeable  // moved this line into Source
    def in : I
    def next(in : I) : Option[A]
  }
  def source : Source
  
  def foreach[U](f : A => U) : Unit = {
    val s = source.in
    try {
      def processNext : Unit = source.next(s) match {
        case None => 
          ()
        case Some(value) => 
          f(value)
          processNext
      }
      
      processNext
    } finally {
      // correctly handle exceptions
      s.close
    }
  }
}

Compiling the class results in a compilation error:

jeichar: tmp$ scalac XX.scala
XX.scala:12: error: type mismatch;
found : s.type (with underlying type Foreach.this.Source#I)
required: _2.I where val _2: Foreach.this.Source
def processNext : Unit = source.next(s) match {
^
XX.scala:16: error: type mismatch;
found : value.type (with underlying type Any)
required: A
f(value)
^
two errors found

So what is the problem? The problem is simple but subtle. Notice that source is defined as a def. So calling source 2 times may return 2 different instances of Source. A simple change can fix this. Either change def source : Source to val source : Source. Or change the method foreach to assign the result from source to a val.

trait Foreach {
  trait Source {
    type I <: java.io.Closeable  // moved this line into Source
    def in : I
    def next(in : I) : Option[Int]
  }
  def source : Source
  
  def foreach[U](f : Int => U) : Unit = {
    // this assignment allows this example to compile
    val sameSource = source
    val s = sameSource.in
    try {
      def processNext : Unit = sameSource.next(s) match {
        case None => 
          ()
        case Some(value) => 
          f(value)
          processNext
      }
      
      processNext
    } finally {
      // correctly handle exceptions
      s.close
    }
  }
}

Abstract Types vs Parameter

2010-05-03T01:15:00.000-07:00

This topic (and the next) are intended to discuss abstract types. A class/trait with an abstract type is quite similar to a class/trait type parameter. For example:

trait C[A] {
  def get : A
  def doit(a:A):A
}
trait C2 {
  type A
  def get : A
  def doit(a:A):A
}

Both implementations have similar properties. However they are NOT the same. At first I thought that I could used them inter-changeably. However, consider the following examples:

//compiles
def p(c:C[Int]) = c.doit(c.get)
// doesn't compile
def p2(c:C2) = c.doit(c.get)

So why doesn't p2 compile? Because it returns A. From the signature of p2 it is impossible to know what p2 returns. There are several ways to fix this problem. One make the method return Unit:

// compiles because the internals of C2 does not leak out
def p(c:C2):Unit = c.doit(c.get)

Another fix would be to change doit to return Unit or an explicit return value like Int

trait C2 {
  type A
  def get : A
  def doit(a:A):Int
}
// compiles correctly
def p(c:C2) = c.doit(c.get)

A second difference between parameterized types and types with abstract type values is illustrated below:

trait C2 {
   type A
   def get : A
}
scala> var c : C2 = new C2 { 
     | type A = Int
     | def get = 3
     | }
c: C2 = $anon$1@11a40fff
// what is the type of result if at compile time the
// value of c is not known
scala> var result = c.get
result: C2#A = 3
scala> c = new C2 {      
     |    type A = String
     |    def get = "hi"
     | }
c: C2 = $anon$1@5f154718
// crazy eh :) the variable can be anything but does not
// have type Any so you cannot assign arbitrary values
scala> result = c.get
result: C2#A = hi
scala> result.isInstanceOf[String]
res0: Boolean = true
// while the dynamic type of result is a string the
// static type is not so you cannot assign a string to result
scala> result = "4"
< console> :8: error: type mismatch;
 found   : java.lang.String("4")
 required: C2#A
       result = "4"
                ^

The obvious question is what use are abstract types. I don't claim to know them all but the main point is that they do not expose the internal implementation details to the world. The famous cake pattern is one such example usage of abstract types.

I read the following as well (wish I could remember where):

Abstract types are good when extending and there will be concrete subclasses. Param type good for when a type is useful without extension but can handle several types.

A simpler example is examined here. It is loosely based on a real world usecase.
The example below is contrived so that it is smaller than the actual usecase, so consider the design and not the fact that the example could be easier done with other examples. In the real scenario this design reduced the lines of duplicated code from around 500 to 10.

The example below shows how a Traversable like object can be created from InputStreams and Readers. The important aspect is that the type signature of Foreach does not leak information about the implementation. Users of a Foreach object don't care whether it is backed onto an InputStream or Reader. They just care about the type of object contained.

I am leaving this already long post here. The next post will investigate different ways you can get in trouble trying to implement using abstract types.

import java.io.{InputStream, Reader, ByteArrayInputStream, StringReader}
import java.net.URL
object Foreach {
  def fromStream(s: => InputStream) = new Foreach[Int] {
    type I = InputStream
    def source = new Source {
      def in = s
      def next(_in : InputStream) = _in.read match {
        case -1 => None
        case i => Some(i)
      }
    }
  }
  
  def fromReader(s: => Reader) = new Foreach[Char] {
    type I = Reader
    def source = new Source {
      def in = s
      def next(_in : Reader) = _in.read match {
        case -1 => None
        case i => Some(i.toChar)
      }
    }
  }
  
  
  def fromInputAndFunction[A](s: => InputStream, f: Int => A) = new Foreach[A] {
    type I = InputStream
    def source = new Source {
      def in = s
      def next(_in : InputStream) = _in.read match {
        case -1 => None
        case i => Some(f(i))
      }
    }
  }
  
  
}
trait Foreach[A] {
  type I <: java.io.Closeable
  trait Source {
    def in : I
    def next(in : I) : Option[A]
  }
  def source : Source
  
  def foreach[U](f : A => U) : Unit = {
    val s = source.in
    try {
      def processNext : Unit = source.next(s) match {
        case None => 
          ()
        case Some(value) => 
          f(value)
          processNext
      }
      
      processNext
    } finally {
      // correctly handle exceptions
      s.close
    }
  }
}
object Test {
  def main(args : Array[String]) = {
    val data = "Hello World"
    val bytes = data.toArray.map { _.toByte }
    import Foreach._
    fromStream(new ByteArrayInputStream(bytes)).foreach {a => print(a.toChar)}
    
    println
    fromReader(new StringReader(data)) foreach print
    
    println
    
    fromInputAndFunction(new ByteArrayInputStream(bytes), i => i.toChar) foreach print
    
    println
  }
}

Filter with FlatMap (or collect)

2010-04-29T00:44:00.001-07:00

I picked up this tip from one of Daniel Spiewak's tweets. He tweeted a pro tip that uses flatMap to create a filtered list:

list flatMap {
  case st: String => Some(st)
  case _ => None
}

At a glance one might wonder why not simply use list.filter{_.isInstanceOf[String]}. The difference is that the flatMap will return a List[String].

However Scala 2.8 offers the collect method for doing a similar thing.

def strings(list: List[Any]) = list flatMap {
  case st: String => Some(st)
  case _ => None
}
// returned list is a List[String]
scala> strings("hi" :: 1 :: "world" :: 4 :: Nil)
res11: List[String] = List(hi, world)
// returned list is a List[Any] (not as useful)
scala> "hi" :: 1 :: "world" :: 4 :: Nil filter {_.isInstanceOf[String]}
res12: List[Any] = List(hi, world)
// collect returns List[String]
scala> "hi" :: 1 :: "world" :: 4 :: Nil collect {case s:String => s}           
res13: List[String] = List(hi, world)

Implicit Parameter Resolution

2010-04-27T23:38:00.001-07:00

This topic is a continuation of the previous implicit parameter topics:

This topic provides some explanation about how implicit parameters are resulted. There are very strict rules for which implicit value is to be applied to a implicit parameter. A simple way to think about it is that the "closest" definition will be used. Local scope, enclosing class, parent class, companion object of the desired type.

class X(val i:Int)
class Y(val i:Int)
object X {
  implicit def xx = new X(1)
}
class Method {
  def x(implicit x:X)=println(x.i)
  def y(implicit y:Y)=println(y.i)
}
trait M { 
  self : Method =>
  implicit def x1 = new X(10)
  implicit def y1 = new Y(100)
  def testy = y
  def testx = x
}
trait SM extends M {
  self : Method =>
  implicit def x2 = new X(20)
  implicit def y2 = new Y(200)
  
  def testy2 = y  
}
// implicit resolved from companion object of X
new Method().x
// explicit applied so that value is used
new Method().x(new X(3))
// implicit resolved from companion object of X
// NOT from M.  This is because the call site of x 
// is not within M therefore does not use the implicits in M
// for resolution.
(new Method with M).x
implicit def x = new X(30)
// local scope overrides companion object implicit
new Method().x
// explicit applied so that value is used
new Method().x(new X(3))
// local scope overrides companion object implicit
(new Method with M).x
// testy is defined within M so the implicits within M
(new Method with M).testy
// testx is defined within M so the implicit within M
// overrides the companion object implicit
(new Method with M).testx
// testy is within M (not SM) so the implicit within M
// is used
(new Method with SM).testy
// testy2 is within SM so the implicit within SM 
// overrides the implicit in M and the companion object
(new Method with SM).testy2

Output:

1
3
1
30
3
30
100
10
100
200

Implicit Parameters

2010-04-26T00:08:00.000-07:00

Evidently the topic of implicit parameters has not yet been correctly addressed. There have been several topic that refer to implicit parameters but none that directly discuss them. So before I continue with the topic of implicit parameter resolution I will discuss implicit parameters.

First, implicit parameters are not the same as implicit object conversions. Implicit parameters provide a way to allow parameters of a method to be "found". This is similar to default parameters at a glance but in fact is a different mechanism for finding the "default" value. It differs from implicit object conversion in that it is only a way for parameters for a method to be resolved. Implicit object conversion allows methods to appear to be called on one object when in fact that object is being converted behind the scenes to another type. (more or less)

An implicit parameter is a parameter to method or constructor that is marked as implicit. This means that if a parameter value is not supplied then the compiler will search for an "implicit" value defined within scope (according to resolution rules.) Implicit parameter resolution rules will be discussed soon.

Example:

scala> def p(implicit i:Int) = print(i)
p: (implicit i: Int)Unit
// defining a val/var/def as implicit 
// means that it will be considered during implicit resolution
scala> implicit val v=2
v: Int = 2
// scope is searched for a implicit value to sue
// v is found as marked implicit
scala> p               
2
// explicit declarations always overrides implicit values
scala> p(1)
1

Implicit parameters are very nice for simplifying APIs. For example the collections use implicit parameters to supply CanBuildFrom objects for many of the collection methods. This is because normally the user does not need to be concerned with those parameters. Another example is supplying an encoding to an IO library so the encoding is defined once (perhaps in a package object) and all methods can use the same encoding without having to define it for every method call.

One important restriction is that there can only be a single implicit keyword per method. It must be at the start of a parameter list (which also makes all values of that parameter list be implicit). I further understand that only the last parameter list may be implicit.

Here are several illegal examples:

// implicit is not in last parameter list
scala> def pp(implicit i:Int, a:Int)(b:Int) = println(a,i)                 
< console>:1: error: '=' expected but '(' found.
       def pp(implicit i:Int, a:Int)(b:Int) = println(a,i)
// there are 2 implicit parameters
scala> def pp(implicit j:Int, a:Int)(implicit i:Int,b:Int) = println(a,i)
< console>:1: error: '=' expected but '(' found.
      def pp(implicit j:Int, a:Int)(implicit i:Int,b:Int) = println(a,i)
// implicit is not the first parameter of the parameter list
scala> def pp(a:Int, implicit i:Int) = println(i,j)         
< console>:1: error: identifier expected but 'implicit' found.
       def pp(a:Int, implicit i:Int) = println(i,j)
                     ^

Here are several legal examples (Updated with useage examples):

scala> implicit def v = 7
v: Int
scala> implicit var x = 10L
x: Long
// i is implicit
scala> def pp(a:Int)(implicit i:Int) = println(a,i)
pp: (a: Int)(implicit i: Int)Unit
scala> pp(3)
(3,7)
// both i and b are implicit
scala> def pp(a:Int)(implicit i:Int, b:Long) = println(a,i,b) 
pp: (a: Int)(implicit i: Int,implicit b: Long)Unit
scala> pp(4)               
(4,7,10)
// both i and b are implicit
scala> def pp(implicit i:Int, b:Long) = println(i,b)  
pp: (implicit i: Int,implicit b: Long)Unit
scala> pp
(7,10)
// all or none of the parameters must be supplied
scala> pp(2)
< console>:13: error: not enough arguments for method pp: (implicit i: Int,implicit b: Long)Unit.
Unspecified value parameter b.
       pp(2)
// This is syntactically legal but I cannot seem to implicitly invoke this
// I would recommend: def pp(b:Long*)(implicit i:Int) = println(i,b)
scala> def pp(implicit i:Int, b:Long*) = println(i,b)
pp: (implicit i: Int,implicit b: Long*)Unit
scala> pp(3,1,2,3)
(3,WrappedArray(1, 2, 3))
scala> def pp(b:Long*)(implicit i:Int) = println(i,b)
pp: (b: Long*)(implicit i: Int)Unit
scala> pp(1,2,3)
(7,WrappedArray(1, 2, 3))

A related topic is Companion Object implicits.

Break Performance

2010-04-23T00:32:00.000-07:00

In the Breaks comments there were several questions about the performance of the Scala break command vs the Java break command. So I decided to take a look.

The code for the tests is available on GitHub at: Scala Benchmarks. Feel free to play around with it.

I personally don't think these tests say anything of particular import because they only test how fast the Java break is vs the Scala break without doing any work in the loop. So I don't expect these number would ever been seen in the real world. However that said if you have a tight loop with minimal processing then a Scala break may not be the correct construct to use.

Here is the Java test (labelled JavaSimpleBreak)

int i = 0;
while (i < 10) {
  if(i==1) break;
  i += 1;
}

Here is the Scala test (labelled ScalaSimpleBreak)

var i = 0;
breakable {
  while (i < 10) {
    if(i==1) break;
    i += 1;
  }
}

Out of curiosity I also added a test that created a new Exception each iteration (labelled ScalaException):

var i = 0;
  while (i < 10) {
    if(i==1) throw new Exception();
    i += 1;
  }

And also a test that just throws the same ScalaBreak exception each time. This one is weird since Scala Simple Break also throws the same exception but is much much faster so I think there is something about popping the stack in the example compared to the ScalaSimpleBreak test.

var i = 0;
breakable {
 while (i < 10) {
  if(i==1) break;
  i += 1;
 }
}

The results of the tests:

First, don't compare the break tests to the Exception tests. They are sufficiently different to not be worth comparing.
Second, remember that this is a micro benchmark and has very little relationship to reality.

90000000 iterations. Swapping every 90000000 tests
JavaSimpleBreak = 254 (0.0016279129387033098)
ScalaSimpleBreak = 2475 (0.015862537493270438)
ScalaBreakException = 18806 (0.12052964852462379)
ScalaException = 156028 (1.0)

90000000 iterations. Swapping every 500000 tests
JavaSimpleBreak = 772 (0.005138547761203965)
ScalaSimpleBreak = 2351 (0.015648608531853004)
ScalaBreakException = 19346 (0.12876987692778744)
ScalaException = 150237 (1.0)

90000000 iterations. Swapping every 500 tests
JavaSimpleBreak = 790 (0.005242446563543097)
ScalaSimpleBreak = 2247 (0.014911110668710557)
ScalaBreakException = 19213 (0.1274976276270298)
ScalaException = 150693 (1.0)

Companion Object Implicits

2010-04-21T23:46:00.000-07:00

When a method requires an implicit there are several ways that the implicit is resolved. One way is to search for an implicit definition in the companion object of the required type. For example: def x(implicit m:MyClass) parameter m will search local scope, class hierarchy and the MyClass companion object for an implicit val or def. (More on implicit resolution later).

To demonstrate the method put the following code block into a file and run the script:

class X(val i:Int) {
  def add(implicit x:X)=println(x.i+i)
}
object X {
  implicit def xx = new X(3)
}
// implicit is obtained from companion object of X
new X(3).add
val other = new {
  def print(implicit x:X)=println(x.i)
}
// implicit is obtained from companion object of X
other.print
implicit def x = new X(32)
// implicit is obtained local scope
other.print

Running: scala impl.scala should produce:
6
3
32

Breaks

2010-04-20T23:30:00.001-07:00

Scala 2.8 added the break control flow option. It is not implemented as a special language feature. Rather it is simply implemented as an object/trait using standard Scala mechanisms. If you are interested in creating a control flow object similar to this look at the Defining Custom Control Structures post.

The Break functionality is works basically how you would expect:

// Import the control flow methodsmethods
scala> import util.control.Breaks._
import util.control.Breaks._
// pass a function to the breakable method
scala> breakable {
     | for (i <- 1 to 10 ) {
     | if(i > 5) break  // call break when done
     | println(i)
     | }
     | }
1
2
3
4
5

Pretty intuitive but beware, break only breaks out to the first enclosing breakable. Here is an example of the issue:

scala> def loop(f : Int => Boolean) = breakable {
     | for (i <- 1 to 300) if (f(i)) break else println(i)
     | }
loop: (f: (Int) => Boolean)Unit
// This never ends because break is caught by breakable in the loop method
scala> breakable {
     | while(true) {
     | loop{ i => break; true }
     | }
     | }

Fortunately the implementers provide an elegant way to handle these sorts of cases. The Breaks object extends the Breaks class. By instantiating other instances of Breaks it is possible to control which breaks capture

scala> import scala.util.control._
import scala.util.control._
scala> 
scala> def loop(f : Int => Boolean) = {
     |   val Inner = new Breaks
     |   Inner.breakable {
     |     for (i <- 1 to 4) if (f(i)) Inner.break else println(i)
     |   }
     | }
loop: (f: (Int) => Boolean)Unit
scala> 
scala> val Outer = new Breaks
Outer: scala.util.control.Breaks = scala.util.control.Breaks@1ba4806
scala> Outer.breakable {
     |   while(true) {
     |     loop{ i => if(i==4) Outer.break; false}
     |   }
     | }
1
2
3

Scala 2.8 Arrays are not Traversables

2010-04-19T04:23:00.000-07:00

One performance/consistency change that has been make in Scala 2.8 is to make Scala Array always be a Java Array. This has some consequences which we will examine in this post. The biggest one is that Array is not a Scala Collection/Traversable. It is implicitly converted to one but it is not an instance of a Traversable. There are several reasons this was done. Probably the biggest is for performance. Because a Scala array is a Java array there is no overhead when using a Scala array.

Thanks to implicit type conversion all the normal collection methods are useable with an array. Even better, after running a method like map the result will again be a Java array. So the API is much more consistent.

An example illustrating that an Array is not a Traversable:

// This does not compile (which is good) 
// because Traversable[Int] can never be an array
scala> def x(t:Traversable[Int]) = t match {
     | case x : Array[Int] => true          
     | }
< console>:13: error: pattern type is incompatible with expected type;
 found   : Array[Int]
 required: Traversable[Int]
       case x : Array[Int] => true
                ^
< console>:13: error: type mismatch;
 found   : Array[Int]
 required: Traversable[Int]
       case x : Array[Int] => true
              ^

Another example:

scala> def x(t:Traversable[Int]) = t.isInstanceOf[Array[_]]
x: (t: Traversable[Int])Boolean
/* this evaluates to false because Array is converted
 * to WrappedArray because it has to be implicitly converted
 * to a Traversable.  Since Array is not a Traversable the resulting 
 * object is not an Array
 */
scala> x(Array(1,2,3))                                     
res24: Boolean = false
scala> def x(t:Traversable[Int]) = println(t)
x: (t: Traversable[Int])Unit
// This method call demonstrates the previous assertion
scala> x(Array(1,2,3))                                            
WrappedArray(1, 2, 3)

So suppose you want to be able to accept and use arrays and Traversables in a method but you want to be able to
check that the parameter is an Array. Why not match against WrappedArray. You probably can, but you may get performance improvements in some cases if you don't require wrapping the array.

For a more concrete example of why you may want to do this. In a Input/Output routine I wrote I would write the data one way if the input was an Array: stream.write(array). But if the input was a traversable then I would have to handle it differently. My particular issue was more complicated than that but that is the basic issue.

So the work around is to define a Generic parameter for the method:

scala> def x[T <% Traversable[Int]](t:T) = t match { 
     | case x : Array[Int] => true                                
     | }
x: [T](t: T)(implicit evidence$1: (T) => Traversable[Int])Boolean
scala> x(Array(1,2,3))                               
res27: Boolean = true

2010-04-16T01:56:00.000-07:00

A quick note. ScalaDays Rocks! Wish you were here :)

This topic just demonstrates a cute little trick that can occasionally be quite useful:

scala> List(1,2,3) ++ Some(4)
res0: List[Int] = List(1, 2, 3, 4)
scala> List(1,2,3) ++ None   
res1: List[Int] = List(1, 2, 3)

Options are implicitly converted to Iterables, so Options can be appended to collections.

scala> val x : Iterable[Int] = None       
x: Iterable[Int] = List()
scala> val x : Iterable[Int] = Some(4)
x: Iterable[Int] = List(4)

Creating Custom Traversable implementations

2010-04-13T01:57:00.000-07:00

One of the most talked about features of Scala 2.8 is the improved Collections libraries. Creating your own implementation is trivial, however if you want your new collection to behave the same way as all the included libraries there are a few tips you need to be aware of.

Note: All of these examples can either be ran in the REPL or put in a file and ran

Starting with the simple implementation:

import scala.collection._
import scala.collection.generic._
class MyColl[A](seq : A*) extends Traversable[A] {
    // only abstract method in traversable is foreach... easy :) 
  def foreach[U](f: A => U) = util.Random.shuffle(seq.toSeq).foreach(f)
}

This is a silly collection I admit but it is custom :).

This example works but if you test the result of a map operation (or any other operation that returns a new instance of the collection) you will notice it is not an instance of MyColl. This is expected because unless otherwise defined Traversable will return a new instance of Traversable.

To demonstrate run the following tests:

val c = new MyColl(1, 2, 3)
println (c mkString ",")
println(c mkString ",")
println(c drop 1 mkString ",")
// this two next assertions fail (see following explanation)
assert(c.drop(1).isInstanceOf[MyColl[_]])
assert((c map {_ + 1}).isInstanceOf[MyColl[_]])

Both assertions will fail. The reason for these failures is because the collection is immutable which dictates by necessity that a new object must be returned from filter/map/etc... Since the Traversable trait returns instances of Traversable these two assertions fail. The easiest way to make these methods return an instance of MyColl is to make the following changes/additions.

import scala.collection._
import scala.collection.generic._
/*
Adding GenericTraversableTemplate will delegate the creation of new
collections to the companion object.  Adding the trait and
companion object causes all the new collections to be instances of MyColl
*/
class MyColl[A](seq : A*) extends Traversable[A] 
                             with GenericTraversableTemplate[A, MyColl] {
  override def companion = MyColl
  def foreach[U](f: A => U) = util.Random.shuffle(seq.toSeq).foreach(f)
}
// The TraversableFactory trait is required by GenericTraversableTemplate
object MyColl extends TraversableFactory[MyColl] {
/* 
If you look at the signatures of many methods in TraversableLike they have an
implicit parameter canBuildFrom.  This allows one to define how the returned collections
are built.  For example one could make a list's map method return a Set
In this case we define the default canBuildFrom for MyColl
*/
  implicit def canBuildFrom[A]: CanBuildFrom[Coll, A, MyColl[A]] = new GenericCanBuildFrom[A]
/*  
The method that builds the new collection.  This is a simple implementation
but it works.  There are other implementations to assist with implementation if
needed
*/
  def newBuilder[A] = new scala.collection.mutable.LazyBuilder[A,MyColl[A]] {
    def result = {
      val data = parts.foldLeft(List[A]()){(l,n) => l ++ n}
      new MyColl(data:_*)
    }
  }
}

Now instances of MyColl will be created by the various filter/map/etc... methods and that is fine as long as the new object is not required at compile-time. But suppose we added a method to the class and want that accessible after applying methods like map and filter.

Adding val o : MyColl[Long] = c map {_.toLong} to the assertions will cause a compilation error since statically the class returned is Traversable[Long]. The fix is easy.

All that needs to be done is to add with TraversableLike[A, MyColl[A]] to MyColl and we are golden. There may be other methods as well but this works and is simple.

Note that the order in which the traits are mixed in is important. TraversableLike[A, MyColl[A]] must be mixed in after Traversable[A]. The reason is that we want methods like map and drop to return instances of MyColl (statically as well as dynamically). If the order was reversed then those methods would return Traversable event though statically the actual instances would still be MyColl.

import scala.collection._
import scala.collection.generic._
class MyColl[A](seq : A*) extends Traversable[A]
                             with GenericTraversableTemplate[A, MyColl] 
                             with TraversableLike[A, MyColl[A]] {
  override def companion = MyColl
  def foreach[U](f: A => U) = util.Random.shuffle(seq.toSeq).foreach(f)
}
object MyColl extends TraversableFactory[MyColl] {  
  implicit def canBuildFrom[A]: CanBuildFrom[Coll, A, MyColl[A]] = new GenericCanBuildFrom[A]
  def newBuilder[A] = new scala.collection.mutable.LazyBuilder[A,MyColl[A]] {
    def result = {
      val data = parts.foldLeft(List[A]()){(l,n) => l ++ n}
      new MyColl(data:_*)
    }
  }
}

Now add in a new method to demonstrate that the new collection works as desired and we are done.

The following is the complete implementation with the tests. You can put it in a file and run scala <filename> or paste it into a REPL

import scala.collection._
import scala.collection.generic._
import scala.collection.mutable.{ Builder, ListBuffer }
class MyColl[A](seq : A*) extends Traversable[A]
                             with GenericTraversableTemplate[A, MyColl] 
                             with TraversableLike[A, MyColl[A]] {
  override def companion = MyColl
  def foreach[U](f: A => U) = util.Random.shuffle(seq.toSeq).foreach(f)
  def sayhi = println("hi!")
}
object MyColl extends TraversableFactory[MyColl] {  
  implicit def canBuildFrom[A]: CanBuildFrom[Coll, A, MyColl[A]] = new GenericCanBuildFrom[A]
  def newBuilder[A] = new ListBuffer[A] mapResult (x => new MyColl(x:_*))
}
val c = new MyColl(1, 2, 3)
println (c mkString ",")
println(c mkString ",")
assert(c.drop(1).isInstanceOf[MyColl[_]])
assert((c map {_ + 1}).isInstanceOf[MyColl[_]])
val o : MyColl[Int] = c filter {_ < 2}
println(o mkString "," )
o.sayhi

Elidable (remove method calls at compile time)

2010-04-09T00:59:00.000-07:00

In Scala 2.8 there is a useful annotation called 'elidable'. This annotation flags a method so that given certain compiler flags all calls to the method will be removed. This is handy if you are writing a logger. The log methods can be annotated so that when compiling for production all log calls below a certain level would be removed from the compiled code. Several of the methods in Predef are annotated with elidable. Specifically, assume, assert and require.

The elidable annotation takes an Int parameter which specifies the priority of the method. The lower the integer the more likely the method would be removed during compilation. The elidable object defines several values that are used in our example.

When compiling with the -Xelide-below , the compiler parameter will remove all calls to elidable methods value and below.

To try the following example copy the example into a scala file (elidable.scala for example) and compile as indicated below:

package example
import scala.annotation.elidable
import scala.annotation.elidable._
object ElidableExamples {
    @elidable(ALL) def all = println("all")
    @elidable(ASSERTION) def assertion = println("assertion")
    @elidable(CONFIG) def config = println("config")
    @elidable(FINE) def fine = println("fine")
    @elidable(FINER) def finer = println("finer")
    @elidable(FINEST) def finest = println("finest")
    @elidable(INFO) def info = println("info")
    @elidable(OFF) def off = println("off")
    @elidable(SEVERE) def severe = println("severe")
    @elidable(WARNING) def warning = println("warning")
}
object Main extends Application {
println("starting")
import ElidableExamples._
all
assertion
config
fine
finer
finest
info
off
severe
warning
println("ending")
assert(false, "boom!")
}

Output from scalac elidable.scala && scala example.Main

starting
assertion
off
ending
java.lang.AssertionError: assertion failed: boom!
at scala.Predef$.assert(Predef.scala:93)
at example.Main$.(elidable.scala:34)
at example.Main$.(elidable.scala)
at example.Main.main(elidable.scala)

Output from scalac -Xelide-below 0 elidable.scala && scala example.Main

starting
assertion
config
fine
finer
finest
info
off
severe
warning
ending
java.lang.AssertionError: assertion failed: boom!
at scala.Predef$.assert(Predef.scala:93)
at example.Main$.(elidable.scala:34)
at example.Main$.(elidable.scala)
at example.Main.main(elidable.scala)

Output from scalac -Xelide-below 1000 elidable.scala && scala example.Main

starting
assertion
off
severe
ending
java.lang.AssertionError: assertion failed: boom!
at scala.Predef$.assert(Predef.scala:93)
at example.Main$.(elidable.scala:34)
at example.Main$.(elidable.scala)
at example.Main.main(elidable.scala)

Output from scalac -Xelide-below 3000 elidable.scala && scala example.Main

starting
off
ending

Scala 2.7 to 2.8 migration help

2010-04-06T00:03:00.001-07:00

Scala 2.8 has some very significant differences from Scala 2.7 so if you want to migrate from Scala 2.7 to 2.8 you might want to view this Stack overflow topic:

What are the biggest differences between Scala 2.8 and Scala 2.7?

It has some good tips.

Variant Positions 3

2010-04-06T00:01:00.001-07:00

The last few topics all discussed variance in its different forms. The following is a cheat sheet of the where the different variances exist within a class.

See Variant positions 1 for a discussion on one position in a class that is a contravariant position.

The example is:
scala> class Output[+A] {def write(a : A) = () /*do write*/ }
In this example A in the method write is an contravariant position. Which means the previous definition is not legal because A is defined as Covariant in the class definition. In a class there are several positions with different variance characteristics. Here's the example from Programming in Scala:

abstract class Cat[-T, +U] { 
    def meow[W<sup>-</sup>](volume: T^-, listener: Cat[U<sup>+</sup>, T<sup>-</sup>]^-): Cat[Cat[U<sup>+</sup>, T<sup>-</sup>]^-, U⁺]⁺
}

If you remove the superscript + and - the above example actually compiles. The + and - indicate if the position is a covariant or contravariant position. Its not critical to memorize the positions (in my opinion). Just look it up as needed. The rule of thumb is that each nested position is inverted (flipped) value of it enclosing position.

That is all I will say about that :)

Working on Scala IO

2010-04-04T05:00:00.000-07:00

I just want to let everyone know that I am working hard on a Scala IO library right now and as such I want to dedicate all my spare time to doing that. So (as has been the case lately) I plan to only update the blog 2-3 times a week. Once done with the library I will try to increase the frequency of updates on the blog.

If you have any topics or code snippets you think would be fit good with the block please send them to me at jeichar.w@gmail.com.

Jesse

Variant Positions 2

2010-03-31T05:13:00.000-07:00

This is a continuation of post: Variant Positions 1

...

My first attempt at the Verified was to make it a mutable object (my Java tourettes kicking in). But it cannot be covariant and mutable. Look at the code to see why:

class Verified[+A <: V,V](assertion : (V) => Boolean, private var value : A){
    assert(assertion(value))
    
    def a = value
// update is illegal.  See the example below
    def update[ B >: A <: V](a : B) = value = a
}
def notNull(obj : AnyRef) = obj != null
val v = new Verified(notNull, "hi")
/*
Up to this point everything looks ok but the next line
will assign an object to value which is a reference to a String
*/
v update (new Object())

For Verified to be mutable A must be invariant. If you look at the Mutable collections in Scala they are all invariant.

Here is an interesting example of both invariant and covariant type parameters in a class hierarchy:

scala> class X[+A](val x :A)
defined class X
scala> class Y[A](var a: A) extends X[A](a)
defined class Y
scala> val x: X[Any] = new Y[String]("hi")
x: X[Any] = Y@1732a4df
scala> x.asInstanceOf[Y[String]].a="ho"

This example is perfectly legal because no matter how X[Any] is used no illegal assignment in Y can occur. The interesting thing is that the object can be used in covariant usecases when only X is required. This is now the collections in Scala can work.

Here is a little example of collections invariance and covariance in action. In List the parameter is covariant but in Buffer it is invariant

scala> def printList(l : List[Any]) = print(l mkString " :: ")
printList: (l: List[Any])Unit
scala> val buffer = Buffer(1,2,3)
buffer: scala.collection.mutable.Buffer[Int] = ArrayBuffer(1, 2, 3)
scala> printList(buffer)
1 :: 2 :: 3
/*
++ is part of Iterable.  Since Iterable is covariant ++ 
returns a new buffer it does not modify the existing buffer
All mutators are only defined on invariant traits
*/
scala> buffer ++ List(4)
res16: scala.collection.mutable.Buffer[Int] = ArrayBuffer(1, 2, 3, 4)
scala> res16 eq buffer
res17: Boolean = false
/*
buffer defines += to add an element to the buffer
so res27 is the same buffer with an extra element
*/
scala> buffer += 10
res27: buffer.type = ArrayBuffer(1, 2, 3, 10)
scala> res27 eq buffer
res28: Boolean = true

Variant Positions 1

2010-03-30T13:47:00.000-07:00

An additional topic on variance to finish up the major points on the topic. The previous two posts:

contain required information for following this post.

In-, co- and contra- variance are the three types of variance expressible in Scala. I showed how this affects assignments and arguments being pass to methods in the last two topics. This looks at how the different types of variance influences how classes can be defined. In the last post we saw how the compiler complained about a method definition in a covariant class because the compiler recognized that such a definition was inherently dangerous and must be prohibited. The example was:

scala> class Output[+A] {def write(a : A) = () /*do write*/ }
< console>:5: error: covariant type A occurs in contravariant position in type A of value a
       class Output[+A] {def write(a : A) = () /*do write*/ }
                                   ^

For an class like Output it does not make sense to have A be covariant so we changed A to be contravariant. However suppose we have a collection type class.

class Verified[+A] (assertion : (A) => Boolean, value : A){
    assert(assertion(value))
    
    def a = value
    def a_=(a : A) = new Verified(assertion, a)
}

The previous definition is not legal because value and a in the parameter of a_= "occur in a contravariant position." What to do? Making A contravariant isn't an option:

class Verified[+A <: V,V](assertion : (V) => Boolean, val value : A){
    assert(assertion(value))
/*
this is the key.  Restrict possible types of
A Since B is a super (or equal) type of A
*/
    def update[ B >: A <: V](a : B) = new Verified(assertion, a)
}
// example useage
scala> def notNull(obj : AnyRef) = obj != null
notNull: (obj: AnyRef)Boolean
scala> val v = new Verified(notNull, "hi")
v: Verified[java.lang.String,AnyRef] = Verified@307b37df
scala> val newV = v update (new Object())
newV: Verified[java.lang.Object,AnyRef] = Verified@36f72f09
// 3 is not legal because the type parameter 'V' is AnyRef.  Int is a subclass of Any NOT AnyRef
scala> val newV = v update (3)           
< console>:8: error: inferred type arguments [Any] do not conform to method update's type parameter bounds [B >: java.lang.String <: AnyRef]
       val newV = v update (3)
                  ^

Contravariance

2010-03-27T05:57:00.000-07:00

Continuing on with variance and type parameters, this topic will discuss contravariance. See the post In- and Co- variance of type parameters for the intro material required for this topic.

Covariant parameters allow for an additional dimension of type compatibility:
val l : List[Object] = List("this is legal")
Contravariance provides the opposite:

// If the type parameter of list was contravariant this would be legal:
val l : List[String] = List(new Object())

As covariance is indicated by a '+' before the type contravariance is indicated by a '-'

scala> class X[-A]
defined class X
scala> val l : X[String] = new X[Object]
l: X[String] = X@66201d6d

I can almost hear the "cool... but why?". Following the lead in the Programming In Scala book. Consider OutputStream first and a method in a Collection second. (The following code is illegal but consider it)

class Output[+A] {def write(a : A) = () /*do write*/ }
def writeObject(out : Output[Object]) = out.write("hello")
/*
Uh oh you this only is for outputting lists not Objects 
(certainly not the String that is actually written)
Runtime error for sure!
*/
writeObject(new Output[List[String]])

The previous example (if it would compile) would explode because an Output that can only write lists is passed to the method. In the example a String is written to the Output object. The Output[List[String]] cannot handle that.

Fortunately the compiler sees the definition of the class and recognizes this is an error waiting to happen and makes it illegal:

scala> class Output[+A] {def write(a : A) = () /*do write*/ }
< console>:5: error: covariant type A occurs in contravariant position in type A of value a
       class Output[+A] {def write(a : A) = () /*do write*/ }
                                   ^

Consider the implications of making A contravariant?

// The definition of object is now legal
class Output[-A] {def write(a : A) = () /*do write*/ }
// this is now a safe method definition since the parameter of Output must be a Object or a super class
def writeObject(out : Output[Object]) = out.write("hello")
// Now this is illegal as it should be
scala> writeObject(new Output[List[String]])
< console>:8: error: type mismatch;
 found   : Output[List[String]]
 required: Output[java.lang.Object]
       writeObject(new Output[List[String]])
       
// this is legal... 
scala> writeObject(new Output[Any])

In this example Output[Any] can be passed to the method. This makes sense. If the Output object knows how to write Any oject then it knows how to write an Object; its all good.

In- and Co- variance of type parameters

2010-03-24T01:08:00.000-07:00

In Java most parameterized types are considered to be "invariant". What does that mean? Here is an example to explain:

/*
This is an example of a parameterized class that with an invariant parameter B
In both Scala and Java parameters are invariant by default.
*/
scala> class Invariant[B]
defined class Invariant
scala> var x : Invariant[Object] = new Invariant[Object]
x: Invariant[java.lang.Object] = Invariant@2e0c5575
/*
Note: Invariant[String] cannot be assigned to Invariant[Object]
      even though logically it seems like it should be.
      This is the effect of invariance.  Covariant parameters do not have
      this restriction.
*/
scala> var x : Invariant[Object] = new Invariant[String]
< console>:6: error: type mismatch;
 found   : Invariant[String]
 required: Invariant[java.lang.Object]
 var x : Invariant[Object] = new Invariant[String]
                             ^
scala> class Sub[A] extends Invariant[A]   
defined class Sub
/*
Since Sub is a subclass of Invariant it can be assigned
(but not Sub[String])
*/
scala> val x : Invariant[Object] = new Sub[Object]
x: Invariant[java.lang.Object] = Sub@26ced1a8

Assignment compatibility has multiple dimensions: the object type and the types of the parameters. Unlike object type the compatibility of the type-parameters can be covariant, contravariant and invariant. Java has invariant parameters and that is demonstrated by the previous example. Covariant parameters allow subclassing. Contravariant parameters need their own topic.

// The '+' indicates the parameter is covariant
scala> class Covariant[+B]
defined class Covariant
scala> var x : Covariant[Object] = new Covariant[Object]
x: Covariant[java.lang.Object] = Covariant@315cb235
// Now this is legal
scala> var x : Covariant[Object] = new Covariant[String]
x: Covariant[java.lang.Object] = Covariant@26e2e276
/*
Warning: The following is not legal because 
         you cannot supply an invariant parameter 
         with a covariant value.
*/
scala> class Sub[+A] extends Invariant[A]
< console>:7: error: covariant type A occurs in invariant position in type [+A]Invariant[A] with ScalaObject{def this(): Sub[A]} of class Sub
       class Sub[+A] extends Invariant[A]
             ^
scala> class Sub[+A] extends Covariant[A]
defined class Sub
scala> class Sub[A] extends Covariant[A] 
defined class Sub

Implicit '=' operator

2010-03-22T01:01:00.000-07:00

Continuing on with operators, There is a special type of operator in Scala. It is an operator that ends with =. If a class has operation (methods with an operator identifer) the = can be appended to the effectively creating a new method. In truth a new method is not created instead the compiler rewrites the line.

For example. If a method (like Int) defines + then a method call += can be used. It can be used to mutate a variable:

scala> var i = 1
i: Int = 1
scala> i += 1
scala> i
res3: Int = 2

To illustrate this is not a special case for Int the next example defines several operations and demonstrates in place variable mutation.

scala> case class MyClass(i:Int) {      
     | def +(j:Int) = new MyClass(j + i)
     | def -(j:Int) = new MyClass(i - j)
     | def ^(j:Int) = MyClass(j)
     | def +|(j:Int) = new MyClass(j + i / 3)
     | }
defined class MyClass
scala> var c = MyClass(1)
c: MyClass = MyClass(1)
scala> c+=6
scala> c
res5: MyClass = MyClass(7)
scala> c -= 2
scala> c
res7: MyClass = MyClass(5)
scala> c ^= 10
scala> c
res23: MyClass = MyClass(10)
scala> c +|= 5
scala> c
res25: MyClass = MyClass(8)

Here are several more examples using existing classes in Scala. They are all immutable examples.

scala> var l = Set(1,2,3) 
l: scala.collection.immutable.Set[Int] = Set(1, 2, 3)
scala> l += 10
scala> l
res7: scala.collection.immutable.Set[Int] = Set(1, 2, 3, 10)
scala> var seq = Seq(5,6,3)
seq: Seq[Int] = List(5, 6, 3)
scala> seq :+= 10
scala> seq                 
res9: Seq[Int] = List(5, 6, 3, 10)
scala> seq +:= 10   
scala> seq       
res11: Seq[Int] = List(10, 5, 6, 3, 10)
scala> var list = List(32)
list: List[Int] = List(32)
scala> list ::= 12
scala> list
res13: List[Int] = List(12, 32)

Note: assignment operators can also be defined as methods to mutate an object

scala> case class MyClass(var i:Int) {
     | def += (j:Int) = { i+=j ; this }
     | }
defined class MyClass
scala> val m = MyClass(6)
m: MyClass = MyClass(6)
scala> m += 7
res0: MyClass = MyClass(13)
scala> m += 9
res1: MyClass = MyClass(22)
scala> res1 eq m
res2: Boolean = true

Operators

2010-03-19T06:50:00.000-07:00

Since Scala allows one to define the behavior of operators there are some rules involving operators and assignment like +=. One of the standard method calls in most languages is i += 1.

Since i+=1(no spaces) is also valid, Scala has some rules regarding how statements like i+=1 should be broken up. Obviously we know it should be 'i' '+=' and '1'. So there is a special class of characters called operators. I don't know all of them but a few are: + - ^ * / % ! | & =( ':' is sort of part of this group but has some special properties as well).

These characters can be method names but they cannot be combined with other identifier characters.

Update: These characters can be combined with other identifier characters if there is an under score so:

def x+ = 3   // not valid
def x_+ = 3  // valid
def +x = 3   // not valid

However these characters are special because they can be combined in a special way with '=' for a special assignment construct as shown in the next post.

(end update)

scala> case class MyClass(i:Int) {
     | def +(j:Int) = new MyClass(j + i)
     | def -(j:Int) = new MyClass(i - j)
     | def ^(j:Int) = MyClass(j)
     | def +|(j:Int) = new MyClass(j + i / 3)
     | }
 
 scala> val c = MyClass(3)
 c: MyClass = MyClass(3)
 scala> c + 4
 res26: MyClass = MyClass(7)
 scala> c-2 
 res27: MyClass = MyClass(1)
 scala> c -6
 res28: MyClass = MyClass(-3)
 scala> c ^ 3
 res29: MyClass = MyClass(3)
 scala> c+|5
 res31: MyClass = MyClass(6)

Geoscript.scala

2010-03-17T06:39:00.000-07:00

This is a bit of a departure from the standard daily-scala format, but it is one of the elements I had always invisioned. This topic takes a look at Geoscript.scala. (Github repo is at: http://github.com/dwins/geoscript.scala)

Geoscript has been given a Scala implementation. It is based on the Geotools Java library and thus far provides a script interface to access some of the basic spatial functions.

For setting up the console with the required libraries follow the instructions at: http://geoscript.org/scala/quickstart.html#quickstart

Here is an example of Geoscript in action:

scala> import org.geoscript.GeoScript._
import org.geoscript.GeoScript._
scala> import org.geoscript.geometry._
import org.geoscript.geometry._
scala> val line = LineString((10, 10), (20, 20), (30, 40))
line: org.geoscript.geometry.LineString = LINESTRING (10 10, 20 20, 30 40)
/*
create a polygon by buffering the line (essentially expanding the line by 10 units that is degrees if not otherwise specified)
*/
scala> val poly = line buffer 10
poly: org.geoscript.geometry.Geometry = POLYGON ((11.781455848733053 25.923591472464004, 21.05572809000084 44.47213595499958, 22.100060210309515 46.13114600374718, 23.447982586398712 47.55453954995706, 25.04769531727891 48.68761637789669, 26.837722339831622 49.48683298050514, 28.74927391943886 49.921475911950004, 30.708890200906794 49.97484208812642, 32.64126422950409 49.6448806768120...
// query the area of the polygon
scala> poly.area
res0: Double = 1041.9912814842407
// get the centroids of the polygon and line
scala> line.centroid
res1: org.geoscript.geometry.Point = POINT (21.12574113277207 24.188611699158105)
scala> poly.centroid
res2: org.geoscript.geometry.Point = POINT (20.79088988611118 24.43096430943361)
/*
Obviously the polygon and line intersect since the polygon is a buffer of the line
*/
scala> poly.intersects(line)
res3: Boolean = true
scala> val poly2 = Geometry.fromWKT("POLYGON ((10 10, 10 20, 20 20, 20 15, 10 10))")
poly2: org.geoscript.geometry.Geometry = POLYGON ((10 10, 10 20, 20 20, 20 15, 10 10))
// less trivial intersects operation
scala> poly intersects poly2
res3: Boolean = true
// not make a new geometry from the intersection of the two geometries
scala> val intersecting = poly intersection poly2
intersecting: org.geoscript.geometry.Geometry = POLYGON ((10 10, 10 20, 20 20, 20 15, 10 10))
scala> intersecting.area 
res6: Double = 75.0
scala> import org.geoscript.projection._
import org.geoscript.projection._
/*
None of the previous geometries has a projection associated.  
A new geometry can have one created with a projection by using the in(Projection) method
*/ 
scala> val latLongPoly = poly2 in Projection("epsg:4326")
latLongPoly: org.geoscript.geometry.Geometry = POLYGON ((10 10, 10 20, 20 20, 20 15, 10 10))
// now reproject the latlong projection to a french projection 
scala> latLongPoly in Projection("epsg:21781")
res12: org.geoscript.geometry.Geometry = POLYGON ((950650.7690658928 -4203986.192880551, 900363.7533498043 -2900002.601715782, 2061411.5566836582 -2774908.8442438124, 2174910.791185147 -3393231.5380846346, 950650.7690658928 -4203986.192880551))

Assert, Require, Assume

2010-03-16T05:00:00.001-07:00

Very simple but useful are the methods assert, require and assume which are built into the Predef object. As you might expect they are methods for performing certain checks during runtime to verify certain conditions. They do not use the Java assert framework and therefore are always evaluated regardless of whether or not assertions are enabled.

Update: Scala 2.8 has an annotation called elidable that will (when 2.8 is complete) allow one to remove method calls at compile time by setting a compiler flag. The assert, etc... methods are all marked with this flag and as a result can be removed at compile time for production environments.

Scala2.8

scala> var called = 0
called: Int = 0
scala> called
res0: Int = 0
/*
assert, require and assume have call by name parameters so the message is only 
calculated when the assertion fails.
*/
scala> assert (called == 0, {called += 1; println("called is not 0")})
scala> require (called == 0, {called += 1; println("called is not 0")})
scala> assume (called == 0, {called += 1; println("called is not 0")}) 
scala> called = 1
called: Int = 1
// demonstrating that the method is in fact called when the assertion fails
scala> assert (called == 0, {called += 1; println("called is not 0")}) 
called is not 0
java.lang.AssertionError: assertion failed: ()
 at scala.Predef$.assert(Predef.scala:95)
 ...
 
scala> called
res4: Int = 2
/*
Require is intended to be used as a precondition of a method so 
it throws an IllegalArgumentException, not an AssertionError
*/
scala> require (called == 0, {called += 1; println("called is not 0")})
called is not 0
java.lang.IllegalArgumentException: requirement failed: ()
 at scala.Predef$.require(Predef.scala:117)
 ...
 
scala> called                                                          
res6: Int = 3
scala> assume (called == 0, {called += 1; println("called is not 0")}) 
called is not 0
java.lang.AssertionError: assumption failed: ()
 at scala.Predef$.assume(Predef.scala:107)
 ...
 
scala> called                                                         
res8: Int = 4

scala 2.7.7

/*
In Scala 2.7 the parameter is evaluated before the 
method is called so the side effect of the message causes
the assertion to fail
*/
scala> assert (called == 0, {called += 1; println("called is not 0")})
called is not 0
scala> called
res2: Int = 1

Unzip

2010-03-15T13:12:00.000-07:00

_ Scala 2.8 only tip _

Unzip is a handy companion to partition.
- Partition divides a traversable into two traversables by a boolean predicate.
- Unzip divides a traversable into two by dividing each element into two parts (each becomes an element in one traversable). If an element is a Tuple2 then each tuple is divided into two otherwise a function is required to divide an element into two.

// basic usage
scala> List((1,2),(2,3)).unzip
res2: (List[Int], List[Int]) = (List(1, 2),List(2, 3))
/* 
tuples can be of different types 
and the resulting traversables reflect the differing types
*/
scala> List((2,"a"),(3,"b")).unzip
res3: (List[Int], List[java.lang.String]) = (List(2, 3),List(a, b))
// Maps are Traversable[Collection] so unzip works with them
scala> Map(1 -> 2, 3 -> 4).unzip
res1: (scala.collection.immutable.Iterable[Int], scala.collection.immutable.Iterable[Int]) = (List(1, 3),List(2, 4))
// Of course sets result in sets and duplicates are collected to a single element
scala> Set((1,2),(2,2)).unzip
res7: (scala.collection.immutable.Set[Int], scala.collection.immutable.Set[Int]) = (Set(1, 2),Set(2))
/*
Arbitrary elements can be unziped if a method is provided to decompose each element
*/
scala> List("one word", "another word").unzip {e => (e takeWhile {_ != ' '}, e dropWhile {_ != ' '})} 
res6: (List[String], List[String]) = (List(one, another),List( word,  word))
/*
The following shows the same function 
applied with map.  It results in a single
 list of Tuples rather than two lists of single elements
 */
scala> List("one word", "another word").map {e => (e takeWhile {_ != ' '}, e dropWhile {_ != ' '})}  
res8: List[(String, String)] = List((one, word), (another, word))

Multiple Argument Implicit Conversions

2010-03-12T04:04:00.000-08:00

Suppose you are creating a DSL and you want to implicitly convert 2 values to a particular object:
val v : SomeObject = (2, 3)
This is easily attained. In this example we will support the previous example as well as the standard 1 object to another implicit conversion.
val v : SomeObject = 2
Example:

// first lets define a class
scala> case class Randomly(x : Int, y : Double)
defined class Randomly
// the normal conversion
scala> implicit def intToRandomly(i : Int) = new Randomly(i,0.0)
intToRandomly: (i: Int)Randomly
/*
now a tuple for the other conversion.
Important:  The two conversions must have different names. At least that is the case in Scala 2.8
*/
scala> implicit def tupleToRandomly(i : (Int, Double)) = new Randomly(i._1, i._2)
tupleToRandomly: (i: (Int, Double))Randomly
 
scala> val r1 : Randomly = 4                                                     
r1: Randomly = Randomly(4,0.0)
scala> val r2 : Randomly = (4, 6.0)
r2: Randomly = Randomly(4,6.0)
/*
Suppose you want to do
val r : Randomly = (4,4)
you might think to implicitly convert from in to double
*/
scala> implicit def intToDouble(i : Int) = i.toDouble
intToDouble: (i: Int)Double
// implicit chaining is not permitted
scala> val r3 : Randomly = (4, 6)                    
< console>:10: error: type mismatch;
 found   : (Int, Int)
 required: Randomly
       val r3 : Randomly = (4, 6)
// Here is the legal option
scala> implicit def intTupleToRandomly(t: (Int,Int)) = new Randomly(t._1,t._2.toDouble) 
intTupleToRandomly: (t: (Int, Int))Randomly
scala> val r3 : Randomly = (4, 6)                                                      
r3: Randomly = Randomly(4,6.0)

How to reverse a map

2010-03-10T07:15:00.000-08:00

Suppose you wish to take a map and swap the keys with values. The stackoverflow question Elegant way to revers a map in scala offers some good suggestions

scala> val nodupes = Map(1 -> "a", 2-> "b", 3 -> "c")
nodupes: scala.collection.immutable.Map[Int,java.lang.String] = Map((1,a), (2,b), (3,c))
// Scala 2.8+
scala> nodupes map {_.swap}                          
res4: scala.collection.immutable.Map[java.lang.String,Int] = Map((a,1), (b,2), (c,3))
// Scala 2.7
scala> Map() ++ (nodupes map {case (k,v) => (v,k)})  
res5: scala.collection.immutable.Map[java.lang.String,Int] = Map((a,1), (b,2), (c,3))
// watch out if the values have duplicates you will loose information:
scala> val dupes = Map(1 -> "a", 2-> "b", 3 -> "b")  
dupes: scala.collection.immutable.Map[Int,java.lang.String] = Map((1,a), (2,b), (3,b))
scala> dupes map {_.swap}                          
res6: scala.collection.immutable.Map[java.lang.String,Int] = Map((a,1), (b,3))
// a way to not loose any data
scala> dupes groupBy {_._2} map {case (key,value) => (key, value.unzip._1)}     
res12: scala.collection.Map[java.lang.String,scala.collection.immutable.Iterable[Int]] = Map((a,List(1)), (b,List(2, 3)))
// I wanted to do the following for performance:
scala> dupes.view groupBy {_._2} map {case (key,value) => (key, value.unzip._1)}
java.lang.UnsupportedOperationException: IterableView((1,a), (2,b), (3,b)).newBuilder
 at scala.collection.TraversableViewLike$class.newBuilder(TraversableViewLike.scala:40)
 at scala.collection.IterableLike$$anon$1.newBuilder(IterableLike.scala:363)
 at scala.collection.TraversableLike$$anonfun$groupBy$1.apply(TraversableLike.scala:370)
// but as you can see a view cannot yet be grouped.  Perhaps in the future.

Lazy man's random test data

2010-03-08T00:02:00.000-08:00

A quick tip for generating some random testdata.

Note: This is a poor man's solution to using ScalaCheck. If you can handle the dependency I would really recommend using that library.

scala> object Options extends Enumeration {                       
     | val ONE, TWO, THREE, FOUR = Value                          
     | }
defined module Options
/*
Randomly select zero or more elements from the options enumeration
*/
scala> Options.values.filter {_ => util.Random.nextBoolean} mkString ", "
res2: String = TWO, FOUR
/*
Select a random string.  
Warning:  there is no restriction on the characters so control characters are likely
*/
scala> util.Random.nextString(10)
res5: String = ??????????
/*
ASCII string is oftern more useful for test data.  This selects a random string up to 13 characters long
*/
scala> util.Random.nextASCIIString(13)
res6: java.lang.String = RVPD\#_HqJ8:o
/*
This creates a sequence of 10 random strings
*/
scala> 1 to 10 map {_ => util.Random.nextASCIIString(13)}
res7: scala.collection.immutable.IndexedSeq[java.lang.String] = IndexedSeq(;E8|Q8H8RI;Q=, vM-X;"ksBr\:c, SKyz{uXNQ5E]X, =Jd8_ll08)s%e, gRCs)6wj%C-YF, `x;2Zru?l*c%@, XE*/Rx9:qPfpm, s|u,e.un+-Xm(, M,TpX9Dq-6$+^, w;exER&#0|}Ya)

Blocks within if statements

2010-03-06T05:08:00.000-08:00

his is another topic that examines the consistency of Scala. This topic examines blocks in if statements. It is related to Blocks within for comprehensions and Temporary Variables during object creation.

// standard if
if(1 > 2) -1 else 0
// since blocks return a value you can use a block within the if statement 
// (not sure when you would want to but...)
if ({ val x = 1
      val y = 2
      x == y }) 1 else 2

Zip with a constant value

2010-03-04T06:51:00.001-08:00

A simple tip for zipping a List (or other collection) with a single value.

scala> Stream.continually("h") zip List(1,2,3,4)
res2: scala.collection.immutable.Stream[(java.lang.String, Int)] = Stream((h,1), ?)
scala> res2 mkString ","
res3: String = (h,1),(h,2),(h,3),(h,4)
scala> List(1,2,3,4) zip Stream.continually("h")
res4: List[(Int, java.lang.String)] = List((1,h), (2,h), (3,h), (4,h))

Functions using case statements

2010-03-03T04:46:00.001-08:00

A further tip regarding using case statements to construct functions. If a case statement is assigned to a Function it will construct a Function object not a PartialFunction.

I suppose the question is why do you care about this since PartialFunction is a Function. The fact is that a PartialFunction is a Function1. But using a case statement you can construct a Function4 very easily.

scala> def run(f : Function1[Any,Unit]) = println(f.isInstanceOf[PartialFunction[_,_]])
run: (f: (Any) => Unit)Unit
/*
since run expects a Function calling run as shown here will make a 
Function object not a PartialFunction Object
*/
scala> run({case f => ()}) 
false
scala> def pf(f : PartialFunction[Any,Unit]) = println(f.isInstanceOf[PartialFunction[_,_]]) 
pf: (f: PartialFunction[Any,Unit])Unit
// Now a PartialFunction will be created
scala> pf({case f => ()})                                                                         
true
scala> def run(f : Function2[Int,String,Unit]) = f(1,"2")                                     
run: (f: (Int, String) => Unit)Unit
/*
This demonstrates why it is important that a case creates a Function
when assigned to a Function. PartialFunctions are Function1 objects
but the following statement is creating a Function2 object.
*/
scala> run({                 
     | case (1,b) => println(b)
     | case (a,b) => println(a,b)
     | })
2

Methods on PartialFunction

2010-03-02T00:04:00.000-08:00

This topic inspects the methods defined in the PartialFunction Object.

scala> type PF = PartialFunction[Int,Int]
defined type alias PF
// the two partial functions that we will use for the examples
scala> val pf1 : PF = {case 1 => 2}                      
pf1: PF = < function1>
scala> val pf2 : PF = {case 2 => 3}                      
pf2: PF = < function1>
/*
As is well known, when a PartialFunction is called with a value
it must be defined at that value or bad things will happen
*/
scala> pf1 isDefinedAt 1  
res14: Boolean = true
scala> pf1 isDefinedAt 2
res15: Boolean = false
scala> pf1(2)
scala.MatchError: 2
 at $anonfun$1.apply(< console>:5)
 at $anonfun$1.apply(< console>:5)
 at .< init>(< console>:7)
    ...
scala> pf1(1)
res5: Int = 2
/*
It is possible to compose two partial functions so first one partialFunction is called and then the next
*/
scala> (pf1 andThen pf2) isDefinedAt 2 
res16: Boolean = false
scala> (pf1 andThen pf2) isDefinedAt 1
res17: Boolean = true
scala> (pf1 andThen pf2)(2)
scala.MatchError: 2
 at $anonfun$1.apply(< console>:5)
 at $anonfun$1.apply(< console>:5)
 at scala.PartialFunction$$anon$2.apply(PartialFunction.scala:59)
 at .< init>(< console>:8)
    ...
scala> (pf1 andThen pf2)(1)
res8: Int = 3
/*
An alternative way of combining PartialFunctions is to 'or' them
*/
scala> (pf1 orElse pf2) isDefinedAt 1
res18: Boolean = true
scala> (pf1 orElse pf2) isDefinedAt 2
res19: Boolean = true
scala> (pf1 orElse pf2) isDefinedAt 3
res20: Boolean = false
scala> (pf1 orElse pf2)(1) 
res9: Int = 2
scala> (pf1 orElse pf2)(2)
res10: Int = 3
/*
Finally a PartialFunction can be easily converted to a function that returns 
an Option
*/
scala> pf1.lift
res21: (Int) => Option[Int] = < function1>
scala> pf1.lift(1)
res11: Option[Int] = Some(2)
scala> pf1.lift(2)
res12: Option[Int] = None

NullPointer when mixed traits (Warning)

2010-03-01T00:15:00.000-08:00

This tip is mainly to document a 'GOTCHA' that I got caught by recently. It basically goes like this:

Trait Y extends(or has self-type) X. Trait X defines some abstract method 'm'. The initialization code in Y accesses 'm'. Creation of an object new X with Y results in: *Boom* NullPointerException (on object creation).

The example in code:

scala> trait X { val x : java.io.File }
defined trait X
scala> trait Y {self : X => ; val y = x.getName} 
defined trait Y
scala> new X with Y { val x = new java.io.File("hi")}
java.lang.NullPointerException
 at Y$class.$init$(< console>:5)
 at $anon$1.< init>(< console>:7)
 ...

At a glance it seems that x should override the abstract value x in trait X. However the order in which traits are declared is important. In this case first Y is configured then X. Since X is not yet configured Y throws an exception. There are several ways to work around this.
Option 1:

trait X {val x : java.io.File}
trait Y {self : X => ; val y = x.getName}
/*
Declaring Y with X will work because Y is initialized after X
but remember that there may
be other reasons that X with Y is required.  
Method resolution is one such reason
*/
new Y with X { val x = new java.io.File("hi")}

Option 2:

trait X { val x : java.io.File }
trait Y {self : X => ; def y = x.getName}
/*
Since method y is a 'def' x.getName will not be executed during initialization.
*/
scala> new X with Y { val x = new java.io.File("hi")}
res10: java.lang.Object with X with Y = $anon$1@7cb9e9a3

Option 3:

trait X { val x : java.io.File }
trait Y {self : X => ; lazy val y = x.getName}
/*
'lazy val' works for the same reason 'def' works: x.getName is not invoked during initialization
*/
scala> new X with Y { val x = new java.io.File("hi")}
res10: java.lang.Object with X with Y = $anon$1@7cb9e9a3

Option 4:

trait X {val x : java.io.File }
trait Y extends X {def y = x.getName}
/*
if Y extends X then a new Y can be instantiated
*/
new Y {val x = new java.io.File("hi")}

Two more warnings. First, the same error will occur whether 'x' is a def or a val or a var.

trait X { def x : java.io.File }   
trait Y {self : X => ; val y = x.getName}     
new X with Y { val x = new java.io.File("hi")}

Second warning: In complex domain models it is easy to have a case where Y extends X but the final object is created as: new X with Y{...}.

You will get the same error here because (I think) the compiler recognized that Y is being mixed in with X and therefore the X will be initialized as after Y instead of before Y.

First the code:

trait X { def x : java.io.File }   
trait Y extends X { val y = x.getName}        
new X with Y { val x = new java.io.File("hi")}

If the code instantiated new Y{...} the initialization would be X then Y. Because X can only be initialized once, the explicit declaration of new X with Y forces Y to be initialized before X. (X can only be initialized once even when it appears twice in the hierarchy).

This is a topic called linearization and will be addressed in the future.

Methods of PartialFunction object

2010-02-22T04:00:00.001-08:00

This topic looks at the functionality provide in the object PartialFunction.

This example uses Scala 2.8 which I believe is required.

Only two methods are focused on. PartialFunction.cond and PartialFunction.condOpt.
As per request the signature for said methods are:

def cond[T](x: T)(pf: PartialFunction[T, Boolean]): Boolean
def condOpt[T, U](x: T)(pf: PartialFunction[T, U]): Option[U]

Now for some examples:

scala>  import PartialFunction._
import PartialFunction._
scala> def strangeConditional(other: Any): Boolean = cond(other) {
     |  case x: String if x == "abc" || x == "def"  => true
     |  case x: Int => true
     | }
strangeConditional: (other: Any)Boolean
scala> strangeConditional("abc")
res0: Boolean = true
scala> strangeConditional("hello")
res1: Boolean = false
scala> strangeConditional(3)      
res2: Boolean = true
scala> def onlyInt(v: Any): Option[Int] = condOpt(v) { case x: Int => x }
onlyInt: (v: Any)Option[Int]
scala> onlyInt("hi")
res3: Option[Int] = None
scala> onlyInt(3)
res4: Option[Int] = Some(3)
scala> import util.control.Exception.catching
import util.control.Exception.catching
scala> object IntNum { 
     |  val number = catching(classOf[NumberFormatException])
     |  def unapply(x : Any) = condOpt(x) {
     |      case x : Int => x
     |      case y : String if number.opt(y.toInt).isDefined => y.toInt
     |    }
     | }
defined module IntNum
scala> 1 match { case IntNum(x) => println(x+" i win!")}   
1 i win!
scala> 3 match { case IntNum(x) => println(x+" i win!")}
3 i win!

=>? type alias for PartialFunction

2010-02-19T02:03:00.000-08:00

A big thanks to Ben Jackman for this tip.

His tip cleans up the examples in post Chaining Partial Functions with orElse
The secret is to define the type alias type =>?[-A, +B] = PartialFunction[A, B]. This alias may be added to Predef in the future but until it is you can add it yourself.

scala> type =>?[-A, +B] = PartialFunction[A, B]
defined type alias $eq$greater$qmark
scala> val i : (Any =>? Unit) = {case x:Int => println("int found")}
i: =>?[Any,Unit] = < function1>
scala> val j : (Any =>? Unit) = {case x:Double => println("Double found")}
j: =>?[Any,Unit] = < function1>
scala> val * : (Any =>? Unit) = {case x=> println("Something else found")}
*: =>?[Any,Unit] = < function1>
scala> (i orElse j orElse *)(1)
int found
scala> (i orElse j orElse *)(1.0)
Double found
scala> (i orElse j orElse *)(true)
Something else found
scala> def =>?[A, B](id : A =>? B) = id
$eq$greater$qmark: [A,B](id: =>?[A,B])=>?[A,B]
scala> ( =>?[Any, Unit]{case s : String => println("String found")} orElse j orElse *)("hello")
String found

Chaining Partial Functions with orElse

2010-02-18T00:42:00.001-08:00

PartialFunctions are extremely valuable Scala constructs that are used in many APIs. Commonly you will encounter the pattern:

obj match {
case "Val" => // do something
case _ => // handle all other cases
}

It is intuitive and obvious how to share the code of the right hand side if the case statement by factoring that code out to a method. But would it not be useful to be able to factor out an entire case statement (PartialFunction) and later chain them together as needed?

This is indeed possible and very easy to do:

/*
We need to declare Partial Functions so to add brevity I am adding this alias
*/
scala> import scala.{PartialFunction => PF}
import scala.{PartialFunction=>PF}
/*
You have to explicitly declare the type because the type inferencer cannot know what type of PartialFunction to create
 
A PartialFunction is Strictly type so some functions can only be used on Ints for example
*/
scala> val i : PF[Any, Unit] = {case x:Int => println("int found")}
i: PartialFunction[Any,Unit] = < function1>
scala> val j : PF[Any, Unit] =  {case x:Double => println("Double found")}
j: PartialFunction[Any,Unit] = < function1>
scala> val * : PF[Any, Unit] =  {case x=> println("Something else found")}
*: PartialFunction[Any,Unit] = < function1>
/*
one might think that you can do:
1 match (i orElse j orElse *)
but in fact (i orElse j orElse *) forms a PartialFunction not a pattern so cannot be used with match.  Instead it must be used as a function
*/
scala> (i orElse j orElse *)(1)
int found
scala> (i orElse j orElse *)(1.0)
Double found
scala> (i orElse j orElse *)(true)
Something else found
/*
for specific cases it is possible to chain the an anonymous partial function with the common function
This is not so nice so it is probably best to declare a val instead of inline like this
*/
scala> (({case s:String => println("String found")}:PF[Any,Unit]) orElse j orElse *)("hello")
String found

For another example of chaining PartialFunctions the Akka tutorial has a good example in the ChatServer trait: http://jonasboner.com/2010/01/04/introducing-akka.html

And Case Statements

2010-02-16T01:03:00.001-08:00

Recently I encountered a good question on Stack Overflow about matching.
http://stackoverflow.com/questions/2261358/pattern-matching-with-conjunctions-patterna-and-patternb.

As mentioned in an earlier post Matching with Or case expressions suppose 'or' expression combination using the '|' character. However 'and' combinations are not possible.

One solution is to build an && extractor object as follows:

scala> case object && {  def unapply[A](a: A) = Some((a, a))}
defined module $amp$amp
scala> object StartsWith {  def unapply(s: String) = s.headOption}
defined module StartsWith
scala> object EndsWith {  def unapply(s: String) = s.reverse.headOption}
defined module EndsWith
scala> "foo" match {  case StartsWith('f') && EndsWith('o') => println("f*o") }
f*o

Note: this is a scala 2.7 solution Scala 2.8 can be used to improve the EndsWith extractor by using the method lastOption instead of s.reverse.headOption.

Self Annotation vs inheritance

2010-02-15T04:09:00.000-08:00

At first glance the "sself annotation" declaration seems similar to extending another class. (For a look at self annotations read the topic: Self Type.) They are completely different but the comparison is understandable since both of them provide access to the functionality of referenced class.

For example both of the following compile:

class Base {
  def magic = "bibbity bobbity boo!!"
}
trait Extender extends Base {
  def myMethod = "I can "+magic
}
trait SelfTyper {
  self : Base => 
  
  def myMethod = "I can "+magic
}

But the two are completely different. Extender can be mixed in with any class and adds both the "magic" and "myMethod" to the class it is mixed with. SelfType can only be mixed in with a class that extends Base and SelfTyper only adds the method "myMethod" NOT "magic".

Why is the "self annotations" useful? Because it allows several provides a way of declaring dependencies. One can think of the self annotation declaration as the phrase "I am useable with" or "I require a".

The following example demonstrates one possible reason to use self annotations instead of extend.

Note: These examples can be pasted into the REPL but I have shown that here because it would make the examples too long.

import java.io._
import java.util.{Properties => JProps}
trait Properties {
  def apply(key:String) : String
}
trait XmlProperties extends Properties {
  import scala.xml._
  
  def xml(key:String) = Elem(null,key,Null,TopScope, Text(apply(key)))
}
trait JSonProperties extends Properties {
  def json(key:String) : String = "%s : %s".format(key, apply(key))
}
trait StreamProperties {
  self : Properties => 
  
  protected def source : InputStream
  
  private val props = new JProps()
  props.load(source)
  
  def apply(key:String) = props.get(key).asInstanceOf[String]
}
trait MapProperties {
  self : Properties =>
  
  protected def source : Map[String,String]
  def apply(key:String) = source.apply(key)
}
val sampleMap = Map("1" -> "one", "2" -> "two", "3" -> "three")
val sampleData = """1=one
                    2=two
                    3=three"""
val sXml = new XmlProperties() with StreamProperties{
              def source = new ByteArrayInputStream(sampleData.getBytes)
           }
val mXml = new XmlProperties() with MapProperties{
              def source = sampleMap
           }
val sJSon = new JSonProperties() with StreamProperties{
              def source = new ByteArrayInputStream(sampleData.getBytes)
            }
val mJSon = new JSonProperties() with MapProperties{
              def source = sampleMap
            }
sXml.xml("1")
mXml.xml("2")
sJSon.json("1")
mJSon.json("2")

The justification for using self annotations here is flexibility. A couple other solutions would be

Use subclassing - this is poor solution because there would be an explosion of classes. Instead of having 5 traits you would need 7 traits. Properties, XmlProperties, JSonProperties, XmlStreamProperties, XmlMapProperties, JsonStreamProperties and JsonMapProperties. And if you later wanted to add a new type of properties or a new source like reading from a database then you need 2 new subclasses.
Composition - Another strategy is to use construct the XmlProperties with a strategy that reads from the source. This is essentially the same mechanism except that you need to build and maintain the the dependencies. It also makes layering more difficult. For example:
trait IterableXmlProperties { self : MapProperties with XmlProperties => def xmlIterable = source.keySet map {xml _} } new XmlProperties with MapProperties with IterableXmlProperties {def source = sampleMap}

The next question that comes to mind is why use extends then if self annotation is so flexible? My answer (and I welcome discussion on this point) has three points.

The first is of semantics and modeling. When designing a model it is often more logical to use inheritance because of the semantics that comes with inheriting from another object.
Another argument is pragmatism.
Imagine the collections library where there is no inheritance. If you wanted a map with Iterable functionality you would have to always declare Traversable with Iterable with Map (and this is greatly simplified). That declaration would have to be used for virtually all methods that require both the Iterable and Map functionality. To say that is impractical is an understatement. Also the semantics of Map is changed from what it is now. The trait Map currently includes the concept of Iterable.
The last point is sealed traits/classes. When a trait is "sealed" all of its subclasses are declared within the same file and that makes the set of subclasses finite which allows certain compiler checks. This (as far as I know) cannot be done with self annotations.

Specs and Fixtures/Contexts

2010-02-12T04:07:00.001-08:00

This topic revisits the Specs BDD testing library. It is a continuation of the previous post Specs BDD Testing Framework.

This topic will look at how to setup fixtures in Specs. This is only a sample to give a feeling of Specs a much more complete guide is available on the Specs website.

jeichar: git-src$ scala -classpath ~/.m2/repository/org/scala-tools/testing/specs/1.6.1/specs-1.6.1.jar
scala> import org.specs._
import org.specs._
/*
This example demonstrates before and after actions similar to what is found in XUnit.
*/
scala> object mySpec extends Specification {           
     | "my system" should {
     | doFirst{println("before")}  // ran once
     | 
     | doBefore{println("before")} // ran for each test
     | doAfter{println("after")} // ran for each test
     | 
     | "test spec 1" in { println("test1"); 1 must_== 1}
     | "test spec 2" in { println("test2"); 1 must_== 2}
     | 
     | doLast{println("last")} // ran once
     | }}
defined module mySpec
scala> mySpec.main(Array())                             
Specification "mySpec"
  my system should
before
before
test1
after
  + test spec 1
before
test2
after
last
  x test spec 2
    '1' is not equal to '2' (< console>:14)
Total for specification "mySpec":
Finished in 0 second, 307 ms
2 examples, 2 expectations, 1 failure, 0 error

Example using Contexts there many more examples at: Shared contexts

scala> import org.specs._
import org.specs._
/*
This specification uses contexts instead of before and after actions.
My personal preference is to use contexts because they are more flexible and can have names associated with them.  In addition contexts can be shared between specifications and multiple contexts can be used within a single specification.  This example is kept very simple for demonstration purposes
*/
scala> object StackSpecification extends Specification {
     |   var list : List[Int] = _
     |   val empty = beforeContext(list = Nil)  
     |   val nonEmpty = beforeContext(list = List(1,2,3))
     |   
     |   "A full stack" definedAs nonEmpty should { 
     |     "size of 3" in {
     |       list must haveSize (3)
     |     }
     |   }
     |   "A stack" when empty should { 
     |     "is empty" in {
     |       list must beEmpty
     |     }
     |   }
     | }
defined module StackSpecification
scala> StackSpecification.main(Array())
Specification "StackSpecification"
  A full stack should
  + size of 3
  Total for SUS "A full stack":
  Finished in 0 second, 42 ms
  1 example, 1 expectation, 0 failure, 0 error
  A stack should
  + is empty
  Total for SUS "A stack":
  Finished in 0 second, 4 ms
  1 example, 1 expectation, 0 failure, 0 error
Total for specification "StackSpecification":
Finished in 0 second, 85 ms
2 examples, 2 expectations, 0 failure, 0 error

Selection with Implicit Methods

2010-02-11T00:06:00.000-08:00

This tip is a cool tip based on Daniel's Use Implicits to Select Types post. That post goes into more detail.

The idea is that depending on a given parameter of type T a particular type of object is required. There are several ways to do this. One would be to use matching to match the type T and create the correct object. Most likely the biggest draw back to matching is caused by type erasure. The following solution gets around that issue.

Two very interesting points.

Implicit methods do not require parameters. They can be selected based only on type parameters
Implicit methods are not subject to type erasure

// the base class we need
scala> abstract class X[T] { def id :Unit }
defined class X
/*
One of the types we need created.  It is the catch all case
*/
scala> implicit def a[T] =new X[T] { def id =println("generic") }
a: [T]X[T]
/*
A specific subclass for integers
*/
scala> implicit def b =new X[Int] { def id =println("Int") }
b: X[Int]
/*
One simple usage.  The most specific implicit will be used to 
create the object to be passed to g.  
*/
scala> def f[T](a :T)(implicit g :X[T]) = g.id
f: [T](a: T)(implicit g: X[T])Unit
scala> f(5)
Int
scala> f('c')
generic
/*
This example demonstrates how erasure is not an issue because 
the selection of the implicit is done at compile time so 
the correct type is selected.  If a match was used instead 
then a more complicated solution would be required
*/
scala> def g[T](l:List[T])(implicit i:X[T]) = i.id          
g: [T](l: List[T])(implicit i: X[T])Unit
scala> g(List(1,2,3))
Int
scala> g(List("a",2,3))
generic

Structural Types: Multiple Methods and Type Aliasing

2010-02-09T02:01:00.000-08:00

There are two more aspects related to structural typing that are useful to look at. Structural types with multiple methods and type aliases.

For background on this topic also look at:

Structural Types are not limited to defining a single method. In that regard they are very similar to interfaces without the binary incompatibility issues. However do not be fooled into thinking they are the same thing. For one reason reflection is used so performance can be an issue in certain cases and also interfaces/traits have semantics that structural types do not.

/*
Defining a types that has both a length and charAt method.  
Just a warning.  If you leave off the () after length this will not work.  This is not a bug.  Martin kindly left a comment on why it is not.
*/
scala> def foreach(t : {def length():Int; def charAt(i:Int):Char}, f : Char => Unit) = {
     | 0 until t.length foreach {i => f(t.charAt(i))}  
     | }
foreach: (t: AnyRef{def length(): Int; def charAt(i: Int): Char},f: (Char) => Unit)Unit
// A string matches the structural type
scala> foreach ("hello", println _)
h
e
l
l
o

Pretty unexpected I would say. A feature of Scala which complements Structural types are type aliases. They are useful in many situations and one is with use with Structural Types:

/*
With type aliasing you can assign a structural type a name
*/
scala> type IChar = {def length():Int
     |               def charAt(i:Int):Char}
defined type alias IChar
scala> def print( t : IChar) = 0 until t.length() foreach {i => println(t.charAt(i))}
print: (t: IChar)Unit
scala> print("gurk")
g
u
r
k

Dynamic calls using Structural Types

2010-02-08T01:10:00.001-08:00

Using reflection can be a real pain in Java since the API is a Java API and consists of many gets and searches through collections not to mention so many exceptions that need to be handled. In Scala there is a wonderful way to clean up a reflective call down to a single line (assuming you don't want to worry about handling exceptions.) Here structural typing can really be a pleasure.

// I am assigning a string to an Any reference
scala> val s:Any = "hello :D"
s: Any = hello :D
// Any does not have a length method
scala> s.length
< console>:6: error: value length is not a member of Any
       s.length
         ^
/*
But I can cast it to a structural type with a length method
*/
scala> s.asInstanceOf[{def length:Int}].length
res2: Int = 8

There are restrictions to this. For example implicits will not work:

/*
The method r is part of StringLike (or RichString in Scala 2.7) 
and there is an implicit conversion from String to RichString/StringLike.
The structural type does not try to apply the implicits because implicits are a
compile time artifact and that information is not kept at run time.  Perhaps this
will be added in the future but it is not present now.
*/
scala> s.asInstanceOf[{def r:util.matching.Regex}].r
java.lang.NoSuchMethodException: java.lang.String.r()

More examples:

scala> val s:Any = "hello :D"
s: Any = hello :D
scala> s.asInstanceOf[{def charAt(x:Int):Char}].charAt(2)
res9: Char = l
/*
This is interesting.  The actual method returns Char, but Char is
compatible with Any so this cast works.
*/
scala> s.asInstanceOf[{def charAt(x:Int):Any}].charAt(2) 
res10: Any = l
// Now lets test to see if that lenience is available for parameters:
scala> class X 
defined class X
scala> class Y extends X
defined class Y
scala> class T {
     | def x(x:X):Int = 1
     | }
defined class T
scala> val a:Any = new T()
a: Any = T@687c3b99
scala> a.asInstanceOf[{def x(x:X):Any}].x(new Y())
res11: Any = 1
/*
Ok so return values can be subclasses but not params
*/
scala> a.asInstanceOf[{def x(x:Y):Any}].x(new Y())
java.lang.NoSuchMethodException: T.x(Y)
 at java.lang.Class.getMethod(Class.ja

Use with care :-D

Introducing Structural Types

2010-02-05T00:05:00.000-08:00

Structural types allows one to declare types based on the methods the type has. For example you could define a method that takes a class containing a close method. This is fairy analogous to duck-typing in dynamic languages. Except that it is statically enforced.

The main example used here was from a comment on the Code Monkeyism Blog. The commenter further explains that this example is in fact from Beginning Scala chapter 4 (which I would like to read but have not yet had the time.)

/*
A can be any object that has a close method.  
This is statically typed which makes some restrictions which are explained later
*/
scala> def using[A <: {def close(): Unit}, B](param: A)(f: A => B): B = 
     | try {
     |     f(param)
     |   } finally {
     |     try {
     |        param.close()
     |     } catch { case _ => () }
     |   }
using: [A <: AnyRef{def close(): Unit},B](param: => A)(f: (A) => B)B
scala> using(new java.io.ByteArrayInputStream("hello world".getBytes)){ in => 
     | io.Source.fromInputStream(in) foreach (print)                          
     | }
hello world
scala> using(new java.io.ByteArrayInputStream("hello world".getBytes)){ in => 
     | io.Source.fromInputStream(in) mkString ""                              
     | }
res8: String = hello world

That is extremely powerful and the consequences will be visited more in the future. But because structural typing is statically enforced it is not quite as flexible as dynamic language's version of duck typing. For example you cannot do:

scala> val i:Any = new java.io.ByteArrayInputStream("hello world".getBytes)
i: Any = java.io.ByteArrayInputStream@145a25f3
/*
This does not work because 'i' is defined as type Any.  Not a type that is closeable.  
Casting can be used to get around this issue.  I will address that in its own post
*/
scala> using(i){ in => 
     | io.Source.fromInputStream(in) mkString ""
     | }
< console>:8: error: inferred type arguments [Any,B] do not conform to method using's type parameter bounds [A <: AnyRef{def close(): Unit},B]
       using(i){ in =>

An alternative to the first using example is to use call by name to construct the closeable. The reason one might want to do that is because it allows currying of the method:

scala> def using[A <: {def close(): Unit}, B](param: =>A)(f: A => B): B = 
     | val closeable = param  // notice param is only referenced once
     | try {
     |     f(closeable)
     |   } finally {
     |     try {
     |        closeable.close()
     |     } catch { case _ => () }
     |   }
using: [A <: AnyRef{def close(): Unit},B](param: => A)(f: (A) => B)B
/*
if this was accessing a database a new connection would be made automatically each time this function was used
*/
scala> val usingTheWorld = using[BStream,Int](new java.io.ByteArrayInputStream("hello world".getBytes))_
usingTheWorld: ((java.io.ByteArrayInputStream) => Int) => Int = < function1>
scala> usingTheWorld { s => io.Source.fromInputStream(s).length}  
res3: Int = 11
scala> usingTheWorld { s => io.Source.fromInputStream(s).getLines().length}
res5: Int = 0

Regex ReplaceAllIn

2010-02-03T23:54:00.001-08:00

Note: Updated on Feb 13th for the newer API on Scala 2.8 trunk. (This is life on the bleeding edge, thanks Daniel).

A couple new methods have just been added to Scala 2.8 Regex. You will need to download a version of Scala 2.8 more recent than Scala2.8-Beta1.

The methods are related to replacing text using a regular expression and to say they are useful is an understatement. Lets take a look:

scala> val quote = """I don't like to commit myself about heaven and hell - you see, I have friends in both places. 
     | Mark Twain"""                                                                                                
quote: java.lang.String = 
I don't like to commit myself about heaven and hell - you see, I have friends in both places. 
Mark Twain
scala> val expr = "e".r    
expr: scala.util.matching.Regex = e
/* 
This first method is not new or is it interesting.  But the new methods are both related
so lets start with the basic form of replaceAllIn
*/
scala> expr.replaceAllIn(quote, "**")
res1: String = 
I don't lik** to commit mys**lf about h**av**n and h**ll - you s****, I hav** fri**nds in both plac**s. 
Mark Twain
// this does the same thing
scala> quote.replaceAll("e","**")
res2: java.lang.String = 
I don't lik** to commit mys**lf about h**av**n and h**ll - you s****, I hav** fri**nds in both plac**s. 
Mark Twain
/*
Now things get interesting.  Using this form of replaceAllIn we can determine the replacement on a case by case basis.
It provides the Match object as the parameter so you have complete access to all 
the matched groups, the location of the match etc...
The method takes a Match => String function.  Very, very powerful.
*/
scala> expr.replaceAllIn(quote, s => if(util.Random.nextBoolean) "?" else "*")
res5: String = 
I don't lik? to commit mys?lf about h?av?n and h?ll - you s*?, I hav? fri*nds in both plac*s. 
Mark Twain
/*
Another example using some of the matcher functionality
*/
scala> expr.replaceAllIn(quote, m => m.start.toString)                        
res6: String = 
I don't lik11 to commit mys26lf about h37av40n and h48ll - you s5960, I hav68 fri73nds in both plac90s. 
Mark Twain
/*
Another crazy useful method is the replaceSomeIn.  It is similar to the replaceAllIn that takes a function except that the function in replaceSomeIn returns an Option.  If None then there is no replacement.  Otherwise a replacement is performed.  Very nice when dealing with complex regular expressions.
In this example we are replacing all 'e's start are before the 50th character in the string with -
*/
scala> expr.replaceSomeIn(quote, m => if(m.start > 50) None else Some("-"))
res3: String = 
I don't lik- to commit mys-lf about h-av-n and h-ll - you see, I have friends in both places.
Mark Twain

Chaining Options with orElse

2010-02-03T14:25:00.000-08:00

A simple but handy use for Option is to select the first valid option from a selection of possible choices. Sound vague? Well it is because it can be used in many different situations. The one presented here is: the program needs a directory that can be set by the user either as a system variable, and environment variable or the default value. The java code is a nightmare of if (xxx == null) statements. The Scala code is beautiful.

scala> val propsTemplates = Option(System getProperty "MVN_CREATOR_TEMPLATES")
propsTemplates: Option[java.lang.String] = None
scala> val envTemplates = Option(System getenv "MVN_CREATOR_TEMPLATES")
envTemplates: Option[java.lang.String] = None
scala> val defaultHome = Some(System getProperty "user.home")
defaultHome: Some[java.lang.String] = Some(/Users/jeichar)
/* 
chain the different options together in order of priority and get the value
I am taking a small liberty here because I am assuming that user.home is always available
*/
scala> propsTemplates.orElse(envTemplates).orElse(defaultHome).get
res0: java.lang.String = /Users/jeichar
// alternative form:
scala> propsTemplates orElse envTemplates orElse defaultHome get
res1: java.lang.String = /Users/jeichar

Blocks within for-comprehensions

2010-02-02T03:59:00.000-08:00

In another example of uniformity it is possible to use blocks within a for-comprehension when creating a generator or making an assignment. The basic form of a for-comprehension is

for {i <- generator 
     if guard
     j = assignment } yield result

The generator, guard, assignment and result are all expressions which means they can all be blocks or simple statements. Most commonly you will see them as simple statements:

scala> for {i <- 1 to 10
     |      if i % 2 == 0
     |      j = i } yield j
res50: scala.collection.immutable.IndexedSeq[Int] = IndexedSeq(2, 4, 6, 8, 10)

But since they are expressions they can be more complex:

scala> for {  
     |  i <- { 
     |     val start = nextInt(3)
     |     val end = nextInt(10)+start
     |     start to end
     |  }
     |  if {
     |    val cut = nextInt(3)+1
     |    i % cut == 0
     |  }
     |  j = {
     |    val x = i+1
     |    x / 2
     |  }
     |  }  yield {
     |    // do a debug println
     |    println(j)
     |  j
     |  }
1
1
2
3
res53: scala.collection.immutable.IndexedSeq[Int] = IndexedSeq(1, 1, 2, 3)

Temporary variables during object instantiation

2010-02-01T02:04:00.000-08:00

In Java a common pattern with class constructors is to assign field values and often there are several intermediate values used for the calculation. If the code is ported to Scala the resulting class will have the intermediate values as fields, which take up space in the object. However the issue is easily worked around. Lets look at a couple examples.

Example 1: Assigning a single field

//  Java
import java.io.File
/** 
No real logic behind class.  But for some reason it needs the path of a tmp directory in the working directory
*/
class OneAssignment {
  final String field;
  public OneAssignment() {
    File file = new File("tmp");
    if(!file.exists()) {
      file.mkdirs();
    }
    field = file.getAbsolutePath();
  }
}

In Scala the naive way to port this would be:

//  Scala
import java.io.File
class OneAssignment {
  val file = new File("tmp")
  if(!file.exists()) {
    file.mkdirs()
  }
  val field = file.getAbsolutePath()
}

Problem is that it has an extra field "file" now. The correct way to port this would be as follows:

//  Scala
import java.io.File
class OneAssignment {
/* 
notice that assignment is in a block so file is only visible within the block
*/
  val field = {
    val file = new File("tmp")
    if(!file.exists()) {
      file.mkdirs()
    }
    file.getAbsolutePath()
  }
}

Example 2: Assigning multiple fields

//  Java
import java.io.File
/** 
Basically the same as last example but multiple fields are assigned
Notice that 2 fields depend on the temporary file variable but count does not
*/
class MultipleAssignments {
  final String tmp,mvn_repo;
  find int count;
  public OneAssignment() {
    File file = new File("tmp");
    if(!file.exists()) {
      file.mkdirs();
    }
    tmp = file.getAbsolutePath();
    count = file.listFiles.length;
    
    File home = new File(System.getProperty("user.home"));
    mvn_repo = new File(home, ".m2").getPath();
  }
}

The Scala port:

//  Scala
import java.io.File
class MultipleAssignments {
/*
When multiple fields depend on the same temporary variables the fields can be assigned together from one block by returning a tuple and using Scala's matching to expand the tuple during assignment.  See previous topics on assignment for details 
*/
  val (tmp,count) = {
    val file = new File("tmp");
    if(!file.exists()) {
      file.mkdirs();
    }
    val tmp = file.getAbsolutePath();
    val count = file.listFiles.length;
    (tmp, count)
  }
  val mvn_repo = {
    val home = new File(System.getProperty("user.home"));
    new File(home, ".m2").getPath();
  }
}

In some ways the Scala port is cleaner in that it splits the constructor up and decouples the dependencies between fields.