Regex ReplaceAllIn

Note: Updated on Feb 13th for the newer API on Scala 2.8 trunk. (This is life on the bleeding edge, thanks Daniel).

A couple new methods have just been added to Scala 2.8 Regex. You will need to download a version of Scala 2.8 more recent than Scala2.8-Beta1.

The methods are related to replacing text using a regular expression and to say they are useful is an understatement. Lets take a look:

1. scala> val quote = """I don't like to commit myself about heaven and hell - you see, I have friends in both places. 
2.      | Mark Twain"""                                                                                                
3. quote: java.lang.String = 
4. I don't like to commit myself about heaven and hell - you see, I have friends in both places. 
5. Mark Twain
7. scala> val expr = "e".r    
8. expr: scala.util.matching.Regex = e
10. /* 
11. This first method is not new or is it interesting.  But the new methods are both related
12. so lets start with the basic form of replaceAllIn
13. */
14. scala> expr.replaceAllIn(quote, "**")
15. res1: String = 
16. I don't lik** to commit mys**lf about h**av**n and h**ll - you s****, I hav** fri**nds in both plac**s. 
17. Mark Twain
19. // this does the same thing
20. scala> quote.replaceAll("e","**")
21. res2: java.lang.String = 
22. I don't lik** to commit mys**lf about h**av**n and h**ll - you s****, I hav** fri**nds in both plac**s. 
23. Mark Twain
25. /*
26. Now things get interesting.  Using this form of replaceAllIn we can determine the replacement on a case by case basis.
28. It provides the Match object as the parameter so you have complete access to all 
29. the matched groups, the location of the match etc...
31. The method takes a Match => String function.  Very, very powerful.
32. */
33. scala> expr.replaceAllIn(quote, s => if(util.Random.nextBoolean) "?" else "*")
34. res5: String = 
35. I don't lik? to commit mys?lf about h?av?n and h?ll - you s*?, I hav? fri*nds in both plac*s. 
36. Mark Twain
38. /*
39. Another example using some of the matcher functionality
40. */
41. scala> expr.replaceAllIn(quote, m => m.start.toString)                        
42. res6: String = 
43. I don't lik11 to commit mys26lf about h37av40n and h48ll - you s5960, I hav68 fri73nds in both plac90s. 
44. Mark Twain
46. /*
47. Another crazy useful method is the replaceSomeIn.  It is similar to the replaceAllIn that takes a function except that the function in replaceSomeIn returns an Option.  If None then there is no replacement.  Otherwise a replacement is performed.  Very nice when dealing with complex regular expressions.
49. In this example we are replacing all 'e's start are before the 50th character in the string with -
50. */
51. scala> expr.replaceSomeIn(quote, m => if(m.start > 50) None else Some("-"))
52. res3: String = 
53. I don't lik- to commit mys-lf about h-av-n and h-ll - you see, I have friends in both places.
54. Mark Twain

Regular Expression 3: Regex matching

This post covers basically the same things as Matching Regular Expressions but goes into a bit more detail. I recommend reading both posts since there is unique information in each.

The primary new item I show here is that more advanced matching techniques can be used but more importantly all groups are matched even groups that are within another group.

Note: The examples use Scala 2.8. Most examples will work with 2.7 but I believe the last example is Scala 2.8 only.

1. scala> val date = "11/01/2010"
2. date: java.lang.String = 11/01/2010
4. scala> val Date = """(\d\d)/(\d\d)/(\d\d\d\d)""".r
5. Date: scala.util.matching.Regex = (\d\d)/(\d\d)/(\d\d\d\d)
7. /*
8. When a Regex object is used in matching each group is assigned to a variable
9. */
10. scala> val Date(day, month, year) = date          
11. day: String = 11
12. month: String = 01
13. year: String = 2010
15. scala> val Date = """(\d\d)/((\d\d)/(\d\d\d\d))""".r
16. Date: scala.util.matching.Regex = (\d\d)/((\d\d)/(\d\d\d\d))
18. /*
19. This example demonstates how all groups must be assigned, if not there will be a matchError thrown
20. */
21. scala> val Date(day, monthYear, month, year) = date
22. day: String = 11
23. monthYear: String = 01/2010
24. month: String = 01
25. year: String = 2010
27. scala> val Date(day, month, year) = date           
28. scala.MatchError: 11/01/2010
29. at .< init>(< console>:5)
30. at .< clinit>(< console>)
33. // but placeholders work in Regex matching as well:
34. scala> val Date(day, _, month, year) = date
35. day: String = 11
36. month: String = 01
37. year: String = 2010
39. scala> val Names = """(\S+) (\S*)""".r             
40. Names: scala.util.matching.Regex = (\S+) (\S*)
42. scala> val Names(first, second) = "Jesse Eichar"
43. first: String = Jesse
44. second: String = Eichar
46. /*
47. If you want to use Regex's in assignment you must be sure the match will work.  Otherwise you should do real matching
48. */
49. scala> val Names(first, second) = "Jesse"       
50. scala.MatchError: Jesse
51. at .< init>(< console>:5)
52. at .< clinit>(< console>)
54. scala> val M = """\d{3}""".r
55. M: scala.util.matching.Regex = \d{3}
57. /*
58. There must be a group in the Regex or match will fail
59. */
60. scala> val M(m) = "Jan"
61. scala.MatchError: Jan
62. at .< init>(< console>:5)
63. at .< clinit>(< console>)

The following are a few more complex examples

1. scala> val Date = """((\d\d)/(\d\d)/(\d{4}))|((\w{3}) (\d\d),\s?(\d{4}))""".r         
2. Date: scala.util.matching.Regex = ((\d\d)/(\d\d)/(\d{4}))|((\w{3}) (\d\d),\s?(\d{4}))
4. /*
5. The Regex has an or in it.  So only 1/2 of the groups will be non-null.
6. If the first group is a String then it is non-null and the next three elements
7. the pattern will be day/month/year
9. Otherwise if the 5th group is a String then the patter will be month day, year
11. Lastly a catch all
12. */
13. scala> def printDate(date:String) = date match {                                      
14.      | case Date(_:String,day,month,year,_,_,_,_) => (day,month,year)                 
15.      | case Date(_,_,_,_,_:String,month,day,year) => (day,month,year) // process month
16.      | case _ => ("x","x","x")                                                        
17.      | }
18. printDate: (date: String)(String, String, String)
20. scala> printDate("Jan 01,2010") 
21. res0: (String, String, String) = (01,Jan,2010)
23. scala> printDate("01/01/2010") 
24. res1: (String, String, String) = (01,01,2010)
26. /*
27. A silly example which drops the first element of the date string
28. not useful but this demonstrates that we are matching agains a sequence so 
29. the _* can be used to match the rest of the groups
30. */
31. scala> def split(date:String) = date match {         
32.      | case d @ Date(_:String ,_*) => d drop 3       
33.      | case d @ Date(_,_,_,_,_:String,_*) => d drop 4
34.      | case _ => "boom"                              
35.      | }
36. split: (date: String)String
38. scala> split ("Jan 31,2004")
39. res5: String = 31,2004
41. scala> split ("11/12/2004") 
42. res6: String = 12/2004
44. /*
45. This is just a reminder that the findAllIn returns an iterator which (since it is probably a short iterator) can be converted to a sequence and processed with matching
46. */
47. scala> val Seq(one,two,_*) = ("""\d\d/""".r findAllIn "11/01/2010" ).toSeq  
48. one: String = 11/
49. two: String = 01/
51. scala> val Seq(one,two) = ("""\d\d/""".r findAllIn "11/01/2010" ).toSeq   
52. one: String = 11/
53. two: String = 01/
55. // drop the two first matches and assign the rest to d
56. scala> val Seq(_,_,d @ _*) = ("""\d\d/""".r findAllIn "11/01/20/10/" ).toSeq
57. d: Seq[String] = ArrayBuffer(20/, 10/)

Regular Expression 2 : The rest Regex class

This is the second installment of Regular expressions in Scala. In the first installment the basics were shown and a few of the methods in the Regex class were inspected. This topic will look at the rest of the methods in the Regex class.

Regex.findPrefixMatchOf

1. /*
2. returns the match if the regex is the prefix of the string
3. */
4. scala> "(h)(e)|(l)".r findPrefixMatchOf "hello xyz"  
5. res2: Option[scala.util.matching.Regex.Match] = Some(he)
7. scala> "lo".r findPrefixMatchOf "hello xyz"  
8. res3: Option[scala.util.matching.Regex.Match] = None
10. /*
11. The method is essentially the same as adding the boundary regex character
12. */
13. scala> "^ab".r findFirstMatchIn "ababab"
14. res8: Option[scala.util.matching.Regex.Match] = Some(ab)
16. scala> "^ab".r findFirstMatchIn "hababab"
17. res9: Option[scala.util.matching.Regex.Match] = None
19. /*
20. findPrefixOf is the same but returns the matched string instead
21. */
22. scala> "ab".r findPrefixOf "haababab"       
23. res11: Option[String] = None
25. scala> "ab".r findPrefixOf "ababab"    
26. res12: Option[String] = Some(ab)

Regex.replaceAllIn -- Essentially the same as using String.replaceAll
Regex.replaceFirstIn -- Essentially the same as using String.replaceFirst

1. scala> "(h)(e)|(l)".r replaceAllIn ("hello xyz","__")
2. res13: String = ______o xyz
4. scala> "hello xyz" replaceAll ("(h)(e)|(l)","__")    
5. res14: java.lang.String = ______o xyz
7. scala> "hello xyz" replaceFirst ("(h)(e)|(l)","__")  
8. res16: java.lang.String = __llo xyz
10. scala> "(h)(e)|(l)".r replaceFirstIn ("hello xyz","__")
11. res17: String = __llo xyz

This next section is not Scala specific but because Regex does not provide a way to set the flags CASE_INSENSITIVE, DOTALL, etc... The section is useful to demonstrate how to do it as part of the standard regex syntax.

1. // examples based on java blog at: <a href="http://www.javaranch.com/journal/2003/04/RegexTutorial.htm#flags">http://www.javaranch.com/journal/2003/04/RegexTutorial.htm#flags</a>
2. scala> val input = """Hey, diddle, diddle,      
3.      | |The cat and the fiddle,                 
4.      | |The cow jumped over the moon.           
5.      | |The little dog laughed                  
6.      | |To see such sport,                      
7.      | |And the dish ran away with the spoon.""".stripMargin
8. input: String = 
9. Hey, diddle, diddle,
10. The cat and the fiddle,
11. The cow jumped over the moon.
12. The little dog laughed
13. To see such sport,
14. And the dish ran away with the spoon.
16. // by default regex is case sensitive
17. scala> """the \w+?(?=\W)""".r findAllIn input foreach (println _)
18. the fiddle
19. the moon
20. the dish
21. the spoon
23. /* the (?i)  makes the match case insensitive the complete set of options are:
24. (?idmsux)
25. • i - case insensitive
    • d - only unix lines are recognized as end of line
    • m - enable multiline mode
    • s - . matches any characters including line end
    • u - Enables Unicode-aware case folding
    • x - Permits whitespace and comments in pattern
26. */
27. scala> """(?i)the \w+?(?=\W)""".r findAllIn input foreach (println _)
28. The cat
29. the fiddle
30. The cow
31. the moon
32. The little
33. the dish
34. the spoon

Regular Expression 1 : Basics and findAllIn

This is the first of a series on Regular expression use in Scala. There was a previous post related that is also worth looking at but the same tips will be revisited in this series: Matching Regular Expressions.

Perhaps the most important thing for regular expressions in Scala is to be aware of the raw string syntax:

1. /* 
2. normal strings treat the \ character as the escape character 
3. so this fails
4. */
5. scala> val normalString = "\.+((xyz)|(abc))"
6. < console>:1: error: invalid escape character
7.        val normalString = "\.+((xyz)|(abc))"
8.                             ^
10. /*
11. raw strings a great for regular expressions so you don't have 
12. escape \ characters
13. */
14. scala> val rawString = """\.+((xyz)|(abc))"""
15. rawString: java.lang.String = \.+((xyz)|(abc))

The next thing is to realize that one can easily create a Regex object from any string with the r method:

1. scala> val regex = """\.+((xyz)|(abc))""".r
2. regex: scala.util.matching.Regex = \.+((xyz)|(abc))

A regex has the standard matching methods one might expect but lets look at findAllIn and the associated MatchData for now. findAllIn returns a MatchIterator which is an Iterator[String] with MatchData. When iterating over the MatchIterator the full matched string will be returned, if you need the subgroups you will need to convert the MatchIterator to an Iterator[Match].

1. // findAllIn returns an iterator over the matches.  
2. scala> "l|he".r findAllIn "hello xyz" foreach {println _}
3. he
4. l
5. l
7. // Each match can have multiple groups
8. // Note: Each element in MatchIterator are strings (no Match objects)
9. scala> ("(h)(e)|(l)".r findAllIn "hello xyz").matchData foreach { m => println(m.matched mkString ",")}   
10. h,e
11. l
12. l
14. // to access subgroups use the matchData method
15. // Note: there are 3 subgroups in the regex
16. scala> ("(h)(e)|(l)".r findAllIn "hello xyz").matchData foreach { m => println( m.subgroups mkString ",")}
17. h,e,null
18. null,null,l
19. null,null,l
21. /*
22. if matched is called the full match is returned (as if you did not convert the iterator to an Iterator[Match])
23. /*
24. scala> ("(h)(e)|(l)".r findAllIn "hello xyz").matchData foreach { m => println(m.matched)}
25. he
26. l
27. l
29. /*
30. The following demonstrates more of the methods on Match
31. Essentially the elements are:
32. (start index of match, end index of match, string before match, string after match, string the match was performed on)
33. */
34. scala> ("(h)(e)|(l)".r findAllIn "hello xyz").matchData foreach { m => println(m.start, m.end, m.before, m.after, m.source)}
35. (0,2,,llo xyz,hello xyz)
36. (2,3,he,lo xyz,hello xyz)
37. (3,4,hel,o xyz,hello xyz)

The last methods to look at for this topic are findFirstIn, findFirstMatchIn and the Regex constructor:

1. /*
2. Groups names can be assigned if the Regex constructor is used
3. */
4. scala> val withNames = new util.matching.Regex("(h)(e)|(l)", "h", "e", "l")
5. withNames: scala.util.matching.Regex = (h)(e)|(l)
7. scala> withNames findFirstIn "hello xyz"
8. res28: Option[String] = Some(he)
10. /*
11. I know a match will be found so I am extracting the value from the Option by assigning it to Some(he)
12. */
13. scala> val Some(he) = withNames findFirstMatchIn "hello xyz"  
14. he: scala.util.matching.Regex.Match = he
16. scala> he.groupNames
17. res29: Seq[String] = Array(h, e, l)
19. scala> he.group("h")
20. res30: String = h
22. scala> he.group("e") 
23. res31: String = e
25. scala> he.group(1)
26. res32: String = h
28. scala> he.group(2)
29. res33: String = e
31. // Uh oh. NullPointer warning!
32. scala> he.group(3)
33. res34: String = null
35. scala> he.groupCount
36. res35: Int = 3

Matching in for-comprehensions

At a glance a for-comprehension appears to be equivalent to a Java for-loop, but it is much much more than that. As shown in post: for-comprehensions, for-comprehensions can have guards which filter out which elements are processed:

1. scala> for ( x <- 1 to 10; if (x >4) ) println(x)
2. 5
3. 6
4. 7
5. 8
6. 9
7. 10

They can be used to construct new collections:

1. scala>?for(?i?<-?List(?"a",?"b",?"c")?)?yield?"Word:?"+i
2. res1:?List[java.lang.String]?=?List(Word:?a,?Word:?b,?Word:?c)

They can contain multiple generators:

1. scala> for {x <- 1 to 10                           
2.      |      if(x%2 == 0)
3.      |      y <- 1 to 5} yield (x,y)
4. res1: scala.collection.immutable.IndexedSeq[(Int, Int)] = IndexedSeq((2,1), (2,2), (2,3), (2,4), (2,5), (4,1), (4,2), (4,3), (4,4), (4,5), (6,1), (6,2), (6,3), (6,4), (6,5), (8,1), (8,2), (8,3), (8,4), (8,5), (10,1), (10,2), (10,3), (10,4), (10,5))

What has not been covered is that the assignments also does pattern matching:

1. scala> for ( (x,y) <- (6 to 1 by -2).zipWithIndex) println (x,y) 
2. (6,0)
3. (4,1)
4. (2,2)

This is not surprising as this also occurs during normal assignment. But what is interesting is that the pattern matching can act as a guard as well. See Extractor examples and Assignment and Parameter Objects for more information of pattern matching and extractors.

1. scala> val args = Array( "h=2", "b=3")
2. args: Array[java.lang.String] = Array(h=2, b=3)
4. scala> val Property = """(.+)=(.+)""".r 
5. Property: scala.util.matching.Regex = (.+)=(.+)
7. scala> for {Property(key,value) <- args } yield (key,value)
8. res0: Array[(String, String)] = Array((h,2), (b,3))
10. scala> Map(res0:_*)
11. res1: scala.collection.immutable.Map[String,String] = Map(h -> 2, b -> 3)
13. scala> res1("h")
14. res3: String = 2

Now just for fun here is a similar example but using symbols instead of strings for the key values:

1. scala> val args = Array( "h=2", "b=3")
2. args: Array[java.lang.String] = Array(h=2, b=3)
4. scala> val Property = """(.+)=(.+)""".r 
5. Property: scala.util.matching.Regex = (.+)=(.+)
7. scala> for {Property(key,value) <- args } yield (Symbol(key),value)
8. res0: Array[(Symbol, String)] = Array(('h,2), ('b,3))
10. scala> Map(res0:_*)
11. res1: scala.collection.immutable.Map[Symbol,String] = Map('h -> 2, 'b -> 3)
13. scala> res1('h)
14. res2: String = 2

Matching Regular expressions

This topic is derived from the blog post: Using pattern matching with regular expressions in Scala

The Regex class in Scala provides a very handy feature that allows you to match against regular expressions. This makes dealing with certain types of regular expression very clean and easy to follow.

What needs to be done is to create a Regex class and assign it to a val. It is recommended that the val starts with an Uppercase letter, see the topic of matching about the assumptions matching makes based on the first letter of the Match case clause.

There is nothing like examples to help explain an idea:

1. // I normally use raw strings (""") for regular expressions so that I don't have to escape all \ characters
2. // There are two ways to create Regex objects.
3. // 1. Use the RichString's r method
4. // 2. Create it using the normal Regex constructor
5. scala> val Name = """(\w+)\s+(\w+)""".r
6. Name: scala.util.matching.Regex = (\w+)\s+(\w+)
8. scala> import scala.util.matching._
9. import scala.util.matching._
11. // Notice the val name starts with an upper case letter
12. scala> val Name = new Regex("""(\w+)\s+(\w+)""")
13. Name: scala.util.matching.Regex = (\w+)\s+(\w+)
15. scala> "Jesse Eichar" match {
16.      | case Name(first,last) => println("found: ", first, last)
17.      | case _ => println("oh no!")
18.      | }
19. (found: ,Jesse,Eichar)
21. scala> val FullName = """(\w+)\s+(\w+)\s+(\w+)""".r
22. FullName: scala.util.matching.Regex = (\w+)\s+(\w+)\s+(\w+)
24. // If you KNOW that the match will work you can assign it to a variable
25. // Only do this if you are sure the match will work otherwise you will get a MatchError
26. scala> val FullName(first, middle, last) = "Jesse Dale Eichar"
27. first: String = Jesse
28. middle: String = Dale
29. last: String = Eichar