szeiger.de » Scala

A Zipper for scala.xml

Stefan Zeiger — Sun, 27 Dec 2009 00:54:11 +0000

In recent discussions on the scala-internals and scala-xml mailing lists, there were calls for a mutable, DOM-like XML model, where nodes hold references to their parent element, so that all standard XPath axes can be supported. In this post I’d like to present a different representation which is based on the immutable and persistent scala.xml model, is completely immutable itself, yet allows navigation along all axes and “mutation”. It is based on the Zipper technique which was first described by Gérard Huet in this paper [PDF].

The Problem

Take this piece of XML (an Eclipse .project file):


  scala-query
  
  
  
    
      ch.epfl.lamp.sdt.core.scalabuilder
      
    
  
  
    ch.epfl.lamp.sdt.core.scalanature
    org.eclipse.jdt.core.javanature

Its scala.xml model looks like this:

(In reality, a pretty-printed document contains a bunch of additional text nodes with all the whitespace between the elements, but we can safely ignore them and work with a simplified model.)

Imagine you have a reference to the first “nature” element which we will call the context. The only other node reachable from this context is the text node below it. You cannot navigate to its parent or sibling without starting at the root element and searching for the right spot.

An obvious solution to this problem is to add a reference from each node to its parent but that means giving up persistence (the sharing of data between different versions of the tree).

Let’s look at the situation without the parent reference first and assume you wanted to add a third “nature” element before the two existing ones. Since you cannot modify the “natures” element in this immutable data structure, you have to make a copy of it which contains the two original children plus the new one. Now that you have a new “natures” element, you make a copy of the “projectDescription” root element above it which contains all the original children except for the “natures” element which is replaced with the new one. This gives you two separate trees (the original and the modified one) which share most of their nodes:

(For the “natures” element we can even reuse the list of its children because we only prepend an element at the beginning. We need to copy the list of the root element’s children though because we replace its last element.)

Now try the same on a model with parent references. You can create the new “nature” element and a copy of the “natures” element above it. But what about the other two “nature” elements? You cannot reuse them because they contain a reference to the original “natures” element, so you have to create copies of them which point up to their new parent. You cannot even reuse the text nodes below them because they also reference their original parents. And now that you have copied the entire “natures” sub-tree, you face the same problem with the root element. All its children point up to the old root element, so you have to copy them recursively, too. In the end you have copied the tree entirely!

The Zipper

We can do better with the Zipper. The basic idea is to take an immutable data structure for the actual data (in this case XML documents in scala.xml models) and add a representation for a context that allows you to navigate through the data in all directions and modify parts of it (in the same way that you modified a document in the previous section — by creating a new document that shares most of the data).

We start with a type for the context:

sealed case class NodeLoc(node: Node, path: NodePath)

It consists of the node (or sub-tree) which it wraps plus a path to the root element. A path can either be the top (root) of the tree or a hole, i.e. the location of a node without the node itself:

sealed trait NodePath
final case object Top extends NodePath
final case class Hole(left: List[Node],
  parent: NodeLoc, right: List[Node]) extends NodePath

For example, here is a NodeLoc for the first “nature” element:

You can see that the original tree (in grey and yellow) is still intact. There are two NodeLocs with holes which point up to a parent NodeLoc and contain lists of their preceding and following siblings. Note that the list of preceding siblings is in reverse order (compared to the corresponding list of children on the parent node) so that the first node in the list is the next element to the left of the hole.

We can now define the basic navigation methods on NodeLoc. In order to move to the left sibling, we create a new NodeLoc which contains the head of the “left” list as its node and a copy of the current hole with the remaining nodes on the “left” list and the current node appended to the “right” list:

  protected def create(node: Node, path: Hole) = NodeLoc(node, path)

  final def leftOpt = path match {
    case Hole(tl :: l, p, r) =>
      Some(create(tl, Hole(l, p, node :: r)))
    case _ => None
  }

Moving to the right is symmetrical to that:

  final def rightOpt = path match {
    case Hole(l, p, tr :: r) =>
      Some(create(tr, Hole(node :: l, p, r)))
    case _ => None
  }

Methods left and right which return a bare NodeLoc or throw an exception, and isFirst and isLast to ensure that you don’t run into such an exception, can be defined in the same way. I’m using Option here because it will be useful later on for creating the XPath axes.

Moving down to a child with the specified index position is accomplished by splitting the list of the current node’s children at the correct position, removing the child node and creating a Hole for the other children which points up to this NodeLoc:

  final def downOpt(idx: Int) = {
    val ch = node.child
    if(ch.isEmpty) None
    else Some(create(ch.head,
      Hole(ch.tail.take(idx).reverse.toList,
           this, ch.drop(idx+1).toList)))
  }

Mutable Yet Immutable

Replacing the node at the current location is extremely simple. We just create a NodeLoc with the new Node and the current path:

  final def set(n: Node) = NodeLoc(n, path)

There is no need to propagate the change up to the root element at this point. We could, for example, move through all siblings, replacing each one with a new node, all in constant time (per sibling). Parent elements are not copied until we move up and “unzip” the path into a new tree. This is done by constructing a new list of children from the current node and its left and right siblings, and creating a copy of the old parent with the new children:

  def upOpt = path match {
    case h : Hole =>
      val l = h.left.reverse ++ (node :: h.right)
      Some(NodeLoc(h.parent.node match {
        case e: Elem => e.copy(child = l)
        case _: Group => new Group(l)
      }, h.parent.path))
    case _ => None
  }

This is slightly messy with scala.xml because we have to match on the type of node and copy Elem and Group nodes (the only ones which can contain children) in different ways.

Always recreating the parent when moving up is a waste of time when no changes have been made to the current sub-tree. We can optimize for this case with a special CachedParentNodeLoc which returns the original parent when moving up:

final class CachedParentNodeLoc(node: Node, path: Hole)
extends NodeLoc(node, path) {
  override def upOpt = Some(path.parent)
  override protected def create(node: Node, path: Hole) =
    new CachedParentNodeLoc(node, path)
}

final class CachedTopNodeLoc(node: Node) extends NodeLoc(node, Top) {
  override def upOpt = None
  override protected def create(node: Node, path: Hole) =
    new CachedParentNodeLoc(node, path)
}

object NodeLoc {
  def apply(node: Node): NodeLoc = new CachedTopNodeLoc(node)
}

Note that the leftOpt, rightOpt and downOpt methods use create(...) instead of NodeLoc(...), so that navigating left, right or down from a cached location returns another cached location.

The XPath Axes

The XPath specification defines the following axes:

the child axis contains the children of the context node

the descendant axis contains the descendants of the context node; a descendant is a child or a child of a child and so on; thus the descendant axis never contains attribute or namespace
nodes

the parent axis contains the parent of the context node, if there is one

the ancestor axis contains the ancestors of the context node; the ancestors of the context node consist of the parent of context node and the parent’s parent and so on; thus, the ancestor axis will always include the root node, unless the context node is the root node

the following-sibling axis contains all the following siblings of the context node; if the context node is an attribute node or namespace node, the following-sibling axis is empty

the preceding-sibling axis contains all the preceding siblings of the context node; if the context node is an attribute node or namespace node, the preceding-sibling
axis is empty

the following axis contains all nodes in the same document as the context node that are after the context node in document order, excluding any descendants and excluding attribute nodes and namespace nodes

the preceding axis contains all nodes in the same document as the context node that are before the context node in document order, excluding any ancestors and excluding attribute nodes and namespace nodes

the attribute axis contains the attributes of the context node; the axis will be empty unless the context node is an element

the namespace axis contains the namespace nodes of the context node; the axis will be empty unless the context node is an element

the self axis contains just the context node itself

the descendant-or-self axis contains the context node and the descendants of the context node

the ancestor-or-self axis contains the context node and the ancestors of the context node; thus, the ancestor axis will always include the root node

The attribute and namespace axes do not fit well into our model because scala.xml does not represent attributes and namespace declarations as Node values. All other axes can be defined as methods on NodeLoc.

We will represent an axis as an Iterator[NodeLoc]. The following helper method creates an iterator by starting with a given value and continually applying a function to it until it returns None. It is similar to Iterator.iterate in Scala 2.8 except the latter can only create infinite iterators.

  private final def iterate[T](start: Option[T])(f: T => Option[T]): Iterator[T] = new Iterator[T] {
    private[this] var acc = start
    def hasNext = acc.isDefined
    def next() = acc match {
      case None => throw new NoSuchElementException("next on empty iterator");
      case Some(x) => val res = x ; acc = f(x) ; res
    }
  }

We can use it like this to create the child axis: Start with the first child, then go right node by node until you reach the last child:

  final def childAxis = iterate(downOpt(0))(_.rightOpt)

The descendant-or-self axis consists of the current node plus the descendant-or-self axes of all children:

  final def descendantOrSelfAxis: Iterator[NodeLoc] =
    Iterator.single(this) ++ childAxis.flatMap(_.descendantOrSelfAxis)

And the descendant axis consists only of the descendant-or-self axes of all children, without the current node:

  final def descendantAxis = childAxis.flatMap(_.descendantOrSelfAxis)

The parent, ancestor-or-self and ancestor axes work similarly:

  final def parentAxis = upOpt.iterator

  final def ancestorOrSelfAxis: Iterator[NodeLoc] =
    Iterator.single(this) ++ upOpt.iterator.flatMap(_.ancestorOrSelfAxis)

  final def ancestorAxis = upOpt.iterator.flatMap(_.ancestorOrSelfAxis)

The following-sibling and preceding-sibling axes are straight-forward:

  final def followingSiblingAxis = iterate(rightOpt)(_.rightOpt)

  final def precedingSiblingAxis = iterate(leftOpt)(_.leftOpt)

For the following and preceding axes we need to define some helper methods for moving to the following or preceding node in document order:

  final def followingAxis = iterate(followingOpt)(_.followingOpt)

  final def followingOpt: Option[NodeLoc] =
    downOpt(0) orElse rightOutOpt

  private final def rightOutOpt: Option[NodeLoc] =
    rightOpt orElse upOpt.flatMap(_.rightOutOpt)

  final def precedingAxis = iterate(precedingOpt)(_.precedingOpt)

  final def precedingOpt: Option[NodeLoc] =
    leftOpt.map(n => n.downLastTransitiveOpt.getOrElse(n)) orElse upOpt

  private final def downLastTransitiveOpt: Option[NodeLoc] =
    downLastOpt.flatMap(_.downLastTransitiveOpt)

  final def downLastOpt = {
    val ch = node.child
    if(ch.isEmpty) None
    else Some(NodeLoc(ch.head, Hole(ch.reverse.toList.tail, this, Nil)))
  }

And finally the trivial self axis:

  final def selfAxis = Iterator.single(this)

Methods \ and \\ for finding all child or descendant elements with a given name can be defined by filtering the child and descendant axes:

  final def \ (Name: String) =
    for(n @ NodeLoc(Elem(_, Name, _, _, _*), _) <- childAxis) yield n

  final def \\ (Name: String) =
    for(n @ NodeLoc(Elem(_, Name, _, _, _*), _) <- descendantAxis) yield n

You can find the complete source code, including more convenience methods and a short demo here.

New ScalaQuery Features

Stefan Zeiger — Mon, 21 Dec 2009 20:15:03 +0000

It’s been almost 3 months since my last summary of new features in ScalaQuery. I implemented some important changes in the following weeks but I only had little time to work on ScalaQuery after my extended one-month vacation ended in October.

Build system

Builds against newer versions of Scala 2.8 have been going well thanks to sbt 0.6 which separates the Scala version used by the build system from the version used by the project being built. You need to use at least version 0.6.7 of the new xsbt launcher to build ScalaQuery. Other dependencies are downloaded automatically by sbt. I wrote an implementation of sbt’s test interface for JUnit which you can find here (binaries are here on scala-tools.org). It allows you to run ScalaQuery’s JUnit test cases from sbt with the test command.

All published artifacts now contain the Scala version in the artifact ID. You can find the latest ScalaQuery snapshot in scala-tools.org’s snapshots repository as:

groupId:: com.novocode
artifactId:: scala-query_2.8.0.Beta1-RC4
version:: 1.0.0-SNAPSHOT

A ScalaQuery Update

Stefan Zeiger — Sun, 27 Sep 2009 00:03:30 +0000

Since my last post about ScalaQuery one month ago I have wasted quite some time trying to implement the OptionMapper type (which is used to make operations on columns interoperable between base types and Option types) in a way which does not require 2^n implicit functions and 2*n+2 type parameters for operations with n operands. The composable OptionMapper itself is not hard to write (see NewOptionMapper here) but it requires more type fidelity than ScalaQuery offers at the moment: Either Projections need to preserve the subtypes of Columns they are storing, or the required types have to be reconstructed from implicit values where they are needed. You can find my failed attempts in the projection-types and new-option-mapper branches. The first one might actually work once Scala bug #2346 is resolved. The second one would require a much more powerful type inferencer. For now, I just added an OptionMapper3 for 3 parameters (which is needed by the BETWEEN operator).

There are also some interesting new features in ScalaQuery:

Mapped Projections

You can now create a bidirectional mapping of a Projection with the <> operator. This is especially useful for a table’s “*” projection. The <> operator takes a function from the projection tuple type T (either as a tuple or as multiple parameters) to the mapped type R, and a function from R back to Some[T]. The asymmetry with the unnecessary Some wrapper is deliberate, matching exactly the apply und unapply methods of a case class. For example, here is a simple User class and table:

  case class User(first: String, last: String)

  object Users extends Table[User]("users") {
    def first = column[String]("first", O NotNull)
    def last = column[String]("last", O NotNull)
    def * = first ~ last <> (User, User.unapply _)
  }

Whenever you perform an insert, update or query directly on such a table (instead of a projection of individual columns), you can use the mapped type:

  Users.insert(User("Stefan", "Zeiger"))
  val someUsers: List[User] = Users.where(_.first startsWith "S").list

Finders

The new convenience method Table.createFinderBy allows you to create a simple finder query template which selects values from a table by matching on a single column, e.g.:

  val findUserByFirstName = Users.createFinderBy(_.first)

This is equivalent to:

  val findUserByFirstName = for {
    first <- Parameters[String]
    u <- Users if u.first is first
  } yield u

Updatable ResultSets

JDBC allows you to modify a ResultSet which was created with the CONCUR_UPDATABLE result set concurrency. ScalaQuery now offers a high-level interface to this functionality with the mutate method, e.g.:

  val q1 = for(u <- Users if u.last.is("Simpson") || u.last.is("Bouvier")) yield u
  q1.mutate { m =>
    if(m()._3 == "Bouvier") m() = m().copy(_3 = "Simpson")
    else if(m()._2 == "Homer") m.delete()
    else if(m()._2 == "Bart") m.insert((None, "Lisa", "Simpson"))
  }

Statement Options

The new ResultSetType, ResultSetConcurrency and ResultSetHoldability classes allow you to request the corresponding JDBC options when ScalaQuery creates a PreparedStatement, e.g.:

  myDB withSession {
    ResultSetType.ScrollInsensitive {
      // All database calls in this block use TYPE_SCROLL_INSENSITIVE
    }
  }

Or if you don't want to use the implicit threadLocalSession:

  myDB withSession { s1 =>
    ResultSetType.ScrollInsensitive(s1) { s2 =>
      // All database calls on s2 use TYPE_SCROLL_INSENSITIVE
    }
  }

The default for all three options is Auto which gives you values suitable for the invoker method you are calling (e.g. ResultSetConcurrency.Updatable for mutate and ResultSetConcurrency.ReadOnly for all other methods).

Build System

I have rewritten most of the test classes as proper unit tests (using JUnit 4) with assertions. I'd like to investigate ScalaTest and specs further (possibly together with ScalaCheck) in the future to replace JUnit but the binary incompatibilities between different Scala 2.8 snapshot builds would only complicate the build process right now. If you build with sbt, you still need to download a suitable Scala 2.8 snapshot yourself and set the paths in the properties because the build is done in forking mode. This also means that you cannot use the "test" command to run the unit tests from sbt at the moment.

H2 and JUnit are declared as test dependencies in the sbt project definition and the Eclipse build path refers to the local copies in lib_managed. If you want to build ScalaQuery with Eclipse (as I usually do), just run "sbt update" once to pull in the required JARs.

I am using FreeMarker templates and the FMPP tool to create repetitive code for the Projection and Parameters classes. This is not yet integrated into the sbt build process. You can use the supplied Eclipse launcher instead (after downloading FMPP and pointing the launcher to the correct path). The generated sources are checked in, so you only need to do this if you want to change the templates and rebuild the Scala sources.

Profiles, Drivers & More

Stefan Zeiger — Thu, 27 Aug 2009 15:07:47 +0000

It’s been three weeks already since my last post about ScalaQuery so I’d like to provide an update on some recent changes.

Profiles & Drivers

The biggest news is that ScalaQuery now supports database engine-specific features. Feature sets can be bundled in profiles which are then implemented by drivers (Of course, a driver can also add driver-specific features which are not part of a profile). All previously existing features are now part of the BasicProfile which should work on all SQL databases with little or no customization. The BasicProfile provides:

An object called Implicit with all implicit conversions which are required to use the profile’s features (including an implicit value of type BasicProfile which points to the driver implementing the profile).
Some methods for creating various statements and query templates. They have default implementations provided by the profile but can be overridden by other profiles or drivers. You do not usually call them directly.
A number of TypeMapperDelegate objects for the basic types supported by the profile. You still use TypeMappers defined outside the profile but they do not implement the actual type mapping any more. Instead they ask the profile for a delegate implementation. (Alternatively, I could have moved the existing TypeMapper objects entirely into the profile but mappers for some basic types like Int and Boolean are used internally at many places. They would need to be threaded through several methods and constructors, thus complicating the implementation unnecessarily.)

The BasicProfile is implemented by the BasicDriver.

There is also a new ExtendedProfile for features which are expected to be available in every commonly used SQL database system, albeit with a non-standard syntax. It currently provides string concatenation with the ++ operator (plus startsWith and endsWith methods which can be implemented with LIKE and string concatenation) and the take and drop methods needed for pagination.

If you only need to support a single database engine in your application, you can forget about profiles and just import a specific driver’s implicit conversions. Supporting multiple drivers is almost as simple: Give your DAO class (or whatever you have in your application design) a parameter of the required profile type and import this profile’s implicits. You can then call it with any driver which implements the profile. For example:

  def test(profile: ExtendedProfile) {
    import profile.Implicit._
    println("Using driver: "+profile.getClass.getName)
    val q1 = Users.where(_.name startsWith "quote ' and backslash \\").take(5)
    println(q1.selectStatement)
    val q2 = Users.where(_.name startsWith "St".bind).drop(10).take(5)
    println(q2.selectStatement)
    val q3 = Query(42 ~ "foo")
    println(q3.selectStatement)
    println
  }

  def main(args: Array[String]) {
    test(H2Driver)
    test(OracleDriver)
    test(MySQLDriver)
  }

This prints the different statements required for H2, Oracle and MySQL:

Using driver: com.novocode.squery.combinator.extended.H2Driver$
SELECT t1.name FROM users t1 WHERE (t1.name like 'quote '' and backslash \%') LIMIT 5
SELECT t1.name FROM users t1 WHERE (t1.name like (? || '%')) LIMIT 5 OFFSET 10
SELECT 42,'foo'

Using driver: com.novocode.squery.combinator.extended.OracleDriver$
SELECT * FROM (SELECT t1.name FROM users t1 WHERE (t1.name like 'quote '' and backslash \%')) WHERE ROWNUM <= 5
SELECT * FROM (SELECT t0.*, ROWNUM ROWNUM_O FROM (t1.name,ROWNUM ROWNUM_I FROM users t1 WHERE (t1.name like (? || '%'))) t0) WHERE ROWNUM_O BETWEEN (1+10) AND (10+5) ORDER BY ROWNUM_I
SELECT 42,'foo' FROM DUAL

Using driver: com.novocode.squery.combinator.extended.MySQLDriver$
SELECT t1.name FROM users t1 WHERE (t1.name like 'quote \' and backslash \\%') LIMIT 5
SELECT t1.name FROM users t1 WHERE (t1.name like concat(?,'%')) LIMIT 10,5
SELECT 42,'foo' FROM DUAL

Updates

You can now take a simple query (returning only named columns from a single table; no modifiers except WHERE restrictions) and call it as an UPDATE statement:

  val q = for(u <- Users if u.id is 42) yield u.first ~ u.last
  q.update("foo", "bar")

Filtering

You may already have noticed in my previous post that the Query class now has a filter method which works like where for functions returning a Column[Boolean] or Column[Option[Boolean]], so you can use Scala’s standard if clauses in for-comprehensions on queries. Unlike where, filter also accepts functions returning a plain Boolean value to enable the use of refutable patterns in queries (e.g. when destructuring a Join).

Invokers

Result type remapping and parameter application are now available for all Invokers. These features were pushed down from the statement invokers for monadic queries into the invoker framework. Query templates are parameterized invokers now. They can be applied like any other invoker.

Efficient Parameterized Queries in ScalaQuery

Stefan Zeiger — Thu, 06 Aug 2009 17:22:04 +0000

One question about ScalaQuery which keeps coming up is that of perfomance. The question is usually in the form “How does it compare to JDBC?” but that’s like comparing apples and, well, apple trees. After all, ScalaQuery is a layer on top of JDBC which provides mainly two things:

A nicer, more Scala-like way of handling database connections, performing queries and reading result sets. This is not optional when you access a database in your application. You will wrap SQL statement execution and result set reading in some way or another to abstract from the low-level JDBC API. Anyway, the overhead here is quite low.
A way of composing queries with an internal DSL based on a query monad and combinators. That’s the part I want to talk about in this post.

If you want to optimize the query generation away, you can always fall back to the StaticQuery and DynamicQuery classes. These work a bit like iBATIS, except your SQL code is embedded directly in your Scala code and not in some XML files. But you’re still writing SQL! That’s nice for the special cases which are not covered by the combinator queries and which do not need to be composable but it’s probably not the reason why you want to use ScalaQuery in the first place.

When you’re constructing a query in the query monad, you normally have some variables which are used in the query like this:

def userNameByID(id: Int) =
  for(u <- Users if u.id is id) yield u.first

This would insert the user ID directly into the SQL statement, thus requiring a new statement to be generated by ScalaQuery and parsed and optimized by the database server (which can be very expensive) for every invocation of the query. We can improve this by using a bind variable for the user ID:

def userNameByID(id: Int) =
  for(u <- Users if u.id is id.bind) yield u.first

Now the generated SQL statement is always the same (e.g. “SELECT t1.first FROM users t1 WHERE (t1.id=?)“), so the database server needs to calculate an execution plan for it only once and can reuse it on subsequent invocations. But the query is still constructed as a tree of dozens of objects and compiled to the same SQL statement every time you call userNameByID.

This can be remedied with query templates, a recent addition to ScalaQuery:

val userNameByID = for {
  id <- Parameters[Int]
  u <- Users if u.id is id
} yield u.first

If you recall how for-comprehensions are desugared, this gets translated to Parameters[Int].apply(...).flatMap(id => ...). The apply method on the Parameters object takes an implicit TypeMapper for each parameter type you specify and creates a Parameters instance. The flatMap method on Parameters takes a function which creates a Query and returns a QueryTemplate for it:

final class QueryTemplate[P, R](query: Query[ColumnBase[R]]) {
  def apply(param: P) = new AppliedQueryTemplate(built, param, query.value)
  lazy val built = QueryBuilder.buildSelect(query, NamingContext())
}

final class Parameters[P, C](c: C) {
  def flatMap[F](f: C => Query[ColumnBase[F]]): QueryTemplate[P, F] =
    new QueryTemplate[P, F](f(c))
  ...
}

object Parameters {
  def apply[P1](implicit tm1: TypeMapper[P1]) =
    new Parameters[P1, Column[P1]](new ParameterColumn(-1, tm1))
  ...
}

Note that, unlike Query, neither Parameters nor QueryTemplate is a monad. You cannot compose multiple parameter lists or query templates this way but by providing a suitable flatMap method, the parameters can be used as the first generator in a for-comprehension which otherwise operates in the Query monad.

Either one of the userNameByID functions/methods defined above can be used in the same way:

for(t <- userNameByID(3)) println(t)

When you apply the parameters to a QueryTemplate, you get an AppliedQueryTemplate which can be lifted to a suitable Invoker by an implicit conversion (just like a Query). The first time you do this, the SQL code gets generated and then cached in the QueryTemplate for further applications.

If you specify more than one type parameter, the Parameters generator gives you a Projection instead of a single Column. It can be unpacked either with the extractor on the “~” object:

val userNameByIDRangeAndProduct = for {
  min ~ max ~ product <- Parameters[Int, Int, String]
  u <- Users if u.id >= min &&
    u.id <= max &&
    Orders.where(o => (u.id is o.userID) && (o.product is product)).exists
} yield u.first

…or with the Projection extractor (which is slightly more efficient but not as nice to read):

val userNameByIDRangeAndProduct = for {
  Projection(min, max, product) <- Parameters[Int, Int, String]
  u <- Users if u.id >= min &&
    u.id <= max &&
    Orders.where(o => (u.id is o.userID) && (o.product is product)).exists
} yield u.first

ScalaQuery’s invoker framework provides methods for invoking queries with parameters but these are currently used by the simple queries only. I expect to integrate query templates with this system in the future.

ScalaQuery for different Scala versions

Stefan Zeiger — Wed, 22 Jul 2009 20:12:11 +0000

I have just created a new scala-2.7 branch for ScalaQuery. My original plan was to target only Scala 2.8 but since I’ve made lots of progress during the last few weeks and I’ve seen increased interest in ScalaQuery, I tried to build it with 2.7.5.

I had to change the semantics of SimpleFunction and SimpleBinaryOperator for 2.7 but I prefer the new version anyway, so it went into the main line. The code on the scala-2.7 branch is currently identical to the master branch, except for the test classes which are different in two regards:

Although the Scala Language Specification mandates that the part left to the “<-” in a for comprehension is a pattern, the wildcard pattern “_” does not work in 2.7. I have changed it to a dummy variable named “__“.
The type inferencer in 2.7 cannot infer the correct type for the implicit OptionMapper objects. OptionMapper[_,_,_,_] has four type parameters, the last one being used for the return type of functions which use the mapper, so it is not yet known when looking for an implicit mapper and gets inferred as Nothing. Scala 2.8 apparently knows that this type parameter is undetermined, finds the single matching implicit object for the other three parameters and then fills in the fourth. The type-correct interoperability of option and non-option types in ScalaQuery relies heavily on the improved type inferencer and I don’t see any way of making it work nicely with 2.7. The work-around is to add type annotations to the boolean operators, e.g. a && b might become a.&&[Boolean,Option[Boolean]](b). Yuck!

I’m still focused on Scala 2.8 as a target platform and will probably not spend much time integrating new features into the scala-2.7 branch but contributions are always welcome.

Implicits on Implicits

Stefan Zeiger — Thu, 16 Jul 2009 22:51:55 +0000

I have recently made a change to ScalaQuery which enables the use of any type for a column without needing specific Column classes and factory methods on Table for it. For this I used an approach which I had not considered earlier and which I wasn’t even sure would work.

Scala (at least in version 2.7) only considers single function calls for implicit conversions. Take the following definitions for example:

class A
class B(a: A)
class C(b: B)

implicit def aToB(a: A) = new B(a)
implicit def bToC(b: B) = new C(b)

val a = new A

def useB(b: B) = ...
def useC(c: C) = ...

You can call useB(a) because the compiler finds the implicit conversion aToB(a) but you cannot call useC(a) — the compiler does not try the chained call bToC(aToB(c)).

But this does not mean that an implicit conversion may not rely on an implicit value! The following works just fine:

trait Column[T]
class ConstColumn[T](value: T, tm: TypeMapper[T]) extends Column[T]

implicit def valueToColumn[T](value: T)(implicit tm: TypeMapper[T]) =
  new ConstColumn(value, tm)

trait TypeMapper[T]
implicit object IntTypeMapper extends TypeMapper[Int]

val c1: Column[_] = 42

The Scala compiler recognizes that valueToColumn[Int] provides the desired conversion from Int to Column[_] and then looks for the required implicit TypeMapper[Int] value, which is available through the implicit IntTypeMapper object.

In ScalaQuery, the real TypeMapper contains only a few methods and there are predefined TypeMappers for most of the basic types used by JDBC. That’s nice because it removes some redundancy from my code base but the better news is that it allows you to write your own TypeMappers. For example, if you wanted to get the java.lang.Integer columns back which I removed in favor of Int and Option[Int], you could add this implicit object to your code:

implicit object IntegerTypeMapper extends TypeMapper[java.lang.Integer] {
  def zero = null
  def sqlType = java.sql.Types.INTEGER
  def setValue(v: java.lang.Integer, p: PositionedParameters) =
    if(v eq null) p.setIntOption(None) else p.setIntOption(Some(v.intValue))
  def setOption(v: Option[java.lang.Integer], p: PositionedParameters) = v match {
    case Some(null) => p.setIntOption(None)
    case Some(i) => p.setIntOption(Some(i.intValue))
    case None => p.setIntOption(None)
  }
  def nextValue(r: PositionedResult) = r.nextIntOption match {
    case None => null
    case Some(i) => java.lang.Integer.valueOf(i)
  }
}

The same should work for any database engine- or domain-specific types.

Formal Language Processing in Scala, Solutions to Part 5

Stefan Zeiger — Wed, 15 Jul 2009 17:08:57 +0000

This is the solution to the exercise from part 5.

Some of the details I left out of the specification, but which are needed to write the parser, include:

What are the precedence rules for the operators?
When does “+” do string concatenation?
Can you apply a higher-order function multiple times with f(p1)(p2) or do you have to write (f(p1))(p2)?
Can expressions be empty (i.e. can you have multiple consecutive semicolons)?
Is an extra semicolon at the end of a sequence allowed?

And here is the complete parser and interpreter:

import scala.util.parsing.combinator.syntactical.StandardTokenParsers
import scala.util.parsing.input.PagedSeqReader
import scala.collection.immutable.PagedSeq

object Fun1a extends StandardTokenParsers with Application {

  sealed abstract class Expression
  case class Sequence(l: List[Expression]) extends Expression
  case class Lambda(params: List[String], expr: Expression) extends Expression
  case class Let(ident: String, expr: Expression) extends Expression
  case class Var(ident: String) extends Expression
  case class Lit(v: Any) extends Expression
  case class App(expr: Expression, args: List[Expression]) extends Expression
  case class Add(e1: Expression, e2: Expression) extends Expression
  case class Subtract(e1: Expression, e2: Expression) extends Expression
  case class Mult(e1: Expression, e2: Expression) extends Expression
  case class Div(e1: Expression, e2: Expression) extends Expression
  case class Equal(e1: Expression, e2: Expression) extends Expression
  case class If(e1: Expression, e2: Expression, e3: Expression) extends Expression

  lexical.reserved += ("let", "if", "then", "else")
  lexical.delimiters += (";", "(", ")", "+", "-", "*", "/", "\\", "->", ",", "=")

  val tokens = new lexical.Scanner(new PagedSeqReader(PagedSeq.fromFile("src/com/novocode/fun1/test.fun")))

  phrase(sequence)(tokens) match {
    case Success(tree, _) => eval(tree)
    case e: NoSuccess =>
      Console.err.println(e)
      exit(1)
  }

  def sequence = ( rep1sep(expression, ";") ^^ Sequence ) <~ (";"?)

  def expression: Parser[Expression] = lambda | let | equality | ifExpr

  def ifExpr = "if" ~> expression ~ "then" ~ expression ~ "else" ~ expression ^^ {
    case e1 ~ _ ~ e2 ~ _ ~ e3 => If(e1, e2, e3)
  }

  def lambda = "\\" ~> repsep(ident, ",") ~ "->" ~ expression ^^ {
    case params ~ _ ~ expr => Lambda(params, expr)
  }

  def let = "let" ~> ident ~ "=" ~ expression ^^ {
    case ident ~ _ ~ expr => Let(ident, expr)
  }

  def equality = sum * ("=" ^^^ Equal)

  def sum = product * ("+" ^^^ Add | "-" ^^^ Subtract)

  def product = application * ("*" ^^^ Mult | "/" ^^^ Div)

  def application = simpleExpr ~ (args*) ^^ {
    case expr ~ l => l.foldLeft(expr)(App)
  }

  def args = "(" ~> repsep(expression, ",") <~ ")"

  def simpleExpr = ident ^^ Var | "(" ~> expression <~ ")" |
    numericLit ^^ Lit.compose(_.toInt) | stringLit ^^ Lit

  sealed trait Func extends (List[Any] => Any)

  case object Println extends Func {
    def apply(args: List[Any]) = args match {
      case v :: Nil => { println(v); v }
      case _ => error("println needs 1 argument")
    }
  }

  def eval(e: Expression): Any = e match {
    case Sequence(l) => l.foldLeft(():Any) { (z,n) => eval(n) }
    //case Lambda(params, ex) => Closure(ex, params)
    //case Let(id, ex)  => { val v = eval(ex); (v, context + ((id, v))) }
    //case Var(id) => (context getOrElse(id, error("Undefined var "+id)), context)
    case App(expr, args) => eval(expr) match {
      case f:Func => f(args.map(a => eval(a)))
      case _ => error("Only functions can be applied")
    }
    case Lit(v) => v
    case If(e1, e2, e3) => eval(e1) match {
      case b:Boolean => eval(if(b) e2 else e3)
      case _ => error("Not a boolean value in IF expression")
    }
    case Equal(e1, e2) => eval(e1) == eval(e2)
    case Add(e1, e2) => (eval(e1), eval(e2)) match {
      case (v1:String, v2) => v1 + v2.toString
      case (v1, v2:String) => v1.toString + v2
      case (i1:Int, i2:Int) => i1 + i2
      case _ => error("'+' requires two Int values or at least one String")
    }
    case Subtract(e1, e2) => (eval(e1), eval(e2)) match {
      case (i1:Int, i2:Int) => i1 - i2
      case _ => error("'-' requires two Int values")
    }
    case Mult(e1, e2) => (eval(e1), eval(e2)) match {
      case (i1:Int, i2:Int) => i1 * i2
      case _ => error("'*' requires two Int values")
    }
    case Div(e1, e2) => (eval(e1), eval(e2)) match {
      case (i1:Int, i2:Int) => i1 / i2
      case _ => error("'/' requires two Int values")
    }
  }
}

Formal Language Processing in Scala, Part 5

Stefan Zeiger — Sun, 05 Jul 2009 14:43:11 +0000

This is the fifth part in a series of articles on formal language processing in Scala. In this part I will introduce some new parser combinators, provide a specification for the Fun1 language and build an interpreter for it.

Binary Operators

I’ve already shown briefly in the solutions to part 4 how to build a left-recursive AST for left-associative operators. I’d like to go back one step and look at this in more detail. We are always using repetitions (the “+” and “*” operators) instead of left-recursive productions in the grammar. Such a repetition creates a List of the individual results which I have so far drawn in the diagrams like an array, e.g. for the expression 1+2+3:

But the lists are really linked lists consisting of :: (cons) and Nil nodes, so we should draw the diagram like this:

Well, it looks like those repetitions are still right-recursive after all! But we are able to process them in a left-associative way in the interpreter for n-ary operators by starting with a 0 and left-folding the values computed from the list elements with the + operator:

  def compute(e: Expression): Int = e match {
    ...
    case Add(exprs) => exprs.foldLeft(0) { _ + compute(_) }
  }

Using an initial zero value here was for brevity’s sake only. All n-ary Add operations contain at least two elements, so exprs can never be empty and we do not really need the 0. Instead we can start with the first element’s value and just fold the rest:

    case Add(e :: l) => l.foldLeft(compute(e)) { _ + compute(_) }

The same approach allows us to construct a left-recursive AST from a list of Expression nodes directly in the parser. Instead of computing the values of the two sub-expressions and performing the addition, we just create a binary Add expression from them:

  lazy val sum = product ~ (("+" ~> product)*) ^^ {
    case e ~ l => l.foldLeft(e)(Add)
  }

This gives us the AST that we want:

Turning chains of expressions in the form e ~ ((op ~ e)*) into a left-recursive tree is a common operation in many parsers. Even in the tiny Fun0 language we need almost the same code again for the product production:

  lazy val product = simpleExpr ~ (("+" ~> simpleExpr)*) ^^ {
    case e ~ l => l.foldLeft(e)(Mult)
  }

If we have to distinguish between several operators, it gets slightly more difficult, but the basic pattern is still the same:

  lazy val sum = product ~ ( ( ("+"|"-") ~ product)* ) ^^ {
    case e ~ l => l.foldLeft(e) {
      case (z, "+" ~ p) => Add(z, p)
      case (z, "-" ~ p) => Subtract(z, p)
    }
  }

Deducing The chainl1 Combinator

When you’re using a parser generator, you have to implement everything in terms of the built-in abstractions. It’s a bit like procedural programming: You can write all kinds of functions (productions in this case) that work on data but you cannot write new abstractions. With parser combinators and Scala’s flexible syntax and support for functional programming, you can. We’ve already developed some basic elements from EBNF in this series but we don’t have to stop there.

Let’s see how we can abstract the parsers for chains of left-associative operators into a reusable combinator by looking at the differences in our implementations of sum (with one or two operators) and product:

There is a parser p1 of type Parser[Expression] which parses the operands. The result of the combinator is also a Parser[Expression]. In our examples, p1 is either product or simpleExpr. Of course, there shouldn’t be any code specific to expressions in the combinator, so we’ll use a type variable A and make p1 a Parser[A].
A combining function f of type (A,A) => A is used for folding the results of parsing the operands. This is one of the Add, Subtract and Mult constructors.
A parser p2 parses the operators (”+”, “-” or “*”). It doesn’t produce a result per se but if we have different operators it determines which combining function to use, so we’ll make the combining function its result. That gives it a type of Parser[(A,A) => A].

Writing the combinator as a function of p1 and p2 is now straight-forward:

def chain[A](p1: => Parser[A], p2: => Parser[(A,A) => A]): Parser[A] =
  p1 ~ ( (p2 ~ p1)* ) ^^ {
    case e ~ l => l.foldLeft(e) { case (e1, f ~ e2) => f(e1,e2) }
  }

This allows us to construct the sum parser as:

  def sum = chain(product, "+" ^^ { _ => Add })

I mentioned above that chain is used frequently so it’s worth abstracting into a combinator. Actually, it is so useful that Scala’s parser combinator library provides it out of the box under the names chainl1 and * (as a binary operator), so we can just write:

  def sum = product * ("+" ^^ { _ => Add })

The ^^^ Combinator

The parser ("+" ^^ { _ => Add }) which returns a fixed value Add still looks a bit awkward. Fortunately, there is the ^^^ combinator as a shortcut for such cases which allows us to write:

  def sum = product * ("+" ^^^ Add)

A version for multiple operators is equally simple:

  def sum = product * ("+" ^^^ Add | "-" ^^^ Subtract)

The Fun1 Language

We have dealt long enough with operators in Fun0 now and it is time to move on from this simple calculator to a real functional programming language with variables, first-class functions and conditional expressions. We’ll call this language Fun1. I will provide only an informal specification of it for now:

A Fun1 program consists of a sequence of expressions separated by semicolons. Expressions can be nested arbitrarily and grouped with parentheses, as in Fun0. The following kinds of expressions are allowed:

String and Int literals
Operators +, -, * and / for arithmetic. The + operator also does string concatenation.
Conditional expressions of the form if e1 then e2 else e3 where e1 yields a boolean value
Variables
Lambda expressions (anonymous functions) with 0 or more parameters of the form \p1,p2,p3 -> expr
Variable definitions of the form let v = expr. When such a definition occurs in a sequence, the variable is visible to all following expressions in that sequence.
Function applications of the form e1(e2,e3,e4) with 0 or more arguments where e1 yields a function value
Equality checks of the form e1 = e2 which yield a boolean value

The syntactic conventions (e.g. case sensitivity, allowed characters in an identifier, syntax of literals, comments) are those of Scala’s StandardTokenParsers which we already used for Fun0. The supported types are integers, strings, functions and boolean values. A pre-defined println function can be used to print a single value to the console (and also return it to the caller).

Here is a simple program in Fun1:

  let add1 = \a,b -> a+b;
  println(add1(3,4))

Exercise: Create a parser for Fun1 which uses the following AST nodes:

  sealed abstract class Expression
  case class Sequence(l: List[Expression]) extends Expression
  case class Lambda(params: List[String], expr: Expression) extends Expression
  case class Let(ident: String, expr: Expression) extends Expression
  case class Var(ident: String) extends Expression
  case class Lit(v: Any) extends Expression
  case class App(expr: Expression, args: List[Expression]) extends Expression
  case class Add(e1: Expression, e2: Expression) extends Expression
  case class Subtract(e1: Expression, e2: Expression) extends Expression
  case class Mult(e1: Expression, e2: Expression) extends Expression
  case class Div(e1: Expression, e2: Expression) extends Expression
  case class Equal(e1: Expression, e2: Expression) extends Expression
  case class If(e1: Expression, e2: Expression, e3: Expression) extends Expression

What information do you need for the parser that is missing or unclear in the informal description? Feel free to ask for it in a comment or come up with a sensible answer on your own. I have omitted some details deliberately (and probably forgotten some others).

Let’s start with an interpreter in the same way as for Fun0: A function which takes an Expression parameter and matches on the kind of Expression to compute and return its value. We don’t need a separate method for evaluating statements anymore because there are none in Fun1 — Everything is an expression. Evaluating literals and arithmetic operators is basically the same as before except we now have to take into account that an expression may return any type of value and not just integers.

  def eval(e: Expression): Any = e match {
    case Lit(v) => v
    case Add(e1, e2) => (eval(e1), eval(e2)) match {
      case (v1:String, v2) => v1 + v2.toString
      case (v1, v2:String) => v1.toString + v2
      case (i1:Int, i2:Int) => i1 + i2
      case _ => error("'+' requires two Int values or at least one String")
    }
    case Subtract(e1, e2) => (eval(e1), eval(e2)) match {
      case (i1:Int, i2:Int) => i1 - i2
      case _ => error("'-' requires two Int values")
    }
    case Mult(e1, e2) => (eval(e1), eval(e2)) match {
      case (i1:Int, i2:Int) => i1 * i2
      case _ => error("'*' requires two Int values")
    }
    case Div(e1, e2) => (eval(e1), eval(e2)) match {
      case (i1:Int, i2:Int) => i1 / i2
      case _ => error("'/' requires two Int values")
    }
    // ...more cases...
  }

For the new equality expressions we can reuse Scala’s equality checks and return a plain Bool value:

    case Equal(e1, e2) => eval(e1) == eval(e2)

The conditional (if) expressions bring us back to the subject of evaluation strategies. In Fun0 we used call-by-value, i.e. in order to evaluate an operator expression, we first evaluated the operands and then applied the operator to the results. This is the default in most programming languages for most kinds of expressions and we will stick with it, but we have to make at least one exception: An if expression must evaluate either the then clause or the else clause but not both. Even if we did not have any side-effects (which we do, in form of the println function) we couldn’t just compute both clauses and drop the result of one of them because lambdas and function application allow us to create inifinite recursion, so evaluating the wrong side of an if expression might turn a terminating program into a non-terminating one. With that out of the way, the actual implementation is quite simple:

    case If(e1, e2, e3) => eval(e1) match {
      case b:Boolean => eval(if(b) e2 else e3)
      case _ => error("Not a boolean value in IF expression")
    }

For function calls, we first of all need a representation of function values. Since functions in Fun1 can have different numbers of parameters, we’ll make it a subtrait of the function type List[Any] => Any:

  sealed trait Func extends (List[Any] => Any)

We can now implement the println function as:

  case object Println extends Func {
    def apply(args: List[Any]) = args match {
      case v :: Nil => { println(v); v }
      case _ => error("println needs 1 argument")
    }
  }

For interpreting function applications we just evaluate all sub-expressions and then invoke the function:

    case App(expr, args) => eval(expr) match {
      case f:Func => f(args.map(a => eval(a)))
      case _ => error("Only functions can be applied")
    }

Environments

In order to do variable assignments (with let expressions) and lookups, we need an environment which contains a mapping of variable names to their current values. We’ll integrate this into the interpreter in a purely functional style now and save the implementation with mutable state for later. Let’s start with a type alias for the environment:

  type Environment = Map[String, Any]

Notice that the Map here is a scala.collection.immutable.Map which can not be modified but allows a new map to be created which is the same as the existing one plus some new (or modified) entries.

The environment can be changed by a program and different invocations of the eval function must be able to see those changes. The only way to accomplish this without mutable state is to pass the current environment into eval and to have it return a possibly new environment along with the computed value:

  def eval(e: Expression, env: Environment): (Any, Environment) = e match {
    // ...
  }

In the initial call to eval with the parser’s result we provide an environment which contains only the println function we defined above:

  eval(tree, Map("println" -> Println))

We have to modify the existing code to pass the environment down into recursive calls to eval, drop the possibly modified environments they return, and return the result together with the original one. For example, the case for Equal becomes:

    case Equal(e1, e2) => (eval(e1, env)._1 == eval(e2, env)._1, env)

The other cases can be rewritten similarly.

We can now use the environment for variable lookups and definitions. The code for Var just performs the lookup while Let returns a new environment with all previous definitions plus the new binding:

    case Var(id) =>
      (env getOrElse(id, error("Undefined var "+id)), env)
    case Let(id, ex)  =>
      { val v = eval(ex, env)._1; (v, env + ((id, v))) }

For a sequence we go through all the sub-expressions, evaluating each in the environment returned by the previous one and returning the last value + environment. Notice the initial value () (”unit”) which gets returned for an empty sequence.

    case Sequence(l) =>
      l.foldLeft(():Any, env) { (z,n) => eval(n, z._2) }

Lambda expressions are supposed to close over the variables which are visible at their point of creation, so we cannot just return a Lambda expression object directly as its own value. Instead we wrap the parameter list and expression of the Lambda together with the current environment in a Closure object:

    case Lambda(params, ex) => (Closure(ex, params, env), env)

Since the environment is immutable, there is no need to create a copy. We can just keep this value around and use it to apply the lambda function at a later time.

We have already defined the Func trait and implemented function application for it, so we make Closure implement it, too. In order to apply the lambda function, we take the saved environment, add the function’s parameters on top of it, and use this new environment for evaluating the saved expression:

  case class Closure(ex: Expression, params: List[String],
                     env: Environment) extends Func {
    def apply(args: List[Any]) = {
      if(args.length != params.length)
        error("Wrong number of arguments for function")
      eval(ex, env ++ params.zip(args))._1
    }
    override def toString =
      " " + ex + ">"
  }

This concludes the interpreter for Fun1 and it also brings me to the end of the content I had originally planned for this series almost one year ago. There are still lots of interesting topics left and I intend to pursue some of them in further installments but I have not yet decided what will be next. For now, I’ll leave you with a short Fun1 program to test your implementation:

// Lambda application -- prints "81"
println((\x -> x*x)(9));

// Shadowing s does not change the closure -- prints "foo!"
let s = "foo";
let ls = \x -> s + x;
let s = "bar";
println(ls("!"));

// N-ary and curried functions -- both print "7"
let add1 = \a,b -> a+b;
println(add1(3,4));
let add2 = \a -> \b -> a+b;
println(add2(3)(4));

// The Y combinator allows us to define recursive functions
let Y = \f ->
  (\x -> f(\y -> x(x)(y)))
  (\x -> f(\y -> x(x)(y)));

// Define a faculty function using Y -- prints "6! = 720"
let fact = Y(\fact -> \n -> if n = 0 then 1 else n*fact(n-1));
println("6! = "+fact(6));

// Print a nullary function "" and its value "42"
let nullary = \-> 42;
println(nullary);
println(nullary())

I’ll make the source code for my reference implementation available in the solutions to this part, coming up in a few days.

Dealing with NULLs in ScalaQuery

Stefan Zeiger — Sat, 20 Jun 2009 14:56:42 +0000

Previously, ScalaQuery used nullable Java types for all columns. This is straight-forward for reference types like java.lang.String which are mapped 1:1 to Scala types but not for Scala’s value types like scala.Int or scala.Boolean. Although they are often implemented in terms of Java types like java.lang.Integer and java.lang.Boolean, there is no 1:1 mapping and there are no aliases for those wrapper types in Scala. Most of the time this is not a problem because you can use Scala’s value types everywhere and have the compiler generate optimized code with primitive types where applicable.

Except there is one thing you cannot do with Scala’s value types: They may never be null. The reference types are nullable but that’s probably not a good idea, either. Scala needs nullable reference types for interfacing with Java code but for native Scala APIs, the Option type is the preferred solution. It makes None values explicit and thus prevents NullPointerExceptions at runtime when a null value is used in a place where it shouldn’t be allowed.

SQL databases also use NULL values for various things (e.g. missing values, undefined values, horrid abominations) so this should work nicely with Scala’s Option type. I rejected Options for database results in ScalaQuery at first because most columns are not nullable (so you wouldn’t want to have all columns return an Option instead of just the nullable ones) yet some operations like outer joins need to convert non-nullable columns to nullable ones (which can probably not be typed in Scala's type system, so you'd have to make all columns return an Option).

On the other hand, having nullable java.lang.Integer and other Java wrapper types in a Scala API is rather ugly, so I finally removed them in favor of a more practical yet null-free solution. All columns now use proper Scala types (scala.Int, scala.Predef.String, etc.). NULLs from the database are returned as a default value (0 for Int, an empty string for String) which can be changed with the orElse method, e.g.:

  for(u <- Users)
    yield u.first.orElse("no first name")
        ~ u.last.orElse("no last name")

Note that this value is only used when extracting rows from the result set on the client side. It does not change the handling of NULLs inside the database server!

For nullable types like scala.Predef.String you can use orElse(null) to get the old behaviour back. The argument of orElse() is passed by name, so you can also run an arbitrary block of code or throw an exception when a NULL value is encountered in the result set. There is a convencience method called orFail which does just that. Maybe this should be the default instead of returning a zero value?

The preferred solution for distinguishing between regular values and NULL for any type is to use the ? method to turn a SimpleColumn of type T into one of type Option[T]:

  for(u <- Users) yield u.first.? ~ u.last.?

Currently there are no operations defined on option columns so it does not make sense to declare an option column directly as part of a table. Ideally, columns of types T and Option[T] should be fully interoperable because the database server uses three-valued logic anyway.

I have just pushed the new code to the master branch and updated the stable branch to include the previously redesigned AST.