szeiger.de » ScalaQuery

New ScalaQuery Features

Stefan Zeiger — Mon, 21 Dec 2009 20:15:03 +0000

It’s been almost 3 months since my last summary of new features in ScalaQuery. I implemented some important changes in the following weeks but I only had little time to work on ScalaQuery after my extended one-month vacation ended in October.

Build system

Builds against newer versions of Scala 2.8 have been going well thanks to sbt 0.6 which separates the Scala version used by the build system from the version used by the project being built. You need to use at least version 0.6.7 of the new xsbt launcher to build ScalaQuery. Other dependencies are downloaded automatically by sbt. I wrote an implementation of sbt’s test interface for JUnit which you can find here (binaries are here on scala-tools.org). It allows you to run ScalaQuery’s JUnit test cases from sbt with the test command.

All published artifacts now contain the Scala version in the artifact ID. You can find the latest ScalaQuery snapshot in scala-tools.org’s snapshots repository as:

groupId:: com.novocode
artifactId:: scala-query_2.8.0.Beta1-RC4
version:: 1.0.0-SNAPSHOT

A ScalaQuery Update

Stefan Zeiger — Sun, 27 Sep 2009 00:03:30 +0000

Since my last post about ScalaQuery one month ago I have wasted quite some time trying to implement the OptionMapper type (which is used to make operations on columns interoperable between base types and Option types) in a way which does not require 2^n implicit functions and 2*n+2 type parameters for operations with n operands. The composable OptionMapper itself is not hard to write (see NewOptionMapper here) but it requires more type fidelity than ScalaQuery offers at the moment: Either Projections need to preserve the subtypes of Columns they are storing, or the required types have to be reconstructed from implicit values where they are needed. You can find my failed attempts in the projection-types and new-option-mapper branches. The first one might actually work once Scala bug #2346 is resolved. The second one would require a much more powerful type inferencer. For now, I just added an OptionMapper3 for 3 parameters (which is needed by the BETWEEN operator).

There are also some interesting new features in ScalaQuery:

Mapped Projections

You can now create a bidirectional mapping of a Projection with the <> operator. This is especially useful for a table’s “*” projection. The <> operator takes a function from the projection tuple type T (either as a tuple or as multiple parameters) to the mapped type R, and a function from R back to Some[T]. The asymmetry with the unnecessary Some wrapper is deliberate, matching exactly the apply und unapply methods of a case class. For example, here is a simple User class and table:

  case class User(first: String, last: String)

  object Users extends Table[User]("users") {
    def first = column[String]("first", O NotNull)
    def last = column[String]("last", O NotNull)
    def * = first ~ last <> (User, User.unapply _)
  }

Whenever you perform an insert, update or query directly on such a table (instead of a projection of individual columns), you can use the mapped type:

  Users.insert(User("Stefan", "Zeiger"))
  val someUsers: List[User] = Users.where(_.first startsWith "S").list

Finders

The new convenience method Table.createFinderBy allows you to create a simple finder query template which selects values from a table by matching on a single column, e.g.:

  val findUserByFirstName = Users.createFinderBy(_.first)

This is equivalent to:

  val findUserByFirstName = for {
    first <- Parameters[String]
    u <- Users if u.first is first
  } yield u

Updatable ResultSets

JDBC allows you to modify a ResultSet which was created with the CONCUR_UPDATABLE result set concurrency. ScalaQuery now offers a high-level interface to this functionality with the mutate method, e.g.:

  val q1 = for(u <- Users if u.last.is("Simpson") || u.last.is("Bouvier")) yield u
  q1.mutate { m =>
    if(m()._3 == "Bouvier") m() = m().copy(_3 = "Simpson")
    else if(m()._2 == "Homer") m.delete()
    else if(m()._2 == "Bart") m.insert((None, "Lisa", "Simpson"))
  }

Statement Options

The new ResultSetType, ResultSetConcurrency and ResultSetHoldability classes allow you to request the corresponding JDBC options when ScalaQuery creates a PreparedStatement, e.g.:

  myDB withSession {
    ResultSetType.ScrollInsensitive {
      // All database calls in this block use TYPE_SCROLL_INSENSITIVE
    }
  }

Or if you don't want to use the implicit threadLocalSession:

  myDB withSession { s1 =>
    ResultSetType.ScrollInsensitive(s1) { s2 =>
      // All database calls on s2 use TYPE_SCROLL_INSENSITIVE
    }
  }

The default for all three options is Auto which gives you values suitable for the invoker method you are calling (e.g. ResultSetConcurrency.Updatable for mutate and ResultSetConcurrency.ReadOnly for all other methods).

Build System

I have rewritten most of the test classes as proper unit tests (using JUnit 4) with assertions. I'd like to investigate ScalaTest and specs further (possibly together with ScalaCheck) in the future to replace JUnit but the binary incompatibilities between different Scala 2.8 snapshot builds would only complicate the build process right now. If you build with sbt, you still need to download a suitable Scala 2.8 snapshot yourself and set the paths in the properties because the build is done in forking mode. This also means that you cannot use the "test" command to run the unit tests from sbt at the moment.

H2 and JUnit are declared as test dependencies in the sbt project definition and the Eclipse build path refers to the local copies in lib_managed. If you want to build ScalaQuery with Eclipse (as I usually do), just run "sbt update" once to pull in the required JARs.

I am using FreeMarker templates and the FMPP tool to create repetitive code for the Projection and Parameters classes. This is not yet integrated into the sbt build process. You can use the supplied Eclipse launcher instead (after downloading FMPP and pointing the launcher to the correct path). The generated sources are checked in, so you only need to do this if you want to change the templates and rebuild the Scala sources.

Profiles, Drivers & More

Stefan Zeiger — Thu, 27 Aug 2009 15:07:47 +0000

It’s been three weeks already since my last post about ScalaQuery so I’d like to provide an update on some recent changes.

Profiles & Drivers

The biggest news is that ScalaQuery now supports database engine-specific features. Feature sets can be bundled in profiles which are then implemented by drivers (Of course, a driver can also add driver-specific features which are not part of a profile). All previously existing features are now part of the BasicProfile which should work on all SQL databases with little or no customization. The BasicProfile provides:

An object called Implicit with all implicit conversions which are required to use the profile’s features (including an implicit value of type BasicProfile which points to the driver implementing the profile).
Some methods for creating various statements and query templates. They have default implementations provided by the profile but can be overridden by other profiles or drivers. You do not usually call them directly.
A number of TypeMapperDelegate objects for the basic types supported by the profile. You still use TypeMappers defined outside the profile but they do not implement the actual type mapping any more. Instead they ask the profile for a delegate implementation. (Alternatively, I could have moved the existing TypeMapper objects entirely into the profile but mappers for some basic types like Int and Boolean are used internally at many places. They would need to be threaded through several methods and constructors, thus complicating the implementation unnecessarily.)

The BasicProfile is implemented by the BasicDriver.

There is also a new ExtendedProfile for features which are expected to be available in every commonly used SQL database system, albeit with a non-standard syntax. It currently provides string concatenation with the ++ operator (plus startsWith and endsWith methods which can be implemented with LIKE and string concatenation) and the take and drop methods needed for pagination.

If you only need to support a single database engine in your application, you can forget about profiles and just import a specific driver’s implicit conversions. Supporting multiple drivers is almost as simple: Give your DAO class (or whatever you have in your application design) a parameter of the required profile type and import this profile’s implicits. You can then call it with any driver which implements the profile. For example:

  def test(profile: ExtendedProfile) {
    import profile.Implicit._
    println("Using driver: "+profile.getClass.getName)
    val q1 = Users.where(_.name startsWith "quote ' and backslash \\").take(5)
    println(q1.selectStatement)
    val q2 = Users.where(_.name startsWith "St".bind).drop(10).take(5)
    println(q2.selectStatement)
    val q3 = Query(42 ~ "foo")
    println(q3.selectStatement)
    println
  }

  def main(args: Array[String]) {
    test(H2Driver)
    test(OracleDriver)
    test(MySQLDriver)
  }

This prints the different statements required for H2, Oracle and MySQL:

Using driver: com.novocode.squery.combinator.extended.H2Driver$
SELECT t1.name FROM users t1 WHERE (t1.name like 'quote '' and backslash \%') LIMIT 5
SELECT t1.name FROM users t1 WHERE (t1.name like (? || '%')) LIMIT 5 OFFSET 10
SELECT 42,'foo'

Using driver: com.novocode.squery.combinator.extended.OracleDriver$
SELECT * FROM (SELECT t1.name FROM users t1 WHERE (t1.name like 'quote '' and backslash \%')) WHERE ROWNUM <= 5
SELECT * FROM (SELECT t0.*, ROWNUM ROWNUM_O FROM (t1.name,ROWNUM ROWNUM_I FROM users t1 WHERE (t1.name like (? || '%'))) t0) WHERE ROWNUM_O BETWEEN (1+10) AND (10+5) ORDER BY ROWNUM_I
SELECT 42,'foo' FROM DUAL

Using driver: com.novocode.squery.combinator.extended.MySQLDriver$
SELECT t1.name FROM users t1 WHERE (t1.name like 'quote \' and backslash \\%') LIMIT 5
SELECT t1.name FROM users t1 WHERE (t1.name like concat(?,'%')) LIMIT 10,5
SELECT 42,'foo' FROM DUAL

Updates

You can now take a simple query (returning only named columns from a single table; no modifiers except WHERE restrictions) and call it as an UPDATE statement:

  val q = for(u <- Users if u.id is 42) yield u.first ~ u.last
  q.update("foo", "bar")

Filtering

You may already have noticed in my previous post that the Query class now has a filter method which works like where for functions returning a Column[Boolean] or Column[Option[Boolean]], so you can use Scala’s standard if clauses in for-comprehensions on queries. Unlike where, filter also accepts functions returning a plain Boolean value to enable the use of refutable patterns in queries (e.g. when destructuring a Join).

Invokers

Result type remapping and parameter application are now available for all Invokers. These features were pushed down from the statement invokers for monadic queries into the invoker framework. Query templates are parameterized invokers now. They can be applied like any other invoker.

Efficient Parameterized Queries in ScalaQuery

Stefan Zeiger — Thu, 06 Aug 2009 17:22:04 +0000

One question about ScalaQuery which keeps coming up is that of perfomance. The question is usually in the form “How does it compare to JDBC?” but that’s like comparing apples and, well, apple trees. After all, ScalaQuery is a layer on top of JDBC which provides mainly two things:

A nicer, more Scala-like way of handling database connections, performing queries and reading result sets. This is not optional when you access a database in your application. You will wrap SQL statement execution and result set reading in some way or another to abstract from the low-level JDBC API. Anyway, the overhead here is quite low.
A way of composing queries with an internal DSL based on a query monad and combinators. That’s the part I want to talk about in this post.

If you want to optimize the query generation away, you can always fall back to the StaticQuery and DynamicQuery classes. These work a bit like iBATIS, except your SQL code is embedded directly in your Scala code and not in some XML files. But you’re still writing SQL! That’s nice for the special cases which are not covered by the combinator queries and which do not need to be composable but it’s probably not the reason why you want to use ScalaQuery in the first place.

When you’re constructing a query in the query monad, you normally have some variables which are used in the query like this:

def userNameByID(id: Int) =
  for(u <- Users if u.id is id) yield u.first

This would insert the user ID directly into the SQL statement, thus requiring a new statement to be generated by ScalaQuery and parsed and optimized by the database server (which can be very expensive) for every invocation of the query. We can improve this by using a bind variable for the user ID:

def userNameByID(id: Int) =
  for(u <- Users if u.id is id.bind) yield u.first

Now the generated SQL statement is always the same (e.g. “SELECT t1.first FROM users t1 WHERE (t1.id=?)“), so the database server needs to calculate an execution plan for it only once and can reuse it on subsequent invocations. But the query is still constructed as a tree of dozens of objects and compiled to the same SQL statement every time you call userNameByID.

This can be remedied with query templates, a recent addition to ScalaQuery:

val userNameByID = for {
  id <- Parameters[Int]
  u <- Users if u.id is id
} yield u.first

If you recall how for-comprehensions are desugared, this gets translated to Parameters[Int].apply(...).flatMap(id => ...). The apply method on the Parameters object takes an implicit TypeMapper for each parameter type you specify and creates a Parameters instance. The flatMap method on Parameters takes a function which creates a Query and returns a QueryTemplate for it:

final class QueryTemplate[P, R](query: Query[ColumnBase[R]]) {
  def apply(param: P) = new AppliedQueryTemplate(built, param, query.value)
  lazy val built = QueryBuilder.buildSelect(query, NamingContext())
}

final class Parameters[P, C](c: C) {
  def flatMap[F](f: C => Query[ColumnBase[F]]): QueryTemplate[P, F] =
    new QueryTemplate[P, F](f(c))
  ...
}

object Parameters {
  def apply[P1](implicit tm1: TypeMapper[P1]) =
    new Parameters[P1, Column[P1]](new ParameterColumn(-1, tm1))
  ...
}

Note that, unlike Query, neither Parameters nor QueryTemplate is a monad. You cannot compose multiple parameter lists or query templates this way but by providing a suitable flatMap method, the parameters can be used as the first generator in a for-comprehension which otherwise operates in the Query monad.

Either one of the userNameByID functions/methods defined above can be used in the same way:

for(t <- userNameByID(3)) println(t)

When you apply the parameters to a QueryTemplate, you get an AppliedQueryTemplate which can be lifted to a suitable Invoker by an implicit conversion (just like a Query). The first time you do this, the SQL code gets generated and then cached in the QueryTemplate for further applications.

If you specify more than one type parameter, the Parameters generator gives you a Projection instead of a single Column. It can be unpacked either with the extractor on the “~” object:

val userNameByIDRangeAndProduct = for {
  min ~ max ~ product <- Parameters[Int, Int, String]
  u <- Users if u.id >= min &&
    u.id <= max &&
    Orders.where(o => (u.id is o.userID) && (o.product is product)).exists
} yield u.first

…or with the Projection extractor (which is slightly more efficient but not as nice to read):

val userNameByIDRangeAndProduct = for {
  Projection(min, max, product) <- Parameters[Int, Int, String]
  u <- Users if u.id >= min &&
    u.id <= max &&
    Orders.where(o => (u.id is o.userID) && (o.product is product)).exists
} yield u.first

ScalaQuery’s invoker framework provides methods for invoking queries with parameters but these are currently used by the simple queries only. I expect to integrate query templates with this system in the future.

ScalaQuery for different Scala versions

Stefan Zeiger — Wed, 22 Jul 2009 20:12:11 +0000

I have just created a new scala-2.7 branch for ScalaQuery. My original plan was to target only Scala 2.8 but since I’ve made lots of progress during the last few weeks and I’ve seen increased interest in ScalaQuery, I tried to build it with 2.7.5.

I had to change the semantics of SimpleFunction and SimpleBinaryOperator for 2.7 but I prefer the new version anyway, so it went into the main line. The code on the scala-2.7 branch is currently identical to the master branch, except for the test classes which are different in two regards:

Although the Scala Language Specification mandates that the part left to the “<-” in a for comprehension is a pattern, the wildcard pattern “_” does not work in 2.7. I have changed it to a dummy variable named “__“.
The type inferencer in 2.7 cannot infer the correct type for the implicit OptionMapper objects. OptionMapper[_,_,_,_] has four type parameters, the last one being used for the return type of functions which use the mapper, so it is not yet known when looking for an implicit mapper and gets inferred as Nothing. Scala 2.8 apparently knows that this type parameter is undetermined, finds the single matching implicit object for the other three parameters and then fills in the fourth. The type-correct interoperability of option and non-option types in ScalaQuery relies heavily on the improved type inferencer and I don’t see any way of making it work nicely with 2.7. The work-around is to add type annotations to the boolean operators, e.g. a && b might become a.&&[Boolean,Option[Boolean]](b). Yuck!

I’m still focused on Scala 2.8 as a target platform and will probably not spend much time integrating new features into the scala-2.7 branch but contributions are always welcome.

Implicits on Implicits

Stefan Zeiger — Thu, 16 Jul 2009 22:51:55 +0000

I have recently made a change to ScalaQuery which enables the use of any type for a column without needing specific Column classes and factory methods on Table for it. For this I used an approach which I had not considered earlier and which I wasn’t even sure would work.

Scala (at least in version 2.7) only considers single function calls for implicit conversions. Take the following definitions for example:

class A
class B(a: A)
class C(b: B)

implicit def aToB(a: A) = new B(a)
implicit def bToC(b: B) = new C(b)

val a = new A

def useB(b: B) = ...
def useC(c: C) = ...

You can call useB(a) because the compiler finds the implicit conversion aToB(a) but you cannot call useC(a) — the compiler does not try the chained call bToC(aToB(c)).

But this does not mean that an implicit conversion may not rely on an implicit value! The following works just fine:

trait Column[T]
class ConstColumn[T](value: T, tm: TypeMapper[T]) extends Column[T]

implicit def valueToColumn[T](value: T)(implicit tm: TypeMapper[T]) =
  new ConstColumn(value, tm)

trait TypeMapper[T]
implicit object IntTypeMapper extends TypeMapper[Int]

val c1: Column[_] = 42

The Scala compiler recognizes that valueToColumn[Int] provides the desired conversion from Int to Column[_] and then looks for the required implicit TypeMapper[Int] value, which is available through the implicit IntTypeMapper object.

In ScalaQuery, the real TypeMapper contains only a few methods and there are predefined TypeMappers for most of the basic types used by JDBC. That’s nice because it removes some redundancy from my code base but the better news is that it allows you to write your own TypeMappers. For example, if you wanted to get the java.lang.Integer columns back which I removed in favor of Int and Option[Int], you could add this implicit object to your code:

implicit object IntegerTypeMapper extends TypeMapper[java.lang.Integer] {
  def zero = null
  def sqlType = java.sql.Types.INTEGER
  def setValue(v: java.lang.Integer, p: PositionedParameters) =
    if(v eq null) p.setIntOption(None) else p.setIntOption(Some(v.intValue))
  def setOption(v: Option[java.lang.Integer], p: PositionedParameters) = v match {
    case Some(null) => p.setIntOption(None)
    case Some(i) => p.setIntOption(Some(i.intValue))
    case None => p.setIntOption(None)
  }
  def nextValue(r: PositionedResult) = r.nextIntOption match {
    case None => null
    case Some(i) => java.lang.Integer.valueOf(i)
  }
}

The same should work for any database engine- or domain-specific types.

Dealing with NULLs in ScalaQuery

Stefan Zeiger — Sat, 20 Jun 2009 14:56:42 +0000

Previously, ScalaQuery used nullable Java types for all columns. This is straight-forward for reference types like java.lang.String which are mapped 1:1 to Scala types but not for Scala’s value types like scala.Int or scala.Boolean. Although they are often implemented in terms of Java types like java.lang.Integer and java.lang.Boolean, there is no 1:1 mapping and there are no aliases for those wrapper types in Scala. Most of the time this is not a problem because you can use Scala’s value types everywhere and have the compiler generate optimized code with primitive types where applicable.

Except there is one thing you cannot do with Scala’s value types: They may never be null. The reference types are nullable but that’s probably not a good idea, either. Scala needs nullable reference types for interfacing with Java code but for native Scala APIs, the Option type is the preferred solution. It makes None values explicit and thus prevents NullPointerExceptions at runtime when a null value is used in a place where it shouldn’t be allowed.

SQL databases also use NULL values for various things (e.g. missing values, undefined values, horrid abominations) so this should work nicely with Scala’s Option type. I rejected Options for database results in ScalaQuery at first because most columns are not nullable (so you wouldn’t want to have all columns return an Option instead of just the nullable ones) yet some operations like outer joins need to convert non-nullable columns to nullable ones (which can probably not be typed in Scala's type system, so you'd have to make all columns return an Option).

On the other hand, having nullable java.lang.Integer and other Java wrapper types in a Scala API is rather ugly, so I finally removed them in favor of a more practical yet null-free solution. All columns now use proper Scala types (scala.Int, scala.Predef.String, etc.). NULLs from the database are returned as a default value (0 for Int, an empty string for String) which can be changed with the orElse method, e.g.:

  for(u <- Users)
    yield u.first.orElse("no first name")
        ~ u.last.orElse("no last name")

Note that this value is only used when extracting rows from the result set on the client side. It does not change the handling of NULLs inside the database server!

For nullable types like scala.Predef.String you can use orElse(null) to get the old behaviour back. The argument of orElse() is passed by name, so you can also run an arbitrary block of code or throw an exception when a NULL value is encountered in the result set. There is a convencience method called orFail which does just that. Maybe this should be the default instead of returning a zero value?

The preferred solution for distinguishing between regular values and NULL for any type is to use the ? method to turn a SimpleColumn of type T into one of type Option[T]:

  for(u <- Users) yield u.first.? ~ u.last.?

Currently there are no operations defined on option columns so it does not make sense to declare an option column directly as part of a table. Ideally, columns of types T and Option[T] should be fully interoperable because the database server uses three-valued logic anyway.

I have just pushed the new code to the master branch and updated the stable branch to include the previously redesigned AST.

Public GIT repo for ScalaQuery

Stefan Zeiger — Sat, 21 Feb 2009 13:27:55 +0000

The type-safe database query library/DSL for Scala which I introduced in this post is now available in a public GIT repository at GitHub: http://github.com/szeiger/scala-query/tree

I have renamed it to ScalaQuery because SQuery and all other short names I could think of were already used by numerous other projects.

There are several changes compared to the version I used in the introduction:

The combinator/monad-based query classes are now in their own package com.novocode.squery.combinator.
The invokers (which execute a typed query through JDBC) of the combinator queries and the simple SQL queries (in com.novocode.squery.simple) have been unified.
Some minor renaming and refactoring.

A Type-Safe Database Query DSL for Scala

Stefan Zeiger — Sun, 21 Dec 2008 15:13:39 +0000

When the topic of JDBC wrappers came up on the Scala mailing list a few weeks ago, I mentioned that…

I’d prefer a type-safe DSL for database queries but that’s more
complicated. I’m still experimenting with that, too

I think those experiments worked rather well and the code is now in a state where the major features are usable and I can show some example code which compiles and runs successfully as a proof of concept. Many features (like more SQL operators, more data types and column aliasing) are missing but at least I am confident that it is possible to implement them on this basis.

Just like HaskellDB and LINQ to SQL, I am using a query monad which collects the projections and restrictions on database tables in a composable way to build an SQL statement for a query while producing the correct type for that statement’s result in Scala’s type system. Don’t worry if that sounds too abstract, there will be plenty of example code.

The basic scaffolding for the examples looks like this:

import java.lang.Integer
import com.novocode.squery._
import com.novocode.squery.Implicit._
import com.novocode.squery.session._
import com.novocode.squery.session.SessionFactory._

object SQuery2Test2 {
  def main(args: Array[String]) {
    val sf = new DriverManagerSessionFactory(
      "org.h2.Driver", "jdbc:h2:mem:test1")
    sf withSession {
      // Database access code goes here
    }
  }
}

The session._ and SessionFactory._ imports provide a thin wrapper around JDBC connections with session management and query string caching. I am using the H2 Database Engine here because it is very easy to set up for a local in-memory database (just include a single JAR in the classpath and use the driver name and URL from above). SessionFactory contains an implicit session object that can be used in a withSession block. The squery package contains the actual classes and traits needed for the type-safe queries. I will explain the implicit conversions from Implicit._ as we go along.

For every table we want to use in a query, we need a corresponding Table object. Let’s define two tables for Users of some shopping application and the Orders they placed:

object Users extends Table[(Integer, String, String)]("users") {
  def id = intColumn("id", O.AutoInc, O.NotNull)
  def first = stringColumn("first")
  def last = stringColumn("last")
  def * = id ~ first ~ last
}

object Orders extends Table[(Integer, Integer, String)]("orders") {
  def userID = intColumn("userID", O.NotNull)
  def orderID = intColumn("orderID", O.AutoInc, O.NotNull)
  def product = stringColumn("product")
  def * = userID ~ orderID ~ product
}

Like every object which can be used in or returned from a query, a table is parameterized with the type of the values it represents. This is always a tuple of individual column types, in our case Integers and Strings (note the use of java.lang.Integer instead of Int; more about that later). In this respect SQuery (as I’ve named it for now) is closer to HaskellDB than to LINQ because Scala (like most languages) does not give you access to an expression’s AST at runtime. In LINQ you can write queries using the real types of the values and columns in your database and have the query expression’s AST translated to SQL at runtime. Without this option we have to use meta-objects like Table and Column to build our own AST from these.

The rest of the table definitions should be quite obvious if you’ve ever worked with an SQL database. Each table gets a name (”users” and “orders”) and the individual columns consist of a name, a type (depending on the method by which they are created, like intColumn and stringColumn) and options like AUTO_INCREMENT and NOT NULL.

Each table must define the special “*” column which is a projection of all of the table’s columns in their preferred order. The type of this projection is statically enforced to be the same as the table’s (i.e. a Table[T] contains a “*” method of type ConvertableColumn[T]. There are separate projection classes for different numbers of columns just like Scala has different tuple types. You can use the projection operator “~” to build projections incrementally:

trait SimpleColumn[T] extends ConvertableColumn[T] {
  def ~[U](b: SimpleColumn[U]) = new Projection2[T, U](this, b)
}

sealed trait Projection[T <: Product]
  extends ConvertableColumn[T] with Product { ... }

final class Projection2[T1,T2](
  override val _1: SimpleColumn[T1],
  override val _2: SimpleColumn[T2])
extends Tuple2(_1,_2) with Projection[(T1,T2)] {
  def ~[U](c: SimpleColumn[U]) = new Projection3(_1,_2,c)
}

//... and so on with Projection3, Projection4, etc.

Before we get to the queries, we should define our tables in the database. In practice you will probably build the tables first in the database and then write the required meta-objects but it can also be done directly from the Table objects:

Users.createTable
Orders.createTable

The createTable method is not defined on the tables but on the DDLInvoker class which can be instantiated automatically with an implicit conversion:

object Implicit {
  implicit def tableToDDLInvoker[T](t: Table[T]): DDLInvoker[T] =
    new DDLInvoker(t)
  ...
}

class DDLInvoker[T](table: Table[T]) {
  lazy val createTableStatement =
    new DDLBuilder(table).buildCreateTable
  def createTable(implicit session: Session) = ...
}

createTable uses the implicit session I mentioned earlier. If you prefer to make it explicit you can leave out the SessionFactory._ import and use the alternate withSession method:

sf withSession { session =>
  Users.createTable(session)
  Orders.createTable(session)
}

All other database access methods work the same way. I will always use the implicit version in the examples from now on.

You can also print out the generated SQL statements:

> println(Users.createTableStatement)
CREATE TABLE users (id INT NOT NULL AUTO_INCREMENT,first VARCHAR,last VARCHAR)

> println(Orders.createTableStatement)
CREATE TABLE orders (userID INT NOT NULL,orderID INT NOT NULL AUTO_INCREMENT,product VARCHAR)

Let’s insert some users into the database:

val ins1 = (Users.first ~ Users.last).insert("Homer", "Simpson")
val ins2 = (Users.first ~ Users.last).insertAll(
  ("Marge", "Simpson"),
  ("Apu", "Nahasapeemapetilon"),
  ("Carl", "Carlson"),
  ("Lenny", "Leonard") )
val ins3 = Users.first.insertAll(
  "Santa's Little Helper",
  "Snowball" )
println("Inserted "+(ins1+ins2+ins3)+" users")

You can use any single column or projection of columns to insert values of the corresponding type (which is a tuple for projections). All columns of a projection must be from the same table but this is not statically enforced.

Now that we have some content in the database, we can perform queries on it. This is about the simplest SQL query you can write:

SELECT * from users

You can do the same in Scala with a for-comprehension:

val q1 = for(u <- Users) yield u

This is equivalent to

val q1 = Users map { u => u }

There is no map method defined on tables but the table gets lifted automatically to a query with another implicit conversion:

object Implicit {
  implicit def tableToQuery[T <: TableBase.T_](t: T) =
    Query(t.withOp(new ColumnOp.BaseTableQueryOp(t)))
  ...
}

The mapping itself is just an identity mapping, so we might as well have written the query as tableToQuery(Users) to get a Query[Users] without any restrictions.

Note the use of t.withOp(...) in tableToQuery. What we want to get from tableToQuery is a node for our query AST which indicates a uniquely named table. This is needed for queries which use multiple instances of the same table, for example to get the latest version of every document in a table documents(id, version, content) you could use:

SELECT content FROM documents t1
WHERE t1.version = (
  SELECT max(t2.version) FROM documents t2
  WHERE t2.id = t1.id
)

new BaseTableQueryOp(t) provides such a unique table instance but on the outside it doesn’t “look like” the table it wraps. We need an instance of the same type as the original table which is marked as being a BaseTableQueryOp but otherwise behaves like that table. This is what the trait WithOp provides in a rather un-functional and hackish way by cloning the receiver object and setting the clone’s op field to the given object (the actual AST node):

trait WithOp[O >: Null] extends Cloneable {
  def withOp(op: O): this.type = {
    val t = clone
    t._op = op
    t
  }
  private[WithOp] var _op: O = _
  final def op: O = _op
  override def clone(): this.type =
    super.clone.asInstanceOf[this.type]
}

object WithOp {
  def unapply[O >: Null](w: WithOp[O]) =
    if(w.op == null) None else Some(w.op)
}

(Unlike this version the current implementation in SQuery is not generic because of bug #1434.)

We can print out the SQL statement for the query like this:

> println(q1.selectStatement)
SELECT t1.id,t1.first,t1.last FROM users t1

This is just what you’d expect (taking into account that SQuery does not use “*” and aliases all tables). The query does not contain a selectStatement field but again there’s an implicit conversion to a QueryInvoker with that field:

object Implicit {
  implicit def queryToQueryInvoker[T](q: Query[ConvertableColumn[T]]): QueryInvoker[T,T] = QueryInvoker(q)
  ...
}

class QueryInvoker[T,R] private (q: Query[ConvertableColumn[T]], mapper: T => R) {
  lazy val selectStatement = new QueryBuilder(q).buildSelect
  def first(implicit session: Session): Option[R] = ...
  def list(implicit session: Session): List[R] = ...
  def foreach(f: R => Unit)(implicit session: Session): Unit = ...
  def elements(implicit session: Session): CloseableIterator[R] = ...
  def mapResult[U](f: (R => U)) = new QueryInvoker[T,U](q, { v:T => f(mapper(v)) })
}

object QueryInvoker {
  def apply[T](q: Query[ConvertableColumn[T]]) = new QueryInvoker[T,T](q, { v:T => v })
}

Note that you cannot convert just any Query[_], it has to be a Query[ConvertableColumn[_]]. This is the main reason for separating queries and invokers. The same implicit conversion allows you to use a foreach loop to iterate through all elements selected by a query:

> for(t <- q1) println("User tuple: "+t)
User tuple: (1,Homer,Simpson)
User tuple: (2,Marge,Simpson)
User tuple: (3,Apu,Nahasapeemapetilon)
User tuple: (4,Carl,Carlson)
User tuple: (5,Lenny,Leonard)
User tuple: (6,Santa's Little Helper,null)
User tuple: (7,Snowball,null)

The mapResult method can be used to apply a mapping function to every row which is read from the database. Unlike a projection in the query which gets translated to SQL and executed on the database server this mapping is applied on the client side. The Query class has two type parameters for the type returned by the database query and the mapped type. Let’s define a case class for User objects and use a mapped QueryInvoker to read a list of them from the database:

case class User(id: Integer, first: String, last: String)

val allUsers = q1.mapResult {
  case (id,f,l) => User(id,f,l)
}.list
for(u <- allUsers) println("User object: "+u)

If function argument lists and tuples get unified in a future version of Scala, it should be possible to write that mapping simply as “q1.mapResult(User)“.

It is time we looked at a more complex query with a non-trivial projection and a restriction (a WHERE clause in SQL). The following code can be used to find Apu’s last name and ID:

val q2 = for(u <- Users where {_.first is "Apu" }) yield u.last ~ u.id
println("Apu's last name and ID are: " + q2.first)

Compare this with a similar snippet which selects the same values from a List[User] containing all data from the database:

val q2 = for(u <- users if u.first == "Apu") yield (u.last, u.id)
println("Apu's last name and ID are: " + q2.firstOption)

where is a method on Query, so the Users object gets lifted to a query of a new unique instance of the Users table and then where maps this table to a Column[Boolean] which is added to the query as a restriction. Note that you have to use “is” instead of “==” (and “isNot” instead of “!=”) for column expressions because the latter is defined on all objects and cannot be overwritten oder overloaded in a suitable way. The string “Apu” is lifted to a ConstColumn[String] by another implicit conversion:

implicit def stringToConstColumn(v: String) =
  new ConstColumn(v) with StringColumn

Instead of returning the complete Users tuple from the query we use the projection “yield u.last ~ u.id” to extract only the last name and the ID. The call to first consequently returns an Option[(String,Integer)]. The SQL statement for q2 is:

SELECT t1.last,t1.id FROM users t1 WHERE (t1.first='Apu')

At the moment all constants are inserted directly into the SQL statement. A production-grade library should of course use bind variables by default, with inlined values available as an option.

We need data in the Orders table for the next query, so let’s create some rows. We insert two random orders for every user except Apu and Snowball:

for(u <- allUsers
         if u.first != "Apu" && u.first != "Snowball"; i <- 1 to 2)
  Orders.insert(u.id, null, "Gizmo "+((Math.random*10)+1).toInt)

So far the queries have only used a single table instance. They can be written in terms of map and filter without making use of the monadic structure of the Query class. Adding a second table to the query gives us a natural way of expressing a join:

val q3 = for {
  u <- Users
  o <- Orders where { o => (u.id is o.userID) && (u.last isNot null) }
} yield (u.first ~ u.last ~ o.orderID ~ o.product).sortBy(u.first)
println("All Orders by Users with a last name by first name:")
q3.foreach(o => println("  "+o))

Note the use of the “&&” operator in the restriction. This is not the usual boolean “and” but an operator on two Column[Boolean] objects yielding another boolean column. Another new feature seen in this query is the sortBy method in the projection. The SQL statement for this query is:

SELECT t1.first,t1.last,t2.orderID,t2.product
FROM users t1, orders t2
WHERE ((t1.id=t2.userID) and (t1.last is not null))
ORDER BY t1.first

Apart from joins, there are two more ways of combining queries: Unions and subqueries. Unions should map nicely to monadic plus operations with the empty query being the zero value but I haven’t tried that yet. Subqueries are used by constructing a special column from a query. This is already implemented as can be seen in the following example:

val q4 = for (
  u <- Users;
  o <- Orders
    where { o => o.orderID is
      queryToSubQuery(for {
        o2 <- Orders where(o.userID is _.userID)
      } yield o2.orderID.max) }
    where { _.userID is u.id }
) yield u.first ~ o.orderID
println("Latest Order per User:")
q4.foreach(o => println("  "+o))

The queryToSubQuery function should ideally be implicit. It’s currently not because of bug #1579 which prevents all implicit conversions to various invoker classes from working if queryToSubquery is also implicit.

The subquery is used to find the maximum of a column for each separate value in another column. This common pattern can be extracted into a combinator:

def maxOfPer[T <: TableBase.T_]
  (c: T, m: (T => Column[Integer]), p: (T => Column[_])) =
  c where { o => m(o) is queryToSubQuery(
    for { o2 <- c where( n => p(o) is p(n)) } yield m(o2).max
  ) }

We can now write the query with maxOfPer like this:

val q4b = for (
  u <- Users;
  o <- maxOfPer[Orders.type](Orders, _.orderID, _.userID)
    where { _.userID is u.id }
) yield u.first ~ o.orderID

In either case, the generated SQL statement is:

SELECT t1.first,t2.orderID FROM orders t2, users t1
WHERE (t2.userID=t1.id) AND (t2.orderID=(
  SELECT max(t3.orderID) FROM orders t3
  WHERE (t2.userID=t3.userID))
)

Another way to create a subquery implicitly is the in operator (and its negation notIn). To find all users which haven’t placed an order we can use:

val q5 = Users where { _.id notIn Orders.map(_.userID) }
println("Users without Orders:")
q5.foreach(o => println("  "+o))

Any query which returns a single table can be used to delete rows with a DeleteInvoker:

println("q5: " + q5.deleteStatement)
println("Deleting them...")
val deleted = q5.delete
println("Deleted "+deleted+" rows")

And finally, we can count the rows selected by that statement (which should of course be 0 after deleting them):

val q6 = q5.map(_.count)
println("q6: " + q6.selectStatement)
println("Users without Orders left: " + q6.first.get)

As I already mentioned at the beginning of this post (which is getting much longer than I anticipated), I am using java.lang.Integer instead of Int for the integer columns. The underlying problem is nullability. While you can specify which columns are created nullable in the database (by placing or omitting the NotNull option), they are all treated as nullable in SQuery and thus they cannot use value types like Int. Alternatively, you could take the same approach as HaskellDB and use value types (or references types which may never be null for types like String) for the non-nullable columns and an Option of the same type for their nullable counterparts. That works fine for all the features I’ve shown so far but it makes typechecking of outer joins difficult (or even impossible) because an outer join has to selectively promote non-nullable columns to nullable ones. I didn’t want to give up outer joins so easily, so I chose to make everything nullable for now.

There’s no public repository for SQuery yet but you can download the Eclipse project as a ZIP file. Just add the H2 JAR (not included) to the classpath to run the supplied example class test.SQuery2Test2 which contains all the queries from this post.