Scala from a n00b, one idiot functional programmer's struggle with a whole new world

Given how little documentation I've seen out there that is anywhere near close to comprehensive, I figured I'd document some of my trials and tribulations learning Scala, and working in the Scala world. I am adding the very large disclaimer here that as I am just learning things, I am by no means a Scala expert, and take nothing here as perfect, or even good. Feel free to admonish my techniques in the comments if you wish, I'll try to post corrections where I can.

As many who know me know, I play EVE Online. It's a big MMO which has a significant API and database dump available for players to work with. I've been working on and off over the years on tools to work with the EVE universe through the data and the API. I'm going to use this platform as my first experiment with Scala.

I have the Programming in Scala book, I bought it some time ago, and it languished on my shelf for a long time. I finally picked it up and started working with it, but I found it severely wanting. When I start a new language like this, I want working examples that do stuff. I want to see how language concepts work in practice, not in examples so clinical they are more or less worthless. To that end, I started writing code based on what I had read, and what I could google.

I've picked the O/R Broker database API and Jersey to run my Rest service. I remember reading about a Rest server that Twitter used somewhere, but I can't seem to find it now, so I'm going with something else that's pretty well known.

O/R Broker is a bit verbose, but I like how it uses real SQL and case classes to achieve a pretty effective ORMish style system.

What follows is a simple example of serving up two services: a solar system information service, and a route service that shows a path between two solar systems through jump gates. I'm guessing if you're reading this, you can figure out the database structure easily enough, and I'll leave acquiring the EVE database as an exercise for the reader if you really want to do it (though I'll be happy to answer questions on that if anyone cares).

Starting with the SQL and moving upward, using ORBroker you create plain text files with the SQL in them, map them with a Token object then use those to make read calls. I'm using Maven as my build tools, so directory naming conventions follow Maven conventions for the most part, I think I started with a Scala archetype, but I don't fully recall.

First, I'm designing the SQL to return the values I want to use from the appropriate tables. To start with, I'm going to retrieve only basic information about a solar system:

select
  a.solar_system_id,
  a.solar_system,
  a.x,
  a.y,
  a.z,
  a.security
from
  solar_system a
where
  a.solar_system = :solarSystemName

I defined the model class for our SolarSystem object. It's created as a case class as that is what ORBroker is expecting, and it has a number of benefits including public members, but many others that I don't fully understand yet to be honest.

src/main/scala/com/plexq/eve/model/SolarSystem.scala:

case class SolarSystem (id: Option[Long],
                        name: String,
                        x: Double,
                        y: Double,
                        z: Double,
                        security: Double)

The next piece is mapping the SQL and model classes to extractor objects. These objects are responsible for turning the results of the SQL into Scala objects. In the first service, we have only a simple type, so we use the RowExtractor super class:

src/main/scala/com/plexq/eve/db/SolarSystemExtractor.scala:

object SolarSystemExtractor extends RowExtractor[SolarSystem] {
  def extract(row: Row) = {
    new SolarSystem(
      row.bigInt("solar_system_id"),
      row.string("solar_system").get,
      row.decimal("x").get.doubleValue(),
      row.decimal("y").get.doubleValue(),
      row.decimal("z").get.doubleValue(),
      row.decimal("security").get.doubleValue()
    )
  }
}

The row data is mapped into a constructor call for the model object, but because a SQL query can legitimately return a null for a column, the type of a row field is an Option type. The Option type in Scala is a case class used to distinguish explicitly between null values and actual values. It allows nulls to be type safe for one thing. The get method on an Option object retrieves the actual object, assuming there is one (I don't know what happens when there isn't, an exception I'd guess).

Now we want to perform the database operation in the context of a Rest call, which using Jersey is pretty easy:

@Path("/solarSystem")
class SolarSystemService(@QueryParam("name") name: String) {
  @Produces(Array("text/plain"))
  @GET
  def getSystem = {
    val broker = DatabaseContainer.getBroker
    broker.readOnly() { session =>
      session.selectOne(Tokens.selectSolarSystem, "solarSystemName"->name).get match {
        case Some(p) => {
            "Solar System:" + p.name + "\n" +
            "Security:" + p.security +"\n" +
            "x:" + p.x + "\n" +
            "y:" + p.y + "\n" +
            "z:" + p.z + "\n"
        }
        case _ => "Failed to find solar system "+name
      }
    }
  }
}

I'm just sending back text for the time being so that I can easily read if the output is correct. I've found that XML or JSON is surprisingly tricky in Scala as of yet, and the mechanisms I've tried didn't work out of the box or as designed/described.

We also need to create the object to contain our database information, modify to your local environment as usual:

object DatabaseContainer {
  val ds = new PGSimpleDataSource();
  ds.setServerName("localhost")
  ds.setUser("eve")
  ds.setPassword("xxxx")
  ds.setDatabaseName("eve")
  val builder = new BrokerBuilder(ds)
  FileSystemRegistrant(new java.io.File("src/main/sql/orbroker")).register(builder)
  builder.verify(Tokens.idSet)
  val broker = builder.build()

  def getBroker = broker
}

The final piece to the puzzle is a web.xml that initializes the Jersey servlet, and I'm using Jetty as a container here because I can't be arsed to get Glassfish sorted out, and I like Jetty:

src/main/webapp/WEB-INF/web.xml:

We can now run this web service and pull back information on a solar system!

I'm going to throw some stuff up here about getting JSON out of this whole thing, but it seems a subject that is unclear at best with my current knowledge, at least in a concise way and using Jersey. Lift seems overly complex for what I want, so I want to do it outside of that framework. There is some support in Jackson for Scala, but it doesn't seem to work quite right, at least the way I have it configured.

To get it going, I've changed web.xml, SolarSystem.scala and SolarSystemService.scala. I added the POJO option to the Jersey container:

I created a Jackson mapper in my Service object and updated the output to use that instead of my String concatenation:

@Path("/solarSystem")
class SolarSystemService(@QueryParam("name") name: String) {
  @Produces(Array(MediaType.APPLICATION_JSON))
  @GET
  def getSystem = {
    val mapper = new ObjectMapper()
    mapper.registerModule(DefaultScalaModule)

    val broker = DatabaseContainer.getBroker
    broker.readOnly() { session =>
      session.selectOne(Tokens.selectSolarSystem, "solarSystemName"->name) match {
        case Some(p) => {
          mapper.writeValueAsString(p);
        }
        case _ => "Failed to find solar system "+name
      }
    }
  }
}

And for the most perplexing part, I updated the SolarSystem object to define explicit getters. This is a bit odd because Jackson is supposed to cope with public fields, but it's not working for me. I've read some things about version incompatibilities, so maybe that's it, but I'm going with what I have so far:

case class SolarSystem (id: Option[Long],
                        name: String,
                        x: Double,
                        y: Double,
                        z: Double,
                        security: Double) {
  def getName = name
  def getId = id
  def getSecurity = security
  def getX = x
  def getY = y
  def getZ = z
}

Now when I run the service and ask for information on Jita, I get the following:

{"id":1358,"name":"Jita","x":-1.29064861734878E17,"y":6.07553069099636E16,"z":-1.1746922706009E17,"security":0.945913116664839}

Much easier to digest.

In the next service, I want to build a route between two solar systems, kind of a travel plan. To do this I'm going to need to retrieve a list of systems that a given system leads to, a list of destinations. Constructing the SQL for this is a little more interesting, but not particularly challenging:

src/main/sql/orbroker/selectDestinations.sql:

select
  a.solar_system_id,
  a.solar_system,
  a.x,
  a.y,
  a.z,
  a.security,
  d.solar_system_id as destination_id,
  d.solar_system as destination_name
from
  solar_system a,
  stargate b,
  stargate c,
  solar_system d
where
  b.solar_system_id=a.solar_system_id
  and b.destination_id=c.stargate_id
  and c.solar_system_id=d.solar_system_id
  and a.solar_system = :solarSystemName

As you can see above, we're performing a join, so we need to use two of the extractor types provided by ORBroker, a RowExtractor and a JoinExtractor. The information we are retrieving here is a one-to-many relationship between a solar system and its destinations. A RowExtractor is responsible for the most frequent output information, the data from the join that is unique on each row and represents the child objects, which in this case is the destination solar systems. The extractor we already have for SolarSystem is find for that. The low frequency information, which is the parent object, is the source solar system. The source solar system is therefore extracted using the JoinExtractor. The JoinExtractor needs to know what field the identity for the parent record is so that it can separate objects that belong to the parent, and those that belong to the child. The identity column is provided by overriding the 'key' property. All rows that share this identity column are assumed to be a single parent object. All rows within that set that have different entries are mapped as children of that parent object.

src/main/scala/com/plexq/eve/db/SolarSystemDestinationExtractor.scala:

object SolarSystemDestinationExtractor extends JoinExtractor[SolarSystemDestination] {
  val key = Set("solar_system_id")

  def extract(row: Row, join: Join) = {
    new SolarSystemDestination(
      new SolarSystem(
        row.bigInt("solar_system_id"),
        row.string("solar_system").get,
        row.decimal("x").get.doubleValue(),
        row.decimal("y").get.doubleValue(),
        row.decimal("z").get.doubleValue()
      ),
      join.extractSeq(SolarSystemExtractor, Map("solar_system_id"->"destination_id", "solar_system" -> "destination_name"))
    )
  }
}

Given the data structure and the constructors above, we can define the new model class for SolarSystemDestinations:

src/main/scala/com/plexq/eve/model/SolarSystemDestination.scala:

case class SolarSystemDestination(solarSystem : SolarSystem,
                                  destination : IndexedSeq[SolarSystem])

Now we have enough code to store the result of a SQL query that retrieves information about solar system destinations. We need some code to read it and turn it into a route. I'm building a simple b-tree style object here that contains a route-in-progress:

src/main/scala/com/plexq/eve/map/RouteTree.scala:

class RouteTree(solarSystem: SolarSystem) {
  var nodes : List[RouteTree] = List[RouteTree]()

  def contains(v: SolarSystem) : Boolean = (v.name == solarSystem.name) || nodes.exists {_.contains(v)}

  def leaves() : List[RouteTree] = {
    nodes.length match {
      case 0 => List(this)
      case _ => nodes.flatMap {x=>x.leaves()}
    }
  }

  def getSolarSystem : SolarSystem = solarSystem
  def setNodes(n : List[RouteTree]) : RouteTree = {
    nodes = n
    this
  }

  def path(end: SolarSystem, filter: (SolarSystem) => (Double)) : List[SolarSystem] = {
    if (contains(end)) {
      nodes.length match {
        case 0 => List(solarSystem)
        case _ => List((solarSystem, filter(solarSystem))) ::: nodes.find {_.contains(end)}.get.path(end, filter)
      }
    }
    else List()
  }

  def count : Int = (nodes.length/:nodes)(_+_.count)
}

and a builder object to construct a route:

class RouteBuilder {
  def buildRoute(broker: Broker, route: RouteTree, end: SolarSystem) : List[SolarSystem] = {
    var s = route.count

    route.leaves().foreach { x : RouteTree =>
      x.setNodes(SolarSystemDataService.getSolarSystemDestinations(broker, x.getSolarSystem.name).filterNot {
        route.contains(_)
      }.map {
        new RouteTree(_)
      }.toList)
    }

    /* Bug out if the list didn't get any bigger */
    if (route.count == s) {
      return List()
    }

    route.contains(end) match {
      case false => buildRoute(broker, route, end)
      case _ => {
        route.path(end)
      }
    }
  }
}

Now we can create our service class:

@Path("/route")
class RouteService(@QueryParam("start") start: String,
                   @QueryParam("end") end: String
                  ) {
    @Produces(Array(MediaType.APPLICATION_JSON))
    @GET
    def getRoute = {
      val mapper = new ObjectMapper()
      mapper.registerModule(DefaultScalaModule)

      val broker = DatabaseContainer.getBroker
      var error : scala.collection.mutable.ListBuffer[String] = ListBuffer()

      val routeTree = broker.readOnly() { session =>
        session.selectOne(Tokens.selectSolarSystem, "solarSystemName"->start).get match {
          case p : SolarSystem => new RouteTree(p)
          case _ => {
            error+=("Failed to find Start System "+start)
            null
          }
        }
      }

      val endSystem : SolarSystem = broker.readOnly() { session =>
        session.selectOne(Tokens.selectSolarSystem, "solarSystemName"->end).get match {
          case p : SolarSystem => p
          case _ => {
            error+=("Failed to find End System "+end)
            null
          }
        }
      }

      error.length match {
        case 0 => {
          mapper.writeValueAsString(new RouteBuilder().buildRoute(broker, routeTree, endSystem))
        }
        case _ => {
          ("[\""/:error)(_+"\",\""+_)+"]"
        }
      }
    }
}

In our service class we check to make sure the start and end systems exist, and I'm thinking there has to be a better way than this to do it, but this works. The main difference here is the mapper class now registers the DefaultScalaModule. This is provided by the jackson-scala dependency, and will cope with Scala classes that the default Java bindings don't, like List() objects, which is what we get in this case. Now when we ask for the route from Jita to Amarr, we get back a nice JSON list:

[{"id":1358,"name":"Jita","x":-1.29064861734878E17,"y":6.07553069099636E16,"z":-1.1746922706009E17,"security":0.945913116664839},

{"id":1360,"name":"Perimeter","x":-1.29064861734878E17,"y":6.07553069099636E16,"z":-1.1746922706009E17,"security":0.945913116664839},

{"id":1355,"name":"Urlen","x":-1.43265233088943008E17,"y":6.4923714928938896E16,"z":-1.04178623206742E17,"security":0.953123230586721},

{"id":4028,"name":"Sirppala","x":-1.39376796022883008E17,"y":7.1476647043998E16,"z":-9.9524016578104608E16,"security":0.959995210823471},

{"id":4025,"name":"Inaro","x":-1.37550934148756E17,"y":7.8077592063385904E16,"z":-8.6193987480218304E16,"security":0.88322702239899},

{"id":4026,"name":"Kaaputenen","x":-1.3575371976577E17,"y":7.79504770996252E16,"z":-8.2362867465824608E16,"security":0.836977572149063},

{"id":4750,"name":"Niarja","x":-1.38143247136544992E17,"y":6.6032260761458E16,"z":-7.5317306241481296E16,"security":0.779168558516838},

{"id":4749,"name":"Madirmilire","x":-1.84441638429595008E17,"y":4.9352410074477104E16,"z":2.47548529253837E16,"security":0.541991606217488},

{"id":4736,"name":"Ashab","x":-1.86411855995097984E17,"y":5.1383654517254496E16,"z":2.95630167990767E16,"security":0.603228207163472},

{"id":3412,"name":"Amarr","x":-1.95782999035935008E17,"y":5.4527294158362E16,"z":5.51292598732268E16,"security":0.909459990985168}]

After this, I start to descend into madness around providing a filter mechanism to provide routes only in high-sec etc, and a service for capital jump planning, but that's another story!

I'm not a Scala expert by far, this is pretty much my first Scala app, so suggestions and comments welcome.  I hope this was useful.  I can post Maven deps if that's useful, but largely it's just Scala, Jackson, ORBroker and Jersey, I grabbed the latest versions of each using mvnrepository.com.

Scala from a n00b, one idiot functional programmer's struggle with a whole new world

Saturday, March 3, 2012

Meh

Saturday, December 31, 2011

Converting from a List or Seq or similar to varargs

Friday, December 30, 2011

ScalaTest, SBT, debugging mode and IDE

Wednesday, December 28, 2011