Problem accessing /HomepageDashboard/. Reason:

scala/ScalaObject
Caused by:

java.lang.NoClassDefFoundError: scala/ScalaObject

The problem is that the scala library files are not being copied correctly into the WEB-INFO/lib directory. This omission can be verified by looking into the war file:

jar tvf HomepageDashboard.war

for the content the WEB-INFO/lib directory.

Some suggested fixes, such as checking the scala library in the project’s “Java EE Modules Dependencies” have not worked consistently for me. Instead, the simple solution is to just manually copy the files where they need to go. The files are found inside the Scala distribution jar. Just extract and copy the needed files:

jar xvf scala.library_2.7.7.final.jar
cp lib/scala-dbc.jar lib/scala-library.jar lib/scala-swing.jar WebContent/WEB-INF/lib/

When the eclipse plugin deploys correctly, it copies all three. You probably only need the scala-library.jar.

First, to perform a multivariate linear regression in Excel, use the LINEST function. It requires a formula array to see all of the results, so control-U and command-enter are your friends. Select an 5 row by n column group of cells, where n is the number of variables you’re fitting. Hit control-U to edit. Enter “=linest(y,x,1,1)” where y covers your output values and x covers the n columns of input values. Then hit command-enter to set the array formula for all of the cells. Check the Excel help for more information about LINEST usage.

Once you’ve figured out the array formula gymnastics, note that the coefficients are returned backwards! That’s right, the first row of the array formula contains the calculated coefficients for your fit, but in reverse order compared to the order of your input x columns. Go figure. You can even see this dyslexia in the third example in the LINEST help. Look for the section where the regression equation is constructed using the new coefficients.

The third parameter may cause confusion. If you’ve used LINEST for simple regressions, you may have used linest(y,x,,1) as your function, where x and y are columns. In that case, LINEST provides the slope in row 1, column 1 of the output array, and the intercept in row 1, column 2. Above we set the third parameter to 1 or true, which causes Excel to force the regression plane through the origin. Doing so allows us to compare results between Excel and the Commons Math regression functions. But wait, what if I don’t want the fit to go through the origin? What if I have no reason to think my function goes through 0?

It’s actually the standard way of doing regression. Derivations of regression equations don’t treat the intercept calculation separately. Instead, it’s built in. Remember, linear regression is linear on the regression coefficients. The function on the coefficients goes through the origin. The function you construct with those coefficients may or may not…
If you want a constant value in your regressor variables, add it as a column. The resulting coefficient is the intercept. More specifically, regression provides an output coefficient for each column of input. To create a non-zero intercept, add a column of 1s to your data. Regression will find a coefficient for that column. It will be the constant part of your function, the intercept.

For example, when fitting a single column of data, Excel allows you to input an x column and a y column. That’s one input variable, x. Then linest(y,x,,1) will return two values, the slope and the intercept. A more generalized way about thinking about this problem is to use two input columns: the original x values and a columns of 1s. Then linest(y,x,1,1) returns a coefficient for each column. The resulting equation is y=b1 * x1 + b0 * x0, where x0 is always 1.

With the generalized view, we can now compare results between Excel and coded results such as from Commons Math. The Excel coefficients in the first row of the LINEST array function and the values returned by estimateRegressionParameters() should match.

For example, to fit a parabola in both systems, we would perform a linear regression on a*x^2 + b*x + c. Create columns containing x^2, x, and 1. Then the regression will return a, b, and c.

Finally, note that that Excel returns the residual and regressor sum of squares. You can use these to check the the r-squared calculations in your code: r^2 = 1-ssRes/(ssRes+ssReg).

Download the library from here and put the commons-math-2.1.jar file on your classpath. Here’s the example in the documentation converted to Scala:

		val regression = new OLSMultipleLinearRegression()
		// example from apache: http://commons.apache.org/math/userguide/stat.html#a1.5_Multiple_linear_regression
		val y = Array(11.0, 12.0, 13.0, 14.0, 15.0, 16.0)
		val x = new Array[Array[Double]](6,6)
		x(0) = Array(1.0, 0, 0, 0, 0, 0)
		x(1) = Array(1.0, 2.0, 0, 0, 0, 0)
		x(2) = Array(1.0, 0, 3.0, 0, 0, 0)
		x(3) = Array(1.0, 0, 0, 4.0, 0, 0)
		x(4) = Array(1.0, 0, 0, 0, 5.0, 0)
		x(5) = Array(1.0, 0, 0, 0, 0, 6.0)

		this.regression.newSampleData(y, x) 

		val beta = this.regression.estimateRegressionParameters()
		println("betas: " + beta.map("%.3f".format(_)).mkString(", "))

		// residuals, if needed
		val residuals = this.regression.estimateResiduals()

This works so cleanly because Scala compiles Array[Double] to double[]. It’s compatible with the Java library, while still allowing all the cool Scala functions like map().

Commons Math provides the matrix inversion needed to solve the normal equation and return the desired equation coefficients. But it doesn’t provide the correlation coefficient, often called “r-squared,” which tells you how much of the data’s variance is explained by the regression. We can add that calculation with the following code. Given the original Y values and the residuals, use:

	def calcRSquared(y:Array[Double], residuals:Array[Double]) = {
		val ssReg = sumSq(y.zip(residuals).map{case(a,b) => a - b})
		val rMean = sum(residuals) / residuals.size.toDouble
		val ssRes = sumSq(residuals.map(_ - rMean))

		1.0 - ssRes / (ssReg + ssRes)
	}

	def sq(in:Double) = in * in
	def sum(in: Array[Double]) = (0.0 /: in){_ + _}
	def sumSq(in: Array[Double]) = sum(in.map(sq))

The conciseness of Scala means that in reading the code, you can quickly discern both what a function does and how it does it. ssReg, for example is literally “the sum of the squares of pairs of items from y and residuals subtracted” or, more colloquially, “the sum of the squares of differences between y and residuals elements.

A traditional method for ensure single instances of program uses file locking and PIDs. A program, upon start up, would get its process id (PID) from the OS, then write it to a file called “program.lock” in a known, fixed location. Then it would read the file and check the written PID against its own. If they match, great; continue. If they don’t match, it means that another file succeeded at creating the file, ie got the lock, so this application must shut down. You might think that if the file exists, then no check is necessary. However, the idea is to handle the situation when multiple instances start at nearly identical times. Neither sees the file and tries to create it, but only one succeeds because Unix file creation is atomic.

The traditional method is clever, but challenging to get right. Lock files must be removed on exit, requiring more code. If the program terminates unexpectedly, provisions must be made to catch the exit signal and clean up or use additional methods to reap leftover lockfiles.

Here’s a much simpler method. Like the traditional method, each application attempts to hold a shared resource. Only one succeeds, and the resource hold is automatically released on program termination. The resource? A OS network port. So the only risk is accidentally choosing a port needed by another application for actual communication.

All you have to do is choose a port to use as an “application exclusion group id”. Several applications use the id. But only one application among the group sharing the id can run at a time.

Here’s the code. Include the file in your program, then call SingletonApp.performSolo as shown in the demo. To demo, just run multiple copies of the file as shown in the comments. Only one will run at a time…

import java.net.BindException
import java.net.ServerSocket

/**
 * This class enables applications to ensure that only one instance
 * of the application runs at a time.
 *
 * @author  Gary Boone, PhD
 * @version 1.0, 2010/02/24
 */
object SingletonApp {

	var serverSocket:ServerSocket = null

	/**
	 * Pick a port number to use as the application exclusion
	 * group id. Choose a port that won't conflict with other
	 * applications. See
	 * http://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers
	 * for common port numbers.
	 *
	 * Additionally, pass in a closure to be run if the application
	 * is already running.
	 */
	def performSolo( portToHold:Int)( fail: => Unit ) = {
		try {
			serverSocket = new ServerSocket(portToHold)
		} catch {
		  	case e:BindException => fail
		}
	}

	/**
	 * Demo the SingletonApp class
	 *
	 * Compile, or run with:
	 * scala -i SingletonApp.scala -e 'SingletonApp.main(null)' &
	 * Then start another instance to see it fail.
	 */
	def main( args:Array[String] ) : Unit = {
		val DEMO_PORT_TO_HOLD = 15486

		// ensure single instance
		SingletonApp.performSolo(DEMO_PORT_TO_HOLD) {
			println("failing due to already-running instance.")
			exit
		}

		println("Application started. Try to start another instance.")
		Thread.sleep(300000)		// 3e6 ms = 300 sec = 5 min
		println("exiting")
	}
}

To sort just the top k items, you can use a bounded priority queue; after feeding the whole list into it, the queue contains the top k items. This approach is better, but expends unnecessary effort maintaining the sort in the queue as items are added. If n is much larger than k, it would be faster to find the top items without sorting, then sort only the final list of top items.

The code below does a partial sort in linear time. Upon return, the first k positions of the array contain the largest or smallest items of the array, depending on the compare() function provided by the Ordered trait of the class contained in the array. The items in the first k positions are not sorted.

package scala

/*
 * FirstK
 *
 * These functions provide a linear partial sort of an array of Ordered objects. The array is
 * sorted in-place and upon return the the k smallest or largest items are in the first k positions.
 * The first k values are not sorted, but are less than or equal to the values in the rest of the
 * array. Whether the first k items are the largest or the smallest of the array is determined
 * by the compare function of the objects in the array.
 *
 * For example:
 * 	case class Foo(n: Int) extends Ordered[Foo] { def compare(other: Foo) = this.n.compare(other.n) }
 * means that firstK will return the smallest n values in Array[Foo]
 * 	case class Foo(n: Int) extends Ordered[Foo] { def compare(other: Foo) = -this.n.compare(other.n) }
 * means that firstK will return the largest n values in Array[Foo]
 *
 *  Example from scalatest unit test:
 *
 *    case class Foo(n: Int) extends Ordered[Foo] {
 *      def compare(other: Foo) = this.n.compare(other.n)
 *    }
 *
 * 	  test(" Test findFirstK() Length 2" ) {
 *      val array = Array(new Foo(2), new Foo(1))
 *      FirstK.findFirstK(array,  1)       // k = 1
 *      assert(array.deepEquals(Array(Foo(1), Foo(2))))
 *    }
 */
object FirstK {
	def swap[T](array: Array[T], x: Int, y: Int) = {
		val t = array(x)
		array(x) = array(y)
		array(y) = t
	}

	// Lomuto Partition
	def partition[T <% Ordered[T]](array: Array[T], left: Int, right:Int, pivotIndex: Int): Int = {
		val pivotValue = array(pivotIndex)
		swap(array, pivotIndex, right)              // Move pivot to end
		var storeIndex = left
		for (i <- left until right ) {
			if (array(i) <= pivotValue) {			// will use the compare fn, so may use > for desc sort
				swap(array, storeIndex, i)
				storeIndex += 1
			}
		}
		swap(array, right, storeIndex) 				// Move pivot to its final place
		storeIndex
	}

	// Note that p here is the position that ends the range of 'first' values.
	def findFirstToP[T <% Ordered[T]](array: Array[T], left: Int, right:Int, p: Int): Unit = {
		if (right > left) {
			val pivotIndex = (left + right) / 2
			val pivotNewIndex = partition(array, left, right, pivotIndex)
			if (pivotNewIndex > p)
				findFirstToP(array, left, pivotNewIndex - 1, p)
			if (pivotNewIndex < p)
				findFirstToP(array, pivotNewIndex + 1, right, p)
		}
	}

 	// intended entry fn; includes parameter checking. Modifies the array so that the first k items
 	// are largest/smallest of the array.
	def findFirstK[T <% Ordered[T]](array: Array[T],  k: Int) : Unit = {
		if (array==null || k==0 || k>array.size) {
			throw new IllegalArgumentException()
		}
		findFirstToP(array, 0, array.length - 1, k-1)		// fn is position-based so, k-1 for 'first k'
	}
}

scala> val a:Option[Int] = Some(5)
a: Option[Int] = Some(5)
scala> val b:Option[Int] = None
b: Option[Int] = None
scala> a.getOrElse(0)
res1: Int = 5
scala> b.getOrElse(0)
res2: Int = 0

But a problem you’re likely to come across is how to access an instance variable when a class is stored in an option? For example,

scala> case class S(lines:Int, pages:Int)
defined class S
scala> val sOpt:Option[S] = Some(new S(16,5) )
sOpt: Option[S] = Some(S(16,5))
scala> val nOpt:Option[S] = None
nOpt: Option[S] = None

Here, we’ve created a simple class an put an instance of it into an Option. Then we made another option container for the class containing None. So given an Option[S], how do you get the lines value out of it? You can’t do it directly because the Option may contain None.

The answer is to use map, passing in the accessor or other function you want to apply. In this case, we just use the accessor lines. An Option is returned, so getOrElse() then returns the value or a default.

scala> val numLines = sOpt.map{ _.lines }.getOrElse(0)
numLines: Int = 16
scala> val numLines = nOpt.map{ _.lines }.getOrElse(0)
numLines: Int = 0

The underscore is an abbreviation for the object in the map. Don’t forget that you can also pass functions into the map.

scala> sOpt.map{println}
S(16,5)
scala> nOpt.map{println}

Not that the instance toString() was printed by the first line, whereas the second printed nothing because the Option contained a None. (I’ve omitted the REPL result lines.)

Finally, note that complex functions can be built using the normal block “=>” syntax. Here, we print as well as accessing the lines instance variable:

scala> val numLines = sOpt.map{x => println(x); x.lines}.getOrElse(0)
S(16,5)
numLines: Int = 16
scala> val numLines = nOpt.map{x => println(x); x.lines}.getOrElse(0)
numLines: Int = 0
scala>

The first part of solving the problem was to look for clues in the Problems view. Opening it, I saw this:

error while loading Configgy, Scala signature Configgy has wrong version expected: 5.0 found: 4.1 in /Users/…

So the first error was due to the compilation failing and not producing a main class that eclipse could launch.

After further digging, the version error turned out to be caused by a mismatch between the Scala version (2.7.5) I used to compile the Configgy library and the Scala version (2.8) that the eclipse plugin used to compile the rest of the program. I didn’t install Scala 2.8, so how could this occur? The Scala plugin installs its own version of Scala. Because I had selected the nightly build version of the plugin, the Scala versions drifted apart during one of the eclipse updates.

You can tell which version the eclipse Scala plugin uses by looking at the jar name in the eclipse/plugins directory. Eg, scala.tools.nsc_2.75.final.jar

The solution was to revert to the stable, release version of the plugin. Do that by using this url in the eclipse Software Updates | Available Software | Manage Sites… list.

http://www.scala-lang.org/scala-eclipse-plugin

More complete directions here.

The first time I did it, I saw an error about finding a Scala object. Solve that by adding the scala-library.jar to your project. Right-click on the libraries folder in the Projects tab on the left sidebar. Choose Add JAR/Folder and find your scala-library.jar. Because I installed Scala with MacPorts, mine was under

/opt/local/var/macports/software/scala/2.7.4_0/opt/local/share/scala/lib

See this blog entry for more on finding MacPorts installations.

Add any other required libaries this way. Rebuild. Then you can run your program via

$ java -jar YourProgram

in the dist/ directory.

Note that by default, the NetBeans jar mechanism creates a jar file for your program and then adds the libraries you specified into a lib directory. So it’s not a single executable jar. You can then zip and distribute the jar/directory. The advantage of this approach is that you can update the libraries in the lib directory without having to rejar everything. You can probably also save space by using symbolic links in the lib directory to point to your jar locations. I haven’t tried that, though.

You can ignore the lib directory by directly linking to your libraries in the java command using Xbootclasspath. Here, I’ve added two libraries, Configgy and the Scala jar, to my project, but note that I also had to include the path to the Java SDK:

java -Xbootclasspath:/opt/local/var/macports/software/scala/2.7.4_0/opt/local/share/scala/lib/scala-library.jar:../configgy/dist/configgy-1.3/configgy-1.3.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6/Classes/classes.jar -jar ScalaApplication1.jar

If you want to create a single jar with all of the required files, then add a few lines to your build.xml file. There are many varying pages on the web about this; I found these directions simplest:

1. Add

<target name="-pre-jar">
    <unjar dest="${build.classes.dir}" src="${file.reference.theLibraryName.jar}">
</target>

to the build.xml file, replacing theLibraryName with the appropriate jar filename.

2. Rebuild

To get the correct library filename, switch to the Files tab in the left sidebar and look in the project.properties file. Find lines that look like these:

<unjar dest="${build.classes.dir}" src="${file.reference.configgy-1.3.jar}"/>
<unjar dest="${build.classes.dir}" src="${file.reference.scala-library.jar}"/>

You can ls -l your program in the dist directory and see that it’s much larger; it now contains the libraries. The lib directory is still there, but you don’t need it anymore.

First, eclipse, following the post here:
1. Create a Java class to run your Scala main:

package scala.loader;
import java.util.*;
public class ScalaEntryPoint
{
    public static void main(String[] args) {
    List<string> argList = new ArrayList<string>();
    argList.add("YourScalaObjectWithMain");
    for (String s : args) argList.add(s);
        scala.tools.nsc.MainGenericRunner.main(argList.toArray(new String[0]));
    }
}

Be sure to change YourScalaObjectWithMain to your class. Also, you’ll have to add the Scala tools jar to your projects classpath: Project | Properties | Java Build Path tab | Libraries tab | Add External Jars… button. Then pick the scala.tools.nsc file in your eclipse/plugins folder.

2. Create a Run Configuration to run the Java class. Check that it runs.

3. Then choose File | Export…, then Java | Runnable JAR File, then pick the Java run configuration to export and a destination. Optionally, you can save the ant XML file that does the export. Later, after some edits, you can rejar using

ant -f yourjarXMLfile.xml

If you change your libraries, be sure to repeat the complete Export process to make sure your jar scrips includes the correct jars.