New test version of ENMTools with model selection

There's a new test version of ENMTools up here. In addition to fixing a few minor annoyances from previous versions, there's a new function that allows criterion-based model selection using AICc and BIC. The user interface for the function is almost non-existent - all it really does is ask for a script file. Here's a quick-and-dirty rundown of how to use it. In order to correctly calculate likelihoods, the data must be formatted appropriately. For that reason we suggest that users pay very close attention to the requirements below.

1. Build a set of models to compare. It is absolutely crucial that suitability scores be output in RAW format! You will need both the .asc file and the .lambdas file associated with each model.

2. Make sure that each set of occurrence points to be compared is in its own independent file. You do not want to load an occurrence file that has points for multiple species. You also need to eliminate duplicate occurrence points from the file, particularly if you have Maxent set to ignore duplicate occurrences.

3. Build a script. A script is simply a .csv file with the paths to the files you want to analyze. Each line of the script should consist of a .csv file, a .asc file, and a .lambdas file. Relative paths will not work, you need fully qualified path names. A typical line will look like this:

c:\mydata\points.csv,c:\mydata\species.asc,c:\mydata\species.lambdas

You need one line per analysis. Also note that ENMTools will output results into a file with a name based on your script file. If your script file is named myscript.csv, the output will be named myscript_model_selection.csv. At present it will overwrite that output file (if it already exists) without asking, so BE CAREFUL.

4. In ENMTools, choose the "Model Selection" tool under "ENM Measurements". A file dialog will pop up. At this point you should choose the script file that you just made. ENMTools will chug along for a while, and will tell you when it's finished. The process is fairly simple: ENMTools uses your raw suitability scores (after standardization) and occurrence points to calculate the likelihood of observing your data under that model. It then counts the number of parameters from your lambdas file, counting any parameter with nonzero weight. Finally, it uses these values to calculate AICc and BIC.

Preliminary studies (Warren and Seifert, in review) indicate that AICc outperforms BIC in selecting models on simulated data. I'll be talking about this study at Evolution this year, for those who are interested (shameless plug).

Keep in mind that this is a test build and may be buggy. Feedback is appreciated.

New version of ENMTools up

I've tracked down the sorting bug that Rich and I found last night, and there's now a new version of ENMTools that fixes the issue. It does appear that it only affected the ribbon analysis, and only the sorted output. There are some new functions added as well. One is just a simple tool for measuring correlations between raster files using the Pearson correlation coefficient. The second gives you the ability to sample occurrences from a raster file either randomly or with a probability determined by some function of the suitability of habitat in that grid cell. You can download the new version and manual here. There's more information on the new functions in the manual.

Now that the application note is out we're going to start using actual version numbers instead of build dates. This is version 1.1.

Sorting bug with ribbon range-breaking analysis

It has just now come to my attention that ENMTools has a bug involving the sorting of I and D values for the ribbon range-breaking analysis. This bug can cause incorrect values to be inserted into the sorted output. The unsorted output files are still fine, and as far as we can tell at the moment this bug doesn't apply to any other analysis. If you are using the ribbon analysis, please use the .csv files for the unsorted output and sort them in Excel yourself. I am working on this bug right now and will post a corrected version of the software ASAP. My apologies for any inconvenience this might have caused.

Problems with ENMTools and Maxent 3.3.2

Rich and I just hit a problem, but luckily it's one with an easy solution.

As discussed on the Maxent discussion group, Maxent version 3.3.2 does not necessarily like underscores too much. Since ENMTools generates replicate names using underscores, this leads to replicates crashing, often with "number format exception" messages from Maxent and "Can't open ..." messages from ENMTools. Steven is aware of the issue and is working on it, but for now we'd suggest sticking with version 3.3.1 for your ENMTools runs.

ENMTools application note now available

The application note for ENMTools is now available in the Early View section of Ecography's website. Due to the limits of space, the note really only discusses the measurement of overlap and the identity and background tests (i.e., the tests that were presented in Warren et al. 2008). Papers that present the rangebreaking tests and niche breadth measurements are on the way, and should be out fairly soon.

I'm going to be putting out an update of ENMTools shortly which will add a couple of interesting features and fix a few minor issues.

How to run ENMTools tests on a cluster, or using a modeling method other than Maxent

ENMTools tests can take a long time to run, due to the number of Maxent runs necessary to construct a distribution of expected overlaps. For that reason, people occasionally ask whether there's a way to break up analyses and run them on a cluster, or just manually spread them across several computers. People also occasionally ask whether it's possible to use ENMTools with non-Maxent methods of ENM construction. There's no built-in way to do either of these at present, and given the number of different ways that people might want to do this I'm not sure there will be. However, it CAN be done! It requires a bit of extra work, but it's not too bad, particularly with a few simple new tools that I'm presenting here.

Let's say you want to do an identity test, and split it up so that replicates are submitted to a cluster. For starters, we'll just set up the test in ENMTools as usual but de-check the box marked "Run Maxent". This will cause ENMTools to generate the data sets necessary for the test, but the data will not be sent to Maxent to analyze.

Okay, now we have a file that has a whole bunch of replicates in it. What we would like to do is analyze each of those separately. I've got a really simple no-frills script here that simplifies this process. All you need to do is drop this file into the directory where your file of replicates is, then go to a command prompt in that directory. Type "splitcsv.pl INFILE", where INFILE is the name of your .csv file of replicates. In short order, you will have a new .csv file in that directory for each replicate in the input file. Note: you can also use this little tool to split a large file up by species. It will spit out a file for each unique species name in the input file. Another note: this script will overwrite files without asking, so it's best to run it in its own directory!

At this point it's up to you to figure out how to submit jobs to your cluster using the Maxent command line options. One thing to keep in mind: Maxent by default writes to a file called maxentResults.csv. If you've got a shared file system, or just a directory on a single multi-core computer that's being shared by two different simultaneous Maxent jobs, both of those jobs will by default try to write to the same results file. Java will happily allow multiple instances of Maxent to fight over the same filehandle, but what comes out in that results file will be incomplete and quite possibly not useful. I've just set my stuff up so that it writes to different directories, but it's possible that using the "perspeciesresults" option will fix this issue as well. On a side note: if you're doing jackknife or bootstrap on a cluster, you pretty much can't have multiple instances of Maxent writing to the same directory. The perspeciesresults option doesn't work for resampling, and having multiple instances writing to the same output file simultaneously will cause Maxent to fail when calculating summary grids. Just FYI.

All right, now you've gotten all of your runs finished and want to bring your data together to construct a distribution of overlaps. This is where a handy new half-developed tool in ENMTools becomes useful: scripting mode. You'll need the newest test version of ENMTools for this. I'll admit that it's a little weird to have a scripting interface for a program that is itself just an elaborate scripting interface for Maxent, but that doesn't mean it's not useful. At present the scripting interface is only hooked up for a few functions, and I'm not even close to writing a comprehensive manual entry for it. However, it works for our purposes.

What you need to do is build a .csv file that has a line for each comparison to be made. The script command is "measureOverlap". Capitalization is not important. A typical line looks like this:

measureOverlap,c:/sample data/species1_rep1.asc,c:/sample data/species2_rep1.asc,testing_measureOverlap1

The first entry is the command, the next two are the two files to compare, and the last entry is the name for the analysis. If you're doing a comparison between two species using 100 replicates, you need 100 lines in your script file, each with its own name for generating output files. In cases like this, the "concatenate" function in Excel is your best friend. Once you've got your script file, just go to Options->Run Script File in ENMTools and let it do its thing.

Now you've got two output files per replicate (I and D), and you'd really like to have all of those in one file. In Windows, the easy way to do this is to go to a command prompt and use the "copy" command with some clever wildcards. In the above example, let's say I've got 100 files named "testing_measureOverlap1_I_output.csv" to "testing_measureOverlap100_I_output.csv", and likewise for D. The appropriate command would be something like:

copy testing_measureOverlap*_I_output.csv collected_I_scores.csv

This would concatenate all of those output files for I into one csv file that can then be edited in Excel. There's going to be a bit of cleanup to do, since each of those files had its own header line and two copies of each score. That's all fairly easy, though, and should only take a couple of minutes with some clever sorting.

Everything I've said here goes for other modeling methods as well - you can generate data sets and analyze them in whatever software you like, and then use the scripting interface to build your distributions. When I get a chance I plan to make an interface that will make the comparison of multiple files easier (i.e., send everything to one formatted outfile). Seeing as I'm currently in a blind rush to finish my dissertation before my postdoc starts, though, I don't think that's going to happen too soon.

New manual!

We've finally bashed together a first take on the full manual for ENMTools. You can find it here, or in the zip file with the new version of ENMTools.

The new manual outlines for the first time the niche breadth, range breaking, spatial cross-validation, and jackknife/bootstrap functionality of ENMTools. Papers are in progress discussing all of these tools in more detail, but there should be enough information in the manual to get the general idea now.

Now I'm almost immediately going to start adding new features so that the manual is obsolete again. We'll try to keep it more up to date, though. Scout's honor.