The feature id's are expected to be in ascending numerical order. The lowest allowable feature-id is 1 (0 is reserved for the bias term internally.) Any feature not specified is assumed to have value 0 to allow for sparse representation.

The class label for test data is required but not used; it's okay to put in a dummy placeholder value such as 0 for test data. For binary-class classification problems, the training labels should be 1 or -1. For ranking problems, the labels may be any numeric value, with higher values being judged as "more preferred".

Currently, the comment string is not used. However, it is available for use in other algorithms, and can also be useful to aid in bookkeeping of data files.

Examples:

Class label is 1, feature 1 has value 1.2, feature 2 (not listed) has value 0,

and feature 3 has value -0.5.

1 1:1.2 3:-0.5

Class label is -1, belongs to qid 3, and all feature values are zero except

for feature 5011 with value 1.2.

-1 qid:3 5011:1.2

Class label is -1, feature 1 has value 7, comment string is

"This example is especially interesting."

-1 1:7 3:-0.5#This example is especially interesting.

==Commandline Details==

File Input and Output

--model_in

• Read in a model from this file before doing any training or testing.

--model_out

• Write the model to this file when done with everything.

--training_file

• File to be used for training. When set, causes model training to occur.

--test_file

• File to be used for testing. When set, causes current model (either loaded from --model_in or trained from --training_file to be tested on test data.

--results_file

• File to which to write predictions, when --test_file is used. Results for each line are in the format \t\n and correspond line-by-line with the examples form the --test_file.

Learning Options

--learner_type

• Type of learner to use.
• Options are: o pegasos Use the Pegasos SVM learning algorithm. --lambda sets the regularization parameter, with values closer to zero giving less regularization. Note that Pegasos enforces a hard constraint that the model weight vector must lie within an L2 ball of radius at most 1/sqrt(lambda). Also relies on --eta_type. o sgd-svm Use the SGD-SVM learning algorithm. --lambda sets the regularization parameter, with values closer to zero giving less regularization. Also relies on --eta_type o passive-aggressive Use the Passive Aggressive Perceptron learning algorithm. --passive-aggressive-c sets the largest step size to be taken on any update step; this operates as a capacity term with values closer to zero encouraging simpler models. --passive-aggressive-lambda will force the model weight vector to lie within an L2 ball of radius 1/sqrt(passive-aggressive-lambda) o margin-perceptron Use the Perceptron with Margins algorithm. --perceptron-margin-size sets the update margin. When set to 0, this is exactly equivalent to the classical Perceptron by Rosenblatt. When set to 1, this is equivalent to optimizing SVM hinge-loss without regularization. Increasing values may give additional tolerance to noise. Also relies on --eta_type. o romma Use the ROMMA algorithm. No parametert to set. o logreg-pegasos Use Logistic Regression with Pegasos updates; we optimize logistic loss and enforce Pegasos-style regularization and constraints, with --lambda being the regularization parameter. Also relies on --eta_type.
• Default: pegasos

--loop_type

• Type of sampling loop to use for training, controlling how examples are selected.
• Options are: o stochastic
  Perform normal stochastic sampling for stochastic gradient descent, for training binary classifiers. On each iteration, pick a new example uniformly at random from the data set. o balanced-stochastic
  Perform a balanced sampling from positives and negatives in data set. For each iteration, samples one positive example uniformly at random from the set of all positives, and samples one negative example uniformly at random from the set of all negatives. This can be useful for training binary classifiers with a minority-class distribution. o rank Perform indexed sampling of candidate pairs for pairwise learning to rank. Useful when there are examples from several different qid groups. o roc
  Perform indexed sampling to optimize ROC Area. o query-norm-rank Perform sampling of candidate pairs, giving equal weight to each qid group regardless of its size. Currently this is implemented with rejection sampling rather than indexed sampling, so this may run more slowly.
• Default: stochastic

--eta_type

• Type of update for learning rate to use.
• Options are: o basic On the i-th iteration, the learning rate eta is set to: 1000 / (i + 1000) o pegasos On the i-th iteration, the learning rate eta is set to: 1 / (i * lambda) o constant Always use learning rate eta of 0.02.
• Default: pegasos

--dimensionality

• Index id of largest feature index in training data set, plus one.
• Default: 2^17 = 131072

--iterations

• Number of stochastic gradient steps to take.
• Default: 100000

--lambda

• Value of lambda for SVM regularization, used by both Pegasos SVM and SGD-SVM.
• Default: 0.1

--passive_aggressive_c

• Maximum size of any step taken in a single passive-aggressive update.

--passive_aggressive_lambda

• Lambda for pegasos-style projection for passive-aggressive update.
• When set to 0 (default) no projection is performed.

--perceptron_margin_size

• Width of margin for perceptron with margins.
• Default of 1 is equivalent to unregularized SVM-loss.

--hash_mask_bits

• When set to a non-zero value, causes the use of a hashed weight vector with hashed cross product features. This allows learning on conjunction of features, at some increase in computational cost. Note that this flag must be set both in training and testing to function properly.
• The size of the hash table is set to 2^--hash_mask_bits.
• Default value of 0 shows that hash cross products are not used.

Other Options

--random_seed

• When set to non-zero value, use this seed instead of seed from system clock.
• This can be useful in testing and in parameter tuning.
• Default: 0

--training_objective

• Compute value of objective function on training data, after training.
• Default is not to do this.

==References==

If you use this source code for scientific research, please cite the following:

* D. Sculley. Large Scale Learning to Rank. NIPS Workshop on Advances in Ranking, 2009. Presents the indexed sampling methods used learning to rank, including the rank and roc loops. 

Additional reading and references:

* K. Crammer, O. Dekel, J. Keshet, S. Shalev-Shwartz, and Y. Singer. Online passive-aggressive algorithms. J. Mach. Learn. Res., 7, 2006. Presents the Passive-Aggressive Perceptron algorithm. 

* T. Joachims. Optimizing search engines using clickthrough data. In KDD ’02: Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, 2002. Presents the RankSVM objective function, a pairwise objective function used by the rank loop method in sofia-ml. 

* Y. Li and P. M. Long. The relaxed online maximum margin algorithm. Mach. Learn., 46(1-3), 2002. Presents the ROMMA algorithm. 

* S. Shalev-Shwartz, Y. Singer, and N. Srebro. Pegasos: Primal estimated sub-gradient solver for SVM. In ICML ’07: Proceedings of the 24th international conference on Machine learning,