rsget.pl is a powerful perl application designed to retrieve files from download services (like RapidShare, MegaUpload and many more). It has a long list of features implemented already and even longer TODO list.

You can find more details on http://rsget.pl/ web page.

Features:

• Perfect for screen session
• Support for many download services
• Supports multiple network interfaces
• Dead interfaces are kicked out (useful with unreliable vpn tunnels)
• Continues partially downloaded files (if download service allows it)
• Incorporates small HTTP server allowing to check the status of downloads, and add new links.
• Automatically updates itself from SVN.

TODO:

• Write more documentation
• When continuing partially downloaded data start few kb before the end and compare old with new.
• Add commands: pause downloads, allow/don't allow captcha, and more
• File group support (first step for multiuser support)
• File priorities and group priorities.
• Selectable temporary directory.
• Fix bugs in http server, and speed it up.
• Better file writing methods, to allow downloading from multiple sources.
• Backoff if captcha fails too many times; lower uri prio.
• If multi-download problem appears while checking files force check on another interface/ip.
• Possibility to mark uri as "cannot continue".

Planned features:

• Multiuser support (may require major changes in the code).
• XML-RPC (or similar) to allow writing GUIs.

For full list of command-line options check: rsget.pl --help. All those options may also be set in config file ( $HOME/.rsget.pl/config ). Check 'README.config' file for example config. Each option requires a value, which may be specified immediately after '=' sign, or as next argument. Underscores may be replaced with dashes. Following command-line declarations are equivalent: --use_svn=update --use-svn=update --use_svn update --use-svn update

============================= URI list / get.list =============================

Understanding URI list syntax may be very important for people unable or not willing to use HTTP interface. It also is crucial if writing any software for manipulating the list.

Empty lines and lines starting with '#' are ignored. If line does not start with '#', the sign and text after it are treated as part of the line, and not as a comment.

Long lines may be truncated between any two words in following manners:

• Append '\' at the end of unfinished line: word1 word2 \ word3 \ word4 word5
• Start added line with '+' sign (preferred): word1 word2
  • word3
  • word4 word5

Each line specify one file to download, it is composed of following parts:

[COMMAND:] [GLOBAL_OPTS] URI1 [URI1_OPTS] [URI2 [URI2_OPTS] [URI3 ...]]

When introducing new source URI1 is the only part required. URIs must valid identifiers, as seen in location bar in your web browser; i.e. they may not contain spaces, replace them with '%20'.

COMMAND may be one of: GET, DONE, STOP or ADD. Note the upper case, and colon sign (:) after command. If none specified GET is assumed. Their meanings are:

• GET: download the file, or continue partially downloaded one
• DONE: file download completed, don't try to download
• STOP: file download aborted or never started, don't try to download
• ADD: try to find other uris on the list with the same name and size and add this uri as clone, if none found add it as new source. Command possibly dangerous, SEE (1)

GLOBAL_OPTS are options applied to all URIs for this file. Options specified after URI are applied only to the URI immediately proceeding them. Each option has syntax: 'name=value', where both name and value must be URI-encoded, in most cases it is enough to replace '%' sign and any white space with corresponding '%XX' code. There are many options saved by rsget.pl itself, but only a few which should be introduced, before starting download, by the user:

• (global) dir=VALUE - directory within workdir and outdir where downloaded file should be placed
• (local) pass=VALUE - file password, sometimes required to download a file, SEE (2)

Some of automatic options (saved by rsget.pl):

• (global) fname=VALUE - exact name of file, set after starting download
• (global) fsize=VALUE - exact size of file, set after starting download
• (local) error=VALUE - reason of last failure, URI won't be considered if it contains error option, if all URIs for file have errors COMMAND will change to STOP
• (local) name, iname, aname, ainame - name of file as reported by the website
• (local) size, asize - size of file as reported by the website

rsget.pl always writes each line in the following style: COMMAND: [GLOBAL_OPTS]

• URI1 [URI1_OPTS]
• URI2 [URI2_OPTS] ...

NOTE to programmers: rsget.pl rereads list file each time it is willing to change something, or file's mtime has changed. For that matter you should newer write directly to that file, unless you are appending only. On the other hand, rsget.pl will not make any changes to the list file if lock file exists. There is one exception tho: if new sources are added from HTTP interface and lock file exists it is removed and new URIs are written. Assuming your list_file is get.list, and lock file used is: get.list.lock, you should: touch get.list.lock # this ensures consistency read get.list process if needs update: write get.list.new if exists get.list.lock: rename get.list.new to get.list else: # get.list has changed restart from beginning remove get.list.lock

NOTES: (1) Some download services (e.g. MegaUpload) abbreviate file names on their download pages, which may lead to incorrect clone guesses. If there is a three part rar, first two parts will have the same size, and same abbreviated name. ADD command will think both links specify the same file. Moreover MegaUpload allows continuing partial downloads, so after restarting rsget.pl you may end up with a broken file containing data chunks from both sources.

(2) It is not user password nor rar file password. Some sites (in fact MegaUpload is the only one) may require to introduce resource password to be able to download a file. That password is set by the uploader, separately for each resource, and it should be provided by the uploader along with the URIs.