NAME
xrootd - The eXtended ROOT file server daemon
SYNOPSIS
xrootd [options] [exported directories]
DESCRIPTION
This manual page documents briefly the xrootd program.
xrootd is a ROOT remote file server daemon.
STARTING XROOTD
The simplest possible setup is one server and one or more clients:
+-- kan001 -----------+
+-----------+ +--------+ +---------+ |
| Client |------->| Server | | Storage | |
| TXNetFile | | xrootd |--| /data |-+
+-----------+ +--------+ +---------+
As with the original rootd(1) server, this is relatively easy to setup.
You can start it as:
kan001> xrootd
Unlike the rootdserver(1), this doesn’t immediately go into background,
but in fact produces some amount of logging information.
Using some xrootd client application, you should be able to access a
file on the server using the same syntax as with TNetFile/rootd, for
example:
root://kan001//tmp/mydatafile.root
From a client-side application such as a ROOT interactive session, you
should be able to open the file and perform operations on it:
root [1] myfile = TFile::Open("root://kan001//tmp/mydatafile.root");
root [2] myfile->ls();
XTNetFile** root://kan001//tmp/mydatafile.root
XTNetFile* root://kan001//tmp/mydatafile.root
KEY: TF1 fooHist;1 gaus
KEY: TF1 barHist;1 gaus
root [3] myfile->Close();
root [4] .q
ACCESSING AREAS OTHER THAN /tmp
One important difference from rootd(1) is that xrootd by default only
allows one to access files in /tmp. This is for security reasons so
that a running xrootd does not give remote clients access to read (or
write to!) everything visible as xrootd.
A client accessing an area which is not explicitly exported by xrootd
e.g., root://kan001//data/mydatafile.root will get an error.
You must specify explicitly any directories you wish to export via
xrootd. This can be done on the command line when starting xrootd, for
example:
kan001> xrootd /data
will export the /data directory.
MULTIPLE SERVERS
Often it is the case that the data that an application wants to access
is spread over multiple data servers.
The client application thus needs to know (or determine from some
external source) on which server any given data file is located. It can
then access those files directly with URL’s like:
root://kan001//data/aaaFile.root
root://kan002//data/bbbFile.root
While this may work for small systems with few users, it has some
disadvantages:
If users know statically where files are located it doesn’t
easily permit load balancing by moving files between servers or
by replicating them to spread the load.
If an external catalog is read by the client in order to
determine which server has the file, any changes to that catalog
must be synchronized with what actually happens to the files on
disk.
If the file isn’t actually on disk, but is retrieved from
somewhere else when the client asks for it (e.g. mass storage,
discussed in later examples), there is no easy way to manage the
disk space to insure that it is used efficiently and uniformly.
The xrootd system provides a method for solving these problems via its
open load balancer component. This provides a method for providing
transparent access data servers.
+-- kan001 ----------------+
| +--------+ +---------+ |
+--->| Server | | Storage | |
+ kanrdr + | | | olbd | | /data | |
+-----------+ +--------+ | | +--| xrootd |--| |-+
| Client |--->| Server |------+ +--------+ +---------+
| TXNetFile | | xrootd | | |
+-----------+ | olbd |---+ | +-- kan002 ----------------+
+--------+ | | +--------+ +---------+ |
+--->| Server | | Storage | |
| | | olbd | | /data | |
| +--| xrootd |--| |-+
| +--------+ +---------+
|
...
In addition to the dataservers an additional machine (the redirector -
kanrdr above) is added. It does not need to have any disk space itself
as its sole role is to redirect the client dynamically to an actual
data server machine which can serve the requested file. Thus client
applications open all files via the redirector kanrdr:
root://kanrdr//data/aaaFile.root
root://kanrdr//data/bbbFile.root
The redirection to the appropriate dataserver will take place
automatically (with the proper server configuration, discussed below)
and hence the user running the application doesn’t need to know
anything about the actual layout of the data over the data servers.
Note also that after redirection the client is reading directly from
the data server to which it was redirected. It does need the redirector
any longer (at least until it needs to open a different file).
To accomplish this a second daemon, the olbd, must be started on both
the redirector and each of the dataservers as show in the figure above.
Configuration of an xrootd/olbd system
OPTIONS
-l logfile
The primary advantage of using this option as opposed to simply
redirecting the output to a file is that there is also a feature
which does automatic log file rotation. At midnight each night
xrootd will move the current logfile to logfile.yyyymmdd and
then continue writing to logfile.
-p port
As with rootd(1), xrootd will started automatically on the IANA
assigned port number 1094. If you need to start it on another
port, that can be done using this option. Clients will need to
specify this port when accessing the server, using an URL like
root://kan001:2525//tmp/mydatafile.root. In general it is
probably simplest to run xrootd on the standard port 1094.
-c configfile
xrootd can be configured via a config file as well as via
command line options. There are in fact many more options
available via the config file and hence this is the recommended
way of starting xrootd is to use this option. The config file
configfile can have configuration directives as described below.
-r Flag this server as a redirector.
CONFIGURATION FILES
A configuration file for xrootd looks like
# Start the server on port 2525
xrd.port 2525
# Export files in /data only
xrootd.export /data
in order to achieve (for example) what was done with command line
options in the sections above. Note that lines beginning with # are
taken to be comments and ignored. For the moment we just note that all
config lines are of the form:
configclass.option [optval1] [optval2] [...]
It is also important to note that spelling errors in either configclass
or option will result in the entire line being (silently) ignored.
Read-only data access
An important thing to note about xrootd access to files is that by
default it allows any accessed file to be opened in update mode (and/or
new files to be created). For many situations this is undesirable: the
goal is simply to make data files available for others to read. The
easiest thing to do in that case is simply to disallow all write
access. To do this you need to add two lines to your config file:
xrootd.fslib /opt/xrootd/lib/libXrdOfs.so
oss.readonly
in your configuration file. The first of these will cause the shared
library for the optional ofs/oss file layer to be loaded (the exact
path the lib area will depend on how you installed xrootd). This shared
library provides a number of filesystem extensions. The second line
disallows all write/update access using one of these options.
More fine-grained configurations are possible, but globally disallowing
writes like this should cover a common use case.
Multiple servers
As an example we take a redirector (called kanrdr) and two data servers
(called kan001 and kan002) with files aaaFile.root and bbbFile.root as
above.
For the redirector we use a config file:
#
# redirector.cf
#
# xrootd
xrootd.fslib /opt/xrootd/lib/libXrdOfs.so
xrootd.export /data
odc.manager kanrdr 3121
odc.trace redirect
# olbd
olb.port 3121
olb.allow host kanrdr.slac.stanford.edu
olb.allow host kan001.slac.stanford.edu
olb.allow host kan002.slac.stanford.edu
and for the data servers we use:
#
# dataserver.cf
#
# xrootd
xrootd.fslib /opt/xrootd/lib/libXrdOfs.so
xrootd.export /data
oss.readonly
odc.manager kanrdr 3121
# olbd
olb.port 3121
olb.subscribe kanrdr 3121
As in the earlier examples, the actual path the to the libXrdOfs.so may
depend on your installation. (And you should obviously substitute your
machine names for kanrdr/kan001/kan002.)
It is then possible to start the xrootd and olbd servers on each of the
machines.
On dataserver kan001:
xrootd -l kan001.xrdlog -c dataserver.cf &
olbd -s -l kan001.olblog -c dataserver.cf &
On dataserver kan002:
xrootd -l kan002.xrdlog -c dataserver.cf &
olbd -s -l kan002.olblog -c dataserver.cf &
And on the redirector kanrdr:
xrootd -r -l kanrdr.xrdlog -c redirector.cf &
olbd -m -l kanrdr.olblog -c redirector.cf &
This should give you a working system capable of providing (readonly)
access to data files on two servers transparently masqueraded behind a
single redirector machine.
Testing the example load balanced system
Once the system is running we can try to open one of the files, for
example:
root [1] myfile = TFile::Open("root://kanrdr//data/aaafile.root");
root [2] myfile->ls();
XTNetFile** root://kanrdr//data/aaafile.root
XTNetFile* root://kanrdr//data/aaafile.root
KEY: TF1 fooHist;1 gaus
KEY: TF1 barHist;1 gaus
Your client will be redirected and connect itself to kan001, the actual
location of the requested file.
SEE ALSO
rootd(1), proofd(1), system.rootdaemonrc(1)
See also http://xrootd.slac.stanford.edu for more on the extended ROOT
server.
For more information on the ROOT system, please refer to
http://root.cern.ch
ORIGINAL AUTHORS
The ROOT team (see web page above):
Rene Brun and Fons Rademakers
COPYRIGHT
This library is free software; you can redistribute it and/or modify it
under the terms of the GNU Lesser General Public License as published
by the Free Software Foundation; either version 2.1 of the License, or
(at your option) any later version.
This library is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301
USA
AUTHOR
This manual page was originally written by Christian Holm Christensen
<cholm@nbi.dk>, for the Debian GNU/Linux system and ROOT version 3.