Coda File System

Next Previous Contents

7. Installing a Coda Server

7.1 Introduction

A Coda cell is a set of servers which all believe one member of the set to be the master, or SCM server. All modifications to important Coda databases should be made on the SCM, otherwise the SCM plays the role of an ordinary Coda server. The updateclt / updatesrv daemons will then propagate changes from the SCM to the other servers.

The first server setup must be the SCM. This chapter is divided into three sections: installing the binaries, configuring the system using the configuration script vice-setup (for both SCM and non-SCM servers) and finally, a detailed look at what needs to be done to setup a server manually. Before beginning, we have provided some overview information to assist in setting up the server.

Recoverable Virtual Memory

To help ensure that data is not lost or left in inconsistent state between server restarts, Coda uses Recoverable Virtual Memory (RVM) which is a transaction system that maintains the system state of the Coda server meta data. RVM is a data logging transaction system which writes modifications into the RVM log and upon truncation or restart incorporates such modifications into the RVM data file. Both log and data file for RVM are ideally held on raw partitions.

NOTE: this should not be confused with Virtual Memory.

Upon startup, the Coda servers use RVM to restore the Coda system state. To efficiently use this feature, you should have dedicated disk partitions for optimal performance the log partition ideally on its own disk. However, a disk sharing the log partition with other disk partitions or the use of a log file may be used at corresponding loss of optimal performance. See section XXX for more details on RVM.

Server Disk Organization

Coda servers require a minimum of 2 disk partitions for optimal performance (one raw partition for the RVM Log, one raw partition for RVM Data and one regular UNIX filesystem to hold file data store in Coda), data security and protection from accidental deletion. For additional performance gains the RVM Log partition should be on its own disk to avoid head movement latencies which reduces disk seek times on log operations. Optionally, /vice can be a separate partition for the same reasons it is advantages to have /var as a separate partition.

However, other configuration may be used such as having the RVM Data and Log information stored as regular UNIX files at a loss in performance and data security. Also, if more than one Storage Area Data is needed on a Coda Server (the default directory is called /vicepa ), the additional storage areas must be separate partitions (different partition from /vicepa the default, initial storage area for data under Coda) and mounted as /vicepb , for example.

The table below shows a possible partitioning of disks on Coda servers with their respective purpose, mount points, typical sizes and consistency check program. Please note that the sizes of these partitions were taken from one of the Coda servers at CMU-SCS: the actual sizes may vary at other installations.

Example of Partitions for a Coda Server

Partition
Storage Purpose Mounted Typical Size Whether fscked
hda2 RootandUser File System / 650MB Yes
hda5 Var file system /var 100MB Yes
hda3 Vice File System /vice 300MB Yes
hdc1 RVM Log Not 12MB No
sda1 RVM Data Not 130MB No
sda2 Coda FS Data0 /vicepa 1.6GB Yes
sda3 Coda FS Data1 /vicepb 1.6GB Yes
sda5 Coda FS Data2 /vicepc 1.6GB Yes

7.2 Obtaining Coda

Currently, the binary and source packages are available via ftp from

( ftp://ftp.coda.cs.cmu.edu/pub/coda / < platform > )

as well as the Coda Project's local Coda servers ( /coda/project/release/objs / < platform > ) for CMU users. The platform layout for finding the Coda binaries relative to the base directory indicated above is:


linux/
<
coda-release
>
/{i386,sparc,coda-fs-module,SRPMS} # For Linux
freebsd/
<
coda-release
>
/
<
OS_RELEASE_VERSION
>
/i386          # For FreeBSD
netbsd/
<
coda-release
>
/
<
OS_RELEASE_VERSION
>
/i386        # For NetBSD

Where < coda_release > refers to the Coda version and < OS_RELEASE_VERSION > is the version of the Operating System. For example, if you want Coda release 4.3.13 for i386 FreeBSD 2.2.5, the correct platform directory is: fbsd/4.3.13/2.2.5/i386/ . For i386 Linux, it is: linux/4.3.13/i386/ , and for i386 NetBSD 1.3, it is: netbsd/4.3.13/1.3/i386/ . For more information on the layout, please check the document LAYOUT available in the top-level of the locations listed above for obtaining Coda. However, the following files are needed by the various platforms:

Linux

coda-debug-server-
<
release
>
.i386.rpm 
coda-doc-
<
release
>
.i386.rpm
coda-debug-backup-
<
release
>
.i386.rpm

Note that the Coda Project supports the glibc (GNU C Library) version of Red Hat (5.X and higher). Support for the older Linux standard Lib C is no longer provided although Coda is known to run on many other Linux systems.

FreeBSD and NetBSD

coda-server-
<
release
>
.tgz 
coda-doc-
<
release
>
.tgz

7.3 Installing and Configuring A Coda Server

Currently, Linux, NetBSD and FreeBSD are supported (CMU-MACH is no longer supported). For Linux, the Red Hat Package Management tool is supported. The package system of FreeBSD 2.X and higher as well as NetBSD 1.3+ is now supported.

TIP : Always check the INSTALL. < platform > on ( ftp://ftp.coda.cs.cmu.edu/pub/coda / < platform > ) for last minute changes and updates that have not yet found their way into the manual.

Installing the Coda Server Binaries and Documentation

Currently, server excutables for the supported platforms are installed into the following directories:

Linux

installs files/binaries in /etc , /usr/bin and /usr/sbin .

FreeBSD

installs files/binaries in /usr/local/etc , /usr/local/bin and /usr/local/sbin

NetBSD

installs files/binaries in /usr/pkg/etc , /usr/pkg/bin and /usr/pkg/sbin

NOTE: Please make sure your PATH environment variable is set properly for your platform. That is, ensure FreeBSD has /usr/local/{bin,sbin} and NetBSD has /usr/pkg/{bin,sbin} in its default search paths. Linux installs Coda into the default system paths, so Linux shouldn't need its default seach path modified.

There is one directory used by all platfroms: /vice which is used to store the supporting Coda configuration and Coda databases. It does not need to be in the PATH.

Specific steps to install the binaries and documentation are:

Linux

rpm -Uvh coda-debug-server-
<
release
>
.i386.rpm
rpm -Uvh coda-doc-
<
release
>
.i386.rpm

NetBSD and FreeBSD

The installation procedure is identical for both platforms (we assume you are using GNU tar which is the default tar for both platforms):


pkg_add coda-server-
<
OS_RELEASE_VERSION
>
-
<
coda_release
>
.tgz
tar zxfv coda-doc-
<
release
>
.tgz -C /usr/share

Please note that NetBSD uses a different location for NetBSD packages (/usr/pkg instead of /usr/local). Please make sure /usr/pkg/{bin,sbin} are in your paths under NetBSD.

Configuring A Coda Server

Server setup is similar for all platforms.

Before Beginning

The /vice directory will be created by the setup script automatically if it does not already exist. However, if your root partition does not have enough additional space (e.g. the SrvLog file can become huge & plus;10MB under certain circumstances), you will need to create a partition to be mount as /vice , format the partition as normal UNIX file system, create the mount point /vice , add the partition and mount information to /etc/fstab and mount it by hand BEFORE running the setup script.

Alternatively, /vice may be a symbolic link to a directory on an existing partition with enough space but it also must be created before running the setup script, vice-setup . Sub-directories needed under /vice will be created by vice-setup .

Also note that the setup of an SCM differs in some important ways from a non-SCM setup. So, when vice-setup asks: Will this machine will be the SCM? it is very important this question is answered correctly.

Only one SCM should be setup per Coda Cell and Coda does not support multiple-homing cells at this time.

While vice-setup sets up things that are common for both the SCM and non-SCM servers, the setup of scripts called by answering yes to the Is this the SCM? are:

SCM Setup

vice-setup


vice-setup-scm
vice-setup-user
vice-setup-rvm
vice-setup-srvdir

The set of scripts called by answering no are:
non-SCM Setup

vice-setup


vice-setup-rvm
vice-setup-srvdir

in the order listed above.

Setting Up A Coda Server

If this is the first (or only) server being setup for a Coda cell, it must be setup as the ``SCM'' by answering yes to the question when asked by vice-setup . The SCM coordinates the sync'ing of Coda database and global configuration files needed to keep track of data stored in Coda. The SCM also coordinates the authentication of a Coda user.

If you are adding a non-SCM server which keeps copies of databases, but does not distribute them to the rest of the machines comprising a Coda Cell answer no to the question.

Other than the distinctions indicated above, the SCM plays no special role in a Coda cell and can actually be down for a short time without denying service in a multiple-machine cell site. However, is extremely important to only have one SCM

NOTE: that running a server seriously dips into system virtual memory. Running a Coda server, a Coda client and X11, the Coda Group as observed that a slightly over 64MB of available VM is needed. The command top gives information on memory, cpu and process usage. Therefore, we recommend not running a Coda client on a Coda server and only run X when performing server maintenance if X is convenient to use.

Whether setting up an SCM Coda server or adding a server to an existing Coda Cell, make sure the following items are taken care before running the setup script:

  1. an empty directory ( /vicepa ) where the file server will put user files. If more than one storage are are is needed for file space on an existing server, such as needed /vicepb , additional space must be a separate partition from /vicepa .
  2. a raw partition for RVM metadata (you can use a file but this will be slow on a medium to large server) This partition must be around 4% of the total size of the files you wish to store under /vicepa (e.g. on a 3.3GB server we use around 130M of rvm data). Consider 10M to be the minimum.
  3. a LOG partition, preferably on a disk by itself. This needs not be large (12MB is large enough for a 3.3GIG partition which is the largest default configuration provided by vice-setup-rvm script).
  4. two secret tokens of _exactly_ 8 characters (eg elephant). The RVM files involve the journalling/transactional aspects of Coda and communicating this information between servers and via the loop-back device when the SCM updates itself. The tokens are used to secure intra-server communication.
  5. If the root partition is not large enough to hold /vice , having alternative measures setup as outlined in Before Begining .

Once you have the above items available, run:

vice-setup

This provides several useful ``canned'' configuration for setting up a Coda server. It does a number of things ``behind the scenes'' as well (such as setting up the directory /vice if it doesn't exist and populating it with the sub-directories backup db srv vol bin ) while asking the following questions:

Enter a random token for auth2 authentication :

This must be _exactly_ eight characters in length. This requirement is due to a bug that hasn't been completely fix yet.

Enter a random token for volutil authentication :

This must be _exactly_ eight characters in length and different that the auth2 token. This requirement is due to a bug that has not been completely fixed yet.

Do you want to start the server at boot time? (y/n)

Answering yes to this question creates the file /vice/srv/STARTFROMBOOT which must be present for either rc.vice or vice.init to start the server at boot time. Removal of this file will prevent the Coda server from starting automatically at boot time.

Is this the master server, aka the SCM machine? (y/n)

For the first Coda machine being setup in a Coda cell, the answer must be "yes". If you are adding a server to an existing Coda cell, the answer must be "no".

If you have answered ``no'' to the above question, you will be asked for the HOSTNAME of an existing SCM:

If you have answered ``yes'' to the above question, you will see the message:


An initial administrative user "admin" with password "changeme" now exists

. and pick up at the question Are you ready to setup up RVM?

If you have answered ``no'' to the above question, you will be asked for the HOSTNAME of an existing SCM:

Enter the hostname of the SCM machine :

This question is only asked if the machine is not the SCM. The hostname entered must be that of the SCM which distributes the Coda cells'database and global configuration files.

Are you ready to set up RVM? [yes/no]

Answer yes to this question or the setup will abort.

What is your log partition?

RVM is the Recoverable Virtual Memory used to keep a copy of Coda Virtual memory on local disk. While this can be a file, it is strongly recommended that it be a small partition on a dedicated disk. If you have set aside, for example, /dev/sdc1 under Linux as the log partition, enter the partitions' device name at the prompt. If you are setting up a *BSD system, you must enter the raw partition name. For example, if you are going to use /dev/sd2e you must enter the raw device name /dev/rsd2e . The maximum template size we provide for and have tested is 30MB.

What is your log size? (enter as e.g. '12M')

The log partition keeps transaction records for Coda volume updates that may not have been writen to the Coda data partition yet. We have not extensively tested values above 30MB and don't recommend more than this. 12M is a good value for a small to medium sized server. The value may be entered as, 12M or 12288 for 12 mega-bytes. Please see section The Data Partition for a more detailed explanation

What is your data partition (or file)?

This allows you to specify a partition or file. We strongly recommend a partition which can reside with other system partitions. Remember, if you use a partition, you are using it "raw" . it does not contain a file system of any kind. Again, if you are using Linux, you may enter a block device name such as /dev/sdc1 . If you are using a *BSD system, you must enter a raw device such as /dev/rsd2e .

What is the size of you data partition (or file) [22M,44M, 90M, 130M]: "

This specifies the the size of the RVM data partition. It must be typed exactly as 22M , 44M , 90M , or 130M . These are associated with default parameters that must be feed to codasrv when it is started. The following table provides the log size to total data storage conversion:

  • 22M is good for up to 500MB
  • 44M is good for up to 1.0GIG
  • 90M is good for up to 2.2GIG
  • 130M is good for up to 3.3GIG
For a detailed explanation of the trade off between different Data log sizes vs. Data Storage Space, please see the section The Log Partition .
Proceed, and wipe out old data? [y/N]
WARNING

if you have entered erronous entries for RVM log and RVM data partitions, you could damage your system. Be sure you entered correct information!

This is the last chance before any action is taken by the setup script. Enter y to commit or N to abort.
Where shall we store your data [/vicepa]?

Older development versions of coda require /vicepa to be present. This version does not. You give a name other than the default we suggest. However, we strongly advise not calling it /code as that can be confused with venus.

Once you successfully complete vice-setup :

And an SCM server is being setup

you are ready to start the update server, the update client and the auth server, as well as the fileserver by typing:

For Linux:

/etc/rc.d/init.d/update.init start
/etc/rc.d/init.d/auth2.init start
/etc/rc.d/init.d/codasrv.init start

For FreeBSD:

/usr/local/sbin/auth2
&
/usr/local/sbin/updateclnt -h `cat /vice/db/scm`  -q coda_udpsrv
&
/usr/local/sbin/updatesrv -p /vice/db
&

For NetBSD:

/usr/pkg/sbin/auth2
&
/usr/local/sbin/updateclnt -h `cat /vice/db/scm`  -q coda_udpsrv
&
/usr/local/sbin/updatesrv -p /vice/db
&

One final step is needed to make the SCM server function. Use the root volume you specified in vice-setup in place of your-root-volume. This step creates the volume, which will be mountable by the client software.


createvol_rep your-root-volume E0000100 /vicepa

Once you have done this, check that the server has started by viewing the log:


xterm -e tail -f  /vice/srv/SrvLog
&

If this is a non-SCM server

You should start auth2, updatesrv and updateclnt first by typing:

For Linux:

/etc/rc.d/init.d/update.init start
/etc/rc.d/init.d/auth2.init start

For FreeBSD:

/usr/local/sbin/auth2 -chk
&
/usr/local/sbin/updateclnt -h `cat /vice/db/scm`  -q coda_udpsrv
&
/usr/local/sbin/updatesrv -p /vice/db
&

For NetBSD:

/usr/pkg/sbin/auth2 -chk
&
/usr/pkg/sbin/updateclnt -h `cat /vice/db/scm`  -q coda_udpsrv
&
/usr/pkg/sbin/updatesrv -p /vice/db
&

This will cause the new server to first obtain the current server files such as /vice/db/ROOTVOLUME and /vice/db/VRList as well as others needed for this server to participate in the Coda cell. Once these files have been retrieved, you may start the server manually for the first type by typing:

For Linux:

/usr/sbin/startserver
&

For FreeBSD:

/usr/local/sbin/startserver
&

For NetBSD:

/usr/pkg/sbin/startserver
&

Once you have done this, check that the server has started by viewing the log:


xterm -e tail -f  /vice/srv/SrvLog
&

NOTE: the above distinction between SCM and non-SCM server is non-trival.

If the codasrv process started correctly, you should see ``File Server started'', if not you need look at the messages in the SrvLog to determine what went wrong. If the messages are not helpful, you should kill the server ( NOTE: The server is designed to continue to stay up so gdb can be attached for debugging):


kill -9 `cat /vice/srv/pid`

and then you can restart the server with:

startserver -d 10
&

Basic Trouble Shooting and Misc.

However, assuming that the server came up, also make sure updatesrv and updateclnt are running by using either top or ps .

NOTE:

you need the package bc (binary calculator on the server). you can either obtain from our ftp site or a mirror site containing your operating system and contrib software.

NOTE2:

E0000100 is the Volume Storage Group set up for you by vice-setup when setting up an SCM. With more servers you can define other groups in /vice/db/VSGDB.

Once this is complete, you should setup a client machine and run Venus and point venus point to this server. Setting up a client machine with venus is covered in the next chapter. Once you have tested and verified the Coda server works from a client, you are ready to adduser more volumes and users!

NOTE: A current limitation of the current vice-setup-user script is that the initial admin account needed is assigned a Coda uid of 500. In order for a client to authenticate a user, that user must exist with the same corresponding user-id in the UNIX password file. If you need to change the admin account uid to suite local uid space, please edit the script vice-setup-user , change the default Coda uid to the needed value and re-run the script. This only applies if you are setting up the SCM.

Otherwise, if this is a non-SCM server, you might consider setting up a singlely replicated volume on the new server to test it.

7.4 Underneath vice-setup

The sections contained here describe what vice-setup does for you. This information is useful for those who either wish to customize our default vice-setup script or wish to have a custom server setup outside the scope of Coda's setup script..

RVM Initialization

RVM initialization requires the selection of several parameters, each of which involve tradeoffs. Although the RVM log and data can be kept as regular UFS files, this is not recommended : Raw partitions have much stronger data consistency guarantees as well performance advantages. It is probably best to plan out the RVM partitions on paper first, taking into consideration both the effect on RVM performance as well as overall disk usage if you choose not to use the values provided in the vice-setup script.

The Log Partition

The size of the log device is based on available space and issues involving truncation. Larger logs provide a longer accessible history of operations, are truncated less frequently, but each truncation will take a longer period of time. Shorter logs truncate more often, but each truncation takes less time. Log size is also strongly related to server startup time as well. We use a 90M log size, on a storage size roughly 3.2GIG spread between two UNIX partitions. (We suggest leaving a little space at the end of the RVM log partition for safety, as RVM automatically adds about one extra page to the amount you specify). Our vice-setup script provides for several default values, the largest of which is 130MB. If you decide to follow the recommendation and use a dedicated partition, creating a 130MB partition will leave plenty of room to expand even if your initial use only suggests a size of 22M, for example..

The log is initialized with rvmutl . At rvmutl s prompt, use the command i , and then specify the size of the log segment. In specifying the size, you can use M for megabyte and K for kilobyte. For example, to initialize a log on partition 0g to eight megabytes:


# @B(rvmutl)
* i
Enter name of log file or device: /dev/sdc1
Enter length of log data area: 22M
* q

The Data Partition

The data segment contains the meta-data of the system such as volume headers, Coda directories, vmond lists, resolution logs, etc. The size of the data segment depends on the amount of disk space for file data, i.e. the size of the /vicep? partitions. As a rule of thumb, you will need about 3-5% of the total file data space for recoverable storage. We currently use 4% as a good value under most circumstances. In our systems the data segment is 90Meg for approximately 3.2 gigabytes of disk space. By making it smaller, you can reduce server startup time. However, if you run out of space on the RVM Data partition, you will be forced to reinitialize the system, a costly penalty. So, plan accordingly.

In initializing (or reinitializing), you need to pick several parameters. The first is the starting address of the recoverable segment in your address space. On our servers we start the RVM segment at 0x20000000 for both Linux and BSD based systems on Intel architecture. The second is the amount of space to give the recoverable heap. The heap will obviously grow over use, so plan accordingly. Our heap space is 0x1000000. We suggest that for the last parameters you use 1Meg (0x100000) for the static area, use 80 free lists (or nlists), and a chunksize of 32. These numbers will work well with the internal structure of the fileserver and are provided as the default vales in the setup script vice-setup .

To perform the data initialization, run the program rdsinit . rdsinit takes two parameters, the names of the RVM log and data devices. For example, to initialize one of our Intel based servers:


# rdsinit /dev/hdc1 /dev/sdb1
Enter the length of the device /dev/hdc1:119070700 
Going to initialize data file to zero, could take awhile.
done.
rvm_initialize succeeded.
starting address of rvm: 0x20000000
heap len: 0x1000000
static len: 0x100000
nlists: 80
chunksize: 32
rds_zap_heap completed successfully.
rvm_terminate succeeded.

NOTE: the use of the decimal value for the length of the device and the use of hex values for the address and lengths of the next three values.

Update Monitor

The update monitor is used to propagate changes to the Coda server databases to all servers from the SCM. Client update processes run on all the Coda servers and connect to a server udpate process running on the SCM. The server process uses the file /vice/db/files to determine which files should be kept consistent on all the servers. See the updateclnt (8) and updatesrv (8) man pages for more details.

Create the file /vice/db/files on the SCM. Currently our /vice/db/files looks like this:


VLDB
auth2.pw
auth2.tk
pro.db
servers
hosts
vice.pdb
vice.pcf
volutil.tk
VRDB
files
VSGDB
dumplist
scm
ROOTVOLUME

Authentication Database

Coda uses an authentication database that is separate from the UNIX password file. This database is maintained by the SCM. When someone authenticates to Coda, their password is checked against this database and that person is issued a token if they successfully authenticate. This section describes how to initialize the authentication database.

On the SCM

If the server you are installing is the SCM, you must set up the initial authentication database. Make sure you add at least one user to the System:Administrators group. You need to do the following:

On Servers OTHER than the SCM

On servers other than the SCM, the Update monitor will ensure that the proper database files are propogated from the SCM. However, you must copy the file /vice/db/volutil.tk to the new server for the Update monitor to work.

Creating the Volume Storage Group DataBase (SCM only)

Every combination of servers setup expects to use as a VSG which must be given a multicast address in /vice/db/VSGDB . This file must be created by hand; please refer to VSGDB (5) in Appendix XXX for more details.

7.5 Starting the File Server

We have provided a script called startserver which reads in the file /vice/srv.conf which contains the startup information needed by codasrv . The syntax of /vice/srv.conf is:


-rvm
<
log_partition
>
<
data_partition
>
<
size_of_log_partition
>

The vice-setup-rvm contains the canned default value for the corresponding value entered through vice-setup . Should you deviate from one of these canned default values, you will need to either create or modify the /vice/srv.conf file to indicate the correct data size of the data partition. For example, one of CMU's Coda servers contains:

-rvm /dev/hdc1 /dev/sda1 119070700

You should be careful to specify the correct partitions for the RVM log, data segment, and size of the data segment.

Once this file has been correctly edited, you can start the server with startserver & .

Creating the Root Volume

If you are installing any server other than the SCM, please use the instructions in section XXX to create new volumes on this server.

The root of the Coda file system must reside on the /vicepa partition for correct functioning. Since the SCM probably also acts as a server, place the root of the Coda file system in the /vicepa partition on the SCM. Create the root volume by using either one of the following commands

% createvol_rep codaroot < VSG_ENTRY > /vicepa

% createvol codaroot < hostname > /vicepa

the former is for replicated root volume, where < VSG_ENTRY > is the entry in the VSGDB for the volume storage group for this volume; the later is for non-replicated root volume, where < hostname > is the name of the host which hosts the root volume. Since it is likely that the root will be cloned and read-only replicated, you might wish to create this volume as singly replicated for simplicity.

These commands will create the /vice/db/VLDB , /vice/db/VRDB , /vice/vol/VolumeList , /vice/vol/BigVolumeList , and the /vice/vol/AllVolume files.

Create or modify the file /vice/db/ROOTVOLUME to contain the volume name (E.g. codaroot ) corresponding to the root volume as listed in the file /vice/vol/VRList .

Creating a readonly root

You may wish for the root to be readonly. To do this, first create a root volume as above. Start a venus which can talk to this server. Setup the directory structure you want in your root volume. Once you are satisfied with the structure, clone the volume and dump it to a disk file with the the volutil program. Now restore the dump file to the servers in the desired VSG for the readonly volume. Be sure to specify the volumeId and the volume Name for the new volume:

% volutil restore < filename > /vicepa < newrootname > < volid >

Then modify the /vice/db/ROOTVOLUME file to contain the new rootvolume name, < newrootname > .


Next Previous Contents