Jan Harkes wrote:

> > Think about a ftp site (using CODA kernel-module and podfuk+ftpfs) which
> > has large INDEX files (~60MB). File manager likes to determinate the
> > datatype - reads the first two bytes - but triggers the download of
> > 60MB.
> > Quite efficient, isn't it ? ;-(((
> >
> > (Possible) solution:
> > Allow random-access (read/write etc.) to remote files.
> > What about an interface to the SYSV-VFS layer, e.g. moving more stuff
> > from kernel to userspace layer ?
>
> This would actually introduce far too much overhead in context switches
> between the application and the cache-manager. Besides, as far as Coda
> is concerned, it makes it impossible to guarantee consistency. However,
> there is another solution which we've been thinking of.
>
> When a large file is opened, CODA_OPEN could return early, f.i. when
> the first 8-64KB have been fetched. The kernel would get a `lease' on
> accessing these first pages (both read and write), while the cache
> manager pulls in the rest of the file.
>
> When the application accessing the file seeks or reads past it's `lease'
> it is blocked until the data is available, and the cachemanager has
> returned a new lease. However when the application is done and closes
> the file before everything has been fetched, the ongoing fetch can be
> aborted.
>
> In this model, we can keep streaming data into the container files as
> efficiently as possible, while at the same time allowing some early
> access to the containerfile. One of the big problems is that most
> applications don't handle read/write errors very well, so an interrupted
> transfer (disconnection) would lead to silently truncated files. Mostly
> due to user `error', when someone opens a file in an editor, makes some
> changes but doesn't notice the end of the file was lost and saves it back.

I was thinking about a completly different solution: What about letting venus/podfuk decide at OPEN to download the complete file to the local system (e.g. into the "cache") or handle each operation (read/write/seek/locking etc.) manually and map them to the remote file. This would give us the control whether file operations should be executed on the "cached" file or on a remote file. This would be a more universal approach as this also supports both your and my idea, too (for example: venus decides at OPEN to access the remote file readonly but triggers the download into the cache in _parallel_. If download is complete it switches from remote to cached file).
And it would allow other projects to use the same interface, either using cached operations, direct operations or both.

Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) Roland.Mainz_at_informatik.med.uni-giessen.de
  \__\/\/__/  gisburn_at_informatik.med.uni-giessen.de
  /O /==\ O\  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
 (;O/ \/ \O;) TEL +49 641 99-13193 FAX +49 641 99-41359