I think that RPC2 needs 3 important changes. Variable keylength instead
  of fixed 8 byte keys, allow for padding (expanding) the encrypted
  packets, and an integrity checksum. I also believe that adding
  encryption to SFTP is a lower priority issue, the bulk data encryption
  could possibly be done (more efficiently) at a higher layer.

Well, it's lower priority if the code that does encryption and
_integrity protection_ of the SFTP data exists.

  Because each RPC2 packet can be lost, a CBC cipher is not doable.
  However it should be possible to add an IV to each packet which makes
  identical packets encrypt differently (similar to WEP encryption).
  (ofcourse that could get us the whole nastiness of trying to avoid weak
  IVs depending on the cipher we use etc).

Sure, and similar to IPsec ESP.

  Which is why 'off-the-shelf' solutions like IPsec and SSL are nice,
  smart people have put some serious thought into it. Too bad there isn't
  an SSL over UDP version.

People are talking about that, but GSS-API's gss_wrap is essentially
what we need.  I realize that the blocking behavior is an issue, but I
am pretty sure that with krb5 this happens only on the client (in
getting a service ticket from the KDC) and not at the server (which is
just verifying).  It should be possible to have the clog program block
and then insert GSS-API state into venus.  Or, to use krb5 natively
and extract the krb tickets and put that in both venus and codasrv.
Then we can use operations that (at least basically, perhaps always)
don't block.

  Authentication is mostly in rpc2a.c, there is one big function
  'RPC2_NewBinding' that steps through the whole authentication sequence.
  I just read through it and it is the suggested alternative in the ban
  paper.

So changing there and in the packet format/encrypting part, as well as
perhaps changing key management so that it is expected that different
servers have different keys (which is good crypto practice).  This may
mean a thread/process to grab tickets for servers as we need them, but
that doesn't sound so bad.

-- 
        Greg Troxel <gdt_at_ir.bbn.com>