[ILUG] Bigger pipes...

[Kenn Humborg]

> On Wed, 25 Apr 2001, Paul Jakma wrote:
> 
> > > takes 20secs of CPU time.  Doing them together takes 40secs,
> > > while, in theory, the I/O and CPU could happen in parallel, giving
> > > a run time of 20secs.
> >
> > hmm... usually i'd defer to your wisdom kenn, but in this case i can

This'll learn you...

> > not see how you think that you can reduce 20secs for reading, 20secs
> > for gzipping to less than 20+20secs.
> 
> just to be clearer and avoid obvious retorts:
> 
> ...i do not see how you think you can get extra parallelism beyond any
> you are getting already...

This is what's happening:

       gzip            NFS
       read()          .
       .               get data from server
       uncompress      .
       read            .
       .               get data from server
       uncompress      .
       read            .
       .               get data from server
       ...

Now, imagine if NFS could read ahead while gzip is uncompressing:

       gzip            NFS
       read()          .
       .               get data from server
       uncompress      get more data
       read            return immediately
       uncompress      get more data
       read            return immediately
       uncompress      get more data
       read            return immediately
       ...

This runs the network and CPU in parallel.

And, in fact, mbuffer does the job nicely.  The 40secs is currently
reduced to 24secs.  I use a dd to pull across NFS which feeds
a nice big mbuffer process, which feeds gzip.

Thanks, Padraig!

BTW, Kevin, I already adjusted rsize & wsize to 8K.

Later,
Kenn