[ILUG] ftp.heanet.ie

[Colm MacCarthaigh]

On Sun, Jan 11, 2004 at 12:29:23AM +0000, Paul Jakma wrote:
> On Sat, 10 Jan 2004, Colm MacCarthaigh wrote:
> 
> > Buying Alpha at this stage is very risky :) More like an itanium !
> 
> Or Opteron, though I dont know how stable x86-64 is. Thing about the
> alpha though is that just about all of the host bus chipsets (Pyxis
> and up, probably even before Pyxis) have built-in IO-MMUs, which 
> probably would be useful for what you are doing :)

We've looked at Opteron, but it's not available on the infrastructure
we need - itanium is there, but it's simply too costly for what we're
doing right now - there are better ways of utilising what we've got :)
And there's still a bit more to come.

ftp.heanet.is is as much about stress testing our hardware and kernels
that then go on to provide our critical and value-added services as
anything else. All of our services run on the same platform, and because
of ftp.heanet.ie we gain knowledge and experience which comes in very
useful in the provision of those services. We see ftp.heanet.ie as 
part of our network and service development and testing infrastructure,
aswell as a very important service in its own right.  So there's a lot 
of factors that lead to us keeping it on x86 for now :)

> Your other (and saner in terms of support) option is sparc64 - i
> think most decent Sun sparc64 hardware also has IO-MMUs. No idea how
> well linux runs on them though (seems there possibly are actually
> people running linux on sparc64 in production judging from posts to
> l-k).

ftp.heanet.ie has been in operation now for about 16 months, in that time
we've spent about one 10th of the annual budget of our European
equivalents. We outperform them all, by a factor of at least 10 and in
most cases more. They all run on Sparc64, and all on multiple Sparc64
boxes at that.

I remember a few months ago someone suggested we look at HP kit, since
we run alphas in production and since ftp.kernel.org runs on one -
it must be good. Then the people at kernel.org added that "Current
utilisation" to http://www.kernel.org/, and it turns out we're busier
than ftp.kernel.org - usually by a factor of 2. After that, comparisons
seemed hardly worthwhile!

> > That it is, but we really need the numbers :)
> 
> Only the first 4GB, at best, of that physical address space is
> IO-able though, as i'm sure you're aware :) the rest has to be bounce
> buffered which isnt conducive to performance - whats the remaining
> 8GB for? (and how, if at all, do you make sure your IO constrained 
> processes get allocated at <= 4GB physical?)

We run the numbers, and tweak :) That extra 8Gb of RAM isnt for direct IO
exactly, it's for caching so that we never have to hit disk - that's it's
major function. Process wise, we mainly have a few thousand instances
of Apache - tuned to within an inch of it's life. On it's own, the
processes don't even go past a gigabyte, rsync will now and then nab a
few hudnred megs for itself - but in general we could run ftp.heanet.ie
on about 2gb of RAM without processes ever having to worry about not
having enough memory.

The rest is there because even with PAE, it's still *much* better to
cache the files in VM than to pull it from disk. So at any given time,
we should be able to serve most requests with memory operations, rather
than disk. PAE is more painful on CPU than we would otherwise be, but
that's why we've upgraded from a 1.8Ghz Xeon to dual 2.8Ghz :)

The route we've taken is a bit unusual - we havn't gone with blindlingly
fast disks and 64bit scalable architechtures, we have 6Tb of IDE disks at 
the back of it all, with x86 and clever caching in the middle - but it 
turns out to be a lot better value for money than alternatives :)

In it's current configuration, ftp.heanet.ie has peaked at 444Mbit/sec,
and our current requests per second threshold is over 1,300. In
testing, we've happily saturated it's Gigabit ethernet without
complaint. That's pretty damn good, all we need now is for those pesky
kernel vulnerabilities to not happen ;)

Actually, does anyone know what the story with XFS for 2.4.24 is?
All they have up is;

	ftp://oss.sgi.com/projects/xfs/patches/2.4.24/README

Which seems ludicrous, as the kernel has been released. Usually XFS
patches are out in a matter of hours.

-- 
Colm MacCárthaigh                        Public Key: colm+pgp at stdlib.net