[ILUG] ftp.heanet.ie

[Paul Jakma]

On Sun, 11 Jan 2004, Liam Bedford wrote:

> Is there an option missing? I think the G5 machines are supported
> in linux now.. possibly even in 64bit mode properly.

That depends on whether the G5 machine has an IO-MMU (ditto for
opteron). A 64bit machine with tonnes of RAM without an IO-MMU has
the same problems as a 32bit PAE machine when it comes to IO. It
seems that at least the IBM iSeries has IO-MMU - no idea about other
G5 platforms.

Yep. not good either. But that affects translation of virtual
addresses -> physical. Not immediately relevant to IO.

> (maybe that's what you mean by all that technical mumbo jumbo :))

Bounce buffers: PCI has a 4GB address space (well PCI can have a
64bit address space using DAC, but that eats one PCI bus cycle (DAC =
dual address cycle) and AFAIK, isnt useable on 32bit PCs, only on
64bit machines). On PC's (which dont have IO-MMUs), the PCI bus
address space maps directly to physical address space.  So, if you
have a process (eg ftpd, httpd) which has data located somewhere
above 4GB in physical ram then you have a problem if you want to tell
a PCI device to DMA that data - as the device has no way to address
that data. Instead, the kernel must first make a temporary copy of
the data to somewhere below 4GB. Ie buffers are kept some where the
device can access and data is bounced via this buffer to/from
process<->device. Having to copy data before/after IO really sucks
performance wise.

Server class 64bit machines have dealt with this problem using
IO-MMUs. Ie ability for the system bus controllers to map bus address
space to arbitrary regions of physical address space. Eg, if data is
located at 6GB physical memory, the kernel programmes the host
controller to map, eg, PCI address 2GB to 6GB physical (for some
length). When the PCI devices DMAs to/from 2GB the controller fetches
to/from 6GB - sort of like NAT for memory :). More sophisticated
IO-MMUs can generally do very fine grained mapping, allowing one
block of PCI address space to be mapped to a whole bunch of
discontigious physical addresses - called scatter-gather. (though,
most PCI devices tend to support this themselves).

Ironically, most /desktop/ PCs actually /do/ have an IO-MMU where
quite a few server PCs do not - the GART is a very basic IO-MMU,
intended for AGP cards to map textures in system RAM. However, the
GART window applies to all of the PCI address space, hence it can be
used to remap memory for any PCI device. It seems that x86-64 uses a
slightly extended version of GART to handle io-mapping (bit icky).

Alpha and USparc machines /definitely/ have good IO-MMU support. IBM
G5 iSeries and Itanic appear to have it too, but no idea how good it
is.

NB: i'm probably completely wrong as usual.

> L.

regards,
-- 
Paul Jakma	paul at clubi.ie	paul at jakma.org	Key ID: 64A2FF6A
	warning: do not ever send email to spam at dishone.st
Fortune:
Never invest your money in anything that eats or needs repainting.
		-- Billy Rose