CORRECTION - Re: [ILUG] Editing unicode text files.

[Niall O Broin]

On 17 Feb 2007, at 20:29, Brian Foster wrote:

>   |  I concur with Kae's point here:  Until you can simply
>   |  cat(1) the file and see what you _should_ see, things
>   |  are not set up correctly.  [ ... ]
>  apologies, that isn't quite what I meant to say:
>   if the files in question are UTF-16, then on a
>  typical Linux system (often UTF-8 nowadays), the
>  above check clearly “won't work”.  the above test
>  is best applied when the terminal's encoding is
>  compatible (read: identical unless you know what
>  you are doing) with the file's.
>
>  however, as Kae has pointed out, vim(1) at least
>  should be able to deal with the situation even if
>  the files are, as I (now) suspect, UTF-16LE (esp.
>  if you are using a UTF-8 terminal window).

Damn, but this whole area is a minefield. The committee has recently  
decided
to have more regular ILUG meetings with a talk format, and I for one  
would
love if somebody with a reasonable level of understanding of this  
whole area
would volunteer to de-mystify the rest of us.

My personal current fight with this with a PHP4 site which I'm trying  
to migrate
off a RHEL 3 box, with a 3.23.xx version of MySQL. There is a small  
number of
characters used which are non ASCII (the German ones with umlauts and  
the sz beta
character) and for the life of me, I cannot persuade those to display  
correctly on
the new site, no matter what magic options I use to dump and restore  
the DB.

I expect if I had a better fundamental understanding of the whole  
business I might
be better off, so do we have a willing expert?


Niall