More compatibility support for encoding (ANSI and UTF-8)

pnotepad.org forums » discussion

More compatibility support for encoding (ANSI and UTF-8)

(16 posts) (4 voices)

Started 1 year ago by rc-flitzer
Latest reply from simon
2 Members Subscribed To Topic

Tags:

rc-flitzer
Member

Hello!
At first I'd like to thank you, Simon, for the newest version 2.1. It has been a while since I looked here for new versions and this one impresses me. Native support for extensions without the need to install Python (and a nice text sorting extension) and native UTF-8 support, finally. You see, UTF-8 is the future, but already for years, so PNotepad was already late. ;-)

If you only edit UTF-8 files everything is nice. But I also have many many old ANSI (1252) files, and it would be great to use PNotepad as all-purpose editor for both encodings. Here is why is that not yet possible:

1. I want to create new files automatically in UTF-8. But when I set „Code Page“ to Unicode (UTF-8), all opened ANSI files (with automatc encoding recognition) are displayed wrong. The editor recognizes ANSI encoding and says so in status line, but instead of an „ü“ I only get a square with „FC“ written in it. So it seems a bug: the text window wants to display the text in UTF-8, but it is ANSI and it should be displayed so.
That's really annoying. Either I can't open my old ANSI files properly or I manually have to change encoding from ANSI to UTF-8 on every single new-created document.

2. There is still no easy conversion between encodings, which I proposed in July 2008: http://pnotepad.org/forums/topic/443#post-1558
And yes, I rechecked the behaviour. It's still the same. It would be so easy to implement an option to set/change the encoding in the save dialog window.

I already created two bug reports (875 and 876) for these issues. Hopefully I'm waiting for them to be finished (at least 875). :-)

Posted 1 year ago #
simon
PN Developer

Hi, thanks for the feedback. I really would like to get this stuff fixed, and sorry I haven't got around to it yet. I'll try and prioritise these bugs, but maybe you could help:

What would make these requests easier to work on would be a set of sample files that I can use to verify results, so for each conversion that I might do:

1. Open source file [x]
2. Convert to encoding blah
3. Save as new file, and compare to file [y]

If I had a set of x and y that were known to be good, then testing this is much easier. If you were willing to help on this (or anyone else for that matter, code or samples) it might speed things along.

Thanks.

Posted 1 year ago #
rc-flitzer
Member

I created a file which has all german umlauts (äöü and ß) in a line. Then there are all 122 non-ASCII characters from Windows-1252 in another line. This has been saved both with encodings Windows-1252 and UTF-8 (with BOM). Is this helpful? Most European Windows versions use 1252. For Kyrillic and other encodings (Chinese, Thai, Turkish) it's a bit more complicated to create these files, but I could also do this if neccassary.

http://rainer-grabbe.de/developer/pnotepad/windows-1252.txt
http://rainer-grabbe.de/developer/pnotepad/utf-8.txt
Maybe your browser does not display any text in one of these files because of encoding. Simply save them.

Posted 1 year ago #
horus
Member

Yeah, I'm also quite annoyed by this problem. But I also see that PN2 is quite good at detecting BOM-less UTF-8 files. So maybe it's not really necessary to set "Code Page" to UTF-8; just default should be good enough.

However, there're two related bugs which prevents us from using UTF-8 without fear of losing data:
http://code.google.com/p/pnotepad/issues/detail?id=1020
http://code.google.com/p/pnotepad/issues/detail?id=1021

If these two bugs are fixed, PN2 could reach another level of perfection :)

Posted 1 year ago #
simon
PN Developer

Happy to say that verison 2.1.5 (due for release shortly) will include encoding conversion between UTF-* and ANSI. This should solve this for good!

My thanks to rc-flitzer, and all those who reported bugs that helped me find the right way to fix this.

Posted 1 year ago #
rc-flitzer
Member

"Right way to fix"? No. Still not fixed. (I'm sorry for not checking the forum a longer time, but I started a new job and it is quite time-intense.)

I set Options -> General -> Defaults -> Code Page to "UTF-8", because I want new documents to be created as UTF-8 ones. But we still are not able to open our old ANSI encoded documents without hassle. See point 1 in my first posting.
And I can't create anymore ANSI documents. If you have (with UTF-8 setting) an open utf-8 document, convert it to ANSI and store it, it is still utf-8 (without BOM). Not that I want to convert _to_ ANSI, but a bug is a bug. *sigh*

Why do you still have this kind of double or triple setting? And all this unicode/ansi not really running code? There is "Code Page", "Character Set" and "Encoding" (in "File" menu). This is not useful. I mentioned this way more than a year ago and also gave you the solution: just make _one_ setting for an encoding/character set the user wants to have.

I'll tell you what. Make UTF-8/encoding support the same as in Notepad++. And make the settings the same. Look at its "Encoding" menu and you'll see what I mean. It is just _simple_ and it works.

... and search / replace still is unusable in UTF-8.

Posted 1 year ago #
simon
PN Developer

Sorry about this, clearly still some work to do.

The problem with converting to ANSI is caused because I use the default code page when converting to ANSI. Of course if your default code page is UTF-8 then you get UTF-8 rather than any ANSI encoding.

I guess what I really want to make available is:

1. Default encoding for new files.
2. Default code page for ANSI files (?)

All other settings should be hidden.

As for Search, I'm surprised that's not working. For example, if I use the above utf-8.txt example file and search for ÊËÌÍÎÏ then it finds both upper and lowercase versions which is as I expect. Could you provide repro steps?

Posted 1 year ago #
simon
PN Developer

Also, I will of course look at the Notepad++ options later and see what they do differently.

Posted 1 year ago #
simon
PN Developer

Ok, I have done the work to switch to default encoding. There are still two options, but they are different and now more clear:

`
Default Encoding: [ANSI ]
UTF-8
UTF-8 No BOM
UTF-16 Little Endian
UTF-16 Big Endian

Multi-Byte Character Support for ANSI: [Default ]
Chinese Big 5
etc.

The character set option is gone and is unlikely to return, the above should be enough. I'm happy to provide a private testing build to anyone to try this out and give feedback before the release. Let me know!

Also still interested in search bugs when UTF-8 encoding is in use.

Posted 1 year ago #
rc-flitzer
Member

I'm sorry about yesterday. This was not an appropriate tone, I was a bit harsh.
Your changes sound good. Do you want to send me a copy of the testing build? The e-mail adress should be available for you as forum administrator.

"Of course if your default code page is UTF-8 then you get UTF-8 rather than any ANSI encoding."
Hm, that's quite wrong. Convert to ANSI should always do what it says. *g* In N++ there is no difference between utf-8, utf-16 and several 8-Bit-Encodings. You either say "this file now has this encoding" or you say "convert to that encoding". For new files there is an option (settings -> new/open/save) where you set either ANSI, UTF-8 (with/out BOM) or some UTF-16 (UCS-2? That's old.) variant.

On second thought, your menu structure for encoding is quite ok. You can set an encoding for new files and you can convert files to other encodings. It just has to work properly.^^

About search / replace: now it works, strange. Sometimes it adds some characters to the beginning or end of the file. And a second bug: if you have marked some characters for replacing, the search box contains the characters as ANSI encoded bytes, e.g. if I mark "ä", then search/replace, the search box contains "Ã¤". Of course only when my current file is UTF-8. This is again something where you give utf-8 some different behaviour instead of treating it as just a different encoding... (Seems to me you never even thought of UTF-8 when writing the first versions. *g*)

Posted 1 year ago #
simon
PN Developer

@rc-flitzer No problem, thanks for continuing to provide feedback!

I've uploaded a portable build with two changes here:

http://pnotepad.googlecode.com/files/portable-pn2152224.zip

Change 1: As detailed above, switch to Default Encoding
Change 2: Fix issue with passing selection into Find Dialog when using UTF-8

As part of Change 1, the UTF-8 conversion thing is fixed. When I explained what I had done above, I meant to say that this was a bug and not what I had intended :)

Hopefully this will get us closer to where it needs to be.

Posted 1 year ago #
rc-flitzer
Member

Very nice :-) Now this is how I wanted PNotepad to be. Search/replace works well. Oh I found another bug: “replace all” doesn’t work in UTF-8 mode. He still says „0 occurences replaced”, though replacing one by one works. But I think this is the last bug. ;)

Posted 1 year ago #
simon
PN Developer

Great, thanks for the feedback - 2.2 will be a much better release thanks to you!

I'll look at the Replace All thing.

Posted 1 year ago #
Nux
Member

In my case I have a file:

zażółć
zażółć

And I'm replacing all "ż" with "z". It says "2 occurrence(s) replaced." but nothing is replaced.

I've also found something that might be related, but I'm not sure if it's about display or selection...
http://code.google.com/p/pnotepad/issues/detail?id=1077

But I'm very happy it works at all :-).

Posted 1 year ago #
rc-flitzer
Member

When it says “2 occurrence(s) replaced.”, it has indeed replaced two characters, but not the ż … Try replacing with e.g. “M”, then you will see the result: Mażółć. Long story short: it replaces the wrong character.
This is a serious thing. Why will a character like ż be treated as z?

Posted 1 year ago #
simon
PN Developer

This is the same as the Replace All bug that you reported above. I don't have the code here as I'm at work, so can't debug it, but I suspect that when you convert "ż" to Windows-1252 rather than UTF-8 we are getting a fallback character which is z. Therefore the replacement is finding z.

When I fix the Replace All bug this will go away too.

Posted 1 year ago #

RSS feed for this topic

Reply

You must log in to post.