Part of the EllisLab Network
   
1 of 2
1
BUG: UTF-8 does NOT work!
Posted: 02 March 2010 11:05 AM   [ Ignore ]  
Grad Student
Avatar
Rank
Total Posts:  31
Joined  08-20-2009

Hello everybody,
I am developing an extremely simple website using CodeIgniter. First of all I’d like to say that I did all the things listed in Phil Sturgeon’s blog post.

Now there are two things that are not working properly:

1) First, UTF-8 is not detected by CodeIgniter. This means that despite all the steps described in Phil Sturgeon’s blog post, data retrieved from the database are not treated as UTF-8.
2) The database system in codeigniter fails to retrieve data in UTF-8.

I will explain how I came to those two conclusions:

Here is a method that I use in one of my models:

public function get_one($id)
{
    $this
->db->select('idTexte, titre_fr AS titre, texte_fr AS texte, intro_fr AS intro')
             ->
from($this->_table)
             ->
where($this->_primary_key$id)
             ->
limit(1);
    
$query $this->db->get();
    
    if (
$query->num_rows() == 1)
        return 
$query->row();
    else
        return 
FALSE;
// End of get_one 

And here is the result I get from the database (result displayed with var_dump()):

object(stdClass)#19 (4) {
  
["idTexte"]=>
  
string(7"aPropos"
  
["titre"]=>
  
string(42"À Propos - ÃƒÂ  propos Présentation"
  
["texte"]=>
  
string(156"<p>&#160;c'est cool, c'est super!!!!!! HéHé!</p>
<p>&#160;À Propos - ÃƒÂ  propos</p>
<p>À Propos - ÃƒÂ  propos</p>"
  
["intro"]=>
  
string(13"<p>&#160;</p>"

Now look at the equivalent method using a new mysqli connection:

public function get_one($id)
{
    $mysqli 
= new mysqli('localhost''root''root''the_database');
    
    
$_q 'SELECT titre_fr AS titre, texte_fr AS texte, intro_fr AS intro FROM table_textes';
    
    if (
$result $mysqli->query($_q))
    
{
        
while ($row $result->fetch_object())
        
{
            var_dump
($row); // DEBUG <-
            
die();
        
}
    }
// End of get_one 

And here is the var_dump() output:

object(stdClass)#19 (3) {
  
["titre"]=>
  
string(35"À Propos - Ã  propos Présentation"
  
["texte"]=>
  
string(142"<p>&#160;c'est beau, c'est cool, c'est super!!!!!! HéHé!</p>
<p>&#160;À Propos - Ã  propos</p>
<p>À Propos - Ã  propos</p>"
  
["intro"]=>
  
string(13"<p>&#160;</p>"

There is clearly something wrong with the CodeIgniter database system. The difference between the two var_dump outputs is that if I use PHP’s utf8_decode on the retrieved data, the mysqli output is displayed correctly:

CodeIgniter’s output:

object(stdClass)#19 (4) {
  
["idTexte"]=>
  
string(7"aPropos"
  
["titre"]=>
  
string(35"Ã? Propos - Ã  propos Présentation"
  
["texte"]=>
  
string(142"<p>&#160;c'est beau, c'est cool, c'est super!!!!!! HéHé!</p>
<p>&#160;Ã? Propos - Ã  propos</p>
<p>Ã? Propos - Ã  propos</p>"
  
["intro"]=>
  
string(13"<p>&#160;</p>"

MySQLi’s output:

object(stdClass)#19 (3) {
  
["titre"]=>
  
string(32"À Propos - à propos Présentation"
  
["texte"]=>
  
string(136"<p>&#160;c'est beau, c'est cool, c'est super!!!!!! HéHé!</p>
<p>&#160;À Propos - à propos</p>
<p>À Propos - à propos</p>"
  
["intro"]=>
  
string(13"<p>&#160;</p>"

These two last code snippets prove my first point: CodeIgniter doesn’t recognize UTF-8:the use of utf8_decode() should not be necessary for MySQLi’s output to get displayed correctly.

Now, I prepended the title of this post with BUG because it’s the most likely explenation. Also, I downloaded a fresh version of CodeIgniter and reran all the tests and I got the same results, which proves once again that those two problems I have are actually bugs.

Can anyone confirm what I am saying, and help me solve the problem please?

 Signature 

GitHub: http://github.com/AzizLight/
Forrst: http://forr.st/-AzizLight
StackOverflow: http://stackoverflow.com/users/123016/aziz-light

Profile
 
 
Posted: 02 March 2010 12:13 PM   [ Ignore ]   [ # 1 ]  
Summer Student
Total Posts:  2
Joined  03-02-2010

Hello,

It’s weird you’re having this issue. I’ve developed several sites with CodeIgniter and no problems with UTF-8.

Can you tell me something.. what IDE are you using? Please tell me you’re not using Dreamweaver.

Regards.

Profile
 
 
Posted: 02 March 2010 12:17 PM   [ Ignore ]   [ # 2 ]  
Grad Student
Avatar
Rank
Total Posts:  31
Joined  08-20-2009

I use Textmate. (I will never use Dreamweaver, or pay for it for that matter…)

Also I am sure that this is not the issue anyway. I created a small script that is independent and not connected to CodeIgniter in any way. All this script does is connect to the database to retrieve data. I didn’t have any issue with UTF-8. Hence, the problem comes from CodeIgniter.

 Signature 

GitHub: http://github.com/AzizLight/
Forrst: http://forr.st/-AzizLight
StackOverflow: http://stackoverflow.com/users/123016/aziz-light

Profile
 
 
Posted: 02 March 2010 12:24 PM   [ Ignore ]   [ # 3 ]  
Summer Student
Total Posts:  2
Joined  03-02-2010

Okay, I remember having issues with accents / encoding. And the problam was the way I was saving the scripts as. If you save your scripts encoded in ANSI, 100% sure you’ll have issues with accents and data retrieved from a database.

I use Notepad++, this awesome opensource IDE has a menu called: “Format” which allows you to convert any text file into UTF-8 without BOM.

All your scripts must be saved to this format in order to get rid of all the symbols, and of course add the meta tag with utf-8 encoding.

What collation are your database tables? I use utf8_general_ci for spanish accents.

Regards.

Profile
 
 
Posted: 02 March 2010 12:31 PM   [ Ignore ]   [ # 4 ]  
Research Assistant
Avatar
RankRankRank
Total Posts:  741
Joined  09-11-2008

First of all your database char set should be set for UTF-8 and just set you html meta header as UTF-8 and it works fine, now a days working on multi language site which has English and Arabic for the site, and its working fine just setting header UTF-8 and database accepting UTF-8 char.

 Signature 

CI,JQuery,Google Maps | widget with CI loader | Thumbnail, Image Resize, Image Crop Helper | CI shortcode

Profile
 
 
Posted: 02 March 2010 05:43 PM   [ Ignore ]   [ # 5 ]  
Research Assistant
Avatar
RankRankRank
Total Posts:  335
Joined  05-09-2008

I’ve had a similar experience as @umefarooq; setting the db char set and html header to uTF8 does the trick.

BTW, the post name ‘BUG: UTF-8 does NOT work!’ is a bit excessive, no? Seems like you’re doing sth wrong…

 Signature 

http://www.starfishwebconsulting.co.uk


Quicksnaps - CI Photo gallery app

Profile
 
 
Posted: 03 March 2010 08:56 AM   [ Ignore ]   [ # 6 ]  
Grad Student
Avatar
Rank
Total Posts:  31
Joined  08-20-2009

Thanks for all your replies. The reason I called this a “BUG” is because accents don’t appear correctly even though my db has a UTF-8 charset, I have have a meta tag that sets the content to UTF-8, I used php to set the content type to UTF-8 too (using the header() function), so it should work, but it doesn’t. The weirdest thing is that if I take the same data from the same DB using plain php/mysqli, everything works fine. But if I use it withing a codeigniter install, it does’t work at all…

I also tried to create a new CI project and connect to the same database but it still didn’t work.

About the texteditor, as I said before, I use Textmate, and the file encoding is set to UTF-8 in the preferences, so there shouldn’t be any problems…

At this point I have no clue what the problems is, but my logic says that it comes from CodeIgniter…

 Signature 

GitHub: http://github.com/AzizLight/
Forrst: http://forr.st/-AzizLight
StackOverflow: http://stackoverflow.com/users/123016/aziz-light

Profile
 
 
Posted: 03 March 2010 11:10 AM   [ Ignore ]   [ # 7 ]  
Summer Student
Total Posts:  5
Joined  01-15-2008

Try to send yourself the encoding,

header('Content-Type:text/html; charset=UTF-8'); 

Other solution is to use (if you can) PHP ini directive per directories

create a php.ini file in the directory where the view resides en put this code:

ini_set('default_charset''UTF-8'); 

If nothing works, then maybe the problem is the encoding of apache, you can try:

Create a .htaccess file to change the character encoding and put this code:

AddCharset UTF-.php 

You can set the extension to whatever yo need .html, .htm etc.

Hope this helps.

Profile
 
 
Posted: 05 April 2010 06:49 AM   [ Ignore ]   [ # 8 ]  
Summer Student
Total Posts:  3
Joined  04-05-2010

How did you resolve this finally? I am facing the same issue as you…plain php/mysqli works fine, but codeigniter doesn’t. Note that the data was inserted into DB by java program (not php).

AzizLight - 03 March 2010 01:56 PM

Thanks for all your replies. The reason I called this a “BUG” is because accents don’t appear correctly even though my db has a UTF-8 charset, I have have a meta tag that sets the content to UTF-8, I used php to set the content type to UTF-8 too (using the header() function), so it should work, but it doesn’t. The weirdest thing is that if I take the same data from the same DB using plain php/mysqli, everything works fine. But if I use it withing a codeigniter install, it does’t work at all…

I also tried to create a new CI project and connect to the same database but it still didn’t work.

About the texteditor, as I said before, I use Textmate, and the file encoding is set to UTF-8 in the preferences, so there shouldn’t be any problems…

At this point I have no clue what the problems is, but my logic says that it comes from CodeIgniter…

Profile
 
 
Posted: 05 April 2010 07:12 AM   [ Ignore ]   [ # 9 ]  
Grad Student
Avatar
Rank
Total Posts:  31
Joined  08-20-2009

I didn’t resolve the issue…

I just retrieved what I needed using MySQLi, it’s the only solution I found. Until I find a real solution I’m not using CodeIgniter for client projects anymore.

 Signature 

GitHub: http://github.com/AzizLight/
Forrst: http://forr.st/-AzizLight
StackOverflow: http://stackoverflow.com/users/123016/aziz-light

Profile
 
 
Posted: 05 April 2010 07:33 AM   [ Ignore ]   [ # 10 ]  
Summer Student
Total Posts:  3
Joined  04-05-2010

As you mentioned, when I use utf8 decode it throws up garbage. I used this to decode the data.

For others, I have stored hindi unicode data using java application in mysql db (4.x). See example patrika. Does mysql version have anything to do with this issue? Any help in this regard is appreciated. I use codeigniter 1.7.2 and notepad++.

AzizLight - 05 April 2010 11:12 AM

I didn’t resolve the issue…

I just retrieved what I needed using MySQLi, it’s the only solution I found. Until I find a real solution I’m not using CodeIgniter for client projects anymore.

Profile
 
 
Posted: 05 April 2010 10:26 AM   [ Ignore ]   [ # 11 ]  
Summer Student
Total Posts:  3
Joined  04-05-2010

Got it fixed. 3 potential causes:

1) Old version (4.x) of mysql db: Upgraded to the latest version (5.x) of mysql db.

2) The data was being inserted by a java application using jdbc/odbc connector. Changed connector to Connector/J. Reinserted all data.

3) Changed mysql db and table collation from utf8_general_ci to utf8_unicode_ci

No other changes or tweaks.

Vasu Adiga - 05 April 2010 11:33 AM

As you mentioned, when I use utf8 decode it throws up garbage. I used this to decode the data.

For others, I have stored hindi unicode data using java application in mysql db (4.x). See example patrika. Does mysql version have anything to do with this issue? Any help in this regard is appreciated. I use codeigniter 1.7.2 and notepad++.

Profile
 
 
Posted: 05 April 2010 10:08 PM   [ Ignore ]   [ # 12 ]  
Lab Assistant
Avatar
RankRank
Total Posts:  119
Joined  03-24-2007

This is not a bug. I’ve made a lot of UTF-8 apps and I’m not encountering the problem.

You should change the title of your post.

 Signature 

Twitter | Facturation en ligne avec iScriba (one of my CodeIgniter apps)

Profile
 
 
Posted: 07 April 2010 12:21 PM   [ Ignore ]   [ # 13 ]  
Summer Student
Total Posts:  7
Joined  04-07-2010

I think there is no utf-8 problem with codeigniter. This related to mysql charset problem. Second application insert’s data to mysql db other than utf-8 , but your codeigniter db connection is utf-8.

I have created 5 project with codeigniter , I haven’t seen any utf-8 problem.

Profile
 
 
Posted: 08 May 2010 06:47 AM   [ Ignore ]   [ # 14 ]  
Summer Student
Total Posts:  2
Joined  05-08-2010

Hi AzizLight,
I had an issue like yours, I mean everything is set right but the data fetched from db isn’t in appropriate style. As Vasu Adiga declared, it doesn’t work if you have inserted the data by a java application or maybe sth like that. Instead you need to insert the data yourself by ci queries, scaffolding feature or even by phpMyAdmin, then fetch them. It worked for me.
Although I found the solution but I don’t know what the deal with the previous data was. I need to insert all data again.

Profile
 
 
Posted: 08 May 2010 10:00 AM   [ Ignore ]   [ # 15 ]  
Research Assistant
Avatar
RankRankRank
Total Posts:  335
Joined  10-18-2009
AzizLight - 02 March 2010 04:05 PM

Can anyone confirm what I am saying, and help me solve the problem please?

no, sorry.. I always used utf-8 and never had a problem with CI

 Signature 

Wallpapers and Images Site: Desktop Wallpapers

Profile
 
 
   
1 of 2
1