ITek Blog

Development Specialist, Artist and Activist
Personal Website

FIXED: UnicodeEncodeError ‘ascii’ codec can’t encode characters in position ordinal not in range(128)

9 Replies

The Error

This is a solution for UnicodeEncodeError raised when saving a ‘POST’ in Django form where filename is in different encoding then ‘ASCII’. ( ‘ascii’ codec can’t encode characters in position )

‘ascii’ codec can’t encode characters in position

Posting the form raising error in Django:

UnicodeEncodeError at /upload/add/
‘ascii’ codec can’t encode characters in position 52-54: ordinal not in range(128)

UnicodeEncodeError raised when saving a ‘POST’ in Django form where filename is in UTF-8 encoding and converted by Django to ‘ASCII’.

Posting the form raising error in Django:

Trackback

File “/usr/lib/python2.6/site-packages/django/core/handlers/base.py” in get_response
111. response = callback(request, *callback_args, **callback_kwargs)
File “/usr/lib/python2.6/site-packages/django/views/generic/base.py” in view
47. return self.dispatch(request, *args, **kwargs)
File “/usr/lib/python2.6/site-packages/django/views/generic/base.py” in dispatch
68. return handler(request, *args, **kwargs)
File “/usr/lib/python2.6/site-packages/django/views/generic/edit.py” in post
138. return self.form_valid(form)
File “/var/www/websites/mysite/fileupload/views.py” in form_valid
54. obj.save()
File “/var/www/websites/mysite/fileupload/models.py” in save
25. super(Picture, self).save(*args, **kwargs)
File “/usr/lib/python2.6/site-packages/django/db/models/base.py” in save
460. self.save_base(using=using, force_insert=force_insert, force_update=force_update)
File “/usr/lib/python2.6/site-packages/django/db/models/base.py” in save_base
543. for f in meta.local_fields if not isinstance(f, AutoField)]
File “/usr/lib/python2.6/site-packages/django/db/models/fields/files.py” in pre_save
255. file.save(file.name, file, save=False)
File “/usr/lib/python2.6/site-packages/django/db/models/fields/files.py” in save
92. self.name = self.storage.save(name, content)
File “/usr/lib/python2.6/site-packages/django/core/files/storage.py” in save
48. name = self.get_available_name(name)
File “/usr/lib/python2.6/site-packages/django/core/files/storage.py” in get_available_name
74. while self.exists(name):
File “/usr/lib/python2.6/site-packages/django/core/files/storage.py” in exists
218. return os.path.exists(self.path(name))
File “/usr/lib64/python2.6/genericpath.py” in exists
18. st = os.stat(path)

Exception Type: UnicodeEncodeError at /upload/add/
Exception Value: ‘ascii’ codec can’t encode characters in position 52-54: ordinal not in range(128)

The Code

In the model I have:

file = models.ImageField(upload_to=”pictures”)

the error raised on the line in the view.py:

obj.save()

Explanation of the error

Ticket #11030 is talking about this error.

Reverted a change that assumed the file system encoding was utf8, and changed a test to demonstrate how that assumption corrupted uploaded non-ASCII file names on systems that don’t use utf8 as their file system encoding (Windows for one, specifically).

Some servers do not have the necessary files to allow successfully setting the locale to one that supports utf-8 encoding. See here.

The meaning of this is that Django assumes the file system is non UTF-8 and validates that the filename is ASCII. The error raised when the file name is in UTF-8.

Test FileSystem

The problem may be in different places. We need to search for the problem:

Sys.getfilesystemencoding()

from here:

Django is passing a unicode string “path” to the os.stat() function. On many operating systems, Python must actually pass a bytestring, not unicode, to the underlying OS routine that implements “stat”. Therefore Python must convert the unicode string to a bytestring using some encoding. The encoding it uses is whatever is returned by os.getfilesystemencoding

To get the system encoding using the sys.getfilesystemencoding(), enter python at bash and then:

import sys
sys.getfilesystemencoding()

If the output is:

‘UTF-8’

You don’t have problem with your system encoding.

If the problem is here, and you get back ‘ASCII’, change it according to your system.

Locale

Check the locale Object (again in python shell)

import locale
locale.getdefaultlocale()

Again, if the output is: (‘en_US’, ‘UTF8’) – the problem is not here. if it is – change it according to your system.

If the system is ok, then probably the problem is with you web server (Apache, Nginx, etc)

Test Apache

Are you using apache? mod_wsgi? Maybe the problem is here.

LC_ALL & LANG

To see locale on your centos type at bash:

locale

You should see something like this:

# locale
LANG=en_US.UTF-8
LC_CTYPE=”en_US.UTF-8″
LC_NUMERIC=”en_US.UTF-8″
LC_TIME=”en_US.UTF-8″
LC_COLLATE=”en_US.UTF-8″
LC_MONETARY=”en_US.UTF-8″
LC_MESSAGES=”en_US.UTF-8″
LC_PAPER=”en_US.UTF-8″
LC_NAME=”en_US.UTF-8″
LC_ADDRESS=”en_US.UTF-8″
LC_TELEPHONE=”en_US.UTF-8″
LC_MEASUREMENT=”en_US.UTF-8″
LC_IDENTIFICATION=”en_US.UTF-8″
LC_ALL=en_US.UTF-8

if you see:

LC_ALL=

then probable using this python script:

import locale
locale.getlocale()

will return (None, None)

type for all available locale:

locale -a

Unfortunately LANG is often set incorrectly when running under Apache. Documenting the need to set LANG properly under Apache is the subject of #10426,

In [11170] Added note on language variables required for Apache to survive non-ASCII file uploads:

If you get a UnicodeEncodeError
===============================

If you’re taking advantage of the internationalization features of Django (see
:ref:`topics-i18n`) and you intend to allow users to upload files, you must
ensure that the environment used to start Apache is configured to accept
non-ASCII file names. If your environment is not correctly configured, you
will trigger “UnicodeEncodeError“ exceptions when calling functions like
“os.path()“ on filenames that contain non-ASCII characters.

To avoid these problems, the environment used to start Apache should contain
settings analogous to the following::

export LANG=’en_US.UTF-8′
export LC_ALL=’en_US.UTF-8′

Consult the documentation for your operating system for the appropriate syntax
and location to put these configuration items; “/etc/apache2/envvars“ is a
common location on Unix platforms. Once you have added these statements
to your environment, restart Apache.

Check your Django app settings.py to see if I18N is enabled:

USE_I18N = True

Check locale using Django View/Template

Create view:

import locale
import sys
 
def view_locale(request):
    loc_info = "getlocale: " + str(locale.getlocale()) + \
        "<br/>getdefaultlocale(): " + str(locale.getdefaultlocale()) + \
        "<br/>fs_encoding: " + str(sys.getfilesystemencoding()) + \
        "<br/>sys default encoding: " + str(sys.getdefaultencoding())
        "<br/>sys default encoding: " + str(sys.getdefaultencoding())
    return HttpResponse(loc_info)

import locale
import sys

def view_locale(request):
    loc_info = "getlocale: " + str(locale.getlocale()) + \
        "<br/>getdefaultlocale(): " + str(locale.getdefaultlocale()) + \
        "<br/>fs_encoding: " + str(sys.getfilesystemencoding()) + \
        "<br/>sys default encoding: " + str(sys.getdefaultencoding())
        "<br/>sys default encoding: " + str(sys.getdefaultencoding())
    return HttpResponse(loc_info)

and also create a url pattern:

url(r’^locale/$’, ‘myapp.views.view_locale’),

Browse to ‘yoursite.com/locale‘, to check for problems:

getlocale: (None, None)
getdefaultlocale(): (None, None)
fs_encoding: ANSI_X3.4-1968
sys default encoding: ascii

If the view return something like the above, and everything we checked is ok until now, It’s mean that maybe the problem is with your web-server (apache, nginx, etc.):

Solution for Apache encoding problem

Set LANG & LC_ALL

non-ascii filenames with the Django storage system with the default apache settings on most systems will trigger UnicodeEncodeError exceptions when calling functions like os.path(). To avoid these issues, ensure that the following lines are included in your apache envvars file (typically found in /etc/apache2/envvars).

export LANG='en_US.UTF-8'
export LC_ALL='en_US.UTF-8'

To see your active envvars use:

printenv

This error likely wont rear its head during development on the test server as, when run from the command line, the ./manage.py script inherits the users language and locale settings.

Consult the documentation for your operating system for the appropriate syntax and location to put these configuration items; /etc/apache2/envvars is a common location on Unix platforms (Not all Apache distributions have a envvars file). Once you have added these statements to your environment, restart Apache.

if the ‘envvars’ file doesn’t exist. In that case you will need
to modify the environment of the startup script which is used to
startup Apache in the first place. I believe that for most Linux
systems this can be done by modifying:

/etc/sysconfig/httpd

/etc/init.d/httpd

/etc/init.d/apache

depending on the distro.

If everything fine, when you’ll add those lines and restart the httpd (apache) server you should get at the /locale view:

getlocale: (‘en_US’, ‘UTF8’)
getdefaultlocale(): (‘en_US’, ‘UTF8’)
fs_encoding: UTF-8
sys default encoding: utf-8

And your app should work now!

Some had also add the lines to ~/.bashrc or to the .htaccess, but I haven’t tested it.

Do not use the .wsgi script!

I tried to add the LANG and LC_ALL to the .wsgi instead (from some instructions) and failed becuase:

Some are adding the LANG & LC_ALL to the .WSGI loading script:

os.environ['LANG']='en_US.UTF-8'
os.environ['LC_ALL']='en_US.UTF-8'

Using the view we created earlier (if you had problem) you can see now that

getdefaultlocale(): (‘en_US’, ‘UTF8’).

But the others function may still return ASCII values:

getlocale: (None, None)
getdefaultlocale(): (‘en_US’, ‘UTF8’)
fs_encoding: ANSI_X3.4-1968
sys default encoding: ascii

adding:

import sys
reload(sys)
sys.setdefaultencoding('utf-8')

And now you can see that the sys default encoding is: UTF-8.
reload is important This is python 2.x problem, not the django.

BUT

as you can see:

1	(sys.getfilesystemencoding()

(sys.getfilesystemencoding()

return:

ANSI_X3.4-1968

and that is the problem we have. Django doesn’t recognize the filesystem as UTF-8

SetEnv directive does not modify process environment variables, except
for CGI scripts spawned from Apache. In Apache/mod_wsgi they only
affect the per request WSGI environment.

Setting them in the WSGI script file also will have no affect, as
Python works out the default encoding when the interpreter is first
initialised, which means that doing it in the script file is too late.

What this is mean that you should insert export the LANG and LC_ALL earlier.

Test AddDefaultCharset (httpd.conf)

Check the httpd.conf for:

AddDefaultCharset

The problem may be there.

You can try to set it to

AddDefaultCharset UTF-8

or to off:

AddDefaultCharset Off

Test Nginx

If you have nginx installed, Add

1	charset utf-8;

line in

http

section in main Nginx config file (

1	/etc/nginx/nginx.conf

) or in section

server

in your virtual server config file.

Read more about Nginx HttpCharsetModule.

Django Admin

So, you’ve fixed the app.. now you can upload non ascii files but the Django admin return UnicodeEncodeError when you try to view the row in the admin panel?

Just fix your model __UNICODE__ function to return unicode (u”):

1 2	def __unicode__(self): return u'%s' % (self.file)

def __unicode__(self):
   return u'%s' % (self.file)

Django Centos 6 Beginners Installation Guide

Leave a reply

NOTE: This article was rewritten for CentOS 6.4. please read the updated post.

Django Centos 6

So you want your first django centos based web site? This is quite easy to install and configure. I’ll cover how to install Python/Django on your centos.

Prerequisite

Centos 6.x
Apache (httpd)

What is Python?

“Python is a general-purpose, high-level programming language whose design philosophy emphasizes code readability. Its syntax is said to be clear and expressive.” from Wikipedia.

Visit http://www.python.org/

What is Django?

“Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design.” from https://www.djangoproject.com/

Installing Python

Prerequisites

you may need the EPEL repositories for Centos.

cd /opt/
wget http://mirrors.nl.eu.kernel.org/fedora-epel/6/i386/epel-release-6-8.noarch.rpm
rpm -Uvh epel-release-6-8.noarch.rpm
rm epel-release-6-8.noarch.rpm -f

cd /opt/
wget http://mirrors.nl.eu.kernel.org/fedora-epel/6/i386/epel-release-6-8.noarch.rpm
rpm -Uvh epel-release-6-8.noarch.rpm
rm epel-release-6-8.noarch.rpm -f

Installation Process

Simple as:

1	yum install python

yum install python

test python by typing: “python” and you should see something similar to:

As for this time being of writing (June 18 2012), the version of python@EPEL is 2.6.6

Installing SQL

If you need other SQL Server else then SQLite, you can follow:
MySQL/MongoDB/CouchDB on RHEL/Centos 6.3
If you don’t need SQL and you installed Python 2.5 or later, you can skip this step for now.

Installing Django

1	yum install Django

yum install Django

Testing Django

Test Django by typing python on your command and then:

1 2	import django print django.get_version()

import django
print django.get_version()

As you can see, the RPM version is version 1.3.1, while the current release version is 1.4.1. while you are not too far behind using EPEL, you won’t get to play with the last options. Internationalization: in template code is not available in 1.3.1 for example but only from Django 1.4

Creating Project

From the command, cd into a directory where you’d like to store your app, then run the following command:

1 2	django-admin startproject mysite cd mysite

django-admin startproject mysite
cd mysite

Starting Server

1	python manage.py runserver

python manage.py runserver

Browse to http://127.0.0.1:8000/ with your Web browser. You’ll see a “Welcome to Django” page. It worked!

To change port:

1	python manage.py runserver 8080

python manage.py runserver 8080

If you need to start the server to answer not only locally, use:

1	python manage.py runserver 0.0.0.0:8000

python manage.py runserver 0.0.0.0:8000

Configure more…

Config Database

Edit mysite/settings.py. It’s a normal Python module with module-level variables representing Django settings.

Help here.

Django using Apache

Install mod_python

For this to work, you must have apache installed and configured.

1	yum install mod_python

yum install mod_python

python using mod_python

you need to configure you apache/VirtualHost to:

    AddHandler mod_python .py
    PythonHandler mod_python.publisher | .py
    AddHandler mod_python .psp .psp_
    PythonHandler mod_python.psp | .psp .psp_
    PythonDebug On</pre>
[/code]

 
<h3>Testing mod_python</h3>
Create a '<em>test.py'</em> file in your apache server. put inside:
[code lang="py"]<% req.write("Hello World!") %>[/code]

and browse to www.your.server/test.py and you should see the "Hello World!" there.
<h3>Django using mod_python</h3>
Edit your <tt>httpd.conf</tt> file:

[code lang="apache"]
<pre><Location "/mysite/">
    SetHandler python-program
    PythonHandler django.core.handlers.modpython
    SetEnv DJANGO_SETTINGS_MODULE mysite.settings
    PythonOption django.root /mysite
    PythonDebug On
</Location>

Install mod_wsgi

Deploying Django with Apache and mod_wsgi is the recommended way to get Django into production.

1	yum install mod_wsgi

yum install mod_wsgi

to use mod_wsgi, create an apache folder inside your project and create a django.wsgi file:

import os, sys sys.path.append('/var/www/django') sys.path.append('/var/www/django/mysite') os.environ['DJANGO_SETTINGS_MODULE'] = 'mysite.settings'


import django.core.handlers.wsgi

application = django.core.handlers.wsgi.WSGIHandler()

and configure your apache with:

<VirtualHost *:80>


ServerName www.example.com

ServerAlias www.example.com

WSGIScriptAlias / /var/www/django/mysite/apache/django.wsgi
# Alias /robots.txt /var/www/django/mysite/static/robots.txt

# Alias /favicon.ico /var/www/django/mysite/static/favicon.ico

Alias /static/admin/ /usr/lib/python2.6/site-packages/django/contrib/admin/media/

Alias /static/ /var/www/django/mysite/static/

Alias /media/ /var/www/django/mysite/media/
<Directory /var/www/django/mysite>

Order allow,deny

Allow from all

</Directory>
<Directory /var/www/django/mysite/media>

Order deny,allow

Allow from all

</Directory>

<Directory /var/www/django/mysite/static> Order deny,allow Allow from all </Directory> </VirtualHost>

That’s it!

You’ve Python / Django installed on your Centos 6.3 and a new project template waiting for you to work on.

Continue reading…

Django URL dispatcher routing beginners tutorial
Django documentation
WSGI Reloading Source Code – If you find the server reload is frasturating.

* this article is an edited version of an older article.

+37

Development Specialist, Artist and Activist
Personal Website

Django: Preserve table data when changing model (adding fields)

Leave a reply

Overview

Django, out-of-the-box doesn’t knows how to add rows to your db when the model changed, You have 2 options:

manually add the field to the db. (I won’t demonstrate this)
export data > reset your db > import data.

+27

Development Specialist, Artist and Activist
Personal Website

Django: return rows from model as JSON

1 Reply

Overview

I’ll demostrate in this article how to create a simple view to return JSON formatted rows from your model.

+28

Development Specialist, Artist and Activist
Personal Website

Django URL dispatcher routing beginners tutorial

Leave a reply

How to use Django URL dispatcher

Django routing module called URL dispatcher

url.py

Normally, When a user requests a page from your Django-powered site Django loads the URLconf module from a file called urls.py at the root of your project, Ordinarily, this is the value of the ROOT_URLCONF setting, The URL dispatcher can be overridden when the incoming HttpRequest object has an attribute called urlconf (set by middleware request processing), its value will be used in place.

First line in your urls.py should be:

1	from django.conf.urls import patterns

urlpatterns

Django loads that file and looks for the variable urlpatterns. This should be a Python list, in the format returned by the function django.conf.urls.patterns().

1
2
3
4
5
6
7
8
9
10
11
12
13
14

from django.conf.urls.defaults import patterns

urlpatterns = patterns('',
# ex: /
(r'^$', 'myapp.views.home', {}, 'index'),
# ex: /about
(r'^about', 'myapp.views.about', {}, 'about'),
....
)

# To add more urlpatterns in the same urls.py you can use:
urlpatterns += patterns('',
....
)

URL Pattern Arguments

First Argument is URL pattern.
Second is the View to send back.
Third is Dictionary to send to the view.
Last argument is Name. Naming enables calling {% url name %} from templates.

Notes

Django runs through each line, in order, and stops at the first one that matches the request.
The ‘r’ in front of each regular expression string is optional but recommended. It tells Python that a string is “raw” – that nothing in the string should be escaped. See Dive Into Python’s explanation.

url() function

You can use the url( convenience wrapper.

The url() function is passed five arguments, two required: regex and view, and three optional: kwargs, name, and prefix.

url(regex, view, kwargs=None, name=None, prefix=”)

You can use the url() function, instead of a tuple, as an argument topatterns(). This is convenient if you want to supply a name but not extra_context you’d still have to include an empty optional extra arguments dictionary in the tuple version

Remember you need to include the url, from django.conf.urls.defaults.

1	from django.conf.urls.defaults import patterns, url

These are the same:

1 2	(r'^$', 'views.home', {}, 'index'), url(r'^$', views.index, name='index'),

Import views to shorten code

You can use either way, as you feel convenient for you. It’s the same code above, but using Import to shorten code:

Method 1: Import Views Class

1
2
3
4
5
6
7

from django.conf.urls.defaults import patterns
from myapp import views

urlpatterns = patterns('',
(r'^$', 'views.home', {}, 'index'),
.....
)

Method 2: Import Views (or *)

1
2
3
4
5
6

from django.conf.urls.defaults import patterns
from myapp.views import home

urlpatterns = patterns('',
(r'^$', 'home', {}, 'index'),
.....

Method 3: Use Default Pattern

1
2
3
4

urlpatterns = patterns('myapp.views',
(r'^$', 'home', {}, 'index'),
.....
)

Include

You can include from another file, Django app or object. remember to import the include() function.

1
2
3
4
5
6

from django.conf.urls.defaults import patterns, url, include
urlpatterns = patterns('',
# ex: /help
url(r'^help/', include('help.urls')),
.....
)

This code will search for url.py file inside help app in your django project. construct the file the same as the url.py file.

Include in the same file

1
2
3
4
5
6
7
8
9
10

from django.conf.urls.defaults import patterns, url, include

extra_patterns = patterns('',
.......
)

urlpatterns = patterns('',
(r'^extra/', include(extra_patterns)),
.....
)

These are the URL dispatcher basics. Maybe I’ll be post later another post on more advanced aspects of the URL dispatcher include URL grouping, passing extra parameters to view and more…

CYA,

And good luck with Django – It worth it!

+23

Development Specialist, Artist and Activist
Personal Website

Django Tutorial: Adding Foreign User ID to Model

Leave a reply

Adding Foreign Key

First, declare the authentation User object:

+23

Development Specialist, Artist and Activist
Personal Website

HTML Secrets – Web Application Architecture Design Tips

Leave a reply

Whatever site or web application you’re working on, you probrably going to use HTML.

These are the tips I’ve learned over time:

+28

Development Specialist, Artist and Activist
Personal Website

Installing Python / Django on Centos 6.3 is Easy!

4 Replies

NOTE: This article was rewritten (for CentOS 6.4). please read the updated post.

What is Python?

“Python is a general-purpose, high-level programming language whose design philosophy emphasizes code readability. Its syntax is said to be clear and expressive.” from Wikipedia.

Visit http://www.python.org/

What is Django?

“Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design.” from https://www.djangoproject.com/

+39

Development Specialist, Artist and Activist
Personal Website