Category Archives: Development

how to speed website

How to speed up website load time

How to speed up website load time

How to speed up website load time is important question for those that performance matters to them.

Nobody Likes a Slow Website!

 

Overview

How to speed up website?

Website optimization is a required step for production, and when is automated when possible correctly it can help you deliver your website as fastest as possible without compromising on development comfortability and code readability.

 

Test your website load time speed

How to speed up website question could be answered for your case better if you’ll use a website speed test tools. There several good websites load time speed tests available online for free. check out pingdom website speed test, Google PageSpeed tools. this are very good tools which also gives you important and valuable information and tips to speed up your website.

 

pingdom

You can also check: WebPagetest, GTmetrix or PageScoring Website Speed Test.

Some of them rates your website and/or explain what could be done to make your site faster. you can learn also a lot from loading time graphs.collect information from all the above to optimize your website to maximum.

  • If it takes more then 10 seconds for your website to load. you are doing a very bad job optimizing your website (if at all).
  • If your page load time takes several seconds then you’re ok (but you can do better then that!).
  • if your page load time takes up to two seconds then your site is in very good shape!
  • if your page load time takes less then a second then your site is sooooo fast!! and you are doing a excellent job!

if your score is bad.. continue reading How to speed up website

 

How to speed up website load time

So, how to speed up your website load time?

You can do that using several tweaks.. one at a time…

 

1. learn the waterfall. where are your bottom neck ?

Use the mentioned websites speed test graphs to learn what are the slowest parts of your website and start with them. does the problem is with your webserver or maybe the infrastructure?

How to speed up website ? Does your static files load time is the problem?

And maybe it is the dns? processing request? loading the images? loading the scripts? maybe you have bad requests?

This extreme powerfull tool will help you learn your weaks and strongs.

how to speed website

how to speed website

2. Minimize redirects

If your website on load check if it’s mobile or not, then it redirect to another page when checks the language and then redirects to another page who check the cookies and then redirect… well you’ve got the idea. Bad design pattern!

Minimize redirects!

 

3. Browser caching

Set up your server to return Expire and Cache-Control headers with static files. Setup you expire date to be within week or more is a very good timeframe (depend on how frequently you update your website static files you can setup the expire date as required).

 

4. use CDN (if possible)

Content Delivery Network (CDN) can be an important tool in achieving decent page load and web application speeds. Providers such as AkamaiEdgeCast, Amazon (CloudFront), Rackspace (CloudFiles), Google (PageSpeed) and Microsoft (Azure CDN) are providing the means to distribute your content to locations geographically closer to your customers/users, which improves the responsiveness of the application or website.

 

5. put cache server between your server and your client when possible

Although using a CDN can help with page load speeds, a CDN is not always the best solution in terms of Cost–benefit.

when configuring a cache server before your website server, and no matter if it’s apache, tomcat, gunicorn or any other html generator you can boost up clients page load dramatically.

 

caching the compiled html pages is very important speed tweak. if you have any static and/or low-rate updated pages with no problem if some users will not see the lastest updates of the html when it changes (until cache reloads).. check varnish-cache or squid-cache or nginx (with HttpProxy module) or Apache with mod_proxy.

You should think of your webapp as in several seperate layers. where one layer could answering html requests and other serving your static files. always try to find the suitable tool for this mission in terms of speed (and taste). consider serving your static files (those not on CDN) using the fastest web server for static files you can find.

 

6. Minify

A very important stage is to minify data.

  • Minify CSS files for production! there are many simple gui or command tools to do that..
    if you are using LESS, SASS or other, Never compile on client side! compiler only on server side and setup your compiler to minify results.
  • Minify your JS files. if possible, insert them as scripts into the html, or better, minify and join them inside when processing the html (But don’t forget to use Cache server!).
    If you’re using Coffee-script and template engine it should be easier as you can compile the ,coffee file to minified version and include it within your template.
  • Minify HTML. if you have static html files use command or gui tool. if you have more advanced server like python or ruby – minify them before sumbit.
  • If your template engine support preprocessors. use them to minify HTML (and everything other you can). if not – use a library (if possible).

 

7. GZip

Configure you webserver (nginx, apache,etc)  to GZip response.

Set the Vary header (Vary: Accept-Encoding header) correctly for Internet Explorer.

 

8. Caching data

If you have data you can cache in memory for your application server – DO IT.

Use Redis, memcached or any other tool you find is suitable.

It can be small data (like the number of your followers you’ve just counted from facebook, twitter and youtube JSON calls) or the Entire processed HTML.

Put expire time for each resource on your store as needed.

 

 

How to speed up website

‘ How to speed up website ‘ tutorial ends here.

hope you’ll have fun optimizing your website and now you now few important tips on How to speed up website

Using those steps I’ve managed to reduced my website load time from several seconds to only 1-2 seconds (and sometimes less then second). It’s was worth the effort!!

 

Good day (or night),

Good SEO tips – Search engine optimization for your blog posts

Good SEO tips

Good SEO tips

Good SEO tips for your website / blog / posts /  etc.

Focus Keyword

The first thing you should do BEFORE writing your post is to choose a Focus Keyword.

  • A good Focus Keyword whould be the shortest and most common google search you would like to point to your post.
  • avoid using the same keyword in your website twice.

For example: The focus keyword of this post is “Good SEO“.

You can visit google and start typing “Good SEO” and you’ll see using the auto completion tool that this keyword search is the most common Keyword.

goodseosearch

Note: you can actually use other then the most common keyword (in our example, we could use the “Good SEO Tips” keyword, and It could be a great keyword too. It depends on you to choose what you think would fit best your post and is more related to the content of your post.

I could use also “Best SEO” keyword but as you can see below, is not so common used keyword:

bestseosearch

 

REMEMBER the keyword as you will need to use it in page title/url/content/etc.

 

Writing the article

Now you can start working on your article. Make sure you following:

 

Article Heading

In our example with the keyword “Good SEO“… our heading is:
“Good SEO tips – Search engine optimization for your blog posts

  • Your article heading should contain the keyword / Phrase. better when it at the beginning which is considered to improve rankings.
  • Remember that viewable limit of the heading in search engines is: 70 characters; some words will not be visible to users if it will be longer.
  • If you choose to set a longer heading, use a SEO Title

 

SEO Title

If you used title longer then 70 characters in your CMS (Like WordPress Page Title) use the HTML <title></title> to set a maximum of 70 chars title to your page.

this page title is:
<title>Good SEO tips – Search engine optimization for your blog posts</title>

 

Page titles

  • Your page title (<h1>) should contain the Keyword similar as the article heading.
  • Also make sure you have a sub header <h2> with the keyword.

 

Page URL

  • The keyword should appear in your page URL.

for example: this page URL is:
https://itekblog.com/good-seo-tips

 

Content

  • Make sure the keyword appears in your post content at least several times (at least 6+).
  • The keyword should appear in the first paragraph of the copy.
  • Write no less then 300 words in your copy.
  • run the Flesch Reading Ease test on your article. The copy scores 75.1 which is considered fairly easy to read.
  • add at least 1 outbound link(s). (<a href=””>)

 

Images

  • Add images to your article. at least 1 or 2. insert the keyword to the Alt=””. Adding the keyword as a caption for the images would be great too. look in this example:
Good SEO Tips. This is how it should be done!

Good SEO Tips. This is how it should be done!

 

Meta Description

  • Meta Description is the <meta> tag on your html. the preview on google (and other search engines) will show the meta when available instead of showing the first 156 chars of the article (including titles).
  • Use your keyword inside the meta. Preferably at the beginning. More then once will be better.
<meta name=”description” content=”Good SEO: what is the best description for my post with the keyword inside?.”/>

This article Meta is:

First thing you should do BEFORE writing your post is to think of a good SEO plan. good SEO will help your copy to reach more visitors from search engines!.

Which is exactly 156 chars!.

 

Twitter

Support Twitter too!! Using:

<meta name=”twitter:card” content=”summary”/>
<meta name=”twitter:site” content=”@ITekBlog”/>
<meta name=”twitter:domain” content=”ITek Blog”/>
<meta name=”twitter:creator” content=”@ITekBlog”/>
<meta name=”twitter:image:src” content=”https://itekblog.com/wp-content/uploads/2013/11/googlelogo-300×146.png”/>
<meta name=”twitter:image:src” content=”https://itekblog.com/wp-content/uploads/2013/11/goodseosearch.png”/>
<meta name=”twitter:image:src” content=”https://itekblog.com/wp-content/uploads/2013/11/bestseosearch.png”/>
<meta name=”twitter:description” content=”First thing you should do BEFORE writing your post is to think of a good SEO plan. good SEO will help your copy to reach more visitors from search engines!.”/>
<meta name=”twitter:title” content=”Good SEO tips – Search engine optimization for your blog posts”/>
<meta name=”twitter:url” content=”https://itekblog.com/good-seo-tips/”/>

Google+

Add link to your Google+ page. If authenticated right you will have your image near google searches.

<link rel=”author” href=”https://plus.google.com/109370438350838521961/“/>

 

For WordPress users

If you use WordPress: “WordPress SEO by Yoast” will help you do the job right!

Google Web Designer review – It’s the future?

Google Web Designer review

I’ve met today with the new Google Web Designer Beta – the new tool Google released for building adaptive, universally usable rich HTML based designes including ads.

The main purpose appears to be to provide a way to produce Adobe Flash-style animated in HTML5/CSS3 without using Flash.

“We think that Google Web Designer will be the key to making HTML5 accessible to people throughout the industry, getting us closer to the goal of ‘build once, run anywhere,’” Google engineer Sean Kranzberg posted

The Google Web Designer Youtube page is full of examples:

 

Google Web Designer

Starting Google Web Designer for the first time…

gwd-icon

Google Web Designer is available as a free (as in beer) download for Windows PC and Mac OS X, so at this stage, it won’t run on Linux, or Chromebooks.

Google Web Designer loading

Google Web Designer loading

Google Web Designer

Google Web Designer main screen

Product review

You have several presets to start from. You can start by creating a new Ad banner, or HTML / CSS / JS / XML as you can see:

gwb-presets

 

The UI is slick, and yet powerfull. You have great tools to work with creating your HTML file.

gwb-tools3D Object tools: You can move/rotate any object in 3D mode.

Text tool: You can write wherever you want in the document and not in HTML oriented direction (Position: absolute;)

Tag tool: Create blocks (DIV) of objects.

Pen tool: You can draw vectors. Just like Flash or Illustrator.

You can also find the Stroke tool and Fill tool as expected. also you can find the normal Zoom tool, Selection tool and Hand tool

One idea. Any screen.

Google Web Designer helps you create a responsive HTML. everything you create is accessible on any screen – desktop, tablet or mobile – without compatibility issues.

Amplify with code

If you’re feeling more hands-on, all the code behind your designs is hand-editable.

Two animation modes

In Quick mode, build your animations scene by scene and google web designer take care of the frames in between. In Advanced mode, animate individual elements using layers, easily changing the location of elements within the stack.

Full 3D authoring environment

Create and manipulate 3D content utilizing an array of tools and the power of CSS3. Rotate objects and even 2D designs along any axis, visualizing 3D transformations and translations as you author.

 

 

Summary

Although other Hacker News users reckon the code that GWD generates compares unfavorably to Microsoft FrontPage 2000. I found the code to be somehow fine, if inspected and modified manually. This tool is great for creating HTML parts (widgets) that are cross browser responsible (like ads, menu, carousel, etc.).

I still gonna play with Google Web Designer for sometime, but I must admit I was never the ‘HTML generator’ guy.. I don’t think that tools like that will take the job for the great HTML/CSS designers out there… How write their code using decent IDE.

It’s worth noting that projects like Adobe Muse/Dreamweaver, Reflow and others include most of Web Designer’s features, too. But by making Web Designer available for free, Google is putting quite a bit of pressure on the competitors.

 

That’s it. What do you think about Google Web Designer?

Django CentOS 6.4 installation instructions for noobs.

Django Centos 6.4 installation instructions for noobs.

So you want to build your first django centos 6.4 based web site? This is quite easy to install and configure. I’ll cover how to install Python/Django on your centos.

Django Centos

Prerequisite

  • CentOS 6.4 (For the matter of fact, this tutuorial is good for all the CentOS 6.x series)
  • Apache (httpd)

 

What is Python?

Python is a general-purpose, high-level programming language whose design philosophy emphasizes code readability. Its syntax is said to be clear and expressive.” from Wikipedia.

Visit http://www.python.org/

 

What is Django?

“Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design.” from https://www.djangoproject.com/

Installing Python

Prerequisites

you may need the EPEL repositories for Centos.

1
2
3
4
cd /opt/
wget http://mirrors.nl.eu.kernel.org/fedora-epel/6/i386/epel-release-6-8.noarch.rpm
rpm -Uvh epel-release-6-8.noarch.rpm
rm epel-release-6-8.noarch.rpm -f
cd /opt/
wget http://mirrors.nl.eu.kernel.org/fedora-epel/6/i386/epel-release-6-8.noarch.rpm
rpm -Uvh epel-release-6-8.noarch.rpm
rm epel-release-6-8.noarch.rpm -f

 

Installation Process

Simple as:

1
yum install python
yum install python

test python by typing: “python” and you should see something similar to: (CTRL+D to exit)

At the time of writing (September 30 2013), the version of python@EPEL is 2.6.6

Django works with any Python version from 2.6.5 to 2.7. It also features experimental support for versions 3.2 and 3.3. All these versions of Python include a lightweight database called SQLite so you won’t need to set up a database just yet unless you need other SQL

 

Installing SQL

Django works with several DB engines. for simplicity, Install SQLite:

yum install sqlite

If you need other SQL Server else then SQLite,  you can follow:
MySQL/MongoDB/CouchDB on RHEL/Centos 6.3
If you don’t need SQL and you installed Python 2.5 or later, you can skip this step for now.

 

 

 

 

Installing Django

You can install using EPEL repositories any yum (like in the old article I wrote), but I recommend to use the easy_install method from the python-setuptools package, which I’ll show next:

1
2
yum install python-setuptools
easy_install django
yum install python-setuptools
easy_install django

 

Testing Django

Test Django by typing python on your command and then:

1
2
import django
print django.get_version()
import django
print django.get_version()

python-django-1.5.4

As you can see, the RPM version is version 1.5.4, while the current release in the EPEL version is 1.3.1. Internationalization: in template code is not available in 1.3.1 for example but only from Django 1.4

 

 

Creating Project

From the command, cd into a directory where you’d like to store your app, then run the following command:

1
2
django-admin.py startproject mysite
cd mysite
django-admin.py startproject mysite
cd mysite

 Note: if you are installing Django using EPEL repos, the command will be django-admin and not django-admin.py.

Starting Server

1
python manage.py runserver
python manage.py runserver

You’ve started the Django development server, a lightweight Web server – easier to startwithout having to deal with configuring a production server — such as Apache — until you’re ready for production.

Browse to http://127.0.0.1:8000/ with your Web browser. You’ll see a “Welcome to Django” page. It worked!

To change port:

1
python manage.py runserver 8080
python manage.py runserver 8080

If you need to start the server to answer not only locally, use:

1
python manage.py runserver 0.0.0.0:8000
python manage.py runserver 0.0.0.0:8000

 

Remember: Apache loads Django environment when starting and keep running it even when source is changed! I suggest you to use Django ‘runserver’ (which automatically restarts on source code changes) in development sessions, unless you need some Apache-specific features.

 

Configure more…

Config Database

Edit mysite/settings.py. It’s a normal Python module with module-level variables representing Django settings.

Help here.

 

Django using Apache

To run your Django application inside apache – use either mod_python or mod_wsgi, Support for mod_python will be deprecated in a future release of Django. If you are configuring a new deployment, you are strongly encouraged to consider using mod_wsgi or any of the other supported backends.

 

OPTION A: Install mod_python (NOT recommended)

For this to work, you must have apache installed and configured.

1
yum install mod_python
yum install mod_python

python using mod_python

you need to configure you apache/VirtualHost to:

    AddHandler mod_python .py
    PythonHandler mod_python.publisher | .py
    AddHandler mod_python .psp .psp_
    PythonHandler mod_python.psp | .psp .psp_
    PythonDebug On</pre>
[/code]

 
<h3>Testing mod_python</h3>
Create a '<em>test.py'</em> file in your apache server. put inside:
[code lang="py"]<% req.write("Hello World!") %>[/code]

and browse to www.your.server/test.py and you should see the "Hello World!" there.
<h3>Django using mod_python</h3>
Edit your <tt>httpd.conf</tt> file:

[code lang="apache"]
<pre><Location "/mysite/">
    SetHandler python-program
    PythonHandler django.core.handlers.modpython
    SetEnv DJANGO_SETTINGS_MODULE mysite.settings
    PythonOption django.root /mysite
    PythonDebug On
</Location>

read more…

 

 

Option B: Install mod_wsgi (RECOMMENDED)

Deploying Django with Apache and mod_wsgi is the recommended way to get Django into production.

1
yum install mod_wsgi
yum install mod_wsgi

 

to use mod_wsgi, create an apache folder inside your project and create a django.wsgi file:

import os, sys
sys.path.append('/var/www/django')
sys.path.append('/var/www/django/mysite')
os.environ['DJANGO_SETTINGS_MODULE'] = 'mysite.settings'

import django.core.handlers.wsgi

application = django.core.handlers.wsgi.WSGIHandler()

 

and configure your apache with:

<VirtualHost *:80>

ServerName www.example.com
ServerAlias www.example.com
WSGIScriptAlias / /var/www/django/mysite/apache/django.wsgi

# Alias /robots.txt /var/www/django/mysite/static/robots.txt
# Alias /favicon.ico /var/www/django/mysite/static/favicon.ico
Alias /static/admin/ /usr/lib/python2.6/site-packages/django/contrib/admin/media/
Alias /static/ /var/www/django/mysite/static/
Alias /media/ /var/www/django/mysite/media/

<Directory /var/www/django/mysite>
Order allow,deny
Allow from all
</Directory>

<Directory /var/www/django/mysite/media>
Order deny,allow
Allow from all
</Directory>

<Directory /var/www/django/mysite/static>
Order deny,allow
Allow from all
</Directory>
</VirtualHost>

 

read more on: multiple django sites with apache & mod_wsgi

 

 

That’s it!

 

You’ve Python / Django installed on your Centos 6.4 and a new project template waiting for you to work on.

 

Continue reading…

* this article is an edited version of an older article.

Jinja2 vs Tenjin: Why I moved (back) from Jinja2 to Tenjin

Jinja2 vs Tenjin: The Story

I started a new project: for learning purposes: to learn python gevent api and to implement a simple server – with Authentication & Websockets.

First thing after handling routings and starting server with the famous ‘hello, world!’ message was to find a suitable Framework, if any.

After a search I found Bottle which I think is amazing microframework. Flask was also great, but I loved the idea of a microframework without template engine so I could choose my own.

As I worked with many Template engines in many different languages and environments (.NET/Ruby/Node.js/Python/PHP/and more). I know I can handle any syntax, so the first thing I’ve looked for is the fastest Python template engine.

I’ve choosed to try Tenjin, It worked great, looked bad. the syntax was messy, there was that .cache file near my.pyhtml files… But it worked great and FAST!

I’ve got to admit that SpitFire catched me, and I’d choose it for several reasons:

  • Really FAST!!
  • Trivial Cheetah templates will probably compile in Spitfire

But as I ‘m not using Cheetah and the Languages support Tenjin offers – for me It is Winning case for Tenjin.

 

Jinja2 vs Tenjin

I decided to try more template engine, slower – but friendly : Jinja2.

Tenjin syntax , which was somewhat ugly, and Jinja2 was just beautirful, and reminds me Django’s and Twig which I like… So, I;ve decided to try Jinja2 and to feel the difference.

Found no problems there, Jinja2 was working great, and the .html files are looking great in compare to Tenjin,.. more readable, no .cache file – and .html instead of .pyhtml. I’ve choosed to stick with Jinja2… for now…..

…meanwhile.. on another project I’m working on in Django, I’ve tried to implement Googles AngularJS and found my self in a very bad situation. the AngluarJS Template Engine was similar to Django’s one! There are several solutions and fixes for this, some are really messy but I fixed it… and I’ve learned a very important lesson… Templates engines used today in both sides: both client & server, and this should be taken into consideration when choosing template engines.

The true title of this article should be: Why I moved from Tenjin to Jinja2 and why I moved from Jinja2 to Tenjin again..

As Jinja is very similar to Django (and angular) I found my self thinking about Tenjin again…

  • It’s faster then Jinja.. it’s faster then most teamplte engines!.. ,
  • it support many languages which make the project more trasferable and convertable between servers (Supports PHP, Perl, Ruby & Python).
  • It supports JS templating – wchich I haven’t tried yet,
  • It’s different then the mainstream template engines with {{ }} and {*  *} so mixing Template engines with client side template engines should be safer.

  

Summary

I know it’s not a classical -vs- article which compares between features, and there is a lot to cover in the topic of jinja2 vs tenjin and the feautres of both,.. I choosed to wrote instead on my view and decisions… what made me go back to Tenjin and leave Jinja 2 behind.. This is my story of Jinja2 vs Tenjin… In my case, Tenjin won, but I MUST say that Jinja2 is GREAT!, also: SpitFire, Mako Template, Cheetah, Genshi & others… as always: each tool for different task… It’s your call…

 

 EDIT:

I learned about Jinja2 inheritance feature which is missing in Tenjin. If you need Template Inheritance go for Tenjin (or other).

 

What do you think? Which Python Template engine do you use? and why?

FIXED: UnicodeEncodeError ‘ascii’ codec can’t encode characters in position ordinal not in range(128)

The Error

This is a solution for UnicodeEncodeError raised when saving a ‘POST’ in Django form where filename is in different encoding then ‘ASCII’. ( ‘ascii’ codec can’t encode characters in position )

'ascii' codec can't encode characters in position

‘ascii’ codec can’t encode characters in position

Posting the form raising error in Django:

UnicodeEncodeError at /upload/add/
‘ascii’ codec can’t encode characters in position 52-54: ordinal not in range(128)

UnicodeEncodeError raised when saving a ‘POST’ in Django form where filename is in UTF-8 encoding and converted by Django to ‘ASCII’.

Posting the form raising error in Django:

Trackback

File “/usr/lib/python2.6/site-packages/django/core/handlers/base.py” in get_response
111. response = callback(request, *callback_args, **callback_kwargs)
File “/usr/lib/python2.6/site-packages/django/views/generic/base.py” in view
47. return self.dispatch(request, *args, **kwargs)
File “/usr/lib/python2.6/site-packages/django/views/generic/base.py” in dispatch
68. return handler(request, *args, **kwargs)
File “/usr/lib/python2.6/site-packages/django/views/generic/edit.py” in post
138. return self.form_valid(form)
File “/var/www/websites/mysite/fileupload/views.py” in form_valid
54. obj.save()
File “/var/www/websites/mysite/fileupload/models.py” in save
25. super(Picture, self).save(*args, **kwargs)
File “/usr/lib/python2.6/site-packages/django/db/models/base.py” in save
460. self.save_base(using=using, force_insert=force_insert, force_update=force_update)
File “/usr/lib/python2.6/site-packages/django/db/models/base.py” in save_base
543. for f in meta.local_fields if not isinstance(f, AutoField)]
File “/usr/lib/python2.6/site-packages/django/db/models/fields/files.py” in pre_save
255. file.save(file.name, file, save=False)
File “/usr/lib/python2.6/site-packages/django/db/models/fields/files.py” in save
92. self.name = self.storage.save(name, content)
File “/usr/lib/python2.6/site-packages/django/core/files/storage.py” in save
48. name = self.get_available_name(name)
File “/usr/lib/python2.6/site-packages/django/core/files/storage.py” in get_available_name
74. while self.exists(name):
File “/usr/lib/python2.6/site-packages/django/core/files/storage.py” in exists
218. return os.path.exists(self.path(name))
File “/usr/lib64/python2.6/genericpath.py” in exists
18. st = os.stat(path)

Exception Type: UnicodeEncodeError at /upload/add/
Exception Value: ‘ascii’ codec can’t encode characters in position 52-54: ordinal not in range(128)

 

The Code

In the model I have:

file = models.ImageField(upload_to=”pictures”)

 

the error raised on the line in the view.py:

obj.save()

 

 

Explanation of the error

Ticket #11030 is talking about this error.

Reverted a change that assumed the file system encoding was utf8, and changed a test to demonstrate how that assumption corrupted uploaded non-ASCII file names on systems that don’t use utf8 as their file system encoding (Windows for one, specifically).

Some servers do not have the necessary files to allow successfully setting the locale to one that supports utf-8 encoding. See here.

The meaning of this is that Django assumes the file system is non UTF-8 and validates that the filename is ASCII. The error raised when the file name is in UTF-8.

 

 

Test FileSystem

The problem may be in different places. We need to search for the problem:

Sys.getfilesystemencoding()

from here:

Django is passing a unicode string “path” to the os.stat() function. On many operating systems, Python must actually pass a bytestring, not unicode, to the underlying OS routine that implements “stat”.  Therefore Python must convert the unicode string to a bytestring using some encoding. The encoding it uses is whatever is returned by os.getfilesystemencoding

To get the system encoding using the sys.getfilesystemencoding(), enter python at bash and then:

import sys
sys.getfilesystemencoding()

If the output is:

‘UTF-8’

You don’t have problem with your system encoding.

If the problem is here, and you get back ‘ASCII’, change it according to your system.

 

Locale

Check the locale Object (again in python shell)

import locale
locale.getdefaultlocale()

Again, if the output is: (‘en_US’, ‘UTF8’) – the problem is not here. if it is – change it according to your system.

If the system is ok, then probably the problem is with you web server (Apache, Nginx, etc)

 

Test Apache

Are you using apache? mod_wsgi? Maybe the problem is here.

 

LC_ALL & LANG

To see locale on your centos type at bash:

locale

You should see something like this:

# locale
LANG=en_US.UTF-8
LC_CTYPE=”en_US.UTF-8″
LC_NUMERIC=”en_US.UTF-8″
LC_TIME=”en_US.UTF-8″
LC_COLLATE=”en_US.UTF-8″
LC_MONETARY=”en_US.UTF-8″
LC_MESSAGES=”en_US.UTF-8″
LC_PAPER=”en_US.UTF-8″
LC_NAME=”en_US.UTF-8″
LC_ADDRESS=”en_US.UTF-8″
LC_TELEPHONE=”en_US.UTF-8″
LC_MEASUREMENT=”en_US.UTF-8″
LC_IDENTIFICATION=”en_US.UTF-8″
LC_ALL=en_US.UTF-8

if you see:

LC_ALL=

then probable using this python script:

import locale
locale.getlocale()

will return (None, None)

type for all available locale:

locale -a

 

Unfortunately LANG is often set incorrectly when running under Apache. Documenting the need to set LANG properly under Apache is the subject of #10426,

In [11170] Added note on language variables required for Apache to survive non-ASCII file uploads:

If you get a UnicodeEncodeError
===============================

If you’re taking advantage of the internationalization features of Django (see
:ref:`topics-i18n`) and you intend to allow users to upload files, you must
ensure that the environment used to start Apache is configured to accept
non-ASCII file names. If your environment is not correctly configured, you
will trigger “UnicodeEncodeError“ exceptions when calling functions like
“os.path()“ on filenames that contain non-ASCII characters.

To avoid these problems, the environment used to start Apache should contain
settings analogous to the following::

export LANG=’en_US.UTF-8′
export LC_ALL=’en_US.UTF-8′

Consult the documentation for your operating system for the appropriate syntax
and location to put these configuration items; “/etc/apache2/envvars“ is a
common location on Unix platforms. Once you have added these statements
to your environment, restart Apache.

Check your Django app settings.py to see if  I18N is enabled:

USE_I18N = True

 

Check locale using Django View/Template

Create view:

1
2
3
4
5
6
7
8
9
10
import locale
import sys
 
def view_locale(request):
    loc_info = "getlocale: " + str(locale.getlocale()) + \
        "<br/>getdefaultlocale(): " + str(locale.getdefaultlocale()) + \
        "<br/>fs_encoding: " + str(sys.getfilesystemencoding()) + \
        "<br/>sys default encoding: " + str(sys.getdefaultencoding())
        "<br/>sys default encoding: " + str(sys.getdefaultencoding())
    return HttpResponse(loc_info)
import locale
import sys

def view_locale(request):
    loc_info = "getlocale: " + str(locale.getlocale()) + \
        "<br/>getdefaultlocale(): " + str(locale.getdefaultlocale()) + \
        "<br/>fs_encoding: " + str(sys.getfilesystemencoding()) + \
        "<br/>sys default encoding: " + str(sys.getdefaultencoding())
        "<br/>sys default encoding: " + str(sys.getdefaultencoding())
    return HttpResponse(loc_info)

and also create a url pattern:

    url(r’^locale/$’, ‘myapp.views.view_locale’),

Browse to ‘yoursite.com/locale‘, to check for problems:

getlocale: (None, None)
getdefaultlocale(): (None, None)
fs_encoding: ANSI_X3.4-1968
sys default encoding: ascii

If the view return something like the above, and everything we checked is ok until now, It’s mean that maybe the problem is with your web-server (apache, nginx, etc.):

 

Solution for Apache encoding problem

Set LANG & LC_ALL

non-ascii filenames with the Django storage system with the default apache settings on most systems will trigger UnicodeEncodeError exceptions when calling functions like os.path(). To avoid these issues, ensure that the following lines are included in your apache envvars file (typically found in /etc/apache2/envvars).

export LANG='en_US.UTF-8'
export LC_ALL='en_US.UTF-8'

To see your active envvars use:

printenv

This error likely wont rear its head during development on the test server as, when run from the command line, the ./manage.py script inherits the users language and locale settings.

Consult the documentation for your operating system for the appropriate syntax and location to put these configuration items; /etc/apache2/envvars is a common location on Unix platforms (Not all Apache distributions have a envvars file). Once you have added these statements to your environment, restart Apache.

if the ‘envvars’ file doesn’t exist. In that case you will need
to modify the environment of the startup script which is used to
startup Apache in the first place. I believe that for most Linux
systems this can be done by modifying:

/etc/sysconfig/httpd

or

/etc/init.d/httpd

or

/etc/init.d/apache

depending on the distro.

 

If everything fine, when you’ll add those lines and restart the httpd (apache) server you should get at the /locale view:

getlocale: (‘en_US’, ‘UTF8’)
getdefaultlocale(): (‘en_US’, ‘UTF8’)
fs_encoding: UTF-8
sys default encoding: utf-8

And your app should work now!

Some had also add the lines to ~/.bashrc or to the .htaccess, but I haven’t tested it.

 

Do not use the .wsgi script!

I tried to add the LANG and LC_ALL to the .wsgi instead (from some instructions) and failed becuase:

Some are adding the LANG & LC_ALL to the .WSGI loading script:

os.environ['LANG']='en_US.UTF-8'
os.environ['LC_ALL']='en_US.UTF-8'

Using the view we created earlier (if you had problem) you can see now that

getdefaultlocale(): (‘en_US’, ‘UTF8’).

But the others function may still return ASCII values:

getlocale: (None, None)
getdefaultlocale(): (‘en_US’, ‘UTF8’)
fs_encoding: ANSI_X3.4-1968
sys default encoding: ascii

adding:

import sys
reload(sys)
sys.setdefaultencoding('utf-8')

And now you can see that the sys default encoding is: UTF-8.
reload is important This is python 2.x problem, not the django.

BUT

as you can see:

1
(sys.getfilesystemencoding()
(sys.getfilesystemencoding()

return:

ANSI_X3.4-1968

and that is the problem we have. Django doesn’t recognize the filesystem as UTF-8

SetEnv directive does not modify process environment variables, except
for CGI scripts spawned from Apache. In Apache/mod_wsgi they only
affect the per request WSGI environment.

Setting them in the WSGI script file also will have no affect, as
Python works out the default encoding when the interpreter is first
initialised, which means that doing it in the script file is too late.

What this is mean that you should insert export the LANG and LC_ALL earlier.

 

 

Test AddDefaultCharset (httpd.conf)

Check the httpd.conf for:

AddDefaultCharset

The problem may be there.

You can try to set it to

AddDefaultCharset UTF-8

or to off:

AddDefaultCharset Off

Test Nginx

If you have nginx installed, Add 

1
charset utf-8;

 line in 

1
http

 section in main Nginx config file (

1
/etc/nginx/nginx.conf

) or in section 

1
server

 in your virtual server config file.

Read more about Nginx HttpCharsetModule.

 

Django Admin

So, you’ve fixed the app.. now you can upload non ascii files but the Django admin return UnicodeEncodeError when you try to view the row in the admin panel?

Just fix your model __UNICODE__ function to return unicode (u”):

1
2
def __unicode__(self):
   return u'%s' % (self.file)
def __unicode__(self):
   return u'%s' % (self.file)

 

Read more..

Python vs PyPy installation and comparison (on Centos 6.x)

Python vs PyPy

in this article: Python vs PyPy I’ll explain how-to install both compilers (Python (CPython) and PyPy) on CentOS and the difference between them.

Requirements for examples:

  • Centos 6.x

 

Overview

The Download section of the Python official website (python.org) include list of several Alternative Implementations for python complier. some of them for Linux, some for Windows (.NET) and others.

I am going to review the default Python compiler vs PyPy on my CentOS 6.x machine.

 

python-logo

Python (nicknamed CPython)

Installation

yum install python

From Official website:

Python is a programming language that lets you work more quickly and integrate your systems more effectively. You can learn to use Python and see almost immediate gains in productivity and lower maintenance costs.

Python (nicknamed CPython) is the default Python Compiler you get with you installing Python.

Example

Create a file named ‘example.py‘ and fill it with:

print 'Hello, world!'

and type on bash:

python example.py

Note: you can also insert the following line as the first line in your .py file and then chmod +x this file to execute it as script:

#!/usr/lib/env python
print “Hello, world!”

and then on bash:

chmod +x example.py

and you can run the script (code) using:

./example.py

 

 

pypy-logo

PyPy

fast python implementation with a JIT compiler (generate native code on the fly) written in RPython and currently translated partly to C.
Arguments against PyPy compiled in Python are everywhere, but when C compiler gcc is implemented in C, is there any reason for the Python compiler to not be written in Python?

From Wikipedia:

PyPy is a Python interpreter and just-in-time compiler. PyPy focuses on speed, efficiency and compatibility with the original CPython interpreter.[1]

PyPy started out as a Python interpreter written in the Python language itself. Current PyPy versions are translated from RPython to C code and compiled. The PyPy JIT compiler is capable of turning Python code into machine code at run time.

 

Mission

The mission of PyPy is:

We aim to provide:

  • a common translation and support framework for producing implementations of dynamic languages, emphasizing a clean separation between language specification and implementation aspects. We call this the RPython toolchain.
  • a compliant, flexible and fast implementation of the Python Language which uses the above toolchain to enable new advanced high-level features without having to encode the low-level details. We call this PyPy.

By separating concerns in this way, our implementation of Python – and other dynamic languages – is able to automatically generate a Just-in-Time compiler for any dynamic language. It also allows a mix-and-match approach to implementation decisions, including many that have historically been outside of a user’s control, such as target platform, memory and threading models, garbage collection strategies, and optimizations applied, including whether or not to have a JIT in the first place.

 

How can it possibly beat CPython?

Manual memory management (which is what CPython does with its counting) can be slower than automatic management in some cases.

Limitations in the implementation of the CPython interpreter preclude certain optimisations that PyPy can do (eg. fine grained locks).

The JIT. Being able to on the fly confirm the type of an object can save you the need to do multiple pointer dereferences to finally arrive at the method you want to call.

 

Installation

yum install pypy

 

Using PyPy

To use pypy instead of your default CPython compiler you need to run your .py file with the pypy compiler.

to use pypy compiler you type at bash:

and type on bash:

pypy myapp.py

 

Summary

Python compiler (CPython) is the default and probably the safe choice so most python applications and modules will work; where PyPy can break some.

You should play with them both. Install them both on your CentOS and test / benchmark your applications using both compilers.

PyPy can rebust your web applcations (or any other code).

For example, Django app can be run on top of PyPy, with drawbacks.

 

Read more…

gevent CentOS 6.x Installation Guide

gevent CentOS 6.x Installation Guide

gevent CentOS 6.x Installation Guide

Installation

Requirments

  • yum install python python-devel
  • yum install libevent-devel

 

Install Setuptools

install python-setuptools (for easy_install)

yum install python-setuptools

Install greenlet

Install greenlet:

easy_install greenlet

Install gevent

There are several ways to install gevent:

using easy_install (recommended)

easy as:

easy_install gevent

centos gevent

from source

You need wget to download source:

yum install wget

Now, installing gevent: (Check here for the latest package)

cd /tmp wget http://pypi.python.org/packages/source/g/gevent/gevent-0.13.8.tar.gz tar -xvzf gevent-0.13.8.tar.gz cd gevent-0.13.8 python setup.py install

U may need to add –no-check-certificate to the wget if something is wrong.

from GitHub

You can also install gevent from github using pip.

Test

to test simply run in bash:

python

and then:

import gevent

if no error found, you have installed it correctly python-import-gevent

First App

Example taken from the community documentation:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
from gevent.wsgi import WSGIServer
 
def application(environ, start_response):
    status = '200 OK'
    body = '<p>Hello World</p>'
 
    headers = [
        ('Content-Type', 'text/html')
    ]
 
    start_response(status, headers)
    return [body]
 
WSGIServer(('', 8000), application).serve_forever()
from gevent.wsgi import WSGIServer

def application(environ, start_response):
    status = '200 OK'
    body = '<p>Hello World</p>'

    headers = [
        ('Content-Type', 'text/html')
    ]

    start_response(status, headers)
    return [body]

WSGIServer(('', 8000), application).serve_forever()

copy the example into a file (server.py for ex), and start the server by:

python server.py

and that’s it! browse to the machine on port 8000 using something like this:

http://[ip of the machine]:8000

gevent-working

What’s next..?

 

Production

In production you can host your gevent after several WSGI server racks like: Gunicorn, Circus, Meinheld, uWSGI, etc..

Gunicorn is the most common WSGI server to host gevent applications. one benchmark shows that Circus is faster then Gunicorn. other tested uWSGI vs Gunicorn and found that uWSGI is faster.

recommend you run them behind a reverse proxy such as nginx. Additional advantages of running them behind a reverse proxy include improved handling of slow clients and bad HTTP requests

 

SocketIO

You need socketio? you may want to install it:

easy_install gevent-socketio

 

Read more…

 

need microframework? (flask/bottle)?

You can use a python web framework on top of gevent.

 

PHP GitHub followers count script (v3.0 API)

Php GitHub Followers Count

The following PHP GitHub Followers Count script is using the v3.0 GitHub API.

 

Overview

 I’ll explain how to get your github  followers count using PHP.

 

Get Followers Count

</span>

<?php
function curl_get_contents($url)
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch,CURLOPT_USERAGENT,'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13');
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
$github_data = json_decode(curl_get_contents('https://api.github.com/users/your-user-name'), true);
$github_followers_count = $github_data['followers'];
echo $github_followers_count;
?>

 

 

More information

This article was based on the work of Jimbo and Meenakshi Sundaram R

PHP Twitter Followers Count script (v1.1 API)

Php Twitter Followers Count

The following PHP Twitter Followers Count script is using the new v1.1 Twitter API.

 

Overview

Recently, Twitter have changed their API and v1.0 is obselete. The new API requires OAuth. I’ll explain how to get your twitter followers count using PHP.

 

Create Application

First, you’ll need to get yourself a twitter dev account, if you haven’t done so yet.

Go here and register your app free.

 

TwitterAPIExchange

Download the TwitterAPIExchange library files created by J7mbo into your application.

you can git clone https://github.com/J7mbo/twitter-api-php.

I recommend the git method as you can git pull later to recieve updated.

 

Get Followers Count

Insert your OATH and Consumer Access-Token & Access-Token secret the the following code. Also, change the $getfield screen name to yours.

Now combine everything to make it work

require_once('TwitterAPIExchange.php');

/** Set access tokens here - see: https://dev.twitter.com/apps/ **/
$settings = array(
'oauth_access_token' => "YOUR_OAUTH_ACCESS_TOKEN",
'oauth_access_token_secret' => "YOUR_OAUTH_ACCESS_TOKEN_SECRET",
'consumer_key' => "YOUR_CONSUMER_KEY",
'consumer_secret' => "YOUR_CONSUMER_SECRET"
);

$ta_url = 'https://api.twitter.com/1.1/statuses/user_timeline.json';
$getfield = '?screen_name=replace_me;
$requestMethod = '
GET';
$twitter = new TwitterAPIExchange($settings);
$follow_count=$twitter->setGetfield($getfield)
->buildOauth($ta_url, $requestMethod)
->performRequest();
$data = json_decode($follow_count, true);
$followers_count=$data[0]['
user']['followers_count'];
echo $followers_count;

 

Notes

I’ve placed the entire twitter-api-php folder inside my project. you can extract the files also, but make sure to require_once() from the correct path.

require_once(‘twitter-api-php/TwitterAPIExchange.php’);

if you’ve placed it inside the folder, or

require_once(‘TwitterAPIExchange.php’); if you extracted the files to the same folder as your script.

 

More information

This article was based on the work of Jimbo and Meenakshi Sundaram R

CentOS Tornado server installation tutorial

Centos Tornado Server Installation Tutorial

 

Install using repository

yum install python-tornado

 

First ‘Hello World’ App

I’ve copied the hello world sample from Tornado documentation:

Create a new file called ‘hello.py‘ (or any other name) and fill it with:

1
2
3
4
5
6
7
8
9
10
11
12
13
import tornado.ioloop
import tornado.web
class MainHandler(tornado.web.RequestHandler):
   def get(self):
      self.write("Hello, world")
 
application = tornado.web.Application([
   (r"/", MainHandler),
])
 
if __name__ == "__main__":
   application.listen(8888)
   tornado.ioloop.IOLoop.instance().start()
import tornado.ioloop
import tornado.web
class MainHandler(tornado.web.RequestHandler):
   def get(self):
      self.write("Hello, world")

application = tornado.web.Application([
   (r"/", MainHandler),
])

if __name__ == "__main__":
   application.listen(8888)
   tornado.ioloop.IOLoop.instance().start()

Now start your server with simply typing at bash:

1
python hello.py
python hello.py

Now you can visit ‘127.0.0.1:8888‘ to watch your hello world first torando app.

It should return the ‘Hello, world‘ output into your browser.

 

Nginx

Good implementation of your server would be behind nginx server, where you can run multiple instances of your app on different port and serve them all throw port 80 using nginx load balancer.

http://www.tornadoweb.org/en/latest/overview.html#running-tornado-in-production

 

After you know how to start a torando server, you can jump into the documentation and learn how-to build your first Torando server!