I think ... - python

Install latest Python 3 on Linux CentOS 7

2018-07-07T00:00:00+06:00

Why¶

Not all distro created equal.
Some are created to join the space race, some are to hold unto the leagcy, some are cutting-edge, some are cutting edge. Some are born to boot-up IoT devices some are to push out heavy graphics.

That’s the fun (albeit, power) of Linux.

CentOS 7 is a powerful and stable distro that runs on thousands (probably, millions) production-grade servers.
In the matter of stability, it’s a beast. However, it doesn’t ship with Python 3, by default. You can install it via EPEL repository, or the below simple steps.

Also, take a note. The Python 2 comes with the system, which is probably 2.7.5, do NOT mess with it. Many system components rely on that specific version. If you need the latest versions of 2, use virtualenv or pipenv.

Prepare your system¶

Start with installing pre-requisite utilities for compilation and development support.

$ sudo yum update && sudo yum groupinstall -y "development tools"  
$ sudo yum install -y zlib-devel bzip2-devel openssl-devel ncurses-devel \
       sqlite-devel readline-devel tk-devel gdbm-devel \
       db4-devel libpcap-devel xz-devel expat-devel

Download latest Python source code from Python.org¶

$ wget https://www.python.org/ftp/python/3.6.6/Python-3.6.6.tar.xz
$ tar xf Python-3.6.6.tar.xz
$ cd Python-3.6.6

Enable performance optimizations (optional, but highly recommended)¶

$ ./configure --prefix=/usr/local --enable-shared LDFLAGS="-Wl,-rpath /usr/local/lib"  
$ ./configure --enable-optimizations

Build and install¶

$ make
$ sudo make altinstall

Now, Python 3.6.6 is ready to be used in your system; located in /usr/local/bin/python3.6

$ which python3.6
/usr/local/bin/python3.6

$ python3.6
Python 3.6.6 (default, Jul 10 2018, 14:04:26)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

For convenience, you can create a symbolic-link with a shorter name. If you had system-installed Python3 (unlikely), don’t do this, as some system-components may depend on that specific older version of Python 3.

$ sudo ln -s /usr/local/bin/python3.6 /usr/local/bin/python3
$ python3
Python 3.6.6 (default, Jul 10 2018, 14:04:26)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

Install wheel and pip¶

You don’t need to, because Python 3.6.6 includes these necessary tools included.

$ pip3.6 -V
pip 10.0.1 from /usr/local/lib/python3.6/site-packages/pip (python 3.6)
$ wheel version
wheel 0.29.0

Pelican Static sites - SEO Optimization

2017-01-07T00:00:00+06:00

Writing is a hard job, especially it is your hobby beside of your day job. But what’s the benefit if nobody reads it just because they couldn’t find it.

Usually Pelican static-site generator is not very concerned about SEO of the generated site mainly because it’s not that focused on commercial usage. That’s what I felt. Related themes and their templates also don’t take it much seriously. But you shouldn’t loose SEO just because you migrated from Wordpress or whatever you were using previously, right?

Often times theme-authors focus more on look-n-feel of the theme, but not so much on the SEO concerns.

That’s this blog about; let’s fix that.

Note 1: About other static-site generators¶

Although the following discussions & codes mostly are specific to Pelican templates which uses jinja2 templating language, the concepts and concerns here are applicable to most static-site generators and their themes.

Note 2: I’m, in no way, an SEO expert¶

This writeup is just a collection of my findings while correcting my blog’s SEO course; fixing the stupid mistakes. Also, this isn’t a commercial site. There are lot more and in-depth aspects of SEO optimization other than the following, that can be very important for commercial projects.

Getting started¶

Lookup for missing pieces¶

Make sure all the linked resources(links, images, CSS & JS files) that you’ve used in your pages are valid. For that check browser’s console in Developers tools for any errors e.g. unavailable urls, faulty html/css etc.

Know the critical spots¶

pelicanconf.py - It’s usually in your root folder of the site.
base.html - It’s in the templates folder of the theme folder that you are using. Changes here will impact all HTML pages generated by Pelican.
article.html - In same folder as base.html. Changes here will impact only articles’ pages.

Avoiding Duplication¶

Avoid your source getting indexed by Google¶

If the source of your blog is not sourced-open, meaning the content is not in a open-sourced repo, this isn’t your concern. But if it is, it should be a concern.

The reason is that Github.com(or where your repo is hosted on) is stronger domain than yours, so Google “sees” that your site-contents (even though it’s yours) also is on Github. So, there’s a high chance that it’ll mark your contents as “duplicate”. And, duplicate contents gets heavy hammers from Google’s SEO point-of-view. Searching any writeup from yours, even it’s very unique on Internet, Google search will show both your site and the repo as well, possibly links from your site will be on the lower side.

It begs the question how to avoid that. It’s not complicated.

For example, Github.com’s robot.txt allows Google (or any search engine for that matter) to index only the master branch of any open-source repo. So, if your repo don’t have any branch named master, it won’t be indexed. That’s it. Rename your “master” as “live”, “main”, “production” or whatever you feel like.

Create a new “master”

$ git checkout -b new-master
$ git push -u origin new-master

Tell Github about new move

Now, delete master branch from your repo.

$ git branch -d master
$ git push origin :master

Utilize `rel="canonical"` link¶

This is kind of a must-do for avoiding being marked as duplicate content. This will also defend you against automatic content-scraping schemes by always having a pointer to your original source.

To utilize it on Pelican, add or make sure that a <link> element with the attribute rel="canonical" to the <head> section of your base template named base.html.

{% if article %}
<link rel="canonical" href="{{ SITEURL }}/{{ article.url }}"/>
{% endif%}

For example, if you see this page’s source (pressing CTRL+u), you should see something like

<link rel="canonical" href="https://blog.kmonsoor.com/pelican-how-to-make-seo-friendly/"/>

Proper < title > of each page¶

Every page on your site should have a proper title. For search engines, it represents the page. It should concisely reflect a page’s content. But try to keep it less 60 characters or search-engines may choose to truncate it. Use each characters wisely.

It may look like this in your base.html.

{% if article %}
<title>{{ article.title }} -- {{ TAGLINE }}</title>
{% else %}
<title>{{ TAGLINE }}</title>
{% endif%}

The else clause here is to ensure that non-article pages also get a title, even it’s just your tagline defined in pelicanconf.py.

Meta descriptions¶

Include a meta-description to be added on each page of your site. Though it may don’t directly hit SEO ranking, but it appears as a snippet on the search page. So, user should get a proper glimpse of what your page gonna talk about. Make sure your theme uses Pelican’s summary-tagged text for this purpose. Else, ensure it yourself by editing base.html.

{% if article and article.summary %}
<meta name="description" content="{{ article.summary|striptags }}"/>
{% else %}
<meta name="description" content="{{ SITE_SUMMARY }}"/>
{% endif%}

Use `search-console` extensively¶

Google’s Search-console, previously known as webmaster-tools is your friend. Utilize it as far as you can go.

Extensively use PageSpeed Insights ¶

To understand where are current bottlenecks of your site, this tools gives quite a lot insights. Address those one-by-one.

Set preferred version of your site¶

If you have www, http and https versions of your site, tell Google here which one is preferred. It’s only applicable to your domain-root. Once applied and Google re-indexed your site, all the search-results from your site will show that preferred version of your site.

You have to add and do it same both for http and https version of your site, if you have both.

If you have both, either you can use a javascript-code snippet in the <head> of base.html to redirect any http page to its https counterpart.

<script type="text/javascript">
  var host = "your-site.com";
  if ((host == window.location.host) && (window.location.protocol != "https:"))
    window.location.protocol = "https";
</script>

or, if your site is served through NGINX or Apache, you can do it through site’s .htaccess file, by adding the following.

RewriteEngine On
RewriteCond %{HTTPS} !on
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}

Or, if you are using CloudFlare CDN, you can create a page-rule for that as I have shown below.

Check index-status¶

Once in a while Check for Google’s index status of your site on the search-console. Look for error messages or suggestions.

After every major change in your site’s structure, make sure Google bots “know” about it. You can somewhat expedite the process by manually submitting your site.

Include `OpenGraph` data¶

Make sure each of your pages is including proper OpenGraph tags, e.g. og:title, og:content etc., in your template.

Though, OpenGraph originated from facebook Inc., these tags are now widely used by other social engines, even by Google+. In absence of Twitter tags, Twitter also uses these og tags. Try to include both og: and twitter: tags. Proper data in these tags makes your article cleanly-sharable in these social sites.

The below snippet that I use myself can serve as a starting point.

<!-- OpenGraph protocol tags: http://ogp.me/ -->
<!-- originally adopted to be used for: https://blog.kmonsoor.com -->
<meta property="og:site_name" content="{{ SITENAME }}" />
<meta property="og:type" content="article" />
{% if article and article.title %}    
<meta property="og:title" content="{{ article.title }} -- {{ TAGLINE }}" />
<meta property="og:url" content="{{ SITEURL }}/{{ article.url }}" />
{% endif%}
{% if article and article.summary %}
<meta property="og:description" content="{{ article.summary|striptags }}" />
{% else %}
<meta name="og:description" content="{{ SITE_SUMMARY }}"/>
{% endif%}
{% if article and article.date %}
<meta property="article:published_time" content="{{ article.date }}" />
{% endif%}
{% if article and article.modified %}
<meta property="article:modified_time" content="{{ article.modified }}" />
{% endif%}
<!-- End of OpenGraph protocol tags -->

{% if TWITTER_USERNAME %}
<meta name="twitter:site" content="@{{ TWITTER_USERNAME }}" />
<meta name="twitter:creator" content="@{{ TWITTER_USERNAME }}" />
{% endif%}
<meta name="twitter:image" content="INSERT-YOUR-TWITTER-IMAGE-LINK" />
{% if article and article.summary %}
<meta name="twitter:card" content="{{ article.summary|striptags }}" />
{% else %}
<meta name="twitter:card" content="{{ SITE_SUMMARY }}"/>
{% endif%}

Notes¶

Grab your own Twitter’s avatar link do the following:
Go to your Twitter profile page
Right-click on your profile picture
Select “Copy image address” / “Copy image link”
For OpenGraph tags you may also consider to use pelican-opengraph plugin.
For all these to work properly, make sure SITEURL, TAGLINE, SITE_SUMMARY, TWITTER_USERNAME are properly defined in your pelicanconf.py alongwith in publishconf.py files. Please remember that definitions in publishconf.py only apply when you using make publish command.

Loading Performance¶

Compress everything¶

PageSpeed impacts SEO directly. Google punishes slow-site especially when search is made on a mobile device. Mobile-optimized sites will definitely rank higher on searches from mobile-devices.

So, make sure all static files are compressed. If not, compress your themes theme’s JS, CSS files yourself to a .min. version and then replace those in the template files of the theme.

Or, better to use gzip_cache for gzipping all the HTML files statically, also and yuicompressor plugin for compressing JS & CSS files for Pelican. Those will make sure that, upon build, everything is compressed.

Utilize CDN if you can¶

Use CDN-ed versions of common libraries(e.g. jQuery, Bootstrap etc.) rather than hosting your own copy, unless your theme actively modified it. Look it up on CloudFlare cdnjs, cdnjs, or on jsdelivr etc. and use those links.
Try to use a CDN for edge-distribution of your site. I only know of CloudFlare that provide this service for free for a single site. There might be others. CF also make managing DNS configuration little breezy.

Engage commenting¶

While serving a static site, integrating a commenting-system looks a little far-fetched. However, blogs without proper commenting system feels kinda lame sometimes. Of course, YMMV.

But, it’s not difficult; easily can be done by systems like Disqus etc. I’m not affiliated with them, by the way.

Host images separately¶

Host all the images separately that you’ve used in your articles. Use image-specific hosting e.g. imgur.com, imgpile.com, UltraIMG.com, postimage.org etc.

But, why? Because, these services provide couple of benefits besides being free.

Firstly, while loading the page, browser can parallelize loading from this hosts rather than your original blog hosts.
More often than not, these services use own CDN.
Often these services resize your uploaded images automatically to be used in different contexts, which enables you to choose the best-fit size on the fly but without doing it by hand.

Other tips¶

Name your images properly¶

Because search-engines index images too. With proper names, images becomes relevant with the topic, hence potential to draw traffic.

Use Google Keyword planner¶

Even if you are not willing to blow money on ads, it will immensely help you to find out more searched for keywords.

Page-headers (controversial)¶

Don’t use multiple 1^st-level headers H1 style. The main title link probably already have used it once. Look it up. So, avoid it anymore, meaning avoid underlined-style(=======) or hash-style(single ‘#’) headers in your markdown files.

However, 2^nd-level H2 tags can be (read ‘should be’) used multiple times. In case of markdown files, that’s --------- underlines, or line starting with double-hash(‘##’).

And, there’s lot more … ;)¶

As I’ll gain more insights, I hope to grow this post. For now, this is a work-in-progress.
Thanks for reading down so far. Adios !

R vs Python — A Summary Comparison

2016-02-08T08:02:00+06:00

..	R	Python
Home	http://r-project.org	http://python.org
Birth	1993	1991
Designer	Ross Ihaka, Robert Gentleman	Guido van Rossum
Package count	7,798 (CRAN)	73,402 (PyPI)
Purpose	statistical computing	General purpose
Paradigm	array, object-oriented, imperative, functional, procedural, reflective	object-oriented, imperative, functional, procedural, reflective
License	GPL	MIT
Platform	cross-platform	cross-platform
Implementations	Main, Rejnin, FastR	cPython, PyPy, Jython, IronPython
TIOBE^TM index	19	5
Dev. IDE	R Studio, Tinn-R, Architect	PyCharm, IPython/Jupyter, Spyder, Komodo IDE
Graphics support	built-in, ggplot2, lattice, Vistat, googleVis	matplotlib, Seaborn, Plotly, Bokeh
Machine-learning	e1071, rpart, igraph, nnet, kernlab, caret	scikit-learn, PyML, PyMC, Shogun
Deep-Learning	H₂O, Darch	TensorFlow, Theano, Decaf, PDNN, Keras, Caffe
Image-Analysis	Raster, EBImage, imager	scikit-image, OpenCV, scipy.ndimage
Github project count	34,268	164,852
Latest release	2015-12-10 (Main: R-3.2.3)	2015-12-21 (cPython: Python 3.4.4)

N.B. * Time-dependent data are up-to-date as of: 2016-02-08 *

…

Se7en Deadly Sins to Do in Python code

2015-08-10T17:05:00+06:00

Prelude¶

I have used the word “deadly” to express the potential to diminish the productivity of a Python programmer or his fellow teammate(s) who will work on the same code. Please take all these with quite a bit of salt, due to my limited expertise & limited experience with different types of projects based on Python.*

7 is just a catchy number. And, of course, this top list is subject to change along with my experience.
You are also most welcome to suggest your own-finding to make into this list.

There are a lot of ways someone can make his (or her) Python code extremely difficult for himself and his fellow developers to work with and maintain. However, some are quite destructive by virtue. These ones are in my top-list.

1. The `try: except: pass` trio¶

You know about design patterns, right ? At least, you know a little bit.

From Wikipedia,

Design patterns can speed up the development process by providing tested, proven development paradigms.

Effective software design requires considering issues that may not become visible until later in the implementation. Reusing design patterns helps to prevent subtle issues that can cause major problems, and it also improves code readability for coders and architects who are familiar with the patterns.

Now, think of the complete opposite of design-pattern. It is called anti-pattern which silently “destroys” efficiency in code. The below pattern can be considered the most deadly anti-pattern in Python code.
Aaron Maxwell called it most diabolical or “evil” anti-pattern.

try:
    subtle_buggy_operation()   # possibly with I/O or DB operation
except:
    pass

You thought to save some development time by “pass”ing them by. But, it will take hours, if not days, to find possible bugs, inside the block, later as all the exceptions are masked by the “pass” and the error location will be somewhere else outside this try:except block which may look like the most innocent code.

Again, quoting from Aaron …

In my nearly ten years of experience writing applications in Python, both individually and as part of a team, this pattern has stood out as the single greatest drain on developer productivity and application reliability, especially over the long term.

2. Wildcard imports i.e. `from module import ` *¶

This one single practice can render a nice (clean) module into a nightmare. According to a core Python developer David Goodger,

Wild-card imports are from the dark side of Python.

Never!

The from module import * wild-card style leads to namespace pollution. You’ll get things in your local namespace that you didn’t expect to get. You may see imported names obscuring module-defined local names. You won’t be able to figure out where certain names come from. Although a convenient shortcut, this should not be in production code.

Moral: don’t use wild-card imports!

Also, in light of Yoda’s mythical conversations, David writes:

LUKE: Is from module import * better than explicit imports?
YODA: No, not better. Quicker, easier, more seductive.
LUKE: But how will I know why explicit imports are better than the wild-card form?
YODA: Know you will when your code you try to read six months from now.

If you use this practice in between inter-connected modules in a mid-sized project, worry not. You’ll start to get errors due to circular references soon enough.

Sounds funny ?

3. Thinking that `try:except:else` construct is not a natural control flow in Python¶

If you are coming from Java(or, similar) world, I understand your confusion. However, Python adopted this construct so much different than Java. It helps to realize Python’s philosophy Ask for Forgiveness than Permission, aka “EAFP paradigm”.

Trying to avoid this will result in messy, unpythonic code. As this great answer on StackOverflow, by a core Python developer, Raymond Hettinger, on this matter where he nicely portrays the philosophy behind it.

Quoting him :

In the Python world, using exceptions for flow control is common and normal. Even the Python core developers use exceptions for flow-control and that style is heavily baked into the language (i.e. the iterator protocol uses StopIteration to signal loop termination). In addition, the try-except-style is used to prevent the race-conditions inherent in some of the “look-before-you-leap” constructs.

For example, testing os.path.exists results in information that may be out-of-date by the time you use it. Likewise, Queue.full returns information that may be stale. The try:except:else style will produce more reliable code in these cases. In some other languages, that rule reflects their cultural norms as reflected in their libraries. The “rule” is also based in-part on performance considerations for those languages.

Also, consider checking out this Q&A on StackOverflow on the same premise.

4. Making everything a Class aka Overusing classes¶

What I am referring to is this talk by Jack Diederich on PyCon 2012. You should watch this couple of times and then once in every week.
His summary is like … Stop creating classes, and modules in every now and then. Before creating one, think hard. Probably, what you need is writing just a function.

Zen of Python described it as below. Read it again, again, and again.

Beautiful is better than ugly.

Explicit is better than implicit.

Simple is better than complex.

Flat is better than nested.

Readability counts.

If the implementation is hard to explain, it’s a bad idea.

Though the below is a perfectly valid class, it is a perfect example case of b***sh*t classes:

class Greeting(object):
    def __init__(self, greeting='hello'):
        self.greeting = greeting

    def greet(self, name):
        return '%s! %s' % (self.greeting, name)

greeting = Greeting('hola')
print(greeting.greet('bob'))

It is doing exactly same as:

def greet(greeting, target):
    return '%s! %s' % (greeting, target)

He also showed a practical example how he simplified (aka re-factored) an API’s complete code, consisting:

1 Package, 22 Modules,
20 Classes,
660 Source Lines of Code

into this below, a grand total of 8 lines. Yes, just 8 lines !!!

MUFFIN_API = url='https://api.wbsrvc.com/%s/%s/'
MUFFIN_API_KEY = 'SECRET-API-KEY'

def request(noun, verb, **params):
    headers = {'apikey' : MUFFIN_API_KEY}
    request = urllib2.Request(MUFFIN_API % (noun, verb), \
                              urllib.urlencode(params), headers)
    return json.loads(urllib2.urlopen(request).read())

Moral:

Stop re-inventing the wheel,
use more of built-in library functions,
use much-less own long chains of class-hierarchy.

Still want to see a worst scenario of creating classes? Check this out:

class Flow(object):
    """Base class for all Flow objects."""
    pass

class Storage(object):
    def put(self, data): _abstract()
    def get(self): _abstract()

def _abstract(): raise NotImplementedError

Yes, this is a real piece of code from Google API client code. (which, in total, has 10,000 SLOC, 115 modules, 207 classes). Whereas someone did implemented the same, well maybe not extremely robust, but in 135 SLOC, 3 classes in total.

You see the point, right? Guido did. Check his comment.

5. Saving time by not writing any documentation or inline comments¶

If you don’t write comments with your semi-obfuscated code, and no docstrings as well saving time and meeting deadlines, stay assure that within a short period you’ll hate yourself when you will not remember what (& why) you did something while reading your own code.

Today or tomorrow, you will leave the company. And, that code will haunt all the members of your team who will come across this code-like zombies; unless they totally cut-off-the-head(e.g. replace) of your code.

There is just no excuse that you don’t do “documentation” except you just don’t care. If you would care, you would not only write minimal doc-strings and comments on complex code-sections, but also name your functions, methods, variables to reflect the purpose of the component to make them “self-documented”.

Here is a nice guide to properly documenting your Python code.

However, there will still be deniers out there …

* source: https://xkcd.com/1513/ *

6. Avoiding Unit-tests (and doc-tests) until the doomsday comes¶

Yes, the judgement day will come.
It will happen on the production server, with customer’s downtime due to a “completely” manually-tested new feature, which will break something “almost” unrelated.

Yes, your company can lose millions and can be out of business. Maybe after some sleep-less night of the development team, the “bug” would have found out.

Maybe, this whole mess could be simply avoided if the developer wrote his/her modules’ unit-test as well as doctests for the functions or methods. And, after implementing the feature he would have run the tests once across the project. The online book Dive-in-Python has an excellent introduction on unittest. Also, you can start with Hitchhiker’s guide’s introduction.

7. Mixing `TAB` and `SPACE` in the same file¶

You will need no more reason to curse yourself just a while after. It will haunt you whenever you’ll need to open the source-code in any editor other than your usual one. And, for others, “oh my! I can’t literally even…”.

While Python3 will simply refuse to interpret this “half-breed” file, in Python2, the interpretation of TAB is as if it is converted to spaces using 8-space tab stops. So while executing, you may have no clue how a specific-line is being executed as part of which code-block.

For any code that you think someday someone else will read or use, to avoid confusion, you should stick with PEP-8, or your team-specific coding style. PEP-8 strongly discourage mixing TAB and Space in a same file.

Also, check out this Q&A on StackExchange.

1. The first downside is that it quickly becomes a mess.

… Formatting should be the task of the IDE. Developers have already enough work to care about the size of tabs, how much spaces will an IDE insert, etc. The code should be formatted correctly, and displayed correctly on other configurations, without forcing developers to think about it.

Also, remember this

Furthermore, it can be a good idea to avoid tabs altogether, because the semantics of tabs are not very well-defined in the computer world, and they can be displayed completely differently on different types of systems and editors.
Also, tabs often get destroyed or wrongly converted during copy-paste operations, or when a piece of source code is inserted into a web page or other kind of markup code.

Fin¶

That’s all for now. That’s my list. This list hopes to evolve with my experience and expertise as well as the ever-changing collective wisdom of all the Python community.

What’s your take on the worst “un-pythonic” nightmares in Python code? Please feel free to share your 2-cents.

Python - How to Create All Possible Combinations (a.k.a Permutations)

2014-01-31T00:00:00+06:00

Let’s say the number is 123.

Then the statement:

[int(''.join(x)) for x in permutations(list(str(123)))]

will create a sorted list of all possible numbers using ‘1’,‘2’ and ‘3’. which is, [123, 132, 213, 231, 312, 321]

What it is doing¶

The part, permutations(list(str('123' is creating a permutated tuples’ list of splitted string ‘123’. And the int(''.join(x)) is converting each tuple to Integer.

However, of course, you need to import permutations from itertools.

so, the generalized version would be:

from itertools import permutations
[int(''.join(x)) for x in list(permutations(list(str(n))))]

Python is fun. Isn’t it?