I think ... - programming

R vs Python — A Summary Comparison

2016-02-08T08:02:00+06:00

..	R	Python
Home	http://r-project.org	http://python.org
Birth	1993	1991
Designer	Ross Ihaka, Robert Gentleman	Guido van Rossum
Package count	7,798 (CRAN)	73,402 (PyPI)
Purpose	statistical computing	General purpose
Paradigm	array, object-oriented, imperative, functional, procedural, reflective	object-oriented, imperative, functional, procedural, reflective
License	GPL	MIT
Platform	cross-platform	cross-platform
Implementations	Main, Rejnin, FastR	cPython, PyPy, Jython, IronPython
TIOBE^TM index	19	5
Dev. IDE	R Studio, Tinn-R, Architect	PyCharm, IPython/Jupyter, Spyder, Komodo IDE
Graphics support	built-in, ggplot2, lattice, Vistat, googleVis	matplotlib, Seaborn, Plotly, Bokeh
Machine-learning	e1071, rpart, igraph, nnet, kernlab, caret	scikit-learn, PyML, PyMC, Shogun
Deep-Learning	H₂O, Darch	TensorFlow, Theano, Decaf, PDNN, Keras, Caffe
Image-Analysis	Raster, EBImage, imager	scikit-image, OpenCV, scipy.ndimage
Github project count	34,268	164,852
Latest release	2015-12-10 (Main: R-3.2.3)	2015-12-21 (cPython: Python 3.4.4)

N.B. * Time-dependent data are up-to-date as of: 2016-02-08 *

…

Se7en Deadly Sins to Do in Python code

2015-08-10T17:05:00+06:00

Prelude¶

I have used the word “deadly” to express the potential to diminish the productivity of a Python programmer or his fellow teammate(s) who will work on the same code. Please take all these with quite a bit of salt, due to my limited expertise & limited experience with different types of projects based on Python.*

7 is just a catchy number. And, of course, this top list is subject to change along with my experience.
You are also most welcome to suggest your own-finding to make into this list.

There are a lot of ways someone can make his (or her) Python code extremely difficult for himself and his fellow developers to work with and maintain. However, some are quite destructive by virtue. These ones are in my top-list.

1. The `try: except: pass` trio¶

You know about design patterns, right ? At least, you know a little bit.

From Wikipedia,

Design patterns can speed up the development process by providing tested, proven development paradigms.

Effective software design requires considering issues that may not become visible until later in the implementation. Reusing design patterns helps to prevent subtle issues that can cause major problems, and it also improves code readability for coders and architects who are familiar with the patterns.

Now, think of the complete opposite of design-pattern. It is called anti-pattern which silently “destroys” efficiency in code. The below pattern can be considered the most deadly anti-pattern in Python code.
Aaron Maxwell called it most diabolical or “evil” anti-pattern.

try:
    subtle_buggy_operation()   # possibly with I/O or DB operation
except:
    pass

You thought to save some development time by “pass”ing them by. But, it will take hours, if not days, to find possible bugs, inside the block, later as all the exceptions are masked by the “pass” and the error location will be somewhere else outside this try:except block which may look like the most innocent code.

Again, quoting from Aaron …

In my nearly ten years of experience writing applications in Python, both individually and as part of a team, this pattern has stood out as the single greatest drain on developer productivity and application reliability, especially over the long term.

2. Wildcard imports i.e. `from module import ` *¶

This one single practice can render a nice (clean) module into a nightmare. According to a core Python developer David Goodger,

Wild-card imports are from the dark side of Python.

Never!

The from module import * wild-card style leads to namespace pollution. You’ll get things in your local namespace that you didn’t expect to get. You may see imported names obscuring module-defined local names. You won’t be able to figure out where certain names come from. Although a convenient shortcut, this should not be in production code.

Moral: don’t use wild-card imports!

Also, in light of Yoda’s mythical conversations, David writes:

LUKE: Is from module import * better than explicit imports?
YODA: No, not better. Quicker, easier, more seductive.
LUKE: But how will I know why explicit imports are better than the wild-card form?
YODA: Know you will when your code you try to read six months from now.

If you use this practice in between inter-connected modules in a mid-sized project, worry not. You’ll start to get errors due to circular references soon enough.

Sounds funny ?

3. Thinking that `try:except:else` construct is not a natural control flow in Python¶

If you are coming from Java(or, similar) world, I understand your confusion. However, Python adopted this construct so much different than Java. It helps to realize Python’s philosophy Ask for Forgiveness than Permission, aka “EAFP paradigm”.

Trying to avoid this will result in messy, unpythonic code. As this great answer on StackOverflow, by a core Python developer, Raymond Hettinger, on this matter where he nicely portrays the philosophy behind it.

Quoting him :

In the Python world, using exceptions for flow control is common and normal. Even the Python core developers use exceptions for flow-control and that style is heavily baked into the language (i.e. the iterator protocol uses StopIteration to signal loop termination). In addition, the try-except-style is used to prevent the race-conditions inherent in some of the “look-before-you-leap” constructs.

For example, testing os.path.exists results in information that may be out-of-date by the time you use it. Likewise, Queue.full returns information that may be stale. The try:except:else style will produce more reliable code in these cases. In some other languages, that rule reflects their cultural norms as reflected in their libraries. The “rule” is also based in-part on performance considerations for those languages.

Also, consider checking out this Q&A on StackOverflow on the same premise.

4. Making everything a Class aka Overusing classes¶

What I am referring to is this talk by Jack Diederich on PyCon 2012. You should watch this couple of times and then once in every week.
His summary is like … Stop creating classes, and modules in every now and then. Before creating one, think hard. Probably, what you need is writing just a function.

Zen of Python described it as below. Read it again, again, and again.

Beautiful is better than ugly.

Explicit is better than implicit.

Simple is better than complex.

Flat is better than nested.

Readability counts.

If the implementation is hard to explain, it’s a bad idea.

Though the below is a perfectly valid class, it is a perfect example case of b***sh*t classes:

class Greeting(object):
    def __init__(self, greeting='hello'):
        self.greeting = greeting

    def greet(self, name):
        return '%s! %s' % (self.greeting, name)

greeting = Greeting('hola')
print(greeting.greet('bob'))

It is doing exactly same as:

def greet(greeting, target):
    return '%s! %s' % (greeting, target)

He also showed a practical example how he simplified (aka re-factored) an API’s complete code, consisting:

1 Package, 22 Modules,
20 Classes,
660 Source Lines of Code

into this below, a grand total of 8 lines. Yes, just 8 lines !!!

MUFFIN_API = url='https://api.wbsrvc.com/%s/%s/'
MUFFIN_API_KEY = 'SECRET-API-KEY'

def request(noun, verb, **params):
    headers = {'apikey' : MUFFIN_API_KEY}
    request = urllib2.Request(MUFFIN_API % (noun, verb), \
                              urllib.urlencode(params), headers)
    return json.loads(urllib2.urlopen(request).read())

Moral:

Stop re-inventing the wheel,
use more of built-in library functions,
use much-less own long chains of class-hierarchy.

Still want to see a worst scenario of creating classes? Check this out:

class Flow(object):
    """Base class for all Flow objects."""
    pass

class Storage(object):
    def put(self, data): _abstract()
    def get(self): _abstract()

def _abstract(): raise NotImplementedError

Yes, this is a real piece of code from Google API client code. (which, in total, has 10,000 SLOC, 115 modules, 207 classes). Whereas someone did implemented the same, well maybe not extremely robust, but in 135 SLOC, 3 classes in total.

You see the point, right? Guido did. Check his comment.

5. Saving time by not writing any documentation or inline comments¶

If you don’t write comments with your semi-obfuscated code, and no docstrings as well saving time and meeting deadlines, stay assure that within a short period you’ll hate yourself when you will not remember what (& why) you did something while reading your own code.

Today or tomorrow, you will leave the company. And, that code will haunt all the members of your team who will come across this code-like zombies; unless they totally cut-off-the-head(e.g. replace) of your code.

There is just no excuse that you don’t do “documentation” except you just don’t care. If you would care, you would not only write minimal doc-strings and comments on complex code-sections, but also name your functions, methods, variables to reflect the purpose of the component to make them “self-documented”.

Here is a nice guide to properly documenting your Python code.

However, there will still be deniers out there …

* source: https://xkcd.com/1513/ *

6. Avoiding Unit-tests (and doc-tests) until the doomsday comes¶

Yes, the judgement day will come.
It will happen on the production server, with customer’s downtime due to a “completely” manually-tested new feature, which will break something “almost” unrelated.

Yes, your company can lose millions and can be out of business. Maybe after some sleep-less night of the development team, the “bug” would have found out.

Maybe, this whole mess could be simply avoided if the developer wrote his/her modules’ unit-test as well as doctests for the functions or methods. And, after implementing the feature he would have run the tests once across the project. The online book Dive-in-Python has an excellent introduction on unittest. Also, you can start with Hitchhiker’s guide’s introduction.

7. Mixing `TAB` and `SPACE` in the same file¶

You will need no more reason to curse yourself just a while after. It will haunt you whenever you’ll need to open the source-code in any editor other than your usual one. And, for others, “oh my! I can’t literally even…”.

While Python3 will simply refuse to interpret this “half-breed” file, in Python2, the interpretation of TAB is as if it is converted to spaces using 8-space tab stops. So while executing, you may have no clue how a specific-line is being executed as part of which code-block.

For any code that you think someday someone else will read or use, to avoid confusion, you should stick with PEP-8, or your team-specific coding style. PEP-8 strongly discourage mixing TAB and Space in a same file.

Also, check out this Q&A on StackExchange.

1. The first downside is that it quickly becomes a mess.

… Formatting should be the task of the IDE. Developers have already enough work to care about the size of tabs, how much spaces will an IDE insert, etc. The code should be formatted correctly, and displayed correctly on other configurations, without forcing developers to think about it.

Also, remember this

Furthermore, it can be a good idea to avoid tabs altogether, because the semantics of tabs are not very well-defined in the computer world, and they can be displayed completely differently on different types of systems and editors.
Also, tabs often get destroyed or wrongly converted during copy-paste operations, or when a piece of source code is inserted into a web page or other kind of markup code.

Fin¶

That’s all for now. That’s my list. This list hopes to evolve with my experience and expertise as well as the ever-changing collective wisdom of all the Python community.

What’s your take on the worst “un-pythonic” nightmares in Python code? Please feel free to share your 2-cents.

Generate ER diagram from a SQL-based database

2014-12-18T00:00:00+06:00

When you are “study”-ing (for whatever reason) someone else’s database, and the database has more than 20 tables, you might be in trouble to understand what’s going where.

Now, imagine a database with 300+ tables. It’s like spaghetti, just not as enjoyable.

I faced a similar challenge recently with a database of 250+ tables. Yes, I felt like in a deep sh*t. And, I started looking for tools that can describe the tables, or at least a decent ER diagram. If anything more, better. And, preferably free.

Then, I found SchemaSpy, originally authored by John Currier. It generates a complete in-depth HTML-based description (of course, including clickable ER-diagram) of the database, which you can then browse with your browser. This post is about its primary usage.

The output will be kind of like this:

It is based on Java, but it can work its magic on most of the major database products. However, it would require an appropriate JDBC connector for that database.

Quote from the author:

SchemaSpy uses JDBC’s database metadata extraction services to gather the majority of its information but has to make vendor-specific SQL queries to gather some information such as the SQL associated with a view and the details of check constraints.

In this post, as an example, I have shown how to use it with PostgreSQL. But, it’s not the only one that’s supported. You can use it with any proper RDBMS system as long as it has a JDBC-connector. Now, to use it, you need to get these stuffs.

First of all, your system should have Java runtime properly installed. Download from here.
SchemaSpy, which is a .jar file. Get it here. At the time of writing, it was version 5.0.0.
JDBC connector to PostgreSQL. Make sure to match your PostgreSQL version. You can download it from here.
You can check your PostgreSQL version by executing: SELECT version(); query on psql prompt.
Also, SchemaSpy depends on GarphViz to generate the ER diagrams, so you need to be installed it on your system. Get it from here.(http://www.graphviz.org/Download..php)
And, of course, make sure the target database instance is running & serving the database you are trying to visualize.

Quoting from the author:

SchemaSpy uses the dot executable from Graphviz to generate graphical representations of the table/view relationships. The visual representation of the connections is a fundamental feature of the tool.

Graphviz is not required to view the output generated by SchemaSpy, but the dot program should be in your PATH (not CLASSPATH) when running SchemaSpy, or none of the entity-relationship diagrams will be generated. Or, maybe use the -gv option).

I kept the .jar files (both the JDBC-connector and the SchemaSpy) in my home folder for convenience.
Now, in my case, my OS is Linux, and the database is hosted locally; hence address is 127.0.0.1, running PostgreSQL-9.3 at port 5432. So, I run the command like this:

 $ java -jar ./schemaSpy_5.0.0.jar -t pgsql -host 127.0.0.1:5432 -db your_database_name \
                                   -u your_DB_user_name -p your_password -s public \
                                   -dp ./postgresql-9.3-1102.jdbc3.jar \
                                   -o output_folder

It may take a little while, depending on the size of the schema of the database.
After that, you will find the output folder/directory named output_folder.
You’ll see some output when the magic is going on, similar to this below.

Using database properties: [./schemaSpy_5.0.0.jar]/net/sourceforge/schemaspy/dbTypes/pgsql.properties
Gathering schema details....................(6sec)
Writing/graphing summary....................(2sec)
Writing/diagramming detail..................(31sec)
Wrote relationship details of 113 tables/views to directory 'output' in 41 seconds.

View the results by opening output_folder/index.html

Now, all the generated files are in the output_folder. Start your journey by starting from the index.html in the output folder. Open it by using any browser.

Good luck. :)

If you find this post helpful, you can show your support through Patreon or by buying me a coffee. Thanks!