Python

Install

Ubuntu server 16.04LTS

sudo apt-get update 
sudo apt-get install make dpkg-dev libssl-dev libreadline6 libreadline6-dev libsqlite3-dev build-essential libncursesw5-dev libreadline-gplv2-dev 
sudo apt-get install libssl-dev libgdbm-dev libc6-dev libsqlite3-dev tk-dev libbz2-dev libncurses-dev
wget https://www.python.org/ftp/python/2.7.12/Python-2.7.12.tar.xz
tar xvf Python-2.7.12.tar.xz
./configure
make
make install

Indentation

Python is an indented language, so the code indentation matters. A good practice is to indent with 4 spaces (if you mix spaces and tabs the code won't work.

Operators

Arithmetic

+    Addition
-    Substraction
*    Multiplication
/    Division
**   Power
%    Reminder
//   Floor division (Reminder is removed)

Comparison

==   Equal 
!=   Not equal
>    Greater
>=   Greater or equal
<    Smaller
<=   Smaller or equal
<>   Similar to !=

Assignment

=    Simple assignment
+=   Add and as
-=   Substract and  assignment
*=   Multiply and assignment
/=   Divide and assignment
%=   Modulus and assignment
**=  Exponent and assignment
//=  Floor Divisionn and assignment

Bitwise operators

They perform operations on binary terms. a= 8 → 100; b= 9 → 101; a & b → 100

$    Binary AND
|    Binary OR
^    Binary XOR
~    Binary complement
<<   Binary left shift
>>   Binary right shift

Logic

and
or
not

Membership operators

in
not in

Variables

String

String assingment:
MyString = 'Hello World' OR MyString = "Hello World"
Strings can be subset sliced:

MyString = "Hello world"
MyString[0:4]

Hel

in operator: to check if a substring is contained in a string

String library:

str.lower()
str.upper()
str.capitalize() → Uppercases 1st char
str.center(width[, fillchar])
str.startswith(preffix[, start[, end]])
str.endswith(suffix[, start[, end]])
str.find(substring[, start[, end]])
str.lstrip([chars])
str.rstrip([chars])
str.strip([chars])
str.replace(old, new[, count])see also threading module:

import threading


str.split([char])

len(MyString) → returns the lenght of a string

Collections

List

Collection - allows us to put many values in a single variable. Simple variables are not collections. The previous value is overwritten when changed.

• A List is made up of list 'constants'. Lists are surrounded by square brackets [] and the constants in the list are separated by commas. ([2,4,6,8])
• A List element can be any Python object, even another list
• A List can be empty
• Lists are mutable (they can be changed)
• When len() is used on a list, it counts the number of constants that make up the list. (not the number of characters)
• Lists can be concatenated using +
• Lists can be sliced
• List is a unique type that can be checked using type() (result: <type 'list'>)
• An empty list can be created with list()
• Lists can be tested for contents using in/not in
• List is an ordered sequence
• A list can be sorted with .sort(). Sort changes the list permanently.
• sorted(list) → Returns a list sorted.
• Methods: append, count, extend, index, insert, pop, remove, reverse, sort
• Functions len() - find length, max() - find highest value, min() - find lowest value, sum() - add all values average can be found with sum()/len()

Del Removes an item from a list.
example: a = [1, 2, 3, 4, 5] del a[0] returns [2, 3, 4, 5]

Tuple

Inmutable, declared by ()

• My_tuple = (3, 2, 1)

You can iterate trough tuples.
You can also refer to the items of a tuple like in a list.

My_tuple[0]
3

They support multple assignment, usefull for example to iterate trough dictionaries:

d = dict()
d['steve'] = 2
d['susan'] = 4

for (k, v) in d.items():
    print k, v

Tuples are comparable. They compare the leftmost first, then the next and so on...

>>> (0, 1, 2) < (5, 1, 2)
True
>>> (0, 1, 30000000) < (0, 3, 1)
True

To sort dictionaries by keys.

>>> d = {'c':22, 'b':1, 'a':10}
>>> t = d.items()
>>> t
[('c', 22), ('b',1), ('a', 10)]
>>> t.sort()
>>> t
[('a', 10), ('b',1), ('c', 22)]

To sort by value:

c = {'c':22, 'b':1, 'a':10}
tmp = list()
for k, v in c.items():
    tmp.append( (v, k) )
tmp.sort(reverse = True)
print tmp
[(22, 'c'), (10, 'a'), (1, 'b')]

The same sort by value with list comprehension

c = {'c':22, 'b':1, 'a':10}
print sorted( [ (v,k) for k,v in c.items() ] )

Top 10 most common words.

fhand = open('romeo.txt')
counts = dict()
for line in fhand:
    words = line.split()
    for word in words:
        counts[word] = counts.get(word, 0) +1
lst = list()
for key, val in counts.items():
    lst.append((val, key))
lst.sort(reverse = True)
for val, key in lst[:10]
    print key, val

Dictionary

Key - Value pairs. They are called diferent in diferent languages:
dictionary[key] = value

• Perl → Associative arrays
• Java → Properties, Map or HashMap
• C# → Property bag

Declarationn and assigment:
There are two posible ways to declare them:

• purse = dict()
• puse = {}

purse = dict()
purse['money'] = 12
purse['candy'] = 3

print purse['money']

purse = {'money': 12, 'candy' = 3}

get() method:
If you try to retrieve a value for a key that doesn't exist you would get a traceback error.
To avoid this you should use the get() method, that returns the default value if the key doesn't exist: purse.get(name, default_value) Other methods and functions

• list(purse) → Returns a list of keys.
• dict.keys() → Returns a list of keys.
• dict.values() → Returns a list of values.
• dict.items → Returns a list of tuples ( [(key, value), (key, value)...] )

Word count using files and dictionarys

counts = dict()
file_path = raw_input('Enter file name: ')
file_handle = open(file_path, 'r')
text = file_handle.read()
words = text.split()

for word in words:
    counts[word] = counts.get(word, 0) + 1

bigCount = None
bigWord  = None
for word, count in counts.items():
    if bigCount is None or count > bigCount:
        bigWord = word
        bigCount = count

print 'Most frequent word: ', bigWord, '\nFrequency: ', bigCount'

Conditional

if/elif/else

if a < 10:
    print "Less than 10"
elif a >= 10 and a < 20:
    print "a greater or equal to 10 and less than 20"
else:
    print "a greater or equal to 20"

try/except/finally

try:
    file = open("test.txt")
except:
    print "Could not open file"
finally:
    print "This part will be executed at the end whether the open fails or not"

Loops

For

NumberList = [1, 3, 7, 12, 24]
for number in NumberList:
    print number

Another way:

for i in range(0, len(NumberList)-1):
    print NumberList[i]

while

CtrlNum = 7
while CtrllNum > 3:
    print CtrlNum
    CtrlNum -= 1

Functions

All arguments in Python are passed by reference, if you change a variable value inside a function it will be changed at the calling function.

def MyFunction(formal_arg, optional_arg = None, *variable_lenght_args):
    print formal_arg
    if optional_arg:
        print optional_arg
    for arg in variable_lenght_args:
        print arg

Regular Expressions

   ^     → Matches the beginning of a line
   $     → Matches the end of a line
   .     → Matches any character
  \s     → Matches any whitespace
  \S     → Matches any non-whitespace
   *     → Repeats a character 0 or more times
  *?     → Repeats a character 0 or more times (non-greedy)
  +      → Repeats a character 1 or more times
  +?     → Repeats a character 1 or more times (non-greedy)
 [aeiou] → Matches a single character in the listed set
 [^XYZ]  → Matches a single character NOT in the listed set
[a-z0-9] → The set of characters can include a range
   (     → Indicates where string extraction is to start
   )     → Indicates where string extraction is to end 
  \     → Escape character

Regular Expression Module

It must be imported at the begining of a program:

import re

• re.search(re_string, string) → similar to find()
• re.findall(re_string, string) → similar to find, returns a list
• re.match(re_string, string)

import re
hand = open('mbox-short.txt')
for line in hand:
    line = line.rstrip()
    if re.search('^From:', line);
        print line

Date

Today as string

import datetime
datetime.date.today().strftime("%B %d, %Y")

String date to date type

from datetime import datetime
fecha_string = '2014-01-15 11:20:05.220'
date = datetime.strptime(fecha_string, '%Y-%m-%d %H:%M:%S.%f')

Files

Counting lines in a file

fhand = open("words.txt")
count = 0
for line in fhand:
    count = count + 1
print "line count", count

Reading the whole file

fhand =  open("words.txt")
inp = fhand.read() # reads the whole file into memory
print len(inp)# returns the number of characters in a file
print inp # prints the whole file

Searching through a file

fhand= open("mbox-short.txt")
for line in fhand:
    # line = line.rstrip()
    if line.startswith("From:"):
        line = line.rstrip() # better here
        print line

File names and paths

fname = raw_input("Enter a file name: ")
    if fname[0:2] == "~/": #Check to see if it starts with a ~ and a slash
        #If it doesn't start with the ~/, then 
        #the user could be referring to a valid file
        #like "~.py" (I checked: it is possible.)
        #notice below replace is valid on Mac OSX only (and not a good approach overall, cause not portable at all)
        fname = fname.replace('~',"/Users/"+raw_input("Enter your short user name: "),1)
workingfname = fname.replace("\\",'') #This for proper escaping of a valid folder named '~' as '\~', you can also use './~' as Python automatically escapes for you.
#go back to normal program now
handle = open(workingfname,'r') # . . .
for line in handle:
    print line
print "\n"+("That was "+fname+".").center(40)

CSV

import csv

def createCSV(simpleProducts, configProducts, shopName):
    '''
    Remaps a CSV filce created with tiendalistas magento exporter to
    match the Shopify CSV import file
    '''

    counter = 0
    with open(OUTPUT_DIALETCT_TEMPLATE, 'rb') as csvDialectTemplate:
        exportDialect = csv.Sniffer().sniff(csvDialectTemplate.read(1024))

    # Create file and write headers
    with open(OUTPUT_PATH + shopName + '_shopify.csv', 'w') as csvoutFile:
        csvWriter = csv.writer(csvoutFile, dialect=exportDialect)
        csvWriter.writerow(shopyheader)
        counter += 1

        # Write lines for simple products
        for prod in simpleProducts:
            tmp = shopifyCSVitem(prod)
            csvWriter.writerow(tmp.productCSVline())
            counter += 1

            for subP in prod.subProducts:
                    tmp2 = shopifyCSVitem(subP)
                    csvWriter.writerow(tmp2.subProductCSVline())
                    counter += 1

        # Write lines for configurable products
        for prod in configProducts:
            tmp1 = shopifyCSVitem(prod)
            csvWriter.writerow(tmp1.productCSVline())
            counter += 1
            logWrite(logFile, 'Configurable product should be reviewed', tmp1.productCSVline()[0])

            for subP in prod.subProducts:
                tmp2 = shopifyCSVitem(subP)
                csvWriter.writerow(tmp2.productCSVline())
                counter += 1

    return counter

Class

class MyClass:
    __init__(self, arg1, arg2):
        #This functionn will be executed every time you instantiate an element of this class
        self.var1 = arg1
        self.var2 = arg2
    
    def myFunction(self, param1):
        self.var1 = self.var1 +param1

Built in attributes:
__dict__ → Dictionary containing the class's namespace
__doc__ → Class documentation string or None if undefined
__name__ → Class name
__module__ → Module name in which the class is defined. This attribute is "__main__" in interactive mode.
__bases__ → A possibly empty tuple containing the base classes in the order of their occurrence

Inheritance

class SubClassName(ParentClass1[, ParentClass2, ...]):
    'Optional class documentation string'
    # Class code

• issubclass(sub, sup)
• isinstance(obj, Class)

Generic functionality that can be overriden in own classes
__init__(self[, args...]) → Constructor
__del__(self) → Executed when Python's garbage collector destroys an object
__repr__(self) → Evaluable string representation
__str__(self) → Printable string representation
__cmp__(self, x) → Object comparison
__add__(self, other) → To define the + operator behave

Hiding attributes attributes that start with a __ wont be visible to others

os.system()

To execute a Linux command from Python:

os.system(command) → returns exit status

Database Access

MySQLdb

Table creation

#!/usr/bin/python

import MySQLdb

# Open database connection
db = MySQLdb.connect("DB_HOST","DB_USER","DB_PASSWORD","DB_NAME" )

# prepare a cursor object using cursor() method
cursor = db.cursor()

# Drop table if it already exist using execute() method.
cursor.execute("DROP TABLE IF EXISTS EMPLOYEE")

# Create table as per requirement
sql = """CREATE TABLE EMPLOYEE (
         FIRST_NAME  CHAR(20) NOT NULL,
         LAST_NAME  CHAR(20),
         AGE INT,  
         SEX CHAR(1),
         INCOME FLOAT )"""

cursor.execute(sql)

INSERT operation

#!/usr/bin/python

import MySQLdb

# Open database connection
db = MySQLdb.connect("DB_HOST","DB_USER","DB_PASSWORD","DB_NAME" )

# prepare a cursor object using cursor() method
cursor = db.cursor()

# Prepare SQL query to INSERT a record into the database.
sql = "INSERT INTO EMPLOYEE(FIRST_NAME, \
       LAST_NAME, AGE, SEX, INCOME) \
       VALUES ('%s', '%s', '%d', '%c', '%d' )" % \
       ('Mac', 'Mohan', 20, 'M', 2000)
try:
   # Execute the SQL command
   cursor.execute(sql)
   # Commit your changes in the database
   db.commit()
except:
   # Rollback in case there is any error
   db.rollback()

# disconnect from server
db.close()

Read example Once the query is made you can:

• fetchone()
• fetchall()
• rowcount

#!/usr/bin/python

import MySQLdb

# Open database connection
db = MySQLdb.connect("localhost","testuser","test123","TESTDB" )

# prepare a cursor object using cursor() method
cursor = db.cursor()

# Prepare SQL query to INSERT a record into the database.
sql = "SELECT * FROM EMPLOYEE \
       WHERE INCOME > '%d'" % (1000)
try:
   # Execute the SQL command
   cursor.execute(sql)
   # Fetch all the rows in a list of lists.
   results = cursor.fetchall()
   print "Found ', cursor.rowcount, ' results: ' 
   for row in results:
      fname = row[0]
      lname = row[1]
      age = row[2]
      sex = row[3]
      income = row[4]
      # Now print fetched result
      print "fname=%s,lname=%s,age=%d,sex=%s,income=%d" % \
             (fname, lname, age, sex, income )
except:
   print "Error: unable to fecth data"

# disconnect from server
db.close()

PostgreSQL

Refer to: PostgreSQL

Other

Refer to: Database Interfaces

ssh tunneling

import subprocess
sshcommand = ['ssh', '-o', 'StrictHostKeyChecking=no', '-i', keyFilePath, '-N', '-L', '8888:localhost:3306', 'user@' + Host]
ssh_proc = subprocess.Popen(sshcommand, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
# Do stuff
ssh_proc.terminate()

Multithreading

#!/usr/bin/python

import thread
import time

#Define a function for the thread
def print_time( threadName, delay):
   count = 0
   while count < 5:
      time.sleep(delay)
      count += 1
      print "%s: %s" % ( threadName, time.ctime(time.time()) )

# Create two threads as follows
try:
   thread.start_new_thread( print_time, ("Thread-1", 2, ) )
   thread.start_new_thread( print_time, ("Thread-2", 4, ) )
except:
   print "Error: unable to start thread"

while 1:
   pass

(*) see also threading module:

import threading

Python GUI Programming

• Tkinter: Tkinter is the Python interface to the Tk GUI toolkit shipped with Python. We would look this option in this tutorial.
• wxPython: This is an open-source Python interface for wxWindows http://wxpython.org.
• JPython: JPython is a Python port for Java which gives Python scripts seamless access to Java class libraries on the local machine http://www.jython.org.

Tkinter

#!/usr/bin/python

import Tkinter
top = Tkinter.Tk()
# Code to add widgets will go here...
top.mainloop()

Widgets

• Button The Button widget is used to display buttons in your application.
• Canvas The Canvas widget is used to draw shapes, such as lines, ovals, polygons and rectangles, in your application.
• Checkbutton The Checkbutton widget is used to display a number of options as checkboxes. The user can select multiple options at a time.
• Entry The Entry widget is used to display a single-line text field for accepting values from a user.
• Frame The Frame widget is used as a container widget to organize other widgets.
• Label The Label widget is used to provide a single-line caption for other widgets. It can also contain images.
• Listbox The Listbox widget is used to provide a list of options to a user.
• Menubutton The Menubutton widget is used to display menus in your application.
• Menu The Menu widget is used to provide various commands to a user. These commands are contained inside Menubutton.
• Message The Message widget is used to display multiline text fields for accepting values from a user.
• Radiobutton The Radiobutton widget is used to display a number of options as radio buttons. The user can select only one option at a time.
• Scale The Scale widget is used to provide a slider widget.
• Scrollbar The Scrollbar widget is used to add scrolling capability to various widgets, such as list boxes.
• Text The Text widget is used to display text in multiple lines.
• Toplevel The Toplevel widget is used to provide a separate window container.
• Spinbox The Spinbox widget is a variant of the standard Tkinter Entry widget, which can be used to select from a fixed number of values.
• PanedWindow A PanedWindow is a container widget that may contain any number of panes, arranged horizontally or vertically.
• LabelFrame A labelframe is a simple container widget. Its primary purpose is to act as a spacer or container for complex window layouts.
• tkMessageBox This module is used to display message boxes in your applications.

Standard attributes

• Dimensions
• Colors
• Fonts
• Anchors
• Relief styles
• Bitmaps
• Cursors

Geometry Management

• The pack() Method - This geometry manager organizes widgets in blocks before placing them in the parent widget.
• The grid() Method - This geometry manager organizes widgets in a table-like structure in the parent widget.
• The place() Method -This geometry manager organizes widgets by placing them in a specific position in the parent widget.

For further reference visit: tutorialspoint.com/python/python_gui_programming.htm

Enable auto complete in python interpreter

Create a file in your home directory named: .pythonrc
Content of this file:

import rlcompleter, readline
readline.parse_and_bind('tab:complete')

Set the PYTHONSTARTUP variable in your .bashrc or .bash_profile

echo "export PYTHONSTARTUP=~/.pythonrc" >> .bashrc

Reload your .bashrc or .bash_profile

source ~/.bashrc

To test it import a library, write librariname. and hit tab twice

Delete *.pyc

find . -name "*.pyc" -exec rm -rf {} \;

format hex

Use the format() function with a '02x' format.

>>> format(255, '02x')
'ff'
>>> format(2, '02x')
'02'

The 02 part tells format() to use at least 2 digits and to use zeros to pad it to length, x
means lower-case hexadecimal.

The Format Specification Mini Language also gives you X for uppercase hex output, and you can
prefix the field width with # to include a 0x or 0X prefix (depending on wether you used
x or X as the formatter). Just take into account that you need to adjust the field width
to allow for those extra 2 characters:

>>> format(255, '02X')
'FF'
>>> format(255, '#04x')
'0xff'
>>> format(255, '#04X')
'0XFF'

nose debug

nosetests --debug=nose,nose.importer --debug-log=nose_debug <your usual args>