1000 Examples Programming In Python

First steps....58 What is Python?....58 What is needed to write a program?....58 The source (code) of Python....59 Python 2 vs. Python 3....59 Installation....59 Installation on Linux....60 Installation on Apple Mac OSX....60 Installation on MS Windows....60 Editors, IDEs....60 Documentation....62 Program types....62 Python on the command line....63 First script - hello world....63 Examples....63 Comments....64 Variables....64 Exercise: Hello world....64 What is programming?....64 What are the programming languages....65 A written human language....65 A programming language....65 Words and punctuation matter!....65 Literals, Value Types in Python....66 Floating point limitation....66 Value Types in Numpy....66 Rectangular (numerical operations)....67 Multiply string....67 Add numbers....67 Add strings....67 Exercise: Calculations....67 Solution: Calculations....68 Second steps....69 Modules....69 A main function....69 The main function - called....69 Indentation....70 Conditional main....70 Input - Output I/O....70 print in Python 2....71 print in Python 3....71 print in Python 2 as if it was Python 3....72 Exception: SyntaxError: Missing parentheses in call....72 Prompting for user input in Python 2....72 Prompting for user input in Python 3....73 Python2 input or raw_input?....74 Prompting both Python 2 and Python 3....74 Add numbers entered by the user (oups)....74 Add numbers entered by the user (fixed)....75 How can I check if a string can be converted to a number?....75 Converting string to int....76 Converting float to int....76 Conditionals: if....77 Conditionals: if - else....77 Conditionals: if - else (other example)....77 Conditionals: else if....77 Conditionals: elif....78 Ternary operator....78 Case or Switch in Python....79 Exercise: Rectangular....79 Exercise: Calculator....79 Exercise: Standard Input....79 Solution: Area of rectangular....79 Solution: Calculator....80 Command line arguments....81 Command line arguments - len....82 Command line arguments - exit....82 Exercise: Rectangular (argv)....82 Exercise: Calculator (argv)....82 Solution: Area of rectangular (argv)....82 Solution: Calculator eval....83 Solution: Calculator (argv)....83 Compilation vs. Interpretation....84 Is Python compiled or interpreted?....85 Flake8 checking....85 Numbers....86 Numbers....86 Operators for Numbers....86 Integer division and the future....87 Pseudo Random Number....88 Fixed random numbers....88 Rolling dice - randrange....88 Random choice....89 built-in method....89 Exception: TypeError: ‘module’ object is not callable....89 Fixing the previous code....90 Exception: AttributeError: module ‘random’ has no attribute....90 Exercise: Number guessing game - level 0....91 Exercise: Fruit salad....91 Solution: Number guessing game - level 0....91 Solution: Fruit salad....92 Boolean....93 if statement again....93 True and False....93 Boolean....94 True and False values in Python....94 Comparision operators....95 Do NOT Compare different types....95 Boolean operators....96 Boolean truth tables....97 Short circuit....97 Short circuit fixed....98 Incorrect use of conditions....98 Exercise: compare numbers....99 Exercise: compare strings....99 Solution: compare numbers....100 Solution: compare strings....100 Strings....102 Single quoted and double quoted strings....102 Long lines....102 Triple quoted strings (multiline)....103 String length (len)....104 String repetition and concatenation....104 A character in a string....105 String slice (instead of substr)....105 Change a string....105 How to change a string....106 String copy....106 String functions and methods (len, upper, lower)....107 index in string....107 index in string with range....107 rindex in string with range....108 find in string....108 Find all in the string....109 in string....109 index if in string....109 Encodings: ASCII, Windows-1255, Unicode....110 raw strings....110 ord....110 ord in a file....111 chr - number to character....111 Exercise: one string in another string....113 Exercise: to ASCII CLI....113 Exercise: from ASCII CLI....113 Solution: one string in another string....114 Solution: compare strings....114 Solution: to ASCII CLI....114 Solution: from ASCII CLI....115 Loops....116 Loops: for-in and while....116 for-in loop on strings....116 for-in loop on list....116 for-in loop on range....117 Iterable, iterator....117 for in loop with early end using break....117 for in loop skipping parts using continue....117 for in loop with break and continue....118 while loop....118 Infinite while loop....119 While with complex expression....119 While with break....119 While True....120 Duplicate input call....120 Eliminate duplicate input call....120 do while loop....120 while with many continue calls....121 Break out from multi-level loops....121 Exit vs return vs break and continue....121 Exercise: Print all the locations in a string....121 Exercise: Number guessing game....122 Exercise: MasterMind....123 Exercise: Count unique characters....123 Solution: Print all the locations in a string....123 Solution 1 for Number Guessing....124 Solution for Number Guessing (debug)....124 Solution for Number Guessing (move)....125 Solution for Number Guessing (multi-game)....126 Solution: MasterMind....127 Solution: Count unique characters....127 MasterMind to debug....128 PyCharm....130 PyCharm Intro....130 PyCharm Project....130 PyCharm Files....130 PyCharm - run code....130 PyCharm Python console at the bottom left....131 Refactoring example (with and without pycharm)....131 Formatted printing....132 format - sprintf....132 Examples using format - indexing....132 Examples using format with names....133 Format columns....133 Examples using format - alignment....133 Format - string....134 Format characters and types....134 Format floating point number....134 f-strings (formatted string literals)....135 printf using old %-syntax....135 Format braces, bracket, and parentheses....135 Examples using format with attributes of objects....136 raw f-strings....136 Lists....137 Anything can be a lists....137 Any layout....137 Lists....138 List slice with steps....138 Change a List....138 Change with steps....139 List assignment and list copy....139 join....140 join list of numbers....140 split....140 for loop on lists....140 in list....141 Where is the element in the list....141 Index improved....141 [].insert....142 [].append....142 [].remove....142 Remove element by index [].pop....142 Remove first element of list....143 Remove several elements of list by index....143 Use list as a queue....143 Queue using deque from collections....144 Fixed size queue....144 List as a stack....145 stack with deque....145 Exercies: Queue....145 Exercise: Stack....146 Solution: Queue with list....146 Solution: Queue with deque....147 Solution: Reverse Polish calculator (stack) with lists....147 Solution: Reverse Polish calculator (stack) with deque....148 Debugging Queue....149 sort....149 sort numbers....150 sort mixed....150 key sort....150 Sort tuples....151 sort with sorted....151 sort vs. sorted....151 key sort with sorted....152 Sorting characters of a string....152 range....152 Looping over index....152 Enumerate lists....153 List operators....153 List of lists....153 List assignment....153 List documentation....154 tuple....154 Exercise: color selector menu....155 Exercise: count digits....155 Exercise: Create list....155 Exercise: Count words....156 Exercise: Check if number is prime....156 Exercise: DNA sequencing....156 Solution: menu....156 Solution: count digits....157 Solution: Create list....158 Solution: Count words....158 Solution: Check if number is prime....159 Solution: DNA sequencing....159 Solution: DNA sequencing with filter....159 Solution: DNA sequencing with filter and lambda....160 [].extend....160 append vs. extend....160 split and extend....160 Files....162 Open and read file....162 Filename on the command line....162 Filehandle with and without....162 Filehandle with return....163 Read file remove newlines....163 Read all the lines into a list....163 Read all the characters into a string (slurp)....163 Not existing file....164 Open file exception handling....164 Open many files - exception handling....164 Writing to file....165 Append to file....165 Binary mode....165 Does file exist? Is it a file?....166 Exercise: count numbers....166 Exercise: strip newlines....166 Exercise: color selector....167 Exercise: ROT13....167 Exercise: Combine lists....167 Solution: count numbers....168 Solution: strip newlines....168 Solution: color selector....168 Solution: Combine lists....169 Read text file....169 Open and read file....169 Direct access of a line in a file....170 Example....170 Dictionary (hash)....172 What is a dictionary....172 When to use dictionaries....172 Dictionary....172 keys....172 Loop over keys....173 Loop using items....173 values....173 Not existing key....174 Get key....174 Does the key exist?....175 Does the value exist?....175 Delete key....175 List of dictionaries....176 Shared dictionary....176 immutable collection: tuple as dictionary key....177 immutable numbers: numbers as dictionary key....177 Sort dictionary by value....178 Sort dictionary keys by value....178 Insertion Order is kept....179 Change order of keys in dictionary - OrderedDict....179 Set order of keys in dictionary - OrderedDict....180 Exercise: count characters....180 Exercise: count words....181 Exercise: count words from a file....181 Exercise: Apache log....182 Exercise: Combine lists again....182 Exercise: counting DNA bases....182 Exercise: Count Amino Acids....183 Exercise: List of dictionaries....183 Exercise: Dictinoary of dictionaries....183 Solution: count characters....184 Solution: count characters with default dict....185 Solution: count words....185 Solution: count words in file....186 Solution: Apache log....187 Solution: Combine lists again....187 Solution: counting DNA bases....187 Solution: Count Amino Acids....188 Loop over dictionary keys....188 Do not change dictionary in loop....189 Default Dict....189 Sets....191 sets....191 set operations....191 set intersection....191 set subset....192 set symmetric difference....192 set union....193 set relative complement....193 set examples....194 defining an empty set....194 Adding an element to a set (add)....194 Merging one set into another set (update)....195 Functions (subroutines)....196 Defining simple function....196 Defining a function....196 Parameters can be named....196 Mixing positional and named parameters....197 Default values....198 Several defaults, using names....198 Arbitrary number of arguments *....198 Fixed parmeters before the others....199 Arbitrary key-value pairs in parameters **....200 Extra key-value pairs in parameters....200 Every parameter option....200 Duplicate declaration of functions (multiple signatures)....200 Recursive factorial....201 Recursive Fibonacci....201 Non-recursive Fibonacci....202 Unbound recursion....202 Variable assignment and change - Immutable....202 Variable assignment and change - Mutable....203 Parameter passing of functions....203 Passing references....203 Function documentation....204 Sum ARGV....204 Copy-paste code....205 Copy-paste code fixed....205 Copy-paste code further improvement....206 Palindrome....206 Exercise: statistics....207 Exercise: recursive....207 Exercise: Tower of Hanoi....207 Exercise: Merge and Bubble sort....207 Solution: statistics....207 Solution: recursive....208 Solution: Tower of Hanoi....209 Solution: Merge and Bubble sort....209 Modules....211 Before modules....211 Create modules....211 path to load modules from - The module search path....212 sys.path - the module search path....212 Flat project directory structure....212 Absolute path....212 Relative path....213 Python modules are compiled....213 How “import” and “from” work?....214 Runtime loading of modules....214 Conditional loading of modules....214 Duplicate importing of functions....214 Script or library....215 Script or library - import....215 Script or library - from import....216 assert to verify values....216 mycalc as a self testing module....217 doctest....217 Scope of import....218 Export import....218 Export import with all....219 import module....220 Execute at import time....220 Import multiple times....220 Exercise: Number guessing....221 Exercies: Scripts and modules....221 Exercise: Module my_sum....221 Exercise: Convert your script to module....222 Exercise: Add doctests to your own code....222 Solution: Module my_sum....222 Regular Expressions....224 What are Regular Expressions (aka. Regexes)?....224 What are Regular Expressions good for?....224 Examples....224 Where can I use it ?....225 grep....225 Regexes first match....225 Match numbers....226 Capture....226 Capture more....227 Capture even more....227 findall....227 findall with capture....228 findall with capture more than one....228 Any Character....228 Match dot....229 Character classes....229 Common characer classes....230 Negated character class....230 Optional character....230 Regex 0 or more quantifier....231 Quantifiers....231 Quantifiers limit....231 Quantifiers on character classes....231 Greedy quantifiers....232 Minimal quantifiers....232 Anchors....233 Anchors on both end....234 Match ISBN numbers....234 Matching a section....235 Matching a section - minimal....236 Matching a section negated character class....237 DOTALL S (single line)....237 MULTILINE M....237 Two regex with logical or....238 Alternatives....238 Grouping and Alternatives....239 Internal variables....239 More internal variables....239 Regex DNA....239 Regex IGNORECASE....240 Regex VERBOSE X....240 Substitution....241 findall capture....241 Fixing dates....242 Duplicate numbers....243 Remove spaces....243 Replace string in assembly code....243 Full example of previous....245 Split with regex....246 Exercises: Regexes part 1....247 Exercise: Regexes part 2....248 Exercise: Sort SNMP numbers....248 Exercise: parse hours log file and give report....249 Exercise: Parse ini file....250 Exercise: Replace Python....250 Exercise: Extract phone numbers....250 Solution: Sort SNMP numbers....251 Solution: parse hours log file and give report....252 Solution: Processing INI file manually....253 Solution: Processing config file....254 Solution: Extract phone numbers....254 Regular Expressions Cheat sheet....255 Fix bad JSON....255 Fix very bad JSON....256 Raw string or escape....257 Remove spaces regex....257 Regex Unicode....258 Anchors Other example....258 Python standard modules....260 Some Standard modules....260 sys....260 Writing to standard error (stderr)....261 Current directory (getcwd, pwd, chdir)....261 OS dir (mkdir, makedirs, remove, rmdir)....261 python which OS are we running on (os, platform)....261 Get process ID....262 OS path....262 Traverse directory tree - list directories recursively....262 os.path.join....263 Directory listing....263 expanduser - handle tilde ~....263 Listing specific files using glob....263 External command with system....263 subprocess....264 subprocess in the background....264 Accessing the system environment variables from Python....265 Set env and run command....265 shutil....265 time....266 sleep in Python....266 timer....266 Current date and time datetime now....267 Converting string to datetime....267 datetime arithmeticis....268 Rounding datetime object to nearest second....268 Signals and Python....268 Sending Signal....268 Catching Signal....269 Catching Ctrl-C on Unix....269 Catching Ctrl-C on Unix confirm....270 Alarm signal and timeouts....270 deep copy list....270 deep copy dictionary....272 Exercise: Catching Ctrl-C on Unix 2nd time....273 Exercise: Signals....273 Ctrl-z....273 JSON....274 JSON - JavaScript Object Notation....274 dumps....274 loads....275 dump....275 load....276 Round trip....276 Pretty print JSON....276 Sort keys in JSON....277 Set order of keys in JSON - OrderedDict....278 Exercise: Counter in JSON....278 Exercise: Phone book....279 Exercise: Processes....279 Solution: Counter in JSON....279 Solution: Phone book....280 Command line arguments with argparse....281 Modules to handle the command line....281 argparse....281 Basic usage of argparse....281 Positional argument....282 Many positional argument....283 Convert to integers....283 Convert to integer....284 Named arguments....284 Boolean Flags....285 Short names....285 Exercise: Command line parameters....286 Exercise: argparse positional and named....286 Exception handling....287 Hierarchy of calls....287 Handling errors as return values....287 Handling errors as exceptions....287 A simple exception....288 Working on a list....288 Catch ZeroDivisionError exception....289 Module to open files and calculate something....290 File for exception handling example....290 Open files - exception....291 Handle divide by zero exception....291 Handle files - exception....292 Catch all the exceptions and show their type....292 List exception types....293 Exceptions....294 How to raise an exception....294 Stack trace....294 Exercies: Exception int conversion....295 Exercies: Raise Exception....296 Solution: Exception int conversion (specific)....296 Solution: Exception int conversion (all other)....297 Solution: Raise Exception....297 Classes - OOP - Object Oriented Programming....298 Why Object Oriented Programming?....298 Generic Object Oriented Programming terms....298 OOP in Python....298 OOP in Python (numbers, strings, lists)....298 OOP in Python (argparse)....299 Create a class....299 Import module containing class....300 Import class from module....300 Initialize a class - constructor, attributes....300 Attributes are not special....300 Create Point class....301 Initialize a class - constructor, attributes....301 Methods....301 Stringify class....302 Inheritance....302 Inheritance - another level....303 Modes of method inheritance....303 Modes of method inheritance - implicit....304 Modes of method inheritance - override....304 Modes of method inheritance - extend....304 Modes of method inheritance - delegate - provide....305 Composition - Line....305 Some comments....306 Class in function....306 Serialization of instances with pickle....306 Quick Class definition and usage....307 Exercise: Add move_rad to based on radians....307 Exercise: Improve previous examples....307 Exercise: Polygon....307 Exercise: Number....308 Exercise: Library....308 Exercise: Bookexchange....308 Exercise: Represent turtle graphics....308 Solution - Polygon....309 PyPi - Python Package Index....310 What is PyPi?....310 Easy Install....310 pip....310 Upgrade pip....310 PYTHONPATH....310 Virtualenv....310 Virtualenv for Python 3....311 SQLite Database Access....312 SQLite....312 Connecting to SQLite database....312 Create TABLE in SQLite....312 INSERT data into SQLite database....312 SELECT data from SQLite database....313 A counter....314 MySQL....316 Install MySQL support....316 Create database user (manually)....316 Create database (manually)....316 Create table (manually)....316 Connect to MySQL....317 Connect to MySQL and Handle exception....317 Select data....318 Select more data....318 Select all data fetchall....318 Select some data fetchmany....319 Select some data WHERE clause....319 Select into dictionaries....320 Insert data....321 Update data....321 Delete data....321 Exercise MySQL....322 Exercise: MySQL Connection....322 Solution: MySQL Connection....322 PostgreSQL....324 PostgreSQL install....324 Python and Postgresql....324 PostgreSQL connect....324 INSERT....324 INSERT (from command line)....325 SELECT....325 DELETE....325 SQLAlchemy....327 SQLAlchemy hierarchy....327 SQLAlchemy engine....327 SQLAlchemy autocommit....327 SQLAlchemy engine CREATE TABLE....327 SQLAlchemy engine INSERT....328 SQLAlchemy engine SELECT....328 SQLAlchemy engine SELECT all....328 SQLAlchemy engine SELECT fetchall....329 SQLAlchemy engine SELECT aggregate....329 SQLAlchemy engine SELECT IN....329 SQLAlchemy engine SELECT IN with placeholders....329 SQLAlchemy engine connection....330 SQLAlchemy engine transaction....330 SQLAlchemy engine using context managers....331 Exercise: Create table....331 SQLAlchemy Metada....332 SQLAlchemy types....333 SQLAlchemy ORM - Object Relational Mapping....333 SQLAlchemy ORM create....333 SQLAlchemy ORM schema....334 SQLAlchemy ORM reflection....335 SQLAlchemy ORM INSERT after automap....335 SQLAlchemy ORM INSERT....336 SQLAlchemy ORM SELECT....336 SQLAlchemy ORM SELECT cross tables....336 SQLAlchemy ORM SELECT and INSERT....337 SQLAlchemy ORM UPDATE....337 SQLAlchemy ORM logging....338 Solution: Create table....338 Exercise: Inspector....339 SQLAlchemy CREATE and DROP....339 SQLAlchemy Notes....340 SQLAlchemy Meta SQLite CREATE....340 SQLAlchemy Meta Reflection....341 SQLAlchemy Meta INSERT....341 SQLAlchemy Meta SELECT....341 NoSQL....342 Types of NoSQL databases....342 MongoDB....343 MongoDB CRUD....343 Install MongoDB support....343 Python MongoDB insert....343 MongoDB CLI....343 Python MongoDB find....344 Python MongoDB find refine....344 Python MongoDB update....344 Python MongoDB remove (delete)....345 Redis....346 Redis CLI....346 Redis list keys....346 Redis set get....346 Redis incr....347 Redis incrby....347 Redis setex....347 Web client....348 urllib the web client....348 urllib2 the web client....348 httpbin.org....348 requests get....349 Download image using requests....349 Download image as a stream using requests....349 Download zip file....349 Extract zip file....350 Interactive Requests....350 requests get JSON....350 requests get JSON UserAgent....350 requests get JSON UserAgent....350 requests get header....351 requests change header....351 requests post....351 Tweet....352 API config file....352 bit.ly....352 Exercise: Combine web server and client....353 Python Web server....354 Hello world web....354 Dump web environment info....354 Web echo....355 Web form....355 Resources....356 Python Flask....357 Python Flask intro....357 Python Flask installation....357 Flask: Hello World....357 Flask hello world + test....357 Flask generated page - time....358 Flask: Echo GET....358 Flask: Echo POST....359 Flask: templates....360 Flask: templates....361 Flask: templates with parameters....361 Flask: runner....362 Exercise: Flask calculator....362 Static files....362 Flask Logging....363 Flask: Counter....363 Color selector without session....363 Session management....364 Flask custom 404 page....364 Flask Error page....365 Flask URL routing....366 Flask Path params....366 Flask Path params (int)....366 Flask Path params add (int)....367 Flask Path params add (path)....367 Jinja loop, conditional, include....368 Exercise: Flask persistent....369 Exercise: Flask persistent....369 Flask Exercises....369 Flask login....369 Flask JSON API....370 Flask and AJAX....371 Flask and AJAX....373 passlib....374 Flask Testing....375 Flask Deploy app....375 Flask Simple Authentication + test....376 Flask REST API....377 Flask REST API - Echo....377 Flask REST API - parameters in path....378 Flask REST API - parameter parsing....378 Flask REST API - parameter parsing - required....379 Networking....381 Secure shell....381 ssh....381 ssh from Windows....381 Parallel ssh....382 telnet....382 prompt for password....382 Python nmap....383 ftp....383 Interactive shell....385 The Python interactive shell....385 REPL - Read Evaluate Print Loop....385 Using Modules....386 Getting help....386 Exercise: Interactive shell....387 Testing Demo....388 How do you test your code?....388 What is testing?....388 What is testing really?....388 Testing demo - AUT - Application Under Test....388 Testing demo - use the module....388 Testing demo: doctets....389 Testing demo: Unittest success....389 Testing demo: Unittest failure....390 Testing demo: pytest using classes....390 Testing demo: pytest without classes....391 Testing demo: pytest run doctests....392 Testing demo: pytest run unittest....392 Exercise: Testing demo....392 Solution: Testing demo....392 Types in Python....394 mypy....394 Types of variables....394 Types of function parameters....394 Types used properly....395 TODO: mypy....395 Testing Intro....396 The software testing equasion....396 The software testing equasion (fixed)....396 The pieces of your software?....396 Manual testing....396 What to tests?....397 Continuous Integration....397 Functional programming....398 Functional programming....398 Iterators (Iterables)....398 range....399 range with list....400 range vs. list size....401 for loop with transformation....401 map....402 map delaying function call....404 map on many values....405 map with list....406 double with lambda....406 What is lambda in Python?....407 lambda returning tuple....408 map returning tuples....408 lambda with two parameters....408 map for more than one iterable....409 map on uneven lists....409 replace None (for Python 2)....409 map on uneven lists - fixed (for Python 2)....410 map mixed iterators....410 map fetch value from dict....410 Exercise: string to length....411 Exercise: row to length....411 Exercise: compare rows....411 Solution: string to length....411 Solution: row to length....411 Solution: compare rows....411 filter....412 filter with lambda....412 filter - map example....413 filter - map in one expression....413 Get indexes of values....414 reduce....415 reduce with default....416 zip....416 Creating dictionary from two lists using zip....416 all, any....417 Compare elements of list with scalar....417 List comprehension - double....417 List comprehension - simple expression....418 List generator....418 List comprehension....419 Dict comprehension....419 Lookup table with lambda....419 Read lines without newlines....420 Read key-value pairs....420 Create index-to-value mapping in a dictionary based on a list of values....421 Exercise: min, max, factorial....421 Exercise: Prime numbers....421 Exercise: Many validator functions....421 Exercise: Calculator using lookup table....421 Exercise: parse file....422 Solution: min, max, factorial....423 Solution: Prime numbers....423 Solution: Many validator functions....423 Solution: Calculator using lookup table....424 map with condtion....424 map with lambda....424 map with lambda with condition....425 List comprehension - complex....425 Iterators - with and without Itertools....426 Advantages of iterators and generators....426 The Fibonacci research institute....426 Fibonacci plain....426 Fibonacci copy-paste....426 Iterators Glossary....427 What are iterators and iterables?....427 A file-handle is an iterator....428 range is iterable but it is not an iterator....428 Iterator: a counter....429 Using iterator....430 Iterator without temporary variable....430 The type of the iterator....430 Using iterator with next....431 Mixing for and next....431 Iterable which is not an iterator....432 Iterator returning multiple values....432 Range-like iterator....433 Unbound or infinite iterator....433 Unbound iterator Fibonacci....434 Operations on Unbound iterator....435 itertools....436 itertools - count....436 itertools - cycle....436 Exercise: iterators - reimplement the range function....437 Exercise: iterators - cycle....437 Exercise: iterators - alter....437 Exercise: iterators - limit Fibonacci....437 Exercise: iterators - Fibonacci less memory....437 Exercise: read char....437 Exercise: read section....438 Exercise: collect packets....438 Exercise: compare files....439 Solution: iterators - limit Fibonacci....440 Solution: iterators - Fibonacci less memory....440 Solution: read section....441 Solution: compare files....441 Solution: collect packets....442 Generators and Generator Expressions....444 Generators Glossary....444 Iterators vs Generators....444 List comprehension and Generator Expression....444 List comprehension vs Generator Expression - less memory....445 List comprehension vs Generator Expression - lazy evaluation....446 Generator: function with yield - call next....447 Generators - call next....448 Generator with yield....449 Generators - fixed counter....449 Generators - counter....449 Generators - counter with parameter....450 Generators - my_range....450 Fibonacci - generator....451 Infinite series....452 Integers....452 Integers + 3....452 Integers + Integers....453 Filtered Fibonacci....453 The series.py....454 generator - unbound count (with yield)....454 iterator - cycle....455 Exercise: Alternator....455 Exercise: Prime number generator....456 Exercise: generator....456 Exercise: Tower of Hanoi....456 Exercise: Binary file reader....456 Exercise: File reader with records....456 Logging....458 Simple logging....458 Simple logging - set level....458 Simple logging to a file....458 Simple logging format....459 Simple logging change date format....459 getLogger....459 Time-based logrotation....460 Size-based logrotation....460 Closures....461 Counter local - not working....461 Counter with global....461 Create incrementors....462 Create internal function....462 Create function by a function....463 Create function with parameters....463 Counter closure....464 Make incrementor with def (closure)....464 Make incrementor with lambda....465 Exercise: closure bank....465 Solution: closure bank....466 Solution: counter with parameter....467 Decorators....469 Function assignment....469 Function inside other function....469 Decorator....470 Use cases for decorators in Python....470 A recursive Fibonacci....471 trace fibo....471 tron decorator....472 Decorate with direct call....472 Decorate with parameter....472 Decorator accepting parameter....472 Decorate function with any signature....473 Decorate function with any signature - implementation....473 Exercise: Logger decorator....474 Exercise: memoize decorator....474 Solution: Logger decorator....474 Solution: Logger decorator (testing)....475 Solution memoize decorator....475 Context managers (with statement)....478 Why use context managers?....478 Context Manager examples....479 cd in a function....479 open in function....481 open in for loop....482 open in function using with....482 Plain context manager....483 Param context manager....484 Context manager that returns a value....484 Use my tempdir - return....485 Use my tempdir - exception....486 cwd context manager....486 tempdir context manager....487 Context manager with class....488 Context managers with class....489 Context manager: with for file....490 With - context managers....490 Exercise: Context manager....490 Exercise: Tempdir on Windows....491 Solution: Context manager....491 Advanced lists....493 Change list while looping: endless list....493 Change list while looping....493 Copy list before iteration....494 for with flag....494 for else....494 enumerate....495 do while....495 list slice is copy....496 Advanced Exception handling....497 Exceptions else....497 Exceptions finally....497 Exit and finally....498 Catching exceptions....499 Home made exception....499 Home made exception with attributes....500 Home made exception hierarcy....500 Home made exception hierarcy - 1....501 Home made exception hierarcy - 2....501 Home made exception hierarcy - 3....502 Exercise: spacefight with exceptions....502 Exercies: Raise My Exception....503 Solution: spacefight with exceptions....504 Solution: Raise My Exception....505 Exception finally return....506 Warnings....507 Warnings....507 CSV....508 Reading CSV the naive way....508 CSV with quotes and newlines....508 Reading a CSV file....509 CSV dialects....509 CSV to dictionary....510 Exercise: CSV....511 Solution: CSV....511 Excel....512 Spreadsheets....512 Python Excel....512 Create an Excel file from scratch....512 Worksheets in Excel....513 Add expressions to Excel....513 Format field....513 Number series and chart....514 Read Excel file....515 Update Excel file....515 Exercise: Excel....516 XML....517 XML Data....517 Expat - Callbacks....517 XML DOM - Document Object Model....518 XML SAX - Simple API for XML....518 SAX collect....519 XML elementtree....520 SciPy - for Scientific Computing in Python....522 Data Science tools in Python....522 Data Analysis resources....523 Python and Biology....524 Biopython....524 Biopython background....524 Bio python sequences....524 Download data....524 Read FASTA, GenBank files....525 Search nucleotids....525 Download nucleotids....526 Exercise: Nucleotid....526 Biology background....526 Chemistry....528 Chemistry links....528 Bond length....528 Covalent radius....528 Python energy landscape explorer....529 Other chemistry links....529 numpy....530 What is NumPy....530 Numpy - vector....530 NumPy 2D arrays....530 Numpy - set type....531 NumPy arrays: ones and zeros....531 Numpy: eye....531 NumPy array random....532 NumPy Random integers....532 NumPy array type change by division (int to float)....533 Numpy: Array methods: transpose....533 Numpy: reference, not copy....534 Numpy: copy array....534 Numpy: Elementwise Operations on Arrays....534 Numpy: multiply, matmul, dot for vectors....535 Numpy: multiply, matmul, dot for vector and matrix....535 Numpy: multiply, matmul, dot for matrices....536 Numpy: casting - converting from strings to integer.....537 Numpy: indexing 1d array....538 Numpy: slice is a reference....538 Numpy: slice - copy....538 Numpy: abs value on a Numpy array....538 Numpy: Logical not on a Numpy array....539 Numpy: Vectorize a function....540 Numpy: Vectorize len....540 Numpy: Vectorize lambda....541 Numpy: Filtering array....541 Numpy: Filter matrix values....541 Numpy: Filter matrix rows....542 Numpy: Stat....543 Numpy: Serialization....543 Numpy: Load from Matlab file....543 Numpy: Save as Matlab file....544 Numpy: Horizontal stack vectors (hstack)....544 Numpy: Append or vertically stack vectors and matrices (vstack)....545 Numpy uint8....545 Numpy int8....546 Pandas....547 Pandas....547 Planets....547 Pandas Planets - Dataframes....547 Pandas Stocks....548 Pandas Stocks....548 Merge Dataframes....549 Analyze Alerts....549 Analyze IFMetrics....549 Create Excel file for experiment with random data....550 Calculate Genome metrics....550 Calculate Genome metrics - add columns....551 Calculate Genome metrics - vectorized....552 Calculate Genome metrics - vectorized numpy....552 Genes using Jupyter....553 Combine columns....553 Pandas more....553 Pandas Series....554 Pandas Series with names....555 Matplotlib....556 About Matplotlib....556 Matplotlib Line....556 Matplotlib Line with dates....556 Matplotlib Simple Pie....557 Matplotlib Simple Pie with params....558 Matplotlib Pie....558 Matplotlib Pie 2....559 Plot, scatter, histogram....560 Seaborn....561 Searborn use examples....561 Seaborn tip....561 Seaborn Anscombes Quartet....562 Jupyter notebooks....563 Jupyter on Windows....563 Jupyter on Linux and OSX....563 Jupyter add....563 Planets....563 Jupyter notebook Planets....565 Jupyter StackOverflow....565 Jupyter StackOverflow - selected columns....566 Jupyter processing chunks....566 Jupyter StackOverflow - selected rows....566 Jupyter StackOverflow - biggest countries (in terms of number of responses)....566 Jupyter StackOverflow - historgram....567 Jupyter StackOverflow - filter by country....567 Jupyter StackOverflow - OpenSourcer....567 Jupyter StackOverflow - cross tabulation....567 Jupyter StackOverflow - salaries....568 Jupyter StackOverflow - replace values....568 Jupyter StackOverflow - selected rows....568 Jupyter notebook Intellisense (TAB completition)....569 Jupyter examples....569 IPy Widgets....569 Testing....570 Traditional Organizations....570 Quality Assurance....570 Web age Organizations....570 TDD vs Testing as an Afterthought....571 Why test?....571 Testing Modes....571 Testing Applications....572 Testing What to test?....572 Testing in Python....572 Testing Environment....572 Testing Setup - Fixture....573 Testing Resources....573 Testing with unittest....574 Use a module....574 Test a module....574 The tested module....575 Testing - skeleton....575 Testing....576 Test examples....576 Testing with PyTest....578 Pytest features....578 Pytest setup....578 Testing with Pytest....578 Testing functions....579 Testing class and methods....579 Pytest - execute....579 Pytest - execute....579 Pytest simple module to be tested....580 Pytest simple tests - success....580 Pytest simple tests - success output....580 Pytest simple tests - failure....580 Pytest simple tests - failure output....581 Exercise: test math functions....581 Exercise: test this app....581 Exercise: test the csv module....582 Solution: Pytest test math functions....582 Solution: Pytest test this app....582 Solution: test the csv module....583 PyTest bank deposit....583 PyTest expected exceptions (bank deposit)....583 PyTest expected exceptions (bank deposit) - no exception happens....584 PyTest expected exceptions (bank deposit) - different exception is raised....584 PyTest expected exceptions....585 PyTest expected exceptions output....585 PyTest expected exceptions (text changed)....585 PyTest expected exceptions (text changed) output....585 PyTest expected exceptions (other exception)....585 PyTest expected exceptions (other exception) output....586 PyTest expected exceptions (no exception)....586 PyTest expected exceptions (no exception) output....586 PyTest: Multiple Failures....586 PyTest: Multiple Failures output....587 PyTest Selective running of test functions....587 PyTest: stop on first failure....587 Pytest: expect a test to fail (xfail or TODO tests)....587 Pytest: expect a test to fail (xfail or TODO tests)....588 PyTest: show xfailed tests with -rx....588 Pytest: skipping tests....588 Pytest: show skipped tests woth -rs....589 Pytest: show extra test summmary info with -r....589 Pytest: skipping tests output in verbose mode....590 Pytest verbose mode....590 Pytest quiet mode....590 PyTest print STDOUT and STDERR using -s....590 PyTest failure reports....591 PyTest compare numbers....591 PyTest compare numbers relatively....591 PyTest compare strings....592 PyTest compare long strings....592 PyTest is one string in another strings....592 PyTest test any expression....593 PyTest element in list....593 PyTest compare lists....593 PyTest compare short lists....594 PyTest compare short lists - verbose output....594 PyTest compare dictionaries....594 PyTest compare dictionaries output....595 PyTest Fixtures....595 PyTest Fixture setup and teardown....595 PyTest Fixture setup and teardown output....596 PyTest: Class setup and teardown....596 PyTest: Class setup and teardown output....597 Pytest Dependency injection....597 Pytest fixture - tmpdir....597 Pytest capture STDOUT and STDERR with capsys....598 Pytest Fixture - home made fixtures....598 More fixtures....599 Pytest: Mocking - why?....601 Pytest: Mocking - what?....601 Pytest: One dimensional spacefight....601 Pytest: Mocking input and output....602 Pytest: Mocking random....602 Pytest: Flask echo....603 Pytest: testing Flask echo....603 PyTest: Run tests in parallel with xdist....604 PyTest: Order of tests....604 PyTest: Randomize Order of tests....604 PyTest: Force default order....604 PyTest: no random order....605 Anagram on the command line....605 PyTest testing CLI....605 PyTest test discovery....606 PyTest test discovery - ignore some tests....606 PyTest select tests by name....607 PyTest select tests by marker....607 PyTest: Test Coverage....608 Exercise: module....608 Exercise: Open Source....609 Pytest resources....609 Pytest and tempdir....609 PyTest compare short lists - output....610 PyTest with parameter....610 PyTest with parameters....611 Pytest reporting in JUnit XML format....611 No test selected....612 Advancted functions....613 Variable scopes....613 Name resolution order (LEGB)....613 Scoping: global seen from fuction....613 Assignment creates local scope....613 Local scope gone wrong....614 Changing global variable from a function....614 Global variables mutable in functions....615 Scoping issues....615 sub in sub....616 Scoping sub in sub (enclosing scope)....616 Function objects....616 Functions are created at run time....617 Mutable default....618 Use None as default parameter....618 Inner function created every time the outer function runs....619 Static variable....620 Static variable in generated function....620 Inspect....621 Variable number of function arguments....623 Python function arguments - a reminder....623 Functions with unknown number of argumerns....623 Variable length argument list with * and **....623 Passing arguments as they were received (but incorrectly)....624 Unpacking args before passing them on....624 Exercise: implement the my_sum function....625 Solution: implement the my_sum function....625 Exercise: implement the reduce function....625 Soluton: implement the reduce function....626 Exercise: sort pairs....626 Solution: sort pairs....626 Python Packages....627 Why Create package....627 Create package....627 Internal usage....627 use module in package - relative path....628 use package (does not work)....628 package importing (and exporting) module....628 use package (module) with import....628 use package with import....629 Creating an installable Python package....629 Create tar.gz file....630 Install Package....630 Dependencies....630 Add README file....631 Add README file (setup.py)....631 Include executables....632 Add tests....632 Add tests calc....633 Add tests all....633 setup.py....633 Run tests and create package....634 Packaging applications (creating executable binaries)....634 Using PyInstaller....634 Other PyInstaller examples....634 Other....634 Py2app for Mac....635 Exercise: package....635 Exercise: create executable....635 Ctypes....636 ctypes - hello....636 concat....637 links....638 Advanced OOP....639 Class count instances....639 Class Attributes....639 Class Attributes in Instances....639 Attributes with method access....640 Instance Attribute....640 Methods are class attributes....641 Monkey patching....642 Classes: instance method....642 Class methods and class attributes....643 Classes: constructor....644 Class methods - alternative constructor....644 Abstract Base Class....645 Abstract Base Class with abc....646 ABC working example....647 ABC - cannot instantiate the base-class....647 ABC - must implement methods....647 Use Python @propery to fix bad interface (the bad interface)....647 Use Python @propery to fix bad interface (first attempt)....648 Use Python @propery to fix bad API....648 Use Python @propery decorator to fix bad API....649 Use Python @propery for value validation....649 class and static methods....651 Destructor: del....652 Destructor delayed....652 Destructor delayed for both....653 Opearator overloading....653 Operator overloading methods....654 Exercise: rectangular....654 Exercise: SNMP numbers....654 Exercise: Implement a Gene inheritance model combining DNA....655 Exercise: imaginary numbers - complex numbers....655 Solution: Rectangular....656 Solution: Implement a Gene inheritance model combining DNA....657 Instance counter....658 2to3....659 Convertig from Python 2 to Python 3....659 division....659 print in Python 2....659 print in Python 3....659 input and raw_input....660 Code that works on both 2 and 3....660 Compare different types....660 Octal numbers....660 2to3 Resources....660 Design Patterns....662 What are Design Patterns?....662 Don’t replace built-in objects....662 Facade - simple interface to complex system....662 Monkey Patching....663 Creation DPs “Just One”....663 Singleton....664 Monostate (Borg)....664 Dispatch table....664 Parallel....666 Types of Problems....666 Types of solutions....666 How many parallels to use?....666 Dividing jobs....667 Performance Monitoring....667 Threads....668 Python Threading docs....668 Threaded counters....668 Simple threaded counters....669 Simple threaded counters (parameterized)....670 Pass parameters to threads - Counter with attributes....671 Create a central counter....672 Lock - acquire - release....672 Counter - plain....673 GIL - Global Interpreter Lock....674 Thread load....674 Exercise: thread files....675 Exercise: thread URL requests.....675 Exercise: thread queue....677 Solution: thread queue....678 Solution: thread URL requests.....678 Forking....680 Fork....680 Forking....680 Fork skeleton....681 Fork with load....682 Fork load results....682 Marshalling / Serialization....683 Fork with random....683 Exercise: fork return data....684 Solution: fork return data....684 Asyncronus programming with AsyncIO....686 Sync chores....686 Async chores....687 Explanation....689 Coroutines....689 More about asyncio....690 Async files....690 Asynchronus programming with Twisted....691 About Twisted....691 Echo....691 Echo with log....692 Simple web client....692 Web client....693 Multiprocess....695 Multiprocess CPU count....695 Multiprocess Process....695 Multiprocess N files: Pool....695 Multiprocess load....696 Multiprocess: Pool....697 Multiprocess load async....697 Multiprocess and logging....698 Exercise: Process N files in parallel....699 Exercise: Process N Excel files in parallel....699 Exercise: Fetch URLs in parallel....699 Exercise: Fetch URLs from one site.....701 Solution: Fetch URLs in parallel....702 Multitasking....704 What is Multitasking?....704 Multitasking example....704 Multitasking example with wait....704 Multitaksing - second loop waits for first one....705 Multitasking counter....706 Multitasking counter with thread locking....706 Improving Performance - Optimizing code....708 Problems....708 Optimization strategy....708 Locate the source of the problem....708 Optimizing tactics....708 DSU: Decorate Sort Undecorate....708 Profile code....709 Slow example....709 profile slow code....710 cProfile slow code....710 Benchmarking....711 Benchmarking subs....711 Levenshtein distance....712 Generate words....712 Levenshtein - pylev....713 Levenshtein - edittidtance....713 Editdistance benchmark....713 A Tool to Generate text files....713 Count characters....714 Memory leak....716 Garbage collection....716 Weak reference....716 Exercise: benchmark list-comprehension, map, for....717 Exercise: Benchmark Levenshtein....717 Exercise: sort files....717 Exercise: compare split words:....718 Exercise: count words....718 GUI with Python/Tk....719 Sample Tk app....719 GUI Toolkits....721 Installation....722 Python Tk Documentation....722 Python Tk Button....723 Python Tk Button with action....723 Python Tk Label....723 Python Tk Label - font size and color....724 Python Tk Keybinding....724 Python Tk Entry (one-line text entry)....725 Python Tk Entry for passwords and other secrets (hidden text)....725 Python Tk Checkbox....725 Python Tk Radiobutton....726 Python Tk Listbox....726 Python Tk Listbox Multiple....727 Python Tk Menubar....727 Python Tk Text....728 Python Tk Dialogs....729 Python Tk Filedialog....729 Python Tk messagebox....730 Python Tk Combobox....731 Python Tk OptionMenu....732 Python Tk Scale....732 Python Tk Progressbar....733 Python Tk Frame....733 Not so Simple Tk app with class....734 Tk: Hello World....734 Tk: Quit button....735 Tk: File selector....735 Tk: Checkbox....736 Tk: Runner....737 Tk: Runner with threads....738 Getting started with Tk....740 Exercise: Tk - Calculator one line....742 Exercise: Tk Shopping list....742 Exercise: Tk TODO list....742 Exercise: Tk Notepad....742 Exercise: Tk Copy files....742 Exercise: Tk....743 Solution: Tk - Calculator one line....743 Solution: Tk....744 Solution: Tk Notepad....745 Simple file dialog....746 Python Pitfalls....747 Reuse of existing module name....747 Use the same name more than once....747 Compare string and number....748 Compare different types....749 Sort mixed data....749 Linters....751 Static Code Analyzis - Linters....751 PEP8....751 F811 - redefinition of unused....751 Warn when Redefining functions....751 Python .NET....752 IronPython....752 Use .NET libraries from Python....752 Python and .NET console....753 Python and .NET examples....753 Exercise Python and .NET....755 Python and Java....756 Jython....756 Calling Java from Python....756 Jython - Python running on the JVM....757 Jython Installation....757 Jython Installation....757 Jython load Java class....757 Jython load Java class in code....757 Jython test Java class....758 PIL - Pillow....759 Install Pillow....759 Create First Image....759 Write Text on Image....759 Select font for Text on Image....759 Font directories....760 Get size of an Image....760 Get size of text....760 Resize an existing Image....760 Crop an existing Image....761 Combine two images....761 Rotated text....761 Rotated text in top-right corner....761 Embed image (put one image on another one)....762 Draw a triangle....762 Draw a triangle and write text in it....763 Draw a triangle and write rotated text in it....763 Draw a rectangular....763 Draw a rectangle....764 Draw circle....764 Draw heart....764 Rectangle with rounded corners....765 TODO....765 FAQ....766 How not to name example scirpts?....766 Platform independent code....766 How to profile a python code to find causes of slowness?....767 pdb = Python Debugger....767 Avoid Redefining functions....768 Appendix....769 print_function....769 Dividers (no break or continue)....769 Lambdas....769 Abstract Class....769 Remove file....770 Modules: more....770 import hooks....771 Python resources....771 Progress bar....771 from future....771 Variable scope....772 scope....773 type....774 Look deeper in a list....775 Exercise: iterators - count....775 Simple function (before generators)....775 Other slides....777 Other slides....777 Atom for Python....777 IDLE - Integrated DeveLopment Environment....777 sh-bang - executable on Linux/Apple....778 Strings as Comments....778 pydoc....778 How can I check if a string can be converted to a number?....779 Spyder Intro....779 Interactive Debugging....779 Parameter passing....780 Command line arguments and main....780 Infinite loop....780 break....780 continue....781 While with many conditions....781 while loop with many conditions....782 Format with conversion (stringifiation with str or repr)....782 Name of the current function in Python....783 Name of the caller function in Python....783 Stack trace in Python using inspect....783 Module Fibonacci....784 PyTest - assertion....785 PyTest - failure....785 PyTest - list....786 SAX with coroutine....786 Getting the class name of an object....787 Inheritance - super....788 Inheritance - super - other class....788 iterator - pairwise....789 iterator - grouped....789 itertools - groupby....790 Circular references....790 Context managers: with (file) experiments....791 itertools - izip....791 mixing iterators....791 mixing iterators....792 itertools - pairwise....792 itertools - grouped....792 range vs xrange in Python....793 profile (with hotshot) slow code....793 Abstract Base Class without abc....794 Abstract Base Class with abc Python 2 ?....795 Abstract Base Class with metaclass....795 Create class with metaclass....796 Python Descriptors....798 alter iterator....798 Create a counter queue....799 A Queue of tasks....799 Filtered Fibonacci with ifilter....800 Python from .NET....800