Standard Library
================

.. note:: Reference document for this section:

 * The Python Standard Library documentation: 
   http://docs.python.org/library/index.html

 * Python Essential Reference, David Beazley, Addison-Wesley Professional
 
.. contents::
   :depth: 2
   
    
Download this page as:
    
    - :download:`a commented Python script <standard_library.py>`
    - :download:`a minimal Python script <standard_library_bare.py>`

 

``os``: operating system functionality
-----------------------------------------------

*"A portable way of using operating system dependent functionality."*

Directory and file manipulation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Current directory:
    
.. ipython::    
    
    In [1]: import os
    
    In [1]: os.getcwd()


List a directory:
    
.. ipython::
    
    In [1]: os.listdir(os.curdir)


Make a directory:
    
.. ipython::
    
    In [1]: os.mkdir('junkdir')
    
    In [1]: 'junkdir' in os.listdir(os.curdir)

Rename the directory:
    
.. ipython::

    In [1]: os.rename('junkdir', 'foodir')

    In [1]: 'junkdir' in os.listdir(os.curdir)

    In [1]: 'foodir' in os.listdir(os.curdir)
    
    In [1]: os.rmdir('foodir')

    In [1]: 'foodir' in os.listdir(os.curdir)
    

Delete a file:
 
.. ipython:: 
    
    In [1]: fp = open('junk.txt', 'w')    # first create an empty file
    
    In [1]: fp.close()

    In [1]: 'junk.txt' in os.listdir(os.curdir)

    In [1]: os.remove('junk.txt')

    In [1]: 'junk.txt' in os.listdir(os.curdir)
    

``os.path``: path manipulations
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

``os.path`` provides common operations on pathnames:
    
.. ipython::
    
    In [1]: fp = open('junk.txt', 'w')

    In [1]: fp.close()

    In [1]: a = os.path.abspath('junk.txt')

    In [1]: a

    In [1]: os.path.split(a)

    In [1]: os.path.dirname(a)

    In [1]: os.path.basename(a)

    In [1]: os.path.splitext(os.path.basename(a))

    In [1]: os.path.exists('junk.txt')

    In [1]: os.path.isfile('junk.txt')

    In [1]: os.path.isdir('junk.txt')

    In [1]: os.path.expanduser('~/local')

    In [1]: os.path.join(os.path.expanduser('~'), 'local', 'bin')
    
    In [1]: os.remove('junk.txt')
    

``subprocess``: running an external command
-------------------------------------------

Call a simple command, wait for it to finish, and get the return code:
    
.. ipython::
  
  In [1]: import subprocess
  
  In [1]: subprocess.call('chmod +x rst2py.py', shell=True)
  

Get the output:
    
.. ipython::
    
  In [1]: output = subprocess.check_output(['ls','-all','-h'])
  
  In [1]: print(output)
  
  
Communicate with the process (try for example with :download:`some_program.f <some_program.f>`, and compile it with ``gfortran some_program.f``). Though you can download and compile the program
in a terminal, you can just as well stay within Python:
                                  
.. ipython::
    
    In [1]: import urllib
    
    In [1]: url = urllib.FancyURLopener()
    
    In [1]: link = 'http://www.ster.kuleuven.be/~pieterd/some_program.f'
    
    In [1]: myfile,msg = url.retrieve(link, filename='some_program.f')
    
    In [1]: url.close()
    
    In [1]: output = subprocess.check_call('gfortran some_program.f',shell=True)
    
    In [1]: print(output) # output flag of zero means successful
    
Now run that program and read the first line from the standard output.
   
.. ipython::
    
    In [1]: p1 = subprocess.Popen('./a.out',stdout=subprocess.PIPE)
    
    In [1]: print(p1.stdout.readline())
    
You can also stop, continue and kill the program after importing the standard
module ``signal``:
   
.. ipython::   
   
    In [1]: import signal
    
    In [1]: p1.send_signal(signal.SIGSTOP) # temporarily halt the executing of a program
    
    In [1]: p1.send_signal(signal.SIGCONT) # continue the program
    
    In [1]: p1.send_signal(signal.SIGKILL) # kill the program

And finally we remove the compiled program again:

.. ipython::
    
    In [1]: os.remove('a.out')

.. raw:: html

   <p class="flip0">Click to Show/Hide: How to communicate with a program during execution</p> <div class="panel0">

Suppose we want to run a program, and check its output while it's running. For
this, we need to read the program's standard output while it is running,
wait for the next line to appear, and end the loop when the output stream is closed.
This can be done with:

.. sourcecode:: python

    def line_at_a_time(fileobj):
        while True:
            line = fileobj.readline()
            if not line:
                return
            yield line
            
Now, we can run the program and check the output. Suppose "myprogram" prints
``ERROR`` to the screen when it encountered an error, and we want to kill the
program whenever that occurs::
    
    >>> p1 = subprocess.Popen('./my_program',stdout=subprocess.PIPE)
    >>> for line in line_at_a_time(p1.stdout):
            if "ERROR" in line:
                p1.send_signal(signal.SIGKILL)
                
Similarly, you can use ``subprocess.PIPE`` to send data to ``stdin``.

.. admonition:: Exercise: run ``some_program.f`` and communicate with it 

   Run :download:`some_program.f <some_program.f>`, and stop the program for
   10 seconds when the number ``4`` is reached. Kill the program when the
   number ``11`` is reached.

.. raw:: html

   <p class="flip9">Click to show/hide solution</p> <div class="panel9">

.. sourcecode:: python
    
    import subprocess
    import signal
    import time

    def line_at_a_time(fileobj):
        while True:
            line = fileobj.readline()
            if not line:
                return
            yield line
        
    p1 = subprocess.Popen('./a.out',stdout=subprocess.PIPE)
    for line in line_at_a_time(p1.stdout):
        number = float(line.split()[0])
        if number==4.:
            print("Reached 4: stop 10 secs, then continue")
            p1.send_signal(signal.SIGSTOP)
            time.sleep(10)
            p1.send_signal(signal.SIGCONT)
        elif number==11:
            print("Reached 11: kill!")
            p1.send_signal(signal.SIGKILL)
        else:
            print("Got: {:s}".format(line.strip()))
    

.. raw:: html

   </div>        



.. raw:: html

   </div>        
   
   
   
   

``shutil``: high-level file operations
---------------------------------------

The ``shutil`` provides useful file operations:

    * ``shutil.rmtree``: Recursively delete a directory tree.
    * ``shutil.move``: Recursively move a file or directory to another location.
    * ``shutil.copy``: Copy files or directories.

``glob``: Pattern matching on files
-------------------------------------

The ``glob`` module provides convenient file pattern matching.

Find all files ending in ``.txt``:

.. ipython::
    
    In [1]: from glob import glob

    In [1]: glob('first_steps/*.rst')

    In [1]: glob('first_steps/[ap]*.rst')

Note that by default the results aren't sorted. If you want the output
list sorted, use:

.. ipython::
    
    In [1]: sorted(glob('[hn]*.txt'))


Environment variables:
~~~~~~~~~~~~~~~~~~~~~~

.. ipython::

    In [1]: import os

    In [1]: list(os.environ.keys())[:2]

    In [1]: os.environ['PYTHONPATH']

    In [1]: os.getenv('PYTHONPATH')



``sys``: system-specific information
--------------------------------------------

System-specific information related to the Python interpreter.

* Which version of python are you running and where is it installed:
    
.. ipython::    

    In [1]: import sys
    
    In [1]: sys.platform

    In [1]: sys.version

    In [1]: sys.prefix

* List of command line arguments passed to a Python script:
    
.. ipython::

    In [1]: sys.argv


``sys.path`` is a list of strings that specifies the search path for
modules.  Initialized from PYTHONPATH:
    
.. ipython::    

    In [1]: sys.path[-2:]