This page demonstrates Python tips and tricks that I use in my everyday programming as an atmospheric science graduate student.
-Brian Blaylock

Friday, April 22, 2016

Python multiprocessing plots

Often I need to create lots of plots. Wouldn't it be great if you could create all the plots at the same time rather than one by one. Summon the multiprocessing module! 


##---- Multiprocessing -------------------------------------------------------------------##
## There are a lot of data files for the two days of TDWR data that need to be plotted.
## We want to make these plots really fast, so use the multiprocessing module
## to create a separate plot on each available processor.
##      1) Place the plotting script in a function. 
##         The input (can only take one) is the file name of the data to be plotted
##      2) In the main program (bottom section of this script) 
##         a) creat a list of the file names (this simple loop is very fast)
##         b) count the number
##         c) creat the pool object p = multiprocessing.Pool(num_proc)
##         d) send each list item to the plot function with the pool function
## This method makes 475 plots in a few minutes rather than over an hour!!!
##---- Multiprocessing -------------------------------------------------------------------##


import multiprocessing #:)
import matplotlib.pyplot as plt
import numpy as np
#etc.

def make_plot(i):    
    # i is the name of the file we will open
    print i
    radar_time = i[8:23]
    
    DATETIME = datetime.datetime.strptime(radar_time,"%Y%m%d_%H%M%S")
    mesowest_time = DATETIME.strftime("%Y%m%d%H%M")
    
    string_time = DATETIME.strftime('%d %b %Y  %H:%M UTC')
    
    
    ### TDWR Radar Data (converted from .nids to .asc grid)
    text = 'SLC_TV0_'+radar_time+'.asc'
    this = np.flipud(np.genfromtxt(text,skip_header=6)) #flip the array

    # etc. plotting functions


## This is the good stuff...utilizing multiprocessors
if __name__ == '__main__':

    # List of file names we want to make plots for
    somelist = []
    for i in os.listdir('./'):
        if i[0:3]=='SLC' and i[-3:]=='asc': # get the file name of a TDWR file for all files
            somelist.append(i)      

    # Count number of processors
    num_proc = multiprocessing.cpu_count()
    p = multiprocessing.Pool(num_proc)
    p.map(make_plot,somelist)




No comments:

Post a Comment

Note: Only a member of this blog may post a comment.