Save a python variable using Numpy

This just might revolutionize the way I return to projects and test functions

You can save a python variable using numpy!!!

Sure wish I would have known this a long time ago. For example, if you have a large python dictionary, you can save the dictionary, and then load it and use it later.

Example:

SomeDictionary = {'Name': np.array([1,2,3,4,5]), 'DATES':np.array([1,2,3,4,5])}

np.save('PyDict.npy', SomeDictionary)

get_the_dict = np.load('PyDict.npy').item()

What a miracle :)

Adding fields to a structured numpy array created from genfromtxt

Is it possible to add a new field in a numpy.genfromtxt output? Yes it is. I asked this question on stackoverflow and got an answer that works...
http://stackoverflow.com/questions/40182466/is-it-possible-to-add-a-new-field-in-a-numpy-genfromtxt-output/40183748#40183748

The trick is using the numpy.lib.recfunctions library. Use the append_fields function to add a new field to a data structure.

Dealing with python datetime arrays

Sometimes it's a pain that you can't perform datetime operations on each element of a numpy array. Here are a couple tipes

1. List of a range of datetime objects

The best way to make a list of datetime objects or numpy array is to use this tip:

which creates a list of datetime objects from today going back a day until it hits the number of days range limit. You can change the - to a + to increment one day in the future, or change the days argument to hours, etc.

2. Convert list of datetime strings to python datetime objects

When you read in date and time data from a file, it will often come in as a string. Often, you'll read these in as a list. To convert that list of strings to a list of python datetime objects do this:

3. Convert list of epoch times to python datetime objects

Alternatively, if the dates come in as a list of epoch times, use this...

4. Convert list of datetime strings to python datetime objects using a numpy vectorized function

This method is slightly appears to be slightly faster than method discussed in 2.

---

In the past I would create an empty array and append to it with a for loop like this...
but this method takes longer to compute, especially for long arrays.

Find the nearest latitude and longitude grid box for a point

🐍 An updated notebook is available for this demonstration on GitHub.

Also look at my demonstration using the KDTree method (scipy.spatial.KDTree.query) to find nearest neighbor for many points.

---

I deal a lot with weather model output (specifically HRRR model data) and often need to verify a data point with observational data. Weather stations we get observations from are not located at the exact grid points as the model's grid points. The hard part is finding the index value of a latitude and longitude grid point nearest to the observation location. This is made easy with numpy's maximum function (not max).

First, you need to have three arrays, one for latitude, one for longitude, and one for the environment variable such as temperature. Load these values into a numpy array. I use pygrib to read model output in grib2 format.

lat = latitude  # a 2D numpy array of your latitudes
lon = longitude # a 2D numpy array of your longitudes
temp = temperature # a 2D numpy array of your temperatures, or other variable

Next you need to know the latitude and longitude for the observation point. We are looking for the nearest grid point in the lat and lon arrays for that grid point.

stn_lat = station's latitude # eg. 40
stn_lon = station's longitude # eg. -110

Now find the absolute value of the difference between the  station's lat/lon with every point in the grid. This tells us how close a point is to the particular latitude and longitude.

abslat = np.abs(lat-stn_lat)
abslon= np.abs(lon-stn_lon)

If you plot abslat and abslon, you'll see what we have done...we have created two new arrays with that tell us how close each latitude and longitude is to the grid point we are interested in.

abslat: the blue line represents the nearest grid latitude line our station is located.

abslon: the blue line represents the nearest grid longitude our station is located.

Now we need to combine these two results.
We will use numpy.maximum, which takes two arrays and finds the local maximum.

c = np.maximum(abslon,abslat)

Which is an array that looks like this...

c: an array expressing the local maximums between the abslat and abslon arrays

The dark blue point is the nearest grid location to our station. You can find the index location on the grid of this by using the min function.

latlon_idx = np.argmin(c)

Now, this latitude/longitude index value is the index for a flattened array, so when you look for that same index value in, say, your temperature array, you should flatten the array to pluck out the value at that point.

grid_temp = temp.flat[latlon_idx]

If you don't like flattened arrays, you can also get the row/column index like this
x, y = np.where(c == np.min(c))
grid_temp = temp[x[0], y[0]

And there you have it!