--- title: Utils keywords: fastai sidebar: home_sidebar summary: "General utilities. Should probably split up into `utils.time` and `utils.download`" description: "General utilities. Should probably split up into `utils.time` and `utils.download`" nb_path: "notebooks/01_utils.ipynb" ---
{% raw %}
{% endraw %} {% raw %}
{% endraw %}

Time format strings

These are the different format strings these utils convert from and to.

An identifier with _dt_ in its name signifies a full datetime format as compared to dates only.

{% raw %}
{% endraw %} {% raw %}

nasa_date_to_iso[source]

nasa_date_to_iso(datestr, with_hours=False)

Convert the day-number based NASA date format to ISO.

Parameters
----------
datestr : str
    Date string in the form Y-j

Returns
-------
Datestring in ISO standard yyyy-mm-ddTHH:MM:SS.MMMMMM
{% endraw %} {% raw %}
{% endraw %} {% raw %}
nasa_date = "2010-110"
iso_date = "2010-4-20"
{% endraw %} {% raw %}
assert nasa_date_to_iso(nasa_date, with_hours=True) == "2010-04-20T00:00:00"
assert nasa_date_to_iso(nasa_date) == "2010-04-20"
{% endraw %} {% raw %}

iso_to_nasa_date[source]

iso_to_nasa_date(datestr)

Convert iso date to day-number based NASA date.

Parameters
----------
datestr : str
    Date string in the form Y-m-d

Returns
-------
Datestring in NASA standard yyyy-jjj
{% endraw %} {% raw %}
{% endraw %} {% raw %}
assert iso_to_nasa_date(iso_date) == nasa_date
{% endraw %} {% raw %}

nasa_datetime_to_iso[source]

nasa_datetime_to_iso(dtimestr)

Convert the day-number based NASA datetime format to ISO.

Note: This is dateTIME vs `nasa_date_to_iso` which is just DATE.

Parameters
----------
dtimestr : str
    Datetime string in the form Y-jTH:M:S

Returns
-------
Datestring in ISO standard yyyy-mm-ddTHH:MM:SS.MMMMMM
{% endraw %} {% raw %}
{% endraw %} {% raw %}
nasa_datetime = "2010-110T10:12:14"
nasa_datetime_with_ms = nasa_datetime + ".123000"
iso_datetime = "2010-04-20T10:12:14"
iso_datetime_with_ms = iso_datetime + ".123000"
{% endraw %} {% raw %}
assert nasa_datetime_to_iso(nasa_datetime) == iso_datetime
assert nasa_datetime_to_iso(nasa_datetime_with_ms) == iso_datetime_with_ms
{% endraw %} {% raw %}

iso_to_nasa_datetime[source]

iso_to_nasa_datetime(dtimestr)

Convert iso datetime to day-number based NASA datetime.

Parameters
----------
dtimestr : str
    Datetime string in the form yyyy-mm-ddTHH-MM-SS

Returns
-------
Datestring in NASA standard yyyy-jjjTHH-MM-SS
{% endraw %} {% raw %}
{% endraw %} {% raw %}
assert iso_to_nasa_datetime(iso_datetime) == nasa_datetime
assert iso_to_nasa_datetime(iso_datetime_with_ms) == nasa_datetime_with_ms
{% endraw %} {% raw %}

replace_all_nasa_times[source]

replace_all_nasa_times(df)

Find all NASA times in dataframe and replace with ISO.

Changes will be implemented on incoming dataframe!

This will be done for all columns with the word TIME in the column name.

Parameters
----------
df: pandas.DataFrame
    DataFrame with NASA time columns.

Returns
-------
Nothing (Changes implemented in-place)
{% endraw %} {% raw %}
{% endraw %}

Network utils

{% raw %}

class ProgressBar[source]

ProgressBar(*_, **__) :: tqdm

Provides `update_to(n)` which uses `tqdm.update(delta_n)`.
{% endraw %} {% raw %}

parse_http_date[source]

parse_http_date(text)

Parse date string retrieved via urllib.request.
{% endraw %} {% raw %}

get_remote_timestamp[source]

get_remote_timestamp(url)

{% endraw %} {% raw %}

download[source]

download(url, local_dir='.', use_tqdm=True, **kwargs)

Simple wrapper of urlretrieve

Adding a default path to urlretrieve

Parameters:
----------
url : str
    HTTP(S) URL to download
local_dir : str,pathlib.Path
    Local directory where to store the download.
**kwargs : {dict}
    Keyword args to be handed to urlretrieve.
Returns
-------
Tuple
    Tuple returned by urlretrieve
{% endraw %} {% raw %}

url_retrieve[source]

url_retrieve(url:str, outfile:str, chunk_size:int=128)

Improved urlretrieve with progressbar, timeout and chunker.

This downloader has built-in progress bar using tqdm and using the `requests`
package it improves standard `urllib` behavior by adding time-out capability.

I tested different chunk_sizes and most of the time 128 was actually fastest, YMMV.

Parameters
----------
url : str, urlpath.URL
    The URL to download
outfile: str, pathlib.Path
    The path where to store the downloaded file.
chunk_size : int, optional
    The size of the chunk for the request.iter_content call. Default: 128

See also
--------
Inspired by https://stackoverflow.com/a/61575758/680232
{% endraw %} {% raw %}

have_internet[source]

have_internet()

Fastest way to check for active internet connection.

From https://stackoverflow.com/a/29854274/680232
{% endraw %} {% raw %}
{% endraw %}

Image processing helpers

{% raw %}

height_from_shadow[source]

height_from_shadow(shadow_in_pixels, sun_elev)

Calculate height of an object from its shadow length.

Note, that your image might have been binned.
You need to correct `shadow_in_pixels` for that.

Parameters
----------
shadow_in_pixels : float
    Measured length of shadow in pixels
sun_elev : angle(float)
    Angle of sun over horizon

Returns
-------
height [meter]
{% endraw %} {% raw %}

get_gdal_center_coords[source]

get_gdal_center_coords(imgpath)

Get center rows/cols pixel coordinate for GDAL-readable dataset.

Check CLI `gdalinfo --formats` to see all formats that GDAL can open.

Parameters
----------
imgpath: str, pathlib.Path
    Path to raster image that is readable by GDLA
{% endraw %} {% raw %}

file_variations[source]

file_variations(filename, extensions)

Create a variation of file names.

Generate a list of variations on a filename by replacing the extension with
the provided list.

Parameters
----------
filename: str, pathlib.Path
    The original file name to use as a base.
extensions: list-like
    A list of file extensions to generate new filenames.

Returns
-------
list of pathlib.Paths

Notes
-----
Adapted from T. Olsens `file_variations of the pysis module for using pathlib.
{% endraw %} {% raw %}
{% endraw %} {% raw %}
fname = 'abc.txt'
{% endraw %} {% raw %}
extensions = ".cub .cal.cub .map.cal.cub".split()
{% endraw %} {% raw %}
file_variations(fname, extensions)
[PosixPath('abc.cub'), PosixPath('abc.cal.cub'), PosixPath('abc.map.cal.cub')]
{% endraw %}