{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Using AgReFed @ Dale data\n", "\n", "This is a small Jupyter notebook to show how one might consume the data provided by\n", "AgReFed @ Dale. It provides some simple plotting examples using [plotly](http://plot.py) and\n", "[hvPlot](hvplot.pyviz.org).\n", "\n", "It is just javascript suitable for cutting and pasting directly into a browser and a bit of Python.\n", "\n", "You can download the *actual* notebook from: http://webapps.plantenergy.uwa.edu.au/agrefed_dale/static/plotly.ipynb (much more fun!)\n", "\n", "Hopefully you can get an idea of how easy it is to incorporate `AgReFed @ Dale` data into your\n", "own investigations.\n", "\n", "see https://plot.ly/javascript/reference/ for a reference to plotly.js" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# from javascript" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## fetch plotly...\n", "\n", "\n", "The notebook uses [requirejs](https://requirejs.org/docs/api.html) so we can use that to download plotly...\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The plotly javascript is at https://cdn.plot.ly/plotly-latest.min.js (all 6MB of it!)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%javascript\n", "// ask for plotly and give it a name \"plotly\"\n", "requirejs.config({\n", " paths: {\n", " plotly: 'https://cdn.plot.ly/plotly-latest.min'\n", " }\n", "});\n", "// helper function\n", "window.plotly = (elem, plots, layout) => {\n", " const div = document.createElement('DIV')\n", " elem.append(div)\n", " // ask requirejs to call us when plotly is loaded\n", " requirejs(['plotly'], Plotly => Plotly.newPlot(div, plots, layout))\n", "}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### set Target to AgReFed @ Dale" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%javascript\n", "window.Dale = 'https://webapps.plantenergy.uwa.edu.au/agrefed_dale'" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Some helper functions" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%javascript\n", "// list of objects to an object of lists\n", "window.unzip = (arr, ...keys) => {\n", " var o = {};\n", " keys.forEach(k => o[k]= []);\n", " arr.forEach(a => {\n", " keys.forEach(k => o[k].push(a[k]))\n", " });\n", " return o;\n", "}\n", "// show failure\n", "window.jqfail = (ele, xhr) => {\n", " const div = $('
');\n", " div.css({\"background-color\": \"orange\"})\n", " div.text(xhr.responseText || xhr.statusText || 'unknown error')\n", " ele.append(div);\n", "}\n", "window.fetch_fail = (ele, ...args) => {\n", " args = args.map(x => x.toString()).join(',')\n", " const div = document.createElement('DIV');\n", " div.textContent = `${args}`\n", " div.style.backgroundColor = 'orange'\n", " ele.append(div)\n", "}\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## using fetch" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "[`fetch`](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API) is an ajax method available in all(most) modern browsers...\n", "\n", "Let's get some site weather data!\n", "\n", "Note that in the `%%javascript` notebook context `element` is a `jQuery` object representing the output text area of the\n", "current cell. This way we can avoid creating random `id` for div elements for plotly to find." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%javascript\n", "const plot1 = element;\n", "\n", "// we only want these columns....\n", "const cols1 = ['datetime', 'soilt', 'airt', 'AT_60cm', 'AT_30cm']\n", "// our query per_page:0 mean *all* data\n", "var params = { per_page:0, datetime: \"2018-02-17/P20D\", select:cols1.join(',') } \n", "\n", "var url = new URL(Dale + '/WeatherSite')\n", "url.search = new URLSearchParams(params)\n", "\n", "\n", "fetch(url).then(resp => resp.json())\n", " .then(json => unzip(json.items, ...cols1))\n", " .then(({datetime, ...rest}) => {\n", " // turn data into plotly plots.\n", " const traces = Object.entries(rest).map(([name, y]) => {\n", " return {\n", " x: datetime,\n", " y: y,\n", " type: 'scatter',\n", " name: name\n", " }\n", " });\n", "\n", " plotly(plot1, traces, {width:900});\n", " }).catch((...args) => fetch_fail(plot1, ...args))\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## using jQuery ajax\n", "\n", "We always have jQuery available in a notebook" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%javascript\n", "const plot2 = element;\n", "const cols2 = ['datetime', 'soilt', 'airt']\n", "var params = {per_page:0, datetime: \"2018-02-17/2018-04-17\", select: cols2.join(',')}\n", "\n", "$.get(Dale + '/WeatherSite', params).done(json => {\n", "\n", " const {datetime, ...rest} = unzip(json.items, ...cols2);\n", " const traces = Object.entries(rest).map(([name, y]) => {\n", " return {\n", " x: datetime,\n", " y: y,\n", " type: 'scatter',\n", " name: name\n", " }\n", " });\n", "\n", " plotly(plot2, traces, {width:900});\n", "}).fail((xhr) => jqfail(plot2, xhr))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Usually you want to compare a column to a value e.g. `airt > 38`. But you can\n", "also compare other columns if you \"quote\" the value with backticks: e.g.airt > `soilt`
. \n",
"\n",
"Here we plot all places where the air temperature was greater than the soil temperature.\n",
"\n",
"We have to target the `/query/suba` endpoint which understands `SUBA` query language. See\n",
"[here](http://webapps.plantenergy.uwa.edu.au/agrefed_dale/search.html)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%%javascript\n",
"const plot3 = element;\n",
"const cols3 = ['datetime', 'soilt', 'airt']\n",
"var params = {per_page:0, query: \"airt > `soilt`\", entity_name: \"WeatherSite\", select: cols3.join(',')}\n",
"\n",
"$.post(Dale + '/suba/query', params).done(json => {\n",
"\n",
" const {datetime, ...rest} = unzip(json.items, ...cols3);\n",
" const traces = Object.entries(rest).map(([name, y]) => {\n",
" return {\n",
" x: datetime,\n",
" y: y,\n",
" type: 'scatter',\n",
" mode: 'markers',\n",
" name: name\n",
" }\n",
" });\n",
"\n",
" plotly(plot3, traces, {width:900});\n",
"}).fail((xhr) => jqfail(plot3, xhr))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## crop yields vs plot\n",
"\n",
"Let's look at how crop yield varied across plots"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%%javascript\n",
"const plot4 = element;\n",
"const cols4 = ['plot_barcode', 'latitude', 'longitude', 'crop_yield']\n",
"$.get(Dale + '/Plot', {per_page: 0, select: cols4.join(',')}).done(page => {\n",
"\n",
" const {plot_barcode, latitude, longitude, crop_yield} = unzip(page.items, ...cols4);\n",
" const traces = [\n",
" {\n",
" y: latitude,\n",
" x: longitude,\n",
" type: 'scatter',\n",
" mode: 'markers',\n",
" text: page.items.map(r => `${r.plot_barcode}: ${r.crop_yield}`),\n",
" marker : {\n",
" size:crop_yield.map(y => 2*y),\n",
" color:crop_yield,\n",
" colorscale: 'Viridis',\n",
" colorbar: {\n",
" title: 'Crop Yield'\n",
" }\n",
" }\n",
" }]\n",
" const axis = {\n",
" tickformat: \".3f\",\n",
" ticksuffix: \"°\",\n",
" }\n",
" const layout = {\n",
" height: 750,\n",
" width: 900,\n",
" title: 'plot yield vs location',\n",
" xaxis : {\n",
" title: \"longitude\", ...axis\n",
" },\n",
" yaxis : {\n",
" title: \"latitude\", ...axis\n",
" }\n",
" }\n",
"\n",
" plotly(plot4, traces, layout);\n",
"}).fail((xhr) => jqfail(plot4, xhr))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"... or maybe you just want the data\n",
"\n",
"Let's get a nice [json renderer](https://github.com/caldwell/renderjson)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%%javascript\n",
"requirejs.config({\n",
" paths: {\n",
" renderjson: 'https://cdn.jsdelivr.net/npm/renderjson@1.4.0/renderjson.min'\n",
" }\n",
"});\n",
"window.renderjson = (elem, json, level=2) => {\n",
" requirejs(['renderjson'], r => {\n",
" r.renderjson.set_show_to_level(level);\n",
" elem.append(r.renderjson(json))\n",
" })\n",
"}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"... and give it some colour"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%%html\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we can just look over the data.\n",
"\n",
"Let's get a page of data\n",
"\n",
"Notice the weird `next_url`? Just \"get\" that endpoint and you'll have the next page of data from\n",
"your query (nice hey!). You can step forward (and back!) through the pages easily."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%%javascript\n",
"const plot5 = element;\n",
"var cols5 = ['datetime', 'soilt', 'airt']\n",
"var params = {per_page:5, query: \"airt > `soilt`\", entity_name: \"WeatherSite\", select: cols5.join(',')}\n",
"\n",
"$.post(Dale + '/suba/query', params).done(json => {\n",
" renderjson(plot5, json, 3)\n",
"}).fail((xhr) => jqfail(plot5, xhr))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"All the metadata about the data can be found from the OAS schema.\n",
"\n",
"In particular `components.schemas.{Table}.properties` gives details on the available data names.\n",
"You can also find all the information [here](\n",
"https://webapps.plantenergy.uwa.edu.au/agrefed_dale/api/oas3.html)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%%javascript\n",
"const plot6 = element;\n",
"$.get(Dale + '/swagger.json').done(json => {\n",
" renderjson(plot6, json, 2)\n",
"}).fail((xhr) => jqfail(plot6, xhr))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# from Python\n",
"\n",
"It's just as easy from Python"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import requests\n",
"import pandas as pd\n",
"Dale = 'https://webapps.plantenergy.uwa.edu.au/agrefed_dale'"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"\n",
"params = {\n",
" 'per_page': 20,\n",
" 'query': \"airt > `soilt`\",\n",
" 'entity_name': \"WeatherSite\",\n",
" 'ordering' : '