My name is Adithi, and I am a Geospatial Data Analyst at ILK Labs in Bengaluru, India. I will now be presenting about how I use shiny to build tools to provide a workflow to analyse air quality data.
R and shiny are a powerful combination.
With the ever increasing global measurements of air pollutants (through stationary, mobile, low-cost, and satellite monitoring) it has become necessary to use management platforms.
So here we present two shiny applications.
R and shiny are a powerful couple which can be used to build interactive platforms to manage and work with the data collected.
With the ever increasing global measurements of air pollutants (through stationary, mobile, low-cost, and satellite monitoring), the amount of data being collected is huge and it necessitates the use of management platforms.
In an effort to address this issue, we developed two Shiny applications to analyse and visualise air quality data.
Shiny apps can be deployed
The user needs no programming knowledge
Help available
Typically, we have two main types of air quality measurements.
This is what stationary monitoring network looks like. Multiple sensors recording data at real time ~ 1 min frequency. Gives information about trends at the city scale
For a neighborhood level measurement, we use mobile monitoring. This schematic shows the many instruments that go in the mobile platform. You can imagine the complexity, diff instruments, have different download methods, from the point of view of analysis - different data formats, different time stamps, etc.
We are looking at the inside of the car now. We have a laptop as a logger for some instruments. And all instruments are secured in the blue tray with inlets outside the vehicle. This is what looks like in practise.
All that to say that many times, things are out of our control in the field.
Build an app to make public use the open source air quality data available and think about science.
Build another app which helps team at ILK to perform quality checks to the high frequency data.
Once we saw that stationary monitors can sometimes need cleaning and also the freely available data was for public use. To facilitate public use why not try to make an app which navigates a user to use the open source air quality data easily and think about science.
And we at ILK Labs we waiting to understand how to perform quality checks on the high dimensional data we were collecting everyday.
We use all tidyverse packages for data cleaning. Ggplot for highly flexible plotting, purrr and map functions for more efficient and faster code instead of for loops, and forcats for functions. Readr, read_csv automatically parses date-time objects, that is is very helpful
lubridate
Date for different instruments can have different
ymd_hms()
, dmy_hms()
...with_tz()
Step 1 is consistent time stamps so that we can join all data sets together. Some instruments are either in UTC time zone, or depending on the country of origin in a diff time zone.
CSTEP, Begaluru, India
The second shiny application "pollucheck" helps processing open source air quality data usually called stationary monitoring.
There are several platforms which provide open source air quality data. We built this application for users of these platforms who can have quick analysis and basic plots of air quality easily generated and think about science.
This is how pollucheck looks like.
sources compatible - CPCB (specific to India), OpenAQ, and AirNow
Data processing options available along with different summary statistics
Generates statistical and summary plots
Implements linear or multilinear regression
Allows users to compare from two sites
"pollucheck" can be used for data downloaded from CPCB (specific to India), OpenAQ and AirNow, it aims at generating a range of statistical plots and summary statistics with several data processing options.
There are options for different averaging periods as well. It also checks for normality, generates density, Q-Q plots along with all these checks for trends in the time series for the selected parameter.
It can implement linear or multilinear regression. pollucheck allows users to upload another set of data to compare selected parameters and generate plots.
It also implements two plots from the openair package.
This is made into a package. Don't forget to check out air quality of your city using our application!
Google Street View car
This is how mmaqshiny looks like.
The first shiny application is called "mmaqshiny" which helps in processing high resolution air quality data collected on a moving platform usually called mobile monitoring.
So we take multiple sensors in a car, and take repeated measurements of each road in an area to generate stable high resolution air pollutant maps (usually daily maps are generated).
There is superior performance in estimating long-term mean concentrations when multiple repeated drives are possible.
Handles high frequency data (~ 1 Hz)
Only mandatory file is GPS and multiple inputs possible
Visualise air pollution hot spots
Pre-processes for various instrumental sensitivities
Provides unit of analysis for further study
Joins output files from different instruments
Reduces computational labour
Alarms user on instrumental errors
Near real time quality check of the data (usually after the ride)
mmaqshiny can handle data of the order 1000's every day from each of the instrument, we had nearly 5 instruments for this study.
It reduces the time consumed for analysing each pollutant individually, helps in visualising the data collected on field each day, it can also be used to look for pollution hotspots, locations that are relatively more polluted than neighboring areas.
Each pollutant or sensor data requires specific kind of pre-processing which depends on the principle on which it operates or its mechanical setup. Eventually, it joins different instrument data into one single corrected file.
This really helps us in achieving the unit of analysis for the rest of the study.
This application has reduced computational labour. Since mobile data contains a huge amount of spatial data, GPS data file is mandatory but other files from different instruments are not necessary.
Since Alarm tab is also present in the application, it helps the user to give a near real-time check on the health of all the instruments used. This application is available as a package on CRAN.
Working with models for low cost sensor data
Working with satellite images
Website: Adithi R. Upadhya
Twitter:
AdithiUpadhya
Github: @adithirgis
Email: adithi@ilklabs.com
Please let me know if you all have any further questions. I am also available in the Lounge if you want to discuss. Have a great day ahead! Enjoy EARL!
My name is Adithi, and I am a Geospatial Data Analyst at ILK Labs in Bengaluru, India. I will now be presenting about how I use shiny to build tools to provide a workflow to analyse air quality data.
Keyboard shortcuts
↑, ←, Pg Up, k | Go to previous slide |
↓, →, Pg Dn, Space, j | Go to next slide |
Home | Go to first slide |
End | Go to last slide |
Number + Return | Go to specific slide |
b / m / f | Toggle blackout / mirrored / fullscreen mode |
c | Clone slideshow |
p | Toggle presenter mode |
t | Restart the presentation timer |
?, h | Toggle this help |
o | Tile View: Overview of Slides |
Esc | Back to slideshow |
My name is Adithi, and I am a Geospatial Data Analyst at ILK Labs in Bengaluru, India. I will now be presenting about how I use shiny to build tools to provide a workflow to analyse air quality data.
R and shiny are a powerful combination.
With the ever increasing global measurements of air pollutants (through stationary, mobile, low-cost, and satellite monitoring) it has become necessary to use management platforms.
So here we present two shiny applications.
R and shiny are a powerful couple which can be used to build interactive platforms to manage and work with the data collected.
With the ever increasing global measurements of air pollutants (through stationary, mobile, low-cost, and satellite monitoring), the amount of data being collected is huge and it necessitates the use of management platforms.
In an effort to address this issue, we developed two Shiny applications to analyse and visualise air quality data.
Shiny apps can be deployed
The user needs no programming knowledge
Help available
Typically, we have two main types of air quality measurements.
This is what stationary monitoring network looks like. Multiple sensors recording data at real time ~ 1 min frequency. Gives information about trends at the city scale
For a neighborhood level measurement, we use mobile monitoring. This schematic shows the many instruments that go in the mobile platform. You can imagine the complexity, diff instruments, have different download methods, from the point of view of analysis - different data formats, different time stamps, etc.
We are looking at the inside of the car now. We have a laptop as a logger for some instruments. And all instruments are secured in the blue tray with inlets outside the vehicle. This is what looks like in practise.
All that to say that many times, things are out of our control in the field.
Build an app to make public use the open source air quality data available and think about science.
Build another app which helps team at ILK to perform quality checks to the high frequency data.
Once we saw that stationary monitors can sometimes need cleaning and also the freely available data was for public use. To facilitate public use why not try to make an app which navigates a user to use the open source air quality data easily and think about science.
And we at ILK Labs we waiting to understand how to perform quality checks on the high dimensional data we were collecting everyday.
We use all tidyverse packages for data cleaning. Ggplot for highly flexible plotting, purrr and map functions for more efficient and faster code instead of for loops, and forcats for functions. Readr, read_csv automatically parses date-time objects, that is is very helpful
lubridate
Date for different instruments can have different
ymd_hms()
, dmy_hms()
...with_tz()
Step 1 is consistent time stamps so that we can join all data sets together. Some instruments are either in UTC time zone, or depending on the country of origin in a diff time zone.
CSTEP, Begaluru, India
The second shiny application "pollucheck" helps processing open source air quality data usually called stationary monitoring.
There are several platforms which provide open source air quality data. We built this application for users of these platforms who can have quick analysis and basic plots of air quality easily generated and think about science.
This is how pollucheck looks like.
sources compatible - CPCB (specific to India), OpenAQ, and AirNow
Data processing options available along with different summary statistics
Generates statistical and summary plots
Implements linear or multilinear regression
Allows users to compare from two sites
"pollucheck" can be used for data downloaded from CPCB (specific to India), OpenAQ and AirNow, it aims at generating a range of statistical plots and summary statistics with several data processing options.
There are options for different averaging periods as well. It also checks for normality, generates density, Q-Q plots along with all these checks for trends in the time series for the selected parameter.
It can implement linear or multilinear regression. pollucheck allows users to upload another set of data to compare selected parameters and generate plots.
It also implements two plots from the openair package.
This is made into a package. Don't forget to check out air quality of your city using our application!
Google Street View car
This is how mmaqshiny looks like.
The first shiny application is called "mmaqshiny" which helps in processing high resolution air quality data collected on a moving platform usually called mobile monitoring.
So we take multiple sensors in a car, and take repeated measurements of each road in an area to generate stable high resolution air pollutant maps (usually daily maps are generated).
There is superior performance in estimating long-term mean concentrations when multiple repeated drives are possible.
Handles high frequency data (~ 1 Hz)
Only mandatory file is GPS and multiple inputs possible
Visualise air pollution hot spots
Pre-processes for various instrumental sensitivities
Provides unit of analysis for further study
Joins output files from different instruments
Reduces computational labour
Alarms user on instrumental errors
Near real time quality check of the data (usually after the ride)
mmaqshiny can handle data of the order 1000's every day from each of the instrument, we had nearly 5 instruments for this study.
It reduces the time consumed for analysing each pollutant individually, helps in visualising the data collected on field each day, it can also be used to look for pollution hotspots, locations that are relatively more polluted than neighboring areas.
Each pollutant or sensor data requires specific kind of pre-processing which depends on the principle on which it operates or its mechanical setup. Eventually, it joins different instrument data into one single corrected file.
This really helps us in achieving the unit of analysis for the rest of the study.
This application has reduced computational labour. Since mobile data contains a huge amount of spatial data, GPS data file is mandatory but other files from different instruments are not necessary.
Since Alarm tab is also present in the application, it helps the user to give a near real-time check on the health of all the instruments used. This application is available as a package on CRAN.
Working with models for low cost sensor data
Working with satellite images
Website: Adithi R. Upadhya
Twitter:
AdithiUpadhya
Github: @adithirgis
Email: adithi@ilklabs.com
Please let me know if you all have any further questions. I am also available in the Lounge if you want to discuss. Have a great day ahead! Enjoy EARL!