RoadCloud and WhereOS Collaborate on Automotive Data Collection and Processing

Traffic Jams in Helsinki

Are there traffic jams in Helsinki? People living in Helsinki would say YES, and people living in central Europe would probably laugh at this. The traffic jams are relatively mild compared to many other places but there are clearly times and locations where the traffic slows down significantly.

RoadCloud and WhereOS have joined forces to collect, analyze and visualize data collected from the fleet of commercial vehicles through RoadCloud sensors. “It’s amazing how detailed information you can collect from road network conditions through RoadCloud sensors”, says JP Partanen, CEO of WhereOS.

In the series of articles, we explain how the data can be collected from vehicle fleets and processed into models, that can further be used to solve different business problems such as traffic conditions, road conditions, or even creating machine learning / artificial intelligence model to predict these conditions based on other external variables such as weather forecasts. In this article, we explain how we created a video visualization of traffic conditions & jams throughout the day. “We are impressed with WhereOS, and how it quick it was to turn our data into an insightful video. We are integrating our data API to WhereOS and making our anonymized vehicle data easily accessible”, says Ari Tuononen, CEO of RoadCloud.

Capturing Vehicle Flow Data with RoadCloud IoT Sensors

RoadCloud has equipped commercial vehicle fleet with RoadCloud IoT sensors to collect and monitor vehicle data and road conditions. The sensors are automatically collecting basic information such speed, heading and acceleration, but more importantly information about road surface conditions such as road friction, road state (dry/snow/ice/water), temperature, bumps and pot holes as a few examples. The data collection is taking place 24/7, as the commercial vehicles are continuously throughout the day and night.

The sensor data is uploaded to the RoadCloud data backbone, where it is stored for further use & analysis. The data can be processed as historical data, or as a real-time feed of continuous updating data.

Analyzing and Visualizing the Sensor Data with WhereOS

WhereOS connects to the RoadCload data backbone and can process the data further. In this visualization, the RoadCloud data is processed and combined with OpenStreetMap (OSM) street network data in several pipelines:

  1. OSM data preparation pipeline extracts the OpenStreetMap street network for the desired region, Helsinki capital area in this case, and splits the streets into segments of desired length (e.g. 200m).
  2. RoadCloud data extraction pipeline loads data from RoadCloud data backbone, and assigns joins the GPS (latitude & longitude) data points into corresponding street segments. Joining uses WhereOS geokey/geohash based operation for matching massive amount of geographical shapes – GPS points and polylines/linestrings (street segments) – together.
  3. NTILE pipeline takes the joined data, and approximates maximum speed for each street segment, by taking average of the 10% highest speeds driven on that specific street segment.
  4. Hourly traffic pipeline calculates average speed driven for each individual street segment for each hour, and also the speed decrease from the maximum speed.
  5. Rendering pipeline takes speed decrease for each street segment for each hour and produces an MP4 video where each frame represents traffic conditions (speed decrease) for each street segment for the given hour. Each street segment is colored so that red equals to high speed decrease (i.e. high traffic / traffic jam) and green means no decrease (i.e. traffic speed close to maximum).

WhereOS uses Spark and Hive as the execution engine for the pipelines. WhereOS pipelines can be created using SQL and R programming languages including built-in functions for ETL (extract, transform, load) operations, statistical analysis, machine learning and artificial intelligence (AI), geographical and geospatial analysis, data visualization etc.

Traffic Jam Video & Further Innovation

The data used for this video has been collected from the vehicles over the time of one full year (2018). You can see how the rush hour traffic affects the average speeds at different locations on different times of the day.

The RoadCloud data can be used for many other analyses such as: How the road conditions – friction, road state (dry/wet/ice/snow), temperature – affect the average speeds. Or how a speed bump or some other new traffic arrangement affects the traffic around it. In the upcoming articles we will dig deeper into the data and create new interesting visualizations out of it.

Ari Tuononen, CEO, RoadCloud: ari@roadcloud.com, +358 50 5604 702
More info: www.roadcloud.fi Twitter LinkedIn

JP Partanen, CEO, WhereOS: jp@eaglepeaks.com, +358 50 486 9257
More info: www.whereos.com Twitter LinkedIn

Most Extensive Visualization of US Income: Over 33000 Zip Codes Visualized

This application visualizes mean income of households in United States, for each zip code (ZCTA, zip code tabulation area): Click here to open the application

Background: Urban Decay

The visualization shows clearly, how large cities in US are surrounded by wealthy zip code areas (red rings). Inner cities, however are areas with less income (green or yellow), with few exceptions, such as New York, Washinton DC and San Francisco.

“In the United States during the 1940s, for the first time a powerful interaction between segregation laws and race differences in terms of socioeconomic status enabled white families to abandon inner cities in favor of suburban living. The result was severe urban decay that, by the 1960s, resulted in crumbling “ghettos“. Prior to national data available in the 1950 US census, a migration pattern of disproportionate numbers of whites moving from cities to suburban communities was easily dismissed as merely anecdotal. Because American urban populations were still substantially growing, a relative decrease in one racial or ethnic component eluded scientific proof to the satisfaction of policy makers. In essence, data on urban population change had not been separated into what are now familiarly identified its ‘components.’ The first data set potentially capable of proving ‘white flight’ was the 1950 census.” Source: Wikipedia

The visualization is based on American Community Survey (ACS) from census.gov

Red = high income
Yellow = medium income
Green = low income

Feedback

I’m looking for bloggers, who want to co-author articles about interesting/surprising facts about US (or European) population demographics, economy etc. along with visualizations. Also, if you have ideas for further visualizations, please contact me via support@eaglepeaks.com.

If you want to embed this app to your web page, read the instructions here.

-JP

Finland: 58 Interesting Indicators Visualized

This is an app, through which you can visualize demographic data from Finland. Try also clicking for example “Relative” and “Higher level university degree”, or “Employed”.

Click here to open the application

The data used in this visualization: Paavo – Open data by postal code area, Statistics Finland. The material was downloaded from Statistics Finland’s interface service with the licence CC BY 4.0.

Check this article how to embed a WhereOS app to your own website. If you have ideas what kind of data you’d like to see in the apps, write me an email support@eaglepeaks.com.

-JP

How to Embed a WhereOS App

Embedding a WhereOS apps to any web page or a blog is easy. Just include the an HTML iframe-tag with width and height properties, and point the link to the application you have created.

URL Format:

https://[domain]/a/[application_name]/[ui_view_name]

[domain] is the name of the service domain you have used when starting the WhereOS service for yourself

[application_name] is the name of the application

[ui_view_name] is the name of the UI view, if more than one UI view exists for the given application

Remeber to replace spaces in the application name with underscore in the URL. Application with name “Reachability Analysis” becomes Reachability_Analysis and so forth.

<iframe frameborder="0" style="width: 100%; height: 400px;" src="https://apps.whereos.com/a/Reachability_Analysis"></iframe>

-JP

WhereOS – A new way of doing Spark, Hive and SQL

WhereOS is a new cloud based operating system for distributed, data driven applications, that’s built on top of Apache Spark. WhereOS makes it easier to build, host, distribute and share applications built using (big) data.   

WhereOS can be programmed through SQL programming language including built-in functions for ETL (extract, transform, load) operations, statistical analysis, geographical and spatial analysis, data visualization etc. Programs written in WhereOS applications are always executed as distributed processes, without developers explicitly design their applications for parallel processing and scalability.

WhereOS uses Spark and Hive as the execution engine for the applications. WhereOS can be extended through two types of drivers: function and data asset drivers. Function drivers can introduce any new Hive/SparkSQL functions, to implement new features for the SQL language. Data asset drivers provide new ways of integrating data formats and/or data transmission protocols into the system.

If you want to discuss more, contact us through support@eaglepeaks.com

-JP

Location Analytics Industry Trends

Here are three relevant industry trends, why I believe WhereOS is going to change the world:

  1. Hyperconnected World – Businesses are getting more and more connected to online, cloud based information systems as a part of their business processes. Customers are accessing companies’ online services 24/7 through different interfaces, generating more and more information about their purchase behavior, location, movement, social interactions and so forth – all tied to geographical locations
  2. Business Digitalization – New businesses are challenging traditional industries by using combinations of static and dynamic information to create new services for their digital customers: They know better who their customers are, what they want, where they are located, and how to serve them best. They know where the operational bottlenecks are and can deliver the right goods and services to right locations to create the best customer experience, maximize the sales and minimize the cost.
  3. Data Explosion – The amount of data businesses are collecting from their customers and business operations is increasing constantly. Data sets are large, and in constant change – the existing IT systems and processes do not facilitate innovation in the fast paced change many industries are undergoing.

With WhereOS, it’s extremely easy to built APIs and applications based on data: instead of months, new applications can be create in a matter of hours or days. This facilitates lean prototyping of how data can be used to create new value-adding services and products, helping both traditional and new businesses to speed up the development.

If you are interested in discussing more, please contact me through support@eaglepeaks.com

-JP

Route Optimization

Route Optimization application calculates the most efficient route that goes through the specified set of waypoints. Route optimization can be used to calculate service routes, delivery routes, pick-up routes etc., that require all waypoints to be visited in the most efficient manner.

The most efficient route is defined calculated through R function, that takes the characteristics of the route and waypoints into account. The default weighting function is duration based, i.e. the optimal route is the one that finishes earliest, with two waypoint categories: critical and low. Critical waypoints are visited first, and low category waypoints last, in a manner that the shortest duration route is the final result. The R function can be modified to take into account different route characteristics, such as route distance, waypoint types, waypoint SLAs, time of day etc.

If you have ideas or questions – or you need help to use the application in your business domain, write to us support@eaglepeaks.com

Open the app: https://apps.whereos.com/a/Route_Optimization

-JP