D8.1 Pilot Evaluation Framework Definition

Work Package 8 of the EU funded project VICINITY contains six tasks. Together they will contribute to the completion of Milestone 9 Pilot demonstration and overall system evaluation. This milestone will be met on completion of the report D8.6 Evaluation of user experience and performance of VICINITY Framework & value-added services, which is due at the end of the project, Month 48 December 2019. 

A review of published methodologies for evaluation was carried out. The proposed methodology is based on the principles of the European Telecommunications Standards Institute (ETSI) Standard TS 103 463 V1.1.1 (2017-07) Key Performance Indicators for Sustainable Digital Multiservice Cities. It includes multiple domains, provides a basis for comparative assessments over time and shows how to use the five-point Likert scale when posing questions for stakeholder surveys with examples.

‘Interoperability as a service’ is a main goal of VICINITY.  Value-Added Services (VAS) are designed to demonstrate VICINITY interoperability while unlocking possibilities VICINITY enables. The four pilots operate in different locations in three different countries and operate within and across domains: health, buildings management, transport and energy management. All use the VICINITY platform to set-up point-to-point data sharing between devices connected to their gateways and seek to add novel and opportunistic VAS over the top of the infrastructure. VAS have assigned key performance indicators which will be used for the final evaluations to be reported in the final year of the project. 

This report describes how data will be collected from stakeholder surveys and autonomously from the IoT devices connected to the VICINITY Platform. An evaluation checklist has been provided summarizing the key steps for evaluation which are described in this report and are required to make a VICINITY evaluation by the Pilot Site Evaluator or VICINITY Platform Evaluator.

Autonomously collected data will be aggregated for display on the evaluation dashboard both from the VICINITY platform and at the pilot level. In the experimental set-up, metadata, such as number of events per day, are recorded together with how many of these events led to true or false notifications. The usefulness of the dashboard, in providing consolidated data from a number of sources for stakeholder reports or for showcasing will be assessed during the final year of the project.

The evaluation of the VICINITY Platform, its nerve centre, is considered to be an important aspect of the evaluation. To do this, it was necessary to define some additional key performance indicators to those given in Deliverable 5.1 for the Pilots. These are included in this report.

VICINITY offers a platform on which a wide range of unmet needs can be met via the Value-Added Services it enables. The evaluation of these from both technical and stakeholder perspectives will lead to business propositions which can be carried forward after the project ends and will be presented in D9.14 VICINITY exploitation and business plan, final version at the end of the project.

In Work Package 9, Open Calls, 1 and 2, from VICINITY have been designed to allow new organisations to join the project to test-out new opportunistic infrastructures and Value-Added Services. These will be evaluated and reported in Work Package 8 during the final year of the project and their future business plans will be reported in D9.14.