D8.6 Evaluation of user experience and performance of VICINITY Framework & value-added services

This document is the final deliverable from WP8 – “Pilot Demonstrations and Overall Evaluation”. It reports on the assessment criteria, evaluation process and the results of the evaluation of the VICINITY Framework, Pilot test sites, and Open Call trials. It should be read in conjunction with more detailed evaluation reports on the individual pilot trials: D8.2, D8.3, D8.4 and D8.5, and D6.4 which evaluates the key USP of providing security and privacy for the personal data of the end users, also D2.4 which evaluates the impact of VICINITY on standards and D9.14 that reports on commercialisation plans. 

The four pilot projects were completed and operated correctly. In addition, there were two rounds of open-call projects, with four projects funded in each call these all completed satisfactorily and formed an independent review of the VICINITY Framework and the supporting documentation. 

In this deliverable we first explain the approach that was taken to gather evidence in order to evaluate the effectiveness of the pilots, open call projects and the overall Framework. The evidence gathered from each pilot is then summarised with links to deliverables where more detailed information can be accessed. 

The evaluation was based on evidence collected automatically and from stakeholder interviews and questionnaires. Several dashboards were created and used to track system-use and performance. 

  • bAvenir tracked usage and issues with the core VICINITY platform. 
  • The data from Pilot demonstration sites and the related value-added services were uploaded in real time to a central registry operated by UniKL. These activity measurements were fed into a dashboard that enabled trends to be observed graphically. 
  • In addition, some pilot sites used their own dashboard to monitor the performance of their systems. 

Subjective evaluations were collected from various stakeholders who were using the VICINITY-based service either as service users or as service providers. Three metrics to evaluate the state of maturity were adopted. The metrics chosen necessarily cover the range of maturity that extends to the fully-commercialised status of being a fully proven, highly integrated system that uses VICINITY-compatible sensors available off the shelf. This approach is necessary to judge how far the project has advanced the concept and what remains to be done. 

The effectiveness of the evaluation processes is also reviewed.

In combination, these evaluations revealed that the VICINITY concept delivers the intended benefits and is relatively easy to use. The level of maturity of the VICINITY Framework and the pilots was assessed using the Technology Readiness Level scale (TRL). This was originally proposed by NASA and adopted by the EU for H2020 projects. In addition to the TRLs , two new criteria were adopted that assess firstly the ease of integration of sensors into a VICINITY network;  and secondly, for pilots, the level of integration that the pilot use-case had with the pre-existing IT systems that were in use.   

Overall, it was concluded that the VICINITY framework had delivered the expected benefits. The teams implementing the Pilots and OpenCall experiments found the framework easy to use and concluded that it provided the intended services. The VICINITY concept has been proved to work well and its adoption is seen to offer benefits to the organisations using VICINITY. We have reached TRL 7, and have technical and some commercial justification to develop the capability to TRL 9.