Scientific Methodology and Work Packages
Work Package 1
In WP1 Requirements, the FANCI software requirements specification (SRS) will be developed comprising a complete description of the behaviour of a system to be developed, encompassing the product perspective, product functions, and specific requirements, functional and non-functional, related to the software. All functional requirements will include a set of use cases that describe interactions the users will have with the software (UX experience). Non-functional requirements imposing constraints on the design or implementation (such as standards compliance, quality standards, or design constraints) will be specified. A comprehensive description of the requirements for both the demonstrator platform and reference form factor board will be provided. Finally, all necessary requirements for the project development and associated risks in implementation will be listed.
Work Package 2
WP2 Architecture is where the foundations will be laid for the system. This will be done in two steps. First the existing tools of the partners will be gathered according to the requirements in WP1, potentially for PC. Then, an analysis will determine the missing parts required for the target application, and these will be added in the functional overview of the system, on the DSP platform. In addition, a detailed plan and schedule of rest of the project will be re-verified against the detailed architecture and fixed where relevant.
Work Package 3
In WP3 Automotive Application and Platform Demonstrator, the focus will be to ramp-up early stage developer and demonstrator platforms as defined in WP2. As the main goal of the project is to bring the innovation of FANCI algorithms and applications into the embedded space, we build these platforms in order to serve us in multiple areas:
Use as internal development platforms during the project duration (until receiving final platforms)
Use as early demonstration vehicles for CES 2016 as well as other shows to generate market feedback and interest
With such goals in mind, the WP will focus on the speed of ramping up the platforms and hence will mostly re-use (as much as possible) existing technologies from the various partners to serve as early integrations of various components in order to achieve the objectives. We differentiate between the engineering developer reference platform and the Automotive demonstrator platform. The developer platform is very generic and could be used for algorithm development for a wide variety of markets and products such as home entertainment, mobile devices, surveillance, home security, automotive and much more. This is a very raw materials platform intended for engineers such as algorithms and application developers to work on, and in fact will be used only internally during the FANCI project. See Figure 7: Sample FANCI Platform Design for an example of such a platform. The developer platform includes various input sensors such as a microphone and a camera and vector DSP to execute FANCI algorithms and connectivity to various outputs such as speakers, display and similar. The platform also enables developers to connect the platform to a PC and load, debug, and execute their applications from the PC onto the platform and is intended to emulate a simple embedded device, incorporating the dedicated vector DSP which executes the sophisticated algorithms.
The Automotive demonstrator platform on the other hand, is intended to serve specifically for demonstrating automotive infotainment and safety applications. It is prototype for a full system and hence includes specific automotive related interfaces and modules such as connectivity to HUD, CAN bus, etc.
Work Package 4
In WP4 Development of User Sensing Methods, the FANCI User Sensing API will be implemented. The User Sensing API will provide classification of user emitted signals captured from sensors related to the following areas: face, eyes, hand, head and voice. The API will be designed to detect and provide access to information such as emotions expressed through facial expressions, eye movements, gaze direction, hand detection and gestures, face authentication, voice presence and pitch, allowing application-specific user state estimation models to be constructed.
Figure 3: User Sensing Data Flow
In addition to a high-level API for application developers, the API will provide an internal low-level feature abstraction from raw sensor data from depth and RGB images via a reconstructed 3D face model and audio data from a voice pitch model optimised for DSP intended for algorithm developers. It will also feature plugin architecture to allow new capabilities from third parties to be added easily in the future that can directly accept raw sensor data or information from the face and voice models. The API will be divided into four main components:
- Face: face detection and authentication, face tracking and emotion recognition, gaze tracking
- Hand: hand detection and gesture recognition.
- Voice: voiced / unvoiced detection, voice authentication and pitch analysis.
- Head: head pose and position estimation.
Work Package 5
Then, in WP5 Development of User State Estimation and System-Reaction Methods, a modular software architecture will be developed which allows fusing sensor data generated by the vehicle demonstrator (as described in WP2), resulting in a reliable and useful user state estimate. In addition, a range of system reaction methods are implemented, taking advantage of the user’s estimated state, and generate multi-modal user feedback that is appropriate to the context of the driver, the car, and the environment. This includes detection of driver fatigue (through eye blink frequency analysis) and inattention (through head pose analysis, for instance the driver looking too long through the side window) and other methods. The core idea behind this task is to go the extra mile in the software development of the system to utilize all existing data, both from the sensor fusion, context information, as well as external information (such as other in-car systems) and generate a sophisticated estimation of the user’s state, calculate their intentions and have the system give back feedback to the user in multiple forms such as voice, vibration, display and even autonomous decision making.
The focus here will be both generic as well as specific implementations for the automotive market to be used in WP6 by the creative application. The sensor fusion encompasses the driver and passenger’s cognitive, attentive, mental, affective, and physiological state. It also includes the activity level of the driver and the passengers, the cognitive load level including the attentive state, the affective state (emotion estimation), physiological data such as breathing rate, blood pressure, and galvanic skin response (all measured through the steering wheel) which are indicative of the driver’s state.
Work Package 6
In WP6 Creative Automotive Applications and UX Validation, the software and the hardware will be validated, documented and packaged, and the FANCI User Sensing API will be ported and optimised onto the universal developer reference platform. The idea behind this reference platform is to supply to application developers a full stack and easy to use platform with a clear FANCI API where they could develop creative and innovative new applications based on the FANCI modules as well as extend those existing modules if the need arises.
As example application, we can take the hot trend of cloud computing. Using the reference platform and the FANCI User Sensing API a great application could be cloud based access via HMI. So the application could, via the FANCI API, authenticate, track, control and adapt according to the users emotions and intentions.
WP6 also includes activities to develop an automotive demonstration platform supporting the FANCI User Sensing API as well as the user state estimation and the system-reaction methods developed in WP5. This activity will be the main technical focus on the project by basically bundling all the work done during the project into a specific application targeted for the automotive market running on the demonstration platform.
As part of the testing and verification of the application, we will fine tune the user experience (UX) to real customer feedback by building some test scenarios
Work Package 7
WP7 Dissemination, Exploitation and Standardisation will take care of the project’s dissemination planning and execution, near-market exploitation planning and contribution to standardisation and certification. In order to be able to get early market feedback from relevant stakeholders in various markets with focus on Automotive we have taken early stage demonstrators. WP3 will result in early demonstrations which will allow presentation starting from CES 2016. In addition a website which will include access to the developed Cloud-based API will be updated as material becomes available. We believe this approach should enable a good path to exploitation for all the partners. This work package will build and execute a detailed dissemination plan based on the Draft Dissemination Plan.
A project Knowledge Portfolio will be developed at project-start, and kept updated with pre-existing knowledge according to the consortium agreement. Preparation of an exploitation plan will also be carried out as an iterative process including advice and support for the consortium with IPR management and protection issues.
As for standardization, the consortium will study, contribute and define standards related to the project. As an example, Khornos OpenVX, an open standard which applies to offloading CV functions from CPU could be extended and applied to portions of the FANCI API which will be addressed during the project.
Work Package 8
WP8 Management and Evaluation will focus on the cross-project coordination, technical leadership and collaboration infrastructure aspects. The idea here is to generate on project-start a good working relationship between the management and engineering levels of all partners and build a stable infrastructure and management plan able to easily monitor, coordinate, implement, test and maintain the project as required.