The end result of this project was the Konnichiwa-Sensei game. The game stands on the back of months of research, development, trial and error into the field of language acquisition and the use of smart home technology/ the Internet of Things. This page details the full timeline of this development and the vast array of problems encountered that led to moving development onto a game proposal.
Development Timeline:
Research begun with investigations into using the Internet of things (IoT) to interact with the various smart devices used in the home. Unfortunately this path had to be dropped later into the project due to unforeseen limitations within the brief. Due to the Covid-19 pandemic, the university was forced to close which had a very direct affect on the development of this project.
Phase 1: The Voice Command Project
The first prototype looked into the application of custom voice command software. Although the original plan would later be scrapped in favour of a game – the use of some kind of voice command software would continue through out the entire project.
This process begun with attempting to alter and function of a brand new Google Home Mini. The objective was to have the device respond to broken Japanese or sentences that use only a few Japanese words within an English sentence. Due to being a complete novice in this area, altering the source code of the pre-built Google Home proved to be too much of a task for the allotted time.
The Google Home runs on the Google Assistant API – this can be downloaded onto other devices and used to turn nearly anything into a DIY Home Assistant; which is well suited to the goals of the Other Today studio. Because of this, the next stage in development was to attempt to install this API onto a Raspberry Pi and code just a few phrases to operate the smart light.
Google Home Mini
A Raspberry Pi model B+ was used for this leg of the research. This model requires a wired ethernet connection and direct input to any arduino devices. More expensive models, such as the Raspberry Pi 3 model B+ have built in wifi and bluetooth. While this is a proven technology – with multiple guides available to show the process of at least installing the API – due to this project being made within the university; the Raspberry pi wasn’t able to connect to the public wifi of the university. This is due to security concerns and restrictions in the universities IT infrastructure and the 2 factor authentication used by the “Eduroam” network, requiring both connection to the network, but also a logging in with a unique username and password.
Raspberry Pi B+
The use of an external mobile hotspot was available, but due to the potential unreliability of this system on the day of the summative assessment and the additional cost; this option was never pursued.
Raspberry Pi :
For this project, a Raspberry Pi model B+ was used. This model was chosen as to not limit potential users of the software due to cost or the effort of needing to go out and buy the latest model of something they already own; causing more E-waste. This is also an important consideration for the Other Today studio.
Connection issues aside, the Raspberry Pi itself provided a plethora of technical challenges, partly caused by the choice to use an older model. The most problematic of which was how long the Pi took to load Chromium. The exact reason for this was never identified but it’s suspected to be because of chromums heave usage of the onboard ram.
This problem could have been bypassed using a Virtual Network Connection (VNC), during this phase of the project the university was still open and one of the deliverable was that the product had to function within the university. Due to the connection problems detailed above, making use of a Virtual Network Connection as a work around wasn’t a viable solution.
Voice Command Software:
Several different existing Voice Command softwares have been developed and are free to use for projects just like this. An effort was made to install each of these onto the Raspberry Pi at home, where a proper internet connection could be established.
The pro’s and con’s of each of these – and of course the problems faced – are detailed below.
Jasper Voice Recognition:
Benefits:
- Works without Wifi, making it the ONLY viable choice for this project before the Covid-19 lockdown.
Issues:
- Complicated and outdated setup.
- Requires some knowledge of linux commands and configuration to install.
- Unknown error during installation of the Sphinxbase Text to Speech program – This may be related to the age of the software,
Link: https://jasperproject.github.io/documentation/
Steven Hickson Voice Recognition:
Benefits:
- Highly customisable allowing for custom inputs – which is important when dealing with multiple language recognition software.
- Far fewer steps to install than the Jasper program.
Issues:
- Encountered unknown issue the Voice Commands dependency, crippling the entire program.
- The documentation was outdated and unclear.
- Requires an internet connection.
link: https://stevenhickson.blogspot.com/2013/05/voice-command-v20-for-raspberry-pi.html
Google Assistant on the Raspberry Pi.
Benefits:
- Large amounts of online documentation.
- Supported by Google.
Issues:
- Runs via Google Assistant API which may cause some data privacy concerns.
- Cannot be installed using the only the Linux command line. Chromium, another web browser or a VNC is required.
- An internet connection is required for access to the API.
Links: https://developers.google.com/assistant/sdk/guides/library/python
https://www.instructables.com/id/Raspberry-Pi-DIY-Google-Assistant-With-Sleek-Wood-/
Based on this comparison, Jasper was the obvious choice to meet the deliverables. However, due to having installation problems with all of these programmes, the entire project moved onto focus more on the development of a mobile app.
Phase 2: App Development
The original wireframe can be seen here.
Phase 3: Arduino
Controllers are expensive – Why not make your own?
The project needed a physical product to be produced alongside the software elements. To meet this requirement, an arduino board and some unsoldered electronic parts were used to develop a “Distributed Design” controller that nearly anyone can build from home. This would later evolve into a full “Controller Package” available alongside the Konnichiwa-Sensei game.
Details on the development of the final prototype can be seen on the Controller Specifications page, but the development journey is shown below.
Controller 1: Joystick
The first iteration of this controller was very strict to the idea that “It must be makeable by anyone, anywhere!” Using purely unaltered arts attached with some duct tape.
Controller 2: On-board Microphone
The second iteration brought in an on-board microphone that would allow for the games speech based gameplay.
This version was used to prototype the initial concept that would eventually become the 3D printed “Scaffolding” around the final version, which hold the parts together more securely and neatly. However, the onboard microphone would be scrapped as the Unity 5 script used in the game works with any PC audio input.
Prototyping of the scaffolding started with simple cardboard templates to find a good size and shape.
After cardboard prototyping, a 3D printable external shell was developed using SolidWorks. The first 2 versions were open top variants that served an ergonomic purpose, but didn’t hold any of the electronics in place.
the third iteration added a top part that would help hold the loose electronics in place, improve usability of the buttons and hide the wires from view.
This evolved over several iteration into the final model. Details on how the 3D printer filament was selected can be found on the Controller Specification page.
Phase 3: Konnichiwa-Sensei
The concept of the video game was a direct response to the difficulties faced with the Raspberry Pi.
A video game bypassed the need for most complex hardware planned for the voice commands project – though a lot of it would still appear in the controller.
The public facing proposal can be found here:
Details on the technical specifications of the Konnichiwa-Sensei game can be found on the Game Specification page.
