Tutorial 2022:3 \\ \\ \\ ====== Furhat robot basics ====== ===== Introduction ===== The [[https://furhatrobotics.com/furhat-robot/|Furhat]] is a life-size robot head on a fixed base. It uses a small internal projector to display an animated face on the outer shell. This allows quite a realistic human face and very detailed facial expressions. The robot has no body and limbs, so it cannot do animations and gestures. However, it can engage with users through gaze, speech and facial expressions. It is ideal for user studies involving face-to-face conversations. Furhat is programmed through an SDK and API using the [[https://kotlinlang.org/docs/reference/|Kotlin language]] and has [[https://docs.furhat.io/getting_started/|excellent documentation]]. It supports speech recognition, natural language processing, dialogue models and text-to-speech synthesis, among other things. The SDK comes equipped with a programmable "wizard of oz" interface for remote controlling aspects of the robot, which is ideal for user studies. The SDK includes a virtual version of the Furhat robot, which supports many of the real robot's features. ===== Installing dependencies ===== ==== Basic - Java JRE 8 ==== If you are only going to use the remote control functions of the robot, this step is sufficient. Before you can use the SDK, please download and install [[https://www.oracle.com/java/technologies/javase-jre8-downloads.html|Java JRE 8]]. Make sure that you download the correct version for your OS. If you are using Windows, you need the x64 ''jre-8uXXX-windows-x64.exe'' version. Note that you need to log in with an Oracle account to download the JRE. ==== Advanced - IntelliJ IDEA + Java JDK 8 ==== If you are going to create more advanced programs for the robot, you will need to install the [[https://www.oracle.com/java/technologies/javase/javase-jdk8-downloads.html|Java JDK 8]] instead. Make sure that you download the correct version for your OS. If you are using Windows, you will need the x64 ''jdk-8uXXX-windows-x64.exe'' version. Note that you need to log in with an Oracle account to download the JDK. {{:images:robots:furhat:download-java-jre.png?450|}} {{:images:robots:furhat:download-java-jdk.png?400|}} After installing the JDK, make sure you also update your [[https://docs.oracle.com/javase/8/docs/technotes/guides/install/windows_jdk_install.html#BABGDJFH|PATH environment variable]] so that it includes the ''java'' and ''javac'' executables located in the ''/bin/'' directory of the Java installation. After installing the JDK, you need to install [[https://www.jetbrains.com/idea/download/|IntelliJ IDEA]], an Integrated Development Environment (IDE) for writing the code. Make sure you download and install the ''Community'' version for your OS. After launching IntelliJ for the first time, it shows a ''Welcome'' screen. You are now ready to create your first skill for the robot. {{:images:robots:furhat:download-intellij.png?400}} {{:images:robots:furhat:intellij-welcome.png?400}} ==== Installing the SDK ==== Follow the [[https://docs.furhat.io/getting_started/|installation process]] outlined in the official documentation. We highlight some important steps below. You must first create a developer account and [[https://furhat.io/|request access to the SDK]]. Mention that you are from the University of Twente and are following a course that uses the Furhat. Do not wait till the last moment as it may take some time for the Furhat team to process your request. {{:images:robots:furhat:request-sdk-access.png?400}} You will then receive an e-mail with an invitation to register a developer account. Follow the steps on the registration page. When you have created your account, use it to log in and [[https://furhat.io/downloads|download]] the latest version of the SDK for your operating system. {{:images:robots:furhat:download.png?400}} Unzip the contents of the file ''Furhat-SDK-xxx-1.xx.x.zip'' somewhere to your hard drive, for example in ''C:/Furhat''. Remember the location, we will refer to it as ''''. The install directory should look similar to this: {{:images:robots:furhat:unzipped.png?400}} ==== Running the SDK ==== > Note: The new Furhat SDK Desktop Launcher will open graphically. In the app, you click ''launch SDK''. If you are using an older version, proceed as described below. Open a terminal and navigate to the ''''. On Windows, open the start menu, type ''cmd.exe'' and press enter. This will open a command prompt. Type ''cd '' and press enter to change to the Furhat installation directory. For example, if you unzipped the SDK in ''C:/Furhat'' you would type ''cd C:/Furhat'' in the command window. Then, type ''dir'' and press enter to verify that you are in the correct directory (this will print all folders and files). {{:images:robots:furhat:open-cmd.png?400}} {{:images:robots:furhat:cd-dir.png?400}} On Windows launch the SDK server by typing ''launchSDK.bat'' and press enter. On Linux and MacOS type ''sh launchSDK.sh''. The terminal will now print a lot of information while it attempts to launch the SDK. (If you encounter an error at this step, see **troubleshooting** below before you continue) If all goes well, your terminal window should print something along the lines of ''[Controller] INFO WebServer: 30 - Web interface available on http://localhost:8080/''. If you changed the default port as part of troubleshooting, it should print the new port number (for example, ''8081''). It will also automatically open a virtual Furhat window. {{:images:robots:furhat:running.png?400|}} {{:images:robots:furhat:virtual-furhat.png?800|}} \\ \\ \\ \\ ==== Connecting to the web interface ==== Open a browser and navigate to the webpage that was printed in the terminal while launching the SDK. By default, this is ''http://localhost:8080'' (If you modified the default port as part of troubleshooting, you will need to use the new port number instead). You will now see a login window, use password ''admin'' to login. {{:images:robots:furhat:web-interface-login.png?400}} ==== Basic - Remote controlling the robot ==== You can use the web interface to remote control the robot's gaze, speech and various facial expressions. For a basic user study this may be sufficient. > Note: The files below are gifs, double click on them to see the animations {{images:robots:furhat:gaze.gif?400}} {{images:robots:furhat:speaking.gif?400}} {{images:robots:furhat:head-roll.gif?400}} {{images:robots:furhat:expressions.gif?400}} You can change the Furhat's default text-to-speech language and voice in the settings. Usually, the ''Neural'' variants tend to sound slightly better. {{images:robots:furhat:settings-voice.png?400}} {{images:robots:furhat:settings-voice-2.png?400}} ==== Advanced - Making your first skill ==== In case you are not be able to work with the physical robot. Instead, you can use the virtual robot that is part of the SDK. It supports most of the interaction capabilities of the real robot: interactive gaze, facial expressions, speech recognition, speech synthesis. It does not support camera vision for detecting the user, but you can add virtual users through the SDK web interface instead. Programs that run on the robot are called ''skills''. Before you continue, make sure you have installed all the requirements first. There is an excellent [[https://docs.furhat.io/tutorials/your_first_skill/|tutorial]] in the official documentation for creating your first skill. There is also more [[https://docs.furhat.io/skills/#creating-a-skill|in-depth documentation about skills]]. Here we highlight some of the important steps to help you get started. ==== Creating the skill ==== > Note: The new Furhat SDK Desktop Launcher makes it easy to create a new skill from within the SDK Launcher. Just click ''Create Skill project'', and select a location for your project. Then proceed to the section Importing the skill in IntelliJ. The Furhat SDK includes a command line tool for generating a new empty skill. Open a terminal (on Windows, open a command prompt ''cmd.exe'') and navigate to your SDK installation folder ''cd '', then type ''gradlew createSkill --name=GettingStarted --folder=skills''. This will first download and install all dependencies, then it will generate an empty project that you can import in your IntelliJ. The generated project files will be located under ''/skills/GettingStarted''. {{images:robots:furhat:gradlew-createskill.png?400}} {{images:robots:furhat:generated-skill.png?400}} ==== Importing the skill in IntelliJ ==== The first time you use IntelliJ it will show the ''Welcome'' screen. From there, select ''Open or Import''. Next time you use IntelliJ you can open a project by using ''File -> Open'' from the main menu. Select the ''build.gradle'' that was generated as part of the previous step. For example, this file could be located in ''C:/Furhat/skills/GettingStarted/build.gradle''. Press OK and then select ''Open as Project''. {{images:robots:furhat:intellij-import.png?400}} {{images:robots:furhat:intellij-import-2.png?400}} {{images:robots:furhat:intellij-import-3.png?400}} IntelliJ will now import the project and download additional dependencies. Wait for the process to finish, this may take a while. {{images:robots:furhat:intellij-import-4.png?400}} ==== Configure the project in IntelliJ ==== After importing your project you should [[https://docs.furhat.io/skills/#setting-up-intellij-idea|configure your IntelliJ to use Kotlin]]. Make sure the Kotlin JVM version is used: from the main menu ''File -> Settings (or Preferences on Mac) -> Build, Execution, Deployment -> Compiler -> Kotlin Compiler'' and set the `Target JVM version field to ''1.8''. {{images:robots:furhat:kotlin-jvm.png?400}} Also configure the Kotlin working directory for the default run configuration template: from the main menu ''Run -> Edit Configurations -> Templates -> Kotlin'' and enter ''$MODULE_DIR$'' in the ''Working Directory'' field. {{images:robots:furhat:kotlin-working-dir.png?400}} ==== Running the skill ==== First, make sure the Furhat SDK is running. The source code for the skill is located in the IntelliJ Project Explorer: ''GettingStarted/src/main/kotlin/furhatos.app.gettingstarted''. Double click the ''main.kt'' file to open it in your editor. {{images:robots:furhat:intellij-run.png?400}} Running for the first time: press the ''Run'' button (the green triangle) next to the ''main'' function and select ''Run 'MainKt' ''. This generates a run configuration based on the ''Kotlin'' template that was configured above. (If the ''Run'' arrow is not there, or if you encounter errors, check troubleshooting below) {{images:robots:furhat:intellij-run-2.png?400}} {{images:robots:furhat:intellij-run-3.png?400}} After doing this once, you can run the script using the generic ''Run'' button at the top of the screen. {{images:robots:furhat:intellij-run-4.png?400}} The Kotlin source code is first compiled, and then it will upload the skill to the robot. You can view the progress in the lower half of the IntelliJ editor. {{images:robots:furhat:intellij-running.png?400}} Now, open the Furhat SDK web interface, which is usually http://localhost:8080. Go to the ''Dashboard'' in the left menu. This will show that the current skill is running on the robot. As you can see, there are two zones around Furhat (shown as grey circles in the situation view). When the user enters the inner zone, it is deemed to enter the interaction, and Furhat will start interacting with the user. To simulate this in the SDK, you can double-click on the situation view somewhere inside Furhat's inner interaction space to add a virtual user. This should trigger the interaction and allow you to test it. {{images:robots:furhat:skill-running.png?400}} {{images:robots:furhat:skill-running.gif?400}} ==== Next steps ==== From here, you can [[https://docs.furhat.io/tutorials/your_first_skill/#inspecting-the-skill-package|inspect how this skill works]] and you can [[https://docs.furhat.io/tutorials/interaction/|start creating a more advanced interaction]]. Also, make sure to check out the various [[https://github.com/FurhatRobotics/example-skills/|example skills]]. ==== Troubleshooting ==== ===== UnsupportedCharsetException: cp65001 ===== You may get this error when you run the SDK. This is not something to worry about, you can ignore the error. It has to do with a default character set encoding not being configured correctly in ''sun.stdout.encoding''. ===== ERROR WebServer ===== If you get an error like ''ERROR WebServer: 43 - Unable to host web interface. Port 8080 is already in use. Please release that port or go and change the port in static/ports.properties, then restart'' that means your computer is already using the SDK's required default port for something else. {{images:robots:furhat:error-port.png?400}} To modify the default port, open the file ''/properties/ports.properties'' in a text editor. For example, you can use ''Notepad.exe'' by clicking on the file with the right mouse button, and select ''Open with...''. Select the `Notepad` app in the list and click OK. {{images:robots:furhat:open-ports.png?400}} {{images:robots:furhat:notepad.png?400}} Modify the default port value for something else, for example `webServerPort=8081`, and make sure to save the file (`Ctrl + s`). {{images:robots:furhat:modify-port.png?400}} Launch the SDK again, following the steps above. Remember: your web interface will now be available on the updated port number, instead of the default `8080`. ===== Green run arrow not available ===== If the green `Run` arrow does not appear next to the `fun main(args: Array)` line in the `main.kt` file, you should try to reload the project files with Gradle. Press the vertical `Gradle` tab at the top right of the IntelliJ editor, this opens the Gradle configuration. Then press the `Reload All Gradle Projects` button. {{images:robots:furhat:gradle-reload.png?400}} ===== Not able to read skill.properties ===== {{images:robots:furhat:intellij-run-error.png?400}} If your skill crashes due to ''not able to read skill.properties'' then ensure that the working directory of your skill is set to the root folder of the skill. Go to ''Run -> Edit configurations'' and edit ''Kotlin -> MainKt'', set the `Working directory` field to ''$MODULE_DIR$''. To fix this for all future skills, make sure to edit your default Run configuration Templates.