Week 14 - Final individual Milestone

For the end of our project, the Geriambience team has created and tested a system that is able to track a user through a virtual bathroom using the skeletal tracking function of the Microsoft Kinect. Three example bathrooms have been modeled, replicating real world bathrooms, all being fitted with Caroma products due to their sponsoring of the overall project. The Kinect is able to track a user and represent them in these virtual bathroom environments through orbs relating to their body parts. The interaction with these environments ranges from simple moving systems to smart, gesture based interactions that the user can perform to alter their bathroom.

Group objectives

Our group had three main objectives throughout the course, although one of these was dropped due to time and technology restraints.
1. Model with high detail a series of three bathrooms. Use Caroma products within these bathrooms.
2.Using the code of the Microsoft Kinect, create a series of interactive elements to place into these bathrooms. These elements will use gesture and positional information taken from the Kinect to register the player in the virtual world and interact with it. This is proof of concept work, and has applications in the real world if it were to be taken further.
3. Create a moving mount for the Kinect system so that it can track the user no matter their position in the bathroom. This is the objective that our group had to stop working on. We kept running into problems while working on this section, including the realisation that finishing this system would take up at least the rest of the semester, and the Adruino work we were doing needed us to order a large number of different parts, and would be very unfeasable to actually setup.

Indivudual Milestones
In order to properly meet the group objectives, I needed to set myself a series of personal milestones to meet. These were things that I both wanted to learn, and needed to learn in order to fully acheive our group goals.

1. C++ coding - I have very limited experience with coding, and it has been something I've wanted to learn more about for a long time. The reason I picked this project to work on over the other ones available is because I saw it as a fantastic way to get some practical experience with coding. This urge to learn about it is why it became my major milestone for the semester.
2. Arduino - Linking with my interest in learning the C++ language was my wanting to learn how to use Adruino kits. I've never used anything like them before, and they seemed like a fantastic prototyping and learning tool, so I was very eager to learn how to use them.
3. Project leadership - When we were choosing teams, Laura asked me to take over for her in the role of group leader. I've worked in groups for major projects before as the leader, and found it to be a good experience so I wasn't upset about this. Working within a group is never an easy task, so I wanted to make sure that as group leader I allowed our group to finish the project and deliver what we set out to do.
4. Presentation of work - After the first milestone submission, I realised that my presentation of my work was severely lacking, so I wanted to then put a lot more effort into presenting the work I had been doing. This also translated over into the group wiki, which I then started making sure all group members were looking after and uploading to.

My contributions
Kinect Interactivity:
I have only ever used a Kinect in a developmental sense briefly once, and it was running through Grasshopper and Rhino. Using C++ to alter the code for the Kinect was completely new to me. I had a very basic knowledge of programming, so I understood Stephen Davey's explanation to myself and Matt about how to setup a new CryEngine Flowgraph node and how to write the code to run them. This was the starting point for our development. I had to sit back and let Matt work through the start of developing our gesture detection node, but once I had seen him do some work on it, I was able to take over and contribute my own parts to it.
Our final node looks like this:

This node took part of one Stephen Davey already had written, which was the top section gathering the walk speed, turn amount, jump, leaning and pointing mechanics. We then added on parts that gave us the output information we needed. The positional elements are the ones with the pink boxes for outputs. These give us the vector coordinates for each of the parts of the users body and allowed us to create a tracking skeleton. The blue boxes are booleans, and show whether the Kinect is tracking someone, and whether they have fallen over. While the white second from the bottom is our gesture detection, and counts a value within the C++ code and outputs that as a float.

The first step we took on developing our system was making sure our tracking system was working properly. This video demonstrates the first joint we put in, the hip joint. It tracks the hip joint of the room, and showed us that the orientation of the axis was different to the Kinect than it was in the CryEngine, as Matt's relative distance away from the kinect (Z axis), changed the balls height (Y axis). This was easy to swap around once we realised.
Once we had this first positional data working, we got to work putting the rest of the person in.
We decided to use balls to represent the parts of the body because we couldn't get an actual person in the game to move.
Throughout this part of the development, Stephen suggested that I record some of our work using ChronoLapse, a program that takes screenshots of your computerscreen every X seconds and then stitches them into a video. Here is the first one I took. It shows me doing some work in C++, then compiling. Most of our time in this part of the project was spent compiling.

We had two focuses for our interaction, gesture based systems and positional based ones. For out positional based systems, we started out by looking at altering something based on the relative position of the users hand and the object itself. This video demonstrates this, by resizing a ball based on how close the hand is to it.
This system was the basis for what our moving bench and moving toilet systems were based on.
The next test we ran was the moving toilet one. I set this one up using a sideways door to simulate a bench. During this stage of the development I looked into the ergonomics of seats. Regular seats have a specific ratio of the persons height to seat height to make it optimal for comfort and not putting strain on a person. This translates into a specific bending angle of the knee, which for a regular seat is slightly over 90 degrees. But for a toilet this angle is shortened to just less than 90, optimally around 80 degrees. This angle is taken when the user is sitting comfortably with their feet planted naturally on the floor infront of them. This system demonstrated below takes the leg height of the person and adjusts the seat to match this ratio.

The area of our interaction that I was most interested in was the temperature control. We used a gesture for this one, which was set up so that if your right forearm was horizontal, your left hand then alters a value by being above or below your waist. This initial test took the value and adjusted the scale of an orb, but this was later changed to a block that scaled in height.

This was the extent of our basic work before we began to impliment it with the work of our bathroom design part of the group. Those guys gave us a fully developed level in which we then brought our flowgraph and linked it up to everything we needed to.
The video below is a timelapse of us setting up some of the interactivity in the finished bathroom.
This next video is one I made as a backup for our presentation. It demonstrates all of the interactive elements of the bathroom, and annotates how they work.

Below are images of our final flowgraph. I'll try to explain what each section does, although it may be difficult to tell just from the pictures.

Full flowgraph

 This section takes all of the output coordinates from the Kinect and creates positional vectors from them to input into the balls that represent the player.

creates positional vectors

 This one assigns those vectors to the balls. I laid it out like a person so we could easily determine which entity related to which body part. This actually became very useful later on when we were using these entity names in other parts of the flowgraph.

assigns the entity positions

These next few all relate to the light controls, and their UI messages.

The math behind the light brightness

 This section shows the area of the flowgraph responsible for the gesture controls for the shower temperature and the UI messages.

Shower temp controls and UI.

As well as working on the Aruino and Kinect/Crysis work, I was incharge of creating the documentation for the programming side of things. This involved the screen captures using ChronoLapse, as well as creating the demonstration videos of our work in progress. I filmed Matt interacting with the Kinect on my phone, while using a screen capture program to record CryEngine. I would then edit these two videos together and upload them to my youtube channel so they were available to the group to use. I also took a number of screenshots of the flowgraph and C++ code to share around.
This all culminated in the final demonstration video that I created incase our system failed on the day of presentation (which it did, but that was a CryDev problem not ours so it was postponed). This video demonstrates all of our interactive elements as well as giving an annotated explanation of them all.

 Individual Development
Interaction development:
For this project the interaction was based both in C++ and in the CryEngine Flowgraph, and so it was essential to learn to properly use both for the logic operations we needed to perform. I learned a lot about the C++ language and its implimentation throughout the course. The main thing I learned was the proper way to actually structure and setup a piece of code in order for it to work. This was specifically important for the Kinect project because unless every part of the new node you try to create is structured properly it won't work.
In addition to this, trying to setup the gestures really showed me some new things with coding that I hadn't ever thought about. For example we used a counter within a piece of code to count how long something was true for. We used refresh frames for the kinect system to measure it, and then a large number of "if" and "then" statements to determine what to do afterwards. This ended up making a huge loop that would reset the counter if it was broken. This sort of logic was something I would never have learned if not for trying to do this sort of gesture detection.

In terms of the CryEngine Flowgraph, I already had a fair amount of experience in using it, but never in conjunction with another system like the Kinect. I had never used actual mathematics within the flowgraph, which in this case was essential to get the system to work because we needed to constantly be calculating new variables. For example to add two vectors you need to use pythagoras' theorem to calculate the new values, which ends up a mess of nodes when implimented in flowgraph.

Working with Matt on the coding side of things was brilliant, because of his huge amount of background knowledge with programming. He helped me to learn a lot through this project, and I definitely wouldn't have been able to do anywhere near as much as I did without him. I found it interesting when we got to the Flowgraph side of things because he kept thinking in terms of coding, which doesn't translate very well to the flowgraph even when it makes logical sense. This is because you can't have things like loops and conditions in the flowgraph like you can with a piece of code. I'm very good with logic, so on a number of occassions I was able to think of ways Matt wouldn't have, purely because I wasn't thinking about them in a programming sense, but in a more general way. A good example of this was using the gate nodes to properly trigger booleans.

In this picture, the gate is the one flowing into the large collection of nodes at the bottom. This turned out to be a good work around to some problems we were having with booleans triggering.

Intellectual Property:
 I chose our group's presentation topic for the group, based on the idea that our project was taking a large number of other people's intellectual property and utilising it for our own gains. I thought it woudl be the most fitting subject for our project, and would be interesting to look into how to properly go about using other peoples property, as well as protecting our own. My area of research for the presentation was into the different types of IP that our group was using that belonged to other people, and how to properly go about using them. This had me look at the licencing agreements for the different IP that we were using, and gave me a good idea of how to use them for non financial gain as well as commerically.  This side of the project also became very relevant to me as I had to investigate the same area for another course weeks after, and this gave me some very specific information to take with me to that project.

Collaboration:
 Our course seems to favour group work, so we were no strangers to collaboration, especially in large groups. For this project, we split our group into two smaller groups, myself and Matt working on the programming and interactivity side, while Laura, Dan and Siyan worked on the modelling and visualisation side.
I've worked with Matt on a number of projects before, and knew from the outset that we would work well together. Our main hinderance with the project was that his computer wouldn't run CryEngine because of a windows 8 limitation on the program, and the fact that working on one piece of code on multiple computers would prove impossible. This saw us coming into uni multiple days a week purely to work together on this project. This turned out to be very benefitial to the both of us, because it let us bounce ideas off each other, and when one of us was struggling with a problem the other would usually be able to help solve it. Having the two of us also meant that one person could test the kinect system while the other was editing it, which proved to save a lot of time.

With the group split in two, we had a very disjointed start to the project, because the first part of each groups work was very unrelated to the other. During this time we had very little contact with the other group, which as the group leader I should have rectified much sooner than I did. However, once we reached the stage where we integrated our Kinect interaction with their level, we became a much more collaborative team.
The design side of the group set up a large amount of detail on the wiki which was amazingly useful during the later stages of the project, especially in setting up our final presentation. They had all the plans for the bathrooms which allowed us to map out where to place everything, as well as deciding upon where the kinect and CryEngine camera view should be placed.

What I would do differently
I'll split this section into two. Firstly, what I would do differently as a team leader, and secondly what I would do differently as a member of the team.

As leader, I feel like I should have taken a much bigger role in organising the team at the beginning of the project. If I had done this it could have saved us a lot of time and allowed us to be much more efficient.   I would have set up a very clear list of deliverables that we expected to have accomplished at the end of the project. As it was, our deliverables changed dramatically over the course of the project, possibly because they weren't locked down at the start.

I would have also made a much bigger effort on team cohesion. The way we split the group into two worked well enough, but it left us not knowing what the other side of the group was doing most of the time. If done again, I would spent a lot more time making sure that the entire group was interacting with each other the whole time, possibly by splitting the roles differently. For example we could have all been involved in little parts of each side on a weekly basis. While this might have slowed things down because we would be all over the place, if it was structured strictly it could have made us work better knowing what everyone was up to at all times.

As an individual within the group there are a number of things I would have done differently as well. Firstly, I would have been much more diligent about documenting my progress in the beginning weeks of the project. The first milestone submission was a good kick in the right direction for this and forced me to become much more aware of recording progress, both through video and written.
Secondly, I would have, if possible, done more research and individual work on the C++ side of things. As it was, I learned most of what I did through watching or learning from Matt. It would have been nice to have a chance to delve a little deeper into it all.


Summary
At the end of this project, we have created a very effective and easy to learn system for controlling different aspects of a virtual bathroom. While this project would only serve as a proof of concept, I feel like it definitely proves that this sort of system could work. It is all very well documented on the wiki, and through the personal blogs of the members, which act as a resource for anyone interested in looking into these sorts of systems. I feel like this is a project we would be able to take to a much higher level were we funded by Caroma to actually try and produce these sorts of systems for real life purposes. I can see these sorts of systems being in everybodies homes in 10 or so years, and it would be fantastic to be part of the reason why they are there.