Application for GSoC 2017 | Thilanka Munasinghe
IoT: Android Things Integration on the MIT App Inventor


This proposal is available at: http://thilankam.github.io/gsoc17.html



Interest in App Inventor

I first learnt of MIT App Inventor when I was a visiting student at MIT in 2014. I have been using MIT App Inventor ever since: to prototype android apps for my research, and as the introductory mobile app development platform in the Android Programming class I teach at WVU.

Last summer I was lucky to be part of Google Summer of Code 2016, where I successfully completed the Raspberry Pi integration on the MIT App Inventor platform. I would like to contribute again this summer and extend the IoT capabilities using the Android Things SDK.

Interest in introductory programming

I have been teaching at the WVU LaunchLab as a codelab instructor for the past 2.5 years. I teach students who do not have prior mobile coding knowledge or even any programming experience to develop their own mobile apps. In my classroom sessions, I have successfully utilized MIT App Inventor to teach the fundamentals before delving into the Android Platform internals.

Here are some of the lesson plans I have used, some of which modified from the official MIT App Inventor tutorials: lesson 1, lesson 2, lesson 3, lesson 4, and lesson 5.

Proposed summer project
(IoT: Android Things Integration On The MIT App Inventor)

I am interested in the Internet of Things project listed on the appinventor-sources wiki. Specifically, I would like to incorporate the Android Things SDK to MIT App Inventor Platform. Android Things was unveiled fairly recently (Developer Preview 1 in December 2016, and Developer Preview 2 in February 2017). Thus, application development utilizing Android Things has just begun by experienced developers, but using the the SDK is not yet simple enough for a typical MIT App Inventor user such as a high school student.

Therefore, the goal of this Summer of Code 2017 project is to investigate ways to integrate the Google Things API features into MIT App Inventor, and contribute an MIT App Inventor extension component supporting Android Things to build IoT Android apps to the vast number of people worldwide.

Why me?

I completed my bachelors in Aerospace Engineering, my Masters in Mechanical Engineering, and I am currently pursuing a second Masters in Applied Mathematics at the West Virginia University. Coming from a multi disciplinary background, IoT is a special interest of mine. I have done several IoT hobbyist projects for fun and as part of the android programming classes I teach at WVU. In addition, I have been using App Inventor in my introductory app development classes at WVU codelab. As part of my graduate studies, I have conducted lot of research on autonomous vehicles, which has even led to several journal publications as listed in my resume.

More importantly, as was mentioned before, I contributed the Raspberry Pi Integration for MIT App Inventor last summer. The experience from working on that project has situated me to be the best candidate to work on the Android Things integration on MIT App Inventor. Through this Google Summer of Code project, I hope to contribute good quality code to the MIT App Inventor project, and also hope this project will result in a research worthy outcome that will lead to a publication.

Benefit to MIT App Inventor

Since there are many famously known starter kits (Adafruit Project Kit, Sparkfun Project Kit) as featured in the official Google Android Things developer kits site, a direct integration of the Android Things will open the door for large number of school students, novel experimentalists, enthusiasts, and hobbyists to be able to do IoT projects along with App Inventor. Android Things also support different boards including Raspberry Pi, Intel Edison, Intel Joule, and NXP Pico via a common interface. Thus, the integration will supersede the capabilities offered by previous Arduino and Raspberry Pi integrations on MIT App Inventor.

As was mentioned before, the Android Things API is relatively new compared to other APIs. The Developer Preview 2 was only released in February 2017 for developers and early adopters to use for development and compatibility testing on supported hardware platforms. If MIT App Inventor becomes one of the early adopters and supporting platforms for the Android Things, it will hopefully lead to many interesting easily programmable DIY projects that are integrated with mobile phones.

Special Note about Android Things License

Android Things is part of the Android SDK, thus licensed under Apache 2. Therefore it will be compatible with the App Inventor license. This will enable the App Inventor user community to develop Android based IoT applications without any licensing implications, as was the case with integrating with Raspberry Pi, since the Raspberry Pi integration software relied on a GPL licensed Pi4J library.

Project Overview

Internet of Things (IoT) is a fast growing field of where sensors are monitoring appliances, actuators, and machines, as well as controlling them remotely, over the Internet. It has helped to create pervasive smart things that integrate computation into everyday environments and link them through Web and Cloud applications. This evolution is impacting our lifestyle and thus quickly and easily creating applications to leverage full potential is of paramount importance. This project is an effort to realize that goal, by integrating the Android Things SDK on the MIT App Inventor platform.

I am proposing to develop an MIT App Inventor Extension that integrates Android Things. This project will be different from the last Summer of Code project I did on integrating Raspberry Pi on the App Inventor, because the Android Things API will allow a generic interface to several different types of IoT platforms, including Arduino, Raspberry Pi, NXP Pico, Intel Edison and Joule.

Supporting the Basic IoT Capabilities from MIT App Inventor

Android Things extends the core Android framework with additional APIs that allow apps to integrate with new types of hardware not found on mobile devices. Apps for embedded devices bring developers closer to hardware peripherals and drivers than phones and tablets. However, support for designing apps that extend the capabilities of IoT is not yet fully supported in MIT App Inventor in a device agnostic manner (the only available integrations being the Arduino and the Raspberry Pi integration I did last summer). This project will aim to bridge that gap by integrating the the Android Things API to MIT App Inventor.

The specific goals of this project is as follows:

  • Control Devices attached to a given GPIO pin.
  • Serial communication in Device to a display port; for e.g. with UART (Serial/RS232) or SPI (Serial Peripheral Interface).
  • Integrate Communication Protocol capabilities with the MQTT protocol and Google Weave.
  • Based on the above criteria, the necessary properties and methods will be added to the MIT AI Things Extension Component.

    Testing and Evaluating the Component Developed

    Since IoT applications space is very broad, I would like to develop several sample applications using the newly created Things Extension component. I have identified two different applications that I can develop. I am hoping to take an iterative approach to refine the component as new requirements get elicited when these sample apps are created.

    • Weather Station App
    • Home Automation App
    • Plant Watering System
    • Basement/Storage Unit Flood Detector

    Implementation Plan

    The following diagram shows the rough time-line I have allocated to various aspects of the project. This is by no means the final time allocation, and with the guidance of the mentors I hope to update this.

    Experience with the development tools

    Javascript

    The Javascript project that I am the most proud of is FlightR. I completed this project as part of a hackathon, and won the Most Technically Interesting Project Award at the Big Travel Data Hackathon 2013, organized by Hack Reduce. The project uses Javascript and the d3.js library to visualize actual flight ticket prices and the search results for tickets in a given time period. It is designed to be user friendly and interactive! The tool provides a windowing method to correlate the date of the flight to the price and the time of the day it was taken. (source code)

    Android Development with the Java SDK

    I was an intern android developer for MyLingoApp, a mobile app that lets movie goers enjoy their theater experience in their language of choice. As part of this internship, I got the opportunity to work on Android Material Design, downloading large file sizes, and Android NDK for audio syncing. Furthermore, I have also developed, though not yet fully completed, an android app for making Selfie Rap Videos (source code).

    Experience with teams, online developer communities and large code bases

    Last summer I was able to comfortably navigate the App Inventor code base, and understand the different pieces to successfully complete the Raspberry Pi integration (many thanks to my mentor William Byrne for his help during that ramp up phase!). I am much more comfortable with App Inventor now, and I believe I should be able to start working on my project from day one.

    Non Trivial Apps Developed

    Sample Apps using the MIT App Inventor Raspberry Pi Components

    I have created two samples apps to test the Raspberry Pi Components that I contributed to MIT App Inventor last summer. The first app is a Remote Controlled Home Appliance such as a light that can be remotely controlled with an MIT App Inventor app. The second app is Sensor Controlled Appliance, that will turn on or off based on a certain sensor input. Please see the comprehensive documentation for more details.

    I am in Class

    An app that puts the phone to silent mode as soon as the user toggles a switch, or enters a classroom as specified by a certain geo fence. For any message that is received during the period, and automatic message as customized by the user will be sent.

    | AIA | APK |

    Record My Location

    This app helps drivers remember where they parked in a busy parking lot. Simply press remember location, and upon return to the parking lot, the app will display the parked location on a map.

    | AIA | APK |

    Let's Go Mountaineers

    A simple app to record "cheering chants" during big game day using text to speech, and play back to create nice sound effects when the played back in large groups.

    | AIA | APK |

    Mesh WV Game

    A simple game.

    | AIA | APK |

    App Inventor Design Challenge

    useFront Property

    According to this issue, "[the useFront] property relied on an undocumented feature of the android camera system". This feature has been removed from nearly all androids, including older ones including 2.3 devices. There exists no intent to specify which camera to use, and the user should decide what camera to use. Furthermore, according to this comment, App Inventor has always used an intent to launch the external camera app, and that intent does not have any way of specifying, and it definitely cannot query device state. Therefore, this property was removed from the AI design and block interfaces.

    Design Solution

    Properties To Be Added: frontCamera, backCamera, (and externalCamera if connected).

    Methods To Be Added: IsSelected, TakePicture (already there)

    Implementation

    Due to the limitations across different APIs and devices in how selecting the camera upon intent passing, we might have to rewrite the existing Camera component. The source code for this component is available at: appinventor/components/src/com/google/appinventor/components/runtime/Camera.java.

    According to the Camera API documentation, we can utilize the open(int id) method of the Camera class, where id refers to the camera the user has specified in the property list.

    The SimpleFunction TakePicture() in the existing Camera AI component should be modified such that it would call camera = Camera.open(findFrontFacingCamera()) if the frontCamera property is set, where the findFrontFacingCamera is implemented as follows. Then we can call the Android API's camera.takePicture() method instead of launching the camera app via the intent.

    Note: This is by no means the complete code, and I probably have overlooked some important things, but I sure hope this is in the right direction to achieve what is requested in the challenge.

                        
                        private int findFrontFacingCamera() {
                            int cameraId = -1;
                            // Search for the front facing camera
                            int numberOfCameras = Camera.getNumberOfCameras();
                            for (int i = 0; i < numberOfCameras; i++) {
                              CameraInfo info = new CameraInfo();
                              Camera.getCameraInfo(i, info);
                              if (info.facing == CameraInfo.CAMERA_FACING_FRONT) {
                                cameraId = i;
                                break;
                              }
                            }
                            return cameraId;
                          }