Android Instrumentation Using Aspect-oriented Programming
ANDROID INSTRUMENTATION
ABSTRACT
Android codebase has been changing extensively over the last few years to support a wide range of mobile devices and attractive features. Although it is an open source project, due to its dynamic nature and continuous growth, there is an ever-increasing need for a tool that facilitates developers to instrument the code at the platform level and aid them in analysis of the source code by providing handles into the platform. The objective of this project is to develop a tool that provides the developers with the ability to instrument the Android platform using Aspect-Oriented Programming.
The Instrumentation tool enables its users to inject custom Java-based code into Android Open Source Project (AOSP) which can be very useful in many situations such as providing the developers with better insights of Android source code, understanding method and class usage, monitoring the behavior of applications and checking any security violations by Android apps. Furthermore, the tool can also be extended and customized to meet the individual needs.
I certify that the Abstract is a correct representation of the content of this thesis.
Chair, Thesis Committee Date
iv
PREFACE AND/OR ACKNOWLEDGEMENTS
This research project was supported and supervised by Dr. Arno Puder, whom I would like to gratefully and sincerely thank for his guidance, encouragement, and patience during my study at San Francisco State University. Also, I would like to thank my peer reviewers Syed Omer Khureshi, Sai Krishna Undurthi and Anshul Vyas from San Francisco State University.
I would like also to thank Dr. William Hsu for reviewing my report in his capacity as the second member of the committee.
v
Table of Contents
1. Introduction
1.1 Motivation
1.2 Objective
1.3 Contribution
1.4 Organization of Document
2. Use Cases
3. Background & Related work
3.1 Android Open Source Project(AOSP)
3.2 Android Architecture
3.2 Android Build System
3.2.1 Configuration
3.2.2 Function Definitions
3.2.3 Cleaning
3.2.4 Module Build Templates
3.2.5 Output
3.2.6 Build Recipes
3.2.7 Compiling with Jack and Jill
3.3 Aspect Oriented Programming
3.3.1 AspectJ Library
3.4 Byte Code Engineering Library (BCEL)
vi
4. Implementation Process
4.1 Build Flow Modification
4.2 Weaving Custom Code
4.2.1 Using Bytecode Manipulation
4.2.2 Using Aspect Oriented Programming
5. Demonstration
5.1 Launcher Process
5.2 Activity Manager
5.3 Zygote
5.4 Application Process
5.4.1 Android Platform Variations
5.4.1.1 Gingerbread (2.3.7)…………………………………………..
5.4.1.2 Gingerbread (2.3.7) with Android support library
5.4.1.3 Nougat (7.1.1)……………………………………………….
5.4.1.4 Nougat (7.1.1) with support library……………………………………
5.4.1.5 Comparison of the results of analysis on Android platform variations………………..
6. Summary and Conclusion
6.1 Project Summary
6.2 Limitations
6.3 Conclusion
7. References
8. Appendix
Steps to instrument and build Android
vii
List of Figures
Figure 1: Working of the Instrumentation tool
Figure 2: Android architecture
Figure 3: Working of Zygote process
Figure 4: Android build architecture
Figure 5: Jack toolchain structure
Figure 6: Jack library format
Figure 7: Android Build flow with Jack and Jill toolchain.
Figure 8: Android build process
Figure 9: Android build process with AspectJ instrumentation
Figure 10: Application startup process
Figure 11: Call-stack graph of the APIs invoked by launcher process
Figure 12: Call-stack graph of the APIs invoked by Activity Manager
Figure 13: Call-stack graph of the APIs called by Zygote Process
Figure 14: Call-stack graph of the APIs invoked during application startup on Gingerbread
Figure 15: Comparison of the results of analysis on different versions of Android
viii
List of Tables
Table 1: Directory contents of Android Open Source Project(AOSP)
Table 2: Set of values that can be provided for TARGET_PRODUCT in the Android build
Table 3: List of functions added by build/envsetup.sh script
Table 4: Configuration information set up by lunch for generic-eng combo
Table 5: List of module build templates and their corresponding .mk files in AOSP
Table 6: Comparison of custom code weaving approaches: Bytecode manipulation and
Aspect-oriented programming
Table 7: List of processes observed at the application startup
Table 8: Metrics used in the analysis of a call-stack graph
Table 9: Graph analysis of the call-stack graph of application process on Gingerbread (2.3.7)
Table 10: Graph analysis of call-stack graph of the application process on Gingerbread (2.3.7) with Android support library
Table 11: Graph analysis of call-stack graph of the application process on Nougat (7.1.1)
Table 12: Graph analysis of the application startup sequence on Nougat (7.1.1) with support library
Table 13: Percentage comparison of the results of analysis on different versions of Android
ix
List of Code snippets
Code snippet 1: Makefile of AOSP
Code snippet 2: List of choices offered by lunch command on Gingerbread
Code snippet 3: Part of the code in envseup.sh which adds lunch combinations…………..
Code snippet 4: vendorscript.sh script in Gingerbread 2.3.7………………..
Code snippet 5: Example of AspectJ around advice
Code snippet 6: Changes made in build/core/definitions.mk file to support
custom weaving……………………………………………………
Code snippet 7: Program to insert log statements at the beginning of every method in a class using BCEL library
Code snippet 8: Program to insert log statements at the beginning of every method in a class using AspectJ library
Code snippet 9: Content of post-compile.sh script
Code snippet 10: AspectJ program to add enter/exit logs in every method of AOSP
x
CHAPTER 1
Introduction
This chapter outlines the objective and the motivation behind this project and explains the organization of this document.
1.1 Motivation
The idea for this project first formed when we learned about an interesting situation involving a law suit targeting a major Android app on the PlayStore. The allegation was that the app was accessing private information in situations where it didn’t require such access and was misusing the information collected. This called for the analysis of the app to verify if such claims were valid. This was not an easy task, more importantly, many other situations arise where we are left wondering what sort of information an app might be privy to. Since the access to source is not available, it was very difficult, almost impossible, to accurately ascertain if the app accesses the private information. Furthermore, there are other major hurdles such as,
- No Access to handles in the platform for APIs accessing private information
Android framework provides multiple ways to access the same piece of information, making it difficult to determine if the information was accessed. There was a need to have a handle in the platform to detect when such APIs are invoked to make the illegal accesses to private information more visible for analysis.
- App level visibility is insufficient
Although the current Android permission model allows a user to control what information/resources an app can access, it does not enable the users to control how frequent the access is or what the app does with the private information. This level of control requires platform level visibility.
- Lack of access to source code
Access to Android application source code is not always available. In fact, often, companies do not make source code open. This leaves us with the only option to perform analysis on the bytecode.
- Lack of existing tool
Although there are tools available in the market that perform analysis at the application level, they do not provide flexibility and visibility needed for a conclusive and definitive answer to the question of whether the app is misusing its permissions. No tool is available for analyzing the APIs at the platform level.
To summarize, there was a need for a tool that could resolve the mentioned hurdles above and since we could not find it, we started to develop our own tool which could instrument the Android platform with which it would be possible to insert handles into the platform.
1.2 Objective
The aim of the project is to provide an end-to-end tool that enables users to inject any custom Java code based on user specified constraints into the Android platform using Aspect Oriented Programming. Figure 1 explains the objective of the project.
Figure 1: Working of the Instrumentation tool
The end user can write the custom Java code that they want to execute every time a condition is met in the base code which is the Android Open Source Project (AOSP). The user can then use the Instrumentation tool to inject the custom code into the base code. The Instrumentation tool weaves the custom code into the base code through the build process and generates a custom emulator i.e. a final executable image of the AOSP. The custom emulator can be used by the end users to execute the APKs and see the output of their custom code every time the condition specified by them is met.
1.3 Contribution
- Custom code injection
The Instrumentation tool provides an ability to inject a custom code into Android Open Source Project that can be very useful in many ways such as understanding APIs and class usage in AOSP, profiling applications and monitoring behavior of applications. There is no need to generate an emulator image every time. Preconfigured emulator images can be reused to analyze multiple apps.
- Support for multiple versions of Android
The Instrumentation tool supports the different versions of Android. It has been specifically developed for all the versions from Gingerbread (2.3.3) to the latest version Nougat (7.1.1).
- Consolidated documentation to understand AOSP and Android Build system.
Though there are several documents available that provide information about Android Open Source Project and Android build system, the information is fragmented. The document consolidates all the relevant information in one place.
4. Environment setup guide
The appendix attached at the end of the document provides step by step instructions on how to download, build and generate a custom emulator with the Instrumentation tool.
- Framework for other projects
The Instrumentation tool provides a necessary framework for other projects like Android best practices verification tool by Sai Undurthi, security inspection tool by Anshul Vyas and visualization tool by Syed Omer Khureshi.
- A detailed analysis of application startup process
The demonstration of the Instrumentation tool provides an insight into the sequence of actions that take place when an app is launched on the Android OS. An analysis of the application startup process is carried out on different versions of Android with different configurations, which helps in understanding the phenomenal growth of Android over the last few years and impact of Android support library on the application startup process.
1.4 Organization of Document
Chapter 2 outlines the use cases of the tool. Chapter 3 discusses the relevant background knowledge, including Android Open Source Project (AOSP), Android build architecture and Aspect-oriented programming. Chapter 4 gives a detailed walkthrough of our implementation process to support custom weaving into the AOSP. Chapter 5 provides a demonstration of the tool using a sample app. Finally, Chapter 6 summarizes the project. The appendix provided at the end explains how to get the Android sources from the Android website and how to compile them with custom aspects to generate a functional emulator image.
CHAPTER 2
Use Cases
In this chapter, we present four use-cases of the Instrumentation tool which highlight its applications.
- Understanding Android APIs and features
Sydney’s team has a published app on Android PlayStore which is designed to detect earthquakes by using mobile sensors. The app uses a background service to record the sensor data constantly, process it and upload to a server. Starting from Android 6.0 (API level 23), Android has introduced a new feature Standby and App Doze to reduce battery consumption. It does so by deferring background services and network activity for apps when the device is unused for long periods of time. To understand how the Doze and App Standby affects the functionality of the app, she utilizes the functionality of the Instrumentation tool by writing custom aspects to add logs in the relevant Java classes and builds an instrumented emulator image. She then installs her app on the emulator and tests it with a set of test cases. After analyzing the logs generated by the tool, she is able to check how the functionality of the app is affected. She can then modify her app functionality accordingly.
- Profiling Android application
Jen is a part time Android app developer who developed a map application on the PlayStore. She has a crash reporting tool ACRA setup in her app. When she checks the recent crash report, she sees a few crashes due to Application Not Responding(ANR) error. To find out the root cause of the issue, she proceeds to use the Instrumentation tool. With the help of the logs generated by the tool, she realizes that her app is spending more than normal time in the Android callbacks, triggering the ANR error. This makes it easy for her to fix the bug and speed up the execution time of bottlenecks in her software code.
- Ensuring if an application is following best practices.
Kevin is a team lead responsible for Android game development. He has been receiving feedback from the app users about excessive battery drainage when using the app. Having heard about the Instrumentation tool, he writes a set of custom aspects to check if the app follows the best practices suggested by Android. With this analysis, Kevin discovers that the app is violating some of the best practices. He incorporates the use of the tool in his team’s standard testing protocol. The issues such as not unregistering sensor listener and broadcast receivers on onPause(), not stopping background services are detected by the tool early in the testing phase. His team is able to fix the code to ensure the best practices are followed. After a couple of months, the app has got increased number of installations and an incredible rating on the PlayStore.
- Detecting privacy breach in an app
Raymond is an Android enthusiast with an extremely deep understanding of the Android framework. He is contacted by a Law firm that is working on a client’s complaint about a potential privacy breach by a leading social networking app and request Raymond, in his capacity as an Android expert, to ascertain the validity of this claim by providing him access to the app’s APK file. Since he does not have access to the source code of the app, it becomes very difficult to understand the working and to examine the behavior of the app. Being aware of the Instrumentation tool, Raymond writes custom aspects on potential APIs that can access private information like contacts, location, etc. and runs the app on an instrumented VM instance. By analyzing the generated logs, he is able to understand the flow of data in the app. He provides his findings to the Law firm.
As can be observed from the presented use cases, there is a need to instrument the Android platform when the source code of Android apps is not accessible, or the application level access is insufficient. In such scenarios, the Instrumentation tool can be used with minimal efforts.
CHAPTER 3
Background & Related work
This chapter introduces the relevant background knowledge for our project by providing a brief explanation of the key concepts related to the project. Section 3.1 gives an overview of the Android Open Source Project (AOSP), whereas section 3.2 explains the Android architecture. Section 3.2 starts with the introduction of Android build system and then digs into Android’s internals such as Android build architecture, build configuration parameters, module template, reusable functions and the newly introduced build toolchain ‘Jack and Jill’. Section 3.3 discusses Aspect-Oriented Programming paradigm. Section 3.4 covers a high-level overview of Apache’s Byte Code Engineering Library (BCEL) used for bytecode manipulation.
3.1 Android Open Source Project(AOSP)
Android is an open source software project which works on a range of devices. The purpose of Android is to develop an open source platform available for developers and to create a product that enhances the mobile experience for users [5]. AOSP is available for download, customization, experimentation and porting.
Android source can be downloaded and built to get custom Android OS running on a device or an emulator. Step by step instructions on how to download and build Android can be found in Appendix. AOSP is a fairly large project consisting of more than 14,000 directories and 100,000 files in Gingerbread (2.3.7) and more than 1340000 files and 132,000 directories in Nougat (7.1.1).
Table 1 provides an overview of the important directories and their contents in the AOSP project.
Directory | Content |
Abi | Minimal C++ Run-Time Type Information support |
bionic | Android’s custom C library |
bootable | OTA, recovery mechanism and reference bootloader |
build | Build system |
cts | Compatibility Test Suite |
Dalvik | Dalvik VM |
development | Development Tools |
device | Device specific files and components |
docs | Documentation of the source code |
external | External projects imported into the AOSP |
frameworks | Core components such as system services |
hardware | HAL and hardware support libraries |
libcore | Apache Harmony Java source |
ndk | Native Development Kit |
pdk | Platform Development Kit |
prebuilt | Prebuilt binaries, including toolchains |
sdk | Software Development Kit |
system | “Embedded Linux” platform that houses Android |
tools | Various IDE tools |
Table 1: Directory contents of Android Open Source Project(AOSP)
‘prebuilt’ and ‘external’ are the two major directories in the AOSP tree. They account for close to 70% of its size. Both the directories mostly have content from other open source projects such as kernel images, GNU toolchains, common libraries such as OpenSSL and WebKit. ‘libcore’ is also a part of another open source project, Apache Harmony [1]. The key components of Android reside in frameworks/. Inside frameworks, System services can be found in frameworks/base/services and frameworks/base/media. frameworks/base /core contains core components including Runtime and Zygote, the system key elements which are explained in the following section. Native daemons can be found in frameworks/base/cmds.
3.2 Android Architecture
Figure 2: Android architecture
This section describes the key components of Android exhibited in Figure 2. in the order in which they are loaded during system startup.
Bootloader
When the Android system is started, a bootloader is the first program executed by CPU. The bootloader initializes the RAM, loads the kernel and RAM disk, and jumps to the kernel.
Kernel
The main goal of Android Kernel is to load the necessary things for CPU. It initializes several subsystems and invokes the ‘init’ functions of all built-in drivers. It also mounts the root filesystem and fires up the init process of Android.
Init
The init process executes instructions stored in init.rc file. The init.rc file contains instructions to create mount points, mount filesystems, start native daemons and set up environment variables such as system-path.
Android Runtime
Init process sends an app_process command to Android Runtime. On receiving the command, the Runtime kicks off the first Dalvik VM, which invokes Zygote’s main() method to launch the Zygote process.
Zygote
Zygote is a daemon process responsible for starting an app as its child process. During system startup, the Zygote preloads all necessary Java classes and resources, starts System Server process and opens a socket /dev/socket/zygote to listen to incoming requests for launching new applications [9].
When the Zygote receives a request to launch a new app through the socket, it takes advantage of the fork() system call to create a new process. With the fork() call, it creates a replica of itself. Therefore, a new Dalvik VM is preloaded with all the necessary classes and resources that any app might need. This makes the process of creating a VM and loading of resources more efficient. With the Copy On Write (COW) technique implemented by the Linux system, the memory pages are shared between the parent and the child processes. When one of the processes attempts to modify the shared memory, the kernel intercepts the call and makes a copy of the shared pages. In the case of Android, the pages are not writable. This means that all the process forked from the Zygote use the same copy of the system classes and resources. In addition to being efficient, this approach also saves physical memory space on a device; regardless of how many applications have started, the increase in memory usage will be a lot smaller [1]. The above process is summarized in Figure 3 provided below.
Figure 3: Working of Zygote process
System Server
System Server is started by the Zygote during system startup and stays as a background process. It hosts the majority of the system services that run on Android within a single process. Some system services are written in Java whereas rest of them are written in C/C++. It also includes some native code access through JNI to allow some of the Java-based services to interface to Android’s lower layers.
Activity Manager
Activity Manager is one of the services hosted by the System server. It handles the starting of new components, such as Activities and Services, fetching of Content Providers and intent broadcasting. It is also involved in the maintenance of out of memory adjustments used by the in-kernel low- memory handler, permissions, task management, etc.
Launcher
Launcher app is started by the Activity Manager by sending an intent of type Intent.CATEGORY_HOME. The launcher is responsible for displaying the home screen which is familiar to the Android users.
3.2 Android Build System
Android build system resembles make-based build system, with a few notable differences. The Android build system does not count on recursive makefiles, unlike the make-based build system. Instead, it makes use of a special makefile, Android.mk. The Android.mk file defines how a local module is built. A module is any part of the AOSP such as a binary, an app package or a library that needs to be built. The Android build system invokes a script that traverses all the subdirectories till it locates an Android.mk file. After finding an Android.mk, it stops. It does not explore the subdirectories beneath that file’s location, unless explicitly specified in the Android.mk. [14].
Another difference between the make and the Android build system is in the way the Android build system is configured. Android depends on a set of variables that are are defined statically in a buildspec.mk file or either set dynamically by envsetup.sh and lunch scripts. Also, the level of configurability allowed by the Android’s build system is limited. Although the properties of the target can be specified, there is no way to enable or disable most of the features. For instance, it is not possible to disable power management support or Location Service.
Also, the Android build system does not store an intermediate output within the same location as the AOSP source files. Instead, the build system generates the intermediate output as well as final output in a new directory out/. Therefore, removing the out/ directory also removes everything that was generated during build. In other words, ‘make clean’ is the same thing as rm -rf out/.
The root directory of AOSP includes a single Make File. That file is mostly empty; its main use is to include the entry point for the Android’s build system as can be seen in Code snippet 1.
### DO NOT EDIT THIS FILE ###
include build/core/main.mk
### DO NOT EDIT THIS FILE ###
Code snippet 1: Makefile of AOSP
The build/core/main.mk file is the entry point to the build system. The build system pulls everything into a single makefile. Therefore, each .mk file in the end becomes a part of a single huge makefile. Therefore, this single makefile contains the rules for building all the modules in the system. Figure 4 presents the components of the build system. The components are explained in detail in the subsequent sections.
Figure 4: Android build architecture
3.2.1 Configuration
The build configuration is specified in config.mk file. The build system pulls in the build configuration by including config.mk. The config.mk file defines the following environment variables. [14].
- TARGET_PRODUCT:
Android flavor to be built. The set of values that can be provided to the TARGET_PRODUCT variable, includes the following:
Value | Description |
generic | the most basic build of the AOSP parts |
full | With most apps and the major locales enabled |
full_crespo | Same as full but for Crespo (Samsung Nexus S) |
full_grouper | Same as full but for Grouper (Asus Nexus 7) |
Sim | Android simulator |
sdk | The SDK; includes a vast number of locales |
Table 2: Set of values that can be provided for TARGET_PRODUCT in the Android build
- TARGET_BUILD_VARIANT:
Dictates which modules to install. Each module sets a LOCAL_MODULE_TAGS variable in its Android.mk from the list: user, debug, eng, tests, optional, or samples. With the selection of the variant, module subsets to be included can be specified.
3. TARGET_BUILD_TYPE:
Decides on whether to use release or debug build type.
- ARGET_TOOLS_PREFIX:
By default, the build system uses one of the cross-development toolchains in the prebuilt/ directory. To use a different toolchain, the value of the TARGET_TOOLS_PREFIX variable should point to the location of the toolchain.
- OUT_DIR:
By default, the build system generates the build output into the out/ directory. This variable is used to provide a different output directory.
- BUILD_ENV_SEQUENCE_NUMBER:
If the default template build/buildspec.mk.default is used to create buildspec.mk file, this value is set correctly. However, if the buildspec.mk created with an older AOSP release and is used in the newer AOSP release having important build system changes, this variable acts as a safety net. It causes the build system to update the users that buildspec.mk file is not compatible with the build system.
envsetup.sh
envsetup.sh sets up the build environment for Android. Primarily, it defines a chain of shell commands useful to many AOSP jobs. Invoking build/envsetup.sh from a shell adds the functions described in Table 3 to the environment.
Variable | Description |
croot | Changes directory to the top of the tree |
m | Makes from the top of the tree |
mm | Builds all the modules in the current directory |
mmm | Builds all the modules in the supplied directories |
cgrep | Greps on all local C/C++ files |
jgrep | Greps on all local Java files |
resgrep | Greps on all local res/*.xml files |
godir | Go to the directory containing a file |
Table 3: List of functions added by build/envsetup.sh script
‘m’ and ‘mm’ are quite useful commands. ‘m’ command allows users to build from top level regardless of the current path, whereas ‘mm’ builds the modules located in the current directory. For example, if a modification is made to the Launcher and the current path is packages/apps/Launcher2, then the module can be rebuilt by with mm instead of going back to the topmost level, and typing make. Since mm does not rebuild the entire tree, it does not rebuild AOSP images even though a dependent module has modified. mm can be still helpful to tryout local changes and to check if they crash the build [14].
lunch
lunch command is defined by envsetup.sh. When lunch is executed without any arguments, it displays a list of alternatives. For example, on Gingerbread, the following list appears.
$ lunch
You’re building on Linux
Lunch menu… pick a combo:
1. generic-eng
2. simulator
3. full_passion-userdebug
4. full_crespo4g-userdebug
5. full_crespo-userdebug
Which would you like? [generic-eng]
Code snippet 2: List of choices offered by lunch command on Gingerbread
Basically, the menu asks the users to select a combination of the TARGET_PRODUCT and the TARGET_BUILD_VARIANT. The menu gives the default combination, but the other options can be chosen as parameters on the command line. These choices are not generated dynamically based on content of AOSP. They are individually added using the add_lunch_combo() function defined in the envsetup.sh [14]. For instance, in Gingerbread 2.3.7, envsetup.sh includes generic-eng and simulator as illustrated in Code snippet 3.
# add the default one here | |
add_lunch_combo generic-eng | |
# if we’re on linux, add the simulator. There is a special case | |
# in lunch to deal with the simulator | |
if [“$(uname)” = “Linux”] ; then | |
add_lunch_combo simulator | |
fi | |
Code snippet 3: Part of the code in envseup.sh which adds lunch combinations
lunch also offers a way to add vendor-specific scripts. Here is how it is done in 2.3.7 Gingerbread:
# Execute the contents of any vendorsetup.sh files we can find. | ||
for f in `/bin/ls vendor/*/vendorsetup.sh vendor/*/build/vendorsetup.sh device/*/*/vendorsetup.sh 2> /dev/null` | ||
do | ||
echo “including $f” | ||
$f done | ||
unset f | ||
Code snippet 4: vendorscript.sh script in Gingerbread 2.3.7
Table 4 describes the required configuration information set up by lunch for default ‘generic-eng’ combo.
Variable | Value |
PATH | $ANDROID_JAVA_TOOLCHAIN:$PATH:$ANDROID_BUILD_PATHS |
ANDROID_EABI_TOOLCHAIN | aosp-root/prebuilt/linux-x86/toolchain/arm- eabi-4.4.3/bin |
ANDROID_QTOOLS | aosp-root/development/emulator/qtools |
ANDROID_BUILD_PATHS | aosp-root/out/host/linux-x86:$ANDROID_TOOLCHAIN:$AN DROID_QTOOLS:$ANDROID_TOOLCHAIN:$ANDROID_EABI_TOOLCHAIN |
ANDROID_BUILD_TOP | aosp_root |
ANDROID_JAVA_TOOLCHAIN | $JAVA_HOME/bin |
ANDROID_PRODUCT_OUT | Aosp- root/out/target/product/generic |
BUILD_ENV_SEQUENCE_NUMBER | 10 |
OPROFILE_EVENTS_DIR | aosp-root/prebuilt/linux-x86/oprofile |
TARGET_BUILD_TYPE | release |
TARGET_PRODUCT | Generic |
TARGET_BUILD_VARIANT | eng |
TARGET_BUILD_APPS | empty |
TARGET_SIMULATOR | false |
PROMPT_COMMAND | ”33]0;[${TARGET_PRODUCT}-${TARGET_BUILD_VARIANT}] $
{USER}@${HOSTNAME}: ${PWD}07” |
Table 4: Configuration information set up by lunch for generic-eng combo
Using ccache
ccache stands for Compiler Cache. It is a mechanism used to accelerate the build process. It caches the object files generated by a compiler based on the preprocessor’s output. Therefore, if two separate builds generate an identical preprocessor’s output, with the use of ccache, the second build will not use a compiler to build the file; the cached object file will be copied to the destination folder.
To enable the use of ccache in the build, the USE_CCACHE variable should be set to 1 before starting the build, as shown here.
$ export USE_CCACHE=1
When run for the first time, there will not be any gain in the acceleration, since the cache will be empty at that point. But whenever the build is run afterwards, the cache will assist in accelerating the build process. The downside of ccache is that it is for C/C++ files only [14].
3.2.2 Function Definitions
Since the build system is enormous, to make reuse of code, the build system defines many functions in the definitions.mk file. Various functions specified in the definitions.mk perform common operations, such as transformation (transform-c-to-o, transform-java-to-classes.jar), copying (copy-file-to-target), file lookup (all-makefiles-under, all-c-files-under) and utility (my-dir). The build system components as well as in modules’ Android.mk files make use of these functions [1].
3.2.3 Cleaning
‘make clean’ is same as removing the out/ directory. The main.mk defines the clean target. Another cleanup target installclean is defined in cleanbuild.mk, and is automatically invoked whenever the values of the variables TARGET_PRODUCT, TARGET_BUILD_VARIANT or PRODUCT_LOCALES are changed.
3.2.4 Module Build Templates
In AOSP,module build recipes are independent of the build system’s internals. The purpose of the build templates is to facilitate the module authors to get their modules built. Each template is adapted for a particular type of module. The template’s behavior and output can be controlled by a set of documented variables. The templates are located in build/core, where also the rest of the build system is located. To get access to the templates, Android.mk makes the use of the include directive. For instance,
include $(BUILD_PACKAGE)
As shown above, Android.mk files do not add the .mk templates by name. Instead, they include a variable which is set to the corresponding .mk file. Because of the build templates, the Android.mk files are very lightweight [1].
Table 5 lists the module build templates supplied by the Android build system.
Variable | Template | What it builds |
BUILD_HOST_SHARED_LIBRARY | host_shared_library.mk | Host shared libraries |
BUILD_HOST_STATIC_LIBRARY | host_static_library.mk | Host static libraries |
BUILD_RAW_STATIC_LIBRARY | raw_static_library.mk | Target static libraries that run on bare metal |
BUILD_PREBUILT | prebuilt.mk | Copies prebuilt target files |
BUILD_HOST_PREBUILT | host_prebuilt.mk | Copies prebuilt host files |
BUILD_MULTI_PREBUILT | multi_prebuilt.mk | Copies prebuilt like Java libraries or executables |
BUILD_PACKAGE | package.mk | Built-in AOSP apps |
BUILD_KEY_CHAR_MAP | key_char_map.mk | Device character maps |
Table 5: List of module build templates and their corresponding .mk files in AOSP
3.2.5 Output
out/ contains two directories, signifying its operating modes: host/ and target/.
The build process generates an output in three stages:
- The module sources generate intermediates. The format and location of the intermediates depend on the type of module sources. For example, .o files for C/C++ code, .jar files for Java code.
- By using the intermediates, the build system creates binaries and packages. For example, linking .o files into an actual binary.
- Finally, the binaries and packages are assembled to create the final output. For instance, binaries are copied into /system directory which are used on the actual device.
3.2.6 Build Recipes
$ make droid
The default build target is droid which is defined in main.mk. So, running ‘make’ is equivalent to the above command.
$ make showcommands
When the AOSP is built, it does not print out the commands it is running. Instead, it shows only a summary of each step. To see the commands the build is executing, the showcommands target can be added to the command line.
$ make –j4
With a -jN argument, parallel tasks can be started to accelerate the build process. The common value of N is usually between 1 and 2 times the number of hardware threads on the system utilized for the build.
$ make sdk
It is possible to build a custom SDK using the AOSP. To do so, it is required to select a series of commands, given below:
$ . build/envsetup.sh
$ lunch sdk-eng
$ make sdk
$ make update-api
The build systems have safeguards to detect if the AOSP’s core API are modified. If the modifications are detected, the build fails by default. To continue the build, the update-api target needs to be added.
$ make Launcher2
Building individual modules is possible. The above command, for example, shows how to build Launcher2 module.
$ make clean-Launcher2
The command allows the users to clean modules individually.
$ make Launcher2 snod
To include an updated module in the system image, the snod target can be added to the command line. It forces the build system to regenerate the system image. [6].
3.2.7 Compiling with Jack and Jill
Jack is a new Android toolchain introduced to reduce the Java code compilation time. It compiles Java source files directly into Android Dex bytecodes, replacing the previous Android toolchain consisting of multiple tools, such as javac, ProGuard, jarjar and dx [13]. The Jack toolchain offers following advantages:
- Completely open source
- Speeds compilation
- Handles shrinking, obfuscation, repackaging, and multidex.
Figure 5: Jack toolchain structure
As can be seen from Figure 5, Jack takes Java source and Jack libraries as an input, performs the operations such as shrinking, obfuscation, repackaging and multidexing, and finally, produces Dex files as output. Since it converts the Java source to Dex in a single step, the compilation time is reduced.
Jack library format
As illustrated in Figure 6, Jack has its own library format, which contains pre-compiled Dex code for the library and therefore allows faster compilation.
Jill
As presented in Figure 7, to include the existing third-party libraries compiled in Java bytecode format, Jill is used. It processes the class files from the libraries and transforms them into Jayce format which can be used as an input for the Jack compiler.
Figure 7: Android Build flow with Jack and Jill toolchain.
Using Jack in Android build
Jack is the default build toolchain in Android 6 (Marshmallow) and 7 (Nougat). When the Jack is used for the first time, a local Jack compilation server is launched. This server speeds up the build since it eliminates the need to start a new host JVM/JRE, to load Jack code, to initialize Jack and to warm up the JIT at each compilation. It also provides excellent compilation times during the incremental mode. The server is also a solution to control the number of parallel Jack compilations to avoid overloading issues since it limits the number of parallel compilations. The Jack server shuts down itself after an idle time without compilation [13].
However, there are a few downsides of the Jack and Jill toolchain. Since the toolchain converts Java source directly into Dex, the existing tools and plugins dependent on the Java class file format cannot be used anymore. Also, as Jack does not currently support annotation processing, the libraries like Dagger and AutoValue can no longer be used. The toolchain also increases the cost of adding Java 8 support. Therefore, despite its advantages, the Jack toolchain is now deprecated by the Android community.
3.3 Aspect-Oriented Programming
Aspect-Oriented Programming (AOP) is a programming model that addresses cross-cutting concerns. AOP provides a pluggable way to add the cross-cutting concerns in the business logic [2].
Some of the key terms and concepts frequently used in AOP are described below.
- Cross-cutting concerns: a concern which affects the entire application and is applicable throughout the application. For instance, security, logging, and data transfer are the concerns which are required in every component of an application.
- Advice: Additional code that you want to apply to your existing model, i.e. the code corresponding to the cross-cutting concerns.
- Join point: The point of execution at which cross-cutting concern needs to be applied in the application.
- Pointcut: A predicate that matches join points.
- Aspect: The pointcut and the advice together form an aspect [6,10].
3.3.1 AspectJ Library
AspectJ is an implementation of Aspect-oriented programming for Java. With just a few new constructs, AspectJ supports an implementation of a number of crosscutting concerns. AspectJ can be used for following tasks.
- Checking permissions
- Interrupting action that takes too long
- Monitoring
- Preparing any data/environment before call and processing results after call
- Opening/closing resources [8]
AspectJ weaving
There are three different ways to inject the AOP code in an application. They all require an extra step to be applied to the code. This extra step is called weaving.
- Compile-time weaving
If both the source code and the aspect code are available, the source code and the aspects can be directly compiled with the help of AspectJ compiler. This is known as compile-time weaving.
- Post-compile weaving / Binary weaving
If source code is not available, the aspects can be injected into already compiled classes or jars using post-compile weaving.
- Load-time weaving
Load-time weaving acts the same way as post-compile weaving/binary weaving but waits to inject aspects into the code until the class loader loads the class file. This requires one or more weaving class loaders [8].
Types of advice
Advice can be added either before, after or around a join point.
- Before advice: Advice that is invoked before a join point.
- Afterreturning advice: Advice that is invoked after a join point completes normally. For instance, if a method returns without throwing an exception.
- After throwing advice: Advice that is executed if a method exits unexpectedly by throwing an exception.
- After (finally) advice: Advice that is invoked irrespective of how a join point exits (normal or exceptional return).
- Around advice: Advice that surrounds a join point. It is the most powerful kind of advice. For instance, it can introduce a custom behavior before and after the method invocation. [7].
The example given in Code snippet 5 uses Around advice. It adds log statements before and after the execution of method print()in the class com.itool.HelloWorld.
HelloWorld class
package com.itool; |
public class HelloWorld{ |
public void print(){ |
System.out.println(“Hello World!”) |
} |
} |
}
@Aspect | |||
public class TestAspect { | |||
@Around(“execution(* com.itool.HelloWorld.print())”) | |||
public Object myAspect(final ProceedingJoinPoint pjp) | |||
throws Throwable{ | |||
System.out.println(“Entering method ” + pjp.getName()); | |||
Object retVal = pjp.proceed(); | |||
System.out.println(“Exiting method ” + pjp.getName()); | |||
return retVal; | |||
} | |||
} | |||
} | |||
Pointcut
Advice
Code snippet 5: Example of AspectJ around advice
3.4 Byte Code Engineering Library (BCEL)
The Byte Code Engineering Library (BCEL) is one of the most widely used frameworks for Java classworking. BCEL is aimed to give users a convenient way to analyze, create, and manipulate Java class files. It enables its users to examine the bytecode of Java classes. BCEL library repsents the classes by objects that contain all the symbolic information of a given class: methods, fields and bytecode instructions. Such objects can be created from an existing file, be transformed by a program and written to a file again. It can also be used to create classes from scratch at the run-time [11].
The BCEL APIs abstract the bytecode representation of Java classes. They can be used for static analysis and dynamic creation or transformation of Java class files. They assist developers to implement the desired functionality on a high-level abstraction without a need to handle the internal details of the Java bytecode format.
The APIs mainly consist of three parts:
- Static Information: A package that describes static constraints of class files. The classes can be used to read and write class files from or to a file. This is useful especially for analyzing Java classes without having access to the source files.
- Dynamic Information: A package to dynamically generate or modify JavaClass or Method objects. It can be used to insert, analyze or remove a piece of code from class files.
- Utilities: Various code examples and utilities like a class file viewer, a tool to convert class files into HTML, full verifier implementation to validate a binary class according to the JVM specification, a disassembler that creates a well framed JVM-level view of a binary class.
CHAPTER 4
Implementation Process
This chapter provides a detailed walkthrough of the implementation process of the project. Section 4.1 explains the modifications done to the Android build flow to enable custom code weaving into the base code. Section 4.2 gives an overview of the techniques that can be used for weaving a custom code, namely bytecode manipulation and Aspect-oriented programming. Pros and cons of the two techniques are discussed further in detail.
4.1 Build Flow Modifications
The Instrumentation tool enables the developers to weave a custom code (in the form of aspects) into Android Open Source Code project. To understand the working of the Instrumentation tool, let’s first look at the build flow of Android. The Android source code is mostly written in Java. As demonstrated in Figure 8, the source code is first compiled to Java bytecode. Dex tool then converts the Java bytecode to Dalvik bytecode.
Figure 8: Android build process
To add a weaving step into the build process, bytecode weaving technique is used which injects the aspects into the compiled Java bytecode binaries. The build process after adding the weaving step can be represented by Figure 9. The process of weaving the custom aspects into the Android binaries is transparent to the end users. The instrumented emulator built this way can be used for the intended purpose.
Figure 9: Android build process with AspectJ instrumentation
As demonstrated in Figure 9, the build flow needs to be modified to add the weaving step. To achieve so, some changes are made in the file build/core/definitions.mk. The definitions.mk file defines many useful functions for file lookup, transformation, and utility. The ‘transform-java-to-classes.jar’ function defined in the definitions.mk file is responsible for compiling Java source files and packaging the compiled files into a jar. The jar file is then converted to Dex bytecode by the Dex tool. The ‘transform-java-to-classes.jar’ function is modified to add the weaving step. After the source files are compiled, custom aspects are injected. The modified classes are packaged into a jar and then the build process continues from there. A program to inject the aspects is defined in script post-compile-weaving.sh. Code snippet 6 shows the changes made to the definitions.mk file. The changes are highlighted with red color.
Code snippet 6: Changes made in build/core/definitions.mk file to support custom weaving
# compile Java classes |
$(hide) tr ‘ ‘ ‘ ’ < $(dir $(PRIVATE_CLASS_INTERMEDIATES_DIR))/java-source-list |
| sort -u > $(dir $(PRIVATE_CLASS_INTERMEDIATES_DIR))/java-source-list-uniq |
$(hide) $(TARGET_JAVAC) -encoding ascii $(PRIVATE_BOOTCLASSPATH) |
$(addprefix -classpath ,$(strip |
$(call normalize-path-list,$(PRIVATE_ALL_JAVA_LIBRARIES)))) |
$(PRIVATE_JAVACFLAGS) $(strip $(PRIVATE_JAVAC_DEBUG_FLAGS)) $(xlint_unchecked) |
-extdirs “” -d $(PRIVATE_CLASS_INTERMEDIATES_DIR) |
@$(dir $(PRIVATE_CLASS_INTERMEDIATES_DIR))/java-source-list-uniq |
|| ( rm -rf $(PRIVATE_CLASS_INTERMEDIATES_DIR) ; exit 41 ) |
# the script below injects custom code into the compiled classes |
$(ASPECTJ_HOME)/post-compile-weaving.sh $(PRIVATE_CLASS_INTERMEDIATES_DIR)
$(PRIVATE_CLASS_INTERMEDIATES_DIR) $(strip $(call normalize-path-list, $(PRIVATE_ALL_JAVA_LIBRARIES))); |
$(hide) rm -f $(dir $(PRIVATE_CLASS_INTERMEDIATES_DIR))/java-source-list |
$(hide) rm -f $(dir $(PRIVATE_CLASS_INTERMEDIATES_DIR))/java-source-list-uniq |
$(hide) mkdir -p $(dir $@) |
# package the compiled classes into jar |
$(hide) jar $(if $(strip $(PRIVATE_JAR_MANIFEST)),-cfm,-cf) |
$@ $(PRIVATE_JAR_MANIFEST) -C $(PRIVATE_CLASS_INTERMEDIATES_DIR) . |
4.2 Weaving Custom Code
To inject a Java-based code into already compiled classes, two alternatives are available in the market:
- Using Bytecode manipulation
- Using Aspect-Oriented Programming
4.2.1 Using Bytecode Manipulation
Java compiler compiles Java source code to bytecode. The custom code can be injected at the bytecode level. Many well-maintained Java bytecode manipulation libraries with an intuitive APIs are available such as BCEL, Javaassist, ASM [12]. As described in section 3.4, BCEL library can be used to inject the custom code into the compiled library. As shown in Code snippet 7, BCEL library is used to add print statements at the beginning of every method in CustomWeaving class.
// create a parser with the name of the class to be transformed.
ClassParser parser = new ClassParser(“com.example.CustomWeaving”);
JavaClass javaClass = parser.parse();
ClassGen cg = new ClassGen(javaClass);
// iterate over the methods of the class
for (Method m : cg.getMethods()) {
MethodGen mg = new MethodGen(m, cg.getClassName(), cg.getConstantPool());
// get the instructions of the method
InstructionList il = mg.getInstructionList();
// create the instruction set to be added
InstructionList modiIL = new InstructionList();
modiIL.append(new GETSTATIC(cg.getConstantPool().
addFieldref(“java.lang.System”, “out”, “Ljava/io/PrintStream;”)));
modiIL.append(new LDC(cg.getConstantPool().addString
(“Calling: ” + m + ” in class: ” + javaClass.getClassName())));
modiIL.append(new INVOKEVIRTUAL(cg.getConstantPool()
.addMethodref(“java.io.PrintStream”, “println”,
“(Ljava/lang/String;)V”)));
// insert the new instructions at the beginning of the existing
// instruction set
il.insert(modiIL);
mg.setInstructionList(il);
// replace the existing method with the modified method in the class.
cg.replaceMethod(m, mg.getMethod());
}
Code snippet 7: Program to insert log statements at the beginning of every method in a class using BCEL library
The main advantages of BCEL are its stability, extensive JVM instruction-level support and commercial-friendly Apache licensing. These features have made it a very popular choice for classworking applications. The disadvantage of BCEL is that it is not easy to use. The users of BCEL need to have a very detailed knowledge of JVM instructions. Therefore, we use an alternative option Aspect-Oriented Programming in this project.
4.2.2 Using Aspect-Oriented Programming
As described in section 3.3, Aspect-Oriented Programming (AOP) provides a powerful way to link the aspects with the pure business logic code. AspectJ is currently the most concrete implementation of AOP language for Java. AspectJ uses bytecode manipulation technique underneath but provides an abstract way to specify the code to be injected (advice) and where it should be injected (pointcut). The example shown in Code snippet 8 adds a log at the beginning of all the methods in the class com.example.CustomWeaving
@Before(“execution (* com.example.CustomWeaving.*(..))”)
public void adviceEnter(JoinPoint joinPoint) {
System.out.printf(“Calling method: ‘%s’%n”,
joinPoint.getSignature().getName() +
“in class ” +
joinPoint.getSignature().getDeclaringType());
}
Code snippet 8: Program to insert log statements at the beginning of every method in a class using AsepectJ
A comparison of the two code weaving approaches is provided below in Table 6.
Bytecode Manipulation | Aspect Oriented Programming (AspectJ) |
Provides a set of APIs to read from an existing class file, transform and write to a file again. | Provides an abstract way to inject custom code with the help of aspects. It uses bytecode manipulation technique underneath. |
The users need to understand the details of JVM instructions such as the bytecode format of a class, opcodes and the operand stack since the level of abstraction is little. | It provides powerful Java-based annotations to represent the code to be injected and the place where the code needs to be inserted. Therefore, the knowledge of JVM instructions is not required. |
The custom code is injected programmatically using the APIs provided. | AspectJ compiler (ajc) weaves the custom aspects into the source code. |
It is not possible to insert the same piece of code into multiple methods at the same time. | With the help of Pattern matching pointcut expressions, it is feasible to insert the same piece of code into multiple methods at the same time. |
Since the code injection is done by manipulating the class file directly, there is no way to verify the modified bytecode. The errors introduced due to the code manipulation are exposed at run-time only. | The ajc compiler compiles the aspect code. Therefore, the errors in the aspects can be caught at the earlier stage. |
There is no run-time dependency on the third-party libraries after the code injection. | The modified code depends on the AspectJ runtime library. Therefore, the library must be present at run-time. |
This technique is the most powerful since it is at low-level. For example, it is possible to create a new class from scratch, remove a method from a class, etc. | Thought AspectJ provides an expressive language, its capability is limited as compared to the Bytecode manipulation technique. |
Table 6: Comparison of custom code weaving approaches: Bytecode manipulation and
Aspect-oriented programming
Because of its simplicity and the support of expressive pointcuts, we used the AspectJ library in this project. We implemented the post-compile weaving technique to instrument the Android source. Using the post-compile weaving, the custom aspects are applied to the compiled Java source classes. Since the woven code depends on the AspectJ library, the library is linked to the final executable image of the emulator. The above process is implemented in post-compile-weaving.sh script. The content of post-compile-weaving.sh script is displayed in Code snippet 9. The script is called in the modified build flow as described in section 4.1.
#!/bin/bash
function error_exit
{ |
echo “$1” 1>&2 |
exit 1 |
} |
export INSTR_HOME=$ANDROID_HOME/instrumentation_tool |
TEMP_DIR=$INSTR_HOME/target_classes/ |
CLASSES_DIR=$1 |
CLASSPATH=$INSTR_HOME/lib/aspectjrt.jar:$INSTR_HOME/lib/aspectjtools.jar:$2 |
echo “Compiling aspects” |
find $INSTR_HOME/src -type f -name “*.java” > sources.txt |
javac -classpath $CLASSPATH -g -d $CLASSES_DIR @sources.txt ||
error_exit “Error in compiling Aspects..Aborting “ |
rm -f sources.txt |
echo “Weaving aspects” |
java -cp $CLASSPATH org.aspectj.tools.ajc.Main -Xmx2048M -source 1.5 –inpath $CLASSES_DIR/ -aspectpath $INSTR_HOME/src -d $TEMP_DIR ||
error_exit “Error while weaving aspects.. Aborting” |
echo “Replacing the original classes with the woven classes” |
cp -rf $TEMP_DIR/* $CLASSES_DIR/ |
rm -rf $TEMP_DIR |
echo “Include aspectJ runtime library “ |
mkdir $INSTR_HOME/extracted_classes |
unzip $INSTR_HOME/lib/aspectjrt.jar -d $INSTR_HOME/extracted_classes > /dev/null |
cp -r $INSTR_HOME/extracted_classes/* $CLASSES_DIR/ |
rm -rf $INSTR_HOME/extracted_classes |
Code snippet 9: Content of post-compile.sh script
CHAPTER 5
Demonstration
This chapter provides a demonstration of the Instrumentation tool to understand the sequence of actions that take place when an app is launched on the Android OS. Multiple versions of Android OS have been instrumented to aid us in our analysis. There are four primary processes that participate in the application startup and are paid special attention in our analysis. These are the Zygote process, the launcher process, the Activity Manager process, and the application process. The study gives us an insight into the changes that have taken place in the mentioned processes above through the different versions of Android OS. The analysis is carried out with the help of logs that are generated by various versions of instrumented emulators corresponding to the different version of Android OS. This entire process is explained in more detail in the next few paragraphs.
The instrumentation is carried out on different versions of Android OS ranging from Gingerbread to Nougat. Aspects are used to aid the injection of logging statements.
@Around(“(execution(*..*(..)) &&
!within(com.android.internal.os.LoggingPrintStream) && !within(com.android.internal.os.AndroidPrintStream) &&
!within(andoid.os.Process) &&
!within(android.util.Log)”)
public Object logAspect(final ProceedingJoinPoint pjp) throws Throwable{
AspectjLog.log(pjp,”Enter”,true);
Object retVal = pjp.proceed();
AspectjLog.log(pjp,”Exit”, false);
return retVal;
}
Code snippet 10: AspectJ program to add enter/exit logs in every method of AOSP
private String getLogMessage(ProceedingJoinPoint pjp,String message, boolean printArgumentsFlag){
char separator = ‘ ‘;
StringBuilder builder = new StringBuilder();
builder.append(‘(‘);
builder.append(android.os.Process.myPid());
builder.append(‘)’);
builder.append(separator);
builder.append(pjp.getSignature());
builder.append(separator);
builder.append(message);
builder.append(separator);
if (printArgumentsFlag) {
logBuilder.append(“[“);
if (pjp.getArgs().length > 0) {
for (final Object argument : pjp.getArgs()) {
logBuilder.append(argument + “,”);
}
logBuilder.deleteCharAt(logBuilder.length()-1);
}
logBuilder.append(“]”);
}
}
The above aspects instrument the complete OS (except the APIs that could cause recursive method calls) and enable us to generate logs and analyze them later.
After instrumentation, we installed a sample HelloWorld app, run on the instrumented emulators for various versions of Android and collected the logs are generated by the tool through the emulators ranging through the entire app launch process. A few lines of the generated log file are shown here. Each log statement has the following fields:
<process-ID> <method signature> <Enter/Exit message> < (only with Enter message) [comma separated method arguments]>
We repeated the process on two major Android platforms namely Gingerbread (2.3.7) and Nougat (7.1.1); Gingerbread being the first publicly available version and Nougat being the latest stable version of Android. Then, we compared the results from these platforms. The analysis carried out on the platforms helps us in understanding the phenomenal growth of Android over the last few years.
(481) void com.android.internal.os.RuntimeInit.commonInit() Enter []
(481) String com.android.internal.os.RuntimeInit.getDefaultUserAgent() Enter []
(481) String com.android.internal.os.RuntimeInit.getDefaultUserAgent() Exit
(481) String android.os.SystemProperties.get(String) Enter[ro.kernel.android.tracing]
(481) String android.os.SystemProperties.get(String) Exit
(481) void com.android.internal.os.RuntimeInit.commonInit() Exit
(481) void com.android.internal.os.RuntimeInit.invokeStaticMain(String,
String[]) Enter [android.app.ActivityThread,[Ljava.lang.String;@4056bd0]
(481) void com.android.internal.os.ZygoteInit.MethodAndArgsCaller.run() Enter[]
(481) void android.app.ActivityThread.main(String[])
Enter [[Ljava.lang.String;@4056bd0]
(481) void com.android.internal.os.SamplingProfilerIntegration.start() Enter []
(481) void com.android.internal.os.SamplingProfilerIntegration.start() Exit
(481) void android.os.Looper.prepareMainLooper() Enter []
(481) void android.os.Looper.prepare() Enter []
(481) void android.os.Looper.prepare() Exit
(481) Looper android.os.Looper.myLooper() Enter []
(481) Looper android.os.Looper.myLooper() Exit
(481) String android.os.Looper.toString() Enter []
(481) String android.os.Looper.toString() Exit
(481) void android.os.Looper.setMainLooper(Looper) Enter [Looper{40572360}]
(481) String android.os.Looper.toString() Enter []
(481) String android.os.Looper.toString() Exit
(481) void android.os.Looper.setMainLooper(Looper) Exit
(481) Looper android.os.Looper.myLooper() Enter[]
(481) Looper android.os.Looper.myLooper() Exit []
(481) void android.os.Looper.prepareMainLooper() Exit
(481) Looper android.os.Looper.myLooper() Enter []
(481) Looper android.os.Looper.myLooper() Exit
Figure 10: Application startup process
Figure 10 shows the application startup process and the processes that participate in the application startup. The subsequent sections describe the analysis of the processes in detail. Through the analysis, we find that, except for the application process, the working of other processes has not changed much between the two Android platforms. Therefore, an explanation aided through visual graphs is provided for these four processes. For the application process, a detailed comparison is provided that highlights various metrics related to the APIs invoked during the startup process.
A relevant part of the output is generated by executing ‘adb shell ps’ command is provided below. The ps command shows a list of currently running processes.
———————————————————————————————————
USER PID PPID NAME
———————————————————————————————————
root 1 0 /init
root 32 1 zygote
media 33 1 /system/bin/mediaserver
bluetooth 34 1 /system/bin/dbus-daemon
keystore 36 1 /system/bin/keystore
root 38 1 /system/bin/qemud
shell 40 1 /system/bin/sh
system 68 32 system_server
system 125 32 com.android.systemui
app_28 135 32 com.android.launcher
app_6 145 32 com.android.inputmethod.latin
radio 161 32 com.android.phone
app_29 481 32 com.example.pooja.test_launch
root 490 41 /system/bin/sh
————————————————————————————————————
Table 7: List of processes observed at the application startup
5.1 Launcher Process
Launcher process is started by the Activity Manager during system startup. It is responsible to register onClick() listeners to enable users to start apps from the home screen. When an app icon is clicked, it invokes startActivity() method of the Activity Manager along with the parameters:
{act = android.intent.action. MAIN, cat = [android.intent.category. LAUNCHER], flg=0x10200000, cmp=com.example.pooja.test_launch/.MainActivity }.
The graph displayed below provides the sequence of APIs called by the launcher process. The numbers affixed on the nodes of the graph indicate the sequence in which the APIs are called. For example, in this graph, Method 4 is calling Method 5 first, Method 6 afterward and then Method 7 at last.
Figure 11: Call-stack graph of the APIs invoked by launcher process
5.2 Activity Manager
Activity Manager is one of the services hosted by the System Server (Process ID 1744). It is created by the Zygote using fork() call during the system boot and runs as a background service. Among others, the Activity Manager is primarily responsible, during the application startup, for validation tasks such as checking if the application has sufficient permissions, checking security violations and gathering stats. It also sends a request to the Zygote to create a new process for the app to be launched over a socket connection.
Figure 12 illustrates the sequence of APIs invoked by the Activity Manager. The traversal order of the graph is indicated by the numbers on the nodes of the graph.
Figure 12: Call-stack graph of the APIs invoked by Activity Manager
5.3 Zygote
As described in section 3.2, Zygote preloads all the classes that might be required by Android apps during system startup. After loading all the required classes, Zygote process listens on a socket and goes in an infinite loop, as can be seen in the code snippet shown below.
Code snippet 11: Code snippet of Zygote process which accepts a new socket request
while (true) {
try {
fdArray = fds.toArray(fdArray);
index = selectReadable(fdArray);
} catch (IOException ex) {
throw new RuntimeException(“Error in select()”, ex);
}
if (index < 0) {
throw new RuntimeException(“Error in select()”);
} else if (index == 0) {
ZygoteConnection newPeer = acceptCommandPeer();
peers.add(newPeer);
fds.add(newPeer.getFileDesciptor());
} else {
boolean done;
done = peers.get(index).runOnce();
if (done) {
peers.remove(index);
fds.remove(index);
}
}
}
The Zygote is responsible for launching applications by replicating itself. Therefore, it is the parent process of all the applications on Android. This can be confirmed by looking at the process IDs of the processes. As is evident from Table 7, process Id of the Zygote daemon is 32, which is also the parent ID of the application ‘com.example.pooja.test_launch’ with process Id 481.
When a new app is launched, ActivityManager sends arguments to Zygote process over the socket connection. After receiving a new request, Zygote calls runOnce() method in the ZygoteConnection class. As might be seen from Code snippet 12, from the runOnce() method, a fork() system call is made. The fork() call creates a new process which becomes a child process of the caller. After a new child process is created, both processes execute the next instruction following the fork() call. The fork() returns a zero to the newly created child process and process ID of the child process (process ID 481) to the parent. The handleChildProc() method is called from the child process, whereas handleParentProc() method is called from the parent process to handle post-fork setup.
pid = Zygote.forkAndSpecialize(parsedArgs.uid, parsedArgs.gid,
parsedArgs.gids, parsedArgs.debugFlags, rlimits);
if (pid == 0) {
System.out.println(“fork in ZygoteConnection called”);
// in child
handleChildProc(parsedArgs, descriptors, newStderr);
// should never happen
return true;
} else { /* pid != 0 */
// in parent…pid of < 0 means failure
return handleParentProc(pid, descriptors, parsedArgs);
}
Code snippet 12: Code snippet of Zygote process to create a child process using fork()
The child process then throws MethodAndArgsCaller exception which gets caught in ZygoteInit.main(). It responds by invoking the exception’s run() method. This arrangement clears up all the stack frames required in setting up the process. In order to load Application-specific classes and libraries into memory, call to bindApplication() method is made. The method sends AppBindData message to the message queue, which leads to an invocation of makeApplication() method. The makeApplication() method loads the application-specific classes into memory.
A call-stack of the primary APIs invoked by the Zygote process during the application launch is illustrated in Figure 13.
Figure 13: Call-stack graph of the APIs called by Zygote Process
5.4 Application Process
Application process (Process Id 481) is a child process of the Zygote. The details of the evaluation results of the application process are provided in the following section.
5.4.1 Android Platform Variations
A HelloWorld application is executed on different versions and configurations of Android. The two different versions are Gingerbread (2.3.7) and Nougat (7.1.1) while the two different configurations are with and without the Android Support Library. The aim of this approach was to compare the code complexity across the various versions and configurations. For each version, the logs are generated from the creation of an application process until the onCreate() method invocation of the launching activity. The onCreate() is the first callback of the Activity life cycle and it is called when the application launch is complete. The sequence of APIs called between the application process creation and the invocation of the onCreate() method are logged and represented in the form of a graph. The call-stack of API invocations was studied based on the metrics summarized in Table 8.
Metric | Description |
Number of classes loaded into memory | A total number of classes loaded in memory, the value of this parameter is calculated using Debug.getLoadedClassCount() API. |
Number of classes used in Application startup | A total number of classes used in launching an application. |
Number of method calls | A total number of methods invoked during the application launch |
Maximum out-degree of the graph | Maximum number of edges coming out of a node in the graph |
Maximum depth of graph | Number of edges from root to the deepest nodes of the graph |
Average children per node | Total number of child nodes
Total number of internal nodes |
Table 8: Metrics used in the analysis of a call-stack graph
The observations made from the graphs analysis on the four variations are explained separately in the subsequent sections.
5.4.1.1 Gingerbread (2.3.7)
Gingerbread was the first publicly available Android platform. When an application is launched on the instrumented emulator, total 302 APIs were called during application startup. Figure 14 shows a representative part of the graph depicting primary API invocations, as the entire graph would be too large to fit here. However, the evaluation analyzes the complete stack.
Figure 14: Call-stack graph of the APIs invoked during application startup on Gingerbread
The complete graph is evaluated and the values of the evaluation parameters are calculated programmatically. Table 9 presents the values of the metrics calculated during the assessment.
Metric | Value |
Number of classes loaded into memory | 254 |
Number of classes used in Application startup | 48 |
Number of method calls | 302 |
Maximum out-degree of graph | 14 |
Maximum depth of graph | 18 |
Average children per Node | 2.43 |
Table 9: Graph analysis of the call-stack graph of application process on Gingerbread (2.3.7)
5.4.1.2 Gingerbread (2.3.7) with Android support library
When developing apps that support multiple API versions, Android support library acts as a compatibility layer and provides a uniform way to provide newer features on previous versions of Android. In addition, the support library also provides additional classes and features not available in the standard Framework API for easier development and support across multiple devices.
The support library is linked as a part of the application’s APK file at the compilation time. To aid our analysis, we used an instrumented version of the support library. With the use of the support library, additional APIs are called during the application. Table 10 provides the summary of the evaluation performed on the call stack graph generated on the instrumented platform. Since the support library API invocations are scattered across the various framework API, obtaining a smaller relevant subset of the graph does not help in understanding the differences between the two configurations of Gingerbread.
Metric | Value |
Number of classes loaded into memory | 2177 |
Number of classes used in Application startup | 76 |
Number of method calls | 510 |
Maximum out-degree of graph | 31 |
Maximum depth of graph | 28 |
Average children per Node | 2.36 |
Table 10: Graph analysis of call-stack graph of the application process on Gingerbread (2.3.7) with Android support library
5.4.1.3 Nougat (7.1.1)
Comparing the Nougat version to the Gingerbread (without support library), we see that there is a considerable increase in complexity in terms of the number of classes loaded and methods invoked. This can be observed by looking at the graph analysis described in Table 11. Although the most part of the startup process remains the same, a lot of additional method calls are observed in the log files as compared to Gingerbread (2.3.7). This difference is made apparent in Table 13.
Parameter | Value |
Number of classes loaded into memory | 377 |
Number of classes used in Application startup | 53 |
Number of method calls | 329 |
Maximum out-degree of graph | 14 |
Maximum depth of graph | 21 |
Average children per Node | 2.45 |
Table 11: Graph analysis of call-stack graph of the application process on Nougat (7.1.1)
5.4.1.4 Nougat (7.1.1) with support library
The increased code complexity of the Nougat version with the support library is reflected in the results of the evaluation described in Table 12. From the results, it can be observed that this configuration has the highest complexity.
Parameter | Value |
Number of classes loaded into memory | 2189 |
Number of classes used in Application startup | 82 |
Number of method calls | 567 |
Maximum out-degree of graph | 32 |
Maximum depth of graph | 39 |
Average children per Node | 3.08 |
Table 12: Graph analysis of the application startup sequence on Nougat (7.1.1) with support library
5.4.1.5 Comparison of the results of analysis on Android platform variations
Comparison of the results of the analysis carried out on different versions and configurations are compared and presented with reference to the analysis on the Gingerbread (without support library) version in Table 13. Figure 15 depicts the graphical representation of the comparison.
|
Table 13: Percentage comparison of the results of analysis on different versions of Android
Figure 15: Comparison of the results of analysis on different versions of Android
CHAPTER 6
Summary and Conclusion
6.1 Project Summary
The project was divided into two phases with the first phase carried out during our CSC 899 and second phase carried out in CSC 895. We started with understanding the structure of AOSP and the build process and then rebuilt the AOSP on our local machines. Our next step was to speed up this process of modifying and rebuilding the AOSP using the Amazon EC2 instance that provided us with faster computing resources. Additionally, we explored the possibility of using different ways to inject custom code and generate logging statements. We also explored different ways to modify the AOSP such that the changes done to the Android framework API were reflected across the entire operating system. Since the recent releases of the Android codebase have expanded extensively, we started with the Gingerbread release (Android 2.3.7).
Next step was to understand Aspect-Oriented Programming better. We researched various alternatives to support AOP and selected the AspectJ library which enabled modification of the existing code by allowing runtime injection of custom code, at the same time being Java compatible. We executed post-compile time Aspect weaving, rebuilt the AOSP and generated custom log files. Once the aspect weaving was successful on Gingerbread, we extended the work to support the latest versions of Android (Nougat).
6.2 Limitations
- Versions prior to Android 5.0 (API 21) have a limit on the total number of references that can be invoked by the code within a single Dalvik Executable (DEX) bytecode file. For each Dex file, a maximum number of allowed references is 65536. While weaving custom code, AspectJ library creates additional methods. These additionally injected methods may cause the number of references to exceed the limit, leading to a build failure. From Android 5.0, Android supports multidexing which resolves this problem.
- Android 6.0 onwards, a new build toolchain called Jack is used to speed up the source compilation time. This toolchain replaces the previous toolchain, which consisted of multiple tools, such as javac, ProGuard, jarjar and dx. With the new toolchain, Java source code is directly compiled into Dex format reducing compilation time. Since AspectJ only works on compiled class files, the Jack toolchain has to be disabled in order to use the Instrumentation tool. This may lead to increase in the compilation time. However, since the Jack toolchain is now deprecated by the Android community, this is no longer an issue.
6.3 Conclusion
We developed a tool which enables an instrumentation for the Android platform using Aspect-Oriented Programming. With this tool, a custom code can be injected into the platform. The ability to inject the custom code can be very useful in many ways such as providing developers with better insights of Android source code, monitoring the behavior of applications and checking for security violations by Android apps. The tool will be helpful in Android instrumentation for developers as well as end-users. It can also be extended and customized to meet the individual needs.
References
- Karim Yaghmour, Porting, Extending and Customizing Embedded Android, O’Reilly media Inc., March 2013.
- Joseph Gradecki, Nicholas Lesiecki, Mastering AspectJ- Aspect Oriented Programming in Java, Willey Technology Publishing, 2003
- Ramnivas Laddad, AspectJ in Action, Manning publications, 2010.
- Yliès Falcone, Sebastian Currea, Weave droid: aspect-oriented programming on Android devices: fully embedded or in the cloud, ACM publication, August 2012
- Android Developer community, The Android Source Code [online]
Available: https://source.android.com/source/index.html
- Fernando Cejas, Aspect Oriented Programming in Android [online]
Available: https://fernandocejas.com/2014/08/03/aspect-oriented-programming-in- android/
- Espen, AspectJ cheatsheet, March 2010, [online]
Available: https://blog.espenberntsen.net/2010/03/20/aspectj-cheat-sheet/
- Gregor Kiczales, Erik Hilsdale, Jim Hugunin, Mik Kersten, Jeffrey Palm and William G. Griswold, An Overview of AspectJ, [online]
Available: http://www.ccs.neu.edu/home/lieber/com3362/w02/lectures/working-
lectures/ECOOP2001-Overview.pdf
- Ankit Singh Android Inside, November 2011, [online]
Available: http://riteattitude.blogspot.com/2011/11/android-zygote-and-processes.html
- Graham O’Regan, Introduction to Aspect-Oriented Programming, January 2014, [online] Available: http://archive.oreilly.com/pub/a/onjava/2004/01/14/aop.html
- Apache Software Foundation, Commons BCEL, July 2016, [online]
Available: https://commons.apache.org/proper/commons-bcel/
- Jaku Holy, Practical Introduction into Code Injection with AspectJ, Javassist, and Java Proxy, September 2011, [online] Available: https://www.javacodegeeks.com/2011/09/practical-introduction-into-code.html
- Android Developer Community, Compiling with Jack, [online]
Available: https://source.android.com/source/jack.html
- Tim Bird, Android Build System, June 2010, [online]
Available http://elinux.org/Android_Build_System
Appendix
Steps to instrument and build Android
This section provides step by step instructions to download and build Android source with AspectJ instrumentation.
1. Establish build environment and Download Android source
- Establish your environment
Follow instructions on Android source page to establish a build environment: https://source.android.com/source/initializing.html
- Install Repo
i. Create a bin/ directory in your home directory and make sure that it is included in your path:
$ mkdir ~/bin
$ PATH=~/bin:$PATH
ii. Download the Repo tool and ensure that it is executable.
$curl https://storage.googleapis.com/git-repo-downloads/repo > ~/bin/repo
$ chmod a+x ~/bin/repo
- Create an empty directory to hold your working files.
$ mkdir Working_Directory
$ cd Working_Directory
- Configure git with your name and email address.
$ git config –global user.name “Your Name”
$ git config –global user.email “you@example.com”
- Initialize repo with the following command. (for a different version, provide a branch name with -b option. For a list of branches, see Source Code Tags and Builds.
For example,
$repo init –u https://android.googlesource.com/platform/manifest -b android-2.3.7_r1
- Download Android source
To pull down the Android source tree to your working directory from the repositories
as specified in the default manifest, run
$ repo sync
This step may take a couple of hours depending upon network speed.
- Download folder “instrumentation_tool” from git repository into Working_Directory
$git clone https://github.com/poojakanchan/instrumentation_tool.git
- Apply patch to the build files. Make sure you take a backup before making the changes. Provide execution permissions to the scripts in the instrumentation_tool folder.
$cp build/core/definitions.mk build/core/definitions_bak.mk
$ patch build/core/definitions.mk < instrumentation_tool/definitions.patch
$ chmod +x instrumentation_tool/*.sh
2. Build Android source with custom aspects
- Write custom aspects and place them in the src/ folder. (Example Aspect class “TestAspect.java” is provided) The files under the src/ folder are automatically compiled and injected by the Instrumentation tool while running a build.
- Initialize environment
$ export ANDROID_HOME=<path to Working_Directory>
$ source build/envsetup.sh
$ lunch
- Build Android source
$make showcommands
To redirect logs to a log file, run
$make showcommands 2>&1 | tee log_file
To use multiple threads, run it with -j option as
$ make -j4
Note that, building the source may require up to 2-3 hours.
- After the build is successful, launch an emulator using the command:
$ emulator
To see adb logs, use command:
$ adb logcat