Debugging a truly remote android device

01 Sep, 2023

There has come a time when remote work isn't seen as a reward anymore, where you'd get a day a week to work from home because you dotted all your i's and crossed all your t's, so we need to adapt.

Lucky for us the client-server architecture has prepared us for just this occasion, all you need to do is start a server and connect to it through a client, be it for just getting a secure shell, securely copying a file, or even developing through Visual Studio Code and building docker images on different machines.

One thing that I needed, though, wasn't readily available. I needed to debug a native C++ application on truly remote android devices (off site, but connected to a machine I had access to). This led me on an adventure of digging down into the guts of a real dragon of an open source project and eventually submitting back a solution that even trickled down to Android Studio itself!

We will use a starter C++ application that Android Studio can generate for us in order to test all of this.

If you're not interested in my ramblings how this got implemented and just want to know how to debug native android apps on a remote device, jump to Part 4.

Part 1: how it's done locally (clients and servers all the way down)

Developing an android application through Android Studio makes deploying it as easy as one-two-three. Plug the phone in, select it in the dropdown menu and press ▶️. The application is built, installed on the device, and launched, all on a click of a button.

Debugging said application is just as easy, but you press the 🐞 button instead of the ▶️ button and Android Studio will do everything automagically for you.

Let's see what is actually going on behind the curtains.

Android Debug Bridge (adb)

The communication between the device and Android Studio goes through adb, a client-server architecture program bundled into one binary that can do lots of things, from listing all the connected devices (and/or emulators), giving info about them (SDK version, CPU ABI, brand, manufacturer, model etc.), as well as pushing things onto said devices, and even forwarding ports between the two. It's a Swiss army knife when it comes to interacting with android devices.

It consists of two parts on the host side (adb client and adb server) and adb daemon on the device side.

When you run adb in the terminal, it checks if the server is running and if it isn't, it spins it up. The server listens on the port 5037 by default, and that's where Android Studio expects it to be.

Pressing the ▶️ button in Android Studio builds the project, installs the .apk on the device using adb install and launches the application through, you've guessed it, adb.

Pressing the little 🐞 button is a bit more involved.

In addition to installing the application, we also need a debugger on the device since that's where the code will run. Android Studio uses lldb (a part of the llvm-project) in the background and does the following:

• launches the application in debug mode
• attaches the java debugger (jdb)
• copies the lldb-server for the device's CPU architecture onto the device
• launches the lldb-server in platform mode¹
• attaches the lldb client

The reason why we need two debuggers is that one is for the Java part of the app (jdb), and the other is for the native part (lldb).

The LLVM debugger (lldb)

Surely it comes to you as a shock that lldb is also using a client-server architecture to handle the debugging.

When executing a binary through lldb, it in fact brings up a gdb-stub process that launches the binary, and all the communication goes through that gdb-stub. In order to reduce code complexity and improve remote debugging experience, this happens even when debugging a process locally by going through the loopback interface, but what happens when we want to debug something on a different platform, in this case android?

lldb-server is the counterpart to the lldb client, and it comes bundled with Android Studio in x86, x86_64, armv7, and armv8 flavours. It can be launched in platform mode which provides the client (your CLI lldb) with something to connect to and allows it to do more complex stuff in the background automagically.

Part 2: the problem

The device we are trying to debug is not connected to the machine where Android Studio is running (codename earth) we need to get to it somehow, which we will do by lying to Android Studio (in a way).

We will drill an ssh tunnel to the remote machine (codename mars) so that Android Studio thinks it's talking to a local adb instance, while in reality it's nowhere near the local machine.

Let's prepare the remote machine the phone is connected to by starting the adb server on it, we will launch it on the default port of 5037:

(mars) $> adb nodaemon server start
adb I 02-28 07:47:09 574697 574697 auth.cpp:417] adb_auth_init...
adb I 02-28 07:47:09 574697 574697 auth.cpp:417] loaded new key from '/home/user/.android/adbkey' with fingerprint 0C663E80DC28AEF63EC510C0B34415C410BA9675C123DE7D21B66F9F759ACB39
adb I 02-28 07:47:09 574697 574697 auth.cpp:417] adb_auth_inotify_init...

Let's check that the device (codename deimos) is plugged in and ready for work:

(mars) $> adb devices
List of devices attached
deimos          device

Now that that's working, we need to put the ssh tunnel from earth to mars into place.

We will forward all the traffic going through a port on earth to a port on mars, so that we can fool Android Studio we have an adb server on port 6000, while in reality it just passes through the commands to adb on mars on the port 5037.

And to save you the trouble of all the future forwards, let's just forward all the ports we will need on this journey and explain them along the way through the blogpost:

# to clarify, this means: -L<earth-port>:localhost:<mars-port>
(earth) $> ssh -CN -L6000:localhost:5037 -L54321:localhost:54321 -L15000:localhost:15000 -L16000:localhost:16000 user@mars

There are other ways to use a remote adb², but we will stick with ssh tunnelling for the time being.

Forwarding earth's port 6000 to mars' port 5037 allows us to talk to the remote adb instance, and setting the variable ANDROID_ADB_SERVER_PORT³ to 6000 allows us not to specify the port in all future invocations of adb.

Does it work from our local machine?

# note that this export has effect only on the current shell!
# if you don't wish to export variables, feel free to use `adb -P 6000` for all invocations
(earth) $> export ANDROID_ADB_SERVER_PORT=6000
(earth) $> adb devices
List of devices attached
deimos          device

Great!

Now that that's working, let's try connecting to it through Android Studio! Open up the settings in Android Studio and search for Existing ADB server port. Set it to 6000. Android Studio should detect deimos as connected, and we can launch the app by pressing ▶️ with no hiccups.

Pressing the 🐞 icon however gets us...

Failed to connect port

Process finished with exit code 0

Well that's not really helpful.

This all works as expected when debugging android apps locally, and in theory, it should all work even when that's not the case, judging by the client-server architecture, surely someone expected this, right?

Maybe we're just holding it wrong, let's do it manually!

For the app to be able to launch a server a debugger can connect to (which is what we're trying to do), we have to modify the app/src/main/AndroidManifest.xml to give it the permission to use the internet:

<manifest xmlns:android="http://schemas.android.com/apk/res/android"
     xmlns:tools="http://schemas.android.com/tools">

+    <uses-permission android:name="android.permission.INTERNET" />
     <application
         android:allowBackup="false"

And now build and install it again (the task installDebug rebuilds the app, no worries there):

(earth) $> ./gradlew installDebug

Now we are ready to launch the app in debug mode.

(earth) $> adb shell am start -D com.example.myapplication/com.example.myapplication.MainActivity -a android.intent.action.MAIN, -c android.intent.category.LAUNCHER

Since android allows the developers to write code either in a JVM based language (using the android SDK) or in C/C++ (using the android NDK (which is also technically an SDK, but I digress)), launching the app in debug mode expects that a Java debugger will be connected so it brings a popup stating "Waiting for debugger..." and it won't go away until we do so, for which we need to forward (through ssh) yet another port, let's take 54321 for that.

Let's get the process ID (PID) of our application using pidof.

(earth) $> adb shell pidof -s "com.example.myapplication"
32003

Note: For android with versions older than I can't be bothered to bisect it, pidof doesn't work as expected so there's a workaround⁴

This is where the already forwarded port (earth:54321) -> (mars:54321) comes into play, we will point it to the device using the jdwp⁵ protocol since adb knows how to handle such connections and we're all set to actually debug the app!

(earth) $> adb forward tcp:54321 jdwp:<insert PID here>
(earth) $> jdb -attach localhost:54321

That's that for the Java part of the app, judging by the lldb documentation, we need to launch the lldb-server on deimos, so let's copy it onto the device. Since the external storage is mounted with the noexec flag, the executable needs to be copied to the debuggee's private storage before launching.

We will be using the lldb-server that comes bundled with Android Studio, if you're using the bundle downloaded from the official website, it's in <extracted-bundle>/plugins/android-ndk/resources/lldb/android/<architecture>/.

First get the architecture of deimos and then push the right lldb-server to it:

(earth) $> adb shell getprop ro.product.cpu.abi
arm64-v8a
(earth) $> adb push /path/to/correct/architecture/lldb-server /data/local/tmp/lldb-server
(earth) $> adb shell run-as com.example.myapplication cp /data/local/tmp/lldb-server .

We can specify the port on which the platform server will listen for a client, as well as the port for gdb server; once it gets running the platform will communicate with it through that port.

In order to connect to the lldb-server running on the device through our lldb client, we need to forward said ports from mars to deimos, and from earth to mars (that's why we forwarded 15000 and 16000 using ssh earlier).

(earth) $> adb forward tcp:15000 tcp:15000   # from mars to deimos
(earth) $> adb forward tcp:16000 tcp:16000   # from mars to deimos
(earth) $> adb shell run-as com.example.myapplication ./lldb-server platform --listen "*:15000" --gdbserver-port 16000

And now let's connect with our lldb client:

# don't forget to export ANDROID_ADB_SERVER_PORT=6000
(earth) $> lldb
(lldb)> platform select remote-android
  Platform: remote-android
 Connected: no
(lldb)> platform connect connect://localhost:15000
error: Failed to connect port

Well that was a bust.

Part 3: the solution

What's going on here? Judging by the error, we're not doing what we think we are.

Let's try to do it manually again, but this time with the logs enabled.

(earth) $> lldb
(lldb)> log enable lldb platform
(lldb)> log enable gdb-remote all
(lldb)> platform select remote-android
lldb             PlatformAndroid::CreateInstance(force=true, arch={<null>,<null>})
lldb             PlatformAndroid::CreateInstance() creating remote-android platform
  Platform: remote-android
 Connected: no
(lldb)> platform connect connect://localhost:15000
lldb             Connected to Android device "deimos"
lldb             Forwarding remote TCP port 15000 to local TCP port 34667
lldb             Rewritten platform connect URL: connect://127.0.0.1:34667
error: Failed to connect port

Huh? Where did the port 34667 come from?

Let's look at the source code of lldb⁶, searching through it for the error messages we get in the logs, and reading through the code brings us to the function MakeConnectURL.

Status PlatformAndroidRemoteGDBServer::MakeConnectURL(
    const lldb::pid_t pid, const uint16_t remote_port,
    llvm::StringRef remote_socket_name, std::string &connect_url) {
  static const int kAttempsNum = 5;

  Status error;
  // There is a race possibility that somebody will occupy a port while we're
  // in between FindUnusedPort and ForwardPortWithAdb - adding the loop to
  // mitigate such problem.
  for (auto i = 0; i < kAttempsNum; ++i) {
    uint16_t local_port = 0;
    error = FindUnusedPort(local_port);
    if (error.Fail())
      return error;

    error = ForwardPortWithAdb(local_port, remote_port, remote_socket_name,
                               m_socket_namespace, m_device_id);
    if (error.Success()) {
      m_port_forwards[pid] = local_port;
      std::ostringstream url_str;
      url_str << "connect://127.0.0.1:" << local_port;
      connect_url = url_str.str();
      break;
    }
  }

  return error;
}

Alright, what do we got? Five attempts, a comment about race possibility, yadda yadda yadda, and then we... we find an unused port for the local port?

Huh.

It would seem the local ports forwarded from the machine running adb to the android device are chosen at random, which means that when our earth stationed lldb client tried accessing them, it fell on deaf ears because they were forwarded from mars to deimos, but not from earth to mars.

Now I don't know if you've built llvm before, but it takes a while, so this might have actually been the time when it was in fact better to read the documentation, logs, and the code, rather than use just sprinkle printf everywhere and recompile all of it every other keystroke. That being said, I still had to try out the solution so my CPU got the short end of the stick one way or the other.

The fix is simple enough, we need to allow the user to configure the local ports somehow. Using environment variables seemed like the easiest approach so that's what I went with.

We just have to modify the MakeConnectURL signature to take another parameter, find all the references to it and supplant the calls with the value from an environment variable if it exists.

There are only two callers, one for launching the gdb stub, and one for the platform communication.

The diff for both cases is pretty much the same:

-  auto error =
-      MakeConnectURL(pid, remote_port, socket_name.c_str(), connect_url);
+  uint16_t local_port = 0;
+  const char *gdbstub_port = std::getenv("ANDROID_PLATFORM_LOCAL_GDB_PORT");
+  if (gdbstub_port)
+    local_port = std::stoi(gdbstub_port);
+
+  auto error = MakeConnectURL(pid, local_port, remote_port, socket_name.c_str(),
+                              connect_url);
   if (error.Success() && log)
     LLDB_LOGF(log, "gdbserver connect URL: %s", connect_url.c_str());

MakeConnectURL assumes that if the given port is non-zero, it's free so it can just be forwarded, no checks needed. The burden of making sure it actually is available lies on the user.

 Status PlatformAndroidRemoteGDBServer::MakeConnectURL(
-    const lldb::pid_t pid, const uint16_t remote_port,
-    llvm::StringRef remote_socket_name, std::string &connect_url) {
+    const lldb::pid_t pid, const uint16_t local_port,
+    const uint16_t remote_port, llvm::StringRef remote_socket_name,
+    std::string &connect_url) {
   static const int kAttempsNum = 5;

   Status error;
+
+  auto forward = [&](const uint16_t local, const uint16_t remote) {
+    error = ForwardPortWithAdb(local, remote, remote_socket_name,
+                               m_socket_namespace, m_device_id);
+    if (error.Success()) {
+      m_port_forwards[pid] = local;
+      std::ostringstream url_str;
+      url_str << "connect://127.0.0.1:" << local;
+      connect_url = url_str.str();
+    }
+    return error;
+  };
+
+  if (local_port != 0)
+    return forward(local_port, remote_port);
+
   // There is a race possibility that somebody will occupy a port while we're
   // in between FindUnusedPort and ForwardPortWithAdb - adding the loop to
   // mitigate such problem.
   for (auto i = 0; i < kAttempsNum; ++i) {
     uint16_t local_port = 0;
     error = FindUnusedPort(local_port);
     if (error.Fail())
       return error;

-    error = ForwardPortWithAdb(local_port, remote_port, remote_socket_name,
-                               m_socket_namespace, m_device_id);
-    if (error.Success()) {
-      m_port_forwards[pid] = local_port;
-      std::ostringstream url_str;
-      url_str << "connect://127.0.0.1:" << local_port;
-      connect_url = url_str.str();
+    if (forward(local_port, remote_port).Success())
       break;
-    }
   }

   return error;

It's worth it to refactor the actual forwarding logic into a lambda given that we have to use it twice.

And with the patch in place, we're ready to take it for a spin!

Launching the app:

(earth) $> adb shell am start -D com.example.myapplication/com.example.myapplication.MainActivity -a android.intent.action.MAIN, -c android.intent.category.LAUNCHER

Launching the lldb-server:

(earth) $> adb shell run-as com.example.myapplication ./lldb-server platform --listen "*:15000" --gdbserver-port 16000

Launching the patched up lldb client:

(earth) $> export ANDROID_PLATFORM_LOCAL_PORT=15000
(earth) $> export ANDROID_PLATFORM_LOCAL_GDB_PORT=16000
(earth) $> ./llvm-project/build/bin/lldb
(lldb)> platform select remote-android
  Platform: remote-android
 Connected: no
(lldb)> platform connect connect://localhost:15000
(lldb)> attach <PID of the app>
(lldb)> continue

Huzzah!

Part 4: back to Android Studio

Now let's try doing the same through Android Studio!

First thing we have to do is dig the tunnels from our local machine (earth) to the remote server (mars) running adb:

(earth) $> ssh -CN -L<local-port-goes-here, e.g. 6000>:localhost:5037 -L15000:localhost:15000 -L16000:localhost:16000 user@mars

Second step, check that locally running an adb command passes it through to adb running mars:

(earth) $> adb -P 6000 devices
List of devices attached
deimos          device

Next, in order for Android Studio to take our required environment variables seriously, we will set them through ~/.lldbinit, a configuration file that lldb sources when launching:

script os.environ['ANDROID_ADB_SERVER_PORT'         ] = '6000'
script os.environ['ANDROID_PLATFORM_LOCAL_PORT'     ] = '15000'
script os.environ['ANDROID_PLATFORM_LOCAL_GDB_PORT' ] = '16000'

Lastly, open up the settings in Android Studio and search for Existing ADB server port. Set it to the locally forwarded port, in our case that's 6000.

Android Studio should be seeing the deimos device now and we are ready for liftoff! Press the 🐞 icon, and the breakpoints in C++ code should now be hittable!

(don't forget to revert these changes if you wish to debug locally, use a leading # for commenting in lldbinit)

Miscellaneous

If you need to interact with deimos to trigger a breakpoint, there's a program called scrcpy that brings a window representing your phone's screen. It needs the port 27183 by default, but reverse forwarded:

(earth) $> ssh -CN -L6000:localhost:5037 -R27183:localhost:27183 user@mars
(earth) $> ANDROID_ADB_SERVER_PORT=6000 scrcpy

And if you're trying out parts 1-3 (i.e. not through Android Studio), be sure to use llvm17, since the neccesary patch didn't make it into previous versions.

Closing thoughts

And that brings this little adventure to a close! Hopefully you're now able to debug android applications on truly remote devices!

This was a rollercoaster for me but I learned a ton so I wanted to share (a part of) the experience with others! There was reversing shared libraries, hardcoding values, undocumented adb variables, trying to takeup all the ports except 2 so that the lldb random port allocation was predictable, and many other things and ideas along the way, I'm just glad I finally got it working and that it trickled down to Android Studio.

It wasn't a one-person-adventure, so thanks to my coworkers, and thank you for reading! :)

#android #debug #lldb #remote