The current state of Tensorflow Lite on Mobile

I am very interested in machine learning, especially pose recognition for training purposes that runs on mobile devices. Pose recognition uses computer vision to estimate the position and angle of body joints. It can convert a 2D video into a 3D model.

OpenPose is the best accurate body (including hands and feet) complete positioning model I can find. This does not run on mobile so it’s no good for this project.

Posenet has a total of 17 key body joints and is built lite for mobile. It doesn’t offer hand and foot positioning and is not as accurate as OpenPose but for my first project, and for realtime simple analysis of body positions, it will suit just fine.

CHECKOUT AN EXAMPLE

First I am going to have a go at getting this example on my phone – https://github.com/tensorflow/examples/blob/master/lite/examples/image_classification/android/README.md

A quick git clone,

git clone https://github.com/tensorflow/examples

and ready to open Android studio, after upgrading from version 1.x because I haven’t used Android Studio for fifty-thousand years. Now I can open the downloaded example.

To build the demo on my phone I need to enable developer options and USB debugging (new phone). Then download the latest USB drivers for your phone and install a few packages required by the demo, all pretty straightforward and explained at the git page.

The build worked ok on my Nokia. Latency was around 1.5 seconds, actually pretty bad. I think it has to do with Antivirus/Firewall screwing things, add exceptions and check firewall, all good. Maybe I need more SDK/API’s, can never have enough. This is definitely it as I do need NDK 21, quick 1gig download later…no improvement.

This is a current issue with Tensorflow Posnet – https://github.com/tensorflow/tensorflow/issues/34230, It is laggy. It will run on the high end device and has speed improvements when running on 4 threads on CPU, but still, it’s 150ms delay, not good enough for real-time.

CONCLUSION

Real-time pose estimation is not ready for mobile. High-end devices like new iPhones and Pixel run it well. For future projects, I will avoid realtime and probably use OpenPose because of its hands and feet abilities.