AI Food classification and nutrient facts from photo

In the age of digital information when all information is one or two clicks away, but there are many users who still don’t know nothing about what they eat and how eating affects their body. It’s common knowledge to know you need to drink tea and eat agrumes (oranges and lemons) when you feel sick and not so common to know how much vitamins are actually in lemon and that broccoli has more vitamin C than a lemon. But eating broccoli can challenge itself if you are not up for the challenges you better stop reading now :)

Nutrient information is quite easy to find, even google search will show you nutrient facts about some of most common food items. Google fetches this data from large database USDA Food Composition Database. Luckily for us, this service also has RESTful API access and we will come back to that topic.

There are numerous apps available with information of nutrients in common foods or prepared meals. They all rely on search from a large database of foods but some of them already use artificial intelligence to ease your search with the use of similar architecture as our own implementation.

Why we have built this app?! To be honest we started this project while ago before there were other solutions similar to our own and we needed a project with which we can test our knowledge about image classification neural networks and step into the age of machine learning. Progress in applied artificial intelligence projects, especially deep neural nets for image classification, has been and will remain fast due ease of use one of many open source neural net platforms & libraries released in past years and of course advances in hardware making everything possible, but that was predictable by the Moore’s Law. Tensorflow from Google Brain Team, Caffe from UC Berkeley, Torch, MXNET, Keras, just to name few of the platforms and high-level libraries that made machine learning, especially deep learning, easier and contribute significantly to faster research and popularity of the field within computer scientists. Such a good age to be a software developer and AI enthusiast.

I’ll try to make this subject easy to follow, talk more about challenges and less technical. You will be able to read the more detailed technological guide in one of our next articles. I’ve started to learn about computer vision as a result of a couple of projects in our company. I was always interested in artificial intelligence and vision is its subfield that I understand most, I was always more visual type. Image classification is also the most talk about and most easy to follow the case of applied artificial intelligence. All you need is TensorFlow, training images (lots of them) and time. For this case, we’ve used a method called transfer learning from trained Inception v3 model. Inception v3 model was the most appropriate model at the time and we could easily follow training examples from TensorFlow web page. If we could do this again we would probably use Caffe2 and MobileNet, for faster detection rate on CPU and directly implement a solution in-app on clientside, as our app is the HTML5 angular app we would probably wait for WebAssembly support on mobile browsers.

We’ve created one powerful backend engine for app development and management. NAPP was configured in a couple of minutes, all we need to code was microservice adapter for FCD API, define our data model in JSON for MongoDB adapter to format that was optimal for our app, define our RESTful API methods, enable Redis cache and all the hard work was done by our NAPP platform, we had that part ready in a couple of hours.

Apps markets and usage today is not as it was a couple of years back. Users are less willing to download the app from the store. That’s why we choose Web as a platform for our app. The app runs like native inside web browsers, it uses angular as a framework and has some clever custom caching techniques besides regular progressive web app service worker. We had some issues with camera capture as getUserMedia API (part of WebRTC) is not supported on older browsers, but we managed to solve that with the good old HTML media capture.