In order to add on-device collections to your Android app, it is necessary to:
- create an on-device collection in CraftAR;
- generate a collection bundle for your SDK version (On-device Image Recognition SDK or Augmented Reality SDK v4 +;
- download the bundle of that collection from CraftAR and add it to the app;
- you are ready to go and start using the on-device collection with the SDK
Once the collection is added to the device, you can start using the on-device Image Recognition SDK to perform visual search queries on the device. You can find more details for the first step in the tutorial about how to create an on-device collection in CraftAR. And for the second and third steps, please read the tutorial about how to Manage on-device collections for the Android SDKs.
An Image Recognition app using the native On-device Android Image Recognition SDK can be implemented by following two steps. First, set up the Activity/Fragment and then run image recognition to get the results for each item that is recognized.
If you want to see an example that implements On-device Image Recognition, take a look at the open source samples available in our Github repository.
1. Set up the SDK in your app
Loading collections into memory
Once the collection is added to the local database, we can set the collection that will be used for on-device image recognition using the CraftAROnDeviceIR instance. Loading collections can take a few seconds, depending on the amount of images to load. The SDK provides progress feedback for this process as well.
To load the collection into the device memory we will use the setCollection() method. This call is also asynchronous. To receive the callbacks, you will need an SetCollectionListener . You can implement these listeners in your android Activity.
Note that this could also be done in the same CraftARActivity where you perform the recognition process.
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
|
public class MyLoadCollectionActivity extends Activity implements SetOnDeviceCollectionListener{
CraftAROnDeviceCollectionManager mCollectionManager;
public void onCreate(Bundle savedInstanceState) {
...
mCollectionManager = CraftAROnDeviceCollectionManager.Instance();
mCollection = mCollectionManager.get("your_collection_token");
if(mCollection != null){
//The collection is available in the local database. Load it!
loadCollection(collection);
}else{
//TODO: Load the collection to the local database
//
}
}
public void loadCollection(CraftARCollection collection){
//Load this collection in memory!
CraftAROnDeviceIR.Instance().setCollection(collection);
}
@Override
public void collectionReady() {
//Collection is ready for recognition
//TODO: Start recognition activity
}
@Override
public void setCollectionFailed(CraftARError error) {
// Error loading the collection into memory. No recognition can be performed
// unless a collection has been set.
}
@Override
public void setCollectionProgress(double progress) {
// The images from the collection are loading into memory.
// You will have to load the collections into memory every time you open the app.
Log.d(TAG,"SetCollectionProgress:"+progress);
}
}
|
Extending CraftARActivity and implementing CraftARSearchResponseHandler
The activity where the image recognition is running must extend from CraftARActivity. If you want to place the Camera view in a Fragment, you can do it, but ensure that its parent activity extends CraftARActivity. (Check the examples in github to see how to implement the camera view inside a fragment).
The image recognition requests are asynchronous. To receive the search responses, you need a CraftARSearchResponseHandler. In this case, and for simplicity, we will implement the interface in the same activity.
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
|
public class MyImageRecognitionActivity extends CraftARActivity
implements CraftARSearchResponseHandler {
...
@Override
public void onPostCreate() {
}
@Override
public void searchResults(ArrayList result,
long searchTimeMillis, int requestCode) {
// You will receive here the search results
}
@Override
public void searchFailed(CraftARError error, int requestCode) {
// Some search request has failed.
}
}
|
Set up the camera capture
The CraftARSDK class manages the camera capture. Once the view is loaded, get the instance of the CraftARSDK and start the camera capture.
If the camera initialization fails, the method onCameraOpenFailed() will trigger. This can happen for multiple reasons, but it basically means that the camera could not be accessed. Sometimes, restarting the camera works. Sometimes you will have to restart the device. Note that this can also happen if another application is using the camera (for example, the flashlight), or if you haven’t added the necessary permissions in the AndroidManifest
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
|
CraftARSDK mCraftARSDK;
@Override
public void onPostCreate() {
View mainLayout= (View) getLayoutInflater().inflate(R.layout.your_activity_layout, null);
setContentView(mainLayout);
// Obtain an instance and initialize the CraftARSDK (which manages the camera interaction).
mCraftARSDK = CraftARSDK.Instance();
mCraftARSDK.startCapture(this);
}
@Override
public void onCameraOpenFailed(){
super.onCameraOpenFailed();
// TODO: Decide what to do in the app
}
|
Prepare the SDK to process searches
The CraftARSDK class also manages the search processes by sending search events to the SearchController, in this case the CraftAROnDeviceIR instance. The CraftAROnDeviceIR instance performs visual searches in the collection of images that is previously set for this app. You can find more details about setting the bundles in the separate tutorial about how to manage collection bundles with the On-device Image Recognition SDK.
We suggest to set the CraftAROnDeviceIR instance as the searchController. The searchController is the object that will manage the singleShotSearch(), startFinder() and stopFinder() calls of the CraftARSDK. We will also set our Activity as the CraftARSearchResposneHandler to receive the responses from the search process.
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
|
CraftAROnDeviceIR mCraftAROnDeviceIR;
@Override
public void onPostCreate() {
...
// Get the instance to the OnDeviceIR singleton
mOnDeviceIR = CraftAROnDeviceIR.Instance(getApplicationContext());
// Tell the SDK who manage the calls to singleShotSearch() and startFinding().
// In this case, as we are using on-device-image-recognition, we will tell the
// SDK that the OnDeviceIR singleton will manage this calls.
mCraftARSDK.setSearchController(mOnDeviceIR.getSearchController());
// Tell the SDK that we want to receive the search responses in this class.
mOnDeviceIR.setCraftARSearchResponseHandler(this);
}
|
2. Implementing the searches and parsing the results
Once you are ready with the previous steps, it’s time to add code to start scanning the real world.
Using the CraftARSDK class we can search the on-device image database in two modes: Finder Mode or Single Shot Mode.
Option A. Use Single Shot Mode to take a single picture
Call singleShotSearch to perform a search with a single image. By calling this method, the SDK triggers the still image capture from the camera and forwards the captured image to the search controller (in this case, the CraftAROnDeviceIR instance)
A common approach consists in triggering the search when the user performs an action on the UI (for example, clicking on a button). Ensure that you have initialized everything before allowing the user to perform the singleShotSearch, otherwise it will fail.
|
1
|
mCraftARSDK.singleShotSearch();
|
The response to a query triggers the searchResults callback with the results available in an array that is ready to parse. If the query failed, the searchFailed() callback will be triggered. A query can fail for multiple reasons, but the most common one is when the query image does not have enough details. If you point to a completely uniform surface (i.e a white wall), you will receive this error.
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
|
@Override
public void searchResults(ArrayList results,
long searchTimeMillis, int requestCode) {
// Callback with the search results
if(results.size() > 0){
// We found something! Log the results
for(CraftARResult result:results){
Log.d(TAG, "Found item :"+result.getItem().getItemName());
}
}else{
//Nothing found
}
// Restart the camera (it freezes after singleShotSearch until you restart the capture)
mCraftARSDK.getCamera.restartCapture();
}
@Override
public void searchFailed(CraftARError error, int requestCode) {
Log.e(TAG, "Search failed( "+error.getErrorCode()+"):"+error.getErrorMessage());
//Some error occurred. We just show log the error
mCraftARSDK.getCamera.restartCapture();
}
|
Option B. Use Finder Mode for continuous scanning
Call startFinder to start searching continuously without user intervention. This method forwards the camera frames to the search controller (in this case, the CraftAROnDeviceIR instance)
|
1
|
mCraftARSDK.startFinder();
|
For every frame that is processed, the SDK generates a query and searches for matches in the local collection. The response to a query produces a callback to searchResults with the results available in an array that is ready to parse.
|
1
2
3
4
5
6
7
8
9
10
11
12
|
@Override
public void searchResults(ArrayList results,
long searchTimeMillis, int requestCode) {
// Callback with the search results
if(results.size() > 0){
// We found something! Stop the finder and log the results
mCraftARSDK.stopFinder();
for(CraftARResult result:results){
Log.d(TAG, "Found item :"+result.getItem().getItemName());
}
}
}
|