Monday 16 June 2014

SemusiSDK Accuracy - A Comparative Analysis


Semusi has been developing cutting-edge Machine Learning solutions with its SemusiSDK for Human Activity Recognition, Demographics recognition, Place labeling and Interest detection. As our systems have matured, we are now benchmarking our solutions against popular Android applications in the market.

This benchmark is against “Moves”, a popular pedometer applications that measures the steps taken by a user and the duration the user has been walking. We asked 6 people - 3 females and 3 male in their mid-20's, to walk 500 steps. The people were between 5 feet 3 inches to 6 feet in height and between 60-80kg in weight. We used 4 high- and low-end devices held in hand, which took 4.47 minutes of walking time. The GPS in all devices was turned off. The devices used were -

  1. Samsung S4
  2. Samsung S2
  3. Moto G
  4. Micromax A21


Summary:

Counting Steps -
As we can see from the graphs, SemusiSDK performs with 63.39% better accuracy on average in counting steps than Moves. SemusiSDK logged 323.5 steps on average, as against 182.58 by Moves.

SemusiSDK is 63.39% more accurate in the best case when using Micromax A21. SemusiSDK logged 285 steps on average, as against 104.33 by Moves.

SemusiSDK performs 59.80% better for females and 25.48% better for males than Moves for counting steps. SemusiSDK logged 340.83 steps on average for females and 306.17 for males, as against 137 and 228.17 respectively by Moves.

Counting Walking Duration -
As can be seen, SemusiSDK is 3.67x better on average for calculating minutes walked than Moves. SemusiSDK logged 3.21 minutes on average, as against 0.69 by Moves.

In the best case, SemusiSDK is 9.18x better when calculating minutes walked than Moves on Samsung Galaxy S4. SemusiSDK logged 3.39 minutes on average, as against 0.33 by Moves.

As in the plots, we can see that SemusiSDK 5.38x better for calculating minutes walked for females and 2.56x better for males than Moves. SemusiSDK logged 3.45 minutes for females and 2.97 minutes for males on average, as against 0.54 and 0.83 respectively by Moves.


 Figure 1: Average steps calculated by SemusiSDK and Moves, and the actual steps walked


 Figure 2: Average steps using Samsung Galaxy S4 over 3 Females, 3 Males


 Figure 3: Average steps using Samsung Galaxy S2 over 3 Females, 3 Males


 Figure 4: Average steps using Moto G over 3 Females, 3 Males


 Figure 5: Average steps using Micromax A21 over 3 Females, 3 Males


 Figure 6: Average steps calculated by SemusiSDK and Moves for 3 Females


 Figure 7: Average steps calculated by SemusiSDK and Moves for 3 Males


 Figure 8: Average minutes walked as calculated by SemusiSDK and Moves, and the actual steps walked


 Figure 9: Average minutes walked using Samsung Galaxy S4 over 3 Females, 3 Males


 Figure 10: Average minutes walked using Samsung Galaxy S2 over 3 Females, 3 Males


 Figure 11: Average minutes walked using Moto G over 3 Females, 3 Males


 Figure 12: Average minutes walked using Micromax A21 over 3 Females, 3 Males


 Figure 13: Average minutes walked by SemusiSDK and Moves for 3 Females



Figure 14: Average minutes walked by SemusiSDK and Moves for 3 Males



Table 1
The actual logged data is as below.
Actual Steps – 500
Actual Minutes – 4.47

User
Device
SemusiSDK
Recorded Steps
SemusiSDK
Recorded Minutes
Moves
Recorded Steps
Moves
Recorded Minutes
Female 1
Samsung Galaxy S4
507
3.37
383
0

Samsung Galaxy S2
259
2.92
0
0

Moto G
53
1.17
60
0

Micromax A21
310
3.9
5
0
Female 2
Samsung Galaxy S4
794
4.9
463
0

Samsung Galaxy S2
380
4.5
0
0

Moto G
281
5.22
18
0

Micromax A21
310
3.9
5
0
Female 3
Samsung Galaxy S4
434
2.78
440
0

Samsung Galaxy S2
295
2.81
270
2

Moto G
265
3.32
0
0

Micromax A21
202
2.55
0
4.47
Male 1
Samsung Galaxy S4
335
3.05
319
0

Samsung Galaxy S2
273
2.6
101
2

Moto G
278
3.29
0
0

Micromax A21
225
2.55
0
0
Male 2
Samsung Galaxy S4
554
3.38
469
2

Samsung Galaxy S2
232
2.78
163
0

Moto G
293
3.94
176
0

Micromax A21
329
2.69
435
0
Male 3
Samsung Galaxy S4
274
2.88
390
0

Samsung Galaxy S2
290
2.58
270
2

Moto G
257
3.31
234
4

Micromax A21
334
2.5995
181
0






No comments:

Post a Comment