Raspberry Pi vs. AWS, v2.0

Earlier this year I ran a few tests comparing a Raspberry Pi (rev 1.0) to an Amazon AWS Micro instance. There were a few interesting and insightful comments in response to the post. I want to run through a few of the open comments and questions in a second post.

What about an AWS small instance?

The most common piece of feedback I received was that I should have used a small AWS instance. I know AWS well (I work for Amazon) and I know the micro instances are capped. I chose a micro instance based on the price. Let’s compare a small and micro instance.

For an EC2 instance (non-reserved) used 100% of the time with 1 elastic IP (0 reassigns), a 10GB EBS volume using 10IOPS, and 5GB of transfer in and out per month the costs of a micro and small instance are:

Micro – $22.42
Small – $51.70

Source – AWS costs.

The Small instance is much faster, sure, but it also costs nearly one Raspberry Pi per month more. That said, I did benchmark the Small instance in the same ways as the Raspberry Pi and the micro instance.

Small AWS nbench performance

BYTEmark* Native Mode Benchmark ver. 2 (10/95)
Index-split by Andrew D. Balsa (11/97)
Linux/Unix* port by Uwe F. Mayer (12/96,11/97)

TEST : Iterations/sec. : Old Index : New Index
: : Pentium 90* : AMD K6/233*
--------------------:------------------:-------------:------------
NUMERIC SORT : 425.67 : 10.92 : 3.59
STRING SORT : 93.044 : 41.57 : 6.44
BITFIELD : 1.854e+08 : 31.80 : 6.64
FP EMULATION : 94.848 : 45.51 : 10.50
FOURIER : 11241 : 12.78 : 7.18
ASSIGNMENT : 13.702 : 52.14 : 13.52

IDEA : 2816.7 : 43.08 : 12.79
HUFFMAN : 910.19 : 25.24 : 8.06
NEURAL NET : 24.263 : 38.98 : 16.39
LU DECOMPOSITION : 675.72 : 35.01 : 25.28
==========================ORIGINAL BYTEMARK RESULTS==========================
INTEGER INDEX : 32.370
FLOATING-POINT INDEX: 25.934
Baseline (MSDOS*) : Pentium* 90, 256 KB L2-cache, Watcom* compiler 10.0
==============================LINUX DATA BELOW===============================
CPU : GenuineIntel Intel(R) Xeon(R) CPU E5507 @ 2.27GHz 2267MHz
L2 Cache : 4096 KB
OS : Linux 3.2.30-49.59.amzn1.x86_64
C compiler : gcc version 4.6.2 20111027 (Red Hat 4.6.2-2) (GCC)
libc : libc-2.12.so
MEMORY INDEX : 8.330
INTEGER INDEX : 7.893
FLOATING-POINT INDEX: 14.384
Baseline (LINUX) : AMD K6/233*, 512 KB L2-cache, gcc 2.7.2.3, libc-5.4.38
* Trademarks are property of their respective holder.

Small AWS nginx/seconds performance

NGINX Benchmark 1.0.11:
    pts/nginx-1.1.0
    Test 1 of 1
    Estimated Trial Run Count:    3
    Estimated Time To Completion: 5 Minutes
        Running Pre-Test Script @ 17:57:34
        Started Run 1 @ 17:57:39
        Started Run 2 @ 18:01:30
        Started Run 3 @ 18:05:21  [Std. Dev: 0.16%]
        Running Post-Test Script @ 18:09:09

    Test Results:
        2205.83
        2210.91
        2212.53

    Average: 2209.76 Requests Per Second

So, yes, a small AWS instance is much faster (about 343%) but also costs ~130% more. You could buy about 11 Raspberry Pis per year with the difference.

ext3 vs ext4 differences

Michael Tomkins pointed out (rightfully so) that my Raspberry Pi was using ext4 and my micro AWS instance was using ext3. As he suggested, this does make a difference. My fault for not noticing that.

I re-ran the test and partially confirmed his results with the Nginx benchmark.
EXT3: 329.83 requests/sec
EXT4: 484.27 requests/sec

Thank you, Michael – you’re totally correct!

(As a note, the v.2 Raspberry Pi scores much better on the Nginx benchmark and is well above an EXT4 micro instance)

Micro instances have inconsistent CPU

Yes, they do. If you run the byte-nbench benchmark, for instance, you can see that the results are so inconsistent, the tests cannot keep them within a 95% confidence interval.

ec2-user@ip-10-112-38-10 nbench-byte-2.2.3]$ ./nbench 

BYTEmark* Native Mode Benchmark ver. 2 (10/95)
Index-split by Andrew D. Balsa (11/97)
Linux/Unix* port by Uwe F. Mayer (12/96,11/97)

TEST                : Iterations/sec.  : Old Index   : New Index
                    :                  : Pentium 90* : AMD K6/233*
--------------------:------------------:-------------:------------
NUMERIC SORT        :
** WARNING: The current test result is NOT 95 % statistically certain.
** WARNING: The variation among the individual results is too large.
                    :          298.58  :       7.66  :       2.51
STRING SORT         :
** WARNING: The current test result is NOT 95 % statistically certain.
** WARNING: The variation among the individual results is too large.
                    :          40.973  :      18.31  :       2.83
BITFIELD            :
** WARNING: The current test result is NOT 95 % statistically certain.
** WARNING: The variation among the individual results is too large.
                    :      7.3277e+07  :      12.57  :       2.63
FP EMULATION        :
** WARNING: The current test result is NOT 95 % statistically certain.
** WARNING: The variation among the individual results is too large.
                    :          48.113  :      23.09  :       5.33
FOURIER             :
** WARNING: The current test result is NOT 95 % statistically certain.
** WARNING: The variation among the individual results is too large.
                    :          4142.6  :       4.71  :       2.65
ASSIGNMENT          :
** WARNING: The current test result is NOT 95 % statistically certain.
** WARNING: The variation among the individual results is too large.
                    :           5.812  :      22.12  :       5.74
IDEA                :
** WARNING: The current test result is NOT 95 % statistically certain.
** WARNING: The variation among the individual results is too large.
                    :          1467.3  :      22.44  :       6.66
HUFFMAN             :
** WARNING: The current test result is NOT 95 % statistically certain.
** WARNING: The variation among the individual results is too large.
                    :           534.7  :      14.83  :       4.73
NEURAL NET          :
** WARNING: The current test result is NOT 95 % statistically certain.
** WARNING: The variation among the individual results is too large.
                    :          7.9804  :      12.82  :       5.39
LU DECOMPOSITION    :
** WARNING: The current test result is NOT 95 % statistically certain.
** WARNING: The variation among the individual results is too large.
                    :          250.29  :      12.97  :       9.36
==========================ORIGINAL BYTEMARK RESULTS==========================
INTEGER INDEX       : 16.251
FLOATING-POINT INDEX: 9.217
Baseline (MSDOS*)   : Pentium* 90, 256 KB L2-cache, Watcom* compiler 10.0
==============================LINUX DATA BELOW===============================
CPU                 : GenuineIntel Intel(R) Xeon(R) CPU           E5430  @ 2.66GHz 2660MHz
L2 Cache            : 6144 KB
OS                  : Linux 3.2.30-49.59.amzn1.x86_64
C compiler          : gcc version 4.6.2 20111027 (Red Hat 4.6.2-2) (GCC) 
libc                : libc-2.12.so
MEMORY INDEX        : 3.495
INTEGER INDEX       : 4.534
FLOATING-POINT INDEX: 5.112
Baseline (LINUX)    : AMD K6/233*, 512 KB L2-cache, gcc 2.7.2.3, libc-5.4.38
* Trademarks are property of their respective holder.

As I mentioned, however, the next step up is quite costly given the fact we’re talking about a $35 computer.

What’s next?

This week I will post my results from testing a Raspberry Pi model 2.0 with 512MB RAM. In summary, it scored much better than the 1.0 model with 256MB of RAM.

One thought on “Raspberry Pi vs. AWS, v2.0

Leave a Reply