Question:
I am running a very simple RESTful API on AWS using Node.js. The API takes a request in the form of ‘/rest/users/jdoe’ and returns the following (it’s all done in memory, no database involved):
|
1 2 3 4 5 6 |
{ username: 'jdoe', firstName: 'John', lastName: 'Doe' } |
The performance of this API on Node.js + AWS is horrible compared to the local network – only 9 requests/sec vs. 2,214 requests/sec on a local network. AWS is running a m1.medium instance whereas the local Node server is a desktop machine with an Intel i7-950 processor. Trying to figure out why such a huge difference in performance.
Benchmarks using Apache Bench are as follows:
Local Network
10,000 requests with concurrency of 100/group
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 |
> ab -n 10000 -c 100 http://192.168.1.100:8080/rest/users/jdoe Document Path: /rest/users/jdoe Document Length: 70 bytes Concurrency Level: 100 Time taken for tests: 4.516 seconds Complete requests: 10000 Failed requests: 0 Write errors: 0 Total transferred: 2350000 bytes HTML transferred: 700000 bytes Requests per second: 2214.22 [#/sec] (mean) Time per request: 45.163 [ms] (mean) Time per request: 0.452 [ms] (mean, across all concurrent requests) Transfer rate: 508.15 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.4 0 2 Processing: 28 45 7.2 44 74 Waiting: 22 43 7.5 42 74 Total: 28 45 7.2 44 74 Percentage of the requests served within a certain time (ms) 50% 44 66% 46 75% 49 80% 51 90% 54 95% 59 98% 65 99% 67 100% 74 (longest request) |
AWS
1,000 requests with concurrency of 100/group
(10,000 requests would have taken too long)
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 |
C:\apps\apache-2.2.21\bin>ab -n 1000 -c 100 http://54.200.x.xxx:8080/rest/users/jdoe Document Path: /rest/users/jdoe Document Length: 70 bytes Concurrency Level: 100 Time taken for tests: 105.693 seconds Complete requests: 1000 Failed requests: 0 Write errors: 0 Total transferred: 235000 bytes HTML transferred: 70000 bytes Requests per second: 9.46 [#/sec] (mean) Time per request: 10569.305 [ms] (mean) Time per request: 105.693 [ms] (mean, across all concurrent requests) Transfer rate: 2.17 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 98 105 3.8 106 122 Processing: 103 9934 1844.8 10434 10633 Waiting: 103 5252 3026.5 5253 10606 Total: 204 10040 1844.9 10540 10736 Percentage of the requests served within a certain time (ms) 50% 10540 66% 10564 75% 10588 80% 10596 90% 10659 95% 10691 98% 10710 99% 10726 100% 10736 (longest request) |
Questions:
- Connect time for AWS is 105 ms (avg) compared to 0 ms on local network. I assume that this is because it takes a lot more time to open a socket to AWS then to a server on a local network. Is there anything to be done here for better performance under load assuming requests are coming in from multiple machines across the globe.
- More serious is the server processing time – 45 ms for local server compared to 9.9 seconds for AWS! I can’t figure out what’s going on in here. The server is only pressing 9.46 requests/sec. which is peanuts!
Any insight into these issues much appreciated. I am nervous about putting a serious application on Node+AWS if it can’t perform super fast on such a simple application.
For reference here’s my server code:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
var express = require('express'); var app = express(); app.get('/rest/users/:id', function(req, res) { var user = { username: req.params.id, firstName: 'John', lastName: 'Doe' }; res.json(user); }); app.listen(8080); console.log('Listening on port 8080'); |
Edit
Single request sent in isolation (-n 1 -c 1)
|
1 2 3 4 5 6 7 8 9 10 11 12 |
Requests per second: 4.67 [#/sec] (mean) Time per request: 214.013 [ms] (mean) Time per request: 214.013 [ms] (mean, across all concurrent requests) Transfer rate: 1.07 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 104 104 0.0 104 104 Processing: 110 110 0.0 110 110 Waiting: 110 110 0.0 110 110 Total: 214 214 0.0 214 214 |
10 request all sent concurrently (-n 10 -c 10)
|
1 2 3 4 5 6 7 8 9 10 11 12 |
Requests per second: 8.81 [#/sec] (mean) Time per request: 1135.066 [ms] (mean) Time per request: 113.507 [ms] (mean, across all concurrent requests) Transfer rate: 2.02 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 98 103 3.4 102 110 Processing: 102 477 296.0 520 928 Waiting: 102 477 295.9 520 928 Total: 205 580 295.6 621 1033 |
Results using wrk
As suggested by Andrey Sidorov. The results are MUCH better – 2821 requests per second:
|
1 2 3 4 5 6 7 8 9 10 |
Running 30s test @ http://54.200.x.xxx:8080/rest/users/jdoe 12 threads and 400 connections Thread Stats Avg Stdev Max +/- Stdev Latency 137.04ms 48.12ms 2.66s 98.89% Req/Sec 238.11 27.97 303.00 88.91% 84659 requests in 30.01s, 19.38MB read Socket errors: connect 0, read 0, write 0, timeout 53 Requests/sec: 2821.41 Transfer/sec: 661.27KB |
So it certainly looks like the culprit is ApacheBench! Unbelievable!
Answer:
It’s probably ab issue (see also this question). There is nothing wrong in your server code. I suggest to try to benchmark using wrk load testing tool. Your example on my t1.micro:
|
1 2 3 4 5 6 7 8 9 10 |
wrk git:master ❯ ./wrk -t12 -c400 -d30s http://some-amazon-hostname.com/rest/users/10 ✭ Running 30s test @ http://some-amazon-hostname.com/rest/users/10 12 threads and 400 connections Thread Stats Avg Stdev Max +/- Stdev Latency 333.42ms 322.01ms 3.20s 91.33% Req/Sec 135.02 59.20 283.00 65.32% 48965 requests in 30.00s, 11.95MB read Requests/sec: 1631.98 Transfer/sec: 407.99KB |