Benchmarking DigitalOcean, Vultr and a Six Year Old Laptop with GitSense

GitSense, if you are learning about it for the first time, brings advanced search capability and code management metrics to GitHub and Bitbucket.  And in this blog post, we'll share what we learned from testing our indexing engine, on low cost platforms.

TL;DR

  • Vultr's $10/month VPS performed better than DigitalOcean's $10/month VPS.
  • A six year old Dell laptop is a viable option for indexing Git repositories.
  • If you can install our indexers on machines with 32GB of RAM or more, with 3 SSDs or more in RAID 0, you really should.

Objective
We wanted to know, how low could we go. Ideally, we would love for everybody to install our indexing engine, on high end machines, but we know this wouldn't be practical for many. So to see how well GitSense would fair on low cost platforms, we decided to test it on $10/month virtual machines from DigitalOcean and Vultr and with VirtualBox on a Dell Latitude E6510, Core i5 520M laptop running Windows 7.

Virtual Machines
DigitalOcean Vultr Laptop VirtualBox VM
Centos 7
1GB Memory
1 Virtual Core
30 GB SSD
Centos 7
1GB Memory
1 Virtual Core
20 GB SSD
Centos 7
4GB Memory
2 Virtual Core
20 GB SSD
And to provide some contrast to the VM indexing times, we've included indexing times from our bare metal install test machine, which has 32GB of RAM, 8 Cores and 4 SSDs configured in RAID 0 mode.

Test Plan
Since we are dealing with low resource virtual machines, indexing was broken into three phases. The first phase would stop indexing at level 4, the second phase would stop indexing at level 6, and the last phase would stop indexing at level 8.

Below is a breakdown of the different GitSense indexing levels:

Level What you can do, once indexing has finished
1 Search for commits by commit attributes (author, committer, title, etc.) and browse the GitSense revisions tree.
2 Search for commits by file attributes (file size, source type, lines of code, etc.)
3 Browse the latest tree
4 Search for code on the latest tree
5 Build commits code churn reports
6 Search for diffs
7 Build daily, weekly and monthly lines of code reports for metrics enabled branches
8 Build daily, weekly and monthly code churn reports for metrics enabled brances

Repositories

Microsoft / vscode

Summary

  • Indexed 3753 commits, 21159 changes, 15824 blobs and 13301 diffs
  • Created 14 searchable branches
  • Created 2 daily, weekly and monthly reportable branches (master and release/0.10.11)
Phases DigitalOcean Vultr Laptop VM Bare Metal
1 - Commits/code searchable 00:07:14 00:04:25 00:05:28 00:01:49
2 - Diffs searchable 00:12:27 00:07:01 00:07:03 00:02:19
3 - Advanced reporting 00:05:16 00:04:27 00:05:26 00:01:55
Total 00:24:57 00:15:53 00:14:50 00:06:03



facebook / react-native

Summary

  • Indexed 6501 commits, 33200 changes, 24517 blobs and 29507 diffs
  • Created 9 searchable branches
  • Created 2 daily, weekly and monthly reportable branches (master and 0.22-stable)
Phases DigitalOcean Vultr Laptop VM Bare Metal
1 - Commits/code searchable 00:04:50 00:03:34 00:03:44 00:01:25
2 - Diffs searchable 00:19:11 00:16:16 00:13:03 00:01:25
3 - Advanced reporting 00:06:26 00:05:49 00:05:01 00:02:38
Total 00:30:27 00:25:39 00:21:48 00:05:28



atlassian / aui

Summary

  • Indexed 13473 commits, 39027 changes, 25543 blobs and 30184 diffs
  • Created 41 searchable branches
  • Created 2 daily, weekly and monthly reportable branches (master and 5.9.x)
Phases DigitalOcean Vultr Laptop VM Bare Metal
1 - Commits/code searchable 00:08:27 00:06:15 00:05:53 00:02:33
2 - Diffs searchable 00:25:49 00:20:24 00:16:42 00:05:06
3 - Advanced reporting 00:07:53 00:06:53 00:06:12 00:03:05
Total 00:42:09 00:33:32 00:28:47 00:10:44



meteor / meteor

Summary

  • Indexed 18197 commits, 52077 changes, 40352 blobs and 40005 diffs
  • Created 39 searchable branches
  • Created 2 daily, weekly and monthly reportable branches (devel and release-1.3)
Phases DigitalOcean Vultr Laptop VM Bare Metal
1 - Commits/code searchable 00:10:09 00:08:09 00:08:09 00:03:13
2 - Diffs searchable 00:38:43 00:37:17 00:27:27 00:08:24
3 - Advanced reporting 00:12:27 00:08:27 00:08:45 00:03:51
Total 01:01:19 00:53:53 00:44:21 00:15:28



rails / rails

Summary

  • Indexed 60620 commits, 135112 changes, 113353 blobs and 115102 diffs
  • Created 11 searchable branches
  • Created 2 daily, weekly and monthly reportable branches (master and 4-2-stable)
Phases DigitalOcean Vultr Laptop VM Bare Metal
1 - Commits/code searchable 00:17:52 00:12:42 00:13:02 00:06:06
2 - Diffs searchable 02:14:45 01:56:06 01:48:31 00:22:40
3 - Advanced reporting 00:49:00 00:36:00 00:17:44 00:05:51
Total 03:21:37 02:44:48 02:19:17 00:34:37


Conclusion

We were definitely not expecting Vultr to perform as well as it did, compared to DigitalOcean. For the rails/rails repository, the difference was over 30 minutes and without knowing the hardware setup for both vendors, we can only speculate the reason for the difference was because DigitalOcean has more saturated servers. This would seem logical, since DigitalOcean is the more popular VPS provider, ergo more provisioned/active instances.

If greater saturation isn't the issue and if it boils down to better hardware, we can see an interesting battle brewing between DigitalOcean and Vultr, which can only be good news for consumers. More testing is obviously needed, but given what we have seen so far, Vultr is the clear cut winner when it comes to $10/month instances.

If you have noticed the same or if you have experienced the complete opposite, send an email to support@gitsense.com and let us know, as we would love to gather more data points on DigitalOcean and Vultr.

The laptop performing better than the cloud virtual machines, didn't really come as a surprise. Throwing more RAM and Cores at a problem, will usually do the trick, provided the CPU is up to the task. Given how close the numbers were, upgrading to the $20/month VPS plans would probably get you the same results, if not better.

Well we hope this was informative and if you are looking for a simple way to enhance your GitHub and Bitbucket browsing experience, make sure to send an email to privatebeta@gitsense.com, to register for the free GitSense beta trial, that will be starting soon.

Blog Posts

© 2016 SDE Solutions, Inc. All rights reserved.