I build a Freenas box using a supermicro X8DTN-F motherboard with dual xeon L5630 24GB ram.
I also build a vmware box using all-in-one board SUPERMICRO X11SBA-LN4F with 32GB ECC ram.
X8DTN-F has 2 onboard Gib RJ-45 ports, they are not fast enough for iSCSI storage share.
SO I bought LOT OF 5 MELLANOX CONNECTX-2 10GbE ETHERNET NETWORK CARD MNPA19-XTR. They cost me $65 plush $12 shipping and $25 for customs.
I also bought 3 Genuine Cisco SFP-H10GB-CU3M 3 Meter 10GbE Twinax Cable 37-0961 COPQAA6J A. They cost me about $15 each.
I installed one card on Freenas box and another one on vmware box.
This is some tests from iperf.
/usr/lib/vmware/vsan/bin/iperf -c 10.8.99.2
Client connecting to 10.8.99.2, TCP port 5001
TCP window size: 32.5 KByte (default)
[ 3] local 10.8.99.4 port 57624 connected with 10.8.99.2 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 8.05 GBytes 6.91 Gbits/sec
And using dd in a vm whos storage is based on iSCSI from Freenas over SFP+
dd if=/dev/zero of=testfile bs=1G count=10
10737418240 bytes (11 GB, 10 GiB) copied, 29.0997 s, 369 MB/s
This is a very good result.
One link of 10 Gib link cost me less than $56, This results are very acceptable.