PVE使用INTEL82576网卡开启IOMMO和 SRIOV

PVE使用INTEL82576网卡开启IOMMO和SRIOV1.建议先安装基本的软件aptupdateapt-getinstallnet-toolsvimethtool-y2.打开IOMMU在/etc/default/grub文件里，GRUB_CMDLINE_LINUX_DEFAULT行添加以下内容如果intelCPU，则intel_iommu=onpcie_acs_override=downstream。sed-i'/GRUB_CMDLINE_LINUX_DEFAULT/cGRUB_CMDLINE_LINUX_DEFAULT="modprobe.blacklist=nouveauintel_iommu=on"'/etc/default/grub如果AMDCPU，则amd_iommu=onpcie_acs_override=downstream。sed-i'/GRUB_CMDLINE_LINUX_DEFAULT/cGRUB_CMDLINE_LINUX_DEFAULT="modprobe.blacklist=nouveauamd_iommu=on"'/etc/default/grub#更新grubupdate-grub#在/etc/modules文件里添加IOMMUvfio>>/etc/modulesvfio_iommu_type1>>/etc/modulesvfio_pci>>/etc/modulesvfio_virqfd>>/etc/modules#更新并重启update-initramfs-u-kallreboot#检查IOMMO功能是否开启dmesg|grep-eDMAR-eIOMMU-eAMD-Vifind/sys/kernel/iommu_groups/-typellspci-vv|lspci-vv|

grepEth如果没有开启，则在/etc/modprobe.d/的任意一个.conf下增加不安全中断：optionsvfio_iommu_type1allow_unsafe_interrupts=13.启用SRIOVlspci-nn|grepEth最前面的数字是设备的BDF号lspci-sBDF号-vvv|grepCapabilities查看是否支持SR-IOV#查看网口最大支持的VF网口数量cat/sys/bus/pci/devices/0000:BDF号/sriov_totalvfs返回结果就是最大支持的VF网口数量echoVF网口数量>/sys/class/net/enp1s0f0/device/sriov_numvfs#添加VF网口成功显示lspci|grepEth首先确定你的intel网卡支持SR-IOV，如果支持，那么继续root@pve:~#echo"optionsigbmax_vfs=6,6,6,6,6,6">/etc/modprobe.d/igb.confroot@pve:~#depmod-aroot@pve:~#update-initramfs-uroot@pve:~#reboot解释一下，max_vfs=6,6,6,6,6,6：这条语句说明最大的虚拟网卡是6个，也就是虚拟6个echo"optionsigbmax_vfs=6,6,6,6,6,6">/etc/modprobe.d/igb.confdepmod-aupdate-initramfs-ureboot解释一下，max_vfs=6,6,6,6,6,6：这条语句说明最大的虚拟网卡是6个，也就是1个物理网卡虚拟6个3.4检查虚拟网卡root@pve:~#dmesg|grep82576[2.527369]igbvf0000:07:10.1:Intel(R)82576VirtualFunction[2.532109]igbvf0000:07:10.3:Intel(R)82576VirtualFunction[2.535583]igbvf0000:07:10.5:Intel(R)82576VirtualFunction[2.539074]igbvf0000:07:10.7:Intel(R)82576VirtualFunction[2.542565]igbvf0000:07:11.1:Intel(R)82576VirtualFunction[2.546055]igbvf0000:07:11.3:Intel(R)82576VirtualFunction[2.768637]igbvf0000:07:10.0:Intel(R)82576VirtualFunction[2.771872]igbvf0000:07:10.2:Intel(R)82576VirtualFunction[2.775080]igbvf0000:07:10.4:Intel(R)82576VirtualFunction[2.778273]igbvf0000:07:10.6:Intel(R)82576VirtualFunction[2.781481]igbvf0000:07:11.0:Intel(R)82576VirtualFunction[2.784682]igbvf0000:07:11.2:Intel(R)82576VirtualFunction[2.920650]igbvf0000:05:10.0:Intel(R)82576VirtualFunction[3.128489]igbvf0000:05:10.2:Intel(R)82576VirtualFunction[3.412422]igbvf0000:05:10.4:Intel(R)82576VirtualFunction[3.540328]igbvf0000:05:10.6:Intel(R)82576VirtualFunction[3.580426]igbvf0000:05:11.0:Intel(R)82576VirtualFunction[3.608433]igbvf0000:05:11.2:Intel(R)82576VirtualFunction[3.940652]igbvf0000:05:10.1:Intel(R)82576VirtualFunction[3.960418]igbvf0000:05:10.3:Intel(R)82576VirtualFunction[3.976422]igbvf0000:05:10.5:Intel(R)82576VirtualFunction[3.992427]igbvf0000:05:10.7:Intel(R)82576VirtualFunction[4.008435]igbvf0000:05:11.1:Intel(R)82576VirtualFunction[4.024426]igbvf0000:05:11.3:Intel(R)82576VirtualFunction[4.359456]igbvf0000:04:10.0:Intel(R)82576VirtualFunction[4.380383]igbvf0000:04:10.2:Intel(R)82576VirtualFunction[4.400395]igbvf0000:04:10.4:Intel(R)82576VirtualFunction[4.424404]igbvf0000:04:10.6:Intel(R)82576VirtualFunction[4.440398]igbvf0000:04:11.0:Intel(R)82576VirtualFunction[4.464396]igbvf0000:04:11.2:Intel(R)82576VirtualFunction[4.795365]igbvf0000:04:10.1:Intel(R)82576VirtualFunction[4.820409]igbvf0000:04:10.3:Intel(R)82576VirtualFunction[4.836453]igbvf0000:04:10.5:Intel(R)82576VirtualFunction[4.856446]igbvf0000:04:10.7:Intel(R)82576VirtualFunction[4.872448]igbvf0000:04:11.1:Intel(R)82576VirtualFunction[4.888444]igbvf0000:04:11.3:Intel(R)82576VirtualFunction出现VF代表成功了，那么在你的ProxmoxVE控制面板上，网络一栏就会出现很多网卡注意把原来网卡标记一下，否则会乱的二、测试SR-IOV网卡生成可以先进行测试譬如在enp2s0f0网卡上开4个VFecho4>/sys/class/net/enp2s0f0/device/sriov_numvfs成功不会有任何反馈，回到PVE节点网络查看网络列表是否有VF网卡出现，也可以使用lspci查看pci设备中是否有vf网卡三、注册系统服务首先创建启动脚本cd/etc/systemd/systemvisriov.service示例是在i350-T4(系统名称enp2s0)的0号和1号口上分别开5个vf网卡、固化MAC地址防止重启MAC随机变动并且设置所有PF和VF为UP状态[Unit]Description=ScripttoenableSR-IOVonboot[Service]Type=simple#startSR-IOVExecStartPre=/usr/bin/bash-c'/usr/bin/echo5>/sys/class/net/enp2s0f0/device/sriov_numvfs'ExecStartPre=/usr/bin/bash-c'/usr/bin/echo5>/sys/class/net/enp2s0f1/device/sriov_numvfs'#setVFMACExecStartPre=/usr/bin/bash-c'/usr/bin/iplinksetdevenp2s0f0vf0mac00:54:00:00:00:00'ExecStartPre=/usr/bin/bash-c'/usr/bin/iplinksetdevenp2s0f0vf1mac00:54:00:00:00:01'ExecStartPre=/usr/bin/bash-c'/usr/bin/iplinksetdevenp2s0f0vf2mac00:54:00:00:00:02'ExecStartPre=/usr/bin/bash-c'/usr/bin/iplinksetdevenp2s0f0vf3mac00:54:00:00:00:03'ExecStartPre=/usr/bin/bash-c'/usr/bin/iplinksetdevenp2s0f0vf4mac00:54:00:00:00:04'ExecStartPre=/usr/bin/bash-c'/usr/bin/iplinksetdevenp2s0f1vf0mac00:54:00:00:01:00'ExecStartPre=/usr/bin/bash-c'/usr/bin/iplinksetdevenp2s0f1vf1mac00:54:00:00:01:01'ExecStartPre=/usr/bin/bash-c'/usr/bin/iplinksetdevenp2s0f1vf2mac00:54:00:00:01:02'ExecStartPre=/usr/bin/bash-c'/usr/bin/iplinksetdevenp2s0f1vf3mac00:54:00:00:01:03'ExecStartPre=/usr/bin/bash-c'/usr/bin/iplinksetdevenp2s0f1vf4mac00:54:00:00:01:04'#setPFupExecStartPre=/usr/bin/bash-c'/usr/bin/iplinksetenp2s0f0up'ExecStartPre=/usr/bin/bash-c'/usr/bin/iplinksetenp2s0f1up'#setVFupExecStartPre=/usr/bin/bash-c'/usr/bin/iplinksetenp2s0f0v0up'ExecStartPre=/usr/bin/bash-c'/usr/bin/iplinksetenp2s0f0v1up'ExecStartPre=/usr/bin/bash-c'/usr/bin/iplinksetenp2s0f0v2up