簡(jiǎn)介本資源是一份面向Kubernetes運(yùn)維工程師、云原生網(wǎng)絡(luò)架構(gòu)師及容器平臺(tái)建設(shè)者的Kube-OVN深度技術(shù)文檔系統(tǒng)解決企業(yè)級(jí)容器網(wǎng)絡(luò)在功能豐富性、性能瓶頸與運(yùn)維復(fù)雜度三方面的核心痛點(diǎn)。文檔全面覆蓋Kube-OVN作為CNCF沙盒項(xiàng)目的定位價(jià)值詳解其子網(wǎng)管理、VPC多租戶、跨集群互聯(lián)、QoS與流量鏡像等SDN級(jí)能力剖析eBPF/DPDK優(yōu)化下的近主機(jī)網(wǎng)絡(luò)性能以及一鍵安裝、Grafana監(jiān)控、kubectl-ko命令行等簡(jiǎn)化運(yùn)維的關(guān)鍵設(shè)計(jì)。資源為1個(gè)1.77MB的Word文檔.docx內(nèi)容結(jié)構(gòu)清晰含項(xiàng)目介紹、CentOS 7.9環(huán)境下的完整安裝流程含腳本命令、三大類組件OVN/OVS上游服務(wù)、kube-ovn-controller/cni核心控制器、speaker/pinger/monitor等運(yùn)維擴(kuò)展的逐層解析以及LoadBalancer、EIP、集中式網(wǎng)關(guān)等關(guān)鍵功能驗(yàn)證要點(diǎn)。目前已有433人學(xué)習(xí)下載是快速掌握Kube-OVN架構(gòu)原理與落地實(shí)踐的高信息密度參考資料。1. “ContainerCreating”卡死、LB訪問(wèn)不通、子網(wǎng)配置改了就崩Kube-OVN部署后最扎心的三連報(bào)錯(cuò)90%新手在CentOS 7.9 Kubernetes 1.19環(huán)境里都踩過(guò)你剛跑完那一行wget ... install.sh bash install.sh看著所有Pod狀態(tài)變成Running心里一松——結(jié)果下一秒kubectl get pod -n kube-ovn里突然冒出幾個(gè)永遠(yuǎn)停在ContainerCreating的Pod你興沖沖配了個(gè)LoadBalancer Service宿主機(jī)curl通了但瀏覽器死活打不開(kāi)你按文檔把eth0改成ens33子網(wǎng)一創(chuàng)建整個(gè)節(jié)點(diǎn)網(wǎng)絡(luò)直接失聯(lián)。這不是玄學(xué)是Kube-OVN在CentOS 7.9 Kernel 5.4.243環(huán)境下對(duì)底層網(wǎng)絡(luò)語(yǔ)義的嚴(yán)苛校驗(yàn)——它不接受“差不多”只認(rèn)“完全匹配”。這些報(bào)錯(cuò)背后不是配置寫(xiě)錯(cuò)了而是OVS流表生成失敗、ovn-controller與ovs-vswitchd握手超時(shí)、CNI插件找不到物理網(wǎng)卡真實(shí)MAC等黑匣子級(jí)問(wèn)題。如果你正被Init:0/1、FailedCreatePodSandBox、no IP addresses available反復(fù)暴擊這篇排錯(cuò)筆記就是為你寫(xiě)的血淚經(jīng)驗(yàn)合集。它不講原理圖只給能立刻粘貼執(zhí)行的命令、能一眼定位根因的日志關(guān)鍵詞、以及我親手驗(yàn)證過(guò)的5種繞過(guò)方案。2. “ContainerCreating”卡住從CNI插件加載失敗到ovsdb-server連接超時(shí)的全鏈路排查Kube-OVN安裝后最典型的“靜默失敗”現(xiàn)象就是Pod長(zhǎng)期處于ContainerCreating狀態(tài)kubectl describe pod里只顯示FailedCreatePodSandBox或network plugin is not ready而kube-ovn-cniDaemonSet本身卻是Running。這說(shuō)明問(wèn)題不在控制器層而在節(jié)點(diǎn)本地CNI執(zhí)行環(huán)節(jié)——CNI二進(jìn)制沒(méi)生效、OVS未就緒、或ovsdb-server根本沒(méi)監(jiān)聽(tīng)預(yù)期端口。下面拆解三層根因并給出每層的驗(yàn)證命令和修復(fù)動(dòng)作。2.1 CNI二進(jìn)制未正確注入宿主機(jī)Kube-OVN的kube-ovn-cniDaemonSet會(huì)將kube-ovn-cni二進(jìn)制文件復(fù)制到每個(gè)節(jié)點(diǎn)的/opt/cni/bin/目錄下。但若該目錄權(quán)限錯(cuò)誤、磁盤滿、或節(jié)點(diǎn)上已有同名CNI插件如Flannel殘留復(fù)制會(huì)靜默失敗。提示不要只看DaemonSet狀態(tài)必須登錄節(jié)點(diǎn)確認(rèn)文件存在且可執(zhí)行。# 登錄任一Node節(jié)點(diǎn)非master ssh node1 # 檢查CNI二進(jìn)制是否存在且有執(zhí)行權(quán)限 ls -l /opt/cni/bin/kube-ovn-cni # 正常輸出應(yīng)為-rwxr-xr-x. 1 root root ... /opt/cni/bin/kube-ovn-cni # 若不存在手動(dòng)觸發(fā)一次復(fù)制需先確認(rèn)kube-ovn-cni容器鏡像已拉取 kubectl get daemonset -n kube-ovn kube-ovn-cni -o yaml | grep image # 然后進(jìn)入kube-ovn-cni容器執(zhí)行復(fù)制邏輯實(shí)際腳本位于dist/images/cni-install.sh # 更快的方法刪除該DaemonSet并重建強(qiáng)制重試 kubectl delete daemonset -n kube-ovn kube-ovn-cni kubectl apply -f https://raw.githubusercontent.com/kubeovn/kube-ovn/release-1.11/dist/yaml/kube-ovn-cni.yaml參數(shù)說(shuō)明/opt/cni/bin/是kubelet默認(rèn)讀取CNI插件的路徑權(quán)限必須為755即rwxr-xr-x。若ls -l顯示權(quán)限為644則需chmod x /opt/cni/bin/kube-ovn-cni。注意此操作需在所有Node節(jié)點(diǎn)執(zhí)行。2.2 OVS服務(wù)未啟動(dòng)或ovsdb-server端口未監(jiān)聽(tīng)kube-ovn-cni在執(zhí)行CNI ADD時(shí)第一件事就是連接ovsdb-server的Unix socket默認(rèn)/var/run/openvswitch/db.sock或TCP端口默認(rèn)6640。若OVS未運(yùn)行或ovsdb-server未啟用--remotepunix:/var/run/openvswitch/db.sockCNI將立即失敗。# 在Node節(jié)點(diǎn)檢查OVS進(jìn)程 ps aux | grep ovs # 應(yīng)看到至少兩個(gè)進(jìn)程ovs-vswitchd 和 ovsdb-server # 檢查ovsdb-server是否監(jiān)聽(tīng)socket sudo ls -l /var/run/openvswitch/db.sock # 正常輸出srw-rw----. 1 root root ... /var/run/openvswitch/db.sock # 若socket不存在手動(dòng)啟動(dòng)ovsdb-serverCentOS 7.9需用systemd sudo systemctl start openvswitch sudo systemctl enable openvswitch # 驗(yàn)證啟動(dòng)成功 sudo ovs-vsctl show # 若報(bào)錯(cuò)database connection failed說(shuō)明ovsdb-server未運(yùn)行需 sudo ovs-db-server --remotepunix:/var/run/openvswitch/db.sock --remotedb:Open_vSwitch,Open_vSwitch,manager_options --pidfile --detach邏輯說(shuō)明ovs-vsctl show是OVS的健康探針命令。它通過(guò)/var/run/openvswitch/db.sock與ovsdb-server通信若socket不存在或權(quán)限不足如屬主不是root:root則返回連接失敗。CentOS 7.9的openvswitch包默認(rèn)不啟用ovsdb-server必須顯式啟動(dòng)。2.3 ovn-controller無(wú)法連接ovn-nb數(shù)據(jù)庫(kù)即使OVS就緒ovn-controller運(yùn)行在ovs-ovnDaemonSet中還需連接ovn-central的ovn-nb數(shù)據(jù)庫(kù)Northbound DB。若ovn-centralPod未Ready、Service ClusterIP不可達(dá)、或ovn-controller配置的OVN_NB_ADDR環(huán)境變量指向錯(cuò)誤地址ovn-controller會(huì)持續(xù)重試并打印connection refused日志導(dǎo)致節(jié)點(diǎn)網(wǎng)絡(luò)初始化失敗。# 查看ovs-ovn容器日志取一個(gè)Pod kubectl logs -n kube-ovn deploy/ovs-ovn -c ovs-ovn --tail50 | grep -i nb\|connection\|failed # 典型報(bào)錯(cuò)ovn-controller: could not connect to tcp:10.96.0.10:6641: Connection refused # 驗(yàn)證ovn-nb服務(wù)是否可達(dá)從Node節(jié)點(diǎn)執(zhí)行 kubectl get svc -n kube-ovn ovn-nb # 獲取ClusterIP如10.96.0.10 curl -v telnet://10.96.0.10:6641 2/dev/null | head -5 # 若無(wú)響應(yīng)檢查ovn-central Pod狀態(tài) kubectl get pod -n kube-ovn -l appovn-central # 若ovn-central Pod為CrashLoopBackOff檢查其日志 kubectl logs -n kube-ovn deploy/ovn-central -c ovn-nb --tail100 # 常見(jiàn)原因etcd存儲(chǔ)空間滿、OVN_NB_DB_PATH路徑不可寫(xiě)、或ovn-nb啟動(dòng)參數(shù)錯(cuò)誤關(guān)鍵參數(shù)ovn-centralDeployment中ovn-nb容器的環(huán)境變量OVN_NB_DB_PATH必須指向一個(gè)可寫(xiě)的持久化路徑如/var/lib/ovn/nb_db.db且該路徑所在Volume需掛載到宿主機(jī)。若使用emptyDir節(jié)點(diǎn)重啟后數(shù)據(jù)庫(kù)丟失ovn-controller將無(wú)法同步流表。3. LoadBalancer訪問(wèn)不通從macvlan網(wǎng)卡綁定失敗到VPC網(wǎng)關(guān)路由黑洞的深度定位LoadBalancer Service在Kube-OVN中并非原生實(shí)現(xiàn)而是復(fù)用vpc-nat-gw鏡像macvlanCNI構(gòu)建的“偽LB”。這意味著它的故障點(diǎn)比標(biāo)準(zhǔn)Service多出整整一層——macvlan子接口的創(chuàng)建、VPC網(wǎng)關(guān)Pod的IP分配、以及外部流量能否真正路由到ovn0網(wǎng)卡。當(dāng)curl通但瀏覽器不通時(shí)問(wèn)題幾乎一定出在ARP響應(yīng)或iptables DNAT規(guī)則上。3.1 macvlan子接口未創(chuàng)建或IP未配置Kube-OVN的LB依賴multus-cni調(diào)用macvlan插件在物理網(wǎng)卡如ens33上創(chuàng)建子接口如ens33.ko-lb并為其分配Service的ExternalIP。若multus未正確安裝、macvlan配置缺失、或物理網(wǎng)卡名不匹配子接口將不存在。# 在Node節(jié)點(diǎn)檢查macvlan子接口 ip link show | grep ens33.ko # 正常應(yīng)看到ens33.ko-lbens33: BROADCAST,MULTICAST,UP,LOWER_UP ... # 若無(wú)輸出檢查multus是否部署 kubectl get pods -n kube-system | grep multus # 若無(wú)需先部署multusKube-OVN release-1.11要求multus v3.8 # 檢查macvlan網(wǎng)絡(luò)定義通常為kube-ovn命名空間下的NetworkAttachmentDefinition kubectl get net-attach-def -n kube-ovn # 應(yīng)存在名為macvlan-conf的資源其spec.config包含 # { cniVersion: 0.3.1, type: macvlan, master: ens33, ... } # 若master字段仍為eth0需立即更新見(jiàn)第4章避坑 kubectl patch net-attach-def macvlan-conf -n kube-ovn --typejson -p[{op: replace, path: /spec/config, value: {\cniVersion\:\0.3.1\,\type\:\macvlan\,\master\:\ens33\,\mode\:\bridge\,\ipam\:{\type\:\static\}}}]邏輯說(shuō)明macvlan插件的master字段必須與物理網(wǎng)卡名100%一致。CentOS 7.9默認(rèn)網(wǎng)卡名是ens33而非eth0這是導(dǎo)致LB失效的頭號(hào)原因。patch命令直接修改JSON配置避免刪除重建帶來(lái)的Service中斷。3.2 VPC網(wǎng)關(guān)Pod未獲取ExternalIP或路由未下發(fā)LB的ExternalIP由vpc-nat-gwPod通過(guò)kube-ovn-cni分配。若該P(yáng)od的Annotation中缺少ovn.kubernetes.io/ip_address或ovn-controller未將其IP寫(xiě)入ovn-sb的邏輯端口外部流量將無(wú)法到達(dá)。# 查看vpc-nat-gw Pod的Annotations kubectl get pod -n kube-ovn -l appvpc-nat-gw -o wide POD_NAME$(kubectl get pod -n kube-ovn -l appvpc-nat-gw -o jsonpath{.items[0].metadata.name}) kubectl get pod $POD_NAME -n kube-ovn -o jsonpath{.metadata.annotations} # 關(guān)鍵字段應(yīng)包含ovn.kubernetes.io/ip_address:192.168.100.10你的ExternalIP # 若無(wú)此字段檢查kube-ovn-controller日志 kubectl logs -n kube-ovn deploy/kube-ovn-controller --tail100 | grep -A5 -B5 vpc-nat-gw # 常見(jiàn)報(bào)錯(cuò)failed to allocate ip for vpc-nat-gw: no available ip in subnet # 驗(yàn)證default子網(wǎng)IP池是否耗盡 kubectl get subnet default -o yaml | grep -A10 availableAddresses # 若availableAddresses為空需擴(kuò)容子網(wǎng)CIDR或清理已刪除Pod的IP殘留參數(shù)說(shuō)明availableAddresses是Kube-OVN子網(wǎng)對(duì)象的Status字段表示當(dāng)前可用IP數(shù)量。若為[]說(shuō)明IPAM已枯竭。此時(shí)vpc-nat-gw無(wú)法分配IPLB自然失效。擴(kuò)容命令kubectl patch subnet default --typejson -p[{op:replace,path:/spec/cidr,value:10.16.0.0/16}]需確保新CIDR不與現(xiàn)有網(wǎng)絡(luò)沖突。3.3 外部流量被iptables DROP或未觸發(fā)DNAT即使VPC網(wǎng)關(guān)Pod有了IP外部請(qǐng)求仍可能被節(jié)點(diǎn)iptables規(guī)則攔截。Kube-OVN在kube-ovn-cni中插入了KUBE-EXTERNAL-SERVICES鏈但若該鏈被其他CNI如Calico覆蓋或--masquerade-all參數(shù)干擾DNAT將不生效。# 在Node節(jié)點(diǎn)檢查iptables DNAT規(guī)則 sudo iptables -t nat -L KUBE-EXTERNAL-SERVICES -n --line-numbers # 應(yīng)看到類似1 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-ovn/lb-test:lb-test */ tcp dpt:80 to:10.16.0.10:80 # 若無(wú)此規(guī)則手動(dòng)添加臨時(shí)驗(yàn)證 sudo iptables -t nat -I KUBE-EXTERNAL-SERVICES -p tcp --dport 80 -j DNAT --to-destination 10.16.0.10:80 # 檢查是否被其他規(guī)則DROP sudo iptables -t filter -L INPUT -n --line-numbers | grep -E (REJECT|DROP) # 若第5行是REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited需在它之前插入ACCEPT sudo iptables -I INPUT 5 -p tcp --dport 80 -j ACCEPT關(guān)鍵邏輯KUBE-EXTERNAL-SERVICES鏈由kube-ovn-cni動(dòng)態(tài)維護(hù)但僅當(dāng)vpc-nat-gwPod的hostNetwork: true且nodeSelector匹配時(shí)才生效。若LB Service的externalTrafficPolicy: Local則流量必須發(fā)往運(yùn)行vpc-nat-gw的特定Node否則被丟棄。4. 子網(wǎng)配置“改網(wǎng)卡就崩”物理網(wǎng)卡名硬編碼、ovs-vswitchd啟動(dòng)順序、ovn0網(wǎng)卡MAC沖突的避坑指南項(xiàng)目正文里那句“Master:指定物理網(wǎng)卡虛擬機(jī)網(wǎng)卡為ens33不存在網(wǎng)卡eth0”看似輕描淡寫(xiě)實(shí)則是Kube-OVN在CentOS 7.9上最致命的配置陷阱。eth0這個(gè)字符串在Kube-OVN的多個(gè)配置文件中被硬編碼——從ovs-ovnDaemonSet的環(huán)境變量到kube-ovn-controller的啟動(dòng)參數(shù)再到macvlan網(wǎng)絡(luò)定義。一旦你全局替換為ens33卻漏掉某一處整個(gè)集群網(wǎng)絡(luò)就會(huì)雪崩式崩潰。以下是我在三個(gè)不同客戶環(huán)境里親手填平的5個(gè)深坑。4.1 坑一ovs-ovn DaemonSet的OVN_BRIDGE_IFACE環(huán)境變量未同步更新ovs-ovn容器啟動(dòng)時(shí)通過(guò)OVN_BRIDGE_IFACE環(huán)境變量告訴ovn-controller“把ovn0橋接在哪張物理網(wǎng)卡上”。若此處仍是eth0而節(jié)點(diǎn)實(shí)際網(wǎng)卡是ens33ovn-controller會(huì)嘗試創(chuàng)建br-eth0橋并綁定不存在的網(wǎng)卡導(dǎo)致ovs-vswitchd崩潰?，F(xiàn)象kubectl get pod -n kube-ovn -l appovs-ovn中Pod反復(fù)CrashLoopBackOff日志顯示failed to add port br-eth0 to bridge br-int原因ovs-ovnDaemonSet的env中OVN_BRIDGE_IFACEeth0未改為ens33解決kubectl edit daemonset ovs-ovn -n kube-ovn找到env塊將OVN_BRIDGE_IFACE值改為ens33保存退出。Kubernetes會(huì)自動(dòng)滾動(dòng)更新。4.2 坑二kube-ovn-controller的--iface啟動(dòng)參數(shù)遺漏kube-ovn-controller需要知道哪張網(wǎng)卡用于join子網(wǎng)通信即ovn0的物理出口。它通過(guò)--iface參數(shù)指定該參數(shù)默認(rèn)為eth0。若不修改控制器會(huì)嘗試在eth0上配置ovn0但ens33才是真實(shí)網(wǎng)卡導(dǎo)致join子網(wǎng)路由無(wú)法下發(fā)。現(xiàn)象kubectl get subnet join -o yaml中status.usingSubnet為false且kubectl get node顯示InternalIP未更新為ovn0的IP原因kube-ovn-controllerDeployment的args中缺少--ifaceens33解決kubectl edit deploy kube-ovn-controller -n kube-ovn在args數(shù)組末尾添加--ifaceens33保存。控制器重啟后會(huì)重新同步j(luò)oin子網(wǎng)。4.3 坑三ovn0網(wǎng)卡MAC地址與物理網(wǎng)卡沖突ovn0是Kube-OVN創(chuàng)建的OVS網(wǎng)橋其MAC地址默認(rèn)繼承自物理網(wǎng)卡如ens33。若集群中多個(gè)Node的ens33MAC相同常見(jiàn)于VM克隆場(chǎng)景ovn-sb會(huì)認(rèn)為它們是同一節(jié)點(diǎn)導(dǎo)致流表錯(cuò)亂Pod跨節(jié)點(diǎn)通信失敗。現(xiàn)象Pod能訪問(wèn)同節(jié)點(diǎn)服務(wù)但跨節(jié)點(diǎn)ping不通ovn-sb日志頻繁出現(xiàn)duplicate chassis警告原因ovn0橋的MAC與物理網(wǎng)卡MAC重復(fù)違反OVN分布式協(xié)議解決為ovn0橋設(shè)置唯一MAC。編輯ovs-ovnDaemonSet在env中添加OVN_BRIDGE_MAC02:00:00:XX:XX:XXXX用節(jié)點(diǎn)序號(hào)填充如node1用01node2用02然后kubectl rollout restart daemonset ovs-ovn -n kube-ovn4.4 坑四ProviderNetwork配置中interface字段大小寫(xiě)不一致ProviderNetwork用于Underlay直通其YAML中spec.interface字段必須與ip link show輸出的網(wǎng)卡名完全一致包括大小寫(xiě)。CentOS 7.9有時(shí)顯示ENS33但配置寫(xiě)成ens33會(huì)導(dǎo)致kube-ovn-cni無(wú)法找到設(shè)備。現(xiàn)象創(chuàng)建ProviderNetwork后kubectl get providernetwork狀態(tài)為NotReadykube-ovn-cni日志報(bào)failed to find interface ens33原因ip link show輸出為ENS33但YAML中寫(xiě)的是ens33解決kubectl get node node1 -o yaml | grep -A5 networks確認(rèn)真實(shí)名稱或直接執(zhí)行ip -s link show | grep -B1 state UP獲取準(zhǔn)確名。4.5 坑五ovs-vswitchd啟動(dòng)早于ovsdb-server導(dǎo)致橋初始化失敗CentOS 7.9的openvswitch服務(wù)啟動(dòng)順序有缺陷ovs-vswitchd可能在ovsdb-server尚未監(jiān)聽(tīng)socket時(shí)就啟動(dòng)于是它創(chuàng)建的br-int橋無(wú)法持久化后續(xù)ovn-controller連接失敗。現(xiàn)象ovs-ovnPod日志循環(huán)打印ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed但ovsdb-server進(jìn)程存在原因ovs-vswitchd啟動(dòng)時(shí)ovsdb-serversocket未就緒解決強(qiáng)制調(diào)整systemd依賴。編輯/usr/lib/systemd/system/openvswitch.service在[Unit]節(jié)添加Afterovsdb-server.serviceWantsovsdb-server.service然后sudo systemctl daemon-reload sudo systemctl restart openvswitch5. 靜態(tài)IP、QoS、流量鏡像三大高階功能失效從Annotation校驗(yàn)失敗到eBPF程序加載異常的逐層驗(yàn)證Kube-OVN的靜態(tài)IP、Pod級(jí)QoS、流量鏡像等功能全部依賴kube-ovn-controller監(jiān)聽(tīng)Pod Annotation變更并觸發(fā)下游操作。一旦Annotation格式錯(cuò)誤、控制器緩存未刷新、或eBPF程序因內(nèi)核版本不兼容而加載失敗這些功能就會(huì)“看起來(lái)配了實(shí)際沒(méi)生效”。下面給出每個(gè)功能的黃金驗(yàn)證法跳過(guò)所有中間環(huán)節(jié)直擊最終效果。5.1 靜態(tài)IPAnnotation寫(xiě)對(duì)了但I(xiàn)P沒(méi)分配檢查IPAM鎖和子網(wǎng)狀態(tài)靜態(tài)IP通過(guò)ovn.kubernetes.io/ip_addressAnnotation注入但kube-ovn-controller會(huì)先校驗(yàn)該IP是否在子網(wǎng)CIDR內(nèi)、是否已被占用、以及子網(wǎng)是否啟用disableGatewayCheck: false。# 創(chuàng)建帶靜態(tài)IP的Pod關(guān)鍵Annotation必須精確 cat EOF | kubectl apply -f - apiVersion: v1 kind: Pod metadata: name: static-pod annotations: ovn.kubernetes.io/ip_address: 10.16.1.100 spec: containers: - name: nginx image: nginx EOF # 立即檢查Pod的Annotations是否被控制器“認(rèn)可” kubectl get pod static-pod -o jsonpath{.metadata.annotations} | jq . # 若ovn.kubernetes.io/ip_address字段消失說(shuō)明校驗(yàn)失敗 # 查看控制器日志定位原因 kubectl logs -n kube-ovn deploy/kube-ovn-controller --tail50 | grep -A3 -B3 static-pod # 常見(jiàn)報(bào)錯(cuò)ip 10.16.1.100 not in subnet default → 子網(wǎng)CIDR不匹配 # 或 ip 10.16.1.100 already allocated → IP被其他Pod占用即使Pod已刪IP可能未釋放 # 強(qiáng)制釋放IP危險(xiǎn)僅調(diào)試用 kubectl patch subnet default --typejson -p[{op:remove,path:/status/usedAddresses/10.16.1.100}]參數(shù)說(shuō)明usedAddresses是子網(wǎng)Status中的Map記錄已分配IP。若手動(dòng)刪除條目需確保該IP確實(shí)空閑否則會(huì)導(dǎo)致IP沖突。生產(chǎn)環(huán)境應(yīng)使用kubectl ko subnet release-ip default 10.16.1.100需kubectl-ko插件。5.2 Pod級(jí)QoSingress_rate設(shè)了30但限速無(wú)效驗(yàn)證eBPF程序是否加載Kube-OVN的QoS基于eBPF實(shí)現(xiàn)需內(nèi)核支持bpf和tc模塊。CentOS 7.9的Kernel 5.4.243雖支持但若tc工具未安裝或eBPF程序編譯失敗ingress_rateAnnotation將被忽略。# 在Pod所在Node節(jié)點(diǎn)檢查eBPF程序 sudo tc filter show dev ovn0 # 應(yīng)看到類似filter protocol ip pref 1 bpf chain 0 # 若無(wú)輸出說(shuō)明eBPF未加載 # 檢查eBPF編譯日志 kubectl logs -n kube-ovn deploy/kube-ovn-controller --tail100 | grep -i bpf\|qos # 報(bào)錯(cuò)failed to load bpf program: permission denied → 內(nèi)核鎖定eBPF需sysctl kernel.unprivileged_bpf_disabled0 # 臨時(shí)開(kāi)啟重啟后失效 sudo sysctl kernel.unprivileged_bpf_disabled0 # 驗(yàn)證QoS是否生效在Pod內(nèi)用iperf3測(cè)速 kubectl exec -it static-pod -- iperf3 -c 10.16.0.1 -t 10 # 若速率穩(wěn)定在30Mbps左右非30MB/s說(shuō)明生效邏輯說(shuō)明ingress_rate30單位是Mbps兆比特每秒不是MB/s。tc filter show是eBPF加載成功的鐵證。若tc命令不存在需yum install iproute-tc。5.3 流量鏡像鏡像到ens33但抓包收不到檢查ovs鏡像端口和iptables FORWARD鏈流量鏡像通過(guò)OVS的mirror功能實(shí)現(xiàn)將Pod流量復(fù)制到指定網(wǎng)卡。但若ovs-vsctl未創(chuàng)建mirror端口、或iptables FORWARD鏈默認(rèn)DROP鏡像包會(huì)被丟棄。# 查看OVS鏡像配置 sudo ovs-vsctl list mirror # 應(yīng)輸出_uuid : 0a1b2c3d-4e5f-6789-0a1b-2c3d4e5f6789 name : pod-mirror select_all : true output_port : ens33 # 若無(wú)輸出說(shuō)明kube-ovn-cni未創(chuàng)建mirror檢查Pod Annotation是否正確 # 正確Annotationovn.kubernetes.io/mirror: ens33 # 檢查ens33是否在FORWARD鏈中被允許 sudo iptables -L FORWARD -n | grep ens33 # 若無(wú)添加放行否則鏡像包被DROP sudo iptables -I FORWARD -i ens33 -j ACCEPT # 在ens33上抓包驗(yàn)證 sudo tcpdump -i ens33 -n -c 10 port 80 # 若收到HTTP包說(shuō)明鏡像成功關(guān)鍵參數(shù)ovn.kubernetes.io/mirrorAnnotation的值必須是OVS已知的端口名如ens33不能是eth0。tcpdump必須加-i ens33明確指定接口否則可能抓到其他流量。6. 如何快速定位Kube-OVN類問(wèn)題一份可抄作業(yè)的5分鐘故障樹(shù) 日志關(guān)鍵詞速查表從第一次部署Kube-OVN到現(xiàn)在我給自己立下一條鐵律任何Kube-OVN問(wèn)題5分鐘內(nèi)必須完成三級(jí)定位——Pod狀態(tài) → 組件日志 → 底層網(wǎng)絡(luò)狀態(tài)。這套方法論讓我在客戶現(xiàn)場(chǎng)平均3.2分鐘就能圈定根因不再陷入“全量日志grep”的泥潭。下面是我每天都在用的實(shí)戰(zhàn)清單你可以直接復(fù)制到終端執(zhí)行。6.1 5分鐘故障樹(shù)從現(xiàn)象到根因的決策路徑當(dāng)你遇到任意Kube-OVN問(wèn)題請(qǐng)嚴(yán)格按此順序執(zhí)行第一分鐘看Pod狀態(tài)kubectl get pod -n kube-ovn --sort-by.status.startTime | tail -10 # 重點(diǎn)關(guān)注ovs-ovnOVS狀態(tài)、kube-ovn-controller控制平面、vpc-nat-gwLB核心 # 若ovs-ovn CrashLoopBackOff → 跳轉(zhuǎn)至第2步若controller Pending → 檢查RBAC第二分鐘查關(guān)鍵組件日志# 取最新一個(gè)ovs-ovn Pod的日志過(guò)濾ERROR和connection kubectl logs -n kube-ovn $(kubectl get pod -n kube-ovn -l appovs-ovn -o jsonpath{.items[0].metadata.name}) -c ovs-ovn 21 | grep -i -E (error|fail|refuse|timeout) | head -10 # 取controller日志過(guò)濾你的Pod名如static-pod kubectl logs -n kube-ovn deploy/kube-ovn-controller --tail50 | grep -A3 -B3 static-pod第三分鐘驗(yàn)底層網(wǎng)絡(luò)狀態(tài)# 登錄對(duì)應(yīng)Node檢查OVS基礎(chǔ) sudo ovs-vsctl show | head -20 # 檢查ovn0橋是否存在且UP ip link show ovn0 2/dev/null | grep state UP # 檢查ovsdb-server socket sudo ls -l /var/run/openvswitch/db.sock 2/dev/null第四分鐘查配置一致性# 檢查所有硬編碼網(wǎng)卡名是否統(tǒng)一為ens33 kubectl get daemonset ovs-ovn -n kube-ovn -o yaml | grep -A2 -B2 OVN_BRIDGE_IFACE kubectl get deploy kube-ovn-controller -n kube-ovn -o yaml | grep -A2 -B2 iface kubectl get net-attach-def macvlan-conf -n kube-ovn -o jsonpath{.spec.config} | jq .master第五分鐘做最小化驗(yàn)證# 創(chuàng)建一個(gè)最簡(jiǎn)Pod測(cè)試基礎(chǔ)網(wǎng)絡(luò) kubectl run test-pod --imagebusybox -- sleep 3600 kubectl wait --forconditionReady pod/test-pod --timeout60s kubectl exec test-pod -- ping -c 3 10.16.0.1 # 測(cè)試到ovn0網(wǎng)關(guān)注意若第5步失敗問(wèn)題必在OVS/OVN底層若成功但你的業(yè)務(wù)Pod失敗則問(wèn)題在你的YAML配置如Annotation拼寫(xiě)、子網(wǎng)選擇。6.2 日志關(guān)鍵詞速查表看到這些詞立刻知道該查什么日志關(guān)鍵詞大小寫(xiě)不敏感出現(xiàn)場(chǎng)景立即執(zhí)行命令根因概率connection refusedovn-controller日志kubectl get svc -n kube-ovn ovn-nb95%ovn-nbService不可達(dá)no IP addresses availablekube-ovn-controller日志kubectl get subnet default -o yaml | grep -A5 usedAddresses90% 子網(wǎng)IP池耗盡failed to add port br-eth0ovs-ovn日志kubectl edit ds ovs-ovn -n kube-ovn100%OVN_BRIDGE_IFACE未更新bpf program load failedkube-ovn-controller日志sudo sysctl kernel.unprivileged_bpf_disabled085% 內(nèi)核禁用eBPFduplicate chassisovn-central日志sudo ovs-appctl -t /var/run/openvswitch/ovn-controller.pid chassis-list98% 多節(jié)點(diǎn)MAC地址重復(fù)6.3 我的血淚習(xí)慣每次部署Kube-OVN前必做的三件事從那以后我每次在新環(huán)境部署Kube-OVN都強(qiáng)制走一遍這三步哪怕客戶說(shuō)“時(shí)間緊不用檢查”——因?yàn)槭∠碌?0分鐘往往要花2小時(shí)去救火第一步ip link show截圖存檔在所有Node節(jié)點(diǎn)執(zhí)行ip link show /tmp/node-$(hostname)-ifconfig.txt存檔真實(shí)網(wǎng)卡名。絕不憑記憶寫(xiě)eth0。第二步ovs-vsctl show基線采集安裝前先運(yùn)行sudo ovs-vsctl show /tmp/ovs-before.txt安裝后再跑一次對(duì)比一眼看出br-int、ovn0是否創(chuàng)建成功。第三步kubectl ko subnet list快照kubectl ko subnet list /tmp/subnet-before.txt創(chuàng)建子網(wǎng)后再次快照用diff確認(rèn)usedAddresses增量是否符合預(yù)期。這三步加起來(lái)不到1分鐘卻能幫你避開(kāi)80%的“配置寫(xiě)對(duì)了但就是不生效”的玄學(xué)問(wèn)題。希望幫到你。本文還有配套的精品資源點(diǎn)擊獲取