Geek with the Peak Always learning · always teaching

Neutron：外部ネットワークへの router-gateway-set が失敗する

2015-11-24

Neutron：外部ネットワークへの router-gateway-set が失敗する

Machine-translated — the English original is authoritative.

本日、過去1ヶ月にわたって本製品を評価していたクライアントから、「動作しなくなった OpenStack ネットワーク」の調査を依頼されました。いつも通り、何も変更していないのに、突然動作しなくなりました 😉

このエンゲージメントにおけるもう一つの課題は、システムへの直接アクセス権がなかったことです。私は別の国にいて、リモートアクセスの許可も得られませんでした。そこで、詳細なコマンドリストを含む Google ドキュメントを作成し、OpenStack 運用担当者と共有しました。私たちはこれを協力と調査の基礎として使用しました。次の2つのブログ投稿は、基本的に生成された最終ドキュメントの内容です。

最初の苦情は、外部ネットワークが機能しないというものでした。

この失敗は以下のようなものでした：

root@overcloud-ce-controller-controller0-dlmy4f5tbc5d:~# neutron router-gateway-set 8c8a0159-2b37-4fae-93e9-b302a9d59573 35510045-decf-491e-9990-87a3f77f0284

Request Failed: internal server error while processing your request.

より詳細な出力を得るために “–debug” スイッチを追加することを忘れないでください。

root@overcloud-ce-controller-controller0-dlmy4f5tbc5d:~# neutron --debug router-gateway-set 8c8a0159-2b37-4fae-93e9-b302a9d59573 35510045-decf-491e-9990-87a3f77f0284

DEBUG: keystoneclient.session REQ: curl -i -X GET https://10.254.25.35:5000/v2.0 -H "Accept: application/json" -H "User-Agent: python-keystoneclient"

DEBUG: keystoneclient.session RESP: [200] content-length: 423 vary: X-Auth-Token keep-alive: timeout=5, max=100 server: Apache/2.4.10 (Debian) connection: Keep-Alive date: Fri, 20 Nov 2015 16:27:30 GMT content-type: application/json

RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}, {"base": "application/xml", "type": "application/vnd.openstack.identity-v2.0+xml"}], "id": "v2.0", "links": [{"href": "https://10.254.25.35:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}

DEBUG: neutronclient.neutron.v2_0.router.SetGatewayRouter run(Namespace(disable_snat=False, external_network_id=u'35510045-decf-491e-9990-87a3f77f0284', request_format='json', router_id=u'8c8a0159-2b37-4fae-93e9-b302a9d59573'))

DEBUG: keystoneclient.auth.identity.v2 Making authentication request to https://10.254.25.35:5000/v2.0/tokens

DEBUG: keystoneclient.session REQ: curl -i -X GET https://10.254.25.35:9696/v2.0/routers.json?fields=id&id=8c8a0159-2b37-4fae-93e9-b302a9d59573 -H "User-Agent: python-neutronclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}04e2fb3010f4069945373862d00e12d9e34c488a"

DEBUG: keystoneclient.session RESP: [200] date: Fri, 20 Nov 2015 16:27:31 GMT connection: keep-alive content-type: application/json; charset=UTF-8 content-length: 61 x-openstack-request-id: req-b6469ccc-b81e-4ae1-82ea-e0413bd46f3d

RESP BODY: {"routers": [{"id": "8c8a0159-2b37-4fae-93e9-b302a9d59573"}]}

DEBUG: keystoneclient.session REQ: curl -i -X GET https://10.254.25.35:9696/v2.0/networks.json?fields=id&id=35510045-decf-491e-9990-87a3f77f0284 -H "User-Agent: python-neutronclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}04e2fb3010f4069945373862d00e12d9e34c488a"

DEBUG: keystoneclient.session RESP: [200] date: Fri, 20 Nov 2015 16:27:31 GMT connection: keep-alive content-type: application/json; charset=UTF-8 content-length: 62 x-openstack-request-id: req-e39fba85-c775-4296-a52f-a83731a5b5f2

RESP BODY: {"networks": [{"id": "35510045-decf-491e-9990-87a3f77f0284"}]}

DEBUG: keystoneclient.session REQ: curl -i -X PUT https://10.254.25.35:9696/v2.0/routers/8c8a0159-2b37-4fae-93e9-b302a9d59573.json -H "User-Agent: python-neutronclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}04e2fb3010f4069945373862d00e12d9e34c488a" -d '{"router": {"external_gateway_info": {"network_id": "35510045-decf-491e-9990-87a3f77f0284"}}}'

DEBUG: keystoneclient.session RESP:

DEBUG: neutronclient.v2_0.client Error message: {"NeutronError": {"message": "Request Failed: internal server error while processing your request.", "type": "HTTPInternalServerError", "detail": ""}}

ERROR: neutronclient.shell Request Failed: internal server error while processing your request.

Traceback (most recent call last):

 File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutronclient/shell.py", line 701, in run_subcommand

   return run_command(cmd, cmd_parser, sub_argv)

 File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutronclient/shell.py", line 92, in run_command

   return cmd.run(known_args)

 File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutronclient/neutron/v2_0/router.py", line 205, in run

   neutron_client.add_gateway_router(_router_id, router_dict)

 File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 98, in with_params

   ret = self.function(instance, *args, **kwargs)

 File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 432, in add_gateway_router

   body={'router': {'external_gateway_info': body}})

 File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1322, in put

   headers=headers, params=params)

 File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1290, in retry_request

   headers=headers, params=params)

 File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1244, in do_request

   self._handle_fault_response(status_code, replybody)

 File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1218, in _handle_fault_response

   exception_handler_v20(status_code, des_error_body)

 File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 66, in exception_handler_v20

   status_code=status_code)

InternalServerError: Request Failed: internal server error while processing your request.

以下のように作成されたルーターのログファイルを確認します：

grep 8c8a0159-2b37-4fae-93e9-b302a9d59573 -l -R /var/log/*

root@overcloud-ce-controller-controller0-dlmy4f5tbc5d:~# grep 8c8a0159-2b37-4fae-93e9-b302a9d59573 -l -R /var/log/*

/var/log/horizon/horizon-json.log

/var/log/horizon/horizon_error.log

/var/log/neutron/neutron-json.log

/var/log/neutron/neutron.log

neutron.log ファイルの調査により、問題の根本原因が判明しました –

“TooManyExternalNetworks: More than one external network exists”

neutron.log

neutron で素早く確認すると、2つの外部ネットワークが存在していることがわかりました – これはこのバージョンの OpenStack ではサポートされていません。

root@overcloud-ce-controller-controller0-dlmy4f5tbc5d:~# neutron net-external-list

+--------------------------------------+---------+-----------------------------------------------------+

| id                                    | name  | subnets                                               |

+--------------------------------------+---------+-----------------------------------------------------+

| 35510045-decf-491e-9990-87a3f77f0284 | ext-net | bfaacf82-7a4f-4e0c-aab9-94f705dc2c11 10.254.27.0/24 |

| f917a3eb-957a-4c0a-baa8-8b20607440a0 | direct  |                                                       |

+--------------------------------------+---------+-----------------------------------------------------+

解決策はシンプルです – 2つ目の外部ネットワークを削除します。

「direct」という新しい外部ネットワークを削除します。

現在すべてのネットワークを一覧表示します

neutron net-list

root@overcloud-ce-controller-controller0-dlmy4f5tbc5d:~# neutron net-list

+--------------------------------------+-------------+-------------------------------------------------------+

| id                                    | name  | subnets                                                    |

+--------------------------------------+-------------+-------------------------------------------------------+

| 35510045-decf-491e-9990-87a3f77f0284 | ext-net | bfaacf82-7a4f-4e0c-aab9-94f705dc2c11 10.254.27.0/24   |

| 4259ab26-a1c4-427e-a7e0-b941368c4e23 | default-net | d20f33e0-82f1-4ce1-80cf-17a27c86d11c 172.0.100.0/24   |

| 73ff3b91-77ad-4ef4-aa67-a99a5ef57650 | demo_net1   | 4a20b788-da40-48a0-b459-5034d2831adf 192.168.111.0/24 |

| a493f23f-3646-49c5-a342-82c0ac7601f6 | lab2-net| 75542425-b437-4433-a9d8-3009527e1435 192.168.1.0/24   |

| d2ad4daa-d004-40bd-afa8-b7e19b8dde67 | HPinternal  | 28d80588-9bf1-415e-a1f3-b08d95d5bf14 10.0.0.0/24  |

| e70da3b0-ad8d-4ec0-9dd7-40ba563d2416 | SVC    | 57929850-9c6a-4917-a8a4-61c033d72705 10.254.26.0/24   |

| e99f2cdd-eb42-4466-9b17-7cb85636c013 | lab1-net1   | 60b9efc0-081f-45fd-b06c-0ce88fc04d50 192.168.113.0/24 |

| f917a3eb-957a-4c0a-baa8-8b20607440a0 | direct  |   |

+--------------------------------------+-------------+-------------------------------------------------------+

外部ネットワークを特定します

neutron net-external-list

root@overcloud-ce-controller-controller0-dlmy4f5tbc5d:~# neutron net-external-list

+--------------------------------------+---------+-----------------------------------------------------+

| id                                    | name      | subnets |

+--------------------------------------+---------+-----------------------------------------------------+

| 35510045-decf-491e-9990-87a3f77f0284 | ext-net | bfaacf82-7a4f-4e0c-aab9-94f705dc2c11 10.254.27.0/24 |

| f917a3eb-957a-4c0a-baa8-8b20607440a0 | direct  |                                                      |

+--------------------------------------+---------+-----------------------------------------------------+

追加の外部ネットワークを削除します

neutron net-delete direct

root@overcloud-ce-controller-controller0-dlmy4f5tbc5d:~# neutron net-delete direct

Deleted network: direct

root@overcloud-ce-controller-controller0-dlmy4f5tbc5d:~# neutron net-list

+--------------------------------------+-------------+-------------------------------------------------------+

| id                                    | name          | subnets   |

+--------------------------------------+-------------+-------------------------------------------------------+

| 35510045-decf-491e-9990-87a3f77f0284 | ext-net    | bfaacf82-7a4f-4e0c-aab9-94f705dc2c11 10.254.27.0/24   |

| 4259ab26-a1c4-427e-a7e0-b941368c4e23 | default-net | d20f33e0-82f1-4ce1-80cf-17a27c86d11c 172.0.100.0/24   |

| 73ff3b91-77ad-4ef4-aa67-a99a5ef57650 | demo_net1   | 4a20b788-da40-48a0-b459-5034d2831adf 192.168.111.0/24 |

| a493f23f-3646-49c5-a342-82c0ac7601f6 | lab2-net   | 75542425-b437-4433-a9d8-3009527e1435 192.168.1.0/24   |

| d2ad4daa-d004-40bd-afa8-b7e19b8dde67 | HPinternal  | 28d80588-9bf1-415e-a1f3-b08d95d5bf14 10.0.0.0/24  |

| e70da3b0-ad8d-4ec0-9dd7-40ba563d2416 | SVC        | 57929850-9c6a-4917-a8a4-61c033d72705 10.254.26.0/24   |

| e99f2cdd-eb42-4466-9b17-7cb85636c013 | lab1-net1   | 60b9efc0-081f-45fd-b06c-0ce88fc04d50 192.168.113.0/24 |

+--------------------------------------+-------------+-------------------------------------------------------+

次のブログ投稿では、Helion OpenStack の基本サービスと Neutron ネットワークの両方の健全性を検証します。

Originally published on allthingscloud.eu (2015-11-24).

← All posts